Project Info

CareBot

Inspiration

People with disabilities and the elderly can face significant challenges in performing everyday tasks independently. Many rely heavily on caregivers, which can be limiting and stressful. Plus, caregivers can be hard to find, as demand often far exceeds supply, leaving many without reliable support. There are countless stories of someone falling or having some other mishap, being stuck for hours on end with no way to call help. This inspired us to create a supportive robot friend that empowers users to regain independence through easy-to-use technology.

What it does

CareBot provides voice-controlled robotic assistance in daily tasks such as cooking, laundry, and in our simulated demonstration, cleaning up. By integrating real-time voice transcription with Sonnet's intuitive capabilities and precise robot arm control, CareBot enables custom object manipulation without requiring specific key words or commands.

How we built it

We combined cutting-edge voice transcription technology with a two-step hierarchal LLM pipeline to control a 7-DOF robot arm in a custom-built simulation environment. We used React Native for the speech-to-text UI, a prompt-engineered Sonnet analyzer and a Sonnet planner for our LLM pipeline, and a Python-based MCP (Model Context Protocol) server for robot commands. This culminates in an intuitive system that listens, understands, and executes tasks efficiently.

Challenges we ran into

Managing precise robot movements while avoiding collisions with objects. Engineering strong prompts for our LLM analyzer and planner. Constructing heuristics for task completion (eg. move straight up before and after an object pick). Feeding the LLM the bare-minimum information required for determining gripper pose (position and orientation). Constructing heuristics for task completion (eg. move straight up before and after an object pick). Feeding the LLM the bare-minimum information required for determining gripper pose (position and orientation). Integrating a multi-step pipeline: speech-to-text, LLM, MCP, pybullet simulation.

Accomplishments we're proud of

Developing a modular robot control API and an MCP server that allows easy integration of new robotic skills. Building a Sonnet 4 prompt-processing pipeline with strong in-built language information. Building a pybullet simulation environment with custom objects (it was our first time using pybullet, or any robot simulator for that matter!). Demonstrating real-time task execution with voice commands in a simulated 3D environment.

What we learned

We gained valuable insights into robot motion planning, robot simulations, and LLM integration with an external API. The project highlighted the reasoning capabilities of state-of-the-art chatbots in a robotics context.

What's next

Integrating computer vision API to dynamically obtain object positions before each pick. Deploying on physical robotic hardware to assist in real-world environments. Enhancing natural language understanding for greater task breadth. Attempting more complex tasks: tool use, cooking, folding laundry.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

HTMLIn code
JavaScriptIn code
PythonIn code
ReactIn code
FlaskClaimed

4 of 5 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

61 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.