# Project export: CareBot

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: UC Berkeley AI Hackathon 2025
- Tagline: We built a personal voice-controlled robot caregiver in simulation! Our vision is to empower people with disabilities and the elderly by assisting with daily tasks—cooking, laundry, cleaning (demo).
- Devpost: https://devpost.com/software/carebot-u4qlrz
- GitHub: https://github.com/sohamkundu27/BerkeleyAIHackathon
- Video: https://www.youtube.com/embed/ek3yyC_W1UA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 3 GitHub contributor(s) — Eric Chuang (30 commits), SoKu27 (27 commits), Madhu Lok (1 commits)

## Devpost submission (written by the team)

### Inspiration

People with disabilities and the elderly can face significant challenges in performing everyday tasks independently. Many rely heavily on caregivers, which can be limiting and stressful. Plus, caregivers can be hard to find, as demand often far exceeds supply, leaving many without reliable support. There are countless stories of someone falling or having some other mishap, being stuck for hours on end with no way to call help. This inspired us to create a supportive robot friend that empowers users to regain independence through easy-to-use technology.

### What it does

CareBot provides voice-controlled robotic assistance in daily tasks such as cooking, laundry, and in our simulated demonstration, cleaning up. By integrating real-time voice transcription with Sonnet's intuitive capabilities and precise robot arm control, CareBot enables custom object manipulation without requiring specific key words or commands.

### How we built it

We combined cutting-edge voice transcription technology with a two-step hierarchal LLM pipeline to control a 7-DOF robot arm in a custom-built simulation environment. We used React Native for the speech-to-text UI, a prompt-engineered Sonnet analyzer and a Sonnet planner for our LLM pipeline, and a Python-based MCP (Model Context Protocol) server for robot commands. This culminates in an intuitive system that listens, understands, and executes tasks efficiently.

### Challenges we ran into

Managing precise robot movements while avoiding collisions with objects. Engineering strong prompts for our LLM analyzer and planner. Constructing heuristics for task completion (eg. move straight up before and after an object pick). Feeding the LLM the bare-minimum information required for determining gripper pose (position and orientation). Constructing heuristics for task completion (eg. move straight up before and after an object pick). Feeding the LLM the bare-minimum information required for determining gripper pose (position and orientation). Integrating a multi-step pipeline: speech-to-text, LLM, MCP, pybullet simulation.

### Accomplishments we're proud of

Developing a modular robot control API and an MCP server that allows easy integration of new robotic skills. Building a Sonnet 4 prompt-processing pipeline with strong in-built language information. Building a pybullet simulation environment with custom objects (it was our first time using pybullet, or any robot simulator for that matter!). Demonstrating real-time task execution with voice commands in a simulated 3D environment.

### What we learned

We gained valuable insights into robot motion planning, robot simulations, and LLM integration with an external API. The project highlighted the reasoning capabilities of state-of-the-art chatbots in a robotics context.

### What's next

Integrating computer vision API to dynamically obtain object positions before each pick. Deploying on physical robotic hardware to assist in real-world environments. Enhancing natural language understanding for greater task breadth. Attempting more complex tasks: tool use, cooking, folding laundry.

## README (from the GitHub repository)

Start the mobile app with-
npx expo start

Pipeline-
speak into phone
receive text input in flask

from flask send input to LLMPipeline

send output from LLMPipeline, which will be a list of commands to the MCP server, where it will be ran and control the Robot arm directly


## Detected evidence (automated analysis)

Indexed codebase: 60 recognized source files, 61 KB.
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (74 of 74)

```
.expo/devices.json
.expo/metro/externals/assert/index.js
.expo/metro/externals/async_hooks/index.js
.expo/metro/externals/buffer/index.js
.expo/metro/externals/child_process/index.js
.expo/metro/externals/cluster/index.js
.expo/metro/externals/console/index.js
.expo/metro/externals/constants/index.js
.expo/metro/externals/crypto/index.js
.expo/metro/externals/dgram/index.js
.expo/metro/externals/diagnostics_channel/index.js
.expo/metro/externals/dns/index.js
.expo/metro/externals/domain/index.js
.expo/metro/externals/events/index.js
.expo/metro/externals/fs/index.js
.expo/metro/externals/fs/promises/index.js
.expo/metro/externals/http/index.js
.expo/metro/externals/http2/index.js
.expo/metro/externals/https/index.js
.expo/metro/externals/inspector/index.js
.expo/metro/externals/module/index.js
.expo/metro/externals/net/index.js
.expo/metro/externals/node:sea/index.js
.expo/metro/externals/node:sqlite/index.js
.expo/metro/externals/node:test/index.js
.expo/metro/externals/os/index.js
.expo/metro/externals/path/index.js
.expo/metro/externals/perf_hooks/index.js
.expo/metro/externals/process/index.js
.expo/metro/externals/punycode/index.js
.expo/metro/externals/querystring/index.js
.expo/metro/externals/readline/index.js
.expo/metro/externals/repl/index.js
.expo/metro/externals/stream/index.js
.expo/metro/externals/string_decoder/index.js
.expo/metro/externals/timers/index.js
.expo/metro/externals/tls/index.js
.expo/metro/externals/trace_events/index.js
.expo/metro/externals/tty/index.js
.expo/metro/externals/url/index.js
.expo/metro/externals/util/index.js
.expo/metro/externals/v8/index.js
.expo/metro/externals/vm/index.js
.expo/metro/externals/wasi/index.js
.expo/metro/externals/worker_threads/index.js
.expo/metro/externals/zlib/index.js
.expo/metro/polyfill.js
.expo/metro/polyfill.native.js
.expo/README.md
.gitattributes
.gitignore
.python-version
App.js
app.json
app.py
audio_transcription.py
client.py
components/VoiceTranscription.js
LLMPipeline.py
main.py
objects/apple.urdf
objects/banana.urdf
objects/bottle.urdf
objects/box.urdf
objects/container.urdf
objects/hammer.urdf
objects/plate.urdf
package.json
pyproject.toml
README.md
robot_controller.py
server.py
static/index.html
test.py
```

### Dependencies

- package.json: @babel/core@^7.20.0, @expo/metro-runtime@~5.0.4, @urql/core@^5.1.1, expo@^53.0.0, expo-av@~15.1.6, expo-file-system@~18.1.10, expo-status-bar@~2.2.3, react@19.0.0, react-dom@19.0.0, react-native@0.79.4, react-native-safe-area-context@5.4.0, react-native-screens@~4.11.1, react-native-web@^0.20.0
- pyproject.toml: mcp@>=1.9.4

### Recent commits (newest first)

- stash
- camera angle
- small changes to LLM pipeline
- small changes to LLM pipeline
- Merge branch 'main' of github.com:sohamkundu27/BerkeleyAIHackathon
- python version
- untrack node modules
- updated LLMPipeline
- Updated
- fix
- Merge branch 'main' of https://github.com/sohamkundu27/BerkeleyAIHackathon into main
- app and LLMPipeline
- object location
- Merge branch 'main' of github.com:sohamkundu27/BerkeleyAIHackathon
- improve motion
- full integration verification
- Merge branch 'main' of github.com:sohamkundu27/BerkeleyAIHackathon
- simulation
- Merge pull request #9 from sohamkundu27/full-integration-complete
- full integration

## Key source files (fetched from GitHub, selected and truncated for size)

### pyproject.toml

```
[project]
name = "berkeleyaihackathon"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
    "mcp>=1.9.4",
]

```

### package.json

```
{
  "name": "voice-transcription-mobile",
  "version": "1.0.0",
  "main": "node_modules/expo/AppEntry.js",
  "scripts": {
    "start": "expo start",
    "android": "expo start --android",
    "ios": "expo start --ios",
    "web": "expo start --web"
  },
  "dependencies": {
    "@expo/metro-runtime": "~5.0.4",
    "@urql/core": "^5.1.1",
    "expo": "^53.0.0",
    "expo-av": "~15.1.6",
    "expo-file-system": "~18.1.10",
    "expo-status-bar": "~2.2.3",
    "react": "19.0.0",
    "react-dom": "19.0.0",
    "react-native": "0.79.4",
    "react-native-safe-area-context": "5.4.0",
    "react-native-screens": "~4.11.1",
    "react-native-web": "^0.20.0"
  },
  "devDependencies": {
    "@babel/core": "^7.20.0"
  },
  "private": true
}

```

### main.py

```python
def main():
    print("Hello from berkeleyaihackathon!")


if __name__ == "__main__":
    main()

```

### App.js

```javascript
import React from 'react';
import { StatusBar } from 'expo-status-bar';
import { SafeAreaView, StyleSheet } from 'react-native';
import VoiceTranscription from './components/VoiceTranscription';

export default function App() {
  return (
    <SafeAreaView style={styles.container}>
      <StatusBar style="light" />
      <VoiceTranscription />
    </SafeAreaView>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    backgroundColor: '#667eea',
  },
}); 
```

### server.py

```python
# server.py
# getting inspector run use the following commang:
# npx @modelcontextprotocol/inspector \
#   uv \
#   run \
#   server.py
# Lastly click on the url with the authenticaiton key already on there

from mcp.server.fastmcp import FastMCP
from robot_controller import RobotSim
import os
os.environ["DISPLAY"] = ":0"

# Create an MCP server
mcp = FastMCP("Demo")
sim = RobotSim()


# Add an addition tool
@mcp.tool()
def add(a: int, b: int) -> int:
    """Add two numbers"""
    return a + b


@mcp.tool()
def move_arm(target, target_orn=None):
    sim.move_arm(target, target_orn)
    return f"Arm moved to {target} with orientation {target_orn}"


@mcp.tool()
def open_gripper():
    sim.open_gripper()
    return "Gripper opened"


@mcp.tool()
def close_gripper():
    sim.close_gripper()
    return "Gripper closed"


# Add a dynamic greeting resource
@mcp.resource("greeting://{name}")
def get_greeting(name: str) -> str:
    """Get a personalized greeting"""
    return f"Hello, {name}!"


if __name__ == '__main__':
    mcp.run(transport='stdio')

```

### app.py

```python
from flask import Flask, render_template, request, jsonify
from flask_cors import CORS
import os
import tempfile
import sounddevice as sd
import numpy as np
import scipy.io.wavfile
import io
import warnings
import threading
import time
import asyncio
from datetime import datetime
from LLMPipeline import get_response
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

data = [{
    "Object": "apple",
    "x": "short",
    "y": "short",
    "location": [0.82, -0.3, 0.6849899910813102]
},
    {
    "Object": "bottle",
    "location": [0.7, 0.1, 0.8],
    "x": "short",
    "y": "short"
},
    {
    "Object": "box",
    "location": [1, 0.1, 0.7],
    "x": "long",
    "y": "short"
},
    {
    "Object": "banana",
    "location": [0.893, 0.313, 0.660],
    "x": "long",
    "y": "short"
},
    {
    "Object": "container",
    "location": [0.9, -0.75, 0.73],
    "x": "long",
    "y": "long"
},
    {
    "Object": "hammer",
    "location": [1, -0.2, 0.7],
    "x": "short",
    "y": "long"
}]
# Suppress the FP16 warning
warnings.filterwarnings(
    "ignore", message="FP16 is not supported on CPU; using FP32 instead")

app = Flask(__name__)
CORS(app)  # Enable CORS for all routes

# Global variables for recording
recording_thread = None
is_recording = False
audio_chunks = []
fs = 16000


def log_message(message):
    """Helper function to log messages with timestamp"""
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    print(f"[{timestamp}] {message}")


def record_audio():
    """Record audio in a separate thread"""
    global audio_chunks, is_recording
    audio_chunks = []
    log_message("🎤 Starting audio recording thread")

    def callback(indata, frames, time, status):
        if is_recording:
            audio_chunks.append(indata.copy())
            if len(audio_chunks) % 10 == 0:  # Log every 10 chunks
                log_message(
                    f"📊 Recorded {len(audio_chunks)} audio chunks so far")

    with sd.InputStream(callback=callback, channels=1, samplerate=fs):
        while is_recording:
            time.sleep(0.1)


def transcribe_audio(audio_data):
    """Transcribe audio data"""
    try:
        log_message("🔍 Starting audio transcription")
        # Import whisper here to avoid circular import
        import whisper
        model = whisper.load_model("base")

        # Combine all audio chunks
        if audio_chunks:
            log_message(f"🎵 Processing {len(audio_chunks)} audio chunks")
            audio = np.concatenate(audio_chunks, axis=0).flatten()
            log_message(
                f"📏 Audio length: {len(audio)} samples ({len(audio)/fs:.2f} seconds)")

            # Save temp WAV and transcribe
            with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:
                scipy.io.wavfile.write(f.name, fs, audio.astype(np.float32))
                log_message("🎯 Running Whisper transcription...")
                result = model.transcribe(f.name)
                os.unlink(f.name)  # Clean up temp file

            log_message(f"✅ Transcription completed: '{result['text']}'")
            return result["text"]
        else:
            log_message("⚠️ No audio chunks to transcribe")
            return ""

    except Exception as e:
        log_message(f"❌ Error in transcribe_audio: {e}")
        return ""


def transcribe_uploaded_file(audio_file):
    """Transcribe uploaded audio file"""
    try:
        log_message(f"📁 Processing uploaded file: {audio_file.filename}")
        log_message(f"📋 File content type: {audio_file.content_type}")
        log_message(f"📋 File size: {len(audio_file.read())} bytes")
        audio_file.seek(0)  # Reset file pointer after reading

        # Import whisper here to avoid circular import
        import whisper
        model = whisper.load_model("base")

        # Save uploaded file temporarily
        with tempfile.NamedTemporaryFile(suffix=".m4a", delete=False) as f:
            audio_file.save(f.name)
            log_message(f"💾 Saved temporary file: {f.name}")

            try:
                log_message(
                    "🎯 Running Whisper transcription on uploaded file...")
                result = model.transcribe(f.name)
                log_message(
                    f"✅ Upload transcription completed: '{result['text']}'")
                return result["text"]
            except Exception as transcribe_error:
                log_message(
                    f"❌ Whisper transcription failed: {transcribe_error}")
                # Try with different file extension if m4a fails
                import shutil
                wav_file = f.name.replace('.m4a', '.wav')
                shutil.copy2(f.name, wav_file)
                log_message(f"🔄 Trying with WAV format: {wav_file}")
                try:
                    result = model.transcribe(wav_file)
                    log_message(
                        f"✅ WAV transcription completed: '{result['text']}'")
                    os.unlink(wav_file)  # Clean up WAV file
                    return result["text"]
                except Exception as wav_error:
                    log_message(
                        f"❌ WAV transcription also failed: {wav_error}")
                    os.unlink(wav_file)  # Clean up WAV file
                    raise transcribe_error
            finally:
                os.unlink(f.name)  # Clean up temp file

    except Exception as e:
        log_message(f"❌ Error in transcribe_uploaded_file: {e}")
        return f"Transcription failed: {str(e)}"


@app.route('/')
def index():
    log_message("🌐 Homepage requested")
    return app.send_static_file('index.html')


@app.route('/start_recording', methods=['POST'])
def start_recording():
    global recording_thread, is_recording

    try:
        log_message("▶️ Start recording request received")
        is_recording = True
        recording_thread = threading.Thread(target=record_audio)
        re
[truncated — 5821 more characters]
```

### .expo/metro/externals/fs/index.js

```javascript
module.exports = $$require_external('node:fs');
```

### .expo/metro/externals/os/index.js

```javascript
module.exports = $$require_external('node:os');
```

### .expo/metro/externals/v8/index.js

```javascript
module.exports = $$require_external('node:v8');
```

### .expo/metro/externals/vm/index.js

```javascript
module.exports = $$require_external('node:vm');
```

[50 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]