# Project export: Roomba Nurse
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2025
- Tagline: A personal bedside attendant for every patient
- Devpost: https://devpost.com/software/roomba-nurse
- GitHub: https://github.com/KevinWu098/roomba-ai
- Video: https://www.youtube.com/embed/_tW5jkEJEZE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 4 GitHub contributor(s) — Kevin Wu (27 commits), Justin Wu (16 commits), Jaslavie (11 commits), Sophia Zhang (10 commits)
## Devpost submission (written by the team)
### Overview
Nurses should be saving lives, not running errands. Yet, nearly 40% of their 12-hour shifts are wasted on manual tasks like retrieving food, water, and supplies—time that should be spent on patient care. Nurses spend over 50% of this time diluting attention to delivery tasks rather than patient care. This causes delayed care and means nurses are unable to fulfill their duties. TL;DR Huge understaffing of nurses. There is 1 nurse for every 5 patients which is considered unsafe Hospital robots can deliver items but lack interactivity. Patients have little agency over their needs and must wait for nurses to come We built an autonomous, emotionally intelligent bedside assistant that delivers food, water, and essentials instantly. Patients request items by voice or touchscreen, and the robot navigates hallways for hands-free, nurse-free delivery. Conversational AI interprets needs in real-time, proactively suggesting items (e.g., “Would you like soup?” when a patient says they’re hungry). An expressive screen shows emotions and action cues, making interaction intuitive and engaging. Fully local autonomous navigation from the patient's bed to the nurse and vice versa using facial recognition deployed locally. Concept We built an emotionally intelligent bedside assistant—a roomba robot that delivers food, water, and essential items to patients autonomously, reducing nurse workload and ensuring that patient needs are met instantly. Unlike existing hospital delivery robots that simply drop off items, our system interacts with patients in real-time, understands requests through conversation, and proactively offers recommendations (e.g., “Would you like some soup?” when a patient says they’re hungry).
### How we built it
We hacked into a Roomba 692 using a serial port connection and designed a custom-built robot body from foam and vinyl to create an approachable, bedside-friendly form factor. A mounted iPad serves as the interactive face and interface, dynamically changing expressions and responding to patient needs. The system integrates: Speech-to-Text with NLP – To interpret patient requests in natural conversation. Embedded Navigation – autonomous movement and identify patients/nurses through on-device facial recognition and computer vision on a Raspberry Pi Custom Emotion Engine – Animated expressions displayed based on the emotional context of the conversation. Roomba Nurse essentially responds to emotion.
### Challenges we ran into
Hacking into a Roomba’s hardware and repurposing it for autonomous hospital navigation required low-level serial communication and precise motor control—something not originally designed for this use case. Developing a real-time conversation system that could both understand patient intent and generate appropriate responses was another major hurdle. Additionally, ensuring the robot’s interface felt natural and engaging while remaining functional in a medical setting demanded thoughtful UX design and emotional AI tuning. The most challenging part of the project was making fully autonomous navigation with facial recognition work on the edge. We optimized and deployed a facial-recognition + CV powered navigation system onto a Raspberry Pi, so that all navigation runs locally, which took a lot of experimentation and optimization!
### Accomplishments we're proud of
We successfully hacked into a Roomba to create an expressive, intelligent bedside assistant, capable of navigating hospital spaces, engaging in conversation, and autonomously delivering patient needs. Our system integrates speech-to-text, natural language processing, and facial expressions, making it one of the first emotionally aware hospital robots.
### What we learned
We gained a deep understanding of robotic hardware hacking, real-time AI interaction design, and the challenges of building assistive technology in a high-stakes environment like healthcare. We also learned how important expressiveness and human-like interaction are for making robots feel intuitive and trustworthy, especially for vulnerable patients.
### What's next
We envision a Roomba Nurse at every patient's bedside. We envision a Roomba Nurse for every hospital room, providing seamless, hands-free assistance for both patients and nurses. Future iterations will include enhanced navigation, a broader range of item retrieval, deeper AI-driven patient engagement, and EHR integration for more personalized recommendations. Our goal is to make Roomba Nurse an essential part of every hospital’s care team, reducing nurse burden and improving patient experience at scale.
## README (from the GitHub repository)
## roomba-ai
### Datasets used
1. [Nurses Stress Prediction](https://www.kaggle.com/code/guanlintao/eda-pca-ml-nurse-stress-prediction/output)
## Detected evidence (automated analysis)
Indexed codebase: 65 recognized source files, 384 KB.
- CSS (language) — detected in the code
- Next.js (technology) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Vercel AI SDK (technology) — detected in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (84 of 84)
```
.gitignore
client/.env.example
client/.eslintrc.json
client/.gitignore
client/.prettierignore
client/components.json
client/next.config.mjs
client/package.json
client/postcss.config.mjs
client/prettier.config.cjs
client/src/actions/handle-token.ts
client/src/app/api/chat/route.ts
client/src/app/globals.css
client/src/app/layout.tsx
client/src/app/nurse/page.tsx
client/src/app/page.tsx
client/src/app/patient/page.tsx
client/src/components/faces/faces.tsx
client/src/components/generative/items.tsx
client/src/components/generative/tools.ts
client/src/components/generative/weather.tsx
client/src/components/livekit-room.tsx
client/src/components/lk-chat/ChatMessage.tsx
client/src/components/lk-chat/ChatMessageInput.tsx
client/src/components/lk-chat/ChatTile.tsx
client/src/components/lk-cloud/CloudConnect.tsx
client/src/components/lk-cloud/useCloud.tsx
client/src/components/lk-color-picker/color-picker.tsx
client/src/components/lk-config/AudioInputTile.tsx
client/src/components/lk-config/ConfigurationPanelItem.tsx
client/src/components/lk-config/device-selector.tsx
client/src/components/lk-config/NameValueRow.tsx
client/src/components/lk-playground/playground.tsx
client/src/components/lk-playground/PlaygroundHeader.tsx
client/src/components/lk-playground/PlaygroundTile.tsx
client/src/components/lk-playground/settings-tile.tsx
client/src/components/lk-playground/SettingsDropdown.tsx
client/src/components/lk-toast/PlaygroundToast.tsx
client/src/components/lk-toast/Toast.tsx
client/src/components/lk-toast/ToasterProvider.tsx
client/src/components/lk-transcription/TranscriptionTile.tsx
client/src/components/ui/button.tsx
client/src/components/ui/dropdown-menu.tsx
client/src/hooks/useConfig.tsx
client/src/hooks/useConnection.tsx
client/src/hooks/useTrackVolume.tsx
client/src/hooks/useWindowResize.ts
client/src/lib/tailwindTheme.preval.ts
client/src/lib/themeColors.ts
client/src/lib/types.ts
client/src/lib/utils.ts
client/tailwind.config.ts
client/tsconfig.json
db/chroma.sqlite3
livekit/.env.example
livekit/.gitignore
livekit/agent.py
livekit/LICENSE
livekit/README.md
livekit/requirements.txt
rag/chromaDB/chroma_config.json
rag/chromaDB/main.py
rag/chromaDB/scripts/actions.py
rag/chromaDB/scripts/client.py
README.md
roomba-nav/create_embeddings.py
roomba-nav/drive/calibrate.py
roomba-nav/drive/obj_detect.py
roomba-nav/drive/RoombaController.py
roomba-nav/face_recog_opencv/FaceRecognizer.py
roomba-nav/face_recog_opencv/find_camera.py
roomba-nav/face_recog_opencv/get_photos.py
roomba-nav/face_recog_opencv/test_face_rec.py
roomba-nav/face_recog_opencv/test_recog_drive.py
roomba-nav/face_recog_opencv/test_show_cam_feed.py
roomba-nav/face-recog-tf/facenet.tflite
roomba-nav/face-recog-tf/FaceRecognizer.py
roomba-nav/face-recog-tf/recog.py
roomba-nav/face-recog-tf/TFLiteFaceDetector.py
roomba-nav/face-recog-tf/ultraface.tflite
roomba-nav/find_camera_2.py
roomba-nav/known_embeddings.npy
roomba-nav/nav.py
roomba-nav/websocket/server.py
```
### Dependencies
- client/package.json: @ai-sdk/openai@^1.1.11, @ai-sdk/react@^1.1.16, @eslint/js@^9.17.0, @ianvs/prettier-plugin-sort-imports@^4.4.0, @livekit/components-react@^2.8.1, @livekit/components-styles@^1.1.4, @radix-ui/react-dropdown-menu@^2.1.6, @radix-ui/react-slot@^1.1.2, @types/js-yaml@^4.0.9, @types/lodash@^4.17.15, @types/node@^20, @types/react@^18, @types/react-dom@^18, ai@^4.1.41, class-variance-authority@^0.7.1, clsx@^2.1.1, cookies-next@^5.1.0, eslint@^8.57.1, eslint-config-next@14.2.16, geist@^1.3.1, js-yaml@^4.1.0, livekit-client@^2.9.1, livekit-server-sdk@^2.10.1, lodash@^4.17.21, lucide@^0.475.0, lucide-react@^0.468.0, motion@^11.15.0, next@14.2.16, next-plugin-preval@^1.2.7, postcss@^8, prettier-plugin-tailwindcss@^0.6.9, react@^18, react-dom@^18, tailwind-merge@^2.5.5, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5.7.2, typescript-eslint@^8.18.1, zod@^3.24.2
- livekit/requirements.txt: livekit-agents@>=0.12.11, livekit-plugins-cartesia@>=0.4.7, livekit-plugins-deepgram@>=0.6.17, livekit-plugins-openai@>=0.10.17, livekit-plugins-silero@>=0.7.4, livekit-plugins-turn-detector@>=0.4.0, python-dotenv@~=1.0
### Recent commits (newest first)
- sdf
- foo
- asdf
- Merge branch 'main' of github.com:KevinWu098/roomba-ai
- sdf
- feat: whatever
- a
- a
- more tweaks
- bug in rotate
- create embeddings, small fix to rotation
- tweak second rotation
- full demo without websockets
- imshow comment out
- minor tweaks
- update pi serial port
- fix
- saved embeddings
- face recog + nav
- obj detect
## Key source files (fetched from GitHub, selected and truncated for size)
### livekit/requirements.txt
```
livekit-agents>=0.12.11
livekit-plugins-openai>=0.10.17
livekit-plugins-cartesia>=0.4.7
livekit-plugins-deepgram>=0.6.17
livekit-plugins-silero>=0.7.4
livekit-plugins-turn-detector>=0.4.0
python-dotenv~=1.0
```
### client/package.json
```
{
"name": "site",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"lint": "next lint"
},
"dependencies": {
"@ai-sdk/openai": "^1.1.11",
"@ai-sdk/react": "^1.1.16",
"@livekit/components-react": "^2.8.1",
"@livekit/components-styles": "^1.1.4",
"@radix-ui/react-dropdown-menu": "^2.1.6",
"@radix-ui/react-slot": "^1.1.2",
"ai": "^4.1.41",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"cookies-next": "^5.1.0",
"geist": "^1.3.1",
"js-yaml": "^4.1.0",
"livekit-client": "^2.9.1",
"livekit-server-sdk": "^2.10.1",
"lodash": "^4.17.21",
"lucide": "^0.475.0",
"lucide-react": "^0.468.0",
"motion": "^11.15.0",
"next": "14.2.16",
"next-plugin-preval": "^1.2.7",
"react": "^18",
"react-dom": "^18",
"tailwind-merge": "^2.5.5",
"tailwindcss-animate": "^1.0.7",
"zod": "^3.24.2"
},
"devDependencies": {
"@eslint/js": "^9.17.0",
"@ianvs/prettier-plugin-sort-imports": "^4.4.0",
"@types/js-yaml": "^4.0.9",
"@types/lodash": "^4.17.15",
"@types/node": "^20",
"@types/react": "^18",
"@types/react-dom": "^18",
"eslint": "^8.57.1",
"eslint-config-next": "14.2.16",
"postcss": "^8",
"prettier-plugin-tailwindcss": "^0.6.9",
"tailwindcss": "^3.4.1",
"typescript": "^5.7.2",
"typescript-eslint": "^8.18.1"
}
}
```
### rag/chromaDB/main.py
```python
# main file to run scripts
import sys
import os
# Get the absolute path of the project root
sys.path.append(os.path.abspath(os.path.dirname(__file__)))
from scripts.actions import main_loop
if __name__ == "__main__":
main_loop()
```
### roomba-nav/websocket/server.py
```python
from typing import Dict, Optional
from fastapi.encoders import jsonable_encoder
import asyncio
from fastapi import FastAPI, WebSocket, WebSocketDisconnect, File, UploadFile
from fastapi.middleware.cors import CORSMiddleware
origins = ["*"]
app = FastAPI()
class ConnectionManager:
def __init__(self):
self.active_connections: Dict[str, WebSocket] = {}
async def connect(self, websocket: WebSocket, client_id: str):
await websocket.accept()
self.active_connections[client_id] = websocket
async def disconnect(self, client_id: str):
try:
del self.active_connections[client_id]
except KeyError:
pass
async def send_message(self, data: dict, websocket: WebSocket):
await websocket.send_json(data)
async def send_roomba(self):
await self.active_connections.get("roomba").send_json({"event": ""})
manager = ConnectionManager()
app.add_middleware(
CORSMiddleware,
allow_origins=origins,
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@app.websocket("/ws/getItem")
async def get_item_endpoint(websocket: WebSocket):
await manager.connect(websocket, 'frontend')
try:
while True:
data = await websocket.receive_json()
print("Received item request:", data)
print('active connections', manager.active_connections)
# await manager.send_message(jsonable_encoder("foobar"), manager.active_connections.get("agent"))
except WebSocketDisconnect:
print("Client disconnected from /ws/getItem")
@app.websocket("/ws/register")
async def register(websocket: WebSocket, client_id: Optional[str] = None):
await manager.connect(websocket, client_id)
@app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket, client_id: Optional[str] = None):
await manager.connect(websocket, client_id)
try:
while True:
data = await websocket.receive_json()
event = data["event"]
if event == "get_requests":
requests = [
{
"id": 123,
"title": "Request for Blankets",
"name": "12345",
"time": 1739656070554,
"resolved": False,
}
]
serialized_requests = jsonable_encoder(requests)
message = {
"event": "request_response",
"data": serialized_requests,
}
await manager.send_message(message, websocket)
# Start periodic updates every 5 seconds
while True:
await asyncio.sleep(5)
requests = [
{
"id": 123,
"title": "Request for Blankets",
"name": "12345",
"time": 1739656070554,
"resolved": False,
}
]
serialized_requests = jsonable_encoder(requests)
message = {
"event": "request_response",
"data": serialized_requests,
}
await manager.send_message(message, websocket)
elif event == "send_roomba":
await manager.send_roomba()
except WebSocketDisconnect:
print("Disconnecting...", client_id)
await manager.disconnect(client_id)
except Exception as e:
print(f"Error at line {e.__traceback__.tb_lineno}:", str(e))
await manager.disconnect(client_id)
```
### client/src/app/page.tsx
```typescript
import { LivekitRoom } from "@/components/livekit-room";
import { Toast } from "@/components/lk-toast/Toast";
export default function Home() {
return (
);
}
```
### client/src/app/layout.tsx
```typescript
import "@livekit/components-styles/components/participant";
import type { Metadata } from "next";
import "./globals.css";
import { Poppins } from "next/font/google";
import { ToastProvider } from "@/components/lk-toast/ToasterProvider";
import { ConfigProvider } from "@/hooks/useConfig";
import { ConnectionProvider } from "@/hooks/useConnection";
import { cn } from "@/lib/utils";
const poppins = Poppins({
subsets: ["latin"],
weight: ["300", "400", "500", "600", "700"],
variable: "--font-poppins",
});
export const metadata: Metadata = {};
export default function RootLayout({
children,
}: Readonly<{
children: React.ReactNode;
}>) {
return (
{children}
);
}
```
### client/src/app/patient/page.tsx
```typescript
import { LivekitRoom } from "@/components/livekit-room";
export default function Page() {
return (