# Project export: ICU This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Extra eyes for those who need them the most. With hand gestures or voice commands, ICU's servo-powered camera rotates, analyzes the environment, and describes surroundings for the visually impaired. - Devpost: https://devpost.com/software/icu-2egdxy - GitHub: https://github.com/vyomfadia/berkeley - Video: https://www.youtube.com/embed/playlist?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Cal Hacks: Best Hardware Hack; Vapi: Best Hardware & Voice Integration) - Team: 4 GitHub contributor(s) — Vyom Fadia (22 commits), jayminjhaveri10@gmail.com (3 commits), Sam Mathew (1 commits), Katie (1 commits) ## Devpost submission (written by the team) ### Inspiration Katie's grandfather has been battling diabetic retinopathy (vision loss due to diabetes) for the past 5 years. He's been struggling more and more with everyday tasks like finding his medication, identifying what's on the dinner table, or checking if he left the stove on. He's constantly asking how new tech can help him stay independent of his caregivers -- why not make it a reality? #fathersday Her grandfather expressed that existing solutions for visually impaired people remain too focused on text-to-speech instead of environmental awareness. This isn't very helpful. This capability gap forces many to choose between constant dependence on others or navigating with incomplete information about their surroundings. ICU addresses this challenge directly by combining voice commands with gesture controls to provide real-time environmental context without requiring technical expertise. It's not just her grandfather who needs this; vision impairment affects approximately 2.2 billion people worldwide. ### What it does ICU acts as an intelligent pair of eyes, allowing users to request specific insights about their surroundings using natural voice or hand commands. The system dynamically adjusts a camera based on voice input or the direction a user points toward, streaming visuals to advanced AI models (Gemini and local Mediapipe landmark extraction) to analyze the environment and give precise, contextual feedback. It also speaks back to the user as if it's a human assistant, making it more natural to use. This approach enables visually impaired individuals, like elderly users with age-related vision decline, to independently identify objects, verify safety conditions, and maintain awareness of their surroundings. Extending beyond just convenience, ICU's functions preserve dignity, reduce dependence, and enabled continued participation in daily activities that most take for granted. ### How we built it We integrated a voice agent interface for two-way natural conversations and commands. Cameras are strategically placed: one dynamically moves based on voice instructions interpreted by OpenAI Whisper, and another is fixed on the user, extracting hand landmarks using Mediapipe. Our local ML model runs efficiently on laptops, processing video streams sent from phones. Data and signals are exchanged seamlessly via WebSockets between microservices that manage analysis, logic, and user interactions. AI-driven inference is handled by Gemini, which receives visual streams to deliver insights. ### Challenges we ran into Managing real-time streaming with low latency was initially tough, especially maintaining smooth WebSocket communication between multiple devices and services. The ML pipeline was another challenge. We had Mediapipe running for hand tracking and Gemini for visual analysis, and we had to make them work harmoniously. The most surprising challenge came with the hand tracking. Translating a pointing gesture into actual camera movement directions required accurate inverse kinematics to translate hand landmarks into meaningful directional data. We spent hours fine-tuning our inverse kinematics algorithms because the initial versions would misinterpret slight hand tremors (common in elderly users) as intentional movements. We also wanted to make this project different from other visually-impaired assistance projects, which are quite common but often not very practical. It was initially a struggle to ideate beyond basic screen readers or OCR tools. We knew we wanted something that could actually describe what's happening in a room, and that required solving a completely different set of technical problems that few hackathon projects attempt to tackle. ### Accomplishments we're proud of Successfully integrating multiple complex components (real-time voice commands, dynamic camera movements, and ML-driven analysis) into a cohesive, user-friendly solution was incredibly rewarding. We also achieved near-real-time responsiveness, ensuring the technology could provide practical assistance without noticeable lag. The moment our first end-to-end test worked was super rewarding, and we immediately sent progress videos to Katie's grandfather, whose excitement seeing the prototype validated our approach. We're now excited for Katie's grandfather to test the system in person next month. ### What we learned We learned valuable lessons in real-time data streaming, model integration, and handling asynchronous tasks efficiently. Working with WebSockets taught us how to maintain persistent connections across devices while minimizing latency, which is critical for a responsive assistive technology. We discovered that implementing buffer management and proper error handling significantly improved reliability when network conditions fluctuated. The challenge of coordinating multiple ML models forced us to develop a more sophisticated understanding of resource allocation and parallel processing workflows. Our team strengthened our skills in integrating cloud-based AI models with local ML processing, finding the right balance between offloading complex inference tasks to more powerful cloud services while keeping latency-sensitive operations like gesture tracking running locally. Finally, we gained practical experience in designing intuitive user interactions for individuals with visual impairments. We learned that responsiveness and consistent feedback were even more crucial than technical sophistication. ### What's next Looking ahead, we plan to miniaturize and optimize the system for portability, aiming to run entirely on mobile devices or compact wearables. We're specifically exploring the Raspberry Pi Zero 2 W paired with a lightweight camera module that could reduce the current setup's footprint by 70% while maintaining core functionality. Enhanced ML models for better accuracy and expanded environmental context are also on our roadmap, particularly focusing on improved object recognition in low-light conditions, a common challenge identified during our testing. We're developing a more comprehensive ontology of household objects with specialized detection for medication bottles, appliance states, and potential hazards. Next month, we'll conduct our first extended user test when Katie's grandfather tries the system in his home. We are looking for his feedback regarding voice command natural language processing and the sensitivity of gesture controls. We're also exploring integration with smart home systems to enable ICU to not just identify but interact with the environment (e.g., "turn off the stove" after detecting it was left on). Ultimately, we envision ICU becoming an essential tool for those needing intuitive, accessible, and immediate assistance, with a particular focus on creating affordable deployment options for individuals with limited financial resources. ## README (from the GitHub repository) # ICU – AI-Powered Environmental Awareness for the Visually Impaired **ICU** is an assistive-technology prototype that provides visually impaired users with real-time awareness of their surroundings. A motorized camera, voice commands, and hand-gesture recognition work together to capture the environment, run AI analysis, and speak back contextual descriptions—acting as an “extra pair of eyes” so users can navigate safely and independently. --- ## Key Features - **Natural voice interface** – Ask questions like “What’s in front of me?” or direct the camera with spoken commands. - **Gesture-based control** – Pointing gestures are tracked with MediaPipe; the camera pans automatically to the indicated direction. - **Real-time scene analysis** – Live video streams run through multimodal models (Gemini + local OpenAI Whisper) to identify objects, hazards, and text. - **Low-latency architecture** – WebSockets connect phone cameras, local ML services, and cloud inference for responsive feedback. - **Hardware prototype** - ESP-32 microcontroller driving a stepper-motor pan-tilt camera - Secondary fixed camera for hand-landmark extraction - Python/C++ services running on a laptop (or Raspberry Pi) --- ## Tech Stack | Layer | Tools & Libraries | |--------------|-----------------------------------------------------| | AI / ML | vapi • Google Gemini • MediaPipe • OpenCV | | Hardware | ESP-32 • Stepper motors • motor drivers | | Backend | Flask API • WebSockets | | Miscellaneous| C++, Red Bull, Stethescope, Superglue 😉 | --- ## Roadmap - **Miniaturization** – Port to Raspberry Pi Zero 2 W or an all-mobile deployment. - **Expanded object ontology** – Better recognition of medication bottles, appliance states, and low-light scenes. - **Smart-home integration** – Act on findings (e.g., “turn off the stove”) via IoT APIs. - **Extended user testing** – Gather feedback from real users to refine voice NLP and gesture sensitivity. --- *Built at the **UC Berkeley AI Hackathon 2025** by Vyom Fadia, Jaymin Jhaveri, Katie Cheng, and Sam Mathew.* ## Detected evidence (automated analysis) Indexed codebase: 20 recognized source files, 92 KB. - C++ (language) — detected in the code - CSS (language) — detected in the code - Flask (technology) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (44 of 44) ``` .gitignore api/.python-version api/hello.py api/main.py api/pyproject.toml api/README.md backend/.python-version backend/pyproject.toml backend/README.md backend/src/backend/__init__.py backend/src/backend/__main__.py backend/uv.lock Berkeley_Stepper/.gitignore Berkeley_Stepper/.vscode/extensions.json Berkeley_Stepper/include/README Berkeley_Stepper/lib/README Berkeley_Stepper/platformio.ini Berkeley_Stepper/src/main.cpp Berkeley_Stepper/test/README frontend/.gitignore frontend/app/dashboard/alerts/page.tsx frontend/app/dashboard/history/page.tsx frontend/app/dashboard/page.tsx frontend/app/dashboard/settings/page.tsx frontend/app/globals.css frontend/app/layout.tsx frontend/app/page.tsx frontend/eslint.config.mjs frontend/next.config.ts frontend/package.json frontend/postcss.config.mjs frontend/README.md frontend/tsconfig.json hand_position.txt pyproject.toml README.md uv.lock voice-agent/.env.example voice-agent/.python-version voice-agent/pyproject.toml voice-agent/README.md voice-agent/src/voice_agent/__init__.py voice-agent/src/voice_agent/__main__.py voice-agent/uv.lock ``` ### Dependencies - api/pyproject.toml: flask@>=3.1.1, google-generativeai@>=0.8.5, pydantic@>=2.11.7 - backend/pyproject.toml: mediapipe@>=0.10.21, opencv-python@>=4.11.0.86, pyserial@>=3.5 - frontend/package.json: @eslint/eslintrc@^3, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@15.3.4, next@15.3.4, react@^19.0.0, react-dom@^19.0.0, tailwindcss@^4, typescript@^5 - pyproject.toml: asyncio@>=3.4.3 - voice-agent/pyproject.toml: numpy@>=1.24.0, openai@>=1.0.0, pyaudio@>=0.2.11, pyserial@>=3.5, python-dotenv@>=1.0.0, realtimestt@>=0.3.104, vapi-python@>=0.1.9 ### Recent commits (newest first) - readme - fix: finalise - chee - feat: image api - fix: improve prompting - fix: something - Merge branch 'master' of https://github.com/vyomfadia/berkeley - Fixed horrendous string function - feat: better voice agent - feat: god bless realtime streaming - bombo - Embedded stepper code - fix: dependencies - feat: voice agent - feat: stream linear position through websocket - fix: stupid file - fix: switch to uv workspace - feat: move to linear coordinates - fix: remove concurrent download limit - fix: voice agent readme ## Key source files (fetched from GitHub, selected and truncated for size) ### pyproject.toml ``` [project] name = "berkeley" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.12" dependencies = [ "asyncio>=3.4.3", ] [tool.uv.workspace] members = ["backend", "voice-agent", "api"] ``` ### api/pyproject.toml ``` [project] name = "api" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.12" dependencies = [ "flask>=3.1.1", "google-generativeai>=0.8.5", "pydantic>=2.11.7", ] ``` ### backend/pyproject.toml ``` [project] name = "backend" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.12" dependencies = [ "mediapipe>=0.10.21", "opencv-python>=4.11.0.86", "pyserial>=3.5", ] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["src/backend"] ``` ### voice-agent/pyproject.toml ``` [project] name = "voice-agent" version = "0.1.0" description = "Voice agent for camera control" readme = "README.md" requires-python = ">=3.12" dependencies = [ "openai>=1.0.0", "pyaudio>=0.2.11", "numpy>=1.24.0", "python-dotenv>=1.0.0", "realtimestt>=0.3.104", "pyserial>=3.5", "vapi-python>=0.1.9", ] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["src/voice_agent"] ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --turbopack", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "react": "^19.0.0", "react-dom": "^19.0.0", "next": "15.3.4" }, "devDependencies": { "typescript": "^5", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "@tailwindcss/postcss": "^4", "tailwindcss": "^4", "eslint": "^9", "eslint-config-next": "15.3.4", "@eslint/eslintrc": "^3" } } ``` ### api/main.py ```python import os import time import cv2 import google.generativeai as genai from PIL import Image from flask import Flask, jsonify, request from pydantic import BaseModel import serial app = Flask(__name__) # Configure Gemini API GEMINI_API_KEY = os.getenv('GOOGLE_API_KEY') # Set this in your environment variables genai.configure(api_key=GEMINI_API_KEY) serial = serial.Serial('/dev/tty.usbserial-0001', 115200) angle = None class ToolFunction(BaseModel): name: str arguments: dict class ToolCall(BaseModel): id: str function: ToolFunction class Message(BaseModel): timestamp: int toolCalls: list[ToolCall] class ToolCallMessage(BaseModel): message: Message def capture_photo(): """Capture a photo using the default camera""" try: camera = cv2.VideoCapture(0) if not camera.isOpened(): raise Exception("Could not open camera") ret, frame = camera.read() camera.release() if not ret: raise Exception("Could not capture image") frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) image = Image.fromarray(frame_rgb) return image except Exception as e: raise Exception(f"Camera capture failed: {str(e)}") def describe_image_with_gemini(image, *, context=None): """Send image to Gemini for description""" try: model = genai.GenerativeModel('gemini-2.5-flash-lite-preview-06-17') prompt = f""" Provide a short description of the surroundings in the image. The context of why you are asked would be a visually impaired individual who just asked for a snapshot of their surroundings from a camera. They mentioned a direction so the way you are facing is the direction they are interested in. Additional context you MUST SPECIFICALLY look out for in the image: {context or ''} """ response = model.generate_content([prompt, image]) return response.text except Exception as e: raise Exception(f"Gemini API error: {str(e)}") @app.route('/', methods=['POST']) def describe_surroundings(): message = ToolCallMessage(**request.json) """Endpoint to capture photo and get AI description""" toolCall = message.message.toolCalls[0] direction = toolCall.function.arguments.get('direction', None) context = toolCall.function.arguments.get('context', None) dont_change = toolCall.function.arguments.get('dontChange', False) full_sweep = toolCall.function.arguments.get('fullSweep', False) print("direction", direction) print("context", context) print("dont_change", dont_change) print("full_sweep", full_sweep) print("🎥 TRIGGERING CAMERA ROTATION ACTION") print("📡 This is where we would call the MediaPipe backend") global angle if not dont_change and not full_sweep: if direction == "left": angle = 90 elif direction == "right": angle = -90 elif direction == "forward": angle = 0 else: with open("hand_position.txt", "r") as f: lines = f.readlines() if lines: last_position = float(lines[-1].strip()) angle = last_position * -90 angle = round(angle) # Send angle over serial serial.write(f"move {angle} degrees\n".encode()) time.sleep(1.5) try: if not full_sweep: image = capture_photo() description = describe_image_with_gemini(image, context=context) if full_sweep: angle = -90 serial.write(f"move {angle} degrees\n".encode()) time.sleep(2) print("capturing 1") time.sleep(1) image1 = capture_photo() angle = 0 serial.write(f"move {angle} degrees\n".encode()) time.sleep(2) print("capturing 2") time.sleep(1) image2 = capture_photo() angle = 90 serial.write(f"move {angle} degrees\n".encode()) time.sleep(2) print("capturing 3") time.sleep(1) image3 = capture_photo() description1 = describe_image_with_gemini(image1, context=context) description2 = describe_image_with_gemini(image2, context=context) description3 = describe_image_with_gemini(image3, context=context) description = f"LEFT FACING IMAGE: {description1}\n\n FORWARD FACING IMAGE: {description2}\n\n RIGHT FACING IMAGE: {description3}" return jsonify({ 'success': True, 'message': 'Image captured and analyzed successfully', 'results': [ { "toolCallId": toolCall.id, "result": f"{description}\n\nThe camera is facing {angle} degrees from forward (positive is clockwise, to the right)." } ] }) except Exception as e: return jsonify({ 'success': False, 'error': str(e), 'message': 'Failed to capture or analyze image' }), 500 @app.route('/health', methods=['GET']) def health_check(): """Health check endpoint""" return jsonify({'status': 'healthy', 'service': 'Visual Assistant API'}) if __name__ == '__main__': # Check if Gemini API key is set if not GEMINI_API_KEY: print("Warning: GOOGLE_API_KEY environment variable not set") app.run(debug=True, host='0.0.0.0', port=8000) ``` ### frontend/app/page.tsx ```typescript import { redirect } from 'next/navigation'; export default function Home() { // Redirect to the dashboard page redirect('/dashboard'); } ``` ### frontend/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "HeartBeat | AI-Powered Heart Health Monitoring", description: "Monitor your heart health with advanced AI technology. Track heart rate, detect irregularities, and receive alerts for potential issues.", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### frontend/app/dashboard/page.tsx ```typescript import React from 'react'; import Link from 'next/link'; export default function Dashboard() { return (
{/* Header with cyberpunk styling */}

HeartBeat Monitor

LIVE

USER ID: #AX-7291-B

{/* Main content */}
{/* Heart rate monitor */}

Heart Rate

78 BPM
{/* This would be replaced with an actual heart rate visualization */}
{[...Array(20)].map((_, i) => (
))}
Last updated: Just now Normal
{/* Abnormality Alert */}
{/* Warning stripes overlay */}

Irregular Heartbeat Detected

Arrhythmia pattern detected at 14:32. Please remain calm and seated.

{/* Pulsing corner indicator */}
{/* Heart Health Metrics */}

Blood Pressure

120/80

Normal

Oxygen Level

98%

Optimal

Heart Rhythm

Irregular

Monitoring

Activity Level

Low

Resting

{/* Recent History */}

Recent History

{[ { time: '14:32', event: 'Irregular heartbeat detected', status: 'alert' }, { time: '13:45', event: 'Blood pressure reading', status: 'normal' }, { time: '12:30', event: 'Heart rate increased', status: 'warning' }, { time: '10:15', event: 'Daily health check', status: 'normal' }, ].map((it [truncated — 3970 more characters] ``` ### frontend/app/dashboard/history/page.tsx ```typescript import React from 'react'; import Link from 'next/link'; export default function History() { // Sample data for demonstration const historyData = [ { date: 'Today', time: '14:32', event: 'Irregular heartbeat detected', status: 'alert', details: 'Arrhythmia pattern detected. Heart rate fluctuated between 65-110 BPM within 2 minutes.' }, { date: 'Today', time: '13:45', event: 'Blood pressure reading', status: 'normal', details: '120/80 mmHg - Within normal range' }, { date: 'Today', time: '12:30', event: 'Heart rate increased', status: 'warning', details: 'Heart rate increased to 95 BPM during rest period' }, { date: 'Today', time: '10:15', event: 'Daily health check', status: 'normal', details: 'All vitals within normal parameters' }, { date: 'Yesterday', time: '21:20', event: 'Sleep monitoring started', status: 'normal', details: 'Baseline heart rate: 62 BPM' }, { date: 'Yesterday', time: '18:45', event: 'Exercise session completed', status: 'normal', details: 'Peak heart rate: 142 BPM, Duration: 32 minutes' }, { date: 'Yesterday', time: '14:10', event: 'Stress level elevated', status: 'warning', details: 'Heart rate variability decreased, suggesting elevated stress' }, { date: 'Yesterday', time: '09:30', event: 'Daily health check', status: 'normal', details: 'All vitals within normal parameters' }, { date: '2 days ago', time: '22:15', event: 'Possible sleep apnea', status: 'alert', details: 'Breathing irregularities detected during sleep' }, { date: '2 days ago', time: '16:40', event: 'Medication reminder', status: 'normal', details: 'Beta blocker scheduled dose' }, ]; // Group history data by date const groupedData = historyData.reduce((groups, item) => { const date = item.date; if (!groups[date]) { groups[date] = []; } groups[date].push(item); return groups; }, {}); return (
{/* Header with cyberpunk styling */}

Health History

SYNCED

USER ID: #AX-7291-B

{/* Filter controls */}
{/* Date range selector */}

Date Range

{/* History timeline */}
{Object.keys(groupedData).map((date) => (

{date}

{groupedData[date].map((item, index) => (

{item.event}

{item.time}