Project Info
This project did not submit a demo video on Devpost.
Inspiration
Currently, in war-torn and disaster-struck areas, first responders are risking their lives unnecessarily, as they lack the resources needed to accurately and safely assess a disaster zone. By using robotics, we can prevent the risk of human life.
What it does
The SPOT robot has been enabled to be an independent rescue machine that understands human emotion and natural language using AI with a noted ability to detect a language and adjust output as such.
How we built it
We built it using a variety of tools including Hume for AI/transcription, OpenCV for ML Models, Flask for the backend, and Next.js for the frontend.
Challenges we ran into
Connecting to and controlling SPOT was extremely difficult. We got around this by building a custom control server that connects directly to SPOT and controls its motors. The Hume API was relatively friendly to use and we connected this to a live stream of data via the Continuity Camera.
Accomplishments we're proud of
Fixing SPOT's internal linux dependencies. This is something that blocked all teams from using SPOT and took up most of the first day. But by solving this, we enabled SPOT to be used by all teams.
What we learned
We learned it is quite complex to combine various tech stacks across a variety of products both hardware and software. We learned to approach these problems by introducing levels of abstraction that would allow parts of the team to work parallely.
What's next
We hope to fully autonomize SPOTTER so that SPOT can traverse and navigate disaster environments completely independently. In this way, SPOT can locate survivors and assess the situation globally.
CalHacks 10 monorepo
The spot for our code for running spotter on Spot
backend: web server for multilingual, multimodal AI processing
spotter: frontend that enables easy robot control and shows the robot's camera and data feed
- serves data and object recognition from AI backend
- makes requests to robot control backend based on keyboard and mouse inputs
- Bun
- Next.js
- shadcn-ui components
robot-control-server: web server that enables remote control of the robot
- REST API written in Python using Flask
- Boston Dynamics SDK
Press
-
Demo
-
LinkedIn
-
Twitter
Running the stack:
First, clone this repo. If you don't have access to a Spot, you can still demo the entire frontend and AI part of the app locally, as long as you provide your OpenAI and Hume.ai API keys as an environment variables.
backend
- install Poetry
poetry installpoetry run devto start the backend
frontend
- install Bun
bun installbun devto start the frontend
robot server
- connect to Spot's wifi network
- run
python3 python/examples/wasd_server/app.pyand replace the hostname192.168.80.3with your Spot's IP - enter the
adminusername & password, or the equivalent credentials for your Spot - send requests to control endpoints manually or via the frontend
Analysis
View
Metric
- 24
- 1
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- FlaskIn code
- JavaScriptIn code
- Next.jsIn code
- PythonIn code
- ReactIn code
- Tailwind CSSIn code
- TypeScriptIn code
- OpenAIClaimed
8 of 9 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
3.0 MB
Source files
253
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
raviriley/code-spot
427 files · 6.0 MB · @ 3143c45
Structure
Interface
26 files · 6%Screens, components and styles rendered to the user.
API & routing
130 files · 30%Request entry points: routes, handlers and controllers.
Application logic
151 files · 35%Domain rules, services and shared utilities.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- Python62%
- Markdown35%
- TypeScript2%
- CSS0%
- XML0%
- JavaScript0%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
spotter/package.json
npm · 28- @radix-ui/react-dialog
- @radix-ui/react-dropdown-menu
- @radix-ui/react-icons
- @radix-ui/react-navigation-menu
- @radix-ui/react-popover
- @radix-ui/react-scroll-area
- @radix-ui/react-slot
- @tanstack/react-table
- class-variance-authority
- clsx
- cmdk
- lucide-react
- next
- next-themes
- react
- react-dom
- tailwind-merge
- tailwindcss-animate
- +10 more
backend/requirements.txt
pypi · 19- blinker
- certifi
- charset-normalizer
- click
- colorama
- flask
- gtts
- idna
- itsdangerous
- jinja2
- markupsafe
- numpy
- opencv-python
- pyaudio
- pygame
- python-dotenv
- requests
- urllib3
- +1 more
backend/pyproject.toml
pypi · 16- flask
- gtts
- hume
- opencv-python
- pyaudio
- pygame
- python-dotenv
- +9 more
robot-control-server/python/examples/wasd_server/requirements.txt
pypi · 3- bosdyn-client
- Flask
- Pillow
robot-control-server/python/examples/wasd/requirements.txt
pypi · 3- bosdyn-client
- Pillow
- windows-curses
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
Feature verification
Flask REST backendVerified
REST API written in Python using Flask
Claimed on readmehigh confidencebackend/backend/app.py:18— Flask app with /video and /process_audio routes
Frontend-to-robot-server proxy routesVerified
Frontend makes requests to robot control backend based on keyboard and mouse inputs
Claimed on readmehigh confidencespotter/pages/api/robot/[move].ts:28— Next.js API route proxies validated move commands to http://127.0.0.1:5000/ (robot-control-server) move/turn/sit/stand endpoints
Next.js frontend with robot control and camera/data feedVerified
Frontend that enables easy robot control and shows the robot's camera and data feed
Claimed on readmehigh confidencespotter/components/controls.tsx:12— Controls component sends WASD/keyboard and button inputs to /api/robot/[move]spotter/components/webcam.tsx:51— WebCam component renders the backend's live MJPEG video feed at localhost:5001/videospotter/components/transcription-table.tsx:114— TranscriptionTable fetches /api/audio and displays emotion/urgency/summary data feed
Person detection via OpenCV in live video feedVerified
Uses OpenCV for ML models to assess disaster zones
Claimed on Devposthigh confidencebackend/backend/app.py:22— generate_frames() uses cv2 HOGDescriptor people detector to draw bounding boxes on webcam framesbackend/backend/app.py:42— /video Flask route streams the annotated MJPEG feed
Speech-to-text transcription (OpenAI Whisper)Verified
Understands natural language using AI, multimodal AI processing backend
Claimed on readmehigh confidencebackend/backend/app.py:147— records 5s of mic audio and calls openai.Audio.transcribe('whisper-1', ...) to get transcript text
SPOT robot motor control server (Boston Dynamics SDK)Verified
Custom control server that connects directly to SPOT and controls its motors; robot-control-server REST API using Boston Dynamics SDK
Claimed on readmehigh confidencerobot-control-server/python/examples/wasd_server/app.py:3— imports bosdyn.client, authenticates and powers on the robot, exposes Flask routes like /move/forward, /sit, /stand that call WasdInterface methodsrobot-control-server/python/examples/wasd_server/wasd.py:1— WasdInterface implementation referenced by app.py for movement/estop/power control
GPT-4 conversational response generationCode-supported
Uses GPT-4 for AI processing (README built-with)
Claimed on readmemedium confidencebackend/backend/app.py:157— Calls openai.Completion.create with engine 'text-davinci-002', a GPT-3 completion model, not GPT-4, contradicting the README claim
Hume AI emotion/prosody analysis of audioCode-supported
Uses Hume for AI/transcription and emotion understanding
Claimed on Devpostmedium confidencebackend/backend/app.py:47— postAudioHume() submits recorded audio to HumeBatchClient with ProsodyConfig and parses top emotionsbackend/backend/app.py:20— HUME_API_KEY is hardcoded empty string, so the call would fail without a real key at runtime; wrapped in try/except that falls back to a hardcoded stub emotion
Language detection and response in the human's languageCode-supported
Noted ability to detect a language and adjust output as such
Claimed on Devpostmedium confidencebackend/backend/app.py:151— GPT prompt instructs the model to 'reply in the language of the human', relying on the LLM to infer language from transcript rather than explicit language-detection codebackend/backend/app.py:161— gTTS text-to-speech call is hardcoded to lang='en', which contradicts multilingual output despite the prompt instruction
Urgency level and paramedic summary generationCode-supported
Generates actionable insights/summary for first responders
Claimed on Devpostmedium confidencebackend/backend/app.py:175— urgency_level is hardcoded to 8 rather than computed from the emotion/text databackend/backend/app.py:178— summary_prompt asks GPT to synthesize an actionable paramedic summary from transcript, emotions, and urgency level
Autonomous/independent disaster-zone navigation and survivor locationClaimed only
The SPOT robot has been enabled to be an independent rescue machine ... traverse and navigate disaster environments completely independently to locate survivors
Claimed on Devposthigh confidenceBark text-to-speechClaimed only
Uses Bark (suno-ai) for speech synthesis
Claimed on readmehigh confidenceFacial emotion detection via HumeClaimed only
Understands human emotion via AI (implies facial + audio)
Claimed on Devpostmedium confidenceGeolocation data transmission to rescue teamsClaimed only
Images, voice, and geolocation data is being sent in real time to rescue teams
Claimed on Devpostmedium confidence
An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.