Project Info

Winner

Chroma: Build your project with Chroma in an AI application

Lucy

Inspiration

We are tinkerers and builders who love getting our hands on new technologies. When we discovered that the Spot Robot Dog from Boston Dynamics was available to build our project upon, we devised different ideas about the real-world benefits of robot dogs. From a conversational companion to a navigational assistant, we bounced off different ideas and ultimately decided to use the Spot robot to detect explosives in the surrounding environment as we realized the immense amount of time and resources that are put into training real dogs to perform these dangerous yet important tasks.

What it does

Lucy uses the capabilities of Spot Robot Dog to help identify potentially threatening elements in a surrounding through computer vision and advanced wave sensing capabilities. A user can command the dog to inspect a certain geographic area and the dog autonomously walks around the entire area and flags objects that could be a potential threat. It captures both raw and thermal images of the given object in multiple frames, which are then stored on a vector database and can be searched through semantic search. This project is a simplified approach inspired by the research "Atomic Magnetometer Multisensor Array for rf Interference Mitigation and Unshielded Detection of Nuclear Quadrupole Resonance" (https://link.aps.org/accepted/10.1103/PhysRevApplied.6.064014).

How we built it

We've combined the capabilities of OpenCV with a thermal sensing camera to allow Spot Robot to identify and flag potentially threatening elements in a given surrounding. To simulate these elements in the surroundings, we built a simple Arduino application that emits light waves in irregular patterns. The robot dog operates independently through speech instructions, which are powered by DeepGram's Speech to Text and Llama-3-8b model hosted on the Groq platform. Furthermore, we've leveraged ChromDB's vector database to tokenize images that allow people to easily search through images, which are captured in the range of 20-40fps.

Challenges we ran into

The biggest challenge we encountered was executing and testing our code on Spot due to the unreliable internet connection. We also faced configuration issues, as some parts of modules were not supported and used an older version, leading to multiple errors during testing. Additionally, the limited space made it difficult to effectively run and test the code.

Accomplishments we're proud of

We are proud that we took on the challenge of working with something that we had never worked with before and even after many hiccups and obstacles we were able to convert our idea in our brains into a physical reality.

What we learned

We learned how to integrate and deploy our program onto Spot. We also learned that to work around the limitations of the technology and our experience working with them.

What's next

We want to integrate LiDar in our approach, providing more accurate results then cameras. We plan to experiment beyond light to include different wave forms, thus helping improve the reliability of the results.

This image contains basic commands to work with camera, microphone and speaker at Spot's computer

Interaction with Spot is based on docker containers with code. You could fork this repo to test your own code. All basic functions are introduced in main.py.

Image link

Image link could be acquired in packages section of the main repository page. After fork, you might need to turn on github action to make it work.

How to use this image

Instal robot agent CLI

pipx install rn-cli

Create a user key

rn keys gen user.key

Share public key with Vataly to get access to robot

Setup environmental variables

export AGENT_RPC=ws://104.131.170.157:8888
export OWNER_KEY=Okkb1brctXS040mWyDun5aCYrG7yIHUjnx/Rza7KDhI=
export USER_KEY_PATH=user.key

You can check, if you see Spot

rn robots list

Send job to robot

Create job.json, change image link to your version

{
  "image": "ghcr.io/otaberu/hackathon-spot-image:main",
  "container_name": "",
  "ports": [],
  "network_mode": "host",
  "volumes": [
    {
      "key": "/dev/video0",
      "value": "/dev/video0"
    },
    {
      "key": "/dev/snd",
      "value": "/dev/snd"
    }
  ],
  "privileged": true,
  "store_data": false,
  "env": [
    "SDL_AUDIODRIVER=alsa",
    "AUDIODEV=hw:1,0",
    "AUDIO_INPUT_DEVICE=hw:2,0"
  ]
}

Create job (after you got access from Vitaly)

rn jobs add job.json spot

If you change last line of Dockerfile to CMD ["/bin/sh"], you could access terminal with

rn jobs terminal spot JOB_ID

Do not forget to exit it with exit command

Analysis

Compare with all teams

View

Metric

Jinash-Rouniyar
21

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

PythonIn code

1 of 1 appear in the indexed code.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

46 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

3 Verified4 Code-supported3 Claimed only

Spot deployment via Docker image and robot-agent CLIVerified
Interaction with Spot is based on Docker containers; use rn-cli / robot agent to deploy jobs to the physical robot
Claimed on readmehigh confidence
- Dockerfile:1 — Builds from a hackathon Spot base image, installs requirements.txt, and runs main.py as the container entrypoint
- job.json:1 — job.json matches the README's job spec for sending the image/container to the robot via the rn CLI
Spot robot control via Boston Dynamics SDKVerified
Lucy uses the Spot Robot Dog to inspect areas and move autonomously
Claimed on Devposthigh confidence
- spot_controller.py:108 — SpotController wraps bosdyn.client to move_to_goal, move_head_in_points, power on/off, lease and estop, using the real Boston Dynamics SDK
- main.py:280 — execute_direction maps instructions (up/down/left/right/turn/scan) to concrete SpotController calls
Voice command control (Deepgram STT + Groq Llama3)Verified
The robot dog operates independently through speech instructions, powered by Deepgram Speech to Text and Llama-3-8b hosted on Groq
Claimed on Devposthigh confidence
- main.py:259 — transcribe_audio uses the Deepgram client to transcribe a recorded wav file
- groq_bot.py:52 — ChatGroq is configured with model_name 'llama3-groq-8b-8192-tool-use-preview' and chained into groq_chain
- main.py:329 — run_complete_sequence feeds the transcript into groq_chain and executes the returned instruction via execute_direction
Autonomous area inspection on command ('inspect a geographic area')Code-supported
A user can command the dog to inspect a certain geographic area and the dog autonomously walks around the entire area
Claimed on Devpostlow confidence
- try.py:104 — explore_room performs a fixed scripted movement/scan pattern (square path, preset head positions), not a general area-bounded autonomous inspection driven by a geographic command
Computer vision threat/object flaggingCode-supported
Lucy flags objects that could be a potential threat using computer vision
Claimed on Devpostmedium confidence
- main.py:25 — SpotThreatDetector uses OpenCV HSV masking to find red-colored LED-like blobs across frames and marks a 'THREAT DETECTED' box when clusters are found
Thermal imaging captureCode-supported
It captures both raw and thermal images of the given object, built with a thermal sensing camera combined with OpenCV
Claimed on Devpostlow confidence
- main.py:165 — create_thermal_vision only applies a CLAHE + COLORMAP_JET false-color effect to the regular RGB frame; there is no code interfacing with an actual thermal sensor
- main.py:205 — capture_head_movement_frames saves 'original', 'threat_detection' and 'thermal_vision' frames per capture
Vector database storage and semantic image search (ChromaDB)Code-supported
Frames are stored on a vector database and can be searched through semantic search, leveraging ChromaDB
Claimed on Devpostlow confidence
- image_retreiver.py:42 — Builds a Chroma vector store from image embeddings, but Chroma.from_embeddings is not a real LangChain Chroma API method and the embedding step (tokenizer.encode on pixel_values) is misused, so this looks like an untested/non-functional script rather than a working semantic search pipeline
Advanced wave sensing (magnetometer/NQR-inspired detection)Claimed only
Advanced wave sensing capabilities inspired by Atomic Magnetometer research for detecting nuclear quadrupole resonance signatures
Claimed on Devposthigh confidence
Arduino light-wave emitter for threat simulationClaimed only
Built a simple Arduino application that emits light waves in irregular patterns to simulate threat elements
Claimed on Devposthigh confidence
LMNT integrationClaimed only
Built with LMNT (listed in Built With)
Claimed on Devposthigh confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.