Project Info

Winner

Best Physical AI Hack by UFB

Omniscient

This project did not submit a description on Devpost.

hackcal / Omniscient

A robot fleet-management system: one operator oversees an autonomous robot line, a VLM correction layer auto-flags failures, and the operator can take over any robot and drive it with their own hand via CV hand-tracking.

Status: the control plane is built as a runnable mock fleet — hub + 2 agents + perception + VLM, all speaking the real wire contract, with the full failure→takeover→ recover loop verified headlessly (no hardware, no API key; python3 -m pytest -q). The ESP32-S3 hand firmware and webcam teleop bridge are the live hardware paths. Remaining hardware (real cameras, ACT policy, Jetson/ZED, CAN) plugs into the mocks one seam at a time — see docs/OPEN_DEVELOPMENT.md. Recent scope decisions: docs/DECISIONS.md.

Repo map

Path	What it is
`shared/schemas.py`	The contract spine — Pydantic wire messages + the transport seam.
`control_plane/`	`fleet.py` (pure state machine) + `hub.py` (FastAPI WebSocket hub).
`agents/`	Mock SO-101 + Piper agents — telemetry, scripted AUTO, staged fault, teleop + auto-relax safety, synthetic MJPEG.
`perception/`	Mock `HandPose` stream — the seam the Jetson/ZED + MediaPipe code plugs into.
`vlm/`	VLM correction layer — offline `/truth` mode + real Claude (Haiku/Opus) path.
`tests/`	State-machine, agent-safety, and end-to-end smoke tests.
`firmware/`	ESP-IDF firmware for the XIAO ESP32-S3 + PCA9685 driving the 5-servo hand. Read `firmware/CLAUDE.md` first — hardware gotchas + servo safety.
`host/`	`hand_teleop.py` — MediaPipe webcam → 5 finger closures → serial. See `host/README.md`.
`docs/`	System design (architecture, plan, topology) + `DECISIONS.md`. Plain markdown; read directly.
`CLAUDE.md`	Agent briefing + the load-bearing contract + what's real vs. planned.

Run the mock fleet (no hardware, no API key)

pip install -r requirements.txt
python3 run.py                 # hub :8000 + 2 agents + perception + VLM in one process
python3 -m pytest -q           # 22 tests: state machine, agent safety, end-to-end loop

Watch http://127.0.0.1:8000/healthz. With ANTHROPIC_API_KEY set the VLM uses real Claude; without it, the offline /truth path, so the loop runs with zero spend.

Run what exists today (hardware)

1. Flash the hand firmware (ESP-IDF; see firmware/CLAUDE.md for the flash-through-hub warning and pin notes):

cd firmware
idf.py build flash monitor      # flash the XIAO ESP32-S3 directly (not through a hub)

In the serial monitor you can drive the hand by hand: open, close, relax, f <i> <c> (one finger), or a bare f0 f1 f2 f3 f4 line of 5 closures in [0,1].

2. Run the webcam teleop bridge (Python 3.12 venv at repo root — see host/README.md for one-time setup):

.venv-cv/bin/python host/hand_teleop.py             # full teleop (auto-detects the port)
.venv-cv/bin/python host/hand_teleop.py --dry-run   # track + preview, DON'T move servos

The contract (host → firmware)

5 space-separated finger closures in [0,1], thumb→pinky, newline-terminated:

f0 f1 f2 f3 f4\n      # 0 = open, 1 = fully curled; ch0=thumb .. ch4=pinky

The firmware owns all servo safety (clamp to calibrated range, slew-limit, auto-relax ~2 s after the stream stops), so the host can only ever send 5 floats in [0,1]. Full protocol: docs/PLAN.md §4.5.

Analysis

Compare with all teams

View

Metric

matthehzhang
26
Claude Opus 4.8
1

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CIn code
CSSIn code
FastAPIIn code
HTMLIn code
JavaScriptIn code
PythonIn code

6 of 6 appear in the indexed code.

AI coding agents

Claude CodeConfig · Commits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

521 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

9 Verified3 Code-supported1 Claimed only

5-servo robotic hand actuated via ESP32-S3 + PCA9685Verified
Built with esp32, espidf, pca9685
Claimed on Devposthigh confidence
- firmware/main/main.c — Firmware maps 5 finger closures to PCA9685 channels per the CLAUDE.md-documented FINGER_CH mapping
- firmware/main/calibration.h — Per-finger calibrated open/closed PWM counts referenced for PCA9685 control
FastAPI WebSocket hub as control planeVerified
Built with fastapi, websockets
Claimed on Devposthigh confidence
- control_plane/hub.py:29 — Imports FastAPI, WebSocket, WebSocketDisconnect; defines /ws websocket endpoint at line 344-345 relaying messages between agents, dashboards, and VLM sockets
Fleet-management dashboard with Three.js 3D visualizationVerified
Built with three.js
Claimed on Devposthigh confidence
- dashboard/assets/app.js:5 — Imports THREE from vendor/three.module.js and constructs a WebGLRenderer, Scene, OrthographicCamera, lights
- dashboard/assets/vendor/three.module.js — Vendored three.js library confirms real dependency, not just a claim
Mock fleet runnable headlessly with no hardware/API key (pytest end-to-end loop)Verified
Full failure to takeover to recover loop verified headlessly via python3 -m pytest -q
Claimed on readmehigh confidence
- tests/test_smoke.py — End-to-end smoke test file present alongside test_fleet.py and test_agent_safety.py exercising the state machine
- run.py — run.py boots hub + agents + perception + VLM in one process per README instructions
Operator can take over any robot for manual controlVerified
The operator can take over any robot and drive it with their own hand
Claimed on Devposthigh confidence
- control_plane/fleet.py:172 — Fleet.take_over() implements AUTO/ALERT/MANUAL -> MANUAL transition, enforcing single active teleop target (I3) and blocking offline/disabled robots (I4)
Pydantic-defined wire message contract (structured schemas)Verified
Built with pydantic, antropicstructuredoutput
Claimed on Devposthigh confidence
- shared/schemas.py:25 — Imports pydantic BaseModel/Field/TypeAdapter and defines _Msg, Wrist, RobotStatus etc. as the wire contract (HandPose/Teleop referenced in CLAUDE.md)
Real-time hand tracking via webcam (MediaPipe) drives robot handVerified
The operator pilots from anywhere via real-time hand tracking
Claimed on Devposthigh confidence
- host/hand_teleop.py:37 — Imports mediapipe HandLandmarker, computes per-finger closures from hand_landmarks, and formats the 'f0 f1 f2 f3 f4' wire message matching the HandPose contract
Staged/scripted failure injection for demoing VLM detectionVerified
VLM flags failures the second they happen (demoed via staged faults)
Claimed on readmemedium confidence
- agents/mock_robot.py:55 — fault_after_s param and _fault_armed flag implement a one-shot staged fault used to demo the failure->alert->takeover loop
VLM correction layer auto-flags robot failuresVerified
A VLM watches each robot against its task and flags failures the second they happen
Claimed on Devposthigh confidence
- vlm/monitor.py:57 — VLMMonitor class with _classify_claude/_verdict_claude methods calling Anthropic Claude models (claude-haiku-4-5, claude-opus-4-8) to classify robot camera frames
- control_plane/fleet.py:156 — Fleet.on_alert() consumes VLM Alert messages and transitions AUTO to ALERT state
ACT policy for autonomous robot control (LeRobot)Code-supported
Built with act, lerobot
Claimed on Devpostmedium confidence
- agents/lerobot_backend.py:34 — ACTPolicyAdapter wraps lerobot's ACTPolicy.from_pretrained and SO101Follower for real inference
- scripts/run_act.py:51 — Script imports lerobot ACTPolicy for standalone inference, but per CLAUDE.md the mock fleet's AUTO mode is scripted, not the trained policy loop
Offline/no-API-key fallback for VLM ('/truth' path)Code-supported
Without ANTHROPIC_API_KEY set, an offline /truth path runs the loop with zero spend
Claimed on readmemedium confidence
- vlm/monitor.py:175 — _verdict_claude is one path; code lazily imports anthropic suggesting an offline fallback exists, but the /truth endpoint itself was not directly inspected
PiperX robotic arm integrationCode-supported
Built with piperx
Claimed on Devpostmedium confidence
- agents/actuator_agent.py:80 — Lazily imports PiperArm from piper_arm/piper_sdk for real CAN control, but CLAUDE.md states the mock fleet moves an EE proxy rather than real Piper CAN hardware today
Depth Anything V2 for perception/depth estimationClaimed only
Built with depthanythingv2
Claimed on Devposthigh confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.