# Project export: Omniscient

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: UC Berkeley AI Hackathon 2026
- Tagline: Supervision layer for robot fleets. A VLM watches each robot against its task and flags failures the second they happen; the operator pilots from anywhere via real-time hand tracking.
- Devpost: https://devpost.com/software/ribbit-rtnfvg
- GitHub: https://github.com/matthehzhang/hackcal
- Video: https://www.youtube.com/embed/uvKU0ZPAZa4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Best Physical AI Hack by UFB)
- Team: 2 GitHub contributor(s) — matthehzhang (26 commits), Claude Opus 4.8 (1 commits)

## Devpost submission (written by the team)

No Devpost description available.

## README (from the GitHub repository)

# hackcal / Omniscient

A robot **fleet-management** system: one operator oversees an autonomous robot line, a VLM
correction layer auto-flags failures, and the operator can **take over any robot** and drive
it with their own hand via CV hand-tracking.

> **Status:** the **control plane is built as a runnable mock fleet** — hub + 2 agents +
> perception + VLM, all speaking the real wire contract, with the full failure→takeover→
> recover loop verified headlessly (no hardware, no API key; `python3 -m pytest -q`). The
> **ESP32-S3 hand firmware** and **webcam teleop bridge** are the live hardware paths.
> Remaining hardware (real cameras, ACT policy, Jetson/ZED, CAN) plugs into the mocks one
> seam at a time — see `docs/OPEN_DEVELOPMENT.md`. Recent scope decisions: `docs/DECISIONS.md`.

## Repo map

| Path | What it is |
|---|---|
| `shared/schemas.py` | The contract spine — Pydantic wire messages + the transport seam. |
| `control_plane/` | `fleet.py` (pure state machine) + `hub.py` (FastAPI WebSocket hub). |
| `agents/` | Mock SO-101 + Piper agents — telemetry, scripted AUTO, staged fault, teleop + auto-relax safety, synthetic MJPEG. |
| `perception/` | Mock `HandPose` stream — the seam the Jetson/ZED + MediaPipe code plugs into. |
| `vlm/` | VLM correction layer — offline `/truth` mode + real Claude (Haiku/Opus) path. |
| `tests/` | State-machine, agent-safety, and end-to-end smoke tests. |
| `firmware/` | ESP-IDF firmware for the XIAO ESP32-S3 + PCA9685 driving the 5-servo hand. Read `firmware/CLAUDE.md` first — hardware gotchas + servo safety. |
| `host/` | `hand_teleop.py` — MediaPipe webcam → 5 finger closures → serial. See `host/README.md`. |
| `docs/` | System design (architecture, plan, topology) + `DECISIONS.md`. Plain markdown; read directly. |
| `CLAUDE.md` | Agent briefing + the load-bearing contract + what's real vs. planned. |

## Run the mock fleet (no hardware, no API key)

```bash
pip install -r requirements.txt
python3 run.py                 # hub :8000 + 2 agents + perception + VLM in one process
python3 -m pytest -q           # 22 tests: state machine, agent safety, end-to-end loop
```

Watch `http://127.0.0.1:8000/healthz`. With `ANTHROPIC_API_KEY` set the VLM uses real
Claude; without it, the offline `/truth` path, so the loop runs with zero spend.

## Run what exists today (hardware)

**1. Flash the hand firmware** (ESP-IDF; see `firmware/CLAUDE.md` for the flash-through-hub
warning and pin notes):

```bash
cd firmware
idf.py build flash monitor      # flash the XIAO ESP32-S3 directly (not through a hub)
```

In the serial monitor you can drive the hand by hand: `open`, `close`, `relax`,
`f <i> <c>` (one finger), or a bare `f0 f1 f2 f3 f4` line of 5 closures in `[0,1]`.

**2. Run the webcam teleop bridge** (Python 3.12 venv at repo root — see `host/README.md`
for one-time setup):

```bash
.venv-cv/bin/python host/hand_teleop.py             # full teleop (auto-detects the port)
.venv-cv/bin/python host/hand_teleop.py --dry-run   # track + preview, DON'T move servos
```

## The contract (host → firmware)

5 space-separated finger **closures** in `[0,1]`, thumb→pinky, newline-terminated:

```
f0 f1 f2 f3 f4\n      # 0 = open, 1 = fully curled; ch0=thumb .. ch4=pinky
```

The firmware owns all servo safety (clamp to calibrated range, slew-limit, auto-relax ~2 s
after the stream stops), so the host can only ever send 5 floats in `[0,1]`. Full protocol:
`docs/PLAN.md` §4.5.


## Detected evidence (automated analysis)

Indexed codebase: 84 recognized source files, 521 KB.
- C (language) — detected in the code
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers

## Codebase structure (from repository index)

### Files (96 of 96)

```
.gitignore
agents/__init__.py
agents/actuator_agent.py
agents/camera_publisher.py
agents/camera.py
agents/cameras.py
agents/frames.py
agents/lerobot_backend.py
agents/livekit_publisher.py
agents/mjpeg.py
agents/mock_robot.py
agents/puck_trigger.py
agents/so101_agent.py
CLAUDE.md
control_plane/__init__.py
control_plane/fleet.py
control_plane/hub.py
dashboard/assets/app.js
dashboard/assets/models/amr.glb
dashboard/assets/models/CREDITS.txt
dashboard/assets/models/piper_rest.obj
dashboard/assets/models/piper_work.obj
dashboard/assets/models/so101_rest.obj
dashboard/assets/models/so101_work.obj
dashboard/assets/style.css
dashboard/index.html
dashboard/README.md
docs/ARCHITECTURE.md
docs/DECISIONS.md
docs/diagrams/FACTORY_LAYOUT.md
docs/OPEN_DEVELOPMENT.md
docs/PLAN.md
docs/README.md
docs/SO101_BRINGUP.md
docs/TOPOLOGY.md
firmware/CLAUDE.md
firmware/CMakeLists.txt
firmware/main/calibrate.c
firmware/main/calibration.h
firmware/main/CMakeLists.txt
firmware/main/main.c
firmware/README.md
firmware/sdkconfig
firmware/sdkconfig.defaults
host/hand_teleop.py
host/README.md
operator/CLAUDE.md
operator/HAND_CONTROL.md
operator/hub_arm.py
operator/install.sh
operator/README.md
operator/test_hub.py
ops/control_plane.sh
ops/env.sh
ops/piper_cam.sh
ops/piper.sh
ops/README.md
ops/so101.sh
perception/__init__.py
perception/mock_perception.py
prompts/factory_ops_assistant.md
README.md
requirements.txt
run.py
scripts/00_check_hardware.py
scripts/01_setup_lerobot.sh
scripts/10_calibrate.sh
scripts/20_teleop_check.sh
scripts/30_record.sh
scripts/40_train.sh
scripts/50_record_web.sh
scripts/delete_dataset.sh
scripts/deploy_nebius.sh
scripts/env.sh
scripts/fleet_extras.py
scripts/hub.sh
scripts/piper_control.sh
scripts/push_dataset.sh
scripts/README.md
scripts/record_buttons.py
scripts/record_web.py
scripts/run_act_robot.sh
scripts/run_act.py
scripts/run_live.py
scripts/run_local.sh
scripts/train_dashboard.py
scripts/view_camera.py
shared/__init__.py
shared/livekit_token.py
shared/schemas.py
tests/__init__.py
tests/test_agent_safety.py
tests/test_fleet.py
tests/test_smoke.py
vlm/__init__.py
vlm/monitor.py
```

### Dependencies

- requirements.txt: fastapi@>=0.115, httpx@>=0.27, pillow@>=10, pydantic@>=2.7, pytest@>=8, uvicorn@>=0.30, websockets@>=13

### Recent commits (newest first)

- last push
- ops
- big moves
- hand control
- agent
- ready for op!?
- operator control
- firmware
- current fixed
- push
- livekit test
- training vla
- psuh
- hand calibration
- training vla
- forklifts yay
- ui + begin vla integration
- ui + vlm
- ui
- ui

## Key source files (fetched from GitHub, selected and truncated for size)

### CLAUDE.md

```markdown
# Omniscient — agent briefing (root)

**Omniscient** is a robot **fleet-management** system: one operator oversees an autonomous
robot line, a **VLM correction layer** auto-flags failures, and the operator can **take over
any robot** and drive it with their own hand via CV hand-tracking. This directory is the
project root. The hardware-real **hand** — five servos driven by the operator's tracked
hand — is the firmware under [`firmware/`](firmware/); the system design lives under
[`docs/`](docs/).

## Read this when…

| File | Read it before you… |
|---|---|
| [`firmware/CLAUDE.md`](firmware/CLAUDE.md) | touch ANYTHING under `firmware/` — hardware gotchas (flash-through-hub corruption, swapped SDA/SCL, I²C NACK lies, **servo-safety pulse limits that already cost one dead servo**). Non-negotiable. |
| [`docs/OPEN_DEVELOPMENT.md`](docs/OPEN_DEVELOPMENT.md) §1 | wire CV/teleop into the hand. It names the exact seam (the synthetic hand-pose generator → `HandPose`) and the contract that must not change. |
| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | make a design decision — explains *why* the system is shaped this way (three planes, contract-first). |
| [`docs/TOPOLOGY.md`](docs/TOPOLOGY.md) | decide where something runs (laptop vs Jetson vs cloud) or how devices wire together. |
| [`docs/PLAN.md`](docs/PLAN.md) | need the demo target, ownership, or milestones. |
| [`docs/README.md`](docs/README.md) | want the system overview / orientation. |

## The load-bearing contract (this is the firmware's input)

The hand's input is the `HandPose.fingers` field — a **design contract**, described in
`docs/PLAN.md` §4 and now implemented in `shared/schemas.py` (`HandPose`/`Teleop`). The
piece this firmware consumes:

```
fingers: list[float]   # length 5, thumb -> pinky, 0.0 = open .. 1.0 = fully curled
```

On the wire (host → firmware) this is 5 space-separated floats: `f0 f1 f2 f3 f4\n`
(see `docs/PLAN.md` §4.5). `firmware/main.c` maps each onto PCA9685 channels **1:1,
thumb→pinky: ch0=thumb, ch1=index, ch2=middle, ch3=ring, ch4=pinky — all five driven**
(`FINGER_CH = {0,1,2,3,4}`). Each finger's `0.0..1.0` maps to its calibrated **open→closed
PWM counts** (`calibration.h`, captured via the interactive jog console — see
`firmware/CLAUDE.md`), bounded by the deliberately narrow safe pulse range. **Keep this
mapping faithful to the contract** so the same `HandPose` that drives the simulated hand
drives this one unchanged — the "mock → real is a backend swap, not a rewrite" property the
docs are built around.

## What is real today vs. what is still design-only

The `docs/` describe the full Omniscient system, but most of it is **not built yet**. Before
you go looking for a file or import, know what actually exists:

| Exists & runs | Design-only (described in docs, no code yet) |
|---|---|
| `firmware/` — ESP32-S3 hand drive firmware (closure protocol, calibration, slew/clamp/auto-relax safety). Done. | Real **camera capture** (mock serves synthetic MJPEG t
[truncated — 1642 more characters]
```

### prompts/factory_ops_assistant.md

```markdown
# Omniscient — factory ops assistant (chatbot master prompt)

This is the system prompt for the operator-facing chatbot. It roleplays as the plant's
control system. Use it as the `system` field with **Claude Opus 4.8** (`claude-opus-4-8`)
or, for cheaper/faster turns, **Claude Sonnet 4.6** (`claude-sonnet-4-6`). Keep it stable
(byte-identical) and put it behind a `cache_control` breakpoint so it caches across turns;
inject the live fleet snapshot *after* it (as a `{"role":"system"}` message on Opus 4.8, or
a leading user-turn context block otherwise), never by editing this text.

---

You are **Omniscient**, the fleet-operations assistant for **NORTHRIDGE PLANT · CELL A ·
KITTING LINE 2** — an autonomous robot kitting line overseen by a single human operator. You
are the conversational layer of the plant's control system: the operator talks to you to
understand line status, diagnose failures, and act on robots.

## The line you run
- Flow: **BUF-IN → conveyor → three arms → BUF-OUT**, with an **HX** handoff between arms and
  a **CHG-01** charge dock.
- Robots (backend id → station / floor label → role):
  - `so101_a` → WS-01 / **AR-1041** — SO-ARM100, picks from infeed and drops at the handoff.
  - `so101_b` → WS-02 / **AR-1042** — SO-ARM100, picks from the handoff and moves to the final bin.
  - `piper` → WS-03 / **AR-2207** — AgileX Piper running a scripted kitting task. This is the
    hero robot and the usual takeover target.
- Each robot is in exactly one **state**: `AUTO` (running autonomously), `ALERT` (a VLM-flagged
  failure needs the operator), or `MANUAL` (the operator is hand-driving it via hand-tracking
  teleop). Only **one** robot can be the active teleop target at a time.
- A **VLM correction layer** watches every camera feed (~1 Hz) and raises an
  `Alert{severity, confidence, issue, suggestion}` when a robot fails. An alert promotes
  `AUTO → ALERT`; it never overrides `MANUAL` (it won't yank a robot out from under the
  operator's hand).

## What you know — and what you must never invent
- On every turn you are given the **current authoritative fleet snapshot**: per-robot state,
  telemetry, the latest alerts, and which robot (if any) is the active teleop target. **That
  snapshot is your only source of truth.**
- **Never fabricate numbers, statuses, or events.** If a value isn't in the snapshot, say you
  don't have it. Do not invent throughput, OEE, uptime, cycle counts, energy, or timestamps —
  an ops console that reports made-up numbers is worse than one that reports fewer real ones.
- For questions the snapshot can't answer (history, root cause beyond the alert text), say
  plainly what you'd need to answer them rather than guessing.

## How to act
- You can take operational actions through your tools: `take_over(robot_id)`,
  `return_to_autonomy(robot_id)`, and `set_control_enabled(robot_id, enabled)`. Use them only
  when the operator clearly asks, and name the target by its floor label, e.g. *"Taking over
  AR-2207 (P
[truncated — 681 more characters]
```

### requirements.txt

```
# Omniscient control plane — runtime deps (the live system today).
# Tested on CPython 3.14. Pin loosely; these are all pure-Python or wheels.
fastapi>=0.115
uvicorn>=0.30
websockets>=13
pydantic>=2.7
httpx>=0.27
pillow>=10        # synthetic frame rendering (mock video plane)

# Optional — only needed for the ONLINE VLM path (real Claude). The offline /truth
# path (and the whole demo loop + smoke test) runs without it.
# anthropic>=0.40

# Dev/test
pytest>=8

```

### run.py

```python
#!/usr/bin/env python3
"""Launch the whole mock fleet in one process: hub + 2 agents + perception + VLM.

    python3 run.py

Then open the dashboard (dashboard/index.html) or watch http://127.0.0.1:8000/healthz.
This is the "run the mock fleet — confirm the loop" step (docs/OPEN_DEVELOPMENT.md §9.1):
everything speaks the real contract, so each piece can later be swapped for hardware.

With ANTHROPIC_API_KEY set, the VLM uses real Claude; without it, the offline /truth
path, so the failure→alert→takeover loop runs with zero spend.
"""

from __future__ import annotations

import asyncio
import logging

import uvicorn

from agents.mock_robot import MockRobotAgent
from control_plane.hub import make_app
from perception.mock_perception import MockPerception
from vlm.monitor import VLMMonitor

HOST = "127.0.0.1"
HUB_PORT = 8000


async def main() -> None:
    logging.basicConfig(level=logging.INFO, format="%(name)s: %(message)s")
    app, _hub = make_app()
    hub_url = f"ws://{HOST}:{HUB_PORT}/ws"

    # lifespan="off": the app has no startup/shutdown hooks, and disabling the lifespan
    # protocol keeps shutdown silent (no lifespan-cancel traceback).
    server = uvicorn.Server(
        uvicorn.Config(app, host=HOST, port=HUB_PORT, log_level="warning", lifespan="off"))
    server_task = asyncio.create_task(server.serve())
    while not server.started:
        await asyncio.sleep(0.05)
    logging.getLogger("omniscient").info("hub up on http://%s:%d", HOST, HUB_PORT)

    components = [
        MockRobotAgent("so101", "so101", hub_url, HOST, 8101),
        # the Piper stages the demo failure ~8 s into its cycle
        MockRobotAgent("piper", "piper", hub_url, HOST, 8102, fault_after_s=8.0),
        MockPerception(hub_url),
        VLMMonitor(hub_url),
    ]
    tasks = [asyncio.create_task(c.run()) for c in components]

    try:
        await asyncio.gather(server_task, *tasks)
    except asyncio.CancelledError:
        pass
    finally:
        # graceful teardown: stop clients, signal the server to exit, and absorb the
        # cancellation of in-flight socket work so shutdown is silent.
        for c in components:
            await c.stop()
        server.should_exit = True
        for t in tasks:
            t.cancel()
        await asyncio.gather(server_task, *tasks, return_exceptions=True)


if __name__ == "__main__":
    try:
        asyncio.run(main())
    except KeyboardInterrupt:
        pass

```

### scripts/hub.sh

```shell
#!/usr/bin/env bash
# Launch the control-plane hub from anywhere (it needs the repo on the import path).
#   bash ~/projects/hackathon/hackcal/scripts/hub.sh
cd "$(dirname "$0")/.." && exec python3 -m control_plane.hub

```

### control_plane/__init__.py

```python
"""Omniscient control plane — the single source of truth for fleet state.

`fleet.Fleet` is the transport-agnostic state machine + registry (no sockets — unit
testable). `hub` wraps it in a FastAPI WebSocket server that does only routing.
"""

```

### shared/__init__.py

```python
"""Omniscient shared contract — the spine.

Every message that crosses a process boundary is a Pydantic model defined in
`shared.schemas`. Import from here:

    from shared import schemas
    from shared.schemas import HandPose, Telemetry, parse_message
"""

```

### agents/__init__.py

```python
"""Mock robot agents — speak the real wire contract so the rest of the system cannot
tell them from hardware. Going live means swapping the frame source (synthetic→camera)
and the action source (scripted→ACT / CAN), not the agent's interface
(docs/ARCHITECTURE.md §6, docs/OPEN_DEVELOPMENT.md §2–§4).
"""

```

### perception/__init__.py

```python
"""Perception / teleop input — produces the operator's HandPose stream.

The mock streams a synthetic Lissajous hand. The real node (Jetson + ZED, or the laptop
webcam + MediaPipe fallback) replaces the generator body; the `HandPose` it emits is
identical, so nothing downstream changes (docs/OPEN_DEVELOPMENT.md §1). This is the exact
seam a teammate's vision code plugs into.
"""

```

### scripts/50_record_web.sh

```shell
#!/usr/bin/env bash
# Web recorder: record SO-101 demos from your browser (live camera + KEEP/DELETE/FINISH
# buttons). Owns the camera + arms, so don't run lerobot-record at the same time.
#   bash scripts/50_record_web.sh        # then open http://127.0.0.1:8088
set -euo pipefail
source "$(dirname "$0")/env.sh"
echo "starting web recorder — open  http://127.0.0.1:8088  in your browser"
exec "$HOME/lerobot-venv/bin/python" "$(dirname "$0")/record_web.py"

```

[65 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]