Project Info

Locked in Vision

Inspiration

Locked in Vision was born from the factory floor: where one missed step, one wrong part, or one unchecked detail can become rework, scrap, or risk. We wanted to build a smarter MPI system that does not just display instructions, but actually sees the work being done and verifies reality in real time.

What it does

Locked in Vision uses computer vision to guide and verify an assembly process. For our demo, the system detects colored blocks inside an assembly area and confirms that they are placed in the correct sequence before unlocking the work order. It turns a normal checklist into a living process: the operator acts, the camera verifies, and the system only passes when the physical work is correct.

How we built it

We built it with a Python vision bridge, a FastAPI backend, and a frontend work-instruction UI. The camera detects the blocks, the backend validates the sequence, and the UI shows the operator what step they are on. The backend is the source of truth, so the process cannot be passed by a fake button or frontend shortcut.

Challenges we ran into

The hardest part was making the real world behave like software. Camera indexes failed, detections froze, ports conflicted, and blocks sometimes appeared visually correct but were not recognized by the system. We also had to cut the demo down to its strongest idea: block verification. Three minutes is short, so we removed extra tool and PPE steps to keep the story sharp.

Accomplishments we're proud of

We are proud that this is not just a dashboard. It connects the digital instruction to the physical assembly space. The system proves a powerful idea: manufacturing software should not only tell operators what to do. It should help verify that the work was actually done right.

What we learned

We learned that computer vision is powerful, but unforgiving. Lighting, camera angle, confidence thresholds, and zone calibration all matter. We also learned that a good demo is not about adding more features. It is about showing one clear truth with confidence.

What's next

Next, we want to make Locked in Vision more reliable, flexible, and production-ready. That means better calibration, stronger detection, snapshots for audit trails, and support for real manufacturing objects like tools, parts, labels, PPE, and fixtures. The bigger vision is a smart MPI system that protects quality, supports operators, and creates trustworthy evidence automatically. Locked in Vision is the first step toward work instructions that do not just guide the process, but verify the truth.

🔒 Locked Vision MPI

A fake MES-connected physical AI workstation. A locked camera supervises a workstation of colored LEGO blocks and simple tools. The operator cannot advance to the next manufacturing step unless the system visually verifies that the correct part/tool moved to the correct zone in the correct order — and the work order cannot close until every tool and part is returned home for a final 6S reset.

Pitch: This is not a camera watching a table. This is visual proof connected to manufacturing execution. The MPI only moves forward when the real world is correct.

Built for UC Berkeley AI Hackathon 2026.

Problem

In real manufacturing, MPI (Manufacturing Process Instruction) compliance is mostly trust-based: the operator clicks "Next" and the MES believes them. There is no physical verification that the right part went to the right place in the right order, and no guarantee the station is reset before the next job. Result: skipped/out-of-order steps, wrong parts, un-reset stations, and audit logs that record clicks, not reality.

Solution

Lock the digital instruction to the physical reality:

A fixed overhead camera + OpenCV detects colored LEGO blocks and tools and maps them to zones.
A fake MES holds the work order and the MPI step sequence (source of truth).
A state machine + validation engine compare vision evidence to the expected step and decide can_advance.
The dashboard physically cannot advance unless the backend returns can_advance=true, and cannot close the work order until final 6S passes.
Every pass and failure is written to an audit log.

Architecture

Fake MES (truth) ─► MPI step data ─► State machine ─► Validation ─► can_advance?
                                          ▲                              │
                                          │ evidence                     ▼
                                    Vision system  ───────────────►  Audit log
                                  (mock OR OpenCV)                       │
                                                                        ▼
                                                  Frontend (obeys can_advance only)

Core rule: MES is truth · vision is evidence · the state machine validates · the frontend never bypasses the backend. Full detail in ARCHITECTURE.md.

Repo structure

locked-vision-mpi/
├── backend/      FastAPI fake MES, state machine, validation, audit log
├── vision/       Camera, OpenCV color detection, zone mapping, mock vision
├── frontend/     Vite + React fake MES dashboard (gated Next button)
├── integrations/ Sponsor scaffolds (Fetch, Sentry, Redis, Deepgram, Arize)
└── docs          README · ARCHITECTURE · TEAM_ROLES · SPONSOR_INTEGRATIONS · DEMO_SCRIPT

Setup

The whole system runs on mocked vision, so you need no camera to demo it.

1. Backend

cd backend
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000     # http://localhost:8000/docs

2. Frontend

cd frontend
npm install
npm run dev                                    # http://localhost:5173

3. (Optional) Real vision

cd vision
pip install -r requirements.txt
python mock_vision_state.py                    # prints all mock scenarios

Or with Docker

docker compose up

Verify the backend

cd backend && .venv/bin/python smoke_test.py   # runs the full WO-1001 flow

Demo flow (4 moments)

Station readiness passes — post station_ready; readiness goes green.
Correct step passes — red → assembly, Verify, can_advance=true, Next unlocks.
Wrong sequence is blocked — blue first, Verify → red banner, Next stays locked, failure logged.
Final 6S blocks close — 6S: tool missing blocks; return the tool (6S: all home) → Work Order Closed.

Full run-of-show in DEMO_SCRIPT.md.

Sponsor tracks targeted

Track	How
Best Physical AI Hack (primary)	Locked camera + real workstation + visual MPI gating
Best Use of Fetch AI	MES Supervisor Agent operates the gated API
Best Use of Anthropic	Built with Claude Code; Claude explains audit logs
Best Use of Sentry	Live backend/frontend/vision error monitoring
Best Use of Deepgram	Optional hands-free operator voice commands
Best Use of Redis	Optional real-time station memory + audit stream
Best Use of Arize / Terac	Optional vision evaluation + labeling loop

Details: SPONSOR_INTEGRATIONS.md.

Team roles

Person	Role
1	Backend / MES Lead
2	Vision / Data Lead
3	Frontend / UI Lead
4	Sponsor / Agents / DevOps Lead

Details: TEAM_ROLES.md.

Safety / compliance

No prior project code was reused. All code in this repo is original and written fresh for UC Berkeley AI Hackathon 2026.
This is a fake/mock MES for demonstration — not a production system and not to be represented as production-grade.
No real PII or customer data; dummy work orders only.
The audit log is demo evidence, not a certified compliance record.
API keys live in .env (see .env.example) and are never committed.
The camera supervises blocks and tools only — no people/biometric tracking.

License

MIT (see hackathon submission).

Analysis

Compare with all teams

View

Metric

Claude Opus 4.8
1
ivancito
1

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CSSIn code
FastAPIIn code
HTMLIn code
JavaScriptIn code
PythonIn code
ReactIn code

6 of 6 appear in the indexed code.

AI coding agents

Claude CodeConfig · Commits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

137 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

8 Verified3 Code-supported4 Claimed only

Audit log of every pass/failVerified
Every pass and failure is written to an audit log
Claimed on readmehigh confidence
- backend/app/audit_logger.py:13 — AuditLogger.log appends JSONL entries; called from main.py on start, validate-step, advance, advance-blocked, and final-6s-check
- frontend/src/components/AuditLogPanel.jsx:1 — Frontend panel displays audit entries fetched via api.auditLog
Backend validates sequence before unlocking next stepVerified
The backend validates the sequence and confirms blocks are placed in the correct order before unlocking the work order / next step
Claimed on Devposthigh confidence
- backend/app/validation_engine.py:31 — validate_step checks vision evidence against expected step, including out-of-sequence detection for wrong-part-in-zone
- backend/app/main.py:230 — advance endpoint re-validates via validate_step and only calls sm.advance when can_advance is true
Computer-vision block detection and zone mappingVerified
The system detects colored blocks inside an assembly area using computer vision (OpenCV)
Claimed on Devposthigh confidence
- vision/color_detector.py:25 — detect_blocks uses OpenCV HSV thresholding and contour detection to find colored LEGO blocks
- vision/zone_mapper.py:40 — map_detections converts bounding boxes to zone assignments using backend/app/data/zones.json
- vision/camera.py:51 — get_vision_state chains camera frame -> detect_blocks -> map_detections into a vision state
FastAPI backend as source of truth (fake MES)Verified
Built with a FastAPI backend that is the source of truth so the process cannot be passed by a fake button or frontend shortcut
Claimed on readmehigh confidence
- backend/app/main.py:49 — FastAPI app 'Locked Vision MPI - Fake MES' exposes work-order/step/validate/advance endpoints
- backend/app/fake_mes_service.py:1 — FakeMESService holds work orders and MPI step sequence as backend-held truth
- backend/app/mpi_state_machine.py:60 — advance() only moves state forward; callers must confirm can_advance first, per docstring
Final 6S reset check gates work order closureVerified
The work order cannot close until every tool and part is returned home for a final 6S reset
Claimed on readmehigh confidence
- backend/app/validation_engine.py:101 — validate_final_6s checks tools_home, parts_home, assembly_clear_zone, and finished_assembly before allowing pass
- backend/app/main.py:271 — final_6s_check endpoint sets rt.status = 'completed' only if result['can_advance'] is true
Mock vision mode (camera-less demo)Verified
The whole system runs on mocked vision, so no camera is needed to demo it
Claimed on readmehigh confidence
- vision/mock_vision_state.py:77 — build_state constructs full vision-state dicts for named scenarios like station_ready, step1_done
- backend/app/main.py:72 — _resolve_vision calls build_mock_state when payload contains a scenario name
- frontend/src/api.js:51 — SCENARIOS list matches mock_vision_state scenario names for the LiveCameraPanel simulate buttons
React frontend work-instruction UI with gated Next buttonVerified
A frontend work-instruction UI shows the operator what step they are on; dashboard cannot advance unless backend returns can_advance=true
Claimed on readmehigh confidence
- frontend/src/App.jsx:100 — onNext calls api.advance and only clears validation state on pass; component re-fetches step/audit after
- frontend/src/components/MPIStepPanel.jsx:1 — MPIStepPanel renders step info and drives onValidate/onNext controls
- frontend/src/api.js:1 — api.js comment states backend is source of truth and only thing that sets can_advance
Station readiness checkVerified
Demo flow: posting station_ready vision evidence turns station readiness green
Claimed on readmehigh confidence
- frontend/src/App.jsx:89 — onSimulate sets stationReady true when scenario is 'station_ready'
- frontend/src/components/StationReadinessPanel.jsx:1 — Panel consumes ready prop to render readiness state
Fetch AI MES Supervisor AgentCode-supported
Best Use of Fetch AI: MES Supervisor Agent operates the gated API
Claimed on readmemedium confidence
- integrations/fetch_agent/mes_supervisor_agent.py:30 — MESSupervisor class calls backend HTTP endpoints (start, validate, 6S check, audit) as a plain requests client; the file's own docstring says wrapping it as a real uAgents/Agentverse agent is still a TODO
Real camera / OpenCV tool and finished-assembly detectionCode-supported
Real manufacturing objects like tools, parts, labels, PPE, and fixtures will be supported; 'a locked camera supervises a workstation of colored LEGO blocks and simple tools'
Claimed on readmemedium confidence
- vision/color_detector.py:72 — TODO comment states tool_1/tool_2 detection is NOT implemented yet, only color blocks are detected
- vision/color_detector.py:81 — Second TODO block states finished_assembly detection is not implemented
Sentry error monitoring integrationCode-supported
Best Use of Sentry: Live backend/frontend/vision error monitoring
Claimed on readmemedium confidence
- backend/app/main.py:41 — sentry_sdk.init is called only if SENTRY_DSN env var is set; this covers backend only, no frontend or vision-side Sentry code found
Arize / Terac vision evaluation and labeling loopClaimed only
Best Use of Arize / Terac: Optional vision evaluation + labeling loop
Claimed on readmehigh confidence
Claude-assisted audit log explanationClaimed only
Best Use of Anthropic: Built with Claude Code; Claude explains audit logs
Claimed on readmehigh confidence
Deepgram hands-free voice commandsClaimed only
Best Use of Deepgram: Optional hands-free operator voice commands
Claimed on readmehigh confidence
Redis real-time station memory + audit streamClaimed only
Best Use of Redis: Optional real-time station memory + audit stream
Claimed on readmehigh confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.