# Project export: Wander This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: A wearable haptic nav system for blind users: a belt with 4 vibrating motors guided by an iPhone fusing Maps, LiDAR, and AI vision. Speak a destination. Wander handles the rest. - Devpost: https://devpost.com/software/wander-18qa5o - GitHub: https://github.com/Sam-T-G/citrus-squad-x-berkeley-hackathon - Video: https://www.youtube.com/embed/Ud9yEybTYGU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Sam.G (78 commits), Claude Opus 4.7 (1M context) (70 commits), Cole (3 commits) ## Devpost submission (written by the team) ### Overview A wearable haptic vision system for blind and low-vision users ### Inspiration Blind and low-vision pedestrians navigate with tools that tell them almost nothing about the space around them. A white cane finds obstacles at ground level. A guide dog handles traffic. Neither gives turn-by-turn directions, and neither warns about a person stepping into your path from the side. We wanted to fuse navigation and obstacle awareness into a single, always-on sense, delivered through touch so the wearer's ears stay free for the street. What It Does Wander is a haptic belt worn around the torso. A chest-mounted iPhone reads Google Maps walking directions, monitors its own LiDAR depth sensor, and watches the camera for people and obstacles. Every heartbeat it compresses all of that into one cue and fires it to four servos arranged as a cross around the body: front, back, left, and right. The belt points the way to go, not at the hazard. A tap on the left motor means turn left, a tap on the right means turn right, and the front motor means you are on course. When something is in your path, the belt steers you clear instead of buzzing the thing itself. An obstacle on your left taps Right to send you toward the open side, and a person stepping in dead ahead taps Back, telling you to stop and step back. Closeness rides on the strength of the tap. The wearer sets a destination by speaking it, and Wander resolves the place, builds the route, and starts walking them there. The safety logic runs at 10 Hz with a strict priority stack: a person in your path beats a LiDAR obstacle, which beats an early-warning looming cue, which beats a navigation turn cue. The belt never confuses "turn here" with "stop, something is in front of you." How We Built It The phone is the entire sensing stack. We dropped the planned Coral accelerator once on-device LiDAR and CoreML carried the safety story, so there is no external compute to wear. iOS app (Swift 6, strict concurrency): AppModel runs a 10 Hz decide loop that arbitrates cues from four sources and fires a single LC2 packet over UDP to the belt. LiDAR obstacle detection: ARKit depth frames, sampled in three lateral bands to decide whether to steer left, steer right, or stop and reorient. Person and object detection: YOLOv8n CoreML model, on-device at the LiDAR frame rate. Depth-crop fusion confirms distance. 21 COCO navigation classes (person, bicycle, car, bus, stop sign, and more). Early warning: A BearingTracker watches for centered, looming objects before LiDAR has a return and fires a soft front tap as a heads-up. Navigation: Google Maps SDK + Directions API. The wearer speaks a destination, PlaceResolver finds it with MKLocalSearch, and the app drives the route off live GPS. Voice layer: Deepgram Voice Agent for speech in and out, with client-side function calling for commands like set_destination, where_am_i, describe_surroundings, and read_sign. Claude reasoning (off the safety path): the describe path runs an evaluator-optimizer, a Haiku draft checked by a Sonnet verify pass against the live scene, with an on-device guard that rejects any "path is clear" line the LiDAR contradicts. read_sign hands one camera frame to Opus vision to read store signs, bus numbers, and door labels, hedged when the read is high-stakes. Claude never touches the obstacle reflex. Belt: ESP32 in Wi-Fi AP mode driving four servos. A FastAPI laptop bridge over USB serial is the fallback if the ESP32 does not come up. Challenges We Ran Into Thermal headroom. Running LiDAR depth, YOLO inference, Google Maps, and a Deepgram WebSocket at once on one phone generates heat. We instrumented a thermal monitor and gated YOLO behind a thermal threshold to keep the phone from throttling mid-demo. ARKit and AVFoundation cannot share a session. We had to collapse the camera preview, LiDAR depth, and YOLO inference onto a single ARSession rather than running a second AVCaptureSession. Swift 6 strict concurrency. Every service and actor boundary had to satisfy the compiler's data-race checker, with a clean build and no warnings. Deepgram Voice Agent API quirks. The plan assumed a client_side flag on function definitions; sending one causes an UNPARSABLE_CLIENT_MESSAGE error, and the real way to mark a function client-side is to omit its endpoint. Push-to-talk with stopAudio tripped CLIENT_MESSAGE_TIMEOUT. We moved to continuous mic streaming with Deepgram's own end-of-speech detection and a tap-to-toggle UI. Chest-mount angle. LiDAR pointed at chest height sees the ground at range. Without ground-plane rejection, the obstacle cue fires constantly on flat pavement. Accomplishments We're Proud Of A complete pipeline from spoken destination to walking directions to haptic belt cues, all running on one iPhone with no cloud compute on the safety path. A belt that guides to safety rather than pointing at danger: every hazard cue taps the direction the wearer should move, and a four-tier arbiter guarantees a navigation hint can never mask a real hazard. The voice layer working end to end on device. Speak a place name, get a route, and the belt starts guiding. An evaluator-optimizer for spoken safety narration: a Haiku draft verified by Sonnet against the live scene, with an on-device guard that blocks a false "all clear." 120+ unit test assertions covering the packet codec, bearing math, routing geometry, obstacle avoidance, person detection, depth fusion, and voice command parsing. A hardware fallback (laptop FastAPI bridge to an Arduino over USB serial) so the demo is not gated on the ESP32 coming up. What We Learned Safety systems need a single, explicit arbitration point. Letting each subsystem fire the belt on its own would have produced chaos. A ranked priority stack with one sender per tick made the behavior predictable and testable. Voice APIs require hands-on device testing. Every assumption we made about the Deepgram API, the function flags, the push-to-talk model, the audio routing, turned out wrong in some detail. The as-built behavior came from running it on the phone, not from reading docs. On-device inference is fast enough to matter. YOLOv8n at the LiDAR frame rate adds real signal without a co-processor. What's Next for Wander Belt bring-up: prove the LC2 round-trip on the real ESP32 and servos, and tune the haptic patterns for clarity at walking speed. Thermal hardening: ground-plane rejection at the chest-mount angle, threshold tuning, and false-positive discipline with settle and hysteresis logic. On-device voice and vision verification: confirm the describe and read-sign paths answer within a single voice turn on the phone against live keys, and tune the tone of what Wander says. YOLO-World upgrade: swap the fixed COCO vocabulary for an open-vocabulary model so we can name any object class without retraining. Fetch.ai integration: an optional transactional tier for booking accessible transit or flagging routing hazards to a shared map. Built With Swift, SwiftUI, Swift 6, ARKit, CoreML, YOLOv8n, Google Maps SDK, Google Directions API, Deepgram Voice Agent, Anthropic Claude API, MKLocalSearch, ESP32, Arduino, FastAPI, Python, WebSockets, XcodeGen Team ## README (from the GitHub repository) # Citrus Squad × Berkeley AI Hackathon 2026 Citrus Squad's entry for the **Berkeley AI Hackathon 2026** at the MLK Jr. Building, UC Berkeley. Hack window opens Saturday June 20 at 11:00 AM and runs 24 hours, closing Sunday June 21 at 11:00 AM. Judging and closing ceremony follow. **Citrus Squad** is a haptic navigation belt for blind and low-vision wearers. A chest-mounted iPhone reads compass direction and Google Maps turn cues, detects nearby obstacles via LiDAR, identifies objects via on-device computer vision, and taps four servos on a belt to tell the wearer which way to turn or move. No screen. No audio required. The phone is the brain; an ESP32 drives the belt. ## Run it yourself Every teammate runs their own instance on their own phone and Mac. See **[RUNNING.md](RUNNING.md)** for the ten-minute setup. The short version: ```sh ./ios/setup.sh # installs XcodeGen, creates your local signing, generates the project open ios/CitrusSquad.xcodeproj # set your team + bundle id in ios/Local.xcconfig, pick your iPhone, press Cmd-R ``` You can run the full app with just a phone (the Navigation card's demo route + simulate mode needs no belt and no API key). The ESP32 belt and live Google Maps are optional add-ons covered in RUNNING.md. ## Computer vision layer (Cole — `cole/computer-vision`) The CV layer adds object awareness on top of the LiDAR obstacle detection already in the base app. It identifies what is in the path (pole, person, car, bench) and feeds that into the same hazard arbitration system that already drives the belt. ### What is built - **`cv/pipeline.py`** — transport-agnostic YOLOv8n inference fused with LiDAR depth. Takes a paired (RGB frame, depth map) and returns a list of `DepthFusedDetection` objects: label, confidence, bounding box, depth at the box, and horizontal position normalized 0–1. - **`cv/detection.py`** — `DepthFusedDetection` dataclass and the `NAVIGATION_CLASSES` filter (21 pedestrian-relevant COCO classes: person, bicycle, car, bench, parking meter, etc.). The on-device iOS filter (`CitrusSquadConfig.visionNavigationClasses`) mirrors this set so both recognize the same things. - **`cv/ingest.py`** — FastAPI WebSocket server. Accepts binary frame pairs from the iPhone over local Wi-Fi, runs the pipeline, and broadcasts JSON detections to any connected haptic client. - **`cv/webcam_test.py`** — local smoke test. Runs the pipeline against a laptop webcam with a synthetic 2.0m depth plane so the full detection path can be verified without a phone. - **`server.py`** — entry point (`uvicorn server:app --host 0.0.0.0 --port 8000`). - **`tests/`** — 17 unit tests covering depth fusion math and the wire protocol parser. ### Planned: on-device CoreML path (no Wi-Fi required) The Wi-Fi server works for prototyping but has a single point of failure at demo time. The target is to run everything on the phone: 1. Export `yolov8n.mlpackage` from the Python model (`YOLO("yolov8n.pt").export(format="coreml")`). 2. **`ObjectDetectionService.swift`** — subscribes to the ARKit session already running in `DepthService`. Each `ARFrame` carries both the camera image and the LiDAR depth map. Runs `VNCoreMLRequest` on the camera image, scales bounding boxes to depth coordinates, samples the inner 50% of each box (same as the Python fusion logic), and calls `VisionHazardSource.report()` with the result. 3. No Wi-Fi dependency. No laptop. All inference runs on the Neural Engine. ### Planned: collision prediction and action layer Pure-logic layer on top of raw detections. For each detection, it asks: is this object in my path, how close, and what is the best move? ``` Input: DepthFusedDetection list + LiDAR band readings Output: NavigationAction (StepLeft(paces: 2), StepRight(paces: 1), Stop, SlowDown, Clear) ``` Decision factors: horizontal position (is it centered?), distance (how urgent?), object type (static pole vs. moving person), and which side has more open space. Belt fires the directional tap; Josh's audio layer can say "pole ahead, step left 2 paces." ### Running the Python CV server ```sh pip3 install -r requirements.txt python3 server.py # or: uvicorn server:app --host 0.0.0.0 --port 8000 ``` Smoke-test the detection pipeline locally (no phone needed): ```sh python3 -m cv.webcam_test # built-in camera python3 -m cv.webcam_test 1 # external camera ``` Run the unit tests: ```sh python3 -m pytest tests/ -v ``` ## System architecture ``` iPhone (Citrus Squad app) ESP32 (belt) Maps directions + compass -> turn cue receives one LC2 packet LiDAR scene depth -> obstacle cue per 100 ms heartbeat, YOLOv8n CoreML (planned) -> object ID + action renders the event as a | servo pattern v arbitrate (safety > direction) one LC2 packet / 100 ms --UDP over Wi-Fi--> 4 servos: Far L, L, R, Far R ``` ## Team Sam (iOS, LiDAR, ESP32, CoreML iOS integration), Cole (computer vision, Python pipeline, collision prediction), Josh (audio), Angelo. ## License MIT. ## Detected evidence (automated analysis) Indexed codebase: 141 recognized source files, 976 KB. - C (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Python (language) — detected in the code - Swift (language) — detected in the code - Anthropic (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 154) ``` .gitignore CLAUDE.md CONTRIBUTING.md cv/__init__.py cv/detection.py cv/ingest.py cv/pipeline.py cv/webcam_test.py docs/00-overview.md docs/01-architecture.md docs/02-hardware.md docs/03-protocol.md docs/04-phone-side.md docs/05-vision-tier.md docs/06-failure-modes.md docs/07-timeline.md docs/08-team-roles.md docs/09-demo-and-pitch.md docs/10-validated.md docs/11-phone-app-design-spec.md docs/12-perception-and-safety-design.md docs/13-sponsor-implementation-ideas.md docs/14-voice-and-reasoning-plan.md docs/15-belt-server-bridge-plan.md docs/README.md firmware/citrus_squad_belt/citrus_squad_belt.ino firmware/citrus_squad_belt/config.h firmware/README.md HANDOFF.md IOS-APP-PLAN.md ios/AI-USAGE-AUDIT-AND-EXPANSION.md ios/BLIND-NAVIGATION-NORTH-STAR.md ios/CO-DESIGN-SESSION-PLAN.md ios/CV-PORT-PLAN.md ios/LAST-50-FEET-SCOPING.md ios/Local.xcconfig.example ios/NAVIGATION-HANDOFF.md ios/PERCEPTION-AVOIDANCE-HANDOFF.md ios/PERCEPTION-EARLY-WARNING-PLAN.md ios/Project.yml ios/README.md ios/run-on-device.sh ios/setup.sh ios/Sources/AI/ClaudeClient.swift ios/Sources/AppModel.swift ios/Sources/Approach/AbsoluteAnchorSource.swift ios/Sources/Approach/AnchorSighting.swift ios/Sources/Approach/AnchorStore.swift ios/Sources/CitrusSquadApp.swift ios/Sources/CitrusSquadConfig.swift ios/Sources/Diagnostics/EventLog.swift ios/Sources/Diagnostics/ThermalMonitor.swift ios/Sources/Info.plist ios/Sources/MapsBootstrap.swift ios/Sources/Networking/LC2Packet.swift ios/Sources/Networking/LC2Transmitter.swift ios/Sources/Networking/WebSocketBeltTransport.swift ios/Sources/Perception/AudioCueSink.swift ios/Sources/Perception/BearingTracker.swift ios/Sources/Perception/Cues.swift ios/Sources/Perception/DetectionStore.swift ios/Sources/Perception/InterferenceStore.swift ios/Sources/Perception/ObstacleAvoidance.swift ios/Sources/Perception/PerceptionSnapshot.swift ios/Sources/Perception/PersonDetector.swift ios/Sources/Perception/PersonFusion.swift ios/Sources/Perception/SpokenLineGuard.swift ios/Sources/Perception/VisionHazardSource.swift ios/Sources/Resources/yolov8n.mlpackage/Data/com.apple.CoreML/model.mlmodel ios/Sources/Resources/yolov8n.mlpackage/Manifest.json ios/Sources/Routing/Bearing.swift ios/Sources/Routing/DirectionsClient.swift ios/Sources/Routing/DirectionsService.swift ios/Sources/Routing/HeadingCalibrator.swift ios/Sources/Routing/HeadingResolver.swift ios/Sources/Routing/Maneuver.swift ios/Sources/Routing/NavigationCueSmoother.swift ios/Sources/Routing/NavTuning.swift ios/Sources/Routing/Polyline.swift ios/Sources/Routing/RouteEngine.swift ios/Sources/Routing/RouteSimulator.swift ios/Sources/Secrets.swift ios/Sources/Sensors/DepthService.swift ios/Sources/Sensors/LocationService.swift ios/Sources/Sensors/MotionService.swift ios/Sources/UI/BeltView.swift ios/Sources/UI/Cards.swift ios/Sources/UI/ControlPanelView.swift ios/Sources/UI/Demo/CameraBackdrop.swift ios/Sources/UI/Demo/CameraPanel.swift ios/Sources/UI/Demo/DepthPanel.swift ios/Sources/UI/Demo/ExpandedMapView.swift ios/Sources/UI/Demo/GoogleMapView.swift ios/Sources/UI/Demo/MapSection.swift ios/Sources/UI/Demo/Minimap.swift ios/Sources/UI/Demo/NavigationOverlay.swift ios/Sources/UI/DemoView.swift ios/Sources/UI/Feedback.swift ios/Sources/UI/NavTuningCard.swift ios/Sources/UI/ProductionView.swift ios/Sources/UI/RootView.swift ios/Sources/Voice/ChimePlayer.swift ios/Sources/Voice/PlaceResolver.swift ios/Sources/Voice/VoiceAudio.swift ios/Sources/Voice/VoiceCommand.swift ios/Sources/Voice/VoiceControlView.swift ios/Sources/Voice/VoiceError.swift ios/Sources/Voice/VoiceModel.swift ios/Sources/Voice/VoiceSession.swift ios/Sources/Voice/VolumeButtonTrigger.swift ios/test-markers.html ios/Tests/BearingTrackerTests.swift ios/Tests/DepthHazardTests.swift ios/Tests/DirectionsServiceTests.swift ios/Tests/HeadingCalibratorTests.swift ios/Tests/HeadingResolverTests.swift ios/Tests/LC2PacketTests.swift ios/Tests/ObstacleAvoidanceTests.swift ios/Tests/PersonDetectorTests.swift ios/Tests/PersonFusionTests.swift [34 more files omitted for size] ``` ### Dependencies - requirements.txt: fastapi@>=0.111, httpx@>=0.27, numpy@>=1.26, opencv-python@>=4.9, pytest@>=8.2, pytest-asyncio@>=0.23, ultralytics@>=8.2, uvicorn[standard]@>=0.29 - server/requirements.txt: aioserial@>=1.3.1, fastapi@>=0.110, uvicorn[standard]@>=0.29, websockets@>=12.0 ### Recent commits (newest first) - Merge sam/ios-app-base into main - Make the nav tolerance tunable live on the phone for a consistent demo - Narrate obstructions: richer, describe-don't-decide cue speech - Fix calibration getting stuck: allow a manual stationary lock - Size the navigation cue dwell to the turn so real corrections take agency - Wire the Local/Cloud belt transport toggle into the app - Add Phase 0 co-design session plan; voice think + TTS changes - Build the last-50-feet detection foundation + test markers - Add WebSocketBeltTransport: the phone-side cloud belt transport - Add hosted WebSocket relay as the internet fallback for the belt link - Feed Claude the full CV+LiDAR scene; scope the last-50-feet wedge - Retier the voice pipeline and add the honest Claude reader - Add on-device Claude reasoning tier; commit heading calibration - Add run.py: one-command belt-bridge bring-up with live link monitor - Add AI usage audit and expansion handoff doc - Re-fold Angelo's belt.ino (8ca8d91): add u_turn, map turn-around to it - Handle the forward event (0x24) explicitly in the belt mapping - Add navigation and collision-avoidance voice tools - Add tolerance handoff for tuning the navigation turn cue - Add hysteresis and dwell to the navigation turn cue so the belt stops chattering ## Key source files (fetched from GitHub, selected and truncated for size) ### CONTRIBUTING.md ```markdown # Contributing — Citrus Squad × Berkeley AI Hackathon 2026 Short rules so the four of us don't step on each other during a 24-hour hack. ## Branching - `main` is protected by convention (no force push, no direct commits to main once anyone else has a branch open). Land via PR. - **Personal branches**: prefix with your name. Use these for early exploration and anything you don't expect others to read yet. - `sam/` - `cole/` - `josh/` - `angelo/` - **Feature branches**: when the work is shareable, rename or merge into a `feat/` branch. - **Fix branches**: `fix/` for bug fixes during the hack. Example flow: `git checkout -b sam/phone-imu-spike` → work for an hour → push → realize it's solid → open PR titled "Phone IMU heading scaffold" → squash-merge. ## Commits - Present-tense, imperative mood. "Add heading service" not "Added heading service." - One concern per commit if you can. Hackathon time means it's fine to bundle when it's faster. - No "WIP" commits on `main`. WIP is fine on personal branches. ## Pull requests - Tag at least one teammate to skim before merging into `main`. A two-minute look is enough. Block only on actual problems, not on style preferences. - **Squash-merge** by default. Keeps `main` history readable when we look back during the pitch. - Self-approve if no teammate is online and the change is small and obviously safe. Note in the PR description that it was self-approved. ## Where things go Project direction is not locked yet. Until the team picks a stack, treat the repo as empty. Once the stack lands: - **Expo (React Native + TS):** standard Expo layout under `app/`, components in `components/`, services in `services/`. Don't check in `node_modules/` (already in `.gitignore`). - **Native iOS (Swift):** an Xcode project at the root or under `ios/`. Don't check in `xcuserdata/`, `DerivedData/`, or `.xcworkspace/` (already in `.gitignore`). ## Spec questions If something isn't clear from the README, `CLAUDE.md`, or the existing code, ask the team in chat first. If no one's online, write your assumption into the PR description and ship. ## When to stop and reconvene - Two people are about to touch the same file in incompatible ways. - A teammate's branch has been red for more than 30 minutes and you don't know why. - An idea pivot that would change the README direction. Mention it in chat first. Outside those cases, ship. ``` ### CLAUDE.md ```markdown # CLAUDE.md — Citrus Squad × Berkeley AI Hackathon 2026 Auto-loaded by Claude Code. Read first every session. ## Start here Read [`STATUS.md`](STATUS.md) before doing anything. It is the living context and handoff doc: current state, locked decisions, what is built, what is in flight, and where everything lives. Any agent that lands meaningful work updates it in the same pass. ## What this repo is Citrus Squad's code repo for the Berkeley AI Hackathon 2026 (June 20-21, MLK Jr. Building, UC Berkeley). This is the production repo. Code that ships at the demo lives here. ## Model context Sam runs Claude Code on **Opus 4.7 (1M context) only**. No fallback to Sonnet or Haiku. If a teammate is using a different model, the working norms in this file still apply, but answers may differ in style or depth. ## Project direction Open as of repo creation (June 20, 2026). Frontrunner is Citrus Squad (haptic navigation belt for blind and low-vision wearers). The team confirms direction at the session-start alignment meeting. Until then, do not assume scope. ## Stack Not yet picked. Team aligns on one of: - **Expo** (React Native + TypeScript) - **Native iOS** (Swift + SwiftUI) Native Android is out because the demo phone is an iPhone. **Stack chosen: native iOS Swift.** The base app is scaffolded in [`ios/`](ios/) and compiles under Swift 6 strict concurrency. Read these before working in it: - [`ios/README.md`](ios/README.md) — how to generate the project and run on the demo phone. - [`SWIFT.md`](SWIFT.md) — how Claude writes Swift in this repo. Read before writing any Swift. - [`IOS-APP-PLAN.md`](IOS-APP-PLAN.md) — the Citrus Squad phone-side app architecture and what is built so far. - [`docs/11-phone-app-design-spec.md`](docs/11-phone-app-design-spec.md) — the per-module build contract. Wins on implementation details. The project file is generated by XcodeGen from `ios/Project.yml`. The `.xcodeproj` is gitignored; run `xcodegen generate` after pulling or adding files. Do not commit the generated project. ## Voice rules Apply Samuel's standing writing voice to any prose Claude writes into this repo. - No em dashes in prose. Restructure or split. Structural dashes in headings and label-value lists are fine. - No "this isn't X, it's Y" contrast patterns. - No AI tells: `delve`, `tapestry`, `at its core`, `in conclusion`, `it is worth noting`, `landscape of`, `realm of`, `navigate the complexities`. - No corporate jargon (`leverage`, `end-to-end`, `deliverable`, `cutover`). - Plain English section headers. - Write like a sharp teammate, not a consultant deck. ## Working norms - **Branching follows `CONTRIBUTING.md`.** Personal branches use `sam/`, `cole/`, `josh/`, `angelo/` prefixes; feature branches use `feat/`. - **No `node_modules/` or `xcuserdata/` in commits.** The `.gitignore` covers both stacks. - **Commits are imperative, present tense.** "Add heading service" not "Added heading service." ## Free / ask first / never **Free [truncated — 839 more characters] ``` ### requirements.txt ``` ultralytics>=8.2 fastapi>=0.111 uvicorn[standard]>=0.29 numpy>=1.26 opencv-python>=4.9 pytest>=8.2 pytest-asyncio>=0.23 httpx>=0.27 ``` ### server/requirements.txt ``` fastapi>=0.110 uvicorn[standard]>=0.29 aioserial>=1.3.1 websockets>=12.0 ``` ### server.py ```python #!/usr/bin/env python3 """ WAND CV server entrypoint. Two WebSocket endpoints: /frames — iPhone app streams paired RGB + LiDAR frames (binary protocol) /haptics — haptic controller connects here to receive detection JSON Run: python server.py # or uvicorn server:app --host 0.0.0.0 --port 8000 """ import json import uvicorn from fastapi import WebSocket, WebSocketDisconnect from cv.detection import DepthFusedDetection from cv.ingest import app, register_output # All connected haptic controller clients. _haptic_clients: set[WebSocket] = set() @app.websocket("/haptics") async def haptics_ws(ws: WebSocket) -> None: await ws.accept() _haptic_clients.add(ws) try: # Block until disconnect. Absorb any keep-alive pings from the client. while True: await ws.receive_text() except (WebSocketDisconnect, Exception): _haptic_clients.discard(ws) async def _broadcast(detections: list[DepthFusedDetection]) -> None: if not detections or not _haptic_clients: return # Sort by closest depth first so the haptic layer can prioritize # the most urgent obstacle without sorting on its end. sorted_detections = sorted( detections, key=lambda d: d.depth_min_m if d.depth_min_m is not None else float("inf"), ) payload = json.dumps([d.to_dict() for d in sorted_detections]) dead: set[WebSocket] = set() for ws in _haptic_clients: try: await ws.send_text(payload) except Exception: dead.add(ws) _haptic_clients -= dead register_output(_broadcast) if __name__ == "__main__": uvicorn.run(app, host="0.0.0.0", port=8000, log_level="info") ``` ### server/app.py ```python """Citrus Squad belt bridge server. Accepts the phone's LC2 cue stream over a WebSocket and forwards each cue to the Arduino over one long-lived USB serial connection. The phone keeps doing all the arbitration and the 100 ms heartbeat; this process is a thin, low-latency forwarder plus a health dashboard. This is the no-ESP32 / no-Wi-Fi path: the laptop hosts the link and tethers to the Arduino over USB. The firmware it drives is `server/arduino/belt.ino`, which reads ONE newline-terminated word per cue: forward | stop | left | right | rotate_left | rotate_right | u_turn | idle Each word latches a continuous pulse pattern that runs until the next command, so `idle` is what stops the belt. This server translates each LC2 cue to a word (see `lc2_to_command`), writes only when the command CHANGES (so the heartbeat does not re-latch the same pattern), and synthesizes `idle` if the link goes silent. Wire in: 4 raw LC2 bytes per packet -> event, mask, intensity, seq over UDP (the phone's real link) or a WebSocket frame (test client / debug) Wire out: 1 newline-terminated word per change -> b"left\\n" / b"stop\\n" / b"idle\\n" / ... Run: uvicorn app:app --host 0.0.0.0 --port 8080 (or `python app.py`, which starts uvicorn for you) Config via env: SERIAL_PORT serial device, e.g. /dev/tty.usbmodem1101. Unset = auto-detect, then mock. SERIAL_BAUD default 9600 (matches Serial.begin(9600) in belt.ino) PORT HTTP/WebSocket port, default 8080 UDP_PORT UDP port the phone sends LC2 to, default 9999 (matches iOS espPort) WARMUP_S seconds to wait after opening serial, for the Arduino auto-reset, default 2.0 SILENCE_TIMEOUT_S send `idle` if no cue arrives within this, default 0.5 See docs/15-belt-server-bridge-plan.md for the why behind every choice here. """ from __future__ import annotations import asyncio import json import os import socket import time from contextlib import asynccontextmanager, suppress from fastapi import FastAPI, WebSocket from fastapi.responses import HTMLResponse, JSONResponse try: import aioserial # pyserial-asyncio for humans; opens the real port from serial.tools import list_ports except ImportError: # the app still runs in mock mode without these aioserial = None list_ports = None SERIAL_BAUD = int(os.environ.get("SERIAL_BAUD", "9600")) HTTP_PORT = int(os.environ.get("PORT", "8080")) # UDP port the phone's LC2Transmitter sends to. Matches the iOS default `espPort` (9999), # so the operator only has to point the belt host at this laptop's IP. UDP_PORT = int(os.environ.get("UDP_PORT", "9999")) WARMUP_S = float(os.environ.get("WARMUP_S", "2.0")) # The firmware pulses continuously until told otherwise, so if the phone link goes silent # we must send `idle` ourselves or the belt buzzes forever. Mirrors the 500 ms silence-to- # quiet rule in docs/03. Slightly longer than two 10 Hz heartbeats so a single dropped # packet does not blip the belt off. SILENCE_TIMEOUT_S = float(os.environ.get("SILENCE_TIMEOUT_S", "0.5")) # LC2 event codes (docs/03-protocol.md). EV_IDLE = 0x00 EV_VISION = 0x10 # vision-danger EV_TURN_SLIGHT = 0x20 EV_TURN_NOW = 0x21 EV_TURN_AROUND = 0x22 EV_ARRIVED = 0x23 EV_FORWARD = 0x24 # on-course / proceed straight EV_OBSTACLE = 0x40 # obstacle-near (LiDAR) # Quadrant mask bits, cardinal layout (matches belt.ino and the iOS QuadrantMask). MASK_FRONT = 0x01 MASK_LEFT = 0x02 MASK_RIGHT = 0x04 MASK_BACK = 0x08 # Synthesized when the link goes silent, to quiet the belt. IDLE_LC2 = bytes([EV_IDLE, 0, 0, 0]) def autodetect_port() -> str | None: """First USB serial port that looks like an Arduino, or None.""" env = os.environ.get("SERIAL_PORT", "").strip() if env: return None if env.lower() == "mock" else env if list_ports is None: return None for p in list_ports.comports(): name = p.device.lower() if any(tag in name for tag in ("usbmodem", "usbserial", "ttyacm", "ttyusb", "wchusb")): return p.device return None def lc2_to_command(lc2: bytes) -> bytes: """Translate a 4-byte LC2 cue to the newline-terminated word belt.ino understands. The firmware (Angelo's) reads `readStringUntil('\\n')` and latches a CONTINUOUS pulse per state, so every cue maps to a command and `idle` is what stops the belt. Intensity, sequence, and the far-left/far-right distinction do not survive this mapping; the Arduino path is the coarse fallback, the ESP32 path keeps full LC2. - idle (0x00) -> b"idle\\n" (stops the belt; must be sent) - vision/obstacle (0x10/0x40) -> b"stop\\n" (all-servo hazard buzz) - turn-around (0x22) -> b"u_turn\\n" (dedicated U-turn pattern) - arrived (0x23) -> b"stop\\n" (no arrival pattern; a buzz reads as "you're here") - directional turns, by mask -> b"left\\n" / b"right\\n" / b"forward\\n" Angelo's firmware also has `rotate_left`/`rotate_right` (in-place reorient) and `low_battery` (a finite alert). No LC2 event maps to those today, so they stay for manual/firmware testing. """ event, mask = lc2[0], lc2[1] if event == EV_IDLE: return b"idle\n" if event in (EV_VISION, EV_OBSTACLE, EV_ARRIVED): return b"stop\n" if event == EV_TURN_AROUND: return b"u_turn\n" if event == EV_FORWARD: return b"forward\n" if mask & MASK_LEFT: return b"left\n" if mask & MASK_RIGHT: return b"right\n" if mask & MASK_FRONT: return b"forward\n" if mask & MASK_BACK: return b"stop\n" # no dedicated back pattern; treat as an alert return b"idle\n" def parse_inbound(message: dict) -> bytes | None: """A WebSocket message -> 4 LC2 bytes, or None if it is not a valid cue. Binary frames are the real pat [truncated — 9207 more characters] ``` ### cv/__init__.py ```python from .detection import NAVIGATION_CLASSES, DepthFusedDetection from .pipeline import CVPipeline __all__ = ["CVPipeline", "DepthFusedDetection", "NAVIGATION_CLASSES"] ``` ### ios/setup.sh ```shell #!/usr/bin/env bash # One-time setup for the CitrusSquad iOS app. Run from anywhere: # ./ios/setup.sh # It installs XcodeGen if needed, creates your local signing config, and generates the project. set -euo pipefail cd "$(dirname "$0")" # 1. XcodeGen if ! command -v xcodegen >/dev/null 2>&1; then if command -v brew >/dev/null 2>&1; then echo "Installing XcodeGen via Homebrew…" brew install xcodegen else echo "XcodeGen is not installed and Homebrew was not found." echo "Install Homebrew from https://brew.sh then re-run, or install XcodeGen another way." exit 1 fi fi # 2. Per-developer signing config if [ ! -f Local.xcconfig ]; then cp Local.xcconfig.example Local.xcconfig echo "" echo "Created ios/Local.xcconfig from the template." echo ">> Open ios/Local.xcconfig and set DEVELOPMENT_TEAM and APP_BUNDLE_ID before building." echo " (You can also leave DEVELOPMENT_TEAM blank and pick your team in Xcode.)" echo "" fi # 3. Generate the Xcode project xcodegen generate echo "" echo "Done. Next:" echo " open ios/CitrusSquad.xcodeproj" echo " pick your iPhone as the run destination, then press Cmd-R." ``` ### ios/run-on-device.sh ```shell #!/usr/bin/env bash # Optional: build, install, and launch on the first connected iPhone without opening Xcode. # Xcode's Cmd-R does the same thing with a nicer signing UI; this is for quick CLI rebuilds. # ./ios/run-on-device.sh set -euo pipefail cd "$(dirname "$0")" if [ ! -f CitrusSquad.xcodeproj/project.pbxproj ]; then echo "No project yet. Run ./setup.sh first." exit 1 fi UDID=$(xcrun xctrace list devices 2>/dev/null \ | grep -i iphone | grep -v Simulator \ | grep -oE '[0-9A-Fa-f]{8}-[0-9A-Fa-f]{16}' | head -1) if [ -z "${UDID:-}" ]; then echo "No connected iPhone found. Plug it in, unlock it, and trust this Mac." exit 1 fi echo "Building for device $UDID…" xcodebuild -project CitrusSquad.xcodeproj -scheme CitrusSquad \ -destination "id=$UDID" -configuration Debug \ -derivedDataPath ./build -allowProvisioningUpdates build APP="./build/Build/Products/Debug-iphoneos/CitrusSquad.app" BUNDLE_ID=$(/usr/libexec/PlistBuddy -c 'Print :CFBundleIdentifier' "$APP/Info.plist") echo "Installing…" xcrun devicectl device install app --device "$UDID" "$APP" echo "Launching $BUNDLE_ID…" xcrun devicectl device process launch --device "$UDID" "$BUNDLE_ID" || \ echo "Install succeeded; launch was blocked (unlock the phone and tap the app icon)." ``` ### cv/detection.py ```python from __future__ import annotations from dataclasses import asdict, dataclass # COCO classes relevant to pedestrian navigation. # YOLOv8n sees all 80 classes; we filter to these after inference. NAVIGATION_CLASSES: frozenset[str] = frozenset( { "person", "bicycle", "car", "motorcycle", "bus", "truck", "chair", "couch", "dining table", "bed", "stop sign", "traffic light", "fire hydrant", "parking meter", "bench", "potted plant", "dog", "cat", "backpack", "suitcase", "umbrella", } ) @dataclass class DepthFusedDetection: label: str confidence: float # Bounding box in RGB pixel space: (x1, y1, x2, y2) bbox_px: tuple[int, int, int, int] # Depth in meters sampled from the LiDAR map at this bbox. # None if the depth region was all NaN / out of range. depth_median_m: float | None depth_min_m: float | None # closest point in bbox — most safety-relevant # Normalized horizontal center [0.0 = full left, 1.0 = full right]. # Downstream haptic layer uses this to pick belt zone without knowing frame size. horizontal_norm: float # Unix timestamp matching the source frame pair. timestamp_s: float def to_dict(self) -> dict: return asdict(self) ``` [126 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]