# Project export: SARchlight This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: An AI rescue intelligence layer that turns drone footage, terrain, and uncertainty into a live map of where to search next - Devpost: https://devpost.com/software/sarchlight - GitHub: https://github.com/Mudit-Arora/SAR-system - Video: https://www.youtube.com/embed/xqCrhM_fjr4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Yousuf (62 commits), Mudit Arora (6 commits), Matt Do (4 commits), Claude Opus 4.8 (1M context) (2 commits) ## Devpost submission (written by the team) ### Inspiration In wilderness search and rescue, time is the enemy. Ground teams cover terrain slowly, aerial spotters get fatigued, and a person under canopy can be almost invisible from above. Even when drones are available, teams still need to answer the hard operational question: where should we look next? We wanted to build something more useful than another object detector. A detector can say "maybe person in this frame." Search teams need a system that combines that evidence with terrain, search coverage, missed detections, and uncertainty. SARchlight is our attempt to build that missing decision layer: a drone-side search brain that keeps updating the probability of where the person is and uses that map to drive the next move. ### What it does SARchlight gives SAR teams a live search brain: Builds a probability map from last-known position, terrain, land cover, and accessibility. Sends drones toward the highest-priority search sectors. Converts image detections into ground locations with a georeferencer. Updates the map from both detections and non-detections. Coordinates multiple drones so they do not search the same area. Declares a subject located only after persistent, concentrated evidence. Shows the mission state in a live React dashboard. After locating the subject, computes a terrain-aware route and simulates a drone guiding them back to operators. The core loop is: prior map -> search path -> drone observation -> Bayesian update -> next search target -> located event ### How we built it Search brain The backend uses a Python and NumPy probability grid. The brain owns a single-writer MapState and applies Bayesian updates as observations arrive. Terrain-aware prior The prior is shaped by the last-known position, elevation, land cover, and accessibility. The real-terrain showcase uses Marin County terrain with DEM hillshade and land-cover-derived visibility. Georeferencing The georeferencer translates detector boxes from image space into ground cells. This lets the search map understand where a detection happened and which ground cells were actually searched. Detection adapter The detector is swappable. The deterministic demo can run on simulated detections, while the integration path supports YOLO-style detector outputs through an adapter. Multi-drone planner The planner overlays coarse sectors on the fine belief grid, ranks sectors by probability and remaining coverage, and assigns drones to disjoint sectors. Dashboard A FastAPI server projects the brain state into a React, TypeScript, Vite, and Tailwind dashboard with probability heat, drone positions, detections, search path, confidence, locate status, and guide-home routing. Voice Layer The voice layer uses Deepgram for both the subject broadcast (text-to-speech) and a Twilio-backed operator phone agent (Deepgram Voice Agent) deployed on Fly.io, which reads the live search state over a dedicated endpoint and answers operator questions through function calls. Reliability For reliability, we instrumented the entire stack with Sentry — the FastAPI brain, the React dashboard, and the remotely deployed phone agent — so failures that would otherwise be invisible (a background search-thread crash, a degraded voice line, or an error mid-phone-call) surface immediately; it's fully env-gated, so it adds observability without adding any risk to the core loop. ### Challenges we ran into Making non-detections useful Search is not only about what the drone sees. Empty ground matters too. We had to model clean passes as evidence without letting repeated misses unfairly erase a subject hidden by canopy. Correlated misses under canopy In real terrain, repeated looks from the same sensor can fail in the same way. We added capped cumulative clearance per sensor so one drone repeatedly missing the same obscured cell does not drive that cell to zero probability. Avoiding premature locate alerts A single false positive should not trigger a rescue declaration. RescueLoop waits for persistent evidence and probability concentration before declaring the subject located. Real terrain was harder than synthetic terrain The real Marin terrain prior made the find more realistic and less forgiving. We redesigned the scenario around a plausible subject cell, canopy visibility, thermal corroboration, and a generalized flight path. Keeping the system integrated The detector, georeferencer, brain, server, dashboard, and voice layer all use different data shapes. We kept clear contracts and adapters so the pieces could connect without rewriting each other. ### Accomplishments we're proud of Built a working closed-loop SAR search system. Ran the brain on real terrain data. Located the planted subject at 0-cell error in the real-terrain showcase. Built multi-drone sector assignment with no overlapping search sectors. Built a live React dashboard for operator situational awareness. Added guide-home routing after the subject is found. Kept the demo honest about what is simulated, what is real, and what still needs field validation. -Deployed a live operator phone agent that answers questions from real-time search state. ## ### What we learned The most important signal in search is not always a detection. Sometimes it is a well-modeled miss. We learned that the strongest AI system here is not a single model. It is the loop connecting perception, geography, probability, planning, and communication. A weaker detector inside a strong search loop can be more useful than a stronger detector with no operational reasoning around it. ### What's next for Next, we want to turn Untitled from a software demo into a field-ready rescue assistant. We would connect it to live drone telemetry, real GPS/camera data, and thermal footage so detections can update the map in real time. We also want to improve the probability model with real SAR behavior data, terrain difficulty, trails, and last-known-position patterns. Long term, Untitled could coordinate multiple drones, support multilingual voice broadcasts to missing people, and use phone-signal or beacon data as another clue. Our goal is simple: help rescue teams search faster, avoid duplicated effort, and make better decisions when every minute matters. ## README (from the GitHub repository) # SARchlight **An AI rescue-intelligence layer that turns drone footage, terrain, and uncertainty into a live map of *where to search next*.** UC Berkeley AI Hackathon 2026 · wide-area Search & Rescue support. SARchlight is the *brain* behind a search-and-rescue drone — not the drone itself. It builds a live probability map of where a missing person likely is from last-known position, terrain, and land cover; directs drones to the highest-probability sectors; turns image detections into ground locations; updates the map from **both detections and clean non-detections**; coordinates multiple drones so they never re-cover the same ground; and declares a subject *located* only after persistent evidence. Once found, a drone guides the subject home and the system speaks to them. > **Scope.** SARchlight is the decision layer, demonstrated **in simulation** — there is no real > drone or live flight here. The demos run on **simulated detections** and a **scripted, simulated > flight** over **real terrain data**, with a **stationary** subject. A real YOLO detector path > exists but has **not been run on real footage**, and no detector has been fine-tuned, so detection > accuracy on real aerial imagery is untested. The "zero-cell locate on real terrain" result below is > produced by the brain, GeoReferencer, and planner driven by the simulator, not by real perception. > Real drone telemetry, footage, thermal sensing, and a moving-target model are future work (see > **What's next**). --- ## The closed loop ``` prior map ─▶ directs search path ─▶ detector on footage ─▶ detections + coverage update map ▲ │ └──────────────── updated map redirects + flags high-probability areas ┘ │ confident, persistent detection ─▶ subject broadcast + operator alert ─▶ guide home ``` In the multi-drone planner demo, the flight path is not hand-drawn — it *emerges* from the map: a detection (or a clean sweep) changes the belief, and the changed belief changes where the simulated drones go next. (The geo+brain feasibility demo uses a fixed scripted path.) ## What's inside - **Probability map (the core).** A NumPy belief grid with Bayesian updates. Single-writer design so map state can't be corrupted by concurrent readers. See `docs/interfaces.md` for the contracts and the Bayesian / `located` math. - **Terrain-aware prior.** Real DEM (elevation) + ESA WorldCover (land cover) shape where the subject is likely to be. See `docs/prior_model.md`. - **Non-detection handling.** A clean sweep *lowers* probability where we've actually looked — but canopy means "didn't see" ≠ "not there", so probability is down-weighted, never erased. - **`located` trigger.** Confidence-as-likelihood-ratio (clipped) + persistence, so a single false-positive aerial frame can't trip an alert. Locates at **zero-cell error on real Marin terrain rasters** in the demo, driven by simulated detections. - **Multi-drone planner.** Probability-of-area ranking with disjoint sector assignment and a boustrophedon sweep — no overlapping coverage. - **GeoReferencer.** Projects pixel detection boxes to ground cells; geography lives in exactly one swappable place, so the detector stays geography-blind. - **Swappable detector.** A simulator *and* a real YOLO path behind one adapter, so the loop runs immediately while the detector is an independent upgrade. - **Guide-home.** After locating, a drone leads the mobile subject home along a terrain-aware route. - **Voice layer.** A synthesized **subject broadcast** (Deepgram TTS) speaks to the found person, and a **Twilio-backed operator phone agent** (Deepgram Voice Agent, deployed on Fly.io) lets ground operators call in and ask the live system questions — coverage, the current highest-probability area, and whether the subject has been found. - **Live dashboard.** A React/TypeScript/Tailwind app showing the probability heat map over real terrain, drone positions, detections, routing, the live video feed, and the call transcript. - **Observability.** The whole stack is instrumented with **Sentry** — the FastAPI brain, the React dashboard, and the remotely deployed phone agent — so failures that would otherwise be invisible (a background search-thread crash, a degraded voice line, an error mid-phone-call) surface immediately. Fully env-gated: with no DSN it is a complete no-op. ## Repository layout ``` src/ the brain: GridSpec + contracts (common/), GeoReferencer (geo/), prior + Bayesian update + located trigger + planner + terrain (search/), and runnable demos (demo/) integration/ the transport seam: FastAPI server, the steppable loop, detector backends, broadcast + Deepgram TTS, map rendering, Sentry init (observability.py) dashboard_app/ the live React/Vite dashboard voice/ the Twilio + Deepgram operator phone agent (deploys to Fly.io) detector/ the real YOLO detector adapter tests/ unit + integration + soak tests (pytest) docs/ technical reference: interfaces, prior_model, core_loop, data, tech_stack data/ terrain rasters (gitignored — see Setup) ``` ## Setup ```bash # 1. Python backend (the brain + integration server) python3 -m venv .venv .venv/bin/python -m pip install -r requirements.txt # 2. Secrets — copy the template and fill in what you need cp .env.example .env # DEEPGRAM_API_KEY, ANTHROPIC_API_KEY; SENTRY_DSN is optional # 3. Terrain rasters are large and gitignored. Verify they are present + intact: .venv/bin/python check_setup.py # see docs/data.md for how to fetch them ``` ## Run ```bash # Tests (unit + integration + soak) .venv/bin/python -m pytest # Brain demo on synthetic terrain (locates) -> demo_output/ .venv/bin/python -m src.demo.run # Multi-drone sector search (closed loop) .venv/bin/python -m src.demo.search_demo --drones 3 # Closed loop CLI: detector -> geo -> brain (simulator backend) .venv/bin/python -m integration.loop # ...or the real YOLO backend: .venv/bin/python -m integration.loop --video --weights # Live dashboard: start the server, then the React app .venv/bin/uvicorn integration.server:app # http://localhost:8000 (serves /state, /map_base.png, /ops, ...) cd dashboard_app && npm install && npm run dev # http://localhost:5173 # Operator phone agent (local dev; needs DEEPGRAM_API_KEY) cd voice && python main.py # deploys to Fly.io via voice/ — see voice/README.md ``` ## Tech stack Python · NumPy · rasterio · FastAPI · YOLO · React · TypeScript · Vite · Tailwind · Claude (Anthropic) · Deepgram · Twilio · Fly.io · Sentry ## What's next Live drone telemetry, GPS/camera and thermal footage for real-time updates; richer probability models from real SAR behavior and trail data; and multilingual voice broadcasts. Known limitations and deferred work are noted alongside the relevant docs in `docs/`. ## Detected evidence (automated analysis) Indexed codebase: 144 recognized source files, 893 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 174) ``` .env.example .gitignore check_setup.py conftest.py dashboard_app/.gitignore dashboard_app/index.html dashboard_app/package.json dashboard_app/postcss.config.js dashboard_app/README.md dashboard_app/src/App.tsx dashboard_app/src/components/ConfidencePanel.tsx dashboard_app/src/components/DetectionsList.tsx dashboard_app/src/components/LiveTranscript.tsx dashboard_app/src/components/LiveVideoFeed.tsx dashboard_app/src/components/MapUpdateSummary.tsx dashboard_app/src/components/MissionSidebar.tsx dashboard_app/src/components/ProbabilityMap.tsx dashboard_app/src/components/ProbabilityTrend.tsx dashboard_app/src/components/SearchLoop.tsx dashboard_app/src/components/StatusBar.tsx dashboard_app/src/components/TopBar.tsx dashboard_app/src/components/VoiceComms.tsx dashboard_app/src/data/mockState.ts dashboard_app/src/hooks/useMapState.ts dashboard_app/src/hooks/useTranscript.ts dashboard_app/src/index.css dashboard_app/src/lib/api.ts dashboard_app/src/lib/colors.ts dashboard_app/src/main.tsx dashboard_app/src/types.ts dashboard_app/src/vite-env.d.ts dashboard_app/tailwind.config.js dashboard_app/tsconfig.json dashboard_app/tsconfig.node.json dashboard_app/tsconfig.node.tsbuildinfo dashboard_app/tsconfig.tsbuildinfo dashboard_app/vite.config.ts data/behavior/.gitkeep data/behavior/koester_references.md data/terrain/.gitkeep detector/.gitignore detector/configs/datasets/hit_uav_thermal.yaml detector/configs/datasets/wizard_rgb_person.yaml detector/configs/datasets/wizard_thermal_person.yaml detector/data/demo_footage/.gitkeep detector/data/processed/.gitkeep detector/data/raw/.gitkeep detector/docs/DATASETS.md detector/models/.gitkeep detector/notebooks/train_hit_uav_kaggle.ipynb detector/outputs/.gitkeep detector/pyproject.toml detector/README.md detector/requirements.txt detector/runs/.gitkeep detector/scripts/download_hit_uav.sh detector/scripts/make_hit_uav_test_video.py detector/scripts/run_demo.sh detector/scripts/run_rgb_demo.sh detector/scripts/train_hit_uav_cpu_quick.sh detector/scripts/train_hit_uav.sh detector/scripts/train_rgb.sh detector/scripts/train_thermal.sh detector/setup.cfg detector/setup.py detector/src/sar_demo/__init__.py detector/src/sar_demo/dataset_tools.py detector/src/sar_demo/infer_video.py detector/src/sar_demo/train_yolo.py docs/core_loop.md docs/data.md docs/interfaces.md docs/prior_model.md docs/tech_stack.md integration/__init__.py integration/backends.py integration/broadcast.py integration/dashboard_projection.py integration/deepgram_tts.py integration/detector_adapter.py integration/loop.py integration/map_render.py integration/observability.py integration/server.py integration/telemetry.py integration/terrain_render.py README.md requirements.txt src/__init__.py src/common/__init__.py src/common/config.py src/common/contracts.py src/common/grid.py src/demo/__init__.py src/demo/detector_sim.py src/demo/guide_home.py src/demo/mock_stream.py src/demo/run.py src/demo/search_and_guide.py src/demo/search_demo.py src/demo/showcase.py src/geo/__init__.py src/geo/georeferencer.py src/search/__init__.py src/search/brain.py src/search/guide.py src/search/planner.py src/search/prior.py src/search/return_path.py src/search/terrain_raster.py src/search/terrain.py src/search/trigger.py src/search/update.py src/search/validation.py tests/test_brain.py tests/test_broadcast.py tests/test_contracts.py tests/test_deepgram_tts.py tests/test_detector_sim.py tests/test_geo_integration.py [54 more files omitted for size] ``` ### Dependencies - dashboard_app/package.json: @sentry/react@^10.59.0, @types/react@^18.3.12, @types/react-dom@^18.3.1, @vitejs/plugin-react@^4.3.3, autoprefixer@^10.4.20, lucide-react@^0.456.0, postcss@^8.4.49, react@^18.3.1, react-dom@^18.3.1, recharts@^2.13.3, tailwindcss@^3.4.15, typescript@^5.6.3, vite@^5.4.11 - detector/requirements.txt: numpy@>=1.24.0, opencv-python@>=4.9.0, PyYAML@>=6.0.0, tqdm@>=4.66.0, ultralytics@>=8.3.0 - requirements.txt: fastapi@>=0.110, httpx@>=0.27, matplotlib@>=3.8, numpy@>=2.0, opencv-python@>=4.9, pytest@>=8.0, pytest-cov@>=5.0, rasterio@>=1.3, sentry-sdk@>=2.0, ultralytics@>=8.3, uvicorn@>=0.29 - voice/requirements.txt: deepgram-sdk@==6.0.0, python-dotenv@==1.2.1, python-multipart@==0.0.22, sentry-sdk@==2.63.0, starlette@==0.52.1, twilio@==9.10.2, uvicorn[standard]@==0.41.0 ### Recent commits (newest first) - Clarify scope (simulation) and fix two README accuracy overclaims - Merge branch 'extra-integrations' - Rewrite README for the full project (SARchlight) - Untrack internal build/planning docs for the public repo - Add Sentry to the Fly-deployed telephony agent - Add Sentry to the dashboard (DSN-gated init + error boundary) - Add Sentry monitoring to the integration backend (server + loop + TTS) - better looking dashboard - live drone footage - changed voice prompt - voice and map - Add check_setup.py: verify the gitignored terrain rasters are present + intact - gitignore: add root-level Node section (node_modules, dist, vite/ts caches) - Dashboard/server: root-cause fix for map flicker + blurred frame - Dashboard: instant frame swap (drop crossfade) to remove residual map flicker - Dashboard: real blink fix (ping-pong buffers), live Map Update Summary, bigger video feed - Dashboard: smooth map animation, offline indicator, remove dead controls - Docs: operator voice agent setup & run guide (voice/RUN_OPERATOR_AGENT.md) - Voice: exclude .venv from Fly build context (.dockerignore) - Voice P6 prep: fly min_machines_running=1 + document SAR_STATE_URL ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/core_loop.md ```markdown # The Core Loop — how the system works This system is the decision **brain behind a search-and-rescue drone** — not the drone itself. At its heart is a **probability map**: a live picture of where a missing person most likely is. The map points the search where to look; the drone's camera feeds a detector; what it finds — *and the empty ground it rules out* — flows back to update the map; and the updated map redirects the next pass. When a detection is confident and repeats across frames, the system speaks a calm message to the person and alerts the operator. The map is the center of gravity — it's the part that visibly *reasons* about where to look. --- ## The loop at a glance ```mermaid flowchart LR Prior["Prior map
where the person likely is"] Search["Search direction
where to look next"] Footage["Drone footage
camera over the terrain"] Detector["Detector
finds people in frames"] Geo["GeoReferencer
puts each hit on the ground"] Brain["THE BRAIN — probability map
a detection SHARPENS it
a clean pass CLEARS it"] Dash["Dashboard
operator's live view"] Located{"Confident
& repeated?"} Cast["Subject broadcast
speaks to the person"] Alert["Operator alert"] Prior --> Search Search -->|"where to fly"| Footage Footage -->|"video frames"| Detector Detector -->|"detections + confidence"| Geo Geo -->|"ground hits + coverage"| Brain Brain -->|"updated map — redirects the search"| Search Brain -->|"live map"| Dash Brain --> Located Located -->|"yes"| Cast Located -->|"yes"| Alert classDef brain fill:#ffe8e8,stroke:#e63946,stroke-width:2px; class Brain brain; linkStyle 5 stroke:#e63946,stroke-width:3px; ``` *The red arrow is the whole point: the map updates and **loops back** to redirect the next search. That's what makes this a search system, not a one-shot detector.* --- ## Same loop, in text ``` ┌─► Search direction ─► Drone footage ─► Detector ─► GeoReferencer ─┐ │ (map picks where (camera over (finds (places each │ │ to look next) the terrain) people in hit on the │ │ frames) ground) │ │ ▼ │ ┌──────────────────┐ │ updated map redirects the next search │ THE BRAIN │ └────────────────────────────────────────────────────────────┤ probability map │ │ │ a detection SHARPENS the map ───────────────► │ (single source of │ a clean, empty pass CLEARS it │ truth for "where │ │ is the person?") │ [truncated — 2552 more characters] ``` ### docs/prior_model.md ```markdown # The Prior Probability Map — Construction Rule **Status: pre-event design draft.** This specifies how the *initial* probability map is built before the search loop runs — the "prior" the judge sees at `t0` that explains where to look and why. It is the concrete combination rule that `docs/demo_scenario.md` §3 leans on and that `docs/interfaces.md` §5.5 named but deferred. Grounded in standard SAR probability-of-area (POA) practice (Koester / ISRID) and Bayesian search theory (Koopman, Stone). > **No-build rule.** Design only — no implementation before Saturday. Numbers below are > defaults to ratify, several flagged to be replaced by real Koester data (the Tier-1 gather). --- ## 1. What the prior represents A probability mass function over the grid: `p_i = P(person in cell i)`, plus a reserved `p_out` ("subject left the searched region"), with `Σ_i p_i + p_out = 1`. This is the **POC / containment prior** in search-theory terms; the loop then applies detection probability (POD) and the Bayesian update from `interfaces.md` §5. The prior is **not** uniform and **not** just a ring around the last-known-position (LKP). It fuses how-far-people-go statistics with where-the-terrain-lets-them-go and where-corridors-pull-them. ## 2. The combination rule (recommended) **Per-cell, multiplicative, then normalized:** ``` prior_i ∝ D(dist_i) · A_i · C_i P_i = (1 − p_out) · prior_i / Σ_j prior_j # normalize in-region mass # p_out held separately so Σ P_i + p_out = 1 ``` **Why multiplicative, not a weighted sum (the load-bearing modeling choice):** terrain is *non-compensatory* — a cliff or deep-water cell should get ~zero probability **regardless** of how close it is to the LKP. A product enforces that (any near-zero factor kills the cell); a weighted sum would let proximity "buy back" an impassable cell. Multiplication is also how real SAR terrain models (Jacobs/MRA "PDEN" layers) and Koester's factor maps actually stack. Keep a single `combine(layers, mode="product")` seam so individual layers could switch to compensatory later without a rewrite. *(Maps to your "explicit over clever" + "build seams, not futures".)* ### Term definitions and default values | Term | Meaning | Default / source | Confidence | |------|---------|------------------|-----------| | `dist_i` | Distance from LKP to cell i. **Euclidean first** (one line); **Tobler cost-distance** is the clean upgrade (§4). | — | — | | `D(dist)` | Distance decay. **Half-normal:** `D = exp(−dist² / (2σ²))`. | `σ ≈ 2.6 km` for a Hiker (see §3). **FLAG: replace with real per-ecoregion Koester quantiles.** | med | | `A_i` | Accessibility ∈ [ε, 1] from slope + land cover. Impassable (cliff, deep water) → **0**. Merely-hard terrain → small `ε ≈ 0.05` (discouraged, not forbidden). | `A = Tobler_speed_i / max_speed`, or a reclassed land-cover table | med | | `C_i` | Corridor attraction ∈ [1, k]. Boost cells on/near trails, roads, **drainag [truncated — 3769 more characters] ``` ### requirements.txt ``` # ============================================================================= # requirements.txt # ----------------------------------------------------------------------------- # Responsible for: Python dependencies for the search-map brain (src/) and its # demo. Install into a local virtualenv: # python3 -m venv .venv # .venv/bin/python -m pip install -r requirements.txt # Role in project: The brain is the critical path and runs standalone on mock # observations, so it deliberately keeps a tiny dependency set # (no geo stack yet — pyproj/rasterio/osmium are deferred behind # seams per the plan). Other tracks (dashboard, voice, detector) # add their own deps. # ============================================================================= numpy>=2.0 # vectorized grid math: half-normal decay, kernels, renormalize matplotlib>=3.8 # demo-only: posterior/coverage heatmap PNGs per beat rasterio>=1.3 # real terrain: sample DEM + ESA WorldCover rasters onto the grid (RasterTerrain) pytest>=8.0 # unit tests for prior / update / trigger pytest-cov>=5.0 # coverage report (pytest --cov=src --cov-report=term-missing) # --- Integration track (Milestone 1): wires the vendored detector + dashboard into # the brain. These are heavier (ultralytics pulls torch) and are needed only to run # integration/, not the brain itself — the brain above stays light. --- ultralytics>=8.3 # YOLO11 inference for the real detector adapter (integration/detector_adapter.py) opencv-python>=4.9 # video frame I/O for the YoloBackend (cv2.VideoCapture) fastapi>=0.110 # serve the projected UI-MapState as JSON to the React dashboard (integration/server.py) uvicorn>=0.29 # ASGI server that runs the FastAPI app httpx>=0.27 # test-only: Starlette/FastAPI TestClient transport (tests/test_integration_server.py) sentry-sdk>=2.0 # optional: error/perf monitoring for the server + loop CLI (Sentry); no-op without SENTRY_DSN ``` ### detector/requirements.txt ``` ultralytics>=8.3.0 opencv-python>=4.9.0 numpy>=1.24.0 PyYAML>=6.0.0 tqdm>=4.66.0 ``` ### voice/Dockerfile ``` FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . EXPOSE 8080 CMD ["python", "main.py"] ``` ### voice/requirements.txt ``` starlette==0.52.1 uvicorn[standard]==0.41.0 deepgram-sdk==6.0.0 twilio==9.10.2 python-dotenv==1.2.1 python-multipart==0.0.22 sentry-sdk==2.63.0 # optional error monitoring for the deployed agent; no-op without SENTRY_DSN ``` ### detector/pyproject.toml ``` [build-system] requires = ["setuptools>=61", "wheel"] build-backend = "setuptools.build_meta" [project] name = "sar-drone-demo" version = "0.1.0" description = "Drone footage search-and-rescue YOLO fine-tuning and inference demo" requires-python = ">=3.9" [tool.setuptools.packages.find] where = ["src"] ``` ### dashboard_app/package.json ``` { "name": "sar-dashboard", "private": true, "version": "0.1.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc --noEmit && vite build", "preview": "vite preview" }, "dependencies": { "@sentry/react": "^10.59.0", "lucide-react": "^0.456.0", "react": "^18.3.1", "react-dom": "^18.3.1", "recharts": "^2.13.3" }, "devDependencies": { "@types/react": "^18.3.12", "@types/react-dom": "^18.3.1", "@vitejs/plugin-react": "^4.3.3", "autoprefixer": "^10.4.20", "postcss": "^8.4.49", "tailwindcss": "^3.4.15", "typescript": "^5.6.3", "vite": "^5.4.11" } } ``` ### voice/main.py ```python """ Telephony Voice Agent - Entry Point Starts a Starlette web server that handles: - POST /incoming-call → Twilio webhook (returns TwiML) - WS /twilio → Twilio audio stream (or dev_client.py) - WS /transcript → live call transcript for the SAR dashboard Usage: python main.py For local development without Twilio: Terminal 1: python main.py Terminal 2: python dev_client.py """ import logging import uvicorn from starlette.applications import Starlette from starlette.routing import Route, WebSocketRoute from starlette.responses import PlainTextResponse from starlette.websockets import WebSocketDisconnect from config import ( SERVER_HOST, SERVER_PORT, SERVER_EXTERNAL_URL, DEEPGRAM_API_KEY, SENTRY_DSN, SENTRY_ENVIRONMENT, SENTRY_TRACES_SAMPLE_RATE, ) from telephony.routes import incoming_call, twilio_websocket from transcript_hub import transcript_hub # --------------------------------------------------------------------------- # Logging # --------------------------------------------------------------------------- logging.basicConfig( level=logging.INFO, format="%(asctime)s.%(msecs)03d %(levelname)s %(name)s - %(message)s", datefmt="%H:%M:%S", ) logger = logging.getLogger(__name__) # --------------------------------------------------------------------------- # Sentry (optional, additive) - error monitoring for the Fly-deployed agent # --------------------------------------------------------------------------- # Initialized at MODULE level so it runs whether the app starts via `python main.py` # (the Dockerfile CMD on Fly) or `uvicorn main:app`. With no SENTRY_DSN the SDK is # never initialized, so the agent behaves exactly as before. sentry-sdk auto-enables # its Starlette + asyncio + logging integrations, so unhandled errors in the HTTP/WS # routes are reported without per-route wiring; session.py adds call_sid context to # the in-call failures it currently swallows. if SENTRY_DSN: import sentry_sdk sentry_sdk.init( dsn=SENTRY_DSN, environment=SENTRY_ENVIRONMENT, traces_sample_rate=SENTRY_TRACES_SAMPLE_RATE, send_default_pii=False, ) logger.info("Sentry initialized (environment=%s)", SENTRY_ENVIRONMENT) else: logger.info("Sentry not configured (SENTRY_DSN unset) - monitoring OFF") async def dashboard(request): return PlainTextResponse( "Telephony Voice Agent is running.\n" "Call your Twilio number or use `python dev_client.py` to test locally." ) async def transcript_websocket(websocket): """Stream the live call transcript to a dashboard client. Subscribes the connecting client to the transcript hub and forwards every published message (call_started / turn / call_ended) as JSON. Cross-origin is fine: browser WebSockets aren't subject to CORS, so the dashboard (on a different port) can connect directly. Why a fixed path (no token): the dashboard's LiveTranscript connects to a single /transcript URL and auto-reconnects; this is a read-only fan-out of what's already spoken on the call, so it needs no per-client auth here. """ await websocket.accept() queue = transcript_hub.subscribe() try: while True: message = await queue.get() await websocket.send_json(message) except WebSocketDisconnect: pass except Exception as exc: # client went away mid-send, etc. logger.debug(f"Transcript client disconnected: {exc}") finally: transcript_hub.unsubscribe(queue) # --------------------------------------------------------------------------- # App # --------------------------------------------------------------------------- app = Starlette( routes=[ Route("/incoming-call/{token:path}", incoming_call, methods=["POST"]), Route("/incoming-call", incoming_call, methods=["POST"]), WebSocketRoute("/twilio/{token:path}", twilio_websocket), WebSocketRoute("/twilio", twilio_websocket), WebSocketRoute("/transcript", transcript_websocket), Route("/", dashboard), ], ) # --------------------------------------------------------------------------- # Startup # --------------------------------------------------------------------------- if __name__ == "__main__": logger.info(f"Deepgram API key: {'configured' if DEEPGRAM_API_KEY else 'MISSING'}") if SERVER_EXTERNAL_URL: logger.info(f"External URL: {SERVER_EXTERNAL_URL}") logger.info(f"Twilio webhook: {SERVER_EXTERNAL_URL}/incoming-call") else: logger.info("Running in local-only mode (no SERVER_EXTERNAL_URL set)") logger.info("Use dev_client.py to test - no Twilio or tunnel needed") uvicorn.run( app, host=SERVER_HOST, port=int(SERVER_PORT), proxy_headers=True, forwarded_allow_ips="*", ) ``` ### dashboard_app/src/main.tsx ```typescript // ============================================================================= // main.tsx // ----------------------------------------------------------------------------- // Responsible for: The dashboard's entry point — mounting into the DOM, and // (additively) initializing Sentry browser monitoring + wrapping the // app in an error boundary so a render-time crash shows a fallback // instead of a blank white screen. // Role in project: Frontend half of the optional Sentry reliability layer. Sentry is // OFF unless VITE_SENTRY_DSN is set at build time (then init runs). The // ErrorBoundary works either way, so white-screen protection is free // even with monitoring disabled. // ============================================================================= import React from 'react' import ReactDOM from 'react-dom/client' import * as Sentry from '@sentry/react' import App from './App' import './index.css' // Initialize Sentry only when a DSN is configured (mirrors the backend's env-gating): // no DSN -> no init -> no network, and the dashboard behaves exactly as before. const SENTRY_DSN = import.meta.env.VITE_SENTRY_DSN if (SENTRY_DSN) { Sentry.init({ dsn: SENTRY_DSN, environment: import.meta.env.VITE_SENTRY_ENVIRONMENT ?? 'demo', // Performance tracing, sampled to keep overhead/quota low (20% of transactions). integrations: [Sentry.browserTracingIntegration()], tracesSampleRate: 0.2, }) } // Minimal, dependency-free fallback shown if a component throws during render. Uses inline // styles (not Tailwind classes) so it still renders even if styling is part of what broke. const CrashFallback = (

Dashboard hit an error

The view crashed and was caught. Reload to retry — the error has been reported if monitoring is enabled.

) ReactDOM.createRoot(document.getElementById('root')!).render( {/* ErrorBoundary catches render-time crashes (and reports them when Sentry is on), so the operator sees a message instead of a blank screen mid-search. It also protects against a bad /state payload that throws somewhere in the render tree below useMapState. */} , ) ``` [133 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]