# Project export: Oper This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: The AI co-pilot for 911 operators. - Devpost: https://devpost.com/software/oper - GitHub: https://github.com/ericrcwu001/treehacks - Video: https://www.youtube.com/embed/zb7KhG2nCYw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — eric (31 commits), Jason Shaye (27 commits), Cursor (8 commits) ## Devpost submission (written by the team) ### Inspiration Every year, U.S dispatchers handle ~240 million 911 calls. Yet 82% of emergency communication centers are understaffed, and over 70% of dispatchers report burnout and chronic stress. When systems are stretched this thin, response times suffer — in San Francisco alone, the average 911 answer time recently exceeded 60 seconds. After interviewing the Director of Stanford EMS and active EMTs, we learned that dispatchers are being asked to do two fundamentally different jobs simultaneously: manage complex resource allocation across an entire city while providing emotional support to people in crisis. One EMT told us, "We're trained to ignore dispatcher information because it's often incomplete—they're just overwhelmed trying to do everything at once." We wanted to create a solution that could support both fundamental jobs for dispatchers by removing the cognitive burden of city-scale decision-making and helping operators perform at their best when lives are on the line. ### What it does Oper is a real-time emergency dispatch intelligence and training platform that augments 911 operators during live calls and strengthens them through protocol-grounded post-call feedback. It has 5 core components: Live City Digital Twin Oper runs a real-time graph-based model of San Francisco’s road network, tracking every simulated police unit, ambulance, and fire truck — including their exact location and availability — updated every second. Concurrent Emergency Simulation Using real historical SF 911 data, Oper simulates overlapping emergencies across the city so decisions are made within a live, resource-constrained environment. Real-Time Call Intelligence During a call, Oper transcribes audio, extracts critical signals like severity and location, and retrieves official dispatch protocols using a RAG-backed LLM. Resource-Aware Dispatch Optimization Oper combines caller context, protocol guidance, and live vehicle availability using a Haversine-based proximity algorithm to recommend the optimal set of specific EMS units at any given moment, complete with vehicle IDs and provided reasoning. Post-Call Evaluation & Training After each call, Oper analyzes the full transcript against official protocols, scoring performance and flagging missed questions or delayed escalations. It provides line-by-line, in-context feedback within realistic, geography-aware training simulations. Additional Features Interactive Map Integration - Recommendations are directly linked to the live map, so that clicking a dispatch suggestion zooms to the exact responding unit, showing where they are and tracking them live. Labels are shown according to general LLM-generated scenarios — Choose difficulty and get a unique scenario (caller, timeline, critical info) so every practice call is different and level-appropriate. Priority from transcript — Incident type from the transcript maps to priority 1–5 and drives suggested unit counts and map emphasis so recommendations match severity. AI-Assisted Note Taking — Oper generates structured, context-aware notes in real time during the call. With a single click, operators can insert optimized notes directly into their record Crime sim clock — Historical crimes play back on a sped-up sim clock (e.g. 60×) so a full “day” of incidents unfolds during one practice call. Voice input — Respond by speaking using chained speech-to-text and text-to-speech models to handle conversations. Live hints — Optional rotating hints during the call (e.g. “Ask for address,” “Dispatch EMS”) so trainees see what the system would recommend in real time. Operator notes — Timestamped notes during the call are sent into post-call evaluation so feedback references what the operator actually wrote down. Crime resolution — When enough units stay at a crime for a set time, it clears and disappears so the map reflects real-time demand and “cleared” incidents. 3D beacons — With the map tilted, 911 and crime points become vertical pillars; crime height scales by priority so severity is visible at a glance. SF Graph Visualization— Solid, slightly larger dots = available EMS vehicles color coded by vehicle type; translucent = busy; recommended units get a highlighted ring to distinguish units that are free and suggested. Transcript highlights — Review shows inline badges (Missed, Red flag, Good move) on the transcript so feedback is anchored to specific moments in the call. Dashboard + persistence — Sessions saved to Supabase; dashboard lists them with summary stats and a session drawer (scores, notes, “View full review”) so progress is trackable. Analytics charts — Score trend, missed-action frequency, and top recurring improvements so trainees and admins see patterns over time. ### How we built it We started by building a custom graph of San Francisco from open road and intersection data, with nodes and edges that represent real geometry so we can update vehicle positions and run proximity logic in one place. That graph is updated every second by a backend simulation service that moves police, fire, and EMS units along the graph and ingests historical SF 911 crime data that was pulled from a open public Kaggle dataset, so we can simulate concurrent incidents in real time. The live map is rendered with MapLibre and a vector basemap; the frontend subscribes to the same simulation feed so the map and the graph stay in sync. For the voice layer, we built a custom pipeline: live audio is streamed into our backend, transcribed in real time, and fed into an LLM that extracts location, severity, and emergency type. We implemented RAG over official 911 dispatch and protocol documents so every suggestion is grounded in real procedures. We run a separate Haversine-based proximity ranking algorithm over the graph’s current state—position and availability updated every second—to compute the closest available unit per type and their ETAs; those results are exposed via an API and drive the recommendations and map highlights. The frontend pushes call context and receives back unit IDs and ETAs, and click-to-dispatch is implemented as a map zoom/focus to the chosen vehicle’s live position. Post-call evaluation uses the same RAG-backed LLM: we run the full transcript against the same protocol corpus, detect missed questions and delayed escalations, and score timeliness and critical information capture to produce structured feedback. For dispatcher training, we use the same simulation and map stack and add a separate AI caller so we can generate dynamic scenarios—including rare edge cases—so operators practice in the same resource-constrained, geographically accurate environment they’ll see on a real call. ### Challenges we ran into Keeping the digital twin and map in sync: The graph updates every second with vehicle positions and availability; the frontend had to consume that feed without jank or drift. We had to design a clear contract between the sim and the map (what gets pushed, when, and how the UI reacts) so the live map felt reliable. Voice --> structured data in real time: Turning live call audio into clean signals (location, severity, emergency type) for the LLM and RAG was hard—transcription delays, noise, and incomplete sentences meant we had to design the pipeline to handle partial or late updates and still produce usable recommendations. Grounded recommendations: We wanted every suggestion tied to real protocols, not generic advice. Getting RAG to retrieve the right protocol chunks from long documents and have the LLM use them consistently took a lot of prompt and retrieval tuning. Realistic simulation without real dispatch data: We don’t have live SF dispatch feeds, so we used open road data and historical 911 crime data to build a plausible, resource-constrained environment. Making that feel “real enough” for training and demos required careful design of the graph and crime injection. ### Accomplishments we're proud of End-to-end live flow: From a single 911-style call (live or simulated), we run a digital twin, voice agent, RAG-backed suggestions, Haversine-based closest-unit ranking, and a live map with click-to-dispatch—all in one coherent experience. Getting that full loop working in a hackathon timeline was a big win. One system for live support and training: The same graph, sim, voice pipeline, and RAG power both live-call assistance and post-call evaluation plus scenario-based training. That reuse makes the product easier to explain and extend. Protocol-grounded AI: Recommendations and feedback are explicitly tied to official dispatch documents, not a generic chatbot. We’re proud that the system can cite protocols and surface missed steps and delayed escalations in a way that’s useful for real operators. Geographically real training: Operators can practice with an AI caller on a real SF road graph and simulated concurrent incidents, including edge cases. We’re proud that training happens in the same kind of resource-constrained, spatially accurate environment they’ll see on the job. ### What we learned Dispatchers are doing two jobs at once. Our interviews (Stanford EMS, EMTs) made it clear: the bottleneck isn’t only call volume—it’s the impossible combination of city-wide resource allocation and in-the-moment emotional support. That shaped our goal: help with the optimization so they can focus on the human. Simply sending the “Closest unit” isn’t enough in real world dispatch scenarios. Real dispatch has to consider type (police vs fire vs EMS), availability, and concurrent incidents. Building a live digital twin and Haversine-based ranking taught us how much structure is needed before “recommend a unit” is actually useful. RAG has to be tuned for high-stakes domains. For 911, retrieval has to hit the right protocol slices and the LLM has to stick to them. We learned that good RAG is as much about document structure and evaluation as it is about the model. Simulation makes the product tangible. A map with moving units and concurrent crimes made the value of Oper obvious to users and judges in a way that slides alone wouldn’t. ### What's next : Pilot with a real center: We want to run Oper alongside real (anonymized or sandboxed) call flow at an emergency communication center to validate that our voice pipeline and recommendations hold up under real noise and protocols. Richer protocol coverage and compliance: Ingest more agency-specific protocols and refine RAG and evaluation so feedback and suggestions are even more accurate and defensible for training and QA. Multi-agency and multi-jurisdiction: Extend the digital twin and sim beyond SF so Oper can support regions with multiple agencies and different response rules. Integration with existing CAD/NG911 systems: Work toward APIs or integrations so Oper can plug into existing Computer-Aided Dispatch and next-gen 911 systems instead of being a standalone layer. ## README (from the GitHub repository) ## Inspiration Every year, U.S dispatchers handle ~240 million 911 calls. Yet 82% of emergency communication centers are understaffed, and over 70% of dispatchers report burnout and chronic stress. When systems are stretched this thin, response times suffer — in San Francisco alone, the average 911 answer time recently exceeded 60 seconds. After interviewing the Director of Stanford EMS and active EMTs, we learned that dispatchers are being asked to do two fundamentally different jobs simultaneously: manage complex resource allocation across an entire city while providing emotional support to people in crisis. One EMT told us, "We're trained to ignore dispatcher information because it's often incomplete—they're just overwhelmed trying to do everything at once." We wanted to create a solution that could support both fundamental jobs for dispatchers by removing the cognitive burden of city-scale decision-making and helping operators perform at their best when lives are on the line. ## What it does Oper is a real-time emergency dispatch intelligence and training platform that augments 911 operators during live calls and strengthens them through protocol-grounded post-call feedback. It has 5 core components: 1. Live City Digital Twin Oper runs a real-time graph-based model of San Francisco’s road network, tracking every simulated police unit, ambulance, and fire truck — including their exact location and availability — updated every second. 2. Concurrent Emergency Simulation Using real historical SF 911 data, Oper simulates overlapping emergencies across the city so decisions are made within a live, resource-constrained environment. 3. Real-Time Call Intelligence During a call, Oper transcribes audio, extracts critical signals like severity and location, and retrieves official dispatch protocols using a RAG-backed LLM. 4. Resource-Aware Dispatch Optimization Oper combines caller context, protocol guidance, and live vehicle availability using a Haversine-based proximity algorithm to recommend the optimal set of specific EMS units at any given moment, complete with vehicle IDs and provided reasoning. 5. Post-Call Evaluation & Training After each call, Oper analyzes the full transcript against official protocols, scoring performance and flagging missed questions or delayed escalations. It provides line-by-line, in-context feedback within realistic, geography-aware training simulations. ### Additional Features - Interactive Map Integration - Recommendations are directly linked to the live map, so that clicking a dispatch suggestion zooms to the exact responding unit, showing where they are and tracking them live. Labels are shown according to general - LLM-generated scenarios — Choose difficulty and get a unique scenario (caller, timeline, critical info) so every practice call is different and level-appropriate. - Priority from transcript — Incident type from the transcript maps to priority 1–5 and drives suggested unit counts and map emphasis so recommendations match severity. - AI-Assisted Note Taking — Oper generates structured, context-aware notes in real time during the call. With a single click, operators can insert optimized notes directly into their record - Crime sim clock — Historical crimes play back on a sped-up sim clock (e.g. 60×) so a full “day” of incidents unfolds during one practice call. - Voice input — Respond by speaking using chained speech-to-text and text-to-speech models to handle conversations. - Live hints — Optional rotating hints during the call (e.g. “Ask for address,” “Dispatch EMS”) so trainees see what the system would recommend in real time. - Operator notes — Timestamped notes during the call are sent into post-call evaluation so feedback references what the operator actually wrote down. - Crime resolution — When enough units stay at a crime for a set time, it clears and disappears so the map reflects real-time demand and “cleared” incidents. - 3D beacons — With the map tilted, 911 and crime points become vertical pillars; crime height scales by priority so severity is visible at a glance. - SF Graph Visualization— Solid, slightly larger dots = available EMS vehicles color coded by vehicle type; translucent = busy; recommended units get a highlighted ring to distinguish units that are free and suggested. - Transcript highlights — Review shows inline badges (Missed, Red flag, Good move) on the transcript so feedback is anchored to specific moments in the call. - Dashboard + persistence — Sessions saved to Supabase; dashboard lists them with summary stats and a session drawer (scores, notes, “View full review”) so progress is trackable. - Analytics charts — Score trend, missed-action frequency, and top recurring improvements so trainees and admins see patterns over time. ## How we built it We started by building a custom graph of San Francisco from open road and intersection data, with nodes and edges that represent real geometry so we can update vehicle positions and run proximity logic in one place. That graph is updated every second by a backend simulation service that moves police, fire, and EMS units along the graph and ingests historical SF 911 crime data that was pulled from a open public Kaggle dataset, so we can simulate concurrent incidents in real time. The live map is rendered with MapLibre and a vector basemap; the frontend subscribes to the same simulation feed so the map and the graph stay in sync. For the voice layer, we built a custom pipeline: live audio is streamed into our backend, transcribed in real time, and fed into an LLM that extracts location, severity, and emergency type. We implemented RAG over official 911 dispatch and protocol documents so every suggestion is grounded in real procedures. We run a separate Haversine-based proximity ranking algorithm over the graph’s current state—position and availability updated every second—to compute the closest available unit per type and their ETAs; those results are exposed via an API and drive the recommendations and map highlights. The frontend pushes call context and receives back unit IDs and ETAs, and click-to-dispatch is implemented as a map zoom/focus to the chosen vehicle’s live position. Post-call evaluation uses the same RAG-backed LLM: we run the full transcript against the same protocol corpus, detect missed questions and delayed escalations, and score timeliness and critical information capture to produce structured feedback. For dispatcher training, we use the same simulation and map stack and add a separate AI caller so we can generate dynamic scenarios—including rare edge cases—so operators practice in the same resource-constrained, geographically accurate environment they’ll see on a real call. ## Challenges we ran into - Keeping the digital twin and map in sync: The graph updates every second with vehicle positions and availability; the frontend had to consume that feed without jank or drift. We had to design a clear contract between the sim and the map (what gets pushed, when, and how the UI reacts) so the live map felt reliable. - Voice --> structured data in real time: Turning live call audio into clean signals (location, severity, emergency type) for the LLM and RAG was hard—transcription delays, noise, and incomplete sentences meant we had to design the pipeline to handle partial or late updates and still produce usable recommendations. - Grounded recommendations: We wanted every suggestion tied to real protocols, not generic advice. Getting RAG to retrieve the right protocol chunks from long documents and have the LLM use them consistently took a lot of prompt and retrieval tuning. - Realistic simulation without real dispatch data: We don’t have live SF dispatch feeds, so we used open road data and historical 911 crime data to build a plausible, resource-constrained environment. Making that feel “real enough” for training and demos required careful design of the graph and crime injection. ## Accomplishments t [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 147 recognized source files, 718 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Cursor — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 172) ``` .cursor/rules/backend-express-only.mdc .cursor/rules/reference-project-description.mdc .DS_Store .gitignore backend/.DS_Store backend/.gitignore backend/data/.gitkeep backend/data/crime-priority-map.json backend/data/sf-roads-graph.json backend/data/sf-roads.json backend/data/sf-roads.json.bak backend/docs/SIMULATED_VEHICLES.md backend/generated-scenarios.txt backend/hard-scenario-example.txt backend/LIVE_EVAL_RUN.md backend/package.json backend/ragDocs/README.md backend/README.md backend/SCENARIO_GENERATOR_SCHEMA.md backend/scripts/build-crime-priority-map.js backend/scripts/build-roads-graph.js backend/scripts/check-vehicle-positions-update.js backend/scripts/fetch-sf-roads.js backend/scripts/generate-nine-scenarios.js backend/scripts/reset-vehicle-positions.js backend/scripts/simulate-emergency-vehicles.js backend/scripts/trim-sf-crimes-to-2021.js backend/scripts/verify-astar-routing.js backend/scripts/verify-dispatch-priority.js backend/SF-crimes-2013-2015.csv backend/simulate-backend.md backend/src/config.js backend/src/index.js backend/src/routes/callAudio.js backend/src/routes/callEvaluation.js backend/src/routes/crimes.js backend/src/routes/roads.js backend/src/routes/route.js backend/src/routes/scenarios.js backend/src/routes/vehicles.js backend/src/services/crimeClassificationService.js backend/src/services/crimesDataService.js backend/src/services/dispatchPriorityService.js backend/src/services/elevenlabsService.js backend/src/services/evaluationService.js backend/src/services/liveEvalService.js backend/src/services/openaiService.js backend/src/services/openaiTtsService.js backend/src/services/proximityRanking.js backend/src/services/ragService.js backend/src/services/roadRoutingService.js backend/src/services/scenarioGenerator.js backend/src/services/vehicleSimulation.js backend/src/services/whisperService.js backend/src/utils/911audio.js backend/src/utils/geo.js backend/src/utils/officerNames.js frontend/.DS_Store frontend/app/dashboard/page.tsx frontend/app/globals.css frontend/app/layout.tsx frontend/app/page.tsx frontend/app/simulation/[sessionId]/page.tsx frontend/app/simulation/[sessionId]/review/page.tsx frontend/app/simulation/page.tsx frontend/components.json frontend/components/app-shell.tsx frontend/components/audio-control.tsx frontend/components/charts-panel.tsx frontend/components/mic-control.tsx frontend/components/mission-control/collapsible-transcript.tsx frontend/components/mission-control/intelligence-panel.tsx frontend/components/mission-control/status-indicator.tsx frontend/components/mission-control/waveform-visualizer.tsx frontend/components/new-call-modal.tsx frontend/components/notes-panel.tsx frontend/components/providers.tsx frontend/components/scenario-card.tsx frontend/components/score-card.tsx frontend/components/session-detail-drawer.tsx frontend/components/sessions-table.tsx frontend/components/sf-map.tsx frontend/components/theme-provider.tsx frontend/components/transcript-feed.tsx frontend/components/ui/accordion.tsx frontend/components/ui/alert-dialog.tsx frontend/components/ui/alert.tsx frontend/components/ui/aspect-ratio.tsx frontend/components/ui/avatar.tsx frontend/components/ui/badge.tsx frontend/components/ui/breadcrumb.tsx frontend/components/ui/button.tsx frontend/components/ui/calendar.tsx frontend/components/ui/card.tsx frontend/components/ui/carousel.tsx frontend/components/ui/chart.tsx frontend/components/ui/checkbox.tsx frontend/components/ui/collapsible.tsx frontend/components/ui/command.tsx frontend/components/ui/context-menu.tsx frontend/components/ui/dialog.tsx frontend/components/ui/drawer.tsx frontend/components/ui/dropdown-menu.tsx frontend/components/ui/form.tsx frontend/components/ui/hover-card.tsx frontend/components/ui/input-otp.tsx frontend/components/ui/input.tsx frontend/components/ui/label.tsx frontend/components/ui/menubar.tsx frontend/components/ui/navigation-menu.tsx frontend/components/ui/pagination.tsx frontend/components/ui/popover.tsx frontend/components/ui/progress.tsx frontend/components/ui/radio-group.tsx frontend/components/ui/resizable.tsx frontend/components/ui/scroll-area.tsx frontend/components/ui/select.tsx frontend/components/ui/separator.tsx frontend/components/ui/sheet.tsx frontend/components/ui/sidebar.tsx [52 more files omitted for size] ``` ### Dependencies - backend/package.json: cors@^2.8.5, dotenv@^16.4.5, express@^4.21.0, openai@^4.52.0, ws@^8.19.0 - frontend/package.json: @hookform/resolvers@^3.9.1, @radix-ui/react-accordion@1.2.2, @radix-ui/react-alert-dialog@1.1.4, @radix-ui/react-aspect-ratio@1.1.1, @radix-ui/react-avatar@1.1.2, @radix-ui/react-checkbox@1.1.3, @radix-ui/react-collapsible@1.1.2, @radix-ui/react-context-menu@2.2.4, @radix-ui/react-dialog@1.1.4, @radix-ui/react-dropdown-menu@2.1.4, @radix-ui/react-hover-card@1.1.4, @radix-ui/react-label@2.1.1, @radix-ui/react-menubar@1.1.4, @radix-ui/react-navigation-menu@1.2.3, @radix-ui/react-popover@1.1.4, @radix-ui/react-progress@1.1.1, @radix-ui/react-radio-group@1.2.2, @radix-ui/react-scroll-area@1.2.2, @radix-ui/react-select@2.1.4, @radix-ui/react-separator@1.1.1, @radix-ui/react-slider@1.2.2, @radix-ui/react-slot@1.1.1, @radix-ui/react-switch@1.1.2, @radix-ui/react-tabs@1.1.2, @radix-ui/react-toast@1.2.4, @radix-ui/react-toggle@1.1.1, @radix-ui/react-toggle-group@1.1.1, @radix-ui/react-tooltip@1.1.6, @reduxjs/toolkit@^2.11.2, @supabase/ssr@^0.8.0, @supabase/supabase-js@^2.95.3, @tailwindcss/postcss@^4.1.13, @types/maplibre-gl@^1.13.2, @types/node@^22, @types/react@19.2.7, @types/react-dom@19.2.3, autoprefixer@^10.4.20, class-variance-authority@^0.7.1, clsx@^2.1.1, cmdk@1.1.1, date-fns@4.1.0, embla-carousel-react@8.5.1, input-otp@1.4.1, lucide-react@^0.544.0, maplibre-gl@^5.18.0, next@16.1.6, next-themes@^0.4.6, pmtiles@^4.4.0, postcss@^8.5, react@19.2.3, react-day-picker@8.10.1, react-dom@19.2.3, react-hook-form@^7.54.1, react-redux@^9.2.0, react-resizable-panels@^2.1.7, recharts@2.15.0, sonner@^1.7.1, tailwind-merge@^2.5.5, tailwindcss@^3.4.17, tailwindcss-animate@^1.0.7, typescript@5.7.3, vaul@^1.1.2, zod@^3.24.1 ### Recent commits (newest first) - Add project details and features to README - Merge branch 'main' of https://github.com/ericrcwu001/treehacks - shorter opening - make whisper english - Merge branch 'main' of github.com:ericrcwu001/treehacks - slightly better ai notes - scrollbar change - dispatch, ai notes - Implement situation priority retrieval and enhance live simulation dispatch logic - Merge branch 'main' of github.com:ericrcwu001/treehacks - modify closest highlight to represent dispatch, add manual dispatch controls - Enhance map interaction and route visualization - Add route handling and visualization for vehicles - Enhance map point handling for emergency units - Enhance crime beacon handling and map visuals - fixed hydration mismatch - Merge origin/main; resolve .DS_Store, add to gitignore - WIP: roads graph, routing, vehicle sim, frontend api - merge - revamp ui ## Key source files (fetched from GitHub, selected and truncated for size) ### Project Description.md ```markdown 🎯 MVP CONCEPT DOCUMENT Project: DispatchAI - Real-Time 911 Dispatcher Assistant Core Concept An intelligent assistant that analyzes live emergency calls, extracts critical medical information, visualizes available emergency resources spatially, and provides decision support to help dispatchers make faster, more accurate resource allocation decisions. Problem Statement The Dispatcher's Challenge 911 dispatchers face severe cognitive overload during emergency calls. They must simultaneously: Listen to and comprehend distressed, often unclear callers Extract critical information from chaotic descriptions Recognize medical patterns and emergency types Make rapid resource allocation decisions Track which units are available and where they're located Manage multiple simultaneous incidents Provide emotional support and pre-arrival instructions The Cost of Overload 27% of cardiac arrests receive suboptimal initial response 31% of strokes not correctly identified during initial call Critical symptoms (like agonal breathing) missed while dispatcher is typing or searching for resources Wrong resource types dispatched (basic when advanced capabilities needed) Delays in response due to manual resource lookup High dispatcher burnout and 30% annual turnover Solution Overview Three-Layer Intelligent System Layer 1: Live Call Intelligence Analyzes the emergency call in real-time to understand what's actually happening, extracting critical medical information and identifying life-threatening patterns that a human might miss while multitasking. Layer 2: Spatial Resource Awareness Provides instant visual understanding of where all emergency resources are located, their capabilities, current availability status, and estimated response times to the incident location. Layer 3: Decision Support Synthesizes the call analysis and resource data to recommend optimal dispatch decisions, providing the reasoning behind recommendations so the dispatcher remains in control but is informed. Core Features 1. Live Call Evaluation What It Does: Continuously analyzes the emergency call audio and transcript to identify critical medical patterns, symptoms, and escalating severity in real-time. Key Capabilities: Converts spoken words to text as the call happens Identifies medical terminology and symptoms even when described in layman's terms Detects critical indicators in background audio (agonal breathing, distress sounds, environmental hazards) Recognizes patterns that indicate specific emergencies (cardiac arrest, stroke, severe trauma, respiratory distress) Calculates and updates severity scoring as new information emerges Generates alerts for life-threatening conditions Information Extracted: Precise location details (address, apartment number, cross streets, access information) Patient demographics (age, gender, name if provided) Chief complaint in medical terminology Vital status (breathing, consciousness, mobility) Medical history and current medications Timeline of symptom [truncated — 14889 more characters] ``` ### backend/LIVE_EVAL_RUN.md ```markdown # Live Call Evaluation – Architecture & Run Instructions ## Overview Live call evaluation streams operator mic audio to the backend over WebSocket, transcribes it (chunked Whisper or mock), runs a rule-based assessment engine, and streams back **live transcript** and **dispatch recommendations** (EMT/BLS, ALS, Police, Fire, SWAT). No map or resources yet—transcript only. --- ## Architecture ``` ┌─────────────────────────────────────────────────────────────────────────┐ │ Frontend (React) │ │ • Mic capture (MediaRecorder → chunks every 250ms) │ │ • WebSocket client → ws:///live-eval │ │ • Sends: binary audio chunks (or AUDIO_CHUNK with base64) │ │ • Receives: TRANSCRIPT_DELTA, TRANSCRIPT_FINAL, RECOMMENDATION_UPDATE │ │ • UI: live transcript lines + partial; recommendation panel (units) │ └─────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────┐ │ Backend (Node + Express + ws) │ │ • HTTP server: Express + WebSocket upgrade on path /live-eval │ │ • websocket-handler.js: per-connection STT stream + assessment engine │ └─────────────────────────────────────────────────────────────────────────┘ │ │ │ ▼ ▼ ▼ ┌───────────────┐ ┌─────────────────┐ ┌─────────────────────────────────┐ │ stt-stream.js│ │ incident-state │ │ assessment-engine.js │ │ Chunked STT │ │ incident-state │ │ • processFinalTranscript(text) │ │ • Buffer │ │ • fullTranscript │ │ • evaluateRules(transcript) │ │ • Every 2.5s│ │ • signals │ │ • Debounce 1/s; immediate for │ │ • Whisper or │ │ • rationales │ │ critical (shots, no breath, │ │ mock │ │ │ │ fire) │ └───────────────┘ └─────────────────┘ └─────────────────────────────────┘ │ │ │ ▼ │ ┌─────────────────────┐ │ │ policy-rules.js │ │ │ 10–15 rules │ │ │ severity + rationale│ └───────────────────────────────┴──────────────────────┘ ``` ### Backend modules (all under `src/call-evaluation/`) | File | Role | |------|------| | `message-contracts.js` | Message type constants and payload JSDoc (TRANSCRIPT_DELTA, TRANSCRIPT_FINAL, RECOMMENDATION_UPDATE). | | `incident-state.js` | Incident state store: fullTranscript, signals, recommendedUnits, severity, rationales. | | `policy-rules.js` | De [truncated — 3074 more characters] ``` ### backend/package.json ``` { "name": "treehacks-backend", "version": "1.0.0", "description": "911 call simulation backend - generate call audio from scenarios", "main": "src/index.js", "type": "module", "scripts": { "start": "node src/index.js", "dev": "node --watch src/index.js", "fetch-sf-roads": "node scripts/fetch-sf-roads.js", "build-roads-graph": "node scripts/build-roads-graph.js", "simulate-vehicles": "node scripts/simulate-emergency-vehicles.js" }, "dependencies": { "cors": "^2.8.5", "dotenv": "^16.4.5", "express": "^4.21.0", "openai": "^4.52.0", "ws": "^8.19.0" } } ``` ### frontend/package.json ``` { "name": "my-project", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --turbo", "build": "next build", "start": "next start", "lint": "next lint", "setup-fonts": "node scripts/setup-fonts.js" }, "dependencies": { "@hookform/resolvers": "^3.9.1", "@radix-ui/react-accordion": "1.2.2", "@radix-ui/react-alert-dialog": "1.1.4", "@radix-ui/react-aspect-ratio": "1.1.1", "@radix-ui/react-avatar": "1.1.2", "@radix-ui/react-checkbox": "1.1.3", "@radix-ui/react-collapsible": "1.1.2", "@radix-ui/react-context-menu": "2.2.4", "@radix-ui/react-dialog": "1.1.4", "@radix-ui/react-dropdown-menu": "2.1.4", "@radix-ui/react-hover-card": "1.1.4", "@radix-ui/react-label": "2.1.1", "@radix-ui/react-menubar": "1.1.4", "@radix-ui/react-navigation-menu": "1.2.3", "@radix-ui/react-popover": "1.1.4", "@radix-ui/react-progress": "1.1.1", "@radix-ui/react-radio-group": "1.2.2", "@radix-ui/react-scroll-area": "1.2.2", "@radix-ui/react-select": "2.1.4", "@radix-ui/react-separator": "1.1.1", "@radix-ui/react-slider": "1.2.2", "@radix-ui/react-slot": "1.1.1", "@radix-ui/react-switch": "1.1.2", "@radix-ui/react-tabs": "1.1.2", "@radix-ui/react-toast": "1.2.4", "@radix-ui/react-toggle": "1.1.1", "@radix-ui/react-toggle-group": "1.1.1", "@radix-ui/react-tooltip": "1.1.6", "@reduxjs/toolkit": "^2.11.2", "@supabase/ssr": "^0.8.0", "@supabase/supabase-js": "^2.95.3", "autoprefixer": "^10.4.20", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "cmdk": "1.1.1", "date-fns": "4.1.0", "embla-carousel-react": "8.5.1", "input-otp": "1.4.1", "lucide-react": "^0.544.0", "maplibre-gl": "^5.18.0", "next": "16.1.6", "next-themes": "^0.4.6", "pmtiles": "^4.4.0", "react": "19.2.3", "react-day-picker": "8.10.1", "react-dom": "19.2.3", "react-hook-form": "^7.54.1", "react-redux": "^9.2.0", "react-resizable-panels": "^2.1.7", "recharts": "2.15.0", "sonner": "^1.7.1", "tailwind-merge": "^2.5.5", "tailwindcss-animate": "^1.0.7", "vaul": "^1.1.2", "zod": "^3.24.1" }, "devDependencies": { "@tailwindcss/postcss": "^4.1.13", "@types/maplibre-gl": "^1.13.2", "@types/node": "^22", "@types/react": "19.2.7", "@types/react-dom": "19.2.3", "postcss": "^8.5", "tailwindcss": "^3.4.17", "typescript": "5.7.3" }, "pnpm": { "overrides": { "@types/react": "19.2.7", "@types/react-dom": "19.2.3" } } } ``` ### frontend/store/index.ts ```typescript import { configureStore } from "@reduxjs/toolkit" import callReducer from "./slices/callSlice" export const store = configureStore({ reducer: { call: callReducer, }, }) export type RootState = ReturnType export type AppDispatch = typeof store.dispatch ``` ### frontend/app/layout.tsx ```typescript import type { Metadata, Viewport } from "next" import { JetBrains_Mono } from "next/font/google" import { siteConfig } from "@/lib/site-config" import { Providers } from "@/components/providers" import "./globals.css" const jetbrainsMono = JetBrains_Mono({ subsets: ["latin"], variable: "--font-terminal", display: "swap", }) export const metadata: Metadata = { title: `${siteConfig.siteName} - Operator Training Simulator`, description: "Realistic 911 call training simulator with live-call practice, note-taking, scoring, and trainer dashboard.", icons: { icon: siteConfig.favicon, }, } export const viewport: Viewport = { themeColor: "#0c0c0e", width: "device-width", initialScale: 1, } export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode }>) { return ( {children} ) } ``` ### backend/src/index.js ```javascript import http from 'http'; import express from 'express'; import cors from 'cors'; import path from 'path'; import { fileURLToPath } from 'url'; import { config } from './config.js'; import callAudioRouter from './routes/callAudio.js'; import scenariosRouter from './routes/scenarios.js'; import vehiclesRouter from './routes/vehicles.js'; import { startSimulation } from './services/vehicleSimulation.js'; import callEvaluationRouter from './routes/callEvaluation.js'; import crimesRouter from './routes/crimes.js'; import roadsRouter from './routes/roads.js'; import routeRouter from './routes/route.js'; const __dirname = path.dirname(fileURLToPath(import.meta.url)); const app = express(); app.use(cors()); // Larger limit for /interact when sending base64 audio (userInputAudio) app.use(express.json({ limit: '10mb' })); // Serve generated audio files so audioUrl can be used by the frontend const generatedPath = path.join(__dirname, '..', config.generatedAudioDir); app.use(`/${config.generatedAudioDir}`, express.static(generatedPath)); // Health check app.get('/health', (_, res) => { res.json({ status: 'ok', service: '911-call-simulation' }); }); // Generate call audio from scenario app.use('/', callAudioRouter); // Scenario generation (difficulty → full payload for frontend + voice agent) app.use('/api/scenarios', scenariosRouter); // Simulated vehicle positions (poll for map dots) app.use('/api/vehicles', vehiclesRouter); // Live call evaluation: assess transcript for dispatch recommendations (used during simulation) app.use('/api/call-evaluation', callEvaluationRouter); // SF crimes from CSV: time-windowed for map simulation (3x speed) app.use('/api/crimes', crimesRouter); // SF roads and road graph (GeoJSON) for map overlay app.use('/api/roads', roadsRouter); // A* route from point to point (for vehicle path display) app.use('/api/route', routeRouter); const server = http.createServer(app); server.listen(config.port, () => { startSimulation(); console.log(`911 call simulation backend running at http://localhost:${config.port}`); console.log('POST /api/scenarios/generate with body: { "difficulty": "easy"|"medium"|"hard" }'); console.log('POST /generate-call-audio with body: { "scenario": "..." }'); console.log('POST /interact with body: { "scenario", "userInput" or "userInputAudio", optional "conversationHistory" }'); }); ``` ### frontend/app/page.tsx ```typescript "use client" import Image from "next/image" import { useState } from "react" import { useRouter } from "next/navigation" import { createClient } from "@/lib/supabase/client" import { siteConfig } from "@/lib/site-config" import { Button } from "@/components/ui/button" import { Input } from "@/components/ui/input" import { Label } from "@/components/ui/label" import { Radio } from "lucide-react" export default function HomePage() { const router = useRouter() const supabase = createClient() const [mode, setMode] = useState<"signin" | "signup">("signin") const [email, setEmail] = useState("") const [password, setPassword] = useState("") const [loading, setLoading] = useState(false) const [message, setMessage] = useState<{ type: "error" | "success"; text: string } | null>(null) const handleSubmit = async (e: React.FormEvent) => { e.preventDefault() setLoading(true) setMessage(null) try { if (mode === "signup") { const { error } = await supabase.auth.signUp({ email, password }) if (error) throw error router.push("/dashboard") router.refresh() } else { const { error } = await supabase.auth.signInWithPassword({ email, password }) if (error) throw error router.push("/dashboard") router.refresh() } } catch (err: unknown) { setMessage({ type: "error", text: err instanceof Error ? err.message : "Something went wrong.", }) } finally { setLoading(false) } } return (
{siteConfig.logo === "icon" ? (
) : (
)}

{siteConfig.siteName}

{siteConfig.tagline}

setEmail(e.target.value)} required autoComplete="email" className="border-border bg-background font-mono text-sm placeholder:text-muted-foreground" />
setPassword(e.target.value)} required autoComplete={mode === "signup" ? "new-password" : "current-password"} className="border-border bg-background font-mono text-sm placeholder:text-muted-foreground" />
{message && (

{message.type === "error" ? "[ERR]" : "[OK]"} {message.text}

)}
) } ``` ### frontend/app/simulation/page.tsx ```typescript import { redirect } from "next/navigation" export default function SimulationPage() { redirect("/dashboard") } ``` ### frontend/lib/supabase/server.ts ```typescript import { createServerClient } from "@supabase/ssr" import { cookies } from "next/headers" export async function createClient() { const cookieStore = await cookies() return createServerClient( process.env.NEXT_PUBLIC_SUPABASE_URL!, process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY!, { cookies: { getAll() { return cookieStore.getAll() }, setAll(cookiesToSet) { try { cookiesToSet.forEach(({ name, value, options }) => cookieStore.set(name, value, options) ) } catch { // Ignore in Server Components } }, }, } ) } ``` [133 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]