Project Info

RELAY – Voice, Search, Stream: Save patients from rigs to ER

Inspiration

The biggest problem is lack of information from delays in accessing patients histories and present information gets lost. 1/3 patients are at risk of treatment mistake in ERs and 40% are critical. And this starts in the ambulance. Paramedics see and do everything; by the time the patient reaches the ER, a lot of that context is lost. Studies put missing information in ambulance-to-ER handovers at around 30%. That means wrong or delayed treatments, duplicated work, and risk that could be avoided. There needs to be a better way to find and request information autonomously, accuratly document and stream it right to the hospital so medics can keep their hands and attention on the patient. This is why we built RELAY.

What it does

Relay is an AI-powered emergency response system that turns paramedic voice and background data into a single, real-time handover from ambulance to ER. It addresses the past, present, and future of the patient in one pipeline: so the ER sees the full picture even before the patient arrives. Devices: laptop with microphone in the ambulance, and a browser-based dashboard in the hospital. Past: patient history (hands-off search) Today, finding past context (e.g., allergies, conditions, medications, prior care, etc.,) often means nurses or medics calling GPs and chasing records by phone, or typing notes into official documentation needed already during the ambulance drive. We outsource that to agents so medics stay with the patient. Once we have core information (full name, address, age, gender) from the paramedic’s voice, we trigger in parallel: GP outreach: We resolve the GP/practice (e.g. lookup number by name and location), place a call (ElevenLabs + Twilio), and request data. No medic on the phone. Medical history: We query FHIR R4 (e.g. Synthea for dev; in production, HIE or Particle Health–style APIs) for corresponding patient records. So the team knows before arrival what drugs are safe and what to avoid. Relative data: We stream the relative data to the hospital much faster such that they can be connected sooner and provide information as they are one of the most critical sources. We decided not to use artificial agents here as this requires sensitive handling. All of that is merged into one patient story. Hands-off: documentation, requesting, and consolidation are done by the system and agents, not by someone on a keyboard or phone. Present: what’s happening now (voice → structure, no scribbling) Voice is the only input. A wearable mic streams the audio records of the treating medic. NEMSIS-compliant information is extracted (e.g., demographics, vitals, procedures, medications, impressions) so the ePCR fills itself as the paramedic speaks. No writing, no tapping; such that the medics can keep their hands on the patient through the full ride. Future: what the hospital needs to be ready (stream + speed) Right now the hospital is often called 1–2 minutes out if the medics have the time and gets the full ePCR at the end. We flip that: we stream information to the hospital en route. The hospital sees a live dashboard over WebSockets—active cases, NEMSIS updates, GP and medical-DB responses—and interfaces for urgency-based summaries for the handover. So the ER can prepare earlier: right team, right room, right drugs. Speed comes from (1) parallel GP + DB fetch, (2) continuous stream instead of one report at the end, and (3) less manual documentation so care isn’t delayed. How it fits together Handover gap: By recording and structuring everything from voice and pulling history in parallel, we aim for a complete handover and lower the risk of missing information. Delay in care: Because there’s only so much one human can do on a ride, we outsource documentation and information-gathering to the system and agents; more information is available faster and more accurately. Delay in hospital prep: We use a stream of information so the hospital can prep quicker. Human labor and error: We reduce calling around, handwritten notes, and late data entry by using secure electronic flows and agents to call, orchestrate, search, and consolidate.

How we built it

We built an end‑to‑end ambulance‑to‑hospital pipeline that streams hands‑free voice capture, extracts structured clinical data, and pushes a live hospital prep dashboard. Core stack Frontend: HTML/CSS/JS dashboards for paramedic + hospital views, real‑time updates via WebSockets. Backend: FastAPI + WebSockets, event bus for fan‑out, SQLite for persistence. AI: LLM‑based NEMSIS ePCR extraction and hospital summaries (Anthropic/OpenAI/Modal‑ready via a model‑agnostic client, Perplexity for targeted search). Voice: ElevenLabs streaming transcription. Data: NEMSIS‑compliant JSON, transcript segments with timestamps, and vitals playback for realism. Pipeline Wearable mic streams audio to the backend, producing continuous, timestamped segments. ElevenLabs transcribes in real time, merging partials into committed transcript updates. Each committed chunk updates a NEMSIS‑compliant ePCR record (patient info, vitals, impressions, procedures, meds). Structured data is persisted and pushed over WebSockets so the hospital view stays 1:1 with the ambulance stream. Once core identity is complete (name, age/DOB, address, gender), the system triggers parallel enrichment: medical history lookup and GP outreach. The hospital dashboard renders an inbound critical brief, contraindication warnings, prep actions, live vitals trends, and a full NEMSIS ePCR tab for deeper drill‑down.

Challenges we ran into

Real‑time coordination: Keeping transcript, NEMSIS extraction, and hospital UI in sync with low latency. Data completeness vs. speed: Balancing early summaries with partial information without misleading clinicians. UI clarity under pressure: Designing a dashboard that is dense but still readable in seconds. Integration complexity: Orchestrating voice, LLM extraction, and streaming updates reliably across the pipeline.

Accomplishments we're proud of

A hands‑free, end‑to‑end pipeline from ambulance voice to a live hospital dashboard. NEMSIS‑compliant structured extraction updated in real time. A hospital UI that highlights critical prep actions and contraindications immediately. A system that streams the ePCR 1:1 to the ER before arrival.

What we learned

Software engineering is dead.

What's next

for Relay – Voice, Search, Stream: Save patients from rigs to ER Production‑grade integrations with HIEs and hospital EHR systems. Clinical decision support tuned to EMS protocols and ER workflows. Voice‑first clinician queries for fast retrieval of critical facts. Hospital prep automation (stroke, STEMI, trauma activations) with tighter rules and audit trails. Security + compliance hardening for real clinical deployment. A custom wearable, 3D printed including the microphone. Adding AI-based predictive and diagnostic tools for early warning systems.

Relay

AI Emergency Response System — Automated ePCR for Paramedics.

Relay captures real-time audio from paramedics, transcribes it, and extracts structured NEMSIS v3.5 medical data using LLMs. It enriches patient records by querying FHIR health information exchanges and contacting GPs, then streams everything to a hospital dashboard for live monitoring.

Architecture

Paramedic (audio) ──WebSocket──▶ Transcription ──▶ NEMSIS Extraction (Claude)
                                                          │
                                          ┌───────────────┼───────────────┐
                                          ▼               ▼               ▼
                                    FHIR Lookup     GP Caller       Medical DB
                                          │          (Twilio +        │
                                          │         ElevenLabs)       │
                                          ▼               ▼           ▼
                                     Clinical Insights + Alerts
                                              │
                                        Event Bus (Pub/Sub)
                                              │
                                              ▼
                                    Hospital Dashboard (WebSocket)

Quick Start

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env with your API keys (at minimum ANTHROPIC_API_KEY)

# Run
uvicorn app.main:app --reload

Paramedic UI: http://localhost:8000/
Hospital Dashboard: http://localhost:8000/hospital

Key Environment Variables

Variable	Description	Default
`ANTHROPIC_API_KEY`	Claude API key for NEMSIS extraction	required
`ELEVENLABS_API_KEY`	Voice agent for GP calls	optional
`TWILIO_ACCOUNT_SID` / `TWILIO_AUTH_TOKEN`	Outbound calling	optional
`LLM_PROVIDER`	`auto`, `anthropic`, or `openai`	`auto`
`DATABASE_URL`	PostgreSQL connection string for production	SQLite default
`SEED_DEMO_CASES`	Populate demo cases on startup	`true`
`GP_CALLS_ENABLED`	Enable outbound GP calls	`false`
`DUMMY_MODE`	Mock data generation for development	`false`
`GCP_PROJECT_ID` / `GCP_PUBSUB_TOPIC`	Multi-instance event streaming	optional

See .env.example for the full list.

Project Structure

app/
  main.py              # FastAPI app entrypoint
  config.py            # Environment configuration
  database.py          # SQLite / PostgreSQL abstraction
  routers/
    stream.py          # WebSocket audio streaming + transcription
    cases.py           # REST API for case CRUD
    hospital.py        # Hospital dashboard WebSocket
    gp_call.py         # GP call workflow endpoints
  services/
    nemsis_extractor.py  # Transcript → NEMSIS v3.5 via LLM
    llm.py               # LLM abstraction (Anthropic / OpenAI)
    transcription.py     # Real-time audio transcription
    voice_agent.py       # ElevenLabs + Twilio voice calling
    fhir_client.py       # FHIR R4 patient record queries
    gp_caller.py         # GP contact workflow
    gp_documents.py      # PDF/OCR extraction of GP records
    gp_lookup.py         # GP practice phone lookup
    clinical_insights.py # Clinical alerts and recommendations
    medical_db.py        # Medical history queries
    event_bus.py         # Pub/Sub or in-memory event streaming
  models/
    nemsis.py          # NEMSIS v3.5 data structures
    case.py            # Case API models
    clinical.py        # Clinical decision support models
    transcript.py      # Transcript segment models
static/                # Frontend UIs (paramedic + hospital)
docs/                  # Deployment guides
tests/                 # pytest test suite

Testing

pytest

Requires 60% minimum code coverage. Config in pyproject.toml.

Deployment

See docs/gcp-setup.md for Cloud Run deployment with Cloud SQL and Pub/Sub.

Tech Stack

Backend: FastAPI, WebSockets, Pydantic
AI: Anthropic Claude, OpenAI (fallback), ElevenLabs
Healthcare: FHIR R4, NEMSIS v3.5
Database: SQLite (dev) / PostgreSQL (prod)
Telephony: Twilio
Cloud: GCP Cloud Run, Pub/Sub, Cloud SQL
Python: 3.11+

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

AnthropicIn code
CSSIn code
FastAPIIn code
HTMLIn code
JavaScriptIn code
OpenAIIn code
PythonIn code

7 of 7 appear in the indexed code.

AI coding agents

Claude CodeCommits
CursorCommits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

433 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.