Project Info

Winner

Best Use of ArmorIQ Platform to Build Secure Agents

Aegis

This project did not submit a demo video on Devpost.

Inspiration

911 call centers buckle under mass-casualty events, earthquakes, mass shootings, multi-vehicle pileups — when hundreds of calls arrive in languages dispatchers don't speak, all at once. I wanted to build the co-pilot that lets a single human dispatcher do the work of ten, without ever letting an AI give freelance medical advice.

What it does

Aegis is a real-time dispatch console. A caller speaks in any language; Aegis live-transcribes, translates, classifies the incident, pulls up the correct vetted protocol card (CPR, choking, childbirth, structure fire, etc.), finds the nearest available unit by real road routing, and speaks instructions back in the caller's own language, all in under a few seconds, with a live latency breakdown per pipeline stage. When the system detects a mass emergency (e.g. our earthquake demo), it goes autonomous: every caller gets their own AI dispatch agent, grouped under area "manager" agents that allocate ambulances/police/fire across a zone with no double-booking, all reporting up to a main coordinator agent that keeps global situational awareness, while a human stays one tap away. ArmorIQ secures every agent at every level: each agent's plan is captured and cryptographically authorized via ArmorIQ's Intent Intelligence SDK before it can act, and any deviation from protocol is blocked, not just logged. We also built a live phone-to-laptop demo: open the app on your phone over a Cloudflare tunnel, hit "Talk," and watch the call appear live on the dispatcher dashboard, exactly like a real incoming 911 call.

How we built it

Frontend: Next.js 16 (App Router) + React 19 + TypeScript + Tailwind CSS v4, a dark "operations console" design system, Leaflet + CARTO dark-matter tiles for the live incident map. Gateway: a standalone Node.js + ws WebSocket server brokering mic audio and orchestrating the whole pipeline, separate from the Next.js process for clean real-time streaming. Speech-to-text: Deepgram nova-3 with language=multi, streamed over a raw WebSocket, with PII redaction and interim results for sub-second feedback. LLM: Anthropic Claude,claude-haiku-4-5 on the hot path using structured outputs to guarantee valid JSON for translation, incident classification, and structured-field extraction; claude-sonnet-4-6 to compile the post-call incident report. Prompt compression: The Token Company compresses every prompt before it hits Claude, visible live on the latency meter. Text-to-speech: ElevenLabs multilingual voices speak instructions back to the caller in their own language. Data layer: Redis Stack — RediSearch vector search to confirm the right protocol card, GEOSEARCH for nearest unit/hospital, Streams for the per-call audit trail, Pub/Sub for the live supervisor board — with a zero-dependency in-memory fallback so the whole stack runs without Docker. Routing: OSRM for real road-network routing and ETA to the dispatched unit. Security: ArmorIQ's Intent Intelligence SDK (capturePlan + getIntentToken) signs and authorizes every agent's plan at the caller, manager, and main-coordinator level, with a local protocol-grounded guardrail as a safe fallback. Observability: Sentry traces every call as a transaction with a child span per pipeline stage, retries once on failure, and fires an SLA alert if the loop exceeds 4 seconds. Phone-as-caller: a Cloudflare Tunnel plus a custom Node reverse proxy puts the page and WebSocket gateway behind one HTTPS origin, so a real phone (which requires a secure context for mic access) can be the live caller while the laptop runs the dispatcher view.

Challenges we ran into

Keeping caller-facing instructions safe by construction: Claude only selects and translates a pre-vetted protocol card; it never invents medical advice. That constraint shaped almost every other design decision. Streaming raw mic audio reliably from a phone, over a tunnel, into Deepgram, while keeping the dispatcher's dashboard perfectly in sync, including a tricky React bug where an unstable callbacks object caused an infinite re-render loop on the caller page, traced back to a missing useMemo. Allocating units across simultaneous incidents in an area without double-booking the same ambulance was solved with a greedy nearest-unit assignment inside each area-manager agent. Securing a 3-tier agent hierarchy with ArmorIQ rather than a single agent, so verification happens independently at the caller, manager, and main-coordinator levels.

Accomplishments we're proud of

A full, real (non-mocked) pipeline: live Deepgram transcription, live Claude extraction, live ElevenLabs speech, live Redis geo/vector search, live OSRM routing, all wired together end to end. A working phone-as-911-caller demo that's indistinguishable from a real emergency call coming into the dashboard. A genuine multi-agent hierarchy, not a single LLM call dressed up as "agents", with cryptographically signed, ArmorIQ-verified intent at every level.

What's next

Wire ArmorIQ's blocking path into a live "agent paused, human takeover required" UI state. Auto-resolve background/demo calls so the supervisor board self-cleans over time. Expand the protocol library and add real multi-jurisdiction unit data.

Aegis — Emergency Dispatch Co-Pilot

Real-time 911 dispatch co-pilot. A caller phones in — panicked, noisy, speaking any language. Aegis live-transcribes and translates for an English-speaking dispatcher, surfaces the correct vetted protocol steps, finds the nearest available unit and an appropriate hospital, speaks instructions back to the caller in their language, and keeps a structured, auditable incident record.

It is a human-in-the-loop co-pilot by default, with a documented autonomous "surge mode" for when dispatchers are saturated.

The whole pipeline runs end-to-end with zero API keys and no Docker. Every external service has a mock fallback, and Redis has an in-memory shim — so you can npm run dev and drive a full deterministic demo offline. Drop in real keys to go live, one service at a time.

What's built (Tier 1)

 mic / demo ─▶ WS gateway ─▶ Deepgram (STT, nova-3/multi)
                   │
                   ├─ assemble prompt  ─▶ Token Company (compress)
                   │                          │
                   │                          ▼
                   │                     Claude Haiku  (translate + classify + extract, strict JSON)
                   │                          │
                   ├─ Redis vector  ◀─────────┤  pick / confirm the protocol card
                   ├─ Redis GEO     ◀─────────┘  nearest unit + hospital
                   │
                   └─▶ dispatcher UI: bilingual transcript · incident card · map ·
                       protocol steps · LATENCY METER · speak-to-caller (→ ElevenLabs)

Tier 2 (Sentry tracing, error fallbacks, SLA alerting) and Tier 3 (full surge mode, supervisor board, audit-stream report → PDF) are fully implemented.

Tech stack

Layer	Choice
Frontend	Next.js (App Router) + React + TypeScript + Tailwind v4
Gateway	Node + TypeScript WebSocket server (`ws`), separate process for clean real-time streaming
Map	Leaflet + CARTO dark-matter tiles (no API key)
State/search	Redis Stack (RediSearch vector + GEO) with a zero-dependency in-memory shim
Embeddings	`@xenova/transformers` (all-MiniLM-L6-v2), local; deterministic mock fallback
LLM	`@anthropic-ai/sdk` — `claude-haiku-4-5` (hot path, structured outputs) · `claude-sonnet-4-6` (reports, Tier 3)
STT	Deepgram `nova-3`, `language=multi` (code-switching)
TTS	ElevenLabs `eleven_flash_v2_5` (low-latency multilingual)

Quick start

# 1. install
npm install

# 2. (optional) configure keys — everything works without this
cp .env.example .env        # then edit .env

# 3. run gateway + frontend together
npm run dev

Open http://localhost:3000, pick a scenario in the top bar, and click Start demo.

That's it — with no .env you'll be running fully mocked (in-memory Redis, scripted multilingual transcript, mock translation/extraction, beep TTS) and the entire loop works.

Going live

Add keys to .env and restart npm run dev. Each service independently flips from mock to live based on whether its key is present:

ANTHROPIC_API_KEY → real translation + extraction (claude-haiku-4-5, structured outputs).
DEEPGRAM_API_KEY → real nova-3/multi streaming STT from the Mic button.
ELEVENLABS_API_KEY → real spoken playback (otherwise a beep cue). (mock by default)
TOKEN_COMPANY_API_KEY + TOKEN_COMPANY_API_URL → real prompt compression. (passthrough by default)

The top bar shows a mock/live badge per service so you always know what's active.

Real Redis Stack (optional)

The default in-memory shim needs nothing. To use real Redis Stack instead:

docker compose up -d     # redis/redis-stack on :6379 (+ RedisInsight on :8001)
npm run seed             # seed GEO sets + build the protocol vector index
npm run dev

With REDIS_URL set and reachable, the gateway uses real RediSearch vector + GEO; if it can't connect, it transparently falls back to the in-memory shim.

Phone caller — real mic over a Cloudflare tunnel

Use your phone as the 911 caller (real audio → Deepgram), with the laptop as the dispatch console showing the call live. Mobile browsers only grant mic access over HTTPS, so a free Cloudflare quick-tunnel provides a trusted HTTPS origin, and a small reverse proxy (scripts/tunnel-proxy.mjs) serves the page and the gateway WebSocket under that one origin (no env editing, no separate cert dance).

cloudflared is bundled at ./cloudflared.exe (Windows, downloaded — not committed). Then, with nothing else running:

npm run phone     # runs gateway + frontend + proxy + cloudflared together

Watch the output for the tunnel URL (https://<random>.trycloudflare.com), then:

Phone: open https://<random>.trycloudflare.com/caller → tap the mic button to start talking, tap again to end.
Laptop: open the same tunnel URL / (or just http://localhost:3000) → the dispatcher dashboard shows the phone's call live: transcript, incident card, map, protocol, latency.

Already running npm run dev? Leave it up and start the two extra pieces in separate terminals instead: npm run proxy and npm run tunnel.

How it works: the phone connects as a caller (streams 16 kHz PCM audio); the laptop connects as a dispatcher (viewer + controls). The gateway forwards a caller's full pipeline to every dispatcher console, so one shared call drives both screens. The browser auto-derives wss://<host>/gateway from the page origin.

Environment variables

Variable	Required?	Effect when missing
`DEEPGRAM_API_KEY`	optional	Mic STT runs in mock mode (use Demo)
`ANTHROPIC_API_KEY`	optional	Translation/extraction use keyword mock
`TOKEN_COMPANY_API_KEY` / `TOKEN_COMPANY_API_URL`	optional	Compression uses heuristic passthrough
`ELEVENLABS_API_KEY`	optional	TTS returns a beep cue
`REDIS_URL`	optional	In-memory Redis shim (no Docker needed)
`SENTRY_DSN` / `NEXT_PUBLIC_SENTRY_DSN`	optional	Sentry disabled (Tier 2)
`NEXT_PUBLIC_GATEWAY_WS_URL`	optional	Defaults to `ws://localhost:8080`
`GATEWAY_PORT`	optional	Defaults to `8080`

Toggles: EMBEDDINGS_MOCK=1 forces the mock embedder; REDIS_MOCK=1 forces the in-memory shim; DEEPGRAM_KEYTERMS=1 enables street/landmark keyterm prompting on live STT.

The deterministic demo

Three scripted, fully-offline scenarios (in data/demo/scenarios.json) let you show the whole loop without any audio hardware or keys:

Scenario	Language	Outcome
ES · Cardiac arrest	Spanish	CPR protocol, Priority 1, nearest ambulance + hospital
ZH · Choking (adult)	Mandarin	Choking protocol, Priority 1
EN · Structure fire	English	Structure-fire protocol, nearest fire unit

Pick one in the top bar → Start demo. Watch, in order:

Transcript streams in the caller's language (interim text greyed), then the English translation fills in beneath each line.
Incident card auto-fills (location, nature, consciousness, breathing, hazards) with a priority accent.
Map drops the incident pin and the nearest available unit + hospital, with a route line.
Protocol card highlights the matched, vetted steps. Use Prev/Next to walk them.
Latency meter (top) shows per-stage timing and total loop time.
Speak to caller: type English (or hit Speak current step) → it's translated to the caller's language and played back.

The latency meter (signature)

The top strip is a live, per-stage breakdown (Deepgram / Compression / Claude / Redis·vec / Redis·geo / TTS) with the total loop time in large mono. The Compression ON/OFF toggle visibly changes the Claude segment and the token counts (original → compressed tok) on the next processed segment — the core demonstration of prompt compression's payoff.

Safety design (human-in-the-loop, protocol-grounded)

Instructions delivered to the caller come only from the retrieved, vetted protocol cards (data/protocols.json), never from freeform model generation. Claude's job is to select which protocol applies and to translate its wording — it must not invent medical or first-aid steps. The dispatcher stays in control: they advance protocol steps and approve what is spoken. Surge mode (Tier 3) reads the approved script autonomously and flags the call on the supervisor board for human takeover the moment a dispatcher is free, logging every autonomous action to the per-call audit stream.

Project structure

aegis/
  data/                 protocols, units, hospitals, keyterms, demo scenarios (JSON)
  docker-compose.yml    Redis Stack (optional)
  packages/shared/      shared TS types + WS protocol (source of truth)
  apps/gateway/         WS server, pipeline, service modules (+ mocks), redis shim
    src/services/       deepgram · anthropic · tokenCompany · elevenlabs · redis · embeddings
    src/redis/          in-memory shim + real Redis Stack backend
    scripts/seed.ts     seed real Redis
  apps/frontend/        Next.js dispatcher dashboard (dark ops console)
    components/         TopBar · LatencyMeter · Transcript · IncidentCard · MapPanel ·
                        ProtocolPanel · SpeakToCaller · SupervisorBoard
    lib/                useGateway (WS client) · audio (mic → PCM16)

Live-key configuration

The Token Company (apps/gateway/src/services/tokenCompany.ts) — configure TOKEN_COMPANY_API_URL and TOKEN_COMPANY_API_KEY for real prompt compression.
ElevenLabs — add ELEVENLABS_API_KEY to enable eleven_multilingual_v2 playback.
Deepgram keyterms (apps/gateway/src/services/deepgram.ts) — set DEEPGRAM_KEYTERMS=1 to use keyterm prompting.
Deepgram medical model — conditionally wire it for English-only medical calls if desired.

Project status

Tier 1 — DONE. End-to-end mock-first loop; real Deepgram + Anthropic wired in.
Tier 2 — DONE. Sentry transaction per call with a child span per stage; error capture + retry-once → last-good-protocol fallback + takeover flag; SLA alert when total loop > 4s.
Tier 3 — DONE. Full surge/autonomous mode; live supervisor pub/sub board; audit-stream → claude-sonnet-4-6 structured incident report → printable PDF via browser.

Notes / troubleshooting

@xenova/transformers / sharp on Windows ARM64: the embeddings model pulls sharp (a native module) which may not build on win-arm64. It's declared optional, so install still succeeds and the embedder transparently falls back to the deterministic mock — vector protocol matching still works. To use the real model on a supported platform, ensure the optional dep installed.
First real embeddings call downloads the MiniLM model (~90 MB) once, then caches it.

Analysis

Compare with all teams

View

Metric

Anuraag
2

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

AnthropicIn code
CSSIn code
Next.jsIn code
ReactIn code
RedisIn code
Tailwind CSSIn code
TypeScriptIn code
Node.jsClaimed

7 of 8 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

175 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

16 Verified1 Code-supported

Audit stream to Claude Sonnet incident report to printable outputVerified
Audit-stream to claude-sonnet-4-6 structured incident report to printable PDF via browser
Claimed on readmemedium confidence
- apps/gateway/src/services/report.ts:71 — reads compressed event log/transcript and calls Claude (Sonnet model) to generate a markdown incident report
- apps/frontend/components/ReportView.tsx:14 — handlePrint calls window.print() for PDF export via the browser print dialog, not a server-generated PDF file
Autonomous surge mode: 3-tier agent hierarchy with greedy no-double-booking allocationVerified
Mass emergency mode spins up per-caller agents grouped under area manager agents reporting to a main coordinator, allocating units with no double-booking via greedy nearest-unit assignment
Claimed on Devposthigh confidence
- apps/gateway/src/coordination.ts:73 — greedy nearest-unit assignment using a usedUnits Set to prevent double-booking
- apps/gateway/src/coordination.ts:141 — builds manager-level nodes per zone
- apps/gateway/src/coordination.ts:169 — builds a single main coordinator node
Claude Haiku structured outputs + Sonnet incident reportVerified
claude-haiku-4-5 for translation/classification/extraction with structured JSON outputs; claude-sonnet-4-6 to compile the post-call incident report
Claimed on Devposthigh confidence
- apps/gateway/src/services/anthropic.ts:274 — calls Claude with output_config json_schema using config.models.hot
- apps/gateway/src/config.ts:36 — hot model = claude-haiku-4-5, report model = claude-sonnet-4-6
- apps/gateway/src/services/report.ts:96 — report generation uses config.models.report (Sonnet)
Deterministic offline demo with scripted multilingual scenariosVerified
Three scripted, fully-offline scenarios let you show the whole loop without audio hardware or keys
Claimed on readmehigh confidence
- data/demo/scenarios.json — contains scripted multilingual utterances with fixed delayMs timings
- apps/gateway/src/services/deepgram.ts:43 — startDemo schedules interim/final transcript events deterministically via setTimeout
ElevenLabs multilingual text-to-speechVerified
ElevenLabs multilingual voices speak instructions back to the caller in their own language
Claimed on Devposthigh confidence
- apps/gateway/src/services/elevenlabs.ts:47 — calls ElevenLabs TTS API with eleven_multilingual_v2 model; falls back to a synthesized beep when unkeyed
- apps/gateway/src/config.ts:49 — modelId configured as eleven_multilingual_v2
Leaflet + CARTO dark-matter map with incident/unit/routeVerified
Leaflet + CARTO dark-matter tiles for the live incident map
Claimed on Devposthigh confidence
- apps/frontend/components/MapPanel.tsx:44 — dynamically imports leaflet and uses CARTO dark_all tile URL, renders incident/unit/hospital markers and route polyline
Live latency meter with per-stage timing breakdownVerified
A live latency breakdown per pipeline stage (Deepgram/Compression/Claude/Redis-vec/Redis-geo/TTS) with total loop time
Claimed on Devposthigh confidence
- apps/frontend/components/LatencyMeter.tsx:61 — renders per-stage bars with ms/mock labels, total loop ms, and SLA breach styling
OSRM real road-network routing and ETAVerified
OSRM for real road-network routing and ETA to the dispatched unit
Claimed on Devposthigh confidence
- apps/gateway/src/services/routing.ts:13 — calls router.project-osrm.org/route/v1/driving with a timeout; falls back to a straight-line route with null ETA on failure
Phone-to-laptop live demo via Cloudflare Tunnel and custom reverse proxyVerified
Open the app on your phone over a Cloudflare tunnel, hit Talk, and watch the call appear live on the dispatcher dashboard
Claimed on Devposthigh confidence
- scripts/tunnel-proxy.mjs:39 — Node reverse proxy serving frontend and gateway WebSocket under one origin, including raw-socket WS upgrade piping, meant to run behind cloudflared
- apps/frontend/lib/audio.ts:29 — startMic captures mic audio and converts to 16kHz mono PCM16 for streaming
Protocol-grounded caller instructions (Claude selects/translates only, never invents)Verified
Instructions delivered to the caller come only from vetted protocol cards; Claude selects and translates, never invents medical advice
Claimed on readmehigh confidence
- apps/gateway/src/services/anthropic.ts:256 — system prompt instructs the model to SELECT which protocol applies and not invent medical/first-aid steps, restricted to the provided protocol list
- data/protocols.json — vetted protocol card data referenced as the source of truth for instructions
Real-time speech-to-text with Deepgram nova-3 multi-language, PII redaction, interim resultsVerified
Deepgram nova-3 with language=multi, streamed over raw WebSocket, PII redaction and interim results
Claimed on Devposthigh confidence
- apps/gateway/src/services/deepgram.ts:92 — startRealMic opens a raw WebSocket to wss://api.deepgram.com/v1/listen with interim_results and redact params for pci/ssn/numbers
- apps/gateway/src/config.ts:43 — model set to nova-3, language set to multi
Redis Stack: RediSearch vector search, GEOSEARCH, Streams, Pub/Sub, with in-memory fallbackVerified
Redis Stack: RediSearch vector search to confirm the protocol card, GEOSEARCH for nearest unit/hospital, Streams for audit trail, Pub/Sub for supervisor board, with zero-dependency in-memory fallback
Claimed on Devposthigh confidence
- apps/gateway/src/redis/realRedis.ts:91 — ft.create/ft.search KNN vector index for protocols
- apps/gateway/src/redis/realRedis.ts:128 — geoSearch used for nearest unit/hospital lookup
- apps/gateway/src/redis/realRedis.ts:201 — xAdd/xRange implement Streams for audit trail
- apps/gateway/src/redis/realRedis.ts:224 — publish/subscribe implement Pub/Sub for supervisor board
- apps/gateway/src/redis/inMemory.ts:1 — implements the same RedisBackend interface in-memory (haversine/cosine/EventEmitter) as a zero-dependency fallback
Sentry observability: transaction + child spans, retry-once, SLA alert over 4sVerified
Sentry traces every call as a transaction with a child span per pipeline stage, retries once on failure, fires an SLA alert if the loop exceeds 4 seconds
Claimed on Devposthigh confidence
- apps/gateway/src/util/sentry.ts:85 — withPipelineTransaction wraps the call loop in a Sentry span (op call.pipeline)
- apps/gateway/src/util/sentry.ts:67 — withStageSpan wraps each pipeline stage as a child span
- apps/gateway/src/util/sentry.ts:99 — retryOnce retries exactly once
- apps/gateway/src/util/sentry.ts:50 — captureSlaBreach fires when totalMs exceeds SLA_TOTAL_MS
- packages/shared/src/index.ts:297 — SLA_TOTAL_MS constant set to 4000
Standalone WebSocket gateway server brokering mic audioVerified
A standalone Node.js + ws WebSocket server brokering mic audio and orchestrating the whole pipeline, separate from the Next.js process
Claimed on Devposthigh confidence
- apps/gateway/src/index.ts:30 — creates a standalone WebSocketServer on config.gatewayPort, separate from the Next.js frontend app
Supervisor board for surge mode via Redis pub/subVerified
Live supervisor board showing autonomous calls, backed by Redis Pub/Sub
Claimed on readmehigh confidence
- apps/frontend/components/SupervisorBoard.tsx:1 — renders active calls sorted by priority with surge/autonomous-action/takeover badges
- apps/gateway/src/index.ts:49 — subscribes to subscribeCallStates and broadcasts supervisor payload to dispatcher sockets
The Token Company prompt compressionVerified
The Token Company compresses every prompt before it hits Claude, visible on the latency meter
Claimed on Devposthigh confidence
- apps/gateway/src/services/tokenCompany.ts:63 — POSTs to configured Token Company API URL with bearer key; falls back to local heuristicCompress when unkeyed
ArmorIQ Intent Intelligence SDK signing/authorizing agent plans at 3 levelsCode-supported
ArmorIQ's Intent Intelligence SDK (capturePlan + getIntentToken) signs and authorizes every agent's plan at caller, manager, and main-coordinator level; deviation from protocol is blocked
Claimed on Devpostmedium confidence
- apps/gateway/src/services/armoriq.ts:83 — calls scoped.capturePlan(...) then scoped.getIntentToken(capture, policy), returning blocked/verified verdicts; degrades to a local protocol-grounded check when unkeyed or on SDK failure
- apps/gateway/src/coordination.ts:94 — invoked at caller level
- apps/gateway/src/coordination.ts:144 — invoked at manager level
- apps/gateway/src/coordination.ts:170 — invoked at main coordinator level

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.