Project Info

FrontLine

Inspiration

Several of us have had loved ones injured or in medical distress where help felt delayed—not because responders didn’t care, but because the path from a phone call to meaningful care requires assembling fragmented information, such as the injured person’s identity, location, and condition. In moments of extreme stress, callers often struggle to describe what’s happening, follow verbal instructions, or clearly advocate for the person in need, causing the most critical early moments to be lost. Those minutes matter, and this project grew from a desire to use every available second more effectively by helping initiate the right actions as early as possible, while professional care is still on the way.

What it does

Frontline is a video-first emergency response system that enables individuals in need to visually connect with 911 operators, allowing responders to assess situations beyond what can be conveyed through audio alone. The system integrates live video access via Zoom, which is analyzed in real time using computer vision through Claude to extract environmental and situational context. Caller audio is processed in parallel to capture spoken descriptions and responder instructions. Rather than independently analyzing raw audio, Perplexity serves as a synthesis layer, combining structured outputs from the computer vision pipeline with parsed audio input to generate a consolidated incident report.

How we built it

Mobile frontend (emergency caller app): The user-facing mobile application is built with React Native and TypeScript, creating a frictionless app experience for emergency callers. Web frontend (operator dashboard): The operator interface is built with HTML, CSS, and JavaScript, providing real-time visibility into incoming emergency calls and active responses. Real-time communication: FrontLine’s video streaming capabilities are planned using Zoom Video SDK to enable live video connection between emergency callers and operators during active responses. AI processing: Claude (Anthropic) analyzes real-time images from emergency scenes to extract environmental context, while Perplexity AI synthesizes audio and visual findings into post-call incident reports.

Challenges we ran into

Zoom meeting SDK integration: Encountered difficulties embedding the Zoom Video SDK into the web application for live video streaming between callers and operators, requiring careful configuration of authentication and session management. Poke integration: The main challenge was migrating Poke from a local environment to deployment on Render, requiring adjustments to environment configuration and asynchronous API handling to ensure reliable execution after the transition. Computer vision analysis: Ensuring accurate analysis requires correctly accessing and isolating the appropriate video stream and frames within the Zoom session, specifically distinguishing and processing the caller’s video feed rather than the operator’s, while maintaining real-time performance.

Accomplishments we're proud of

Built a fully functional emergency response app in 36 hours that could have a real positive impact on people's safety. Successfully integrated Claude AI with Zoom to analyze video/images in real-time for emergency situations. Created an intelligent system that processes visual information to help emergency responders make faster, better-informed decisions.

What we learned

How to integrate multiple third-party APIs and SDKs (Claude AI, Zoom SDK, etc.) into a mobile app, handling authentication, real-time data streaming, and making different services communicate seamlessly. Tackled integration challenges in a niche problem space with limited documentation, learning to troubleshoot and problem-solve without established guides. Built a product that addresses a real public safety need, designing with accessibility and ease-of-use in mind for high-stress emergency situations.

What's next

Scene-specific models: Extend computer vision beyond generic scene understanding to detect medically relevant cues (e.g., body posture, bleeding indicators, unconsciousness, fire/smoke presence). Temporal reasoning: Track changes over time (e.g., patient movement, responsiveness) instead of frame-by-frame analysis to improve situational continuity. Low-bandwidth robustness: Add adaptive frame sampling and degraded-vision fallbacks for poor network conditions common in emergencies.

FrontLine

Prerequisites

Node.js (v18+)
Xcode (for iOS development)
CocoaPods: sudo gem install cocoapods

Setup Instructions

1. Clone the Repository

git clone
cd operatorthing

2. Backend Setup

cd backend
npm install
npm run dev

✅ Server runs on http://localhost:3000

3. Operator Dashboard Setup

cd operator-dashboard
npm install
npm start

✅ Dashboard opens at http://localhost:3000 (or 3001 if backend is on 3000)

4. Mobile App Setup (iOS)

cd EmergencyCallerApp
npm install
cd ios
pod install
cd ..
npx react-native run-ios

✅ iOS simulator launches automatically

Running All Services

Terminal 1 - Backend:

cd backend && npm run dev

Terminal 2 - Dashboard:

cd operator-dashboard && npm start

Terminal 3 - Mobile:

cd EmergencyCallerApp && npx react-native run-ios

Project Structure

operatorthing/
├── EmergencyCallerApp/    # React Native iOS app (emergency caller)
├── operator-dashboard/    # React web app (911 operator)
└── backend/               # Node.js API + WebSocket server

Troubleshooting

iOS Simulator won't launch:

Open Xcode → Preferences → Locations
Set Command Line Tools to your Xcode version
Run sudo xcode-select --switch /Applications/Xcode.app

Port already in use:

Backend: Change PORT in backend/src/index.ts
Dashboard: Set PORT=3001 before npm start

Pod install fails:

cd EmergencyCallerApp/ios
pod deintegrate
pod install

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

AnthropicIn code
CSSIn code
ExpressIn code
HTMLIn code
JavaScriptIn code
KotlinIn code
OpenAIIn code
ReactIn code
SwiftIn code
TypeScriptIn code
Node.jsClaimed

10 of 11 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

730 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Repository

sanjanas18/frontline

162 files · 3.0 MB · @ 769d609

Structure

Interface
10 files · 6%
Screens, components and styles rendered to the user.
API & routing
3 files · 2%
Request entry points: routes, handlers and controllers.
Application logic
35 files · 22%
Domain rules, services and shared utilities.
+1 more

Supporting

Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.

Languages

JavaScript41%
HTML35%
TypeScript19%
Markdown3%
CSS2%
XML0%
Other (2)0%

Share of indexed source by file size. Binary and vendored files are excluded.

Dependencies

operator-dashboard/public/zoom-sdk/zoom-sdk-web-5.1.2/Components/package.json

npm · 49

@zoom/meetingsdk
lodash
react
react-dom
react-redux
redux
redux-thunk
+42 more

operator-dashboard/public/zoom-sdk/zoom-sdk-web-5.1.2/Local/package.json

npm · 34

@zoom/meetingsdk
lodash
react
react-dom
react-redux
redux
redux-thunk
+27 more

EmergencyCallerApp/package.json

npm · 25

@react-native/new-app-screen
@zoom/react-native-videosdk
axios
react
react-native
react-native-safe-area-context
react-native-webview
+18 more

backend/package.json

npm · 16

@anthropic-ai/sdk
axios
axois
cors
dotenv
express
jsonwebtoken
openai
socket.io
+7 more

operator-dashboard/package.json

npm · 16

@testing-library/dom
@testing-library/jest-dom
@testing-library/react
@testing-library/user-event
@types/jest
@types/node
@types/react
@types/react-dom
axios
react
react-dom
react-scripts
socket-io
socket.io-client
typescript
web-vitals

operator-dashboard/public/zoom-sdk/zoom-sdk-web-5.1.2/CDN/package.json

npm · 3

3 development-only dependencies.

package.json

npm · 2

@anthropic-ai/sdk
openai

Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.

FrontLine

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments we're proud of

What we learned

What's next

FrontLine

Prerequisites

Setup Instructions

1. Clone the Repository

2. Backend Setup

3. Operator Dashboard Setup

4. Mobile App Setup (iOS)

Running All Services

Project Structure

Troubleshooting

Technology

AI coding agents

Codebase size

Structure

Interface

API & routing

Application logic

Languages

Dependencies