Project Info

Clampsai

Inspiration

Security cameras are everywhere, but they're only useful if someone is actively watching them. We saw this problem firsthand in retail stores and public spaces where security personnel struggle to monitor multiple feeds simultaneously. This inspired us to create ClampsAI - a system that could intelligently monitor multiple cameras and alert security personnel only when necessary. This enables us to build an autonomous system that intelligently monitors multiple security feeds and takes the appropriate action—whether it's notifying emergency services with real-time details, alerting security personnel, or contacting a designated family member in relevant situations. ClampsAI aims to revolutionize response times, enhance situational awareness, and transform security monitoring.

What it does

ClampsAI is a real-time surveillance system that: Monitors multiple cameras to detect threats simultaneously Utilizes parallel AI agents powered by Gemini AI to detect potential threats in each feed Cross-references multiple cameras to reduce false positives Identifies and highlights threatened areas and malicious activities Makes automated calls to appropriate authorities with realtime incident reporting Stores annotated incident data reports for future review and security analysis

How we built it

We built ClampsAI iteratively: Created a WebRTC-based frontend for multi-camera capture Created parallel threat detection agents using Gemini Flash 2.0 by combining multiple security camera streams. Enhanced real-time Multimodal API for Gemini by processing 3 security feeds at once while also handling phone calling. Integrated cross-camera synthesis using reasoning models to determine threat intensity for individual streams and appropriate next steps, and create context rich call content to speak to emergency service providers or emergency contacts (based on threat type identification) Implemented Twilio and Eleven Labs to initiate and execute appropriate calls with extremely low latency. We boast 5-10 second call times to emergency service providers once a threat has been commenced. Once on the phone line with the emergency service provider, we have sub 3 second responses from the multimodal streaming system to respond to natural language queries. Our tech stack: Frontend: NextJS Backend: Google Gemini Flash 2.0 , Flask, Twilio, Eleven Labs Hardware: HD WebCams

Challenges we ran into

One of the largest challenges we faced was with the gemini multimodal live api endpoint receiving only one stream of video. Our goal was to synchronize multiple video streams and perform natural language understanding tasks over them. In order to do this, we had to reverse engineer the Gemini Multimodal Live API to stream live bits of video from each individual streaming node into Gemini 2.0 FLASH in order to replicate the experience of using Multimodal Live. Gemini 2.0 Flash's video understanding with low latency allowed us to accomplish this. Another large challenge we saw had to do with high latency. We have multiple agentic steps with a lot of data coming in continously but need low latency for our realtime phone calling feature. We fixed this by parallelizing our threat analysis and cross-camera synthesis and experimenting with our video capturing pipeline to reduce latency.

Accomplishments we're proud of

Achieved real-time threat detection with minimal latency Re-engineered multimodal Live API using streamed video through Gemini Video understanding API for realtime querying with multiple video streams Successfully reduced false positives through cross-camera synthesis and smart prompting techniques Accomplished accurate and high threat detection capabilities with context-rich deliverables and content to emergency services providers Built a working multi-camera surveillance system that takes action in real-time and attends to the emergency in real-time by calling the right emergency contact.

What we learned

Performance capabilities of real-time language model APIs Gemini reasoning models, tool calling, Twilio for emergency calls, Eleven labs Importance of cross-referencing data to reduce false positives Multimodal input to enhance security application, threat detection, and response

What's next

for ClampsAI Mobile app for remote monitoring Analytics dashboard for security insights Edge computing for faster processing

ClampsAI

A real-time security monitoring system that uses AI to analyze video feeds and automatically alert emergency services when threats are detected.

What It Does

ClampsAI continuously monitors video feeds from security cameras, analyzing each 5-second video chunk for potential threats. When a threat is detected (robbery, theft, violence, etc.), the system automatically places an emergency call to alert authorities with a detailed description of what was observed.

How It Works

System Architecture

graph TB
    subgraph Client["Client"]
        UI[Web Application]
        Camera[Multi-camera Video Feeds]
        Recorder[MediaRecorder API]
    end
    
    subgraph Server["Server Services"]
        subgraph VideoServer["Video Server (Flask :5002)"]
            VS_API[API Endpoints]
            VS_Service[VideoService]
        end
        
        subgraph ChatServer["Chat Server (FastAPI :8013)"]
            CS_API[Chat Completions API]
        end
        
        subgraph OutboundServer["Outbound Server (Node.js :8000)"]
            OS_API[Outbound Call API]
            OS_WS[WebSocket Handler]
        end
    end
    
    subgraph External["External Services"]
        Gemini[Gemini API<br/>Video Analysis]
        Twilio[Twilio API<br/>Voice Calls]
        ElevenLabs[ElevenLabs<br/>Conversational AI]
    end
    
    UI --> Camera
    Camera --> Recorder
    Recorder -->|POST /save-video| VS_API
    VS_API --> VS_Service
    VS_Service -->|Upload & Analyze| Gemini
    VS_Service -->|Threat Detected| OS_API
    OS_API -->|Initiate Call| Twilio
    Twilio -->|WebSocket Stream| OS_WS
    OS_WS -->|Connect| ElevenLabs
    UI -->|POST /chat/completions| CS_API
    CS_API -->|Query Context| VS_API
    VS_API -->|Get Analysis| Gemini

Video Monitoring & Threat Detection Flow

sequenceDiagram
    participant User
    participant Client
    participant VideoServer
    participant VideoService
    participant Gemini
    
    User->>Client: Click "Monitor Security Feeds"
    Client->>Client: Start MediaRecorder
    loop Every 5 seconds
        Client->>Client: Record video chunk
        Client->>VideoServer: POST /save-video (FormData)
        VideoServer->>VideoServer: Save to ./videos/
        VideoServer->>VideoService: process_video(file_path)
        VideoService->>Gemini: Upload video file
        Gemini-->>VideoService: video_file object
        VideoService->>VideoService: Cache video_file
        VideoService->>Gemini: analyze_video_threat(video_file)
        Note over Gemini: Analyze video clip<br/>Return JSON with:<br/>- threat: 0 or 1<br/>- description: narrative
        Gemini-->>VideoService: {threat: 1, description: "..."}
        alt Threat Detected
            VideoService->>VideoService: make_outbound_call(description)
        end
        VideoService-->>VideoServer: gemini_response
        VideoServer-->>Client: JSON response
        Client->>Client: Display incident card
    end

Emergency Response System

When a threat is detected, the system automatically initiates an emergency call:

flowchart TD
    Start[Video Chunk Uploaded] --> Analyze[Gemini Video Analysis]
    Analyze --> Check{Threat Level?}
    
    Check -->|Threat = 0| NoThreat[Return Normal Response]
    NoThreat --> Display1[Frontend: Green Card]
    
    Check -->|Threat = 1| ThreatDetected[Threat Detected!]
    ThreatDetected --> CallService[CallService.make_outbound_call]
    
    CallService --> PostCall[POST /outbound-call<br/>Outbound Server]
    PostCall --> TwilioCall[Twilio API: Create Call]
    TwilioCall --> TwiML[TwiML Response<br/>WebSocket Stream URL]
    TwiML --> WSConnect[WebSocket Connection<br/>/outbound-media-stream]
    WSConnect --> ElevenLabs[Connect to ElevenLabs<br/>Conversational AI]
    ElevenLabs --> Agent[AI Agent Speaks<br/>Threat Description]
    Agent --> Emergency[Emergency Services<br/>Receives Automated Call]
    
    style ThreatDetected fill:#ff6b6b
    style Emergency fill:#51cf66

The AI agent speaks directly to emergency responders, providing a detailed description of the threat detected in the video.

Chat Interface with Video Context

Users can query the system about what it has observed:

sequenceDiagram
    participant User
    participant Client
    participant ChatServer
    participant VideoServer
    participant Gemini
    
    User->>Client: "What did you see in the last video?"
    Client->>ChatServer: POST /chat/completions
    ChatServer->>ChatServer: Extract user message
    ChatServer->>VideoServer: POST /query
    VideoServer->>VideoServer: Get latest video from cache
    VideoServer->>Gemini: Query video with user question
    Gemini-->>VideoServer: Video analysis response
    VideoServer-->>ChatServer: {response: "..."}
    ChatServer->>ChatServer: Format as chat completion
    loop Stream words
        ChatServer-->>Client: SSE chunk (word-by-word)
    end
    Client->>User: Display streaming response

Data Flow

flowchart LR
    A[Raw Video Chunk<br/>5 seconds] --> B[Save to Disk<br/>./videos/]
    A --> C[Upload to Gemini]
    C --> D[Video File Object<br/>Cached in Memory]
    D --> E[Threat Analysis]
    E --> F{JSON Response}
    F -->|threat: 0| G[Normal Response]
    F -->|threat: 1| H[Emergency Call]
    H --> I[Automated Call<br/>with Description]
    
    style H fill:#ff6b6b
    style I fill:#ffd43b

Key Components

Server Services

Video Server (Flask): Receives video uploads, processes them with Gemini AI, and detects threats
Chat Server (FastAPI): Handles natural language queries about video content
Outbound Server (Node.js): Manages emergency calls via Twilio and ElevenLabs

External AI Services

Google Gemini: Analyzes video content and detects threats
ElevenLabs Conversational AI: Voice agent that speaks to emergency responders
Twilio: Handles the actual phone call infrastructure

Client

Next.js Application (client/): React-based web interface for monitoring feeds and viewing incidents
Records 5-second video chunks continuously
Displays threat detection results in real-time

How to Use

Start the server services:

Video Server (Flask):

cd server
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
source ../.env
python api/video_server.py

Chat Server (FastAPI):

cd server
source venv/bin/activate
source ../.env
python api/chat_server.py

Outbound Server (Node.js):

cd server
npm install
source ../.env
node api/outbound_server.js

Start the client:
```
cd client
npm install
npm run dev
```
Then open http://localhost:3000 in your browser.
The system will:
- Record 5-second video chunks from your camera
- Analyze each chunk for threats
- Display results in the interface
- Automatically call emergency services if a threat is detected

Configuration

All configuration is managed through environment variables (see ENV_EXAMPLE.md):

API keys for Gemini, Twilio, and ElevenLabs
Server ports and directories
Model selection

Analysis

Compare with all teams

View

Metric

Agam Gupta
1

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CSSIn code
FastAPIIn code
FlaskIn code
JavaScriptIn code
Next.jsIn code
PythonIn code
ReactIn code
Tailwind CSSIn code
TypeScriptIn code
Google GeminiClaimed
HTMLClaimed

9 of 11 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

204 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.