# Project export: Clampsai This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Current security systems only detect security incidents. ClampsAI helps SMBs and homeowners automatically identify threats and take appropriate action through realtime multimodal agentic tool calling. - Devpost: https://devpost.com/software/clampsai - GitHub: https://github.com/agamg/treehacks-clampsai - Video: https://www.youtube.com/embed/a0edlET5y3w?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Agam Gupta (1 commits) ## Devpost submission (written by the team) ### Inspiration Security cameras are everywhere, but they're only useful if someone is actively watching them. We saw this problem firsthand in retail stores and public spaces where security personnel struggle to monitor multiple feeds simultaneously. This inspired us to create ClampsAI - a system that could intelligently monitor multiple cameras and alert security personnel only when necessary. This enables us to build an autonomous system that intelligently monitors multiple security feeds and takes the appropriate action—whether it's notifying emergency services with real-time details, alerting security personnel, or contacting a designated family member in relevant situations. ClampsAI aims to revolutionize response times, enhance situational awareness, and transform security monitoring. ### What it does ClampsAI is a real-time surveillance system that: Monitors multiple cameras to detect threats simultaneously Utilizes parallel AI agents powered by Gemini AI to detect potential threats in each feed Cross-references multiple cameras to reduce false positives Identifies and highlights threatened areas and malicious activities Makes automated calls to appropriate authorities with realtime incident reporting Stores annotated incident data reports for future review and security analysis ### How we built it We built ClampsAI iteratively: Created a WebRTC-based frontend for multi-camera capture Created parallel threat detection agents using Gemini Flash 2.0 by combining multiple security camera streams. Enhanced real-time Multimodal API for Gemini by processing 3 security feeds at once while also handling phone calling. Integrated cross-camera synthesis using reasoning models to determine threat intensity for individual streams and appropriate next steps, and create context rich call content to speak to emergency service providers or emergency contacts (based on threat type identification) Implemented Twilio and Eleven Labs to initiate and execute appropriate calls with extremely low latency. We boast 5-10 second call times to emergency service providers once a threat has been commenced. Once on the phone line with the emergency service provider, we have sub 3 second responses from the multimodal streaming system to respond to natural language queries. Our tech stack: Frontend: NextJS Backend: Google Gemini Flash 2.0 , Flask, Twilio, Eleven Labs Hardware: HD WebCams ### Challenges we ran into One of the largest challenges we faced was with the gemini multimodal live api endpoint receiving only one stream of video. Our goal was to synchronize multiple video streams and perform natural language understanding tasks over them. In order to do this, we had to reverse engineer the Gemini Multimodal Live API to stream live bits of video from each individual streaming node into Gemini 2.0 FLASH in order to replicate the experience of using Multimodal Live. Gemini 2.0 Flash's video understanding with low latency allowed us to accomplish this. Another large challenge we saw had to do with high latency. We have multiple agentic steps with a lot of data coming in continously but need low latency for our realtime phone calling feature. We fixed this by parallelizing our threat analysis and cross-camera synthesis and experimenting with our video capturing pipeline to reduce latency. ### Accomplishments we're proud of Achieved real-time threat detection with minimal latency Re-engineered multimodal Live API using streamed video through Gemini Video understanding API for realtime querying with multiple video streams Successfully reduced false positives through cross-camera synthesis and smart prompting techniques Accomplished accurate and high threat detection capabilities with context-rich deliverables and content to emergency services providers Built a working multi-camera surveillance system that takes action in real-time and attends to the emergency in real-time by calling the right emergency contact. ### What we learned Performance capabilities of real-time language model APIs Gemini reasoning models, tool calling, Twilio for emergency calls, Eleven labs Importance of cross-referencing data to reduce false positives Multimodal input to enhance security application, threat detection, and response ### What's next for ClampsAI Mobile app for remote monitoring Analytics dashboard for security insights Edge computing for faster processing ## README (from the GitHub repository) # ClampsAI A real-time security monitoring system that uses AI to analyze video feeds and automatically alert emergency services when threats are detected. ## What It Does ClampsAI continuously monitors video feeds from security cameras, analyzing each 5-second video chunk for potential threats. When a threat is detected (robbery, theft, violence, etc.), the system automatically places an emergency call to alert authorities with a detailed description of what was observed. ## How It Works ### System Architecture ```mermaid graph TB subgraph Client["Client"] UI[Web Application] Camera[Multi-camera Video Feeds] Recorder[MediaRecorder API] end subgraph Server["Server Services"] subgraph VideoServer["Video Server (Flask :5002)"] VS_API[API Endpoints] VS_Service[VideoService] end subgraph ChatServer["Chat Server (FastAPI :8013)"] CS_API[Chat Completions API] end subgraph OutboundServer["Outbound Server (Node.js :8000)"] OS_API[Outbound Call API] OS_WS[WebSocket Handler] end end subgraph External["External Services"] Gemini[Gemini API
Video Analysis] Twilio[Twilio API
Voice Calls] ElevenLabs[ElevenLabs
Conversational AI] end UI --> Camera Camera --> Recorder Recorder -->|POST /save-video| VS_API VS_API --> VS_Service VS_Service -->|Upload & Analyze| Gemini VS_Service -->|Threat Detected| OS_API OS_API -->|Initiate Call| Twilio Twilio -->|WebSocket Stream| OS_WS OS_WS -->|Connect| ElevenLabs UI -->|POST /chat/completions| CS_API CS_API -->|Query Context| VS_API VS_API -->|Get Analysis| Gemini ``` ### Video Monitoring & Threat Detection Flow ```mermaid sequenceDiagram participant User participant Client participant VideoServer participant VideoService participant Gemini User->>Client: Click "Monitor Security Feeds" Client->>Client: Start MediaRecorder loop Every 5 seconds Client->>Client: Record video chunk Client->>VideoServer: POST /save-video (FormData) VideoServer->>VideoServer: Save to ./videos/ VideoServer->>VideoService: process_video(file_path) VideoService->>Gemini: Upload video file Gemini-->>VideoService: video_file object VideoService->>VideoService: Cache video_file VideoService->>Gemini: analyze_video_threat(video_file) Note over Gemini: Analyze video clip
Return JSON with:
- threat: 0 or 1
- description: narrative Gemini-->>VideoService: {threat: 1, description: "..."} alt Threat Detected VideoService->>VideoService: make_outbound_call(description) end VideoService-->>VideoServer: gemini_response VideoServer-->>Client: JSON response Client->>Client: Display incident card end ``` ### Emergency Response System When a threat is detected, the system automatically initiates an emergency call: ```mermaid flowchart TD Start[Video Chunk Uploaded] --> Analyze[Gemini Video Analysis] Analyze --> Check{Threat Level?} Check -->|Threat = 0| NoThreat[Return Normal Response] NoThreat --> Display1[Frontend: Green Card] Check -->|Threat = 1| ThreatDetected[Threat Detected!] ThreatDetected --> CallService[CallService.make_outbound_call] CallService --> PostCall[POST /outbound-call
Outbound Server] PostCall --> TwilioCall[Twilio API: Create Call] TwilioCall --> TwiML[TwiML Response
WebSocket Stream URL] TwiML --> WSConnect[WebSocket Connection
/outbound-media-stream] WSConnect --> ElevenLabs[Connect to ElevenLabs
Conversational AI] ElevenLabs --> Agent[AI Agent Speaks
Threat Description] Agent --> Emergency[Emergency Services
Receives Automated Call] style ThreatDetected fill:#ff6b6b style Emergency fill:#51cf66 ``` The AI agent speaks directly to emergency responders, providing a detailed description of the threat detected in the video. ### Chat Interface with Video Context Users can query the system about what it has observed: ```mermaid sequenceDiagram participant User participant Client participant ChatServer participant VideoServer participant Gemini User->>Client: "What did you see in the last video?" Client->>ChatServer: POST /chat/completions ChatServer->>ChatServer: Extract user message ChatServer->>VideoServer: POST /query VideoServer->>VideoServer: Get latest video from cache VideoServer->>Gemini: Query video with user question Gemini-->>VideoServer: Video analysis response VideoServer-->>ChatServer: {response: "..."} ChatServer->>ChatServer: Format as chat completion loop Stream words ChatServer-->>Client: SSE chunk (word-by-word) end Client->>User: Display streaming response ``` ## Data Flow ```mermaid flowchart LR A[Raw Video Chunk
5 seconds] --> B[Save to Disk
./videos/] A --> C[Upload to Gemini] C --> D[Video File Object
Cached in Memory] D --> E[Threat Analysis] E --> F{JSON Response} F -->|threat: 0| G[Normal Response] F -->|threat: 1| H[Emergency Call] H --> I[Automated Call
with Description] style H fill:#ff6b6b style I fill:#ffd43b ``` ## Key Components ### Server Services - **Video Server** (Flask): Receives video uploads, processes them with Gemini AI, and detects threats - **Chat Server** (FastAPI): Handles natural language queries about video content - **Outbound Server** (Node.js): Manages emergency calls via Twilio and ElevenLabs ### External AI Services - **Google Gemini**: Analyzes video content and detects threats - **ElevenLabs Conversational AI**: Voice agent that speaks to emergency responders - **Twilio**: Handles the actual phone call infrastructure ### Client - **Next.js Application** (`client/`): React-based web interface for monitoring feeds and viewing incidents - Records 5-second video chunks continuously - Displays threat detection results in real-time ## How to Use 1. **Start the server services:** **Video Server (Flask):** ```bash cd server python3 -m venv venv source venv/bin/activate pip install -r requirements.txt source ../.env python api/video_server.py ``` **Chat Server (FastAPI):** ```bash cd server source venv/bin/activate source ../.env python api/chat_server.py ``` **Outbound Server (Node.js):** ```bash cd server npm install source ../.env node api/outbound_server.js ``` 2. **Start the client:** ```bash cd client npm install npm run dev ``` Then open `http://localhost:3000` in your browser. 3. **The system will:** - Record 5-second video chunks from your camera - Analyze each chunk for threats - Display results in the interface - Automatically call emergency services if a threat is detected ## Configuration All configuration is managed through environment variables (see `ENV_EXAMPLE.md`): - API keys for Gemini, Twilio, and ElevenLabs - Server ports and directories - Model selection ## Detected evidence (automated analysis) Indexed codebase: 67 recognized source files, 204 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Flask (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - HTML (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (79 of 79) ``` .DS_Store .env .gitignore client/.gitignore client/app/globals.css client/app/layout.tsx client/app/page.tsx client/components.json client/components/ui/accordion.tsx client/components/ui/alert-dialog.tsx client/components/ui/alert.tsx client/components/ui/aspect-ratio.tsx client/components/ui/avatar.tsx client/components/ui/badge.tsx client/components/ui/breadcrumb.tsx client/components/ui/button.tsx client/components/ui/calendar.tsx client/components/ui/card.tsx client/components/ui/carousel.tsx client/components/ui/chart.tsx client/components/ui/checkbox.tsx client/components/ui/collapsible.tsx client/components/ui/context-menu.tsx client/components/ui/dialog.tsx client/components/ui/drawer.tsx client/components/ui/dropdown-menu.tsx client/components/ui/form.tsx client/components/ui/hover-card.tsx client/components/ui/input-otp.tsx client/components/ui/input.tsx client/components/ui/label.tsx client/components/ui/menubar.tsx client/components/ui/navigation-menu.tsx client/components/ui/pagination.tsx client/components/ui/popover.tsx client/components/ui/progress.tsx client/components/ui/radio-group.tsx client/components/ui/resizable.tsx client/components/ui/scroll-area.tsx client/components/ui/select.tsx client/components/ui/separator.tsx client/components/ui/sheet.tsx client/components/ui/sidebar.tsx client/components/ui/skeleton.tsx client/components/ui/slider.tsx client/components/ui/sonner.tsx client/components/ui/switch.tsx client/components/ui/table.tsx client/components/ui/tabs.tsx client/components/ui/textarea.tsx client/components/ui/toast.tsx client/components/ui/toaster.tsx client/components/ui/toggle-group.tsx client/components/ui/toggle.tsx client/components/ui/tooltip.tsx client/eslint.config.mjs client/hooks/use-mobile.tsx client/hooks/use-toast.ts client/lib/utils.ts client/next.config.ts client/package.json client/postcss.config.mjs client/README.md client/tailwind.config.ts client/tsconfig.json config/ngrok.yml.example ENV_EXAMPLE.md README.md server/__init__.py server/api/__init__.py server/api/chat_server.py server/api/outbound_server.js server/api/video_server.py server/config.py server/package.json server/requirements.txt server/services/__init__.py server/services/call_service.py server/services/video_service.py ``` ### Dependencies - client/package.json: @eslint/eslintrc@^3, @hookform/resolvers@^4.1.0, @radix-ui/react-accordion@^1.2.3, @radix-ui/react-alert-dialog@^1.1.6, @radix-ui/react-aspect-ratio@^1.1.2, @radix-ui/react-avatar@^1.1.3, @radix-ui/react-checkbox@^1.1.4, @radix-ui/react-collapsible@^1.1.3, @radix-ui/react-context-menu@^2.2.6, @radix-ui/react-dialog@^1.1.6, @radix-ui/react-dropdown-menu@^2.1.6, @radix-ui/react-hover-card@^1.1.6, @radix-ui/react-label@^2.1.2, @radix-ui/react-menubar@^1.1.6, @radix-ui/react-navigation-menu@^1.2.5, @radix-ui/react-popover@^1.1.6, @radix-ui/react-progress@^1.1.2, @radix-ui/react-radio-group@^1.2.3, @radix-ui/react-scroll-area@^1.2.3, @radix-ui/react-select@^2.1.6, @radix-ui/react-separator@^1.1.2, @radix-ui/react-slider@^1.2.3, @radix-ui/react-slot@^1.1.2, @radix-ui/react-switch@^1.1.3, @radix-ui/react-tabs@^1.1.3, @radix-ui/react-toast@^1.2.6, @radix-ui/react-toggle@^1.1.2, @radix-ui/react-toggle-group@^1.1.2, @radix-ui/react-tooltip@^1.1.8, @types/node@^20, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7.1, clsx@^2.1.1, date-fns@^3.6.0, embla-carousel-react@^8.5.2, eslint@^9, eslint-config-next@15.1.0, input-otp@^1.4.2, lucide-react@^0.475.0, next@15.1.0, next-themes@^0.4.4, postcss@^8, react@^19.0.0, react-day-picker@^8.10.1, react-dom@^19.0.0, react-hook-form@^7.54.2, react-resizable-panels@^2.1.7, recharts@^2.15.1, sonner@^1.7.4, tailwind-merge@^3.0.1, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@^5, vaul@^1.1.2, zod@^3.24.2 - server/package.json: @fastify/formbody@^8.0.2, @fastify/websocket@^11.0.2, dotenv@^16.4.7, fastify@^5.2.1, twilio@^5.4.4, ws@^8.18.0 - server/requirements.txt: fastapi@==0.115.3, flask@==3.0.0, flask-cors@==4.0.0, google-generativeai@==0.8.3, groq@==0.18.0, pydantic@==2.9.2, python-dotenv@==1.0.0, requests@==2.31.0, uvicorn@==0.32.0 ### Recent commits (newest first) - remove start.sh - restructuring - why the fuck was node modules at root level - more clean up - code clean up, read mes - done - Add clamps and clamps-backend directories as regular directories instead of git repositories - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### ENV_EXAMPLE.md ```markdown # Environment Variables Copy this to `.env` in the root directory: ```bash # API Keys GEMINI_API_KEY=your_gemini_api_key_here GROQ_API_KEY=your_groq_api_key_here ELEVENLABS_API_KEY=your_elevenlabs_api_key_here ELEVENLABS_AGENT_ID=your_elevenlabs_agent_id_here # Twilio Configuration TWILIO_ACCOUNT_SID=your_twilio_account_sid_here TWILIO_AUTH_TOKEN=your_twilio_auth_token_here TWILIO_PHONE_NUMBER=your_twilio_phone_number_here # Server Configuration VIDEO_SERVER_PORT=5002 CHAT_SERVER_PORT=8013 OUTBOUND_SERVER_PORT=8000 OUTBOUND_URL=http://localhost:8000 # Directories VIDEOS_DIR=./videos VIDEO_STREAMS_DIR=./video_streams # Model Configuration GEMINI_MODEL=gemini-2.0-flash GROQ_MODEL=llama-3.3-70b-versatile # Ngrok Configuration (optional - for exposing servers publicly) NGROK_AUTH_TOKEN=your_ngrok_auth_token_here ``` **Note:** If using ngrok, also copy `config/ngrok.yml.example` to `config/ngrok.yml` and add your ngrok auth token there. ``` ### server/requirements.txt ``` # Core dependencies fastapi==0.115.3 uvicorn==0.32.0 flask==3.0.0 flask-cors==4.0.0 python-dotenv==1.0.0 pydantic==2.9.2 requests==2.31.0 # AI/ML dependencies google-generativeai==0.8.3 groq==0.18.0 # Optional: for video streaming (if needed) # flask-socketio==5.5.1 # opencv-python==4.11.0.86 ``` ### server/package.json ``` { "name": "clamps-server", "version": "1.0.0", "description": "Server services for ClampsAI", "type": "module", "main": "api/outbound_server.js", "scripts": { "start:outbound": "node api/outbound_server.js", "start:video": "python api/video_server.py", "start:chat": "python api/chat_server.py" }, "dependencies": { "@fastify/formbody": "^8.0.2", "@fastify/websocket": "^11.0.2", "dotenv": "^16.4.7", "fastify": "^5.2.1", "twilio": "^5.4.4", "ws": "^8.18.0" } } ``` ### client/package.json ``` { "name": "clamps-client", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --turbopack", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@hookform/resolvers": "^4.1.0", "@radix-ui/react-accordion": "^1.2.3", "@radix-ui/react-alert-dialog": "^1.1.6", "@radix-ui/react-aspect-ratio": "^1.1.2", "@radix-ui/react-avatar": "^1.1.3", "@radix-ui/react-checkbox": "^1.1.4", "@radix-ui/react-collapsible": "^1.1.3", "@radix-ui/react-context-menu": "^2.2.6", "@radix-ui/react-dialog": "^1.1.6", "@radix-ui/react-dropdown-menu": "^2.1.6", "@radix-ui/react-hover-card": "^1.1.6", "@radix-ui/react-label": "^2.1.2", "@radix-ui/react-menubar": "^1.1.6", "@radix-ui/react-navigation-menu": "^1.2.5", "@radix-ui/react-popover": "^1.1.6", "@radix-ui/react-progress": "^1.1.2", "@radix-ui/react-radio-group": "^1.2.3", "@radix-ui/react-scroll-area": "^1.2.3", "@radix-ui/react-select": "^2.1.6", "@radix-ui/react-separator": "^1.1.2", "@radix-ui/react-slider": "^1.2.3", "@radix-ui/react-slot": "^1.1.2", "@radix-ui/react-switch": "^1.1.3", "@radix-ui/react-tabs": "^1.1.3", "@radix-ui/react-toast": "^1.2.6", "@radix-ui/react-toggle": "^1.1.2", "@radix-ui/react-toggle-group": "^1.1.2", "@radix-ui/react-tooltip": "^1.1.8", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "date-fns": "^3.6.0", "embla-carousel-react": "^8.5.2", "input-otp": "^1.4.2", "lucide-react": "^0.475.0", "next": "15.1.0", "next-themes": "^0.4.4", "react": "^19.0.0", "react-day-picker": "^8.10.1", "react-dom": "^19.0.0", "react-hook-form": "^7.54.2", "react-resizable-panels": "^2.1.7", "recharts": "^2.15.1", "sonner": "^1.7.4", "tailwind-merge": "^3.0.1", "tailwindcss-animate": "^1.0.7", "vaul": "^1.1.2", "zod": "^3.24.2" }, "devDependencies": { "@eslint/eslintrc": "^3", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "15.1.0", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### client/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "ClampsAI - Security Monitoring", description: "Real-time security monitoring with AI threat detection", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### client/app/page.tsx ```typescript "use client"; import { useState, useRef, useEffect } from "react"; import { Button } from "@/components/ui/button"; import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"; import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"; import { Badge } from "@/components/ui/badge"; import { AlertCircle, Video, Play, Square, Radio } from "lucide-react"; interface Incident { id: string; timestamp: Date; threat: boolean; description: string; filename?: string; loading?: boolean; } export default function Home() { const [isRecording, setIsRecording] = useState(false); const [cameraStream, setCameraStream] = useState(null); const [incidents, setIncidents] = useState([]); const [videoDevices, setVideoDevices] = useState([]); const [selectedDevice, setSelectedDevice] = useState(""); const videoRef = useRef(null); const mediaRecorderRef = useRef(null); const recordingIntervalRef = useRef(null); const videoCounterRef = useRef(0); useEffect(() => { // Get available video devices navigator.mediaDevices.enumerateDevices().then((devices) => { const videoDevs = devices.filter((d) => d.kind === "videoinput"); setVideoDevices(videoDevs); if (videoDevs.length > 0) { setSelectedDevice(videoDevs[0].deviceId); } }); }, []); useEffect(() => { if (cameraStream && videoRef.current) { videoRef.current.srcObject = cameraStream; } return () => { if (cameraStream) { cameraStream.getTracks().forEach((track) => track.stop()); } }; }, [cameraStream]); const startCamera = async () => { try { const stream = await navigator.mediaDevices.getUserMedia({ video: selectedDevice ? { deviceId: selectedDevice } : true, audio: false, }); setCameraStream(stream); } catch (error) { console.error("Error accessing camera:", error); alert("Failed to access camera. Please check permissions."); } }; const stopCamera = () => { if (cameraStream) { cameraStream.getTracks().forEach((track) => track.stop()); setCameraStream(null); } if (isRecording) { stopRecording(); } }; const uploadVideo = async (blob: Blob): Promise => { const fileName = `${videoCounterRef.current}.webm`; videoCounterRef.current++; const formData = new FormData(); formData.append("video", blob, fileName); // Create loading incident card const incidentId = `incident-${Date.now()}`; const loadingIncident: Incident = { id: incidentId, timestamp: new Date(), threat: false, description: "Analyzing video...", filename: fileName, loading: true, }; setIncidents((prev) => [loadingIncident, ...prev]); try { const response = await fetch("http://localhost:5002/save-video", { method: "POST", body: formData, }); if (!response.ok) { throw new Error(`Failed to save video: ${response.statusText}`); } const result = await response.json(); let geminiData; if (typeof result.gemini_response === "string") { geminiData = JSON.parse(result.gemini_response); } else { geminiData = result.gemini_response; } // Update incident with results setIncidents((prev) => prev.map((inc) => inc.id === incidentId ? { ...inc, threat: geminiData.threat === 1, description: geminiData.description || "No description available", loading: false, } : inc ) ); } catch (error) { console.error("Error uploading video:", error); setIncidents((prev) => prev.map((inc) => inc.id === incidentId ? { ...inc, description: `Error: ${error instanceof Error ? error.message : "Failed to analyze video"}`, loading: false, } : inc ) ); } }; const startRecording = () => { if (!cameraStream) { alert("Please start camera first"); return; } setIsRecording(true); const recordChunk = () => { if (!cameraStream) return; const mimeTypes = [ "video/webm;codecs=vp8", "video/webm;codecs=vp9", "video/webm", "video/mp4", ]; let selectedMimeType = null; for (const type of mimeTypes) { if (MediaRecorder.isTypeSupported(type)) { selectedMimeType = type; break; } } if (!selectedMimeType) { console.error("No supported MIME type found"); return; } const mediaRecorder = new MediaRecorder(cameraStream, { mimeType: selectedMimeType, }); const chunks: Blob[] = []; mediaRecorder.ondataavailable = (e) => { if (e.data && e.data.size > 0) { chunks.push(e.data); } }; mediaRecorder.onstop = () => { const blob = new Blob(chunks, { type: selectedMimeType || "video/webm" }); uploadVideo(blob); }; mediaRecorder.start(); mediaRecorderRef.current = mediaRecorder; setTimeout(() => { if (mediaRecorder.state !== "inactive") { mediaRecorder.stop(); } }, 5000); }; // Record first chunk immediately recordChunk(); // Then record every 5 seconds recordingIntervalRef.current = setInterval(recordChunk, 5000); }; const stopRecording = () => { setIsRecording(false); if (recordingIntervalRef.current) { clearInterval(recordingIntervalRef.current); recordingIntervalRef.current = null; } if (mediaRecorderRef.current && mediaRecorderRef.current.state [truncated — 11473 more characters] ``` ### server/__init__.py ```python """Backend package for ClampsAI.""" ``` ### client/next.config.ts ```typescript import type { NextConfig } from "next"; const nextConfig: NextConfig = { /* config options here */ }; export default nextConfig; ``` ### server/config.py ```python """Configuration management for the server.""" import os from dotenv import load_dotenv load_dotenv() class Config: """Application configuration.""" # API Keys GEMINI_API_KEY = os.getenv("GEMINI_API_KEY") GROQ_API_KEY = os.getenv("GROQ_API_KEY") ELEVENLABS_API_KEY = os.getenv("ELEVENLABS_API_KEY") ELEVENLABS_AGENT_ID = os.getenv("ELEVENLABS_AGENT_ID") # Twilio Configuration TWILIO_ACCOUNT_SID = os.getenv("TWILIO_ACCOUNT_SID") TWILIO_AUTH_TOKEN = os.getenv("TWILIO_AUTH_TOKEN") TWILIO_PHONE_NUMBER = os.getenv("TWILIO_PHONE_NUMBER") # Server Configuration VIDEO_SERVER_PORT = int(os.getenv("VIDEO_SERVER_PORT", "5002")) CHAT_SERVER_PORT = int(os.getenv("CHAT_SERVER_PORT", "8013")) OUTBOUND_SERVER_PORT = int(os.getenv("OUTBOUND_SERVER_PORT", "8000")) OUTBOUND_URL = os.getenv("OUTBOUND_URL", f"http://localhost:{OUTBOUND_SERVER_PORT}") # Directories VIDEOS_DIR = os.getenv("VIDEOS_DIR", "./videos") VIDEO_STREAMS_DIR = os.getenv("VIDEO_STREAMS_DIR", "./video_streams") # Model Configuration GEMINI_MODEL = os.getenv("GEMINI_MODEL", "gemini-2.0-flash") GROQ_MODEL = os.getenv("GROQ_MODEL", "llama-3.3-70b-versatile") @classmethod def validate(cls): """Validate that all required configuration is present.""" required_vars = [ "GEMINI_API_KEY", "TWILIO_ACCOUNT_SID", "TWILIO_AUTH_TOKEN", "TWILIO_PHONE_NUMBER", ] missing = [var for var in required_vars if not getattr(cls, var)] if missing: raise ValueError(f"Missing required environment variables: {', '.join(missing)}") ``` ### client/tailwind.config.ts ```typescript import type { Config } from "tailwindcss"; export default { darkMode: ["class"], content: [ "./pages/**/*.{js,ts,jsx,tsx,mdx}", "./components/**/*.{js,ts,jsx,tsx,mdx}", "./app/**/*.{js,ts,jsx,tsx,mdx}", ], theme: { extend: { colors: { background: 'hsl(var(--background))', foreground: 'hsl(var(--foreground))', card: { DEFAULT: 'hsl(var(--card))', foreground: 'hsl(var(--card-foreground))' }, popover: { DEFAULT: 'hsl(var(--popover))', foreground: 'hsl(var(--popover-foreground))' }, primary: { DEFAULT: 'hsl(var(--primary))', foreground: 'hsl(var(--primary-foreground))' }, secondary: { DEFAULT: 'hsl(var(--secondary))', foreground: 'hsl(var(--secondary-foreground))' }, muted: { DEFAULT: 'hsl(var(--muted))', foreground: 'hsl(var(--muted-foreground))' }, accent: { DEFAULT: 'hsl(var(--accent))', foreground: 'hsl(var(--accent-foreground))' }, destructive: { DEFAULT: 'hsl(var(--destructive))', foreground: 'hsl(var(--destructive-foreground))' }, border: 'hsl(var(--border))', input: 'hsl(var(--input))', ring: 'hsl(var(--ring))', chart: { '1': 'hsl(var(--chart-1))', '2': 'hsl(var(--chart-2))', '3': 'hsl(var(--chart-3))', '4': 'hsl(var(--chart-4))', '5': 'hsl(var(--chart-5))' }, sidebar: { DEFAULT: 'hsl(var(--sidebar-background))', foreground: 'hsl(var(--sidebar-foreground))', primary: 'hsl(var(--sidebar-primary))', 'primary-foreground': 'hsl(var(--sidebar-primary-foreground))', accent: 'hsl(var(--sidebar-accent))', 'accent-foreground': 'hsl(var(--sidebar-accent-foreground))', border: 'hsl(var(--sidebar-border))', ring: 'hsl(var(--sidebar-ring))' } }, borderRadius: { lg: 'var(--radius)', md: 'calc(var(--radius) - 2px)', sm: 'calc(var(--radius) - 4px)' }, keyframes: { 'accordion-down': { from: { height: '0' }, to: { height: 'var(--radix-accordion-content-height)' } }, 'accordion-up': { from: { height: 'var(--radix-accordion-content-height)' }, to: { height: '0' } } }, animation: { 'accordion-down': 'accordion-down 0.2s ease-out', 'accordion-up': 'accordion-up 0.2s ease-out' } } }, plugins: [require("tailwindcss-animate")], } satisfies Config; ``` [58 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]