# Project export: Mantis This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Mantis provides real time threat detection to help retail businesses identify and prevent shoplifting as it occurs, protecting your inventory and reducing loss. - Devpost: https://devpost.com/software/mantis-nie02d - GitHub: https://github.com/oviozz/mantis - Video: https://www.youtube.com/embed/JJ5pMXaB2X4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Reka: Best Use of Reka) - Team: 1 GitHub contributor(s) — oviozz (3 commits) ## Devpost submission (written by the team) ### Overview Mantis is an intelligent security system that transforms ordinary cameras into proactive theft prevention tools. By combining real-time computer vision and AI, Mantis detects shoplifting, weapons, and threatening behavior the moment they occur—alerting store owners before losses happen. ### Inspiration We started with one question: Why do most security systems only react after it’s too late? Retailers lose billions every year to theft, yet most cameras just record footage for later review. Humans can’t monitor every feed at once—and by the time incidents are noticed, the culprit is gone. So, we set out to make cameras that could think—systems that recognize suspicious behavior in real time and act before damage is done. What Mantis Does Mantis is an end-to-end intelligent monitoring platform that learns, detects, and alerts: Real-Time Threat Detection: Detects shoplifting, weapons, and aggressive actions within 2 seconds. Face Recognition System: Uses vector embeddings to identify and track repeat offenders across all footage—without storing private identity data. Live Dashboard: LiveKit-powered dashboard overlays detections on the video feed and streams alerts instantly. Smart Querying: Ask questions like “Who stole yesterday?” and get timestamped clips and summaries. Building Mantis taught us invaluable lessons about real-time video processing, multi-agent AI pipelines, and face recognition at scale. Balancing speed, accuracy, and privacy showed us what it takes to move from simple detection to true prevention. How We Built It Architecture Detection Engine: Processes frames at 0.5–1 FPS using specialized AI agents for theft, weapons, and face detection. Face Recognition: Generates and compares facial vector embeddings for cross-incident identification. Live Streaming: Combines LiveKit (for video delivery) and WebSockets (for detection metadata) for synchronized, low-latency updates. Analytics Layer: Uses ChromaDB for semantic vector search and natural-language querying. 💻 Tech Stack Backend: FastAPI + Python AI Models: Hugging Face + Ultralytics Video: OpenCV + LiveKit Database: ChromaDB (for face and clip embeddings) Challenges We Overcame Real-Time Performance at Scale: Achieved <2-second latency across multiple feeds using async job queues, frame sampling, and optimized inference. Accurate Threat Detection: Reduced false positives through a fusion approach combining multiple AI signals. Face Recognition & Privacy: Stored face embeddings separately from personal data, ensuring privacy without losing analytical power. Live Streaming Integration: Engineered a dual-channel pipeline—one for video, one for detection metadata—for seamless synchronization. Accomplishments We’re Proud Of Sub-2-second detection latency—true real-time prevention Face search that scans hours of footage in seconds Multi-agent fusion achieving high accuracy with minimal false alarms A live, intuitive dashboard built for instant response—no training required A system ready for production, not just a demo What We Learned Real-time AI demands millisecond-level optimization—batch processing principles don’t apply 1–2 FPS sampling is the sweet spot for retail: frequent enough to catch incidents, efficient enough for multiple feeds Vector embeddings are game-changing for semantic face tracking and event linking The best AI fails if the UX isn’t intuitive during stressful moments Privacy and ethics must be designed in, not patched later What’s Next Short-Term Mobile alerts with video clips POS integration to detect employee and self-checkout theft Predictive analytics for identifying high-risk locations and times Long-Term Cross-store networks to spot organized retail crime Behavioral analysis for professional shoplifters Edge-deployable systems for privacy-focused retailers Open API for integration with existing security infrastructure Our Vision We envision a world where retail theft becomes economically irrational—where prevention is instant, automated, and intelligent. Mantis isn’t just another security product. It’s the future of proactive protection. ## README (from the GitHub repository) # Mantis — Real-time Shoplifting & Weapon Detection Mantis is an AI-powered surveillance system that detects **shoplifting, weapons, and suspicious behavior in real time**, alerts store owners, and automatically extracts **short, timestamped clips** from long footage for easy review or police. ## Features - ✅ Real-time detection (<2s) of theft, weapons, and aggressive actions - 🔔 Instant alerts to store owners via dashboard or SMS - ✂️ Auto-generates short clips from long recordings (10+ hours) - 🔎 Annotated video frames and forensic summaries ## Demo Footage ### Auto Video Analysis — What happened and how Video Analysis ### Clip Generation from Long Footage — Ready to share with police Clip Generation ## Detected evidence (automated analysis) Indexed codebase: 38 recognized source files, 222 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - React (technology) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel AI SDK (technology) — detected in the code - FastAPI (technology) — claimed on Devpost, not found in the code - Hugging Face (technology) — claimed on Devpost, not found in the code - Python (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (45 of 45) ``` .gitignore components.json convex/_generated/api.d.ts convex/_generated/api.js convex/_generated/dataModel.d.ts convex/_generated/server.d.ts convex/_generated/server.js convex/alerts.ts convex/footageAnalyzer.ts convex/footages.ts convex/schema.ts eslint.config.mjs next.config.ts package.json postcss.config.mjs README.md src/app/(dashboard)/_components/dashboard-sidebar.tsx src/app/(dashboard)/event-breakdown/_components/video-clip-browser.tsx src/app/(dashboard)/event-breakdown/_components/video-clip-generator.tsx src/app/(dashboard)/event-breakdown/page.tsx src/app/(dashboard)/footages/_components/footage-analytics.tsx src/app/(dashboard)/footages/[id]/page.tsx src/app/(dashboard)/footages/page.tsx src/app/(dashboard)/layout.tsx src/app/(dashboard)/overview/_components/camera-model-example src/app/(dashboard)/overview/_components/camera-model.tsx src/app/(dashboard)/overview/_components/live-alerts.tsx src/app/(dashboard)/overview/_components/test-alerts.tsx src/app/(dashboard)/overview/page.tsx src/app/(dashboard)/search/page.tsx src/app/(marketing)/_components/landing-page.tsx src/app/(marketing)/page.tsx src/app/api/add/route.ts src/app/api/generate-clip/route.ts src/app/globals.css src/app/layout.tsx src/app/test/page.tsx src/components/convex-client-provider.tsx src/components/ui/button.tsx src/components/upload-footage.tsx src/lib/supabase.ts src/lib/utils.ts src/store/upload-store.ts src/styles/fonts.ts tsconfig.json ``` ### Dependencies - package.json: @ai-sdk/groq@^2.0.24, @ffmpeg/ffmpeg@^0.12.15, @ffmpeg/util@^0.12.2, @gradio/client@^2.0.0-dev.1, @radix-ui/react-slot@^1.2.3, @supabase/supabase-js@^2.76.1, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, ai@^5.0.79, chromadb@^3.0.17, class-variance-authority@^0.7.1, clsx@^2.1.1, convex@^1.28.0, eslint@^9, eslint-config-next@16.0.0, framer-motion@^12.23.24, lucide-react@^0.548.0, next@16.0.0, react@19.2.0, react-dom@19.2.0, react-icons@^5.5.0, tailwind-merge@^3.3.1, tailwindcss@^4, tw-animate-css@^1.4.0, typescript@^5, zustand@^5.0.8 ### Recent commits (newest first) - Update README.md - update: chromadb, baseten - initial: commit all - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "client", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@ai-sdk/groq": "^2.0.24", "@ffmpeg/ffmpeg": "^0.12.15", "@ffmpeg/util": "^0.12.2", "@gradio/client": "^2.0.0-dev.1", "@radix-ui/react-slot": "^1.2.3", "@supabase/supabase-js": "^2.76.1", "ai": "^5.0.79", "chromadb": "^3.0.17", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "convex": "^1.28.0", "framer-motion": "^12.23.24", "lucide-react": "^0.548.0", "next": "16.0.0", "react": "19.2.0", "react-dom": "19.2.0", "react-icons": "^5.5.0", "tailwind-merge": "^3.3.1", "zustand": "^5.0.8" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.0.0", "tailwindcss": "^4", "tw-animate-css": "^1.4.0", "typescript": "^5" } } ``` ### src/app/layout.tsx ```typescript import type {Metadata} from "next"; import "./globals.css"; import {cn} from "@/lib/utils"; import {outfit} from "@/styles/fonts"; import {ConvexClientProvider} from "@/components/convex-client-provider"; import UploadDialog from "@/components/upload-footage"; export const metadata: Metadata = { title: "Calhacks12.0", description: "project made 10/24/2024", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### convex/_generated/server.js ```javascript /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, componentsGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define a Convex HTTP action. * * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object * as its second. * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`. */ export const httpAction = httpActionGeneric; ``` ### src/app/(marketing)/page.tsx ```typescript import LandingPage from "@/app/(marketing)/_components/landing-page"; export default function MarketingPage() { return } ``` ### src/app/(dashboard)/layout.tsx ```typescript import React from "react"; import DashboardSidebar from "@/app/(dashboard)/_components/dashboard-sidebar"; type DashboardLayoutProps = { children: React.ReactNode } export default function DashboardLayout({ children }: DashboardLayoutProps) { return (
{/* Sidebar */}
{/* Main content */}
{children}
); } ``` ### src/app/test/page.tsx ```typescript "use client"; import {VideoOff} from "lucide-react"; import React, {useEffect, useState, useRef, useCallback} from "react"; import TestAlerts from "@/app/(dashboard)/overview/_components/test-alerts"; import { FaSpinner } from "react-icons/fa"; interface Detection { type: "theft" | "weapon" | "face"; bbox: number[]; // [x, y, width, height] confidence: number; label: string; } interface WSResponse { frame_id: string; ok: boolean; detections: Detection[]; counts: { theft: number; weapon: number; face: number; }; processing_time_ms: number; errors?: string[]; } export default function CameraModel() { const [cameraOn, setCameraOn] = useState(false); const [currentTime, setCurrentTime] = useState(() => new Date()); const [error, setError] = useState(null); const [isLoading, setIsLoading] = useState(false); const [detections, setDetections] = useState([]); const [detectionCounts, setDetectionCounts] = useState({theft: 0, weapon: 0, face: 0}); const [processingTime, setProcessingTime] = useState(0); const [wsConnected, setWsConnected] = useState(false); const videoRef = useRef(null); const canvasRef = useRef(null); const streamRef = useRef(null); const wsRef = useRef(null); const frameIntervalRef = useRef(null); const frameIdRef = useRef(0); // Clock update useEffect(() => { const timer = setInterval(() => { setCurrentTime(new Date()); }, 1000); return () => clearInterval(timer); }, []); // WebSocket connection const connectWebSocket = useCallback(() => { // Update with your actual WebSocket URL const wsUrl = "ws://localhost:8000/analysis/ws/analyze?conf_thresh=0.5"; try { const ws = new WebSocket(wsUrl); ws.onopen = () => { console.log("WebSocket connected"); setWsConnected(true); setError(null); }; ws.onmessage = (event) => { try { const response: WSResponse = JSON.parse(event.data); if (response.ok && response.detections) { setDetections(response.detections); setDetectionCounts(response.counts); setProcessingTime(response.processing_time_ms); } if (response.errors) { console.error("Detection errors:", response.errors); } } catch (err) { console.error("Error parsing WebSocket message:", err); } }; ws.onerror = (err) => { console.error("WebSocket error:", err); setError("WebSocket connection error"); setWsConnected(false); }; ws.onclose = () => { console.log("WebSocket closed"); setWsConnected(false); }; wsRef.current = ws; } catch (err) { console.error("Error creating WebSocket:", err); setError("Failed to connect to analysis server"); } }, []); // Send frame to WebSocket const sendFrameToWS = useCallback(() => { if (!wsRef.current || wsRef.current.readyState !== WebSocket.OPEN) { return; } if (!videoRef.current || !canvasRef.current) { return; } const video = videoRef.current; const canvas = canvasRef.current; const ctx = canvas.getContext("2d"); if (!ctx) return; // Set canvas size to match video canvas.width = video.videoWidth; canvas.height = video.videoHeight; // Draw current video frame to canvas ctx.drawImage(video, 0, 0, canvas.width, canvas.height); // Convert to base64 const base64Image = canvas.toDataURL("image/jpeg", 0.8); // Send to WebSocket const message = { frame_id: `frame_${frameIdRef.current++}`, image_b64: base64Image }; try { wsRef.current.send(JSON.stringify(message)); } catch (err) { console.error("Error sending frame:", err); } }, []); // Draw bounding boxes on canvas const drawBoundingBoxes = useCallback(() => { if (!canvasRef.current || !videoRef.current) return; const canvas = canvasRef.current; const video = videoRef.current; const ctx = canvas.getContext("2d"); if (!ctx) return; // Clear canvas ctx.clearRect(0, 0, canvas.width, canvas.height); // Draw each detection detections.forEach((detection) => { const [x, y, w, h] = detection.bbox; // Determine color based on type let color: string; let bgColor: string; switch (detection.type) { case "theft": color = "#FCD34D"; // yellow bgColor = "rgba(252, 211, 77, 0.2)"; break; case "weapon": color = "#EF4444"; // red bgColor = "rgba(239, 68, 68, 0.2)"; break; case "face": color = "#3B82F6"; // blue bgColor = "rgba(59, 130, 246, 0.2)"; break; default: color = "#9CA3AF"; // gray bgColor = "rgba(156, 163, 175, 0.2)"; } // Check if bbox is normalized (0-1) or pixel coordinates const isNormalized = x <= 1 && y <= 1 && w <= 1 && h <= 1; let boxX, boxY, boxW, boxH; if (isNormalized) { [truncated — 13834 more characters] ``` ### src/app/(dashboard)/overview/page.tsx ```typescript import LiveAlerts from "@/app/(dashboard)/overview/_components/live-alerts"; import CameraModel from "@/app/(dashboard)/overview/_components/camera-model"; export default function VideoIntelligenceDashboard() { return (
{/* Live Feed Section */}

Live Feed

Real-time camera footage

{/* Live Alerts Section */}
); } ``` ### src/app/api/add/route.ts ```typescript import {NextRequest, NextResponse} from "next/server"; import {CloudClient, Collection, Metadata} from "chromadb"; interface AddDataRequest { ids: string[]; documents: string[]; metadatas: Metadata[]; } const chromaClient = new CloudClient(); let myCollection: Collection | null = null; const getMyCollection = async () => { if (!myCollection) { myCollection = await chromaClient.getOrCreateCollection({ name: "myCollection", }); } return myCollection; }; export async function POST(request: NextRequest) { try { const data: AddDataRequest = await request.json(); const collection = await getMyCollection(); await collection.add({ ids: data.ids, documents: data.documents, metadatas: data.metadatas, }); return NextResponse.json({ success: true, message: "Data added successfully", data, }); } catch (error) { console.error(error); return NextResponse.json( {success: false, message: "Failed to add data"}, {status: 500}, ); } } ``` ### src/app/api/generate-clip/route.ts ```typescript import {NextRequest, NextResponse} from "next/server"; import {exec} from "child_process"; import {promisify} from "util"; import fs from "fs/promises"; import path from "path"; import os from "os"; const execAsync = promisify(exec); interface TimestampSegment { start: string; end: string; } function timestampToSeconds(timestamp: string): number { const parts = timestamp.split(":"); if (parts.length === 2) { const [minutes, seconds] = parts.map(Number); return minutes * 60 + seconds; } else if (parts.length === 3) { const [hours, minutes, seconds] = parts.map(Number); return hours * 3600 + minutes * 60 + seconds; } return 0; } function createSegments(timestamps: string[], contextSeconds = 5): TimestampSegment[] { const timestampsInSeconds = timestamps.map(timestampToSeconds).sort((a, b) => a - b); const segments: TimestampSegment[] = []; for (const ts of timestampsInSeconds) { const start = Math.max(0, ts - contextSeconds); const end = ts + contextSeconds; // Merge overlapping segments if (segments.length > 0) { const lastSegment = segments[segments.length - 1]; const lastEnd = timestampToSeconds(lastSegment.end); if (start <= lastEnd) { // Merge with previous segment lastSegment.end = formatSeconds(Math.max(end, lastEnd)); continue; } } segments.push({ start: formatSeconds(start), end: formatSeconds(end), }); } return segments; } function formatSeconds(totalSeconds: number): string { const hours = Math.floor(totalSeconds / 3600); const minutes = Math.floor((totalSeconds % 3600) / 60); const seconds = Math.floor(totalSeconds % 60); if (hours > 0) { return `${hours.toString().padStart(2, "0")}:${minutes .toString() .padStart(2, "0")}:${seconds.toString().padStart(2, "0")}`; } return `${minutes.toString().padStart(2, "0")}:${seconds.toString().padStart(2, "0")}`; } export async function POST(request: NextRequest) { let tempDir: string | null = null; try { const body = await request.json(); const {videoUrl, timestamps} = body; if (!videoUrl || !timestamps || timestamps.length === 0) { return NextResponse.json( {error: "Missing videoUrl or timestamps"}, {status: 400} ); } tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "video-clip-")); const videoPath = path.join(tempDir, "input.mp4"); const videoResponse = await fetch(videoUrl); const videoBuffer = await videoResponse.arrayBuffer(); await fs.writeFile(videoPath, Buffer.from(videoBuffer)); const segments = createSegments(timestamps); const segmentFiles: string[] = []; for (let i = 0; i < segments.length; i++) { const segment = segments[i]; const segmentPath = path.join(tempDir, `segment_${i}.mp4`); // Extract segment using FFmpeg const ffmpegCommand = `ffmpeg -i "${videoPath}" -ss ${segment.start} -to ${segment.end} -c copy -y "${segmentPath}"`; await execAsync(ffmpegCommand); segmentFiles.push(segmentPath); } // Create concat file const concatFilePath = path.join(tempDir, "concat.txt"); const concatContent = segmentFiles.map((file) => `file '${file}'`).join("\n"); await fs.writeFile(concatFilePath, concatContent); // Concatenate all segments const outputPath = path.join(tempDir, "output.mp4"); const concatCommand = `ffmpeg -f concat -safe 0 -i "${concatFilePath}" -c copy -y "${outputPath}"`; await execAsync(concatCommand); // Read the output file const outputBuffer = await fs.readFile(outputPath); // Clean up temp directory await fs.rm(tempDir, {recursive: true, force: true}); tempDir = null; // Return the video file return new NextResponse(outputBuffer, { headers: { "Content-Type": "video/mp4", "Content-Disposition": `attachment; filename="clip-${Date.now()}.mp4"`, }, }); } catch (error) { console.error("Error generating clip:", error); // Clean up on error if (tempDir) { try { await fs.rm(tempDir, {recursive: true, force: true}); } catch (cleanupError) { console.error("Error cleaning up temp directory:", cleanupError); } } return NextResponse.json( {error: "Failed to generate video clip", details: error instanceof Error ? error.message : String(error)}, {status: 500} ); } } ``` ### src/app/(dashboard)/event-breakdown/page.tsx ```typescript "use client"; import React, {useState} from "react"; import { Video, Film, } from "lucide-react"; import {useQuery} from "convex/react"; import {api} from "../../../../convex/_generated/api"; import {Id} from "../../../../convex/_generated/dataModel"; import {formatDateLabel, formatDuration} from "@/lib/utils"; import {VideoClipGeneratorBrowser} from "@/app/(dashboard)/event-breakdown/_components/video-clip-browser"; export default function FootageTimelineBuilder() { const [selectedFootageId, setSelectedFootageId] = useState | null>(null); const availableFootage = useQuery(api.footages.getAll); const getSeverityColor = (severity: string) => { const map: Record = { critical: "text-red-700 bg-red-100 border-red-500", high: "text-orange-700 bg-orange-100 border-orange-500", medium: "text-yellow-700 bg-yellow-100 border-yellow-500", low: "text-blue-700 bg-blue-100 border-blue-500", normal: "text-gray-700 bg-gray-100 border-gray-400", }; return map[severity] || map["normal"]; }; return (
{/* Top section: both headings in one line */}

1. Select Footage

2. Review Events

{/* Left panel - scrollable footage list */}
{availableFootage && availableFootage.length > 0 ? ( availableFootage.map((footage) => (
setSelectedFootageId(footage._id)} className={`border overflow-hidden ${ selectedFootageId === footage._id ? "border-blue-600 bg-blue-50" : "border-gray-200 bg-white hover:border-gray-400" } cursor-pointer transition-all`} >

File name: {footage.footageName}

{formatDateLabel(footage._creationTime)}

)) ) : (

No footage available

)}
{/* Right side - event list and generation */}
{!selectedFootageId ? (
) : (