# Project export: Watchdog This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Help us make our schools safer. - Devpost: https://devpost.com/software/watchdog-2vwqlf - GitHub: https://github.com/oviozz/WatchDog-2024-Hackathon-Berkeley - Result: winner (Hyperbolic: AI Inference Application Bounty) - Team: 1 GitHub contributor(s) — bazalu (3 commits) ## Devpost submission (written by the team) ### Overview What inspired us to build it Guns are now the leading cause of death among American children and teens, with 1 in every 10 gun deaths occurring in individuals aged 19 or younger. School shootings, in particular, have become a tragic epidemic in the U.S., underscoring the urgent need for enhanced safety measures. Our team united with a shared vision to leverage AI technology to improve security in American schools, helping to protect children and ensure their safety. ### What it does Our product leverages advanced AI technology to enhance school safety by detecting potential threats in real-time. By streaming surveillance footage, our AI system can identify weapons, providing instant alerts to security personnel and administrators. In addition to visual monitoring, we integrate audio streaming to analyze changes in sentiment, such as raised voices or signs of distress. This dual approach—combining visual and auditory cues—enables rapid response to emerging threats. ### How we built it We partnered with incredible sponsors—Deepgram, Hyperbolic, Groq, and Fetch.AI—to develop a comprehensive security solution that uses cutting-edge AI technologies. With their support, we were able to conduct fast AI inference, deploy an emergency contact agent, and create intelligent systems capable of tracking potential threats and key variables, all to ensure the safety of our communities. For real-time data processing, we utilized Firebase and Convex to enable rapid write-back and retrieval of critical information. Additionally, we trained our weapon detection agent using Ultralytics YOLO v8 on the Roboflow platform, achieving an impressive ~90% accuracy. This high-performance detection system, combined with AI-driven analytics, provides a robust safety infrastructure capable of identifying and responding to threats in real time. ### Challenges we ran into Streaming a real-time AI object detection model with both low latency and high accuracy was a significant challenge. Initially, we experimented with Flask and FastAPI for serving our model, followed by trying AWS and Docker to improve performance. However, after further optimization efforts, we ultimately integrated Roboflow.js directly in the browser using a Native SDK. This approach gave us a substantial advantage, allowing us to run the model efficiently within the client environment. As a result, we achieved the ability to track weapons quickly and accurately in real time, meeting the critical demands of our security solution. ### Accomplishments we're proud of We are incredibly proud of the features our product offers, providing a comprehensive and fully integrated security experience. Beyond detecting weapons and issuing instant alerts to law enforcement, faculty, and students through AI-powered agents, we also implemented extensive sentiment analysis. This enables us to detect emotional escalations that may signal potential threats. All of this is supported by real-time security data displays, ensuring that key decision-makers are always informed with up-to-the-minute information. Our system seamlessly brings together cutting-edge AI and real-time data processing to deliver a robust, proactive security solution. ### What we learned We learned that the night is darkest right before the dawn... and that we need to persevere and be steadfast as a team to see our vision come to fruition. ### What's next We want to get incorporated in the American school system! ## README (from the GitHub repository) # 🛡️ Watchdog: AI-Powered School Safety System ## Images **Weapon Detection** ![image](https://github.com/user-attachments/assets/32f33874-5178-4714-b463-b8d0b7773938) **Alert System** ![image](https://github.com/user-attachments/assets/54bd45f7-bb25-48e7-8fad-921ca3f0f56b) **Police Report** ![image](https://github.com/user-attachments/assets/98cb0faa-9535-437f-80f2-191a4161c948) **Real-time voice analysis** ![image](https://github.com/user-attachments/assets/62ca2428-68cd-4ee6-bae9-02292b43447d) ![image](https://github.com/user-attachments/assets/729caff5-40ac-4ad1-ac40-7c7328230e11) ## 🚀 Inspiration Gun violence is now the leading cause of death among American children and teens, with 1 in every 10 gun fatalities involving individuals aged 19 or younger. In the U.S., school shootings have escalated into a tragic epidemic, underscoring the urgent need for enhanced security in schools. United by a shared vision, our team leveraged AI technology to create a platform aimed at improving school safety, protecting children, and giving peace of mind to students, parents, and educators alike. ## 🎯 What It Does Watchdog uses advanced AI to enhance school security by detecting threats in real-time. Our platform streams live surveillance footage, leveraging AI to identify weapons and other indicators of potential violence, and issues instant alerts to security personnel and administrators. Additionally, our audio streaming feature detects sentiment changes, such as raised voices or distress signals, allowing security teams to respond rapidly to unfolding threats. By combining visual and auditory cues, Watchdog offers a comprehensive and proactive approach to school safety. ## 🛠️ How We Built It With the support of our incredible sponsors—Deepgram, Hyperbolic, Groq, and Fetch.AI—we developed a robust AI security solution. Here’s an overview of our tech stack and approach: - **Real-Time Data Processing**: Integrated Firebase and Convex for rapid data retrieval and write-back to facilitate real-time communication and alerts. - **Weapon Detection**: Trained an agent using Ultralytics YOLO v8 on the Roboflow platform, achieving ~90% accuracy in weapon detection. - **Audio Analysis**: Implemented sentiment analysis through Deepgram’s API to detect audio patterns like raised voices, signaling potential distress. - **Model Deployment**: We experimented with Flask and FastAPI for model serving and explored AWS and Docker to optimize performance. Ultimately, we chose to implement Roboflow.js directly in the browser with a Native SDK, enabling fast and efficient model inference on the client side. ## 🧩 Challenges We Encountered Achieving low latency and high accuracy with real-time AI object detection presented challenges, especially during initial model deployment. We experimented with various backends and infrastructures but ultimately found that running the model directly in the browser yielded optimal results. By leveraging Roboflow.js's Native SDK, we achieved efficient tracking with fast response times, meeting the stringent performance requirements of our safety solution. ## 🏆 Key Accomplishments We’re proud of the capabilities we built into Watchdog: - **Instant Weapon Detection**: Our AI agents quickly detect weapons and alert law enforcement, faculty, and students in real-time. - **Sentiment Analysis**: Integrated sentiment detection to identify emotional escalations that could indicate potential threats. - **Comprehensive Data Display**: We developed real-time data dashboards that provide school officials and security teams with up-to-the-minute information, helping them make informed decisions swiftly. - **Integrated AI-Driven Safety Solution**: Bringing together visual and auditory detection in a unified system, Watchdog offers a robust, proactive approach to school security. ## 💡 What We Learned Building Watchdog reinforced our belief in the power of perseverance. We learned that each challenge pushed us closer to our goal, and that by standing firm as a team, we could achieve our vision of a safer learning environment. ## 🚀 What's Next for Watchdog Our ultimate vision is to see Watchdog incorporated into the American school system, providing a safer educational environment across the nation. --- ### 🛠️ Built With - **AI & Machine Learning**: Ultralytics YOLO v8, Roboflow - **Audio Analysis**: Deepgram API - **Data Management**: Firebase, Convex - **Backend**: Roboflow.js Native SDK, Flask, FastAPI, AWS, Docker - **Real-Time Streaming & Dashboard**: Custom JavaScript integrations with Convex --- ## Detected evidence (automated analysis) Indexed codebase: 51 recognized source files, 122 KB. - CSS (language) — detected in the code - Firebase (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Express (technology) — claimed on Devpost, not found in the code - Python (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (60 of 60) ``` .eslintrc.json .gitignore .idea/.gitignore .idea/inspectionProfiles/Project_Default.xml .idea/modules.xml .idea/vcs.xml .idea/watchdog.iml components.json convex/_generated/api.d.ts convex/_generated/api.js convex/_generated/dataModel.d.ts convex/_generated/server.d.ts convex/_generated/server.js convex/imageAnalysis.js convex/schema.js convex/securityAlert.js jsconfig.json next.config.mjs package.json postcss.config.mjs README.md src/app/(dashboard)/_components/left/ActionsButtons.jsx src/app/(dashboard)/_components/left/BuildingInfo.jsx src/app/(dashboard)/_components/left/EmergencyResources.jsx src/app/(dashboard)/_components/left/RecentThreats.jsx src/app/(dashboard)/_components/left/ThreatIndicator.jsx src/app/(dashboard)/_components/LeftSide.jsx src/app/(dashboard)/_components/right/BuildingMap.jsx src/app/(dashboard)/_components/right/LiveCameraDashboard.jsx src/app/(dashboard)/_components/right/LiveCameraFeed.jsx src/app/(dashboard)/_components/right/SecurityLog.jsx src/app/(dashboard)/_components/RightSide.jsx src/app/(dashboard)/page.jsx src/app/api/deepgram/route.js src/app/api/transcribe/route.js src/app/cctv/page.jsx src/app/cctv/WeaponObjectDetectionVideo.jsx src/app/globals.css src/app/layout.js src/app/voice/layout.js src/app/voice/page.jsx src/app/voice/SecurityAudioAnalysis.jsx src/app/WeaponObjectDetectionVideo.css src/assests/fonts.js src/components/AudioVisualizerDesign.jsx src/components/AudioVisualizerIndicator.jsx src/components/layout/navbar.jsx src/components/ui/alert.jsx src/components/ui/badge.jsx src/components/ui/button.jsx src/components/ui/card.jsx src/components/ui/dialog.jsx src/components/ui/resizable.jsx src/components/WeaponAlertDialog.jsx src/lib/groq/config.js src/lib/utils.js src/providers/ConvexClientProvider.jsx src/providers/DeepgramContextProvider.jsx src/providers/MicrophoneContextProvider.jsx tailwind.config.js ``` ### Dependencies - package.json: @deepgram/sdk@^3.8.1, @radix-ui/react-dialog@^1.1.2, @radix-ui/react-icons@^1.3.0, @radix-ui/react-slot@^1.1.0, @tensorflow-models/coco-ssd@^2.2.3, @tensorflow/tfjs@^4.21.0, @uploadthing/react@^7.0.3, class-variance-authority@^0.7.0, clsx@^2.1.1, convex@^1.16.6, cross-fetch@^4.0.0, dotenv@^16.4.5, eslint@^8, eslint-config-next@14.2.15, firebase@^10.14.1, firebase-admin@^12.6.0, framer-motion@^11.11.9, inferencejs@^1.0.14, lucide-react@^0.453.0, moment@^2.30.1, next@14.2.15, node-fetch@^3.3.2, nprogress@^0.2.0, openai@^4.68.1, postcss@^8, react@^18, react-dom@^18, react-force-graph-2d@^1.25.7, react-icons@^5.3.0, react-resizable-panels@^2.1.4, react-webcam@^7.2.0, recharts@^2.13.0, tailwind-merge@^2.5.4, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, uploadthing@^7.1.0, ws@^8.18.0 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - updated - first commit - first commit - Initial commit from Create Next App ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "watchdog", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@deepgram/sdk": "^3.8.1", "@radix-ui/react-dialog": "^1.1.2", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-slot": "^1.1.0", "@tensorflow-models/coco-ssd": "^2.2.3", "@tensorflow/tfjs": "^4.21.0", "@uploadthing/react": "^7.0.3", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "convex": "^1.16.6", "cross-fetch": "^4.0.0", "dotenv": "^16.4.5", "firebase": "^10.14.1", "firebase-admin": "^12.6.0", "framer-motion": "^11.11.9", "inferencejs": "^1.0.14", "lucide-react": "^0.453.0", "moment": "^2.30.1", "next": "14.2.15", "nprogress": "^0.2.0", "openai": "^4.68.1", "react": "^18", "react-dom": "^18", "react-force-graph-2d": "^1.25.7", "react-icons": "^5.3.0", "react-resizable-panels": "^2.1.4", "react-webcam": "^7.2.0", "recharts": "^2.13.0", "tailwind-merge": "^2.5.4", "tailwindcss-animate": "^1.0.7", "uploadthing": "^7.1.0", "ws": "^8.18.0" }, "devDependencies": { "eslint": "^8", "eslint-config-next": "14.2.15", "node-fetch": "^3.3.2", "postcss": "^8", "tailwindcss": "^3.4.1" } } ``` ### src/app/layout.js ```javascript import "./globals.css"; import {cn} from "@/lib/utils"; import Navbar from "@/components/layout/navbar"; import {poppins} from "@/assests/fonts"; import {ConvexClientProvider} from "@/providers/ConvexClientProvider"; import WeaponAlertDialog from "@/components/WeaponAlertDialog"; export const metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({children}) { return (
{children}
); } ``` ### convex/_generated/server.js ```javascript /* prettier-ignore-start */ /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define a Convex HTTP action. * * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object * as its second. * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`. */ export const httpAction = httpActionGeneric; /* prettier-ignore-end */ ``` ### src/app/cctv/page.jsx ```javascript import WeaponObjectDetectionVideo from "@/app/cctv/WeaponObjectDetectionVideo"; import Script from "next/script"; export default async function Page(){ return ( <>