# Project export: Lookout
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: Cal Hacks 11.0
- Tagline: Stay on the Lookout. Protect what matters most with our advanced AI-powered surveillance system. Detect dangers in real-time and stay one step ahead.
- Devpost: https://devpost.com/software/lookout-hiknf9
- GitHub: https://github.com/kevintsoii/Lookout
- Demo: https://lookout-xr8a.vercel.app/
- Video: https://www.youtube.com/embed/nvucs6HELoo?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 4 GitHub contributor(s) — kevintsoii (41 commits), xusamson8 (14 commits), huydduong05 (12 commits), Marvin (8 commits)
## Devpost submission (written by the team)
### Overview
💭 Why we built this. / The problem at hand. Many home security camera devices are built solely off motion detection, resulting in repetitive spam alerts to owners’ devices when the slightest motion crosses their feed. If we aren’t quick to detect a potential harmful risk, then oftentimes it may be too late to take action. An example of these occurrences are home fires, if a homeowner can quickly detect the risk such as a spark, then they may be able to stop the risk before their home burns down or spreads into a bigger fire. Warmer states such as California are especially prone to this type of disaster. This functionality extends further than home systems, as government agencies such as the fire department could use surveillance to automatically detect remote fires while they are small. Traditional cameras do not offer ways to detect unique threats such as floods and machine malfunctions. With Lookout, we are introducing a new way for cameras to define their own detection features for analyzing live footage. With each notification tailored to your needs, we are reducing the amount of threat-less alerts. 💪
### What it does
Integrate Fetch AI Agents for a prompt builder to allow users to automatically add features Scan live footage to instantly analyze potential threats using Gemini for video analysis Add / delete custom events to detect Output a live log of events Allow multiple devices and video feeds in real time 💡How does it work? We use React Webcam to take live video feeds and send it to our Flask backend. Users can input custom features on top of the three default ones (theft, weapons, violence) by leveraging a custom Fetch AI uAgents for prompt building. A second agent using the built prompt and Gemini AI analyzes the video within seconds and generates a report in JSON format to pass to MongoDB. The frontend fetches this log and displays alerts with matching timestamps to alert the user. Fetch AI and Gemini AI work in combination to quickly and seamlessly notify the user of any potential danger. 🧱
### How we built it
Sponsors Used: Fetch.ai , Google Gemini Frontend: React, Vite, Javascript, TailwindCSS, React-Webcam Backend: Flask, Python, OpenCV, MongoDB, Fetch.ai UAgents, Gemini AI 😰 Challenges we faced This was our first time working with Fetch AI Agents, so we had to read pages of documents to understand what we needed to implement for them to work with our software. Fetching from Flask for the on_query decorator was especially tough, as we didn’t know immediately how the agents request data from non-agents, especially with other AI such as Gemini. We opted for using custom REST endpoints to leverage the new technology of agent communication while sticking to our old knowledge of REST. Team members had different OS, it was extremely difficult to have some of the APIs working on macOS, which consumed a notable amount of time to debug alongside internet issues. 🏆 Accomplishments Cal Hacks 11.0 was half of our team’s first hackathon, so being able to complete our project was a big accomplishment in itself. We are extremely proud of our group’s collaborative efforts in completing both the frontend and backend in less than 48 hours! On top of all that, Lookout’s threat-seeking functionality has the ability to actually be incredibly useful in security systems ranging from home protection to private security. We were also proud of being able to successfully implement the AI Agents to be able to identify new types of threats as they analyzed the video data. They were challenging to understand, but in the end we were able to bring the whole project together. 🧠
### What we learned
Through our trials with the AI Agents, we learned that they offer a wide range of possibilities, as they are capable of performing tasks without human intervention. These agents can be designed to learn, adapt, and execute tasks based on predefined goals and environmental inputs. They also provide robust communication to pass data and have a chain of events. ⏩
### What's next
We want to improve the API response time significantly and storage space significantly so that it can be implemented in security devices and home cameras around the US without trouble. We also want to improve the AI Agent’s ability to handle the video analysis given by Gemini API, so responses are even faster.
## README (from the GitHub repository)
# Lookout
## **Project Overview**
**Lookout** is a AI video camera monitoring tool with real-time threat detection. It is designed to analyze live footage of both home and security cameras using Google Gemini and FetchAI uAgents. By allowing users to add custom detections for any type of event (fires, floods, package deliveries, robberies, etc.), Lookout has become the best way to enhance your existing security systems.
### **Features**
- **FetchAI** uAgents to create custom prompts based off of user-requested "features" (events to detect)
- **Google Gemini** for real-time analysis of video footage
- WebSockets for video data transfer
- Adding multiple camera sources
- **Live updates** with severity-based color-coded alerts
- Backend storage of previous logs.
---
## **Run the Project**
### **1. Prerequisites:**
- Node.js
- Python
- .env file
```
GEMINI_API_KEY=
MONGO_URI=
```
### **2. Install Dependencies:**
Navigate to the project directory and run:
```
npm install
pip install -r requirements.txt
```
### **3. Running the Program**
- One separate terminal is required for each
```
npm run dev
```
```
py backend/app.py
```
```
py backend/agents/agents.py
```
### **4. Visit the site!**
- http://localhost:8000/
---
## **Use Cases**
- Detect any sort of danger such as crime or robbery
- Detect house fires, or floods and other natural disasters for quick government response
- Detect non-threats such as package deliveries
## Detected evidence (automated analysis)
Indexed codebase: 23 recognized source files, 44 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code
- MongoDB (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (31 of 31)
```
.gitignore
.vercelignore
backend/agents/agents.py
backend/app.py
components.json
eslint.config.js
index.html
jsconfig.json
LICENSE
package.json
postcss.config.js
Procfile
README.md
requirements.txt
src/App.jsx
src/components/button.jsx
src/components/card.jsx
src/components/eyeball/Eye.css
src/components/eyeball/eyeball.jsx
src/components/FeatureList.jsx
src/components/input.jsx
src/components/LogsColumn.jsx
src/components/webcam/Webcam.jsx
src/components/webcam/WebcamSetup.jsx
src/index.css
src/lib/utils.js
src/main.jsx
src/pages/HomePage.jsx
src/pages/LookoutPage.jsx
tailwind.config.js
vite.config.js
```
### Dependencies
- package.json: @eslint/js@^9.11.1, @radix-ui/react-slot@^1.1.0, @shadcn/ui@^0.0.4, @types/react@^18.3.10, @types/react-dom@^18.3.0, @vitejs/plugin-react@^4.3.2, autoprefixer@^10.4.20, axios@^0.21.1, class-variance-authority@^0.7.0, clsx@^2.1.1, eslint@^9.11.1, eslint-plugin-react@^7.37.0, eslint-plugin-react-hooks@^5.1.0-rc.0, eslint-plugin-react-refresh@^0.4.12, framer-motion@^11.11.9, globals@^15.9.0, lucide-react@^0.453.0, postcss@^8.4.47, react@^18.3.1, react-dom@^18.3.1, react-router-dom@^6.27.0, react-webcam@^7.2.0, socket.io-client@^4.8.0, tailwind-merge@^2.5.4, tailwindcss@^3.4.14, tailwindcss-animate@^1.0.7, vite@^5.4.8
### Recent commits (newest first)
- home page changes
- ignore py
- ignore python in vercel
- fix bugs
- fater
- fix time
- Merge pull request #2 from kevintsoii/logs_finalfix
- logs ui fixes
- severity change
- prune
- fix logs
- final
- test123
- Merge branch 'database'
- Update README.md
- Update README.md
- connection w/ mongodb; basic CRUD ops
- readme
- readme
- Merge branch 'main' of https://github.com/kevintsoii/Lookout
## Key source files (fetched from GitHub, selected and truncated for size)
### package.json
```
{
"name": "lookout",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"lint": "eslint .",
"preview": "vite preview"
},
"dependencies": {
"@radix-ui/react-slot": "^1.1.0",
"@shadcn/ui": "^0.0.4",
"axios": "^0.21.1",
"class-variance-authority": "^0.7.0",
"clsx": "^2.1.1",
"framer-motion": "^11.11.9",
"lucide-react": "^0.453.0",
"react": "^18.3.1",
"react-dom": "^18.3.1",
"react-router-dom": "^6.27.0",
"react-webcam": "^7.2.0",
"socket.io-client": "^4.8.0",
"tailwind-merge": "^2.5.4",
"tailwindcss-animate": "^1.0.7"
},
"devDependencies": {
"@eslint/js": "^9.11.1",
"@types/react": "^18.3.10",
"@types/react-dom": "^18.3.0",
"@vitejs/plugin-react": "^4.3.2",
"autoprefixer": "^10.4.20",
"eslint": "^9.11.1",
"eslint-plugin-react": "^7.37.0",
"eslint-plugin-react-hooks": "^5.1.0-rc.0",
"eslint-plugin-react-refresh": "^0.4.12",
"globals": "^15.9.0",
"postcss": "^8.4.47",
"tailwindcss": "^3.4.14",
"vite": "^5.4.8"
}
}
```
### requirements.txt
```
��a i o h a p p y e y e b a l l s = = 2 . 4 . 3
a i o h t t p = = 3 . 1 0 . 1 0
a i o s i g n a l = = 1 . 3 . 1
a n n o t a t e d - t y p e s = = 0 . 7 . 0
a p i s p e c = = 6 . 6 . 1
a s g i r e f = = 3 . 8 . 1
a s y n c - t i m e o u t = = 4 . 0 . 3
a t t r s = = 2 4 . 2 . 0
b e c h 3 2 = = 1 . 2 . 0
b i d i c t = = 0 . 2 3 . 1
b l i n k e r = = 1 . 8 . 2
c a c h e t o o l s = = 5 . 5 . 0
c e r t i f i = = 2 0 2 4 . 8 . 3 0
c h a r s e t - n o r m a l i z e r = = 3 . 4 . 0
c l i c k = = 8 . 1 . 7
c o l o r a m a = = 0 . 4 . 6
c o s m p y = = 0 . 9 . 2
d n s p y t h o n = = 2 . 7 . 0
e c d s a = = 0 . 1 9 . 0
e v e n t l e t = = 0 . 3 7 . 0
F l a s k = = 3 . 0 . 3
F l a s k - C o r s = = 5 . 0 . 0
F l a s k - S o c k e t I O = = 5 . 4 . 1
f r o z e n l i s t = = 1 . 4 . 1
g o o g l e - a i - g e n e r a t i v e l a n g u a g e = = 0 . 6 . 1 0
g o o g l e - a p i - c o r e = = 2 . 2 1 . 0
g o o g l e - a p i - p y t h o n - c l i e n t = = 2 . 1 4 9 . 0
g o o g l e - a u t h = = 2 . 3 5 . 0
g o o g l e - a u t h - h t t p l i b 2 = = 0 . 2 . 0
g o o g l e - g e n e r a t i v e a i = = 0 . 8 . 3
g o o g l e a p i s - c o m m o n - p r o t o s = = 1 . 6 5 . 0
g r e e n l e t = = 3 . 1 . 1
g r p c i o = = 1 . 6 7 . 0
g r p c i o - s t a t u s = = 1 . 6 7 . 0
h 1 1 = = 0 . 1 4 . 0
h t t p l i b 2 = = 0 . 2 2 . 0
i d n a = = 3 . 1 0
i t s d a n g e r o u s = = 2 . 2 . 0
J i n j a 2 = = 3 . 1 . 4
j s o n s c h e m a = = 4 . 2 3 . 0
j s o n s c h e m a - s p e c i f i c a t i o n s = = 2 0 2 4 . 1 0 . 1
M a r k u p S a f e = = 3 . 0 . 2
m s g p a c k = = 1 . 1 . 0
m u l t i d i c t = = 6 . 1 . 0
n u m p y = = 2 . 1 . 2
o p e n c v - p y t h o n = = 4 . 1 0 . 0 . 8 4
p a c k a g i n g = = 2 4 . 1
p r o p c a c h e = = 0 . 2 . 0
p r o t o - p l u s = = 1 . 2 4 . 0
p r o t o b u f = = 5 . 2 8 . 2
p y a s n 1 = = 0 . 6 . 1
p y a s n 1 _ m o d u l e s = = 0 . 4 . 1
p y c r y p t o d o m e = = 3 . 2 1 . 0
p y d a n t i c = = 2 . 8 . 2
p y d a n t i c _ c o r e = = 2 . 2 0 . 1
p y p a r s i n g = = 3 . 2 . 0
p y t h o n - d a t e u t i l = = 2 . 9 . 0 . p o s t 0
p y t h o n - d o t e n v = = 1 . 0 . 1
p y t h o n - e n g i n e i o = = 4 . 1 0 . 1
p y t h o n - s o c k e t i o = = 5 . 1 1 . 4
r e f e r e n c i n g = = 0 . 3 5 . 1
r e q u e s t s = = 2 . 3 2 . 3
r p d s - p y = = 0 . 2 0 . 0
r s a = = 4 . 9
s i m p l e - w e b s o c k e t = = 1 . 1 . 0
s i x = = 1 . 1 6 . 0
t q d m = = 4 . 6 6 . 5
t y p i n g _ e x t e n s i o n s = = 4 . 1 2 . 2
u a g e n t s = = 0 . 1 6 . 2
u r i t e m p l a t e = = 4 . 1 . 1
u r l l i b 3 = = 2 . 2 . 3
u v i c o r n = = 0 . 3 0 . 6
w e b s o c k e t s = = 1 0 . 4
W e r k z e u g = = 3 . 0 . 4
w s p r o t o = = 1 . 2 . 0
y a r l = = 1 . 1 5 . 5
```
### src/main.jsx
```javascript
import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client'
import App from './App.jsx'
import './index.css'
createRoot(document.getElementById('root')).render(
,
)
```
### src/App.jsx
```javascript
import React from 'react';
import { BrowserRouter as Router, Routes, Route, Navigate, useLocation } from 'react-router-dom';
import HomePage from './pages/HomePage';
import LookoutPage from './pages/LookoutPage';
function App() {
const location = useLocation();
console.log("Current URL: ", location.pathname); // Log the current URL
return (
} />
} />
} />
);
}
export default function RootApp() {
return (
);
}
```
### backend/app.py
```python
import os, time, asyncio
import shutil
import threading
import numpy as np
import requests
import cv2
from collections import deque
from threading import Timer
import json
from flask import Flask, request, jsonify
from flask_cors import CORS
from flask_socketio import SocketIO
from uagents.query import query
from uagents import Model
from flask_cors import CORS
from pymongo import MongoClient
from bson.json_util import dumps
from dotenv import load_dotenv
load_dotenv()
VIDEO_ANALYZER_ADDRESS = "agent1qt8r5gxe6q4pyfyg0cf0lz83g7f5zw6787y3ts0ps87qcx9uq3fg78taj5q"
PROMPT_BUILDER_ADDRESS = "agent1qt2zjzgp62m86w65af2yw8rkr749ghu8p724yztg9wvqe3srlyteuhzgedu"
app = Flask(__name__)
CORS(app)
socketio = SocketIO(app, cors_allowed_origins="*", async_mode='eventlet', path='/socket')
UPLOAD_FOLDER = './uploads'
if os.path.exists(UPLOAD_FOLDER):
shutil.rmtree(UPLOAD_FOLDER)
os.makedirs(UPLOAD_FOLDER)
frame_buffer = {}
def gemini_call(file_path):
try:
requests.post(f'http://127.0.0.1:5001/analysis', json={"file_path": file_path})
print('passed', file_path)
except Exception as e:
print("ERROR", e)
def process_buffer(camera_id):
global frame_buffer
if camera_id in frame_buffer:
length = len(frame_buffer[camera_id]["frames"])
if length > 1:
output_video_path = f"./uploads/{camera_id}-{frame_buffer[camera_id]['updated']}.mp4"
print(length, output_video_path)
first_frame = frame_buffer[camera_id]["frames"][0]
height, width, _ = first_frame.shape
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
fps = 10
video_writer = cv2.VideoWriter(output_video_path, fourcc, fps, (width, height))
for _ in range(length):
frame = frame_buffer[camera_id]["frames"].popleft()
video_writer.write(frame)
video_writer.release()
threading.Thread(target=gemini_call, args=(output_video_path,)).start()
if camera_id in frame_buffer:
timer = Timer(5, process_buffer, [camera_id])
frame_buffer[camera_id]["timer"] = timer
frame_buffer[camera_id]["updated"] = time.time()
timer.start()
# cors
@app.after_request
def add_cors_headers(response):
response.headers['Access-Control-Allow-Origin'] = '*'
response.headers['Access-Control-Allow-Methods'] = 'GET,POST,PUT,DELETE,OPTIONS'
response.headers['Access-Control-Allow-Headers'] = 'Content-Type,Authorization'
return response
# sockets
@socketio.on('connect')
def handle_connect():
client_id = request.args.get('id', "Unknown")
print(f'{client_id}: Connected' )
@socketio.on('disconnect')
def handle_disconnect():
client_id = request.args.get('id', "Unknown")
print(f'{client_id}: Disonnected' )
global frame_buffer
if client_id in frame_buffer:
frame_buffer[client_id]["timer"].cancel()
del frame_buffer[client_id]
@socketio.on('end_video')
def handle_end_video(args):
client_id = args["id"]
print(f'{args["id"]}: Paused')
global frame_buffer
if client_id in frame_buffer:
frame_buffer[client_id]["timer"].cancel()
del frame_buffer[client_id]
@socketio.on('video_frame')
def handle_video_frame(args):
camera_id = args["id"]
frame = args["frame"]
#print(f'{camera_id} - {frame[-10:]}')
global frame_buffer
if camera_id not in frame_buffer:
frame_buffer[camera_id] = {"timer": None, "updated": None, "frames": deque()}
np_frame = np.frombuffer(frame, dtype=np.uint8)
img = cv2.imdecode(np_frame, cv2.IMREAD_COLOR)
frame_buffer[camera_id]["frames"].append(img)
if not frame_buffer[camera_id]["timer"]:
timer = Timer(5, process_buffer, [camera_id])
timer.start()
frame_buffer[camera_id]["timer"] = timer
frame_buffer[camera_id]["updated"] = time.time()
# Features
features = []
@app.route('/features', methods=['GET', 'POST'])
def manage_features():
global features
if request.method == 'GET':
return jsonify(features), 200
if request.method == 'POST':
data = request.json # Expecting an array of features
if not data or not isinstance(data, list):
return jsonify({"error": "Invalid data format. Must be an array."}), 400
features = data # Update features list
requests.post(f'http://127.0.0.1:5001/build', json={"prompt_items": features})
return jsonify({"message": "Features updated successfully!", "features": features}), 200
# MongoDB connection
mongo_uri = os.getenv("MONGO_URI") # add your own mongo connection in .env
client = MongoClient(mongo_uri)
db = client["lookout-database"]
logs_collection = db.logs # Access your collection
try:
client.server_info() # Check if connection is successful
print("Connected to MongoDB successfully!")
except Exception as e:
print(f"Error connecting to MongoDB: {e}")
# Get all logs from MongoDB
@app.route('/api/logs', methods=['GET'])
def get_log():
logs = logs_collection.find({})
return dumps(logs) # dumps() to convert BSON to JSON
@app.route('/api/logs', methods=['DELETE'])
def delete_logs():
result = logs_collection.delete_many({}) # Deletes all documents in the collection
return {'message': f'{result.deleted_count} logs deleted.'}, 200
# Add a new user to MongoDB
@app.route('/api/logs', methods=['POST'])
def add_log():
data = request.json # Get data from React frontend
ts = data["ts"]
for feature in data:
if feature != "ts":
new_log = {
"ts": ts,
"severity": data[feature]['severity'],
"description": data[feature]['description']
}
logs_collection.insert_one(new_log)
return jsonify({"message": "User added successfully"}), 201
# Sample logs to demonstrate. Replace this logic with dynamic threa
[truncated — 313 more characters]
```
### postcss.config.js
```javascript
export default {
plugins: {
tailwindcss: {},
autoprefixer: {},
},
}
```
### vite.config.js
```javascript
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";
import path from "path";
// https://vitejs.dev/config/
export default defineConfig({
resolve: {
alias: {
"@": path.resolve(__dirname, "./src"),
},
},
plugins: [react()],
server: {
port: 8000, // Specify your desired port here
},
});
```
### tailwind.config.js
```javascript
/** @type {import('tailwindcss').Config} */
export default {
darkMode: ["class"],
content: ["./index.html", "./src/**/*.{js,ts,jsx,tsx}"],
theme: {
extend: {
borderRadius: {
lg: 'var(--radius)',
md: 'calc(var(--radius) - 2px)',
sm: 'calc(var(--radius) - 4px)'
},
colors: {}
}
},
plugins: [require("tailwindcss-animate")],
};
```
### index.html
```html
Lookout.ai
```
### eslint.config.js
```javascript
import js from "@eslint/js";
import globals from "globals";
import react from "eslint-plugin-react";
import reactHooks from "eslint-plugin-react-hooks";
import reactRefresh from "eslint-plugin-react-refresh";
export default [
{ ignores: ["dist"] },
{
files: ["**/*.{js,jsx}"],
languageOptions: {
ecmaVersion: 2020,
globals: globals.browser,
parserOptions: {
ecmaVersion: "latest",
ecmaFeatures: { jsx: true },
sourceType: "module",
},
},
settings: { react: { version: "18.3" } },
plugins: {
react,
"react-hooks": reactHooks,
"react-refresh": reactRefresh,
},
rules: {
"react/prop-types": "off",
...js.configs.recommended.rules,
...react.configs.recommended.rules,
...react.configs["jsx-runtime"].rules,
...reactHooks.configs.recommended.rules,
"react/jsx-no-target-blank": "off",
"react-refresh/only-export-components": [
"warn",
{ allowConstantExport: true },
],
},
},
];
```
[14 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]