💭 Why we built this. / The problem at hand. Many home security camera devices are built solely off motion detection, resulting in repetitive spam alerts to owners’ devices when the slightest motion crosses their feed. If we aren’t quick to detect a potential harmful risk, then oftentimes it may be too late to take action. An example of these occurrences are home fires, if a homeowner can quickly detect the risk such as a spark, then they may be able to stop the risk before their home burns down or spreads into a bigger fire. Warmer states such as California are especially prone to this type of disaster. This functionality extends further than home systems, as government agencies such as the fire department could use surveillance to automatically detect remote fires while they are small. Traditional cameras do not offer ways to detect unique threats such as floods and machine malfunctions. With Lookout, we are introducing a new way for cameras to define their own detection features for analyzing live footage. With each notification tailored to your needs, we are reducing the amount of threat-less alerts. 💪

What it does

Integrate Fetch AI Agents for a prompt builder to allow users to automatically add features Scan live footage to instantly analyze potential threats using Gemini for video analysis Add / delete custom events to detect Output a live log of events Allow multiple devices and video feeds in real time 💡How does it work? We use React Webcam to take live video feeds and send it to our Flask backend. Users can input custom features on top of the three default ones (theft, weapons, violence) by leveraging a custom Fetch AI uAgents for prompt building. A second agent using the built prompt and Gemini AI analyzes the video within seconds and generates a report in JSON format to pass to MongoDB. The frontend fetches this log and displays alerts with matching timestamps to alert the user. Fetch AI and Gemini AI work in combination to quickly and seamlessly notify the user of any potential danger. 🧱

How we built it

Sponsors Used: Fetch.ai , Google Gemini Frontend: React, Vite, Javascript, TailwindCSS, React-Webcam Backend: Flask, Python, OpenCV, MongoDB, Fetch.ai UAgents, Gemini AI 😰 Challenges we faced This was our first time working with Fetch AI Agents, so we had to read pages of documents to understand what we needed to implement for them to work with our software. Fetching from Flask for the on_query decorator was especially tough, as we didn’t know immediately how the agents request data from non-agents, especially with other AI such as Gemini. We opted for using custom REST endpoints to leverage the new technology of agent communication while sticking to our old knowledge of REST. Team members had different OS, it was extremely difficult to have some of the APIs working on macOS, which consumed a notable amount of time to debug alongside internet issues. 🏆 Accomplishments Cal Hacks 11.0 was half of our team’s first hackathon, so being able to complete our project was a big accomplishment in itself. We are extremely proud of our group’s collaborative efforts in completing both the frontend and backend in less than 48 hours! On top of all that, Lookout’s threat-seeking functionality has the ability to actually be incredibly useful in security systems ranging from home protection to private security. We were also proud of being able to successfully implement the AI Agents to be able to identify new types of threats as they analyzed the video data. They were challenging to understand, but in the end we were able to bring the whole project together. 🧠

What we learned

Through our trials with the AI Agents, we learned that they offer a wide range of possibilities, as they are capable of performing tasks without human intervention. These agents can be designed to learn, adapt, and execute tasks based on predefined goals and environmental inputs. They also provide robust communication to pass data and have a chain of events. ⏩

What's next

We want to improve the API response time significantly and storage space significantly so that it can be implemented in security devices and home cameras around the US without trouble. We also want to improve the AI Agent’s ability to handle the video analysis given by Gemini API, so responses are even faster.