Project Info

Netra AI

Inspiration

Vibha (one of our teammates) was in India last summer visiting her grandparents. Someone stole something in front of her grandma's house, so she was asked through look through hours of footage to determine what exactly happened. While brainstorming, the team decided that AI could be used to address this issue and that's how Netra AI was born.

What it does

Netra AI has two main functionalities. The first one is that the user can query for events. For example, the user can ask if a person came to their house and if so, what time. The second functionality is that users can get alerts on events with detailed descriptions.

How we built it

For the frontend and backend, we used the Reflex framework. To analyze the video files we used Gemini. For the speech to text feature in the chatbot, we used DeepGram. To detect the motion and record the 5 second clip, we used OpenCV. We used ChromaDB to store the alert message produced from the processing.

Challenges we ran into

One challenge that we had was learning the Reflex framework. We wanted to display the alert messages on the website, but we were not able to do so. We also had issues with the motion detection being too sensitive, but we were able to fix it by changing the sensitivity.

Accomplishments we're proud of

One thing we are proud of is being able to successfully implement the speech to text feature. We felt that this feature was important because it would be useful for people on the go to easily communicate with the system. Additionally, we wanted to make our system more accessible. Vibha's grandma struggles to type on the keyboard, so in this case a speech feature would be useful for people like her.

What we learned

Through this experience, we learned many technologies from the sponsors like Gemini, ChromaDB, Reflex, and DeepGram which we used in our project. We also learned how to work better as a team. We were able to make progress by delegating tasks. When we felt stuck, we swapped computers with each other to get a fresh set of eyes. This proved to be very valuable as we were able to resolve many issues this way.

What's next

1) Real Time Text Alerts with Detailed Descriptions of Important Events Text user that, for example “Your daughter has arrived home” or “There is a fire outside your home.” 2) More Training of the AI Model Train the AI model on video data from different countries and people. 3) Improving Security and Privacy Implement Multi-Factor Authentication and Encryption

Analysis

Compare with all teams

View

Metric

Devang Sharma
1

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

OpenAIIn code
PythonIn code
CSSClaimed
Google GeminiClaimed
HTMLClaimed

2 of 5 appear in the indexed code. 3 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

41 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

1 Verified4 Code-supported6 Claimed only1 Blocked

Reflex frontend and backendVerified
For the frontend and backend, we used the Reflex framework.
Claimed on Devposthigh confidence
- frontend/frontend.py:1 — The app is built with `import reflex as rx`, defines rx.State classes, components, and `app = rx.App()` with `app.add_page(index, ...)`.
- frontend/state.py:86 — State and SettingsState classes subclass rx.State, the standard Reflex state pattern.
Query video events via chatbot (Gemini video Q&A)Code-supported
Users can query for events, e.g. ask if a person came to their house and if so, what time.
Claimed on Devpostmedium confidence
- frontend/state.py:50 — Gemini model is instantiated and a chat session is started with an uploaded video file, then user questions are sent via chat_session.send_message(prompt) and displayed in chat history.
- frontend/state.py:56 — The video file is hardcoded to './assets/calhacks.mp4' at module load time rather than a dynamically recorded/uploaded clip, so it works only as a fixed demo, not a general live-monitoring query system.
Speech-to-text chatbot input via DeepGramCode-supported
For the speech to text feature in the chatbot, we used DeepGram.
Claimed on Devpostmedium confidence
- frontend/speechreflex.py:8 — Deepgram API URL and key are defined and used: audio captured via AudioRecorderPolyfill is POSTed to the Deepgram /v1/listen endpoint and the transcript is parsed from the response.
- frontend/speechreflex.py:84 — Transcript result is set into the chat input field (rx.set_value('input1', transcript)), wiring speech-to-text into the chat UI.
- frontend/frontend.py:10 — frontend.py imports 'audio' from speechreflex but the audio() component is never called/rendered in index(); action_bar() in chat.py duplicates a similar Audio state/capture object separately, so it's unclear the feature is fully wired end-to-end in the running app.
Video analysis using GeminiCode-supported
To analyze the video files we used Gemini.
Claimed on Devpostmedium confidence
- frontend/state.py:21 — upload_to_gemini() uploads a video file to the Gemini API and a GenerativeModel chat session is created against it, but only a single hardcoded demo video is analyzed, not a general video pipeline.
Webcam capture/recording componentCode-supported
Implied by video monitoring app functionality (webcam-based recording infrastructure).
Claimed on readmelow confidence
- frontend/webcam.py:189 — A Webcam React component wrapper with start_recording/stop_recording JS helpers is implemented, but the bulk of the file (screenshot/recording state and UI) is commented out and not imported/used by frontend.py.
ChromaDB storage of alert messagesClaimed only
We used ChromaDB to store the alert message produced from the processing.
Claimed on Devposthigh confidence
Future: broader AI model training on diverse video dataClaimed only
More training of the AI model on video data from different countries and people.
Claimed on Devposthigh confidence
Future: Multi-Factor Authentication and EncryptionClaimed only
Improving security and privacy by implementing Multi-Factor Authentication and Encryption.
Claimed on Devposthigh confidence
Future: Real-time text alerts with detailed descriptionsClaimed only
What's next: Real Time Text Alerts with Detailed Descriptions of Important Events (e.g. text user 'Your daughter has arrived home').
Claimed on Devposthigh confidence
Motion detection and 5-second clip recording via OpenCVClaimed only
To detect the motion and record the 5 second clip, we used OpenCV.
Claimed on Devposthigh confidence
Real-time alerts on events with detailed descriptionsClaimed only
Users get alerts on events with detailed descriptions.
Claimed on Devposthigh confidence
README describes a Reflex x LlamaIndex / Llama Deploy full-stack UI tutorial projectBlocked
README title 'Reflex x LLamaIndex' and 'UI For Llama Deploy', pointing to the llama_deploy python_fullstack example tutorial.
Claimed on readmelow confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.