Project Info
Inspiration
A lot of people in our lives have visual impairment issues, our family members, friends, and some of our favorite teachers. We noticed that a lot of the time the accessibility tools that they use are time consuming and tedious to use and our plan was to leverage AI to fix these issues.
What it does
We created an AI Assistant that runs locally on a users machine and is able to control the operating system and different applications with voice commands. Our project also implements the functionality of other AI accessibility tools like OCR and TTS.
How we built it
We actually didn't vibe code it! Our app runs locally on the computer while also running a local web server with Flask so its a little too unique for tools like Vercel. We used Python as our main language, Flask for our UI web server, LMNT for TTS, Google's Imagen 4 for OCR, Claude Anthropic 4 for our main LLM, VAPI for AI phone calls, and then we gave the LLMS access to system control scripts and some pretty overlays.
Challenges we ran into
We spent a lot of time setting up the various AI tools and connecting them together took some debugging. We were also creating software that has an unusual tech stack so we didn't have a template to go off of. We also maxed out API credits a couple times, and at one point we sent our private api keys to a public Github repo and broke our whole app!
Accomplishments we're proud of
Honestly most of us are just proud we finished, when we initially came up with the idea it was like nothing we had ever built before and we weren't sure if we would be able to make a prototype in time. We are also proud of being about to create a unique AI tool that solves a problem that a lot of us have seen the people we love struggle with. I'm excited to share this project with my political science professor who is blind and also my other teammates' visually impaired uncle. We hope it can make their lives a little easier.
What we learned
We learned a lot about the new AI tools out there, we didn't know about groq and lmnts fast outputs. We learned about traditional AI models for edge detection for the restricted OCR. We learned about when to use general purpose AI vs when to use more specialized models. We met a lot of smart people and learned more about the industry. We learned how to create operating system control tools. Using selenium and other power control commands.
What's next
We want to give our tool more access to the computer. Right now we have a list of things we are able to control with the assistant but we want to enable them to control everything. An example would be using the OCR to find the exact pixel value of intractable things on the screen and be able to have better control of the system.
EyesOnAi
A voice controlled Agentic Accessibility tool that lets visually impaired users operate a laptop hands free. The backend runs on Python and Flask, Whisper handles speech to text, Claude as a ReAct Agent and LMNT reads responses back out loud.
Motivation
Many of us have loved ones, family members, friends, and even teachers who live with visual impairments. We noticed that while accessibility tools exist, they are often slow, unintuitive, or difficult to use. With the rapid advancements in AI, we saw an opportunity to reimagine accessibility that is faster, smarter, and more user centric.
How it works
Press the global hotkey to start talking. Speech is transcribed, then Claude reads the request and decides which tools to call to carry it out, using pyautogui for mouse and keyboard actions and subprocess calls to open and control applications. Tool results feed back into the conversation so Claude can chain multiple steps (for example "open TextEdit and write a grocery list" means opening the app, typing the text, and saving the file) before replying, and that reply is read back out loud.
A vision feature lets Gemini look at the screen (or a region the user drags out with an interactive tkinter overlay) and describe what it sees. A summarization step also condenses older parts of the conversation so long sessions do not run out of context.
Destructive actions (deleting files/folders, shutdown, restart) require an explicit spoken "yes confirm" before they run, and file operations are restricted to ~/Documents, ~/Desktop, ~/Downloads, ~/Pictures, ~/Music, and ~/Movies.
Architecture
| File | Role |
|---|---|
| app.py | Entry point. Runs the Flask server, a PyQt5 always-on-top overlay, and a global hotkey listener. Hosts AccessibilityChatbot, the ReAct loop that calls Claude with tool definitions and executes whatever it returns. |
| computer_commands.py | AccessibilityCommands - the actual tool implementations: window/app control, file CRUD, browser navigation, mouse/keyboard automation, screenshots, and vision screen description. |
| voice_commands.py | VoiceRecognition - records microphone audio and transcribes it via Groq's hosted Whisper. |
| lmnt_utils.py | Text-to-speech playback via LMNT. |
| region_selector.py | Standalone tkinter overlay for dragging out a screen region to describe with vision. |
| templates/index.html | Browser UI - chat/status view driven by a server-sent-events stream. |
Setup
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
Copy .env.example to .env and fill in your keys:
ANTHROPIC_API_KEY=
GROQ_API_KEY=
LMNT_API_KEY=
Run it:
python app.py
Analysis
View
Metric
- 5
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- FlaskIn code
- HTMLIn code
- PythonIn code
- CSSClaimed
- Google GeminiClaimed
3 of 5 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
100 KB
Source files
8
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
Balpreetkaur291/EyesOnAI
14 files · 1.5 MB · @ 4b6d144
Structure
Interface
1 file · 7%Screens, components and styles rendered to the user.
Application logic
6 files · 43%Domain rules, services and shared utilities.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- Python67%
- HTML31%
- Markdown3%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
requirements.txt
pypi · 14- flask
- google-generativeai
- keyboard
- lmnt
- psutil
- pyautogui
- pygame
- PyQt5
- replicate
- requests
- selenium
- sounddevice
- vapi_server_sdk
- wavio
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
This project’s features have not been analysed yet.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.