Project Info
This project did not submit a demo video on Devpost.
Inspiration
I was always the designated tech support for my family. My mother in particular would always call me during the middle of the day asking me questions on how to navigate her computer, and I'd find myself FaceTiming her, drawing arrows and instructions on her screen with Facetime's screenshare annotation feature just to explain where to click. Inspired by my experience, we wanted to build an empathetic, patient, visual "Tech Guardian Angel." Our goal was to bridge the divide between the older generation and low-tech literacy users by implementing a tech support AI that uses real-time visual drawings and spoken guidance.
What it does
KindCursor is a cross-platform, real-time desktop companion summoned instantly via a hotkey (Option + Space). It captures the user's active application window, analyzes the interface, and guides them through any task. It talks to the user verbally while painting temporary, fading circles and annotations directly onto their desktop UI, showing them exactly where to click. In addition, there's an option at the users request to take over the mouse and complete the task. Optimized the search and recall system to first call a Claude API for the troubleshooting solutions then search the web only when the API finds no solution.
Challenges we ran into
API Network Latency: Sending raw, lossless PNG screenshots (nearly 1MB each) to Claude caused massive processing delays, ruining the "real-time" feel. We solved this by building an optimization pipeline to compress and downscale payloads to ~80KB JPEGs while mathematically scaling returned coordinates back up. Additionally, we originally built our coordinate-mapping engine on Gemini, but quickly realized its spatial bounding boxes weren't precise enough for safe, automated mouse takeovers. We had to pull off a high-wire mid-hackathon refactor, ripping out Gemini and migrating the core vision payload to Claude 3.5 Sonnet on the fly
Accomplishments we're proud of
We created a mouse tracking system that successfully controls the screen without removing control from the user. The separate mouse feature displays to the client how to resolve the issue for future learning.
What we learned
We learned that building user-facing AI tools requires a deep focus on latency optimization; even a brilliant model feels broken if the loop takes too long to respond. We also learned how to decouple core application logic from platform-specific APIs to ensure our software remains accessible to users regardless of their chosen operating system.
What's next
Persistent User Memory: Integrating a vector/graph database with RAG to build long-term memory profiles, allowing the AI to adapt to a user's specific learning pace and remember past technical roadblocks. Also, integrating Deepgram for ultra-low-latency speech-to-text audio capture, paired with Deepgram TTS tokens to give KindCursor a natural, warm, conversational voice.. Full Computer-Use Automation: Expanding the system into a proactive Computer-Use agent that can safely step in and complete tedious or deeply confusing system configurations
This project did not link a GitHub repository.
Analysis
No indexed repository for this project, so there are no commit stats to show.
Technology
- PythonUnchecked
No repository was indexed for this project, so these Devpost claims have not been checked against code.
AI coding agents
No repository was indexed, so agent usage could not be checked.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
No repository was indexed, so there is no codebase to measure.
This project did not link a GitHub repository, so there is nothing to diagram.
This project did not link a GitHub repository, so its feature claims have not been checked against code.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.