Project Info
Inspiration
Iris takes action for those who cannot. People with paralysis, as well as those in hands-free situations like surgeons, chefs, and mechanics, can all benefit from a browser that truly listens. Even in cases of severe paralysis, eye movement often remains one of the few ways people can still interact with the world. That simple truth inspired us to build Iris. One of our teammates has a family member who lost feeling in their hands. Watching them navigate a webpage using only their eyes reminded us that accessibility is not a feature, it is freedom. What It Does Hands-Free Control Using Eye Tracking, Voice, and AI Real-time gaze tracking highlights elements you look at Voice commands fill in text fields and control navigation AI agent predicts and executes your next action Works universally on any webpage without modifications Learns personal preferences over time (such as name, email, or address) Eye gestures enable clicks, tab switching, and scrolling Iris transforms the web into an adaptive, intelligent interface that moves at the speed of your gaze. How We Built It Real-Time Eye Tracking with Machine Learning Models We trained and deployed custom regression models (SVR, Ridge, Elastic Net, and a Tiny MLP) to predict gaze position from real-time webcam input. Combined with optimized OpenCV and MediaPipe pipelines, Iris achieves sub-100ms latency for seamless visual control. Advanced Kalman and KDE Filtering Pipeline Raw eye data is noisy. To stabilize motion, we implemented a Kalman filter for trajectory prediction and a Kernel Density Estimation (KDE) layer for adaptive smoothing. This multi-stage filter stack eliminates jitter and enables natural cursor flow. Multi-Modal Calibration System Calibration defines precision. Iris supports five- and nine-point calibration as well as continuous Lissajous-curve calibration, dynamically adjusting accuracy during runtime based on user behavior. WebSocket-Based Real-Time Gaze Streaming The Python backend streams gaze coordinates to the Chrome Extension through WebSockets with continuous synchronization and auto-recovery. Every blink, movement, and pause is captured and reflected instantly. Chrome Extension with DOM Element Snapping Iris detects DOM elements under gaze coordinates using a visual feedback system that highlights what you are looking at. Elements snap smoothly under the cursor through adaptive interpolation and dwell-time confirmation. Hybrid Speech Recognition System We combined Whisper.cpp (for local inference) with Vosk for fast, privacy-friendly transcription. The Electron-based speech service handles real-time dictation, enabling continuous speech-to-text input and command execution. Letta Agent Integration with Browser Actions Letta serves as Iris’s cognitive layer, a memory-driven agent that interprets voice commands and gaze context to perform browser automation such as form filling, navigation, and clicking. Virtual Keyboard with Gaze and Voice Input Users can type without hands using a multimodal keyboard that combines gaze focus detection (via Swift native helpers) and speech recognition for character input across applications. Swift Native Focus Watcher A macOS-native accessibility bridge detects active text fields and injects text directly into system-level inputs, allowing Iris to operate beyond browsers, across any desktop application. Custom Gaze Auto-Scroll System Looking near the edge of the screen triggers smooth auto-scrolling, controlled by gaze velocity and viewport position. It feels less like commanding a browser and more like moving through space. Element Highlighting with Adaptive Colors Every highlight adjusts dynamically for contrast and readability using CSS-injected adaptive color algorithms, ensuring accessibility in any theme or website layout. Challenges We Ran Into Our original plan was to use EEG signals for click gestures, but hardware delays forced a pivot toward eye and voice input. That shift was tough mid-hackathon, but it ultimately led to a more stable, scalable, and accessible foundation. Balancing real-time performance with accuracy was another major hurdle. Building a smooth filtering pipeline required countless iterations and fine-tuning. Accomplishments We’re Proud Of Achieved sub-100ms gaze latency for real-time interaction Built a universal browser integration that works without page modifications Implemented a memory-driven Letta agent for context-aware actions Designed a multi-stage filtering and calibration pipeline that rivals research-grade setups What We Learned Filtering is everything. Raw gaze data is unusable without strong temporal smoothing. Calibration defines trust. If the system drifts even slightly, users lose confidence. Accessibility inspires innovation. Designing for people with disabilities makes technology better for everyone. What’s Next for Iris Integrate EEG input for hybrid mind-eye control Expand Letta agent templates for scalability Extend native support beyond browsers to all operating systems Partner with accessibility foundations to deploy Iris at scale Empower millions of users with motor disabilities to regain independence
Iris Vision Module
Vision Module is a Python library that provides webcam-based eye tracking. Extract facial features, train a model and predict gaze with an easy‑to‑use interface.
Features
- Real‑time gaze estimation
- Multiple calibration workflows
- Optional filtering (Kalman / KDE)
- Model persistence – save / load a trained
GazeEstimator - Virtual-camera overlay that integrates with streaming software (e.g., OBS) via the bundled
eyetrax-virtualcamCLI
Installation
From source
git clone https://github.com/tgondil/iris && cd iris
# editable install — pick one
python -m pip install -e .
pip install uv && uv sync
Demo
The Vision Module package provides multiple command‑line entry points
| Command | Purpose |
|---|---|
eyetrax-demo | Run an on‑screen gaze overlay demo |
eyetrax-virtualcam | Stream the overlay to a virtual webcam |
eyetrax-stream | Stream gaze data via WebSocket (for Chrome extension) |
Options
| Flag | Values | Default | Description |
|---|---|---|---|
--filter | kalman, kde, none | none | Smoothing filter |
--camera | int | 0 | Physical webcam index |
--calibration | 9p, 5p, lissajous | 9p | Calibration routine |
--background (demo only) | path | — | Background image |
--confidence (KDE only) | 0–1 | 0.5 | Contour probability |
Quick Examples
eyetrax-demo --filter kalman
eyetrax-virtualcam --filter kde --calibration 5p
Virtual camera demo
https://github.com/user-attachments/assets/de4a0b63-8631-4c16-9901-9f83bc0bb766
🌐 Chrome Extension - Voice & Gaze Control
Control Chrome with your voice and eyes! The Iris Voice & Gaze extension enables:
- 🎤 Voice control - Speak into any text field on any webpage
- 👁️ Gaze tracking - Highlight webpage elements based on where you're looking
- 🤝 Combined workflow - Complete hands-free browsing experience
Quick Start - Voice Control
-
Load the Chrome extension:
- Open
chrome://extensions - Enable Developer mode
- Click "Load unpacked"
- Select the
chrome_gaze_latchfolder
- Open
-
Start using voice:
- Click any text field on any webpage
- Start speaking - your words appear automatically!
-
Keyboard shortcut:
Cmd+Shift+S(Mac) orCtrl+Shift+S(Windows/Linux) to toggle speech
Quick Start - Gaze Tracking
- Start the gaze server:
eyetrax-stream --filter kalman --calibration 9p
- Browse the web - Elements will glow cyan as you look at them!
Features
- 🎤 Real-time speech recognition - Uses Chrome's Web Speech API
- 🎯 Automatic activation - Voice starts when text fields are focused
- 👁️ Real-time gaze tracking - Element highlighting based on eye position
- ⏱️ Dwell-time filtering - Prevents accidental highlights (250ms)
- 🔄 Auto-reconnect - Seamless reconnection to gaze server
- ⌨️ Keyboard shortcuts - Quick toggle for speech recognition
- 🖱️ EEG support - Ready for brain-computer interface integration
Complete Hands-Free Workflow
- Look at a text field (gaze highlights it)
- Click (mouse, keyboard, or future EEG trigger)
- Speak your text (voice recognition fills it in)
- Look at submit button
- Confirm (click or EEG trigger)
Documentation
- Quick Start:
chrome_gaze_latch/QUICKSTART.md - Installation:
chrome_gaze_latch/INSTALL.md - Full Documentation:
chrome_gaze_latch/README.md - Test Page:
chrome_gaze_latch/test.html
See also: VOICE_CONTROL.md for all voice control options.
Library Usage
from vision_module import GazeEstimator, run_9_point_calibration
import cv2
# Create estimator and calibrate
estimator = GazeEstimator()
run_9_point_calibration(estimator)
# Save model
estimator.save_model("gaze_model.pkl")
# Load model
estimator = GazeEstimator()
estimator.load_model("gaze_model.pkl")
cap = cv2.VideoCapture(0)
while True:
# Extract features from frame
ret, frame = cap.read()
features, blink = estimator.extract_features(frame)
# Predict screen coordinates
if features is not None and not blink:
x, y = estimator.predict([features])[0]
print(f"Gaze: ({x:.0f}, {y:.0f})")
More
If you find this Vision Module useful, consider starring the repo or contributing. The project is available under the MIT license.
Based on EyeTrax by Chenkai Zhang.
Letta API (Personal Assistant)
-
Purpose Exposes a small FastAPI service to forward on-screen context to a Letta Cloud personal assistant agent and return a predicted next action.
-
Setup
- Set env vars before starting:
LETTA_PROJECTLETTA_TOKENLETTA_AGENT_ID
- Set env vars before starting:
-
Run
uvicorn letta.app:app --host 0.0.0.0 --port 8001 -
Health check
curl http://localhost:8001/health -
Predict endpoint
curl -X POST http://localhost:8001/letta/predict \ -H 'Content-Type: application/json' \ -d '{ "context_text": "User is in VS Code reviewing API docs; next they want to scaffold an endpoint.", "metadata": {"app": "vscode"} }'Returns JSON with
action,run_id, and rawmessagesfrom Letta.
Analysis
View
Metric
- 75
- 6
- 5
- 2
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- ExpressIn code
- FastAPIIn code
- FlaskIn code
- HTMLIn code
- JavaScriptIn code
- Next.jsIn code
- PythonIn code
- ReactIn code
- SwiftIn code
- Tailwind CSSIn code
- TypeScriptIn code
- Node.jsClaimed
12 of 13 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
651 KB
Source files
140
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
tgondil/iris
190 files · 69.3 MB · @ 80f86b7
Structure
Interface
23 files · 12%Screens, components and styles rendered to the user.
Application logic
89 files · 47%Domain rules, services and shared utilities.
+7 moreData & schema
12 files · 6%Schema definitions, migrations and data access.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- Markdown35%
- JavaScript23%
- Python23%
- TypeScript12%
- HTML2%
- Shell2%
- Other (2)1%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
pyproject.toml
pypi · 12- fastapi
- letta-client
- mediapipe
- numpy
- opencv-python
- python-dotenv
- pyvirtualcam
- scikit-learn
- scipy
- screeninfo
- uvicorn
- websockets
voice/package.json
npm · 12- lucide-react
- next
- react
- react-dom
- +8 more
eyetrax-fresh/pyproject.toml
pypi · 7- mediapipe
- numpy
- opencv-python
- pyvirtualcam
- scikit-learn
- scipy
- screeninfo
voice/electron/package.json
npm · 7- express
- mic
- node-record-lpcm16
- sox-audio
- vosk
- +2 more
fetch_agent/requirements.txt
pypi · 4- flask
- flask-cors
- requests
- uagents
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
Feature verification
Adaptive color element highlighting for contrast/readabilityVerified
Every highlight adjusts dynamically for contrast and readability using CSS-injected adaptive color algorithms
Claimed on Devpostmedium confidencechrome_gaze_latch/speech.js:122— getAdaptiveColor and getAdaptiveBackgroundColor methods used when styling highlight/UI elements
Chrome extension DOM element snapping with dwell-time confirmationVerified
Detects DOM elements under gaze coordinates, highlights them, and snaps under the cursor via adaptive interpolation and dwell-time confirmation
Claimed on Devposthigh confidencechrome_gaze_latch/content.js:19— SNAP_DISTANCE_THRESHOLD and DWELL_TIME constants with dwell tracking state (dwellStartTime, lastDwellElement)
Eye gestures enable clicks, tab switching, and scrollingVerified
Eye gestures enable clicks, tab switching, and scrolling
Claimed on Devposthigh confidencechrome_gaze_latch/content.js:34— detectLeftDoubleBlink/detectRightDoubleBlink/detectLeftTripleBlink functions implement blink-based gesture detectionchrome_gaze_latch/content.js:1892— Triple blink drives tab switching logic with debouncechrome_gaze_latch/bg.js:51— handleTabSwitch uses chrome.tabs API to switch active tab
Gaze-based auto-scroll near screen edgesVerified
Looking near the edge of the screen triggers smooth auto-scrolling controlled by gaze velocity and viewport position
Claimed on Devposthigh confidencechrome_gaze_latch/content.js:1759— handleAutoScroll(yViewport) function triggers window.scrollBy based on gaze viewport position
Hybrid speech recognition using Whisper.cpp and VoskVerified
Combined Whisper.cpp (local inference) with Vosk for fast, privacy-friendly transcription in an Electron speech service
Claimed on Devposthigh confidencevoice/electron/speech-service.js:6— Spawns whisper.cpp binary with model path for transcriptionvoice/electron/vosk-service.js— Separate Vosk-based service file present
Kalman and KDE filtering pipeline for gaze smoothingVerified
Implemented a Kalman filter for trajectory prediction and a KDE layer for adaptive smoothing to eliminate jitter
Claimed on Devposthigh confidencesrc/vision_module/filters/kalman.py:15— KalmanSmoother class with step() and tune() methodssrc/vision_module/filters/kde.py:14— KDESmoother class implementing step()
Letta agent integration for browser action prediction/automationVerified
Letta serves as Iris's cognitive layer, a memory-driven agent that interprets voice/gaze context to perform browser automation (form filling, navigation, clicking)
Claimed on Devposthigh confidenceletta/app.py:529— /letta/predict endpoint builds a prompt from gaze/page context, calls Letta client, parses returned actions, and dispatches them over WebSocket to the extensionchrome_gaze_latch/content.js:283— Extension calls http://localhost:8001/letta/predict, wiring gaze/page context to the Letta backend
Letta FastAPI health check and predict REST APIVerified
Exposes a FastAPI service with /health and /letta/predict endpoints to forward on-screen context to a Letta Cloud agent
Claimed on readmehigh confidenceletta/app.py:29— /health endpoint returns status okletta/app.py:529— /letta/predict endpoint implemented matching README's documented request/response shape
Model persistence (save/load trained GazeEstimator)Verified
Model persistence, save / load a trained GazeEstimator
Claimed on readmehigh confidencesrc/vision_module/models/base.py:44— BaseModel.save() and load() classmethods for persistence
Multi-modal calibration (5-point, 9-point, Lissajous, adaptive)Verified
Supports five- and nine-point calibration as well as continuous Lissajous-curve calibration, dynamically adjusting accuracy during runtime
Claimed on Devposthigh confidencesrc/vision_module/calibration/five_point.py:12— run_5_point_calibration functionsrc/vision_module/calibration/nine_point.py:12— run_9_point_calibration functionsrc/vision_module/calibration/lissajous.py:8— run_lissajous_calibration function with curve()src/vision_module/calibration/adaptive.py:88— run_adaptive_calibration with BlueNoiseSampler for runtime accuracy adjustment
Real-time gaze tracking with regression models (SVR, Ridge, Elastic Net, Tiny MLP)Verified
Trained and deployed custom regression models (SVR, Ridge, Elastic Net, Tiny MLP) to predict gaze position from webcam input
Claimed on Devposthigh confidencesrc/vision_module/models/svr.py:10— LinearSVRModel implements BaseModel with train/predictsrc/vision_module/models/ridge.py:9— RidgeModel implementationsrc/vision_module/models/elastic_net.py:9— ElasticNetModel implementationsrc/vision_module/models/tiny_mlp.py:9— TinyMLPModel implementation
Swift native focus watcher (macOS accessibility bridge)Verified
A macOS-native accessibility bridge detects active text fields and injects text into system-level inputs, allowing Iris to operate beyond browsers
Claimed on Devposthigh confidencevoice/swift-helper/FocusWatcher/main.swift:1— Uses ApplicationServices/AXUIElement APIs to detect focused UI element system-wide and communicates with Electron at 127.0.0.1:5000
Vision Module virtual-camera overlay (eyetrax-virtualcam) for streaming softwareVerified
Virtual-camera overlay that integrates with streaming software (e.g., OBS) via the bundled eyetrax-virtualcam CLI
Claimed on readmehigh confidencesrc/vision_module/app/virtualcam.py— Dedicated virtualcam.py module implementing the CLI entry point referenced in README
Voice commands fill text fields and control navigationVerified
Voice commands fill in text fields and control navigation
Claimed on Devposthigh confidencechrome_gaze_latch/speech.js— Speech recognition module drives text field filling per README Quick Start - Voice Control sectionchrome_gaze_latch/bg.js:15— Background script handles toggle-speech messaging tied to keyboard shortcut
WebSocket-based real-time gaze streaming to Chrome extensionVerified
Python backend streams gaze coordinates to the Chrome Extension through WebSockets with continuous synchronization and auto-recovery
Claimed on Devposthigh confidencesrc/vision_module/app/stream_gaze.py:23— Class streaming gaze coordinates via websockets.serve handlerchrome_gaze_latch/content.js— Extension consumes gaze stream and has auto-reconnect logic per README
Learns personal preferences over time (name, email, address)Code-supported
Iris learns personal preferences over time (such as name, email, or address) via Letta agent memory
Claimed on Devpostmedium confidenceletta/app.py:204— Prompt instructs the agent to 'use your agent memory to supply personal info (name, email, phone, address)', but actual learning/storage happens inside Letta Cloud's agent memory, external to this repo; no local code writes/updates persisted preferencesletta/schemas.py:54— user_profile field passed into the prompt for autofill, but this is caller-supplied context, not evidence of the agent learning over time
Virtual keyboard with gaze and voice inputCode-supported
Multimodal keyboard combining gaze focus detection (via Swift native helpers) and speech recognition for character input across applications
Claimed on Devpostmedium confidencevoice/swift-helper/FocusWatcher/main.swift:5— FocusWatcher polls system-wide focused UI element and communicates with Electron app, supporting text-field detection for keyboard show/hidevoice/src/components— Frontend directory likely contains keyboard UI, but exact gaze+voice combined keyboard component was not individually confirmed in this pass
Works universally on any webpage without modificationsCode-supported
Works universally on any webpage without modifications
Claimed on Devpostmedium confidencechrome_gaze_latch/manifest.json— Manifest V3 content script likely injected on <all_urls>, consistent with universal operation, but exhaustive cross-site compatibility cannot be confirmed by static inspection alone
EEG support ready for brain-computer interface integrationClaimed only
README lists EEG support: 'Ready for brain-computer interface integration'
Claimed on readmehigh confidence
An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.