# Project export: Iris This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Changing lives, one blink at a time :) - Devpost: https://devpost.com/software/iris-ojrmnv - GitHub: https://github.com/tgondil/iris - Video: https://www.youtube.com/embed/T3Psh8Hm7so?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Crater: Play-Do Prize) - Team: 4 GitHub contributor(s) — tgondil (75 commits), skylarsoon (6 commits), Jason (5 commits), Eugene Lacatis (2 commits) ## Devpost submission (written by the team) ### Inspiration Iris takes action for those who cannot. People with paralysis, as well as those in hands-free situations like surgeons, chefs, and mechanics, can all benefit from a browser that truly listens. Even in cases of severe paralysis, eye movement often remains one of the few ways people can still interact with the world. That simple truth inspired us to build Iris. One of our teammates has a family member who lost feeling in their hands. Watching them navigate a webpage using only their eyes reminded us that accessibility is not a feature, it is freedom. What It Does Hands-Free Control Using Eye Tracking, Voice, and AI Real-time gaze tracking highlights elements you look at Voice commands fill in text fields and control navigation AI agent predicts and executes your next action Works universally on any webpage without modifications Learns personal preferences over time (such as name, email, or address) Eye gestures enable clicks, tab switching, and scrolling Iris transforms the web into an adaptive, intelligent interface that moves at the speed of your gaze. How We Built It Real-Time Eye Tracking with Machine Learning Models We trained and deployed custom regression models (SVR, Ridge, Elastic Net, and a Tiny MLP) to predict gaze position from real-time webcam input. Combined with optimized OpenCV and MediaPipe pipelines, Iris achieves sub-100ms latency for seamless visual control. Advanced Kalman and KDE Filtering Pipeline Raw eye data is noisy. To stabilize motion, we implemented a Kalman filter for trajectory prediction and a Kernel Density Estimation (KDE) layer for adaptive smoothing. This multi-stage filter stack eliminates jitter and enables natural cursor flow. Multi-Modal Calibration System Calibration defines precision. Iris supports five- and nine-point calibration as well as continuous Lissajous-curve calibration, dynamically adjusting accuracy during runtime based on user behavior. WebSocket-Based Real-Time Gaze Streaming The Python backend streams gaze coordinates to the Chrome Extension through WebSockets with continuous synchronization and auto-recovery. Every blink, movement, and pause is captured and reflected instantly. Chrome Extension with DOM Element Snapping Iris detects DOM elements under gaze coordinates using a visual feedback system that highlights what you are looking at. Elements snap smoothly under the cursor through adaptive interpolation and dwell-time confirmation. Hybrid Speech Recognition System We combined Whisper.cpp (for local inference) with Vosk for fast, privacy-friendly transcription. The Electron-based speech service handles real-time dictation, enabling continuous speech-to-text input and command execution. Letta Agent Integration with Browser Actions Letta serves as Iris’s cognitive layer, a memory-driven agent that interprets voice commands and gaze context to perform browser automation such as form filling, navigation, and clicking. Virtual Keyboard with Gaze and Voice Input Users can type without hands using a multimodal keyboard that combines gaze focus detection (via Swift native helpers) and speech recognition for character input across applications. Swift Native Focus Watcher A macOS-native accessibility bridge detects active text fields and injects text directly into system-level inputs, allowing Iris to operate beyond browsers, across any desktop application. Custom Gaze Auto-Scroll System Looking near the edge of the screen triggers smooth auto-scrolling, controlled by gaze velocity and viewport position. It feels less like commanding a browser and more like moving through space. Element Highlighting with Adaptive Colors Every highlight adjusts dynamically for contrast and readability using CSS-injected adaptive color algorithms, ensuring accessibility in any theme or website layout. Challenges We Ran Into Our original plan was to use EEG signals for click gestures, but hardware delays forced a pivot toward eye and voice input. That shift was tough mid-hackathon, but it ultimately led to a more stable, scalable, and accessible foundation. Balancing real-time performance with accuracy was another major hurdle. Building a smooth filtering pipeline required countless iterations and fine-tuning. Accomplishments We’re Proud Of Achieved sub-100ms gaze latency for real-time interaction Built a universal browser integration that works without page modifications Implemented a memory-driven Letta agent for context-aware actions Designed a multi-stage filtering and calibration pipeline that rivals research-grade setups What We Learned Filtering is everything. Raw gaze data is unusable without strong temporal smoothing. Calibration defines trust. If the system drifts even slightly, users lose confidence. Accessibility inspires innovation. Designing for people with disabilities makes technology better for everyone. What’s Next for Iris Integrate EEG input for hybrid mind-eye control Expand Letta agent templates for scalability Extend native support beyond browsers to all operating systems Partner with accessibility foundations to deploy Iris at scale Empower millions of users with motor disabilities to regain independence ## README (from the GitHub repository) # Iris Vision Module ![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg) ![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg) ![Demo](https://github.com/user-attachments/assets/1b953a10-442f-4c4a-95e0-52a68f1488bc) Vision Module is a Python library that provides **webcam-based eye tracking**. Extract facial features, train a model and predict gaze with an easy‑to‑use interface. ## Features - Real‑time gaze estimation - Multiple calibration workflows - Optional filtering (Kalman / KDE) - Model persistence – save / load a trained `GazeEstimator` - Virtual-camera overlay that integrates with streaming software (e.g., OBS) via the bundled **`eyetrax-virtualcam`** CLI ## Installation ### From source ```bash git clone https://github.com/tgondil/iris && cd iris # editable install — pick one python -m pip install -e . pip install uv && uv sync ``` ## Demo The **Vision Module** package provides multiple command‑line entry points | Command | Purpose | |---------|---------| | `eyetrax-demo` | Run an on‑screen gaze overlay demo | | `eyetrax-virtualcam` | Stream the overlay to a virtual webcam | | `eyetrax-stream` | Stream gaze data via WebSocket (for Chrome extension) | Options | Flag | Values | Default | Description | |------|--------|---------|-------------| | `--filter` | `kalman`, `kde`, `none` | `none` | Smoothing filter | | `--camera` | *int* | `0` | Physical webcam index | | `--calibration` | `9p`, `5p`, `lissajous` | `9p` | Calibration routine | | `--background` *(demo only)* | *path* | — | Background image | | `--confidence` *(KDE only)* | *0–1* | `0.5` | Contour probability | ## Quick Examples ```bash eyetrax-demo --filter kalman ``` ```bash eyetrax-virtualcam --filter kde --calibration 5p ``` ### Virtual camera demo https://github.com/user-attachments/assets/de4a0b63-8631-4c16-9901-9f83bc0bb766 ## 🌐 Chrome Extension - Voice & Gaze Control Control Chrome with your **voice** and **eyes**! The **Iris Voice & Gaze** extension enables: - 🎤 **Voice control** - Speak into any text field on any webpage - 👁️ **Gaze tracking** - Highlight webpage elements based on where you're looking - 🤝 **Combined workflow** - Complete hands-free browsing experience ### Quick Start - Voice Control 1. **Load the Chrome extension:** - Open `chrome://extensions` - Enable Developer mode - Click "Load unpacked" - Select the `chrome_gaze_latch` folder 2. **Start using voice:** - Click any text field on any webpage - Start speaking - your words appear automatically! 3. **Keyboard shortcut:** - `Cmd+Shift+S` (Mac) or `Ctrl+Shift+S` (Windows/Linux) to toggle speech ### Quick Start - Gaze Tracking 1. **Start the gaze server:** ```bash eyetrax-stream --filter kalman --calibration 9p ``` 2. **Browse the web** - Elements will glow cyan as you look at them! ### Features - 🎤 **Real-time speech recognition** - Uses Chrome's Web Speech API - 🎯 **Automatic activation** - Voice starts when text fields are focused - 👁️ **Real-time gaze tracking** - Element highlighting based on eye position - ⏱️ **Dwell-time filtering** - Prevents accidental highlights (250ms) - 🔄 **Auto-reconnect** - Seamless reconnection to gaze server - ⌨️ **Keyboard shortcuts** - Quick toggle for speech recognition - 🖱️ **EEG support** - Ready for brain-computer interface integration ### Complete Hands-Free Workflow 1. **Look** at a text field (gaze highlights it) 2. **Click** (mouse, keyboard, or future EEG trigger) 3. **Speak** your text (voice recognition fills it in) 4. **Look** at submit button 5. **Confirm** (click or EEG trigger) ### Documentation - **Quick Start:** `chrome_gaze_latch/QUICKSTART.md` - **Installation:** `chrome_gaze_latch/INSTALL.md` - **Full Documentation:** `chrome_gaze_latch/README.md` - **Test Page:** `chrome_gaze_latch/test.html` See also: `VOICE_CONTROL.md` for all voice control options. ## Library Usage ```python from vision_module import GazeEstimator, run_9_point_calibration import cv2 # Create estimator and calibrate estimator = GazeEstimator() run_9_point_calibration(estimator) # Save model estimator.save_model("gaze_model.pkl") # Load model estimator = GazeEstimator() estimator.load_model("gaze_model.pkl") cap = cv2.VideoCapture(0) while True: # Extract features from frame ret, frame = cap.read() features, blink = estimator.extract_features(frame) # Predict screen coordinates if features is not None and not blink: x, y = estimator.predict([features])[0] print(f"Gaze: ({x:.0f}, {y:.0f})") ``` ## More If you find this Vision Module useful, consider starring the repo or contributing. The project is available under the MIT license. Based on EyeTrax by Chenkai Zhang. ## Letta API (Personal Assistant) - **Purpose** Exposes a small FastAPI service to forward on-screen context to a Letta Cloud personal assistant agent and return a predicted next action. - **Setup** - Set env vars before starting: - `LETTA_PROJECT` - `LETTA_TOKEN` - `LETTA_AGENT_ID` - **Run** ```bash uvicorn letta.app:app --host 0.0.0.0 --port 8001 ``` - **Health check** ```bash curl http://localhost:8001/health ``` - **Predict endpoint** ```bash curl -X POST http://localhost:8001/letta/predict \ -H 'Content-Type: application/json' \ -d '{ "context_text": "User is in VS Code reviewing API docs; next they want to scaffold an endpoint.", "metadata": {"app": "vscode"} }' ``` Returns JSON with `action`, `run_id`, and raw `messages` from Letta. ## Detected evidence (automated analysis) Indexed codebase: 140 recognized source files, 651 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - FastAPI (technology) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 179) ``` .env .gitignore .python-version chrome_gaze_latch/ADAPTIVE_COLORS.md chrome_gaze_latch/api_test.html chrome_gaze_latch/AUTO_SCROLL.md chrome_gaze_latch/bg.js chrome_gaze_latch/CHANGES.md chrome_gaze_latch/CLICK_CONFIRMATION.md chrome_gaze_latch/CONSOLE_ERRORS.md chrome_gaze_latch/content.js chrome_gaze_latch/ELEMENT_SNAPPING_CHANGES.md chrome_gaze_latch/ELEMENT_SNAPPING.md chrome_gaze_latch/EYETRAX_FRESH.md chrome_gaze_latch/FINAL_TEXT_MODE.md chrome_gaze_latch/GAZE_CALIBRATION.md chrome_gaze_latch/GOOGLE_PAGES.md chrome_gaze_latch/INSTALL.md chrome_gaze_latch/manifest.json chrome_gaze_latch/MIGRATION.md chrome_gaze_latch/NEW_UI_SUMMARY.md chrome_gaze_latch/QUICKSTART.md chrome_gaze_latch/README.md chrome_gaze_latch/REALTIME_MODE.md chrome_gaze_latch/SMOOTH_CURSOR.md chrome_gaze_latch/speech.js chrome_gaze_latch/test.html chrome_gaze_latch/TROUBLESHOOTING.md chrome_gaze_latch/UI_DESIGN.md chrome_letta_actions/background.js chrome_letta_actions/content.js chrome_letta_actions/manifest.json chrome_letta_actions/options.html chrome_letta_actions/options.js chrome_letta_actions/sample_contact.html chrome_letta_actions/sample_login.html chrome_letta_actions/test_ws_server.py CITATION.cff eyetrax-fresh/.gitignore eyetrax-fresh/.python-version eyetrax-fresh/CHROME_SETUP.md eyetrax-fresh/CITATION.cff eyetrax-fresh/LICENSE eyetrax-fresh/pyproject.toml eyetrax-fresh/README.md eyetrax-fresh/src/eyetrax/__init__.py eyetrax-fresh/src/eyetrax/_version.py eyetrax-fresh/src/eyetrax/app/__init__.py eyetrax-fresh/src/eyetrax/app/build_model.py eyetrax-fresh/src/eyetrax/app/demo.py eyetrax-fresh/src/eyetrax/app/virtualcam.py eyetrax-fresh/src/eyetrax/calibration/__init__.py eyetrax-fresh/src/eyetrax/calibration/adaptive.py eyetrax-fresh/src/eyetrax/calibration/common.py eyetrax-fresh/src/eyetrax/calibration/five_point.py eyetrax-fresh/src/eyetrax/calibration/lissajous.py eyetrax-fresh/src/eyetrax/calibration/nine_point.py eyetrax-fresh/src/eyetrax/cli.py eyetrax-fresh/src/eyetrax/constants.py eyetrax-fresh/src/eyetrax/filters/__init__.py eyetrax-fresh/src/eyetrax/filters/base.py eyetrax-fresh/src/eyetrax/filters/kalman.py eyetrax-fresh/src/eyetrax/filters/kde.py eyetrax-fresh/src/eyetrax/filters/noop.py eyetrax-fresh/src/eyetrax/gaze.py eyetrax-fresh/src/eyetrax/models/__init__.py eyetrax-fresh/src/eyetrax/models/base.py eyetrax-fresh/src/eyetrax/models/elastic_net.py eyetrax-fresh/src/eyetrax/models/ridge.py eyetrax-fresh/src/eyetrax/models/svr.py eyetrax-fresh/src/eyetrax/models/tiny_mlp.py eyetrax-fresh/src/eyetrax/utils/draw.py eyetrax-fresh/src/eyetrax/utils/screen.py eyetrax-fresh/src/eyetrax/utils/video.py eyetrax-fresh/start_gaze.sh eyetrax-fresh/stream_gaze_chrome.py eyetrax-fresh/uv.lock fetch_agent/agent.py fetch_agent/README.md fetch_agent/requirements.txt fetch_agent/test_agent.sh FINAL_SETUP_GUIDE.md letta/__init__.py letta/app.py letta/config.py letta/README_INTEGRATION.md letta/schemas.py LICENSE pyproject.toml QUICK_SETUP_GUIDE.md README.md setup_iris.sh src/vision_module/__init__.py src/vision_module/_version.py src/vision_module/app/__init__.py src/vision_module/app/build_model.py src/vision_module/app/demo.py src/vision_module/app/stream_gaze.py src/vision_module/app/virtualcam.py src/vision_module/calibration/__init__.py src/vision_module/calibration/adaptive.py src/vision_module/calibration/common.py src/vision_module/calibration/five_point.py src/vision_module/calibration/lissajous.py src/vision_module/calibration/nine_point.py src/vision_module/cli.py src/vision_module/constants.py src/vision_module/filters/__init__.py src/vision_module/filters/base.py src/vision_module/filters/kalman.py src/vision_module/filters/kde.py src/vision_module/filters/noop.py src/vision_module/gaze.py src/vision_module/models/__init__.py src/vision_module/models/base.py src/vision_module/models/elastic_net.py src/vision_module/models/ridge.py src/vision_module/models/svr.py src/vision_module/models/tiny_mlp.py src/vision_module/utils/draw.py [59 more files omitted for size] ``` ### Dependencies - eyetrax-fresh/pyproject.toml: mediapipe@>=0.10, numpy@>=1.22, opencv-python@>=4.5, pyvirtualcam@>=0.10, scikit-learn@>=1.3, scipy@>=1.10, screeninfo@>=0.8 - fetch_agent/requirements.txt: flask@==3.1.2, flask-cors@==6.0.1, requests@==2.32.5, uagents@==0.22.10 - pyproject.toml: fastapi@>=0.111, letta-client@>=0.1.0, mediapipe@>=0.10, numpy@>=1.22, opencv-python@>=4.5, python-dotenv@>=1.0, pyvirtualcam@>=0.10, scikit-learn@>=1.3, scipy@>=1.10, screeninfo@>=0.8, uvicorn@>=0.23, websockets@>=12.0 - voice/electron/package.json: electron@^28.0.0, electron-builder@^24.9.1, express@^4.18.2, mic@^2.1.2, node-record-lpcm16@^1.0.1, sox-audio@^0.3.0, vosk@^0.3.39 - voice/package.json: @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@16.0.0, lucide-react@^0.548.0, next@16.0.0, react@19.2.0, react-dom@19.2.0, tailwindcss@^4, typescript@^5 ### Recent commits (newest first) - fix: Prevent duplicate focusTextField calls causing abort errors - fix: Manually dispatch focusin event to trigger speech recognition - debug: Add logging to diagnose focusin event issue + re-enable auto-start - fix: Disable all auto-start/restart to eliminate abort errors - fix: Switch to non-continuous mode to fix speech recognition - fix: Disable auto-restart completely to prevent abort errors - fix: Prevent multiple focusin events from causing speech abort - fix: Prevent speech recognition 'aborted' error with better state management - feat: Make tab switching MUCH harder - requires triple blink - feat: Change tab switching to single blink (left/right eye) - feat: Increase tab switching sensitivity for easier double blink detection - refine: Align Letta UI with chrome gaze style + remove text field blur - refine: Apply Notion × Apple glassmorphic style to all Letta UI - fix: Detect gaze on Letta suggestion overlays including child elements - fix: Correct indentation in letta/app.py (for real this time) - fix: Enable gaze interaction with Letta suggestion overlays - feat: Make confirmation overlay much stickier - fix: Reduce button stickiness during confirmation - fix: Reduce button stickiness during confirmation - feat: Add loading overlay and dual-panel Letta suggestions (left/right) ## Key source files (fetched from GitHub, selected and truncated for size) ### QUICK_SETUP_GUIDE.md ```markdown # 🚀 Quick Setup Guide - Iris Voice & Gaze Control Complete setup guide for hands-free browsing with voice recognition and gaze tracking. ## ✅ What You'll Get - **🎤 Voice Recognition**: Speak into any text field on any website - **👁️ Gaze Tracking**: Look at elements to highlight them with your eyes - **🔄 Combined Control**: Use both voice and gaze together for complete hands-free browsing --- ## 🛠️ Setup Steps ### Step 1: Python Environment (Already Done!) The Python environment is already set up with all dependencies installed. **To activate it:** ```bash cd /Users/jaysuun/Desktop/CodingProj/Hackathons/Calhack2025/iris source .venv/bin/activate ``` ### Step 2: Install Chrome Extension 1. **Open Chrome** and go to: `chrome://extensions` 2. **Enable Developer Mode** - Toggle the "Developer mode" switch in the top-right corner 3. **Load the Extension** - Click "Load unpacked" - Navigate to: `/Users/jaysuun/Desktop/CodingProj/Hackathons/Calhack2025/iris/chrome_gaze_latch` - Click "Select" 4. **Grant Permissions** - The extension will ask for microphone permission - Click "Allow" when prompted ### Step 3: Test Voice Recognition 1. **Open any website** (e.g., google.com) 2. **Click on a text field** (like the search box) 3. **Look for the indicator** in the bottom-right corner (🎤 "Listening...") 4. **Start speaking** - your words should appear in the field! **Keyboard Shortcuts:** - **Mac**: `Cmd + Shift + S` to toggle speech on/off - **Windows/Linux**: `Ctrl + Shift + S` to toggle speech on/off ### Step 4: Test Gaze Tracking (Optional) 1. **Start the gaze server:** ```bash cd /Users/jaysuun/Desktop/CodingProj/Hackathons/Calhack2025/iris/eyetrax-fresh source .venv/bin/activate python stream_gaze_chrome.py --filter kalman --calibration 9p ``` 2. **Complete calibration:** - Follow the moving dots with your eyes (keep head still) - The system will calibrate to your eye movements 3. **Test on any webpage:** - Look around the page with your eyes - Elements should highlight with a cyan glow after ~250ms - A purple cursor shows your gaze position **Gaze Shortcuts:** - `Ctrl + Shift + Up/Down` to adjust gaze cursor offset - `Escape` to close fullscreen gaze visualization --- ## 🎯 Usage Examples ### Voice-Only Workflow 1. Click on any text field 2. Start speaking 3. Your words appear automatically 4. Submit normally ### Gaze-Only Workflow 1. Start gaze server 2. Look at elements to highlight them 3. Click with mouse/keyboard when ready ### Combined Workflow (Recommended!) 1. Look at a search box (gaze highlights it) 2. Click it 3. Speak your search query (voice fills it in) 4. Look at the search button (gaze highlights it) 5. Click to search --- ## 🔧 Troubleshooting ### Voice Recognition Not Working - **Check microphone permission**: Click the lock icon in Chrome's address bar - **Try the keyboard shortcut**: `Cmd+Shift+S` (Mac) or `Ctrl+Shift+S` (Windows) - **Check console**: [truncated — 2257 more characters] ``` ### VOICE_CONTROL.md ```markdown # Voice Control for Iris Iris now supports voice control directly in your browser! Speak into any text field on any webpage. --- ## 🎤 Quick Start ### Browser Extension (Recommended) The easiest way to use voice control with Iris: 1. **Install the extension:** - Open Chrome - Go to `chrome://extensions` - Enable "Developer Mode" - Click "Load unpacked" - Select `chrome_gaze_latch` folder 2. **Start using:** - Click any text field on any webpage - Start speaking - your words appear automatically! **Full documentation:** See `chrome_gaze_latch/README.md` --- ## 📋 Available Options ### Option 1: Browser Extension (Recommended ⭐) **Location:** `chrome_gaze_latch/` **Features:** - ✅ Works on any webpage - ✅ Automatic activation when text fields are focused - ✅ Simple installation (just load in Chrome) - ✅ Integrated with gaze tracking - ✅ Real-time transcription - ✅ Keyboard shortcuts (`Cmd+Shift+S`) **Requirements:** - Chrome browser - Internet connection (uses Google's Web Speech API) **Docs:** - `chrome_gaze_latch/QUICKSTART.md` - Get started in 3 steps - `chrome_gaze_latch/INSTALL.md` - Detailed installation - `chrome_gaze_latch/README.md` - Full documentation --- ### Option 2: Electron App (Legacy) **Location:** `voice/electron/` **Features:** - ✅ System-wide text field detection - ✅ Offline speech recognition (Vosk/Whisper) - ✅ Works outside browser - ⚠️ Complex setup required **Requirements:** - Node.js and Electron - Vosk or Whisper.cpp models - Swift helper app (macOS) **When to use:** - Need offline speech recognition - Need system-wide (non-browser) text input - Need local processing for privacy **Docs:** - `voice/README.md` - Electron app documentation - `voice/ELECTRON_SETUP.md` - Setup guide --- ## 🎯 Which Should I Use? ### Use Browser Extension if: - ✅ You primarily work in the browser (Gmail, Google Docs, web apps) - ✅ You want the simplest setup - ✅ You have internet connection - ✅ You want gaze + voice integration ### Use Electron App if: - ✅ You need system-wide voice control (any app) - ✅ You need offline support - ✅ You require local processing - ✅ You work in desktop apps (not just browser) ### Use Both if: - ✅ You want best of both worlds - ✅ Browser extension for web, Electron for desktop apps - ✅ They can run simultaneously without conflicts --- ## 🔗 Integration with Gaze Tracking Both options can work with the Iris Vision Module for gaze tracking: ```bash # Start the gaze server eyetrax-stream --filter kalman --calibration 9p ``` Then: - **Browser extension** connects automatically via WebSocket - **Electron app** can be enhanced to use gaze data **Complete hands-free workflow:** 1. Look at text field (gaze tracking highlights it) 2. Click (mouse, keyboard, or EEG trigger) 3. Speak (voice recognition fills it in) 4. Look at submit button 5. Click (gaze + confirm trigger) --- ## 🚀 Getting Started ### For Browser-Based Voice Control (Recommended): ```bash # 1. Load e [truncated — 3727 more characters] ``` ### pyproject.toml ``` [build-system] requires = ["hatchling>=1.25"] build-backend = "hatchling.build" [project] name = "vision-module" description = "Webcam-based eye-tracking" readme = "README.md" license = { file = "LICENSE" } authors = [{ name = "Chenkai Zhang (ck-zhang)" }] requires-python = ">=3.9" dynamic = ["version"] dependencies = [ "opencv-python>=4.5", "mediapipe>=0.10", "numpy>=1.22", "scikit-learn>=1.3", "scipy>=1.10", "screeninfo>=0.8", "pyvirtualcam>=0.10", "websockets>=12.0", "fastapi>=0.111", "uvicorn>=0.23", "letta-client>=0.1.0", "python-dotenv>=1.0", ] classifiers = [ "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3 :: Only", "Operating System :: OS Independent", ] [project.urls] homepage = "https://github.com/tgondil/iris" [project.scripts] eyetrax-demo = "vision_module.app.demo:run_demo" eyetrax-virtualcam = "vision_module.app.virtualcam:run_virtualcam" eyetrax-build-model = "vision_module.app.build_model:main" eyetrax-stream = "vision_module.app.stream_gaze:main" [tool.hatch.build.targets.wheel] packages = ["src/vision_module"] [tool.hatch.build] include = [ "src/vision_module/**", "LICENSE", "README.md" ] [tool.hatch.version] path = "src/vision_module/_version.py" ``` ### fetch_agent/requirements.txt ``` uagents==0.22.10 flask==3.1.2 flask-cors==6.0.1 requests==2.32.5 ``` ### voice/package.json ``` { "name": "voice", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint", "electron:dev": "cd electron && npm run dev", "electron:start": "cd electron && npm start", "electron:build": "npm run build && cd electron && npm run build" }, "dependencies": { "lucide-react": "^0.548.0", "next": "16.0.0", "react": "19.2.0", "react-dom": "19.2.0" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.0.0", "tailwindcss": "^4", "typescript": "^5" } } ``` ### eyetrax-fresh/pyproject.toml ``` [build-system] requires = ["hatchling>=1.25"] build-backend = "hatchling.build" [project] name = "eyetrax" description = "Webcam-based eye-tracking" readme = "README.md" license = { file = "LICENSE" } authors = [{ name = "Chenkai Zhang (ck-zhang)" }] requires-python = ">=3.9" dynamic = ["version"] dependencies = [ "opencv-python>=4.5", "mediapipe>=0.10", "numpy>=1.22", "scikit-learn>=1.3", "scipy>=1.10", "screeninfo>=0.8", "pyvirtualcam>=0.10", ] classifiers = [ "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3 :: Only", "Operating System :: OS Independent", ] [project.urls] homepage = "https://github.com/ck-zhang/eyetrax" [project.scripts] eyetrax-demo = "eyetrax.app.demo:run_demo" eyetrax-virtualcam = "eyetrax.app.virtualcam:run_virtualcam" eyetrax-build-model = "eyetrax.app.build_model:main" [tool.hatch.build.targets.wheel] packages = ["src/eyetrax"] [tool.hatch.build] include = [ "src/eyetrax/**", "LICENSE", "README.md" ] [tool.hatch.version] path = "src/eyetrax/_version.py" ``` ### voice/electron/package.json ``` { "name": "voice-keyboard-electron", "version": "1.0.0", "description": "System-wide voice keyboard overlay", "main": "main.js", "scripts": { "start": "electron .", "dev": "ELECTRON_START_URL=http://localhost:3000 NODE_ENV=development electron .", "build": "electron-builder", "pack": "electron-builder --dir" }, "keywords": [ "electron", "voice", "keyboard", "overlay" ], "author": "", "license": "MIT", "dependencies": { "express": "^4.18.2", "mic": "^2.1.2", "node-record-lpcm16": "^1.0.1", "sox-audio": "^0.3.0", "vosk": "^0.3.39" }, "devDependencies": { "electron": "^28.0.0", "electron-builder": "^24.9.1" }, "build": { "appId": "com.voicekeyboard.app", "productName": "VoiceKeyboard", "mac": { "category": "public.app-category.utilities", "icon": "build/icon.icns", "target": [ "dmg", "zip" ], "extendInfo": { "LSUIElement": true } }, "dmg": { "contents": [ { "x": 130, "y": 220 }, { "x": 410, "y": 220, "type": "link", "path": "/Applications" } ] }, "files": [ "main.js", "preload.js", "../out/**/*" ] } } ``` ### src/vision_module/cli.py ```python import argparse def parse_common_args(): parser = argparse.ArgumentParser(description="Common Gaze Estimation Arguments") parser.add_argument( "--filter", choices=["kalman", "kde", "none"], default="none", help="Select the filter to apply to gaze estimation, options are 'kalman', 'kde', or 'none'", ) parser.add_argument( "--camera", type=int, default=0, help="Camera index for video capture, default is 0 (first camera)", ) parser.add_argument( "--calibration", choices=["9p", "5p", "lissajous"], default="9p", help="Calibration method for gaze estimation, options are '9p', '5p', or 'lissajous'", ) parser.add_argument( "--background", type=str, default=None, help="Path to a custom background image (optional)", ) parser.add_argument( "--confidence", type=float, default=0.5, help="Confidence level for KDE smoothing, range 0 to 1", ) parser.add_argument( "--model", default="ridge", help="The machine learning model to use for gaze estimation, default is 'ridge'", ) parser.add_argument( "--model-file", type=str, default=None, help="Path to a previously-trained gaze model", ) return parser.parse_args() ``` ### letta/app.py ```python import asyncio from typing import Optional, Dict, Any, List import json import re import websockets from fastapi import WebSocket, WebSocketDisconnect from fastapi import FastAPI, HTTPException from fastapi.responses import JSONResponse from fastapi.middleware.cors import CORSMiddleware from letta_client import Letta, MessageCreate, TextContent from .config import LettaConfig from .schemas import PredictRequest, PredictResponse app = FastAPI(title="Letta Bridge API", version="0.1.0") # Add CORS middleware to allow requests from Chrome extensions and web pages app.add_middleware( CORSMiddleware, allow_origins=["*"], # Allow all origins for local development allow_credentials=True, allow_methods=["*"], # Allow all methods (GET, POST, OPTIONS, etc.) allow_headers=["*"], # Allow all headers ) @app.get("/health") def health(): return {"status": "ok"} def _format_gazed_element(elem: Optional[Dict[str, Any]]) -> str: """Format the currently gazed element with full surrounding context""" if not elem: return "(none - not gazing at any interactive element)" lines = [] # Primary element info parts = [] tag = elem.get("tag_name") or elem.get("tagName") if tag: parts.append(f"<{tag}>") elem_id = elem.get("id") if elem_id: parts.append(f"#{elem_id}") classes = elem.get("classes") or [] if classes and isinstance(classes, list): parts.append(f".{'.'.join(classes[:2])}") lines.append("PRIMARY ELEMENT: " + " ".join(parts)) # Text/Label/Placeholder label = elem.get("label") if label: lines.append(f" Associated Label: \"{label[:60]}\"") text = elem.get("text_content") or elem.get("textContent") if text and text.strip(): lines.append(f" Text Content: \"{text.strip()[:50]}\"") placeholder = elem.get("placeholder") if placeholder: lines.append(f" Placeholder: \"{placeholder}\"") aria = elem.get("aria_label") or elem.get("ariaLabel") if aria: lines.append(f" ARIA Label: \"{aria}\"") # Attributes role = elem.get("role") elem_type = elem.get("type") if role or elem_type: attrs = [] if role: attrs.append(f"role={role}") if elem_type: attrs.append(f"type={elem_type}") lines.append(f" Attributes: {', '.join(attrs)}") # Parent context parent = elem.get("parent") if parent: parent_parts = [f"<{parent.get('tag_name', '?')}>"] if parent.get("id"): parent_parts.append(f"#{parent['id']}") if parent.get("classes"): parent_parts.append(f".{'.'.join(parent['classes'][:2])}") lines.append(f" Parent: {' '.join(parent_parts)}") # Siblings context siblings = elem.get("siblings") if siblings: if siblings.get("prev"): prev = siblings["prev"] prev_text = prev.get("text", "")[:30] if prev.get("text") else "" lines.append(f" Previous Sibling: <{prev.get('tag_name', '?')}> {('\"' + prev_text + '\"') if prev_text else ''}") if siblings.get("next"): next_elem = siblings["next"] next_text = next_elem.get("text", "")[:30] if next_elem.get("text") else "" lines.append(f" Next Sibling: <{next_elem.get('tag_name', '?')}> {('\"' + next_text + '\"') if next_text else ''}") # Form context form_context = elem.get("form_context") if form_context: lines.append(f" Form Context: {form_context.get('fields_count', 0)} fields | " f"Action: {form_context.get('action', 'N/A')} | " f"Method: {form_context.get('method', 'N/A')}") return "\n".join(lines) if lines else "(unidentified element)" def _build_prompt(req: "PredictRequest") -> str: """Build prompt using actual chrome_gaze_latch extension data""" # Format page content from extension page_summary = [] page_summary.append(f"URL: {req.url}") page_summary.append(f"Title: {req.title}") # Add headings if req.content.headings: page_summary.append("\nPage Headings:") for h in req.content.headings[:10]: # Top 10 headings page_summary.append(f" {h.get('level', 'h?')}: {h.get('text', '')[:100]}") # Add key buttons if req.content.buttons: page_summary.append(f"\nButtons: {', '.join([b[:30] for b in req.content.buttons[:10]])}") # Add key links if req.content.links: page_summary.append("\nKey Links:") for link in req.content.links[:5]: page_summary.append(f" - {link.get('text', '')[:50]}") # Add form inputs if req.content.inputs: page_summary.append("\nForm Fields:") for inp in req.content.inputs[:10]: inp_desc = f" - {inp.get('type', 'input')}" if inp.get('placeholder'): inp_desc += f" (placeholder: {inp['placeholder'][:30]})" page_summary.append(inp_desc) # Add main content if req.main_content: page_summary.append(f"\nMain Content (excerpt):\n{req.main_content[:500]}") page_context = "\n".join(page_summary) # Format gazed element gazed_elem_dict = req.gazed_element.dict() if req.gazed_element else None gaze_info = _format_gazed_element(gazed_elem_dict) # Optional user profile for autofill user_profile_block = "" if req.user_profile: try: profile_str = json.dumps(req.user_profile, ensure_ascii=False) user_profile_block = ( "\nUSER_PROFILE (use to autofill relevant fields; do NOT fabricate unknowns):\n" + profile_str + "\n" ) except Exception: pass return ( "You are an intelligent browser action planner integrated with eye-tracking technology.\n\n" "════════ [truncated — 19326 more characters] ``` ### voice/electron/main.js ```javascript const { app, BrowserWindow, ipcMain } = require('electron'); const express = require('express'); const path = require('path'); const VoskSpeechRecognition = require('./vosk-service'); let mainWindow = null; let isKeyboardVisible = false; const serverPort = 5000; let speechService = null; // Set up permissions for media devices function setupPermissions() { const { session } = require('electron'); // Handle permission requests session.defaultSession.setPermissionRequestHandler((webContents, permission, callback) => { console.log('🔐 Permission requested:', permission); // Allow microphone access for speech recognition if (permission === 'media' || permission === 'microphone') { console.log('✅ Granting microphone permission'); callback(true); } else if (permission === 'geolocation') { callback(false); } else { callback(true); } }); // Check permissions session.defaultSession.setPermissionCheckHandler((webContents, permission, requestingOrigin, details) => { console.log('🔍 Permission check:', permission, 'from', requestingOrigin); if (permission === 'media' || permission === 'microphone') { return true; } return true; }); // Allow speech recognition network requests session.defaultSession.webRequest.onBeforeSendHeaders((details, callback) => { // Allow requests to Google's speech API callback({ requestHeaders: details.requestHeaders }); }); // Set user agent to ensure speech API compatibility session.defaultSession.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'); console.log('🌐 Network access configured for speech recognition'); } // Create the floating keyboard window function createWindow() { mainWindow = new BrowserWindow({ width: 700, height: 400, alwaysOnTop: true, frame: false, transparent: true, skipTaskbar: true, hasShadow: true, resizable: false, focusable: false, webPreferences: { nodeIntegration: false, contextIsolation: true, preload: path.join(__dirname, 'preload.js'), // Enable web APIs including speech recognition enableRemoteModule: false, // IMPORTANT: Disable webSecurity to allow speech API network requests webSecurity: false, // Allow media devices (microphone) allowRunningInsecureContent: false } }); // Load your React app // In development: http://localhost:3000 // In production: load from build folder const startUrl = process.env.ELECTRON_START_URL || `file://${path.join(__dirname, '../out/index.html')}`; mainWindow.loadURL(startUrl); // Make it visible on all workspaces and full screen apps mainWindow.setVisibleOnAllWorkspaces(true, { visibleOnFullScreen: true }); // Initially hide the window mainWindow.hide(); // Prevent window from gaining focus mainWindow.setAlwaysOnTop(true, 'floating'); mainWindow.setVisibleOnAllWorkspaces(true); mainWindow.on('closed', () => { mainWindow = null; }); // Enable web APIs mainWindow.webContents.session.setPermissionRequestHandler((webContents, permission, callback) => { if (permission === 'media' || permission === 'microphone') { callback(true); } else { callback(false); } }); // Open DevTools in development if (process.env.NODE_ENV === 'development') { mainWindow.webContents.openDevTools({ mode: 'detach' }); } // Log when page loads mainWindow.webContents.on('did-finish-load', () => { console.log('📄 Page loaded in Electron window'); console.log('🎤 Speech Recognition should now be available'); }); } // Create express server to receive signals from Swift helper function startControlServer() { const expressApp = express(); expressApp.get('/show', (req, res) => { console.log('📥 Received SHOW signal from Swift helper'); if (mainWindow && !isKeyboardVisible) { showKeyboard(); } res.sendStatus(200); }); expressApp.get('/hide', (req, res) => { console.log('📥 Received HIDE signal from Swift helper'); if (mainWindow && isKeyboardVisible) { hideKeyboard(); } res.sendStatus(200); }); expressApp.get('/toggle', (req, res) => { console.log('📥 Received TOGGLE signal from Swift helper'); if (mainWindow) { if (isKeyboardVisible) { hideKeyboard(); } else { showKeyboard(); } } res.sendStatus(200); }); expressApp.get('/status', (req, res) => { res.json({ visible: isKeyboardVisible }); }); expressApp.listen(serverPort, '127.0.0.1', () => { console.log(`🚀 Control server listening on http://127.0.0.1:${serverPort}`); console.log(' Endpoints:'); console.log(' - GET /show - Show keyboard'); console.log(' - GET /hide - Hide keyboard'); console.log(' - GET /toggle - Toggle keyboard'); console.log(' - GET /status - Get visibility status'); }); } function showKeyboard() { if (mainWindow) { // Position at bottom center of screen const { screen } = require('electron'); const primaryDisplay = screen.getPrimaryDisplay(); const { width, height } = primaryDisplay.workAreaSize; const windowBounds = mainWindow.getBounds(); const x = Math.round((width - windowBounds.width) / 2); const y = height - windowBounds.height - 50; // 50px from bottom mainWindow.setPosition(x, y); mainWindow.showInactive(); // Show without stealing focus isKeyboardVisible = true; console.log('⌨️ Keyboard shown'); } } function hideKeyboard() { if (mainWindow) { mainWindow.hide(); isKeyboardVisible = false; console.log('🙈 Keyboard hidden'); } } // Initialize speech service async function initSpeechService() { const modelPath = path.join(__dirname, '../vosk-model-small-en-us-0.15'); speechService = new VoskSpeechRecognition(modelPath) [truncated — 2115 more characters] ``` [126 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]