# Project export: MindOS This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Control your computer with JUST your mind. - Devpost: https://devpost.com/software/mindos-h6q7ve - GitHub: https://github.com/shawncshen/treehacks-2026/ - Demo: https://treehacks-2026-gamma.vercel.app/ - Video: https://www.youtube.com/embed/v-DJEQgKm-A?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) β€” Ansh2 (7 commits), Shawn Shen (3 commits), Arnav2610 (2 commits) ## Devpost submission (written by the team) ### Overview Technical Overview https://www.loom.com/share/5348b74c2fd14f47992c22fadabc2f57 ### Inspiration Imagine having a perfectly clear mind but being unable to communicate it to the world. For millions of people living with ALS, recovering from a stroke, or dealing with severe paralysis, the standard ways we interact with technology like typing on a keyboard or speaking to a voice assistant are often impossible. We realized that current tools fail exactly where they are needed most and for users who physically cannot use their hands or produce clear speech. This gap in assistive technology is what drove us to build MindOS. We wanted to create a solution that works even when a user is silent to provide a lifeline for communication and control without requiring direct brain implants or invasive procedures. ### What it does MindOS is a silent speech AI agent for your computer powered by micro muscle signals. With sensors placed along the jaw and throat region, a user can silently think about certain actions without making a sound. The system decodes that intent to control a computer. It allows for reliable actions like silent web browsing where a user can scroll, go back, or search. Unlike standard voice control, MindOS works when speech is difficult or impossible and supports hands-free usage. It also incorporates a frontend with inference that enables users to append new training data. This means the software learns based on any new user rather than just guessing. ### How we built it We built MindOS as a modular pipeline that connects signals to digital actions. Hardware & Signal Ingestion: On the hardware side, we used two Myoware muscle sensors attached to the jaw via electrodes. These sensors capture raw EMG data, which is passed to an Arduino Uno. We utilized PySerial to ingest this serial port data into our laptop for real-time processing. Signal Processing: We treated the muscle signals as time-series data, applying noise handling to filter out motion artifacts. We then processed the data with a classification-based Random Forest model to categorize English phonemes into four distinct biometric signatures based on EMG muscle activation. AI Agents: Since raw signals are ambiguous, we implemented a multi-agent workflow to bridge the gap between signal and action: Context Agent: Because we were mapping limited signal categories to the complexity of human language, we built an intelligent agent to suggest the correct character or word based on context. Action Agent: Once the intent is understood, a second agent decides which specific browser action to take. Execution Agent: We used Playwright to drive the actual browser automation based on the Action Agent's decisions. ### Challenges we ran into Our biggest hurdle was the hardware limitation. We ideally wanted to build a 26-class classifier to map signals directly to the alphabet. However, since we were limited to two muscle sensors and fewer than 20 electropads, we couldn't capture enough distinct data points for that level of granularity. We were stuck with just 4 distinct signal categories trying to map to 26 letters. To bridge this gap, we first implemented an exhaustive-but-pruned greedy search that enumerates possible letter sequences from the low-entropy signal stream and filters them using an English lexicon and frequency priors, dramatically shrinking the candidate space to linguistically plausible outputs. On top of that, we deployed a lightweight GPT-mini context agent that evaluates the remaining candidates against preceding text, grammar, and semantic coherence to select the most likely intended sequence in real time. This allows the system to ignore invalid combinations and lock onto the correct word based on probability and context. Finally, ensuring the agent didn't hallucinate actions was difficult. We solved this by leveraging conservative AI prompts rather than giving it total freedom. This balance between speed and reliability was tough to find, but limiting the command set ensured our demos were high-confidence rather than chaotic. ### Accomplishments we're proud of We shipped an end-to-end assistive interface that connects biological sensors to real computer actions within a hackathon timeframe. We are particularly proud of our real-time EMG decoding pipeline and the robust API boundary we built for fast iteration. We achieved 96% accuracy on our model using a dataset of six hours of raw sub-vocal recordings, which we augmented to simulate variance and noise. Additionally, we successfully integrated multi-layer LLMs (using GPT-4o) to drive the decision-making process, proving that we could control a browser with nothing but silent intent. ### What we learned We learned that the hardest part isn't just modeling. It is making the experience reliable when dealing with the messiness of real-world sensors. Electrode placement, skin contact, and small movements can shift signals significantly. We also discovered that constraining the interaction space dramatically improves trust; tool-based execution is essential for predictable agents. On a personal level, we learned the value of perseverance. Even when the hardware signals were noisy or the model failed to generalize, we pushed through to refine our pipeline until it worked. ### What's next Next, we want to improve the calibration features so the system adapts quickly to new users. We plan to expand the vocabulary of commands while keeping reliability high, using our feedback logs to continually improve decoding accuracy. For the future, we hope to implement multi-sensor fusion across the face to capture accurate and faster data. We also aim to move to on-device processing. By moving to an on-edge device, we can reduce latency and ensure that MindOS becomes a private, secure, and viable daily-use product for many people in need. ## README (from the GitHub repository) # MindOS β€” TreeHacks 2026 MindOS is a non-invasive silent speech interface that converts micro muscle signals (EMG) from the jaw and throat into real computer actions. It enables users to communicate and control software without speaking or using their hands. --- ## πŸŽ₯ Demo [![Watch the demo](https://img.youtube.com/vi/v-DJEQgKm-A/maxresdefault.jpg)](https://www.youtube.com/watch?v=v-DJEQgKm-A) --- ## πŸ’‘ Inspiration Imagine having a perfectly clear mind but being unable to communicate it to the world. Millions of people living with ALS, recovering from stroke, or experiencing severe paralysis cannot rely on keyboards or speech interfaces. Existing tools fail exactly where assistive technology matters most. We built MindOS to create a non-invasive interface that translates silent intent into digital action β€” providing a pathway for communication and control without implants, speech, or physical movement. --- ## What it Does - **Silent intent β†’ action** Users silently mouth commands detected through EMG signals - **Hands-free computer control** Browse, search, scroll, and navigate without a keyboard or voice - **Adaptive learning loop** Users can append new training data to personalize decoding - **Assistive-first design** Built for people who cannot rely on speech or motor input --- ## How It Works MindOS is a modular pipeline that connects biological signals to digital actions. ### 1. Hardware & Signal Ingestion - Two **MyoWare EMG muscle sensors** placed along the jaw - Signals routed through an **Arduino Uno** - Real-time streaming via **PySerial** into the processing pipeline ### 2. Signal Processing - Treated EMG as **time-series data** - Applied noise filtering to remove motion artifacts - **Random Forest classifier** maps signals to four biometric phoneme classes ### 3. Multi-Agent Decision Layer Because EMG signals are low-entropy, we designed a structured agent workflow: - **Context Agent** Infers likely characters/words using linguistic priors - **Action Agent** Determines which computer command the user intends - **Execution Agent** Uses **Playwright** to perform browser automation safely ### Reliability Layer A plug-and-play signal interface allows switching between: - Live sensor input - Mock signals for deterministic demos --- **Key Components** - EMG signal ingestion via PySerial - Random Forest classification - Lexicon-filtered candidate generation - LLM-based contextual disambiguation - Tool-constrained agent execution - Browser automation with Playwright ## Tech Stack **Hardware** - MyoWare EMG sensors - Arduino Uno **ML / Signal Processing** - Python - Scikit-learn (Random Forest) - Time-series preprocessing **AI & Agents** - GPT-4o (decision layer) - Custom context + action agents **Automation** - Playwright **Frontend / Interface** - Web UI for calibration + data collection --- ## Challenges ### Limited Signal Resolution With only two sensors, we could not build a full 26-class alphabet classifier. We were constrained to **four signal categories**. --- ## Accomplishments - Built a full end-to-end assistive interface in a hackathon timeframe - Achieved **96% classification accuracy** on six hours of EMG data - Successfully controlled a live browser using silent intent - Designed a robust modular API for rapid iteration --- ## What We Learned - Real-world biosignals are noisy and highly user-dependent - Calibration and UX matter as much as model accuracy - Constrained agent design dramatically improves trust - Hardware-software co-design is essential for assistive tech --- ## What’s Next - Faster calibration for new users - Expanded command vocabulary - Multi-sensor fusion across facial muscle groups - On-device inference for lower latency and better privacy - Continuous learning from feedback logs --- ## Why It Matters - Impact We started this project because we know people personally that would want to work or even just be on their computer, but cannot. MindOS demonstrates that assistive computing doesn’t require invasive brain implants. By decoding neuromuscular signals already present during silent speech, we can create interfaces that restore autonomy and communication for millions of people. --- ## Acknowledgements Thanks to the mentors, organizers, sponsors, and everyone else for their support at TreeHacks 2026. --- ## Detected evidence (automated analysis) Indexed codebase: 74 recognized source files, 534 KB. - CSS (language) β€” detected in the code - Express (technology) β€” detected in the code - FastAPI (technology) β€” detected in the code - JavaScript (language) β€” detected in the code - Next.js (technology) β€” detected in the code - OpenAI (technology) β€” detected in the code - Python (language) β€” detected in the code - React (technology) β€” detected in the code - Tailwind CSS (technology) β€” detected in the code - TypeScript (language) β€” detected in the code - C++ (language) β€” claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (87 of 87) ``` .gitignore actions/__init__.py actions/browser.py actions/config.py actions/cursor.py actions/engine.py actions/gui_overlay.py actions/main.py actions/overlay.py actions/test_agent.py actions/test_cursor.py actions/vision.py actions/word_disambiguate.py actions/word_finder.py actions/words_common.txt README.md silentpilot/.env.example silentpilot/.gitignore silentpilot/agent/package.json silentpilot/agent/src/command_router.ts silentpilot/agent/src/index.ts silentpilot/agent/src/orchestrator.ts silentpilot/agent/src/prompts.ts silentpilot/agent/src/state.ts silentpilot/agent/tsconfig.json silentpilot/alterego_paper.md silentpilot/app_ui/components/CommandOverlay.tsx silentpilot/app_ui/components/ConfusionMatrix.tsx silentpilot/app_ui/components/PTTButton.tsx silentpilot/app_ui/components/SignalPlot.tsx silentpilot/app_ui/lib/api.ts silentpilot/app_ui/lib/ws.ts silentpilot/app_ui/next.config.js silentpilot/app_ui/package.json silentpilot/app_ui/pages/_app.tsx silentpilot/app_ui/pages/calibrate.tsx silentpilot/app_ui/pages/demo.tsx silentpilot/app_ui/pages/index.tsx silentpilot/app_ui/pages/train.tsx silentpilot/app_ui/postcss.config.js silentpilot/app_ui/styles/globals.css silentpilot/app_ui/tailwind.config.js silentpilot/app_ui/tsconfig.json silentpilot/app_ui/tsconfig.tsbuildinfo silentpilot/CLAUDE_SESSION_NOTES.md silentpilot/DATA_CAPTURE_PLAN.md silentpilot/DEMO_IDEAS.md silentpilot/docker-compose.yml silentpilot/emg_core/__init__.py silentpilot/emg_core/api/__init__.py silentpilot/emg_core/api/schemas.py silentpilot/emg_core/api/server.py silentpilot/emg_core/config.py silentpilot/emg_core/dsp/__init__.py silentpilot/emg_core/dsp/features.py silentpilot/emg_core/dsp/filters.py silentpilot/emg_core/dsp/normalize.py silentpilot/emg_core/dsp/segment.py silentpilot/emg_core/ingest/__init__.py silentpilot/emg_core/ingest/base_reader.py silentpilot/emg_core/ingest/mock_reader.py silentpilot/emg_core/ingest/packet_parser.py silentpilot/emg_core/ingest/serial_reader.py silentpilot/emg_core/ml/__init__.py silentpilot/emg_core/ml/infer.py silentpilot/emg_core/ml/model_io.py silentpilot/emg_core/ml/train.py silentpilot/emg_core/pipeline.py silentpilot/firmware/emg_streamer/emg_streamer.ino silentpilot/HACKATHON_3SENSOR_PLAN.md silentpilot/mcp_server/package.json silentpilot/mcp_server/src/server.ts silentpilot/mcp_server/src/tools/browser.ts silentpilot/mcp_server/src/tools/index.ts silentpilot/mcp_server/tsconfig.json silentpilot/pyproject.toml silentpilot/README.md silentpilot/RESEARCH_REPORT.md silentpilot/scripts/e2e_test_full.py silentpilot/scripts/e2e_test.py silentpilot/scripts/emguka_arch_compare.py silentpilot/scripts/emguka_cnn.py silentpilot/scripts/emguka_commands.py silentpilot/scripts/emguka_llm_decode.py silentpilot/scripts/emguka_phones.py silentpilot/scripts/emguka_pipeline.py silentpilot/scripts/emguka_synthetic.py ``` ### Dependencies - silentpilot/agent/package.json: @types/express@^5.0.0, @types/node@^22.0.0, @types/ws@^8.5.0, dotenv@^16.4.0, express@^4.21.0, openai@^4.77.0, tsx@^4.19.0, typescript@^5.7.0, ws@^8.18.0 - silentpilot/app_ui/package.json: @types/node@^22.0.0, @types/react@^18.3.0, @types/react-dom@^18.3.0, autoprefixer@^10.4.0, next@^14.2.0, postcss@^8.4.0, react@^18.3.0, react-dom@^18.3.0, recharts@^2.13.0, tailwindcss@^3.4.0, typescript@^5.7.0 - silentpilot/mcp_server/package.json: @modelcontextprotocol/sdk@^1.12.0, @types/express@^5.0.0, @types/node@^22.0.0, express@^4.21.0, playwright@^1.49.0, tsx@^4.19.0, typescript@^5.7.0, zod@^3.23.0 - silentpilot/pyproject.toml: fastapi@>=0.104.0, joblib@>=1.3.0, matplotlib, numpy@>=1.24.0, pydantic@>=2.5.0, pyserial@>=3.5, pytest, pytest-asyncio, python-dotenv@>=1.0.0, scikit-learn@>=1.3.0, scipy@>=1.11.0, uvicorn[standard]@>=0.24.0, websockets@>=12.0 ### Recent commits (newest first) - Revise README for comprehensive project overview - Update README.md - Initialize README with MindOS project details - updating chromium with amazon persistance - revamped logic for silent pilot implementation - better - EMG sensor input simulated - GUI update and agent understand more - update agent-dev - Add GUI mode, stealth Chrome browser, typing animation, and agent improvements - update to engine - major update - kinda cookin - adding back and forward - making scrolling scroll more and smoother each time - screen control speed is now way faster - Basic GUI Implemented - fixing screen sizing problem - first working version of browser control - Initial commit: SilentPilot + EMG-UKA Trial Corpus ## Key source files (fetched from GitHub, selected and truncated for size) ### silentpilot/CLAUDE_SESSION_NOTES.md ```markdown # MindOS β€” Claude Session Notes ## Project Location - Repo: `/Users/shawnshen/Downloads/treehacks/` (git root) - GitHub: `https://github.com/ansht3/treehacks-2026.git` - Main code: `treehacks/silentpilot/` (Python EMG signal processing + classification) - EMG dataset: `treehacks/EMG-UKA-Trial-Corpus/` (only on `datasets` branch) ## Git Structure - **`datasets` branch**: Contains both `EMG-UKA-Trial-Corpus/` AND `silentpilot/` code - **`algorithm` branch**: Empty (no commits) β€” do NOT use - All work should go on `datasets` branch ## What Was Done (Feb 14, 2026 session) ### 1. EMG-UKA TD0/TD10 Feature Pipeline Integration Replaced the original AlterEgo-inspired features (87 dims) with EMG-UKA TD0/TD10 features. **Files modified:** #### `emg_core/dsp/features.py` - Original `extract_features()` renamed to `extract_features_legacy()` for backward compat - Added TD0/TD10 pipeline functions: - `_double_moving_average(x)` β€” 9-point double-average β†’ smoothed signal w[n] - `_compute_td0_channel()` β€” 5 features per frame: w_bar, P_w, P_r, z_p, r_bar - `_stack_context()` β€” Β±10 frame context stacking (TD0 β†’ TD10) - `extract_features_td10()` β€” frame-level features (420 dims for 4ch) - `extract_features_td10_segment()` β€” segment-level aggregation (mean+std = 840 dims) - `extract_features()` now calls TD10 pipeline by default #### `emg_core/config.py` - Added: `TD10_FRAME_SIZE_MS=27`, `TD10_FRAME_SHIFT_MS=10`, `TD10_CONTEXT=10` - Added: `LDA_COMPONENTS=32`, `CLASSIFIER_TYPE="rf"`, `RF_N_ESTIMATORS=200` #### `emg_core/ml/train.py` - Added `RandomForestClassifier` (default) and `LinearDiscriminantAnalysis` - Pipeline: `StandardScaler β†’ LDA (n_components=min(32, n_classes-1)) β†’ RF` - Factored into `_build_pipeline(n_classes)` #### `emg_core/ml/infer.py` - Updated docstring only (feature calls auto-use TD10 via updated `extract_features`) ### 2. Security Fix - `scripts/emguka_llm_decode.py` line 47: Removed hardcoded OpenAI API key - Replaced with `os.getenv('OPENAI_API_KEY')` ### 3. Test Results - `scripts/e2e_test.py` (4 commands): **100%** accuracy (target β‰₯80%) - `scripts/e2e_test_full.py` (8 commands): **93.3%** accuracy (target β‰₯90%) - Both use `MockReader` with synthetic EMG data, not real sensors ## Feature Dimension Math (4 channels, 250Hz) - TD0 per frame per channel: 5 features - TD0 per frame all channels: 5 Γ— 4 = 20 - TD10 stacking Β±10 frames: 20 Γ— 21 = 420 dimensions - Segment-level (mean + std): 420 Γ— 2 = 840 dimensions - After LDA: 32 dimensions (or n_classes - 1 if fewer classes) ## EMG-UKA Corpus Parameters (for reference) - Corpus path: `~/.cache/kagglehub/datasets/xabierdezuazo/emguka-trial-corpus/versions/1/EMG-UKA-Trial-Corpus` - Sampling rate: 600 Hz, 6 EMG channels (7 total, first 6 used) - Bandpass: 1.3–50 Hz, 4th-order Butterworth; Notch: 60 Hz, Q=30 - LLM decode script uses: RF manner classifier (6 classes) + RF phone classifier (41 phones) β†’ GPT-4.1-nano ## Important Notes - `.env` contains real API keys β€” never c [truncated β€” 270 more characters] ``` ### silentpilot/HACKATHON_3SENSOR_PLAN.md ```markdown # MindOS: 3-Sensor Hackathon Build Plan ## What We Have, What We Can Build, What to Expect --- ## The Hardware **3 Γ— MyoWare 2.0 Muscle Sensors** - Analog output: 0–3.3V (ENV mode = smoothed envelope, RAW mode = amplified EMG) - Onboard gain: 200Γ— base, adjustable via potentiometer - Bandpass: 20.8–498.4 Hz built-in (good β€” captures speech EMG range) - Each sensor has 3 snap electrodes: MID (center of muscle), END (along muscle), REF (nearby bone/non-active area) **ESP32 (from our existing firmware)** - 4 ADC pins available (GPIO 32, 33, 34, 35) β€” we use 3 - 12-bit ADC, 250 Hz sample rate - Binary packet protocol already implemented in `firmware/emg_streamer/` - Serial β†’ Python pipeline already implemented in `emg_core/ingest/serial_reader.py` **What this means:** The software stack is ready. We connect 3 MyoWare outputs to 3 ESP32 ADC pins and the entire pipeline from hardware β†’ features β†’ classifier β†’ WebSocket β†’ UI works out of the box. --- ## Sensor Placement: Where to Put the 3 Sensors With only 3 sensors, every placement decision matters. Here's the optimal configuration, ranked by information value for silent speech: ``` FRONT VIEW SIDE VIEW β”Œβ”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ FACE β”‚ β”Œβ”€β”€β”€ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ ← [1] Jaw (Masseter) [1]β†’ β”‚ β–ˆβ–ˆ β”‚ β”‚ β”‚ Right side of jaw β”‚ β”‚ β”‚ β”‚ β””β”€β”€β”€β”¬β”€β”€β”€β”˜ └─── β”‚ β”‚ β”Œβ”€β”€β”€β”€β”΄β”€β”€β”€β”€β” β”Œβ”€β”€β”€ β”‚ CHIN β”‚ ← [2] β”‚ β”‚ ← [2] Under chin (Submental) β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜ β”‚ β”‚ Centered below jawline β”‚ └─── β”Œβ”€β”€β”€β”€β”΄β”€β”€β”€β”€β” β”‚ β”‚ THROAT β”‚ ← [3] β”‚ ← [3] Throat (Laryngeal) β”‚ β”‚ β”‚ Over the larynx/Adam's apple β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ ``` ### Sensor 1: MASSETER (Jaw) β€” GPIO 32 **Location:** Right cheek, over the jaw muscle. Place your fingers on your cheek and clench your teeth β€” the muscle that bulges is the masseter. **What it captures:** - Jaw opening/closing (distinguishes vowel heights) - Bite force (distinguishes stops like T/D/K from fricatives like S/F) - General "speech is happening" activation **Electrode placement:** - MID electrode: center of the masseter (where it bulges most when clenching) - END electrode: toward the jaw hinge (near the ear) - REF electrode: on the cheekbone (zygomatic arch) β€” bony, minimal muscle **MyoWare physical fit:** Masseter is large enough for the MyoWare board. Orient the board vertically along the jaw. ### Sensor 2: SUBMENTAL (Under Chin) β€” GPIO 33 **Location:** Centered under the chin, on the soft tissue between the jawbone. This captures tongue and hyoid bone movement β€” the **single most informative [truncated β€” 13719 more characters] ``` ### silentpilot/pyproject.toml ``` [build-system] requires = ["setuptools>=68.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "silentpilot-emg-core" version = "0.1.0" description = "sEMG signal processing and classification for silent speech interface" requires-python = ">=3.10" dependencies = [ "fastapi>=0.104.0", "uvicorn[standard]>=0.24.0", "websockets>=12.0", "pyserial>=3.5", "numpy>=1.24.0", "scipy>=1.11.0", "scikit-learn>=1.3.0", "joblib>=1.3.0", "pydantic>=2.5.0", "python-dotenv>=1.0.0", ] [project.optional-dependencies] dev = ["pytest", "pytest-asyncio", "matplotlib"] [tool.setuptools.packages.find] include = ["emg_core*"] ``` ### silentpilot/docker-compose.yml ```yaml # Optional: run all services with Docker Compose # Usage: docker compose up version: "3.9" services: emg-core: build: context: . dockerfile: Dockerfile.emg ports: - "8000:8000" environment: - EMG_READER=mock volumes: - ./data:/app/data - ./models:/app/models mcp-server: build: context: ./mcp_server ports: - "3333:3333" environment: - BROWSER_HEADLESS=true agent: build: context: ./agent ports: - "9000:9000" environment: - EMG_WS_URL=ws://emg-core:8000/ws/live - MCP_SERVER_URL=http://mcp-server:3333/sse - OPENAI_API_KEY=${OPENAI_API_KEY} depends_on: - emg-core - mcp-server ui: build: context: ./app_ui ports: - "3000:3000" depends_on: - emg-core - agent ``` ### silentpilot/mcp_server/package.json ``` { "name": "silentpilot-mcp-server", "version": "0.1.0", "description": "MCP server with Playwright-backed browser tools for MindOS", "type": "module", "scripts": { "dev": "tsx src/server.ts", "build": "tsc", "start": "node dist/server.js" }, "dependencies": { "@modelcontextprotocol/sdk": "^1.12.0", "express": "^4.21.0", "playwright": "^1.49.0", "zod": "^3.23.0" }, "devDependencies": { "@types/express": "^5.0.0", "@types/node": "^22.0.0", "tsx": "^4.19.0", "typescript": "^5.7.0" } } ``` ### silentpilot/agent/package.json ``` { "name": "silentpilot-agent", "version": "0.1.0", "description": "Agent orchestrator for MindOS - bridges EMG commands to OpenAI + MCP", "type": "module", "scripts": { "dev": "tsx src/index.ts", "build": "tsc", "start": "node dist/index.js" }, "dependencies": { "openai": "^4.77.0", "ws": "^8.18.0", "express": "^4.21.0", "dotenv": "^16.4.0" }, "devDependencies": { "@types/node": "^22.0.0", "@types/ws": "^8.5.0", "@types/express": "^5.0.0", "tsx": "^4.19.0", "typescript": "^5.7.0" } } ``` ### silentpilot/app_ui/package.json ``` { "name": "silentpilot-ui", "version": "0.1.0", "description": "Calibration and live demo UI for MindOS", "private": true, "scripts": { "dev": "next dev -p 3000", "build": "next build", "start": "next start" }, "dependencies": { "next": "^14.2.0", "react": "^18.3.0", "react-dom": "^18.3.0", "recharts": "^2.13.0" }, "devDependencies": { "@types/node": "^22.0.0", "@types/react": "^18.3.0", "@types/react-dom": "^18.3.0", "autoprefixer": "^10.4.0", "postcss": "^8.4.0", "tailwindcss": "^3.4.0", "typescript": "^5.7.0" } } ``` ### actions/main.py ```python """Entry point for MindOS β€” supports GUI interactive mode and autonomous agent mode.""" import argparse import asyncio import queue import traceback from actions.config import OPENAI_API_KEY, MAX_AGENT_STEPS # ── Autonomous agent mode ────────────────────────────────────────── async def run_agent(goal: str, url: str, max_steps: int): """Launch browser and run autonomous agent with the given goal.""" from actions.engine import AutonomousAgent if not OPENAI_API_KEY: print("ERROR: OPENAI_API_KEY not set. Check silentpilot/.env", flush=True) return agent = AutonomousAgent(max_steps=max_steps) try: print(" Launching browser...", flush=True) await agent.start(url) print(" Browser ready!\n", flush=True) await agent.run(goal) except KeyboardInterrupt: print("\n Interrupted by user.", flush=True) except Exception as e: print(f"\n Fatal error: {e}", flush=True) traceback.print_exc() finally: await agent.stop() # ── GUI interactive mode ─────────────────────────────────────────── async def run_gui_loop(url: str, command_queue: queue.Queue, overlay): """Main loop: EMG thought-class buttons + free entry. No DOM action suggestions.""" from actions.engine import ActionEngine from actions.word_finder import find_words_for_sequence from actions.word_disambiguate import pick_best_word, suggest_noun_or_use_dict if not OPENAI_API_KEY: print("ERROR: OPENAI_API_KEY not set. Check silentpilot/.env", flush=True) return from actions.vision import AgentPlanner engine = ActionEngine(overlay=overlay) planner = AgentPlanner(api_key=OPENAI_API_KEY) loop = asyncio.get_event_loop() emg_sequence: list[str] = [] accumulated_prompt: list[str] = [] word_candidates: list[str] = [] word_candidate_index: int = 0 def _get_previously_added_words() -> list[str]: return accumulated_prompt try: print(" Launching browser...", flush=True) try: await engine.start(url) print(" Browser ready!", flush=True) except Exception as e: print(f" Browser launch failed: {e}", flush=True) overlay.update_agent_status("No browser β€” enter a goal below") overlay.set_status(True) overlay.update_sequence([]) while True: cmd = await loop.run_in_executor(None, command_queue.get) if not cmd: continue if cmd == "quit": print("\n Goodbye!", flush=True) return if cmd.startswith("emg:"): cls = cmd[4:] if cls == "REST": overlay.set_status(False) overlay.update_agent_status("Looking up…") page_context = "" try: page_context = f"Title: {await engine.browser.get_page_title()}\nText: {(await engine.browser.get_page_text())[:1200]}" except Exception: pass prev_words = _get_previously_added_words() # Ask OpenAI: is this a noun? Noun β†’ Yes/No only. Not noun β†’ dictionary + Yes/No/Retry noun_word, is_noun = await suggest_noun_or_use_dict( emg_sequence, page_context, prev_words ) if is_noun and noun_word: # Proper noun β€” Yes/No only, no Retry word_candidates = [noun_word] word_candidate_index = 0 overlay.show_word_confirmation(noun_word, False) overlay.set_status(True) overlay.update_agent_status("Yes / No") else: # Fall back to dictionary search candidates = find_words_for_sequence(emg_sequence) if not candidates: overlay.show_word_confirmation("(no matches)", False) overlay.update_agent_status("No matches") word_candidates = [] emg_sequence = [] overlay.update_sequence([]) overlay.set_status(True) else: if len(candidates) == 1: suggested = candidates[0] else: suggested = await pick_best_word( candidates, page_context, prev_words ) word_candidates = candidates word_candidate_index = ( candidates.index(suggested) if suggested in candidates else 0 ) overlay.show_word_confirmation( suggested, len(candidates) > 1 ) overlay.set_status(True) overlay.update_agent_status("Yes / No / Retry") else: emg_sequence.append(cls) overlay.update_sequence(emg_sequence) elif cmd == "word_yes": if word_candidates and 0 <= word_candidate_index < len(word_candidates): word = word_candidates[word_candidate_index] accumulated_prompt.append(word) overlay.update_prompt_display(accumulated_prompt) emg_sequence = [] word_candidates = [] overlay.clear_content_area() overlay.update_sequence([]) overlay.update_agent_status("Ready") elif [truncated β€” 10412 more characters] ``` ### silentpilot/app_ui/pages/index.tsx ```typescript /** * Home / navigation page for MindOS UI. */ import Link from "next/link"; import { useEMGWebSocket } from "../lib/ws"; export default function Home() { const { connected } = useEMGWebSocket(); return (

MindOS

Control your computer with silent speech

EMG Core: {connected ? "Connected" : "Disconnected"}

1. Calibrate

Record command samples and build your personal EMG profile.

2. Train

Train your personal classifier and review accuracy metrics.

3. Demo

Live silent speech to computer control with AI agent.

); } ``` ### silentpilot/agent/src/index.ts ```typescript /** * MindOS Agent entry point. * * Connects to the EMG Core FastAPI WebSocket to receive predictions, * routes commands, and drives the orchestrator. * * Also exposes a simple HTTP API for the frontend to set goals and * get agent state. */ import "dotenv/config"; import WebSocket from "ws"; import express from "express"; import { createInitialState, AgentState } from "./state.js"; import { routeCommand, CommandEvent } from "./command_router.js"; import { Orchestrator } from "./orchestrator.js"; const EMG_WS_URL = process.env.EMG_WS_URL || "ws://localhost:8000/ws/live"; const AGENT_PORT = parseInt(process.env.AGENT_PORT || "9000", 10); // --- State --- let agentState = createInitialState(); let orchestrator = new Orchestrator(agentState); let ws: WebSocket | null = null; // --- WebSocket connection to EMG Core --- function connectToEMG(): void { console.log(`[Agent] Connecting to EMG Core at ${EMG_WS_URL}...`); ws = new WebSocket(EMG_WS_URL); ws.on("open", () => { console.log("[Agent] Connected to EMG Core WebSocket"); }); ws.on("message", async (data) => { try { const msg = JSON.parse(data.toString()); if (msg.type === "prediction" && agentState.active) { const prediction = msg.data; console.log( `[Agent] EMG Command: ${prediction.cmd} (p=${prediction.p.toFixed(2)})` ); const event: CommandEvent = { cmd: prediction.cmd, confidence: prediction.p, mode: "AGENT", // can be overridden by router }; const routed = routeCommand(event); console.log(`[Agent] Routed: ${routed.command} -> ${routed.mode}`); const result = await orchestrator.execute(routed); console.log(`[Agent] Result: ${result.action}`); if (result.toolCalls.length > 0) { for (const tc of result.toolCalls) { console.log(` Tool: ${tc.tool}(${JSON.stringify(tc.args)})`); } } } } catch (e) { // Ignore non-prediction messages } }); ws.on("close", () => { console.log("[Agent] EMG WebSocket disconnected. Reconnecting in 3s..."); setTimeout(connectToEMG, 3000); }); ws.on("error", (err) => { console.error("[Agent] WebSocket error:", err.message); }); } // --- HTTP API for frontend --- const app = express(); app.use(express.json()); // CORS app.use((_req, res, next) => { res.header("Access-Control-Allow-Origin", "*"); res.header("Access-Control-Allow-Headers", "Content-Type"); res.header("Access-Control-Allow-Methods", "GET, POST, OPTIONS"); next(); }); app.get("/state", (_req, res) => { res.json(agentState); }); app.post("/goal", (req, res) => { const { goal } = req.body; if (!goal) { res.status(400).json({ error: "goal is required" }); return; } orchestrator.setGoal(goal); console.log(`[Agent] New goal: ${goal}`); res.json({ status: "ok", goal }); }); app.post("/command", async (req, res) => { /** * Manual command injection (for testing without EMG). * POST /command { "cmd": "OPEN", "confidence": 1.0 } */ const { cmd, confidence = 1.0 } = req.body; if (!cmd) { res.status(400).json({ error: "cmd is required" }); return; } const event: CommandEvent = { cmd, confidence, mode: "AGENT" }; const routed = routeCommand(event); const result = await orchestrator.execute(routed); res.json({ action: result.action, result: result.result, toolCalls: result.toolCalls, state: agentState, }); }); app.post("/reset", (_req, res) => { agentState = createInitialState(); orchestrator = new Orchestrator(agentState); res.json({ status: "reset" }); }); // --- Start --- app.listen(AGENT_PORT, () => { console.log(`[Agent] HTTP API on http://localhost:${AGENT_PORT}`); console.log(`[Agent] POST /goal to set a task goal`); console.log(`[Agent] POST /command to inject a command manually`); console.log(`[Agent] GET /state to see agent state`); }); connectToEMG(); ``` [65 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]