# Project export: VisionOS
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2026
- Tagline: Imagine a Blind Person Could Code
- Devpost: https://devpost.com/software/visionos-5euxo7
- GitHub: https://github.com/ledaniel0/treehacks2026
- Video: https://www.youtube.com/embed/aL9MrpwJXOA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner ([OpenAI] Artificial Intelligence Track ([1st] Lunch with OpenAI engineers at the office + 1 year of ChatGPT Pro [2nd] 1 year of ChatGPT Pro [3rd] OpenAI swag))
- Team: 3 GitHub contributor(s) — airjlee (13 commits), Aditya Shirodkar (10 commits), ledaniel0 (7 commits)
## Devpost submission (written by the team)
### Inspiration
My childhood best friend is legally blind. Growing up and navigating the increasingly technology-dependent world was tedious and profoundly boring. Screen readers haven't changed in 15 years. They read pages linearly, break on inaccessible websites, and require dozens of memorized keyboard shortcuts just to do what a sighted person does with a glance. A sighted person doesn't read every element on Amazon, but a screen reader forces it. Why should it work like that? A visually impaired user should get the same experience. For what has become one of the most overlooked issues in modern day society, posing a $7 billion dollar loss to companies every year, we hope to bridge the gap between visual impairment and the growth of technology.
### What it does
Our system is fully conversational agentic operating system for visually-impaired users. Instead of simply listing every element on the screen, speaks to you to learn and take the best possible action on your behalf. Some examples for usage would be allowing those who are visually-impaired a seamless shopping experience, including the purchase of medically-aware items in which our OS will be able to have context on. Our OS also successfully allows one to code a program with just their voice. We architecture a multi-agent, multi-turn system with complete context and several voice-controlled agents capable of controlling your browser, and even your computer. Features: Natural voice conversation with real-time speech recognition and voice response. AI agents see and interact with web pages the way a sighted user would, using vision-based understanding and DOM parsing. This allows our agents to complete complex multi-step tasks with user feedback and context. Beyond the browser, the system controls applications, like skipping songs on Spotify or resuming work in VSCode. Contextual memory that remembers your preferences, budget, and past conversations across sessions.
### How we built it
Conversational Agent: We employed OpenAI GPT-4.1 mini (STT) and Cartesia (TTS) for low latency, accurate conversation. GPT 4.1-mini acts as the “brain,” forming a connection between our agents, contextual layer, and the user-facing conversation. This is all built on an Electron overlay for the UI. Multi Agent Workflow: We employ a variety of agents across tasks, connected via LangGraph. For these agents to delegate tasks smoothly, an orchestration agent routes requests via user intent. Long-term context: ElasticSearch with JINA embeddings powers our semantic memory layer, indexing conversation history and browsing sessions so users can recall past interactions naturally. Web-search: We integrate the Perplexity Sonar API to give our agents grounded, real-time web knowledge. This lets the system make comparisons and recommendations to inform decisions before opening a webpage. Automation Tools: We use Stagehand by Browserbase and Agent-S by Simular AI to create automations for our OS. While Stagehand is useful for all browser related tasks, we use Agent-S for the rest of our desktop experience.
### Challenges we ran into
Implementing a reliable 'Supervisor-Worker' architecture was challenging, as our supervisor initially struggled to accurately route requests among our specialized agents. We decoupled the routing logic into a dedicated classification step using GPT 4.1-mini, enforcing tightly scoped definitions for each worker. By wiring this pipeline with LangGraph, we ensured that the selected agent receives the full conversation context, preventing information loss during handoffs.
### Accomplishments we're proud of
Multi turn agents: We achieved fluid multi-turn refinement by optimizing agent context. Our system uses a persistent context window to maintain the state of the conversation, allowing users to iterate on ideas ('no, I’m allergic to peanuts') and reason about user intent rather than restarting. This turns the friction of prompt engineering into a natural, refining dialogue where the agent gets smarter with every reply. Natural language flow: We built our system to feel like a real conversation. Users speak naturally, get natural speech back, and can interrupt mid-sentence as if they were talking to another person.
### What we learned
Humans rarely speak in perfect commands or say exactly what they mean right away. They can change their mind or forget crucial details. Therefore, we learned to build a system that acknowledges and considers these inconsistencies. We allow a user to interrupt a task, give refinements, or completely switch when they need to.
### What's next
We were working on a "text-to-braille" live physical translation to work alongside our product. Build a full-fledged accessible OS Assumptions are dangerous in ability based design. Our next priority is to partner with visually impaired users for feedback, ensuring our engineering decisions align with lived reality. VisionOS aims to bridge the digital employment gap, building tools/agents that empower blind users to perform more efficiently in administrative and technical jobs.
## README (from the GitHub repository)
# treehacks2026
commands:
visualizer: npm run dev:electron
## Detected evidence (automated analysis)
Indexed codebase: 83 recognized source files, 517 KB.
- Anthropic (technology) — detected in the code
- CSS (language) — detected in the code
- Express (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- LangChain (technology) — detected in the code
- Next.js (technology) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- AI coding agent: Claude Code — evidence: commit authorship or trailers
- AI coding agent: Cursor — evidence: commit authorship or trailers
## Codebase structure (from repository index)
### Files (106 of 106)
```
.gitignore
agent-S-dev-plan.md
agentverse/adapter.py
agentverse/README.md
agentverse/register.py
agentverse/requirements.txt
architecture.txt
client/.gitignore
client/eslint.config.mjs
client/next.config.ts
client/package.json
client/postcss.config.mjs
client/README.md
client/src/app/globals.css
client/src/app/layout.tsx
client/src/app/page.tsx
client/src/components/MicButton.tsx
client/src/components/StatusIndicator.tsx
client/src/components/Transcript.tsx
client/src/components/VoiceAgent.tsx
client/src/hooks/useAudioPlayer.ts
client/src/hooks/useDeepgramSTT.ts
client/src/hooks/useSocket.ts
client/src/lib/types.ts
client/tsconfig.json
docs/01-client-layer.md
docs/02-server-orchestrator.md
docs/03-langgraph-supervisor.md
docs/04-commerce-agent.md
docs/06-general-agent.md
docs/07-execution-layer.md
docs/08-audio-services.md
docs/09-knowledge-base.md
docs/stagehand-integration-notes.md
ELECTRON_README.md
electron/assets/tray-icon.png.placeholder
electron/index.html
electron/package.json
electron/postcss.config.js
electron/QUICKSTART.md
electron/README.md
electron/src/App.tsx
electron/src/components/MicButton.tsx
electron/src/components/StatusIndicator.tsx
electron/src/components/Transcript.tsx
electron/src/components/VoiceAgent.tsx
electron/src/hooks/useAudioPlayer.ts
electron/src/hooks/useDeepgramSTT.ts
electron/src/hooks/useSocket.ts
electron/src/index.css
electron/src/lib/types.ts
electron/src/main.ts
electron/src/main.tsx
electron/src/preload.ts
electron/src/vite-env.d.ts
electron/start.sh
electron/tailwind.config.js
electron/tsconfig.json
electron/tsconfig.main.json
electron/vite.config.ts
package.json
PHASES.md
README.md
server/.gitignore
server/data/seed-user-profile.json
server/package.json
server/scripts/setup-elastic.ts
server/src/agents/coding.ts
server/src/agents/commerce.ts
server/src/agents/desktop.ts
server/src/agents/documentation.ts
server/src/agents/general.ts
server/src/agents/supervisor.ts
server/src/agents/tools.ts
server/src/index.ts
server/src/lib/computerActions.ts
server/src/lib/computerControl.ts
server/src/lib/elasticsearch.ts
server/src/lib/stagehand.ts
server/src/llm/anthropic.ts
server/src/llm/gemini.ts
server/src/llm/index.ts
server/src/llm/openai.ts
server/src/llm/types.ts
server/src/pipeline/voicePipeline.ts
server/src/socket/handler.ts
server/src/stt/deepgram.ts
server/src/test/responses.ts
server/src/tts/cartesia.ts
server/src/types/index.ts
server/tsconfig.json
uagents_core.log
visionimplementationplan.md
visualizer/audio-visualizer@1.0.0
visualizer/electron-main.ts
visualizer/index.html
visualizer/package.json
visualizer/src/App.tsx
visualizer/src/components/AudioVisualizer.tsx
visualizer/src/index.css
visualizer/src/main.tsx
visualizer/src/vite-env.d.ts
visualizer/tsconfig.electron.json
visualizer/tsconfig.json
visualizer/vite
visualizer/vite.config.ts
```
### Dependencies
- agentverse/requirements.txt: fetchai, python-dotenv, requests, uagents, uagents-core
- client/package.json: @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@16.1.6, lucide-react@^0.564.0, next@16.1.6, react@19.2.3, react-dom@19.2.3, socket.io-client@^4.8.3, typescript@^5
- electron/package.json: @types/node@^20, @types/react@^19, @types/react-dom@^19, @vitejs/plugin-react@^4.3.4, autoprefixer@^10.4.20, concurrently@^9.1.2, cross-env@^7.0.3, dotenv@^16.6.1, electron@^33.2.1, electron-builder@^24.13.3, liquid-glass-react@^1.0.0, lucide-react@^0.564.0, postcss@^8.5.1, react@^19.0.0, react-dom@^19.0.0, socket.io-client@^4.8.3, tailwindcss@^3.4.17, tsx@^4.21.0, typescript@^5.9.3, vite@^6.0.7, wait-on@^8.0.1
- package.json: tailwindcss@^4.1.18
- server/package.json: @anthropic-ai/sdk@^0.74.0, @browserbasehq/stagehand@^3.0.8, @deepgram/sdk@^3.9.0, @elastic/elasticsearch@^9.3.1, @google/genai@^1.41.0, @langchain/anthropic@^0.3.34, @langchain/core@^0.3.80, @langchain/langgraph@^0.2.74, @langchain/openai@^0.5.18, @types/cors@^2.8.19, @types/express@^5.0.6, @types/node@^25.2.3, @types/uuid@^10.0.0, @types/ws@^8.18.1, cors@^2.8.6, dotenv@^16.6.1, express@^5.2.1, openai@^6.22.0, socket.io@^4.8.3, socket.io-client@^4.8.3, tsx@^4.21.0, typescript@^5.9.3, uuid@^13.0.0, ws@^8.19.0, zod@^4.3.6
- visualizer/package.json: @types/react@^19, @types/react-dom@^19, @types/three@^0.158.0, @vitejs/plugin-react@^4.3.4, concurrently@^9.1.2, cross-env@^7.0.3, electron@^33.2.1, react@^19.0.0, react-dom@^19.0.0, socket.io-client@^4.8.3, three@^0.158.0, typescript@^5.9.3, vite@^6.0.7, wait-on@^8.0.1
### Recent commits (newest first)
- Add documentation agent
- Merge with test and ui changes
- testing and breaking
- test responses and delays
- INcreased logs for transparency and testing
- Add mute functionality
- Stagehand back to chrome fix
- Arc browser instead for asthetic
- Add coding agent
- Updates to testing and main electron overlay logging
- Update Agentverse implementation for ASI:One
- audiocontext for visualizer and voice to both affect it
- visualizer upgrade
- main electron overlay ui changes
- Pushed agent to agent verse
- electron websocket start closed
- [ff] test ff impl
- small election ui tweaks
- Add memory to agent classification
- bug fix
## Key source files (fetched from GitHub, selected and truncated for size)
### ELECTRON_README.md
```markdown
# 🎙️ RCY - Voice Agent Platform
A real-time voice conversational agent with multiple deployment options: Web (Next.js) and **Desktop Overlay (Electron with Liquid Glass)**.
## 🏗️ Project Structure
```
rcy/
├── client/ # Next.js web client (original)
├── server/ # Express + Socket.IO server with LLM agents
└── electron/ # 🆕 Electron desktop app with liquid glass overlay
```
## ✨ What's New: Electron Edition
The new **Electron app** (`electron/`) provides a beautiful desktop overlay experience:
- **🪟 Liquid Glass Effect**: Apple-inspired glassmorphism UI using [liquid-glass-react](https://github.com/rdev/liquid-glass-react)
- **🔝 Always-on-Top Overlay**: Floats above other applications
- **⌨️ Global Hotkey**: Toggle with `Cmd/Ctrl + Shift + Space`
- **📍 System Tray**: Minimize to background
- **🎤 Full Voice Support**: Web Audio API + Speech Recognition
- **🔄 Same Backend**: Uses the existing server via Socket.IO
## 🚀 Quick Start
### 1. Install Dependencies
```bash
# Server
cd server
npm install
# Electron App
cd ../electron
npm install
```
### 2. Configure Environment
**Server** (`server/.env`):
```bash
ANTHROPIC_API_KEY=your_key_here
CARTESIA_API_KEY=your_key_here
DEEPGRAM_API_KEY=your_key_here
```
**Electron** (`electron/.env`):
```bash
VITE_SERVER_URL=http://localhost:3001
```
### 3. Launch
**Easy Way** (starts everything):
```bash
cd electron
./start.sh
```
**Manual Way** (separate terminals):
```bash
# Terminal 1 - Server
cd server
npm run dev
# Terminal 2 - Electron
cd electron
npm run dev
```
## 🏛️ Architecture
### Option A: Separate Server (Current Implementation)
```
┌─────────────────────────────────────────┐
│ Electron Window (Transparent Overlay) │
│ ┌────────────────────────────────────┐ │
│ │ LiquidGlass Component │ │
│ │ ┌──────────────────────────────┐ │ │
│ │ │ React UI + Voice Agent │ │ │
│ │ │ - Web Audio API │ │ │
│ │ │ - Speech Recognition │ │ │
│ │ │ - Socket.IO Client │ │ │
│ │ └──────────────────────────────┘ │ │
│ └────────────────────────────────────┘ │
└─────────────────────────────────────────┘
│
│ Socket.IO (ws://localhost:3001)
↓
┌─────────────────────────────────────────┐
│ Express Server (Separate Process) │
│ - Socket.IO Server │
│ - LLM Agents (Anthropic) │
│ - TTS (Cartesia) │
│ - STT (Deepgram) │
│ - Browser Automation (Stagehand) │
└─────────────────────────────────────────┘
```
**Pros:**
- ✅ Simple to develop and debug
- ✅ Server can be run separately
- ✅ Easy to update either part independently
- ✅ Stagehand's Chromium is separate from Electron
**Cons:**
- ❌ Two processes to manage
- ❌ Requires server to be running
## 📦 Deployment Options
### Web Client (Original)
```bash
cd client
npm run dev # Development
npm run build # Production build
[truncated — 2557 more characters]
```
### agent-S-dev-plan.md
```markdown
# Plan: Agent-S Desktop Control Integration
## Context
The project is a voice-first accessible browser for blind users (TreeHacks 2026). It has a LangGraph supervisor that routes to commerce/coding/general agents. The coding agent is currently a text-only stub. We want to add full desktop control via Agent-S so the assistant can control any app (VS Code, Terminal, Finder, Spotify, etc.) outside the browser.
**Key decisions made:**
- **Architecture**: Hybrid — Agent-S framework with Claude (Anthropic) as grounding model (no UI-TARS/GPU needed)
- **Routing**: New "desktop" category in the supervisor classifier (4-way routing)
- **Agent design**: Single desktop agent with mode-switching (coding vs general desktop prompts, same tools)
- **Sidecar**: Python FastAPI on port 8001, auto-started by Node.js server
---
## Step 1: Verify Supervisor Routing to "desktop" Category
**Files to modify:**
- `server/src/types/index.ts` — Add `"desktop"` to `AgentCategory` union type
- `server/src/agents/supervisor.ts` — Update classifier prompt, router, and graph edges
**Changes:**
1. **`server/src/types/index.ts:29`** — Extend AgentCategory:
```typescript
export type AgentCategory = "commerce" | "coding" | "general" | "desktop";
```
2. **`server/src/agents/supervisor.ts`** — Update `CLASSIFY_SYSTEM` prompt (line 74-86):
- Add `"desktop"` category: tasks requiring OS-level control outside the browser — opening apps, typing in desktop apps, file management, system settings, coding in VS Code, running terminal commands
- Keep `"coding"` for text-only code help (explain code, debug conceptually, discuss architecture)
- Clarify boundary: "If the user wants something DONE on the computer (not just explained), classify as desktop"
3. **`server/src/agents/supervisor.ts`** — Update `routeByCategory` (line 165-176):
```typescript
case "desktop": return "desktopAgent";
```
4. **`server/src/agents/supervisor.ts`** — Update `createSupervisor` (line 185+):
- Import and create `desktopAgent`
- Add `desktopAgent` node to the graph
- Add edge: `desktopAgent → formatResponse`
- Add `"desktopAgent"` to conditional edges
**Verification**: Send test messages and confirm classification:
- "open VS Code" → desktop
- "explain what a for loop is" → coding
- "buy headphones on Amazon" → commerce
- "read this article" → general
- "run my Python script in Terminal" → desktop
- "open Spotify and play music" → desktop
---
## Step 2: Set Up Agent-S Python Sidecar
**New files:**
- `agent-s-sidecar/server.py` — FastAPI server (follows existing plan in visionimplementationplan.md lines 437-554)
- `agent-s-sidecar/requirements.txt` — Python dependencies
- `agent-s-sidecar/start.sh` — Startup script
**Implementation:**
1. **`agent-s-sidecar/requirements.txt`**:
```
gui-agents
fastapi
uvicorn
pyautogui
pytesseract
```
2. **`agent-s-sidecar/server.py`** — FastAPI app with 4 endpoints:
- `POST /predict` — Single-step: captu
[truncated — 7301 more characters]
```
### package.json
```
{
"devDependencies": {
"tailwindcss": "^4.1.18"
}
}
```
### agentverse/requirements.txt
```
fetchai
python-dotenv
uagents
uagents-core
requests
```
### client/package.json
```
{
"name": "client",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"lint": "eslint"
},
"dependencies": {
"lucide-react": "^0.564.0",
"next": "16.1.6",
"react": "19.2.3",
"react-dom": "19.2.3",
"socket.io-client": "^4.8.3"
},
"devDependencies": {
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"eslint": "^9",
"eslint-config-next": "16.1.6",
"typescript": "^5"
}
}
```
### server/package.json
```
{
"name": "server",
"version": "1.0.0",
"type": "module",
"main": "src/index.ts",
"scripts": {
"dev": "tsx watch src/index.ts",
"dev:test": "TEST=true tsx watch src/index.ts",
"start": "tsx src/index.ts"
},
"keywords": [],
"author": "",
"license": "ISC",
"description": "",
"dependencies": {
"@anthropic-ai/sdk": "^0.74.0",
"@browserbasehq/stagehand": "^3.0.8",
"@deepgram/sdk": "^3.9.0",
"@elastic/elasticsearch": "^9.3.1",
"@google/genai": "^1.41.0",
"@langchain/anthropic": "^0.3.34",
"@langchain/core": "^0.3.80",
"@langchain/langgraph": "^0.2.74",
"@langchain/openai": "^0.5.18",
"cors": "^2.8.6",
"dotenv": "^16.6.1",
"express": "^5.2.1",
"openai": "^6.22.0",
"socket.io": "^4.8.3",
"uuid": "^13.0.0",
"ws": "^8.19.0",
"zod": "^4.3.6"
},
"overrides": {
"undici": "^6.21.1"
},
"devDependencies": {
"@types/cors": "^2.8.19",
"@types/express": "^5.0.6",
"@types/node": "^25.2.3",
"@types/uuid": "^10.0.0",
"@types/ws": "^8.18.1",
"socket.io-client": "^4.8.3",
"tsx": "^4.21.0",
"typescript": "^5.9.3"
}
}
```
### visualizer/package.json
```
{
"name": "audio-visualizer",
"private": true,
"version": "1.0.0",
"type": "module",
"main": "dist/electron-main.cjs",
"description": "Particle synthesis audio visualizer — transparent Electron overlay",
"scripts": {
"dev": "vite",
"dev:electron": "npm run build:main && concurrently \"vite\" \"wait-on http://localhost:5174 && cross-env NODE_ENV=development electron .\" --names \"vite,electron\" --prefix-colors \"magenta,cyan\"",
"build:main": "tsc -p tsconfig.electron.json && mv dist/electron-main.js dist/electron-main.cjs",
"build:renderer": "vite build --outDir dist/renderer",
"build": "npm run build:main && npm run build:renderer",
"start": "electron .",
"preview": "vite preview"
},
"dependencies": {
"react": "^19.0.0",
"react-dom": "^19.0.0",
"socket.io-client": "^4.8.3",
"three": "^0.158.0"
},
"devDependencies": {
"@types/react": "^19",
"@types/react-dom": "^19",
"@types/three": "^0.158.0",
"@vitejs/plugin-react": "^4.3.4",
"concurrently": "^9.1.2",
"cross-env": "^7.0.3",
"electron": "^33.2.1",
"typescript": "^5.9.3",
"vite": "^6.0.7",
"wait-on": "^8.0.1"
}
}
```
### electron/package.json
```
{
"name": "rcy-electron",
"version": "1.0.0",
"description": "Voice Agent Electron Overlay with Liquid Glass Effect",
"main": "dist/main.js",
"scripts": {
"dev": "concurrently \"npm run dev:renderer\" \"npm run dev:electron\" --names \"vite,electron\" --prefix-colors \"magenta,cyan\"",
"dev:renderer": "vite",
"dev:electron": "npm run build:main && wait-on http://localhost:5173 && cross-env NODE_ENV=development electron .",
"build": "npm run build:main && npm run build:renderer",
"build:main": "tsc -p tsconfig.main.json",
"build:renderer": "vite build",
"start": "electron .",
"package": "electron-builder",
"launch": "./start.sh"
},
"dependencies": {
"dotenv": "^16.6.1",
"liquid-glass-react": "^1.0.0",
"lucide-react": "^0.564.0",
"react": "^19.0.0",
"react-dom": "^19.0.0",
"socket.io-client": "^4.8.3"
},
"devDependencies": {
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"@vitejs/plugin-react": "^4.3.4",
"autoprefixer": "^10.4.20",
"concurrently": "^9.1.2",
"cross-env": "^7.0.3",
"electron": "^33.2.1",
"electron-builder": "^24.13.3",
"postcss": "^8.5.1",
"tailwindcss": "^3.4.17",
"tsx": "^4.21.0",
"typescript": "^5.9.3",
"vite": "^6.0.7",
"wait-on": "^8.0.1"
},
"build": {
"appId": "com.rcy.voiceagent",
"productName": "RCY Voice Agent",
"files": [
"dist/**/*",
"package.json"
],
"mac": {
"category": "public.app-category.productivity"
}
}
}
```
### visualizer/src/App.tsx
```typescript
import { AudioVisualizer } from "./components/AudioVisualizer";
export default function App() {
return ;
}
```
### electron/src/main.tsx
```typescript
import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';
import './index.css';
ReactDOM.createRoot(document.getElementById('root')!).render(
);
```
[74 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]