# Project export: ReSight This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Normalizing browsing experiences for the visually impaired. - Devpost: https://devpost.com/software/resight-ah45vl - GitHub: https://github.com/vietnguyen2358/ReSight - Demo: https://resight-xi.vercel.app/ - Video: https://www.youtube.com/embed/hCwBxZFTSWw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Viet Nguyen (10 commits), Steven Le (8 commits), NicholasLe04 (1 commits) ## Devpost submission (written by the team) ### Inspiration A few years back, one of us tutored a blind student at San Jose State who wanted to learn web development. Watching him browse made something obvious: for visually impaired users, modern web browsing is often just a shell of the full experience non-visually impaired users have. Instead of viewing vivid UIs, they are forced to rely on linear text-to-speech options that miss context and layout. With today’s AI and modern web tooling, we believed we could help bridge that gap. ReSight is our attempt to empower visually impaired users to take control of their own browser experience. ### What it does ReSight is a voice-first browsing copilot designed specifically for visually impaired users. It goes beyond simple screen reading to provide true agency. Navigation & Tasks: Helps users navigate websites, compare information, and complete multi-step tasks through natural conversation. Vision-Based Narration: Uses vision models to explain the actual Page UI, not just the DOM text. Memory & Safety: Includes persistent memory and safety support (Guardian) so users can browse with confidence and less friction. Built for zero-vision use ReSight is designed to be used entirely without sight. Voice & Keyboard Control: Users can start and control browsing with voice + keyboard shortcuts (e.g., press Space to start an instruction). Audio Feedback: The system provides spoken progress updates, vision-based page narration, and spoken next-step prompts so users never need to look at the screen. Conversational Interaction: Core interactions (searching, navigating, comparing, form guidance) are designed to be completed through audio feedback and conversational control alone. How is this different than just talking to an LLM? Our goal is not to have AI "do everything for" the user. Our goal is empowerment. ReSight is designed so the user feels like they are actively navigating, deciding, and interacting with the web—with AI as an assistive layer, not a replacement for their agency. ### How we built it We built ReSight as a multi-agent web application utilizing browser automation, screenshot-based vision narration, and conversational voice interaction. The Agent Architecture (Multiturn + “handoffs”): Orchestrator: Handles task coordination and delegates instructions. In multiturn mode it also threads the conversation state forward (what you asked, what we just found, what we’re trying next) and can re-issue improved instructions when the user clarifies mid-flight (“no, the other one”, “go back”, “open the comments”). Orchestrator: Handles task coordination and delegates instructions. In multiturn mode it also threads the conversation state forward (what you asked, what we just found, what we’re trying next) and can re-issue improved instructions when the user clarifies mid-flight (“no, the other one”, “go back”, “open the comments”). Navigator: Executes web actions and extracts data. It runs as a step-wise agent loop (act → observe/extract → narrate) and can pause for a user answer (credentials, “pick option A/B”) without losing its place, then continue from the same page state. Navigator: Executes web actions and extracts data. It runs as a step-wise agent loop (act → observe/extract → narrate) and can pause for a user answer (credentials, “pick option A/B”) without losing its place, then continue from the same page state. Scribe: Manages user memory and preferences. In multiturn interactions, it persists durable context (“prefers concise summaries”, “uses Reddit logged-in context”, “always wants top 3 options”) so later turns don’t re-ask or re-derive basics. Scribe: Manages user memory and preferences. In multiturn interactions, it persists durable context (“prefers concise summaries”, “uses Reddit logged-in context”, “always wants top 3 options”) so later turns don’t re-ask or re-derive basics. Guardian: Performs safety checks for suspicious flows or content. It acts like a specialist agent that can be invoked mid-task when the Navigator hits dark patterns, suspicious redirects, or risky actions—then hands control back with an allow/block + rationale. Guardian: Performs safety checks for suspicious flows or content. It acts like a specialist agent that can be invoked mid-task when the Navigator hits dark patterns, suspicious redirects, or risky actions—then hands control back with an allow/block + rationale. How the agents go back-and-forth (handoffs): Agent tool-calls as contracts: The Orchestrator calls navigate(...) (or remember(...), safety_check(...)) and gets structured results back. That makes the “handoff” explicit and repeatable across turns. Agent tool-calls as contracts: The Orchestrator calls navigate(...) (or remember(...), safety_check(...)) and gets structured results back. That makes the “handoff” explicit and repeatable across turns. Multiturn continuation: A user can interrupt or refine at any time (“stop”, “go back”, “open that one”). The Orchestrator cancels in-flight work, keeps the last useful state, and re-delegates—so it feels like a real back-and-forth instead of a one-shot script. Multiturn continuation: A user can interrupt or refine at any time (“stop”, “go back”, “open that one”). The Orchestrator cancels in-flight work, keeps the last useful state, and re-delegates—so it feels like a real back-and-forth instead of a one-shot script. Clarification loop: The Navigator can ask a single targeted question (via an ask_user bridge), block until the user answers, then resume the same task—no restarting the whole flow. Clarification loop: The Navigator can ask a single targeted question (via an ask_user bridge), block until the user answers, then resume the same task—no restarting the whole flow. Powered by Stagehand + Browserbase (why this works): Stagehand act(): Powers intent-driven interactions (click/type/scroll) without relying on brittle selectors. Stagehand extract(): Provides structured outputs (sponsors, tracks, prices, hours, form fields) from real pages. Stagehand goto(): Enables deterministic fallback routing when act() fails (navigating by URL instead of repeated clicks). Stagehand Screenshot Loop: Combined with vision narration, this gives blind users UI context beyond raw DOM text. Browserbase: Provides persistent context (keeping logins/cookies active across runs) and solves CAPTCHAs to reduce failures on bot-protected flows. ### Challenges we ran into Anti-Bot Protections: Dealing with aggressive CAPTCHAs and blocking on popular sites. Dynamic Actions: Ensuring action reliability on complex, dynamic websites (SPAs). Humanizing the AI: Making the narration feel helpful and human, rather than robotic or overly verbose. The Trade-off: Balancing autonomy, safety, and execution speed. ### Accomplishments we're proud of Built a working voice-first browser experience centered entirely on accessibility. Implemented vision-first narration so users hear what is actually on the screen, not just code. Created a practical multi-agent system that handles navigation, memory, and safety simultaneously. Kept the interaction conversational while still capable of handling complex, multi-step tasks. ### What we learned Accessibility Reading Text: Accessibility is not just "read text aloud." Agency is Key: Real accessibility means giving users agency, context, and confidence while navigating the web. System Design: We learned that multi-agent systems work best when each agent has a clear, strictly defined role to keep the user experience simple. ### What's next Better Stealth: Better CAPTCHA bypassing will enhance browsing capabilities for users (ex. access to yelp.com) Suite for the Visually Impaired: We'd like to expand to other tools that will enhance the experiences of the visually impaired in the context of using technology ## README (from the GitHub repository) # ReSight — Voice Browser for the Visually Impaired > Built for TreeHacks 2026 ReSight is an AI-powered voice browser that enables visually impaired users to navigate the entire internet using only their voice. It features a "Split Brain" architecture: a voice-first UI paired with headless browser automation orchestrated by a Council of AI Agents. ## Architecture ``` User Input → Orchestrator (LLM router) ├── Navigator (browser pilot) → Stagehand → Chromium ├── Scribe (memory store) → user_context.json └── Guardian (safety check) → approve/block Parallel streams: Thoughts: Agent → SSE → ChatPanel (unified feed) Screenshots: Browser → memory cache → LiveFeed UI ``` **The Mind (Left Pane)** — Unified chat interface with real-time agent thought stream, inter-agent communication logs, and text/voice input. The ReSight sphere serves as a compact status indicator. **The World (Right Pane)** — Live screenshot feed of the headless browser session with element overlay detection. **The Council** — Four specialized AI agents working in two LLM layers: - **Orchestrator** — Routes user intent to the right agent (LLM with 3 tools) - **Navigator** — Plans & executes multi-step browser automation via Stagehand (LLM with 4 tools, up to 12 steps) - **Scribe** — Manages user preferences and memory (no LLM, file I/O) - **Guardian** — Safety analysis, dark-pattern detection, purchase confirmation (LLM, single call) **Stagehand** — Browser automation layer using its own internal Gemini-2.0-flash LLM to translate natural language actions ("click the search box") into precise DOM interactions via Playwright. ## Setup 1. Install dependencies: ```bash npm install --legacy-peer-deps npx playwright install chromium ``` 2. Configure environment variables in `.env.local`: ``` GOOGLE_GENERATIVE_AI_API_KEY=your_key_here NEXT_PUBLIC_ELEVENLABS_AGENT_ID=your_agent_id ``` 3. Start the dev server: ```bash npm run dev ``` 4. Open [http://localhost:3000](http://localhost:3000) ## Tech Stack - **Next.js 15** (App Router) + React 19 - **Tailwind CSS 4** + Framer Motion - **ElevenLabs** — Voice conversation - **Stagehand** — Browser automation - **Vercel AI SDK** + Google Gemini — Agent orchestration - **Three.js** + React Three Fiber — 3D visualization ## API Endpoints - `POST /api/orchestrator` — Send a browser instruction - `GET /api/thought-stream` — SSE stream of agent thoughts - `GET /api/screenshot` — Latest browser screenshot (base64) ## Future Roadmap - **Collaborative Browsing (Multiplayer Mode)** — Multiple users sharing and co-navigating a browser session in real-time - **Persistent Memory** — Long-term user preference storage across sessions - **Multi-tab Support** — Navigate multiple pages simultaneously - **Accessibility Audit Agent** — Automatically detect and report accessibility issues on visited pages ## Detected evidence (automated analysis) Indexed codebase: 53 recognized source files, 294 KB. - CSS (language) — detected in the code - Google Gemini (technology) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel AI SDK (technology) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (62 of 62) ``` .env.example .gitignore app/api/clarification/route.ts app/api/dev-logs/route.ts app/api/orchestrator/route.ts app/api/screenshot/route.ts app/api/thought-stream/route.ts app/api/transcribe/route.ts app/api/tts/route.ts app/dev/page.tsx app/globals.css app/layout.tsx app/page.tsx CLAUDE.md components/mind/AgentGraph.tsx components/mind/ChatPanel.tsx components/mind/MindPane.tsx components/mind/ReSightSphere.tsx components/mind/SpeakButton.tsx components/mind/ThoughtStream.tsx components/providers/ReSightProvider.tsx components/SplitLayout.tsx components/ui/background-beams.tsx components/ui/cn.ts components/ui/dot-background.tsx components/ui/moving-border.tsx components/ui/sparkles.tsx components/ui/text-generate-effect.tsx components/voice/VoiceManager.tsx components/world/LiveFeed.tsx components/world/OverlayBox.tsx components/world/WorldPane.tsx docs/architecture/agent-flow.excalidraw docs/architecture/feedback-loop.excalidraw docs/architecture/system-architecture.excalidraw docs/demo-prompts.md docs/sample-conversations.md lib/agents/cancellation.ts lib/agents/clarification.ts lib/agents/guardian.ts lib/agents/navigator.ts lib/agents/orchestrator.ts lib/agents/planner.ts lib/agents/playbook.ts lib/agents/scribe.ts lib/agents/types.ts lib/agents/vision.ts lib/context/user_context.json lib/context/user-context.ts lib/dev-logger.ts lib/stagehand/browser.ts lib/stagehand/screenshot.ts lib/stagehand/session.ts lib/thought-stream/emitter.ts lib/voice/transcribe.ts lib/voice/tts.ts next.config.ts package.json postcss.config.mjs PRD.md README.md tsconfig.json ``` ### Dependencies - package.json: @ai-sdk/google@^3.0.29, @ai-sdk/openai@^3.0.29, @browserbasehq/stagehand@^3.0.8, @playwright/test@^1.58.2, @react-three/fiber@^8.17.0, @tailwindcss/postcss@^4.0.0, @types/node@^22.0.0, @types/react@^19.0.0, @types/react-dom@^19.0.0, @types/three@^0.171.0, ai@^6.0.86, clsx@^2.1.1, dotenv@^17.3.1, framer-motion@^11.15.0, next@^16.1.6, react@^19.0.0, react-dom@^19.0.0, tailwind-merge@^3.4.0, tailwindcss@^4.0.0, three@^0.171.0, typescript@^5.7.0, zod@^3.24.0 ### Recent commits (newest first) - more changes? - stuff - Update Next.js version - refactor + clean ups to simplify logics - remove bad ref - additional context + diagrams - Update prompts - Update - resite -> resight - cleanup context bloat - switch to 2.5, don't push resite cache - Update stt and tts - more update - added demo prompts + sample convos for stronger context - update agent prompts - Abort functionality and conversation history context - More conversational + Rename - Make model more conversational - update pipeline - fixes stuff ## Key source files (fetched from GitHub, selected and truncated for size) ### PRD.md ```markdown # Project: ReSight (TreeHacks 2026) # Role: Principal Software Architect # Objective: Build the "Ultimate Voice Browser" for the Visually Impaired. You are **ReSight**, a highly sophisticated AI Architect. You are building a **Next.js 15 (App Router)** application that acts as a "Voice OS for the Web." Your users are blind or visually impaired. Your mission is to enable them to navigate the entire internet using *only* their voice. You will implement a **"Split Brain"** architecture: 1. **The Mind (Frontend):** A high-contrast, voice-first UI (ElevenLabs). 2. **The Body (Backend):** A "Council of Agents" (Vercel AI SDK) that controls a headless browser (Stagehand). --- ## 1. The Tech Stack (Strict Constraints) * **Framework:** Next.js 15 (App Router) + React 19. * **Styling:** Tailwind CSS + `framer-motion` (for "breathing" UI animations). * **Voice Engine:** `@elevenlabs/react` (Client-side WebSocket connection). * **Browser Automation:** `@browserbasehq/stagehand` (Running in Next.js API Routes). * **AI Orchestration:** `ai` (Vercel AI SDK Core) + `@ai-sdk/anthropic`. * **3D Visuals:** `three` + `@react-three/fiber` (For the "ReSight Sphere"). * **Database:** (Optional) Simple JSON file or `localStorage` for the hackathon (Memory Agent). --- ## 2. The "Split Brain" UI Layout You must create a page layout that serves two masters: the **Blind User** and the **Sighted Judge**. ### A. Left Pane: "The Mind" (User Focus) * **Visual Style:** High-Contrast Neobrutalism. Pitch Black background (`#000000`), Neon Yellow text (`#CCFF00`). * **Component 1: The ReSight Sphere:** A centralized 3D wireframe sphere (`Canvas`) that reacts to audio volume. * *Idle:* Rotating slowly (Cyan). * *Listening:* Pulsing with mic input (Green). * *Thinking:* Spinning rapidly/shattering (Gold). * **Component 2: The Thought Stream:** A live-scrolling log showing the *internal monologue* of the backend agents. * *Format:* `[Agent Name] Action Description` * *Example:* `[Navigator] Locating 'Add to Cart' button...` * **Component 3: The Big Button:** A massive, screen-width button at the bottom labeled "HOLD TO SPEAK." ### B. Right Pane: "The World" (Judge Focus / "Ghost Mode") * **Visual Style:** "Cyber-Security Feed." * **Component:** A live visual feed of the Stagehand browser session. * **Implementation:** Since Stagehand runs on the server, the backend must save screenshots to a public directory (or base64) after every action. The frontend polls this image every 500ms to create a "stop-motion" video feed. * **Overlay:** When Stagehand identifies an element, overlay a CSS bounding box on the screenshot to prove the AI "sees" it. --- ## 3. The Backend: "The Council of Agents" You will NOT write a monolithic API route. You will implement a **Multi-Agent Orchestrator** in `app/api/orchestrator/route.ts`. ### Agent 1: The Orchestrator (The Boss) * **Role:** Receives raw text from ElevenLabs. Decides which sub-agent to deploy. * **Logic:** [truncated — 2850 more characters] ``` ### CLAUDE.md ```markdown # CLAUDE.md — ReSight Codebase Guide ## Project Overview ReSight is a voice-controlled web browser for the visually impaired, built for TreeHacks 2026. Next.js 15 App Router app with a "Split Brain" architecture: voice-first UI (left pane) + headless browser live feed (right pane), orchestrated by a Council of AI Agents. ## Tech Stack - **Framework:** Next.js 15 (App Router) + React 19 - **LLM (Agents):** OpenRouter (`anthropic/claude-haiku-4.5` default) or Google Gemini (`gemini-2.0-flash`) — powers orchestrator + navigator via Vercel AI SDK - **LLM (Browser):** Google Gemini (`gemini-2.0-flash`) — powers Stagehand browser automation (separate from agent LLM) - **Browser Automation:** Stagehand v3.0.8 (`@browserbasehq/stagehand`) — LOCAL mode for dev, BROWSERBASE for production - **Voice:** ElevenLabs (`@elevenlabs/react`) — WebSocket-based conversational AI - **Styling:** Tailwind CSS 4 (PostCSS plugin, no tailwind.config.js) - **3D:** Three.js + React Three Fiber (wireframe icosahedron sphere) - **Animation:** Framer Motion (thought stream entries) - **AI SDK:** Vercel AI SDK v6 (`ai` package) + `@ai-sdk/google` + `@ai-sdk/openai` ## Commands ```bash npm install --legacy-peer-deps # --legacy-peer-deps required (R3F peer dep conflict with React 19) npm run dev # Start dev server on localhost:3000 npm run build # Production build (Next.js) npx playwright install chromium # Required once for Stagehand LOCAL mode ``` ## Environment Variables (.env.local) ``` GOOGLE_GENERATIVE_AI_API_KEY= # Required for Stagehand; also used for agents if no OpenRouter OPENROUTER_API_KEY= # Optional: use OpenRouter for agent LLM calls OPENROUTER_MODEL= # Optional: model name (default: openai/gpt-4o-mini) NEXT_PUBLIC_ELEVENLABS_AGENT_ID= # ElevenLabs conversational agent STAGEHAND_ENV=LOCAL # LOCAL or BROWSERBASE BROWSERBASE_API_KEY= # Required for BROWSERBASE mode BROWSERBASE_PROJECT_ID= # Required for BROWSERBASE mode ``` ## Architecture ### File Structure ``` app/ layout.tsx # Root layout, wraps children in ReSightProvider page.tsx # Renders SplitLayout (dynamically imported, no SSR) globals.css # Tailwind 4 @import + @theme (black/cyan/yellow palette) dev/ page.tsx # Dev logging dashboard — polls /api/dev-logs, themed to match app api/ orchestrator/route.ts # POST: receives {instruction, history?}, runs agent orchestrator thought-stream/route.ts # GET: SSE stream of agent thoughts (EventSource) screenshot/route.ts # GET: returns latest base64 screenshot + bounding boxes dev-logs/route.ts # GET: poll logs (?since=ID), DELETE: clear logs clarification/route.ts # GET: poll pending question, POST: submit answer components/ SplitLayout.tsx # 50/50 flex split: MindPane | WorldPane providers/ ReSightProvider.tsx # React context: status, thoughts, [truncated — 8812 more characters] ``` ### package.json ``` { "name": "resight", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@ai-sdk/google": "^3.0.29", "@ai-sdk/openai": "^3.0.29", "@browserbasehq/stagehand": "^3.0.8", "@playwright/test": "^1.58.2", "@react-three/fiber": "^8.17.0", "ai": "^6.0.86", "clsx": "^2.1.1", "dotenv": "^17.3.1", "framer-motion": "^11.15.0", "next": "^16.1.6", "react": "^19.0.0", "react-dom": "^19.0.0", "tailwind-merge": "^3.4.0", "three": "^0.171.0", "zod": "^3.24.0" }, "devDependencies": { "@tailwindcss/postcss": "^4.0.0", "@types/node": "^22.0.0", "@types/react": "^19.0.0", "@types/react-dom": "^19.0.0", "@types/three": "^0.171.0", "tailwindcss": "^4.0.0", "typescript": "^5.7.0" } } ``` ### app/page.tsx ```typescript import SplitLayout from "@/components/SplitLayout"; export default function Home() { return ; } ``` ### app/layout.tsx ```typescript import type { Metadata } from "next"; import { ReSightProvider } from "@/components/providers/ReSightProvider"; import "./globals.css"; export const metadata: Metadata = { title: "ReSight — Voice Browser", description: "ReSight: The Ultimate Voice Browser for the Visually Impaired", }; export default function RootLayout({ children }: { children: React.ReactNode }) { return ( {children} ); } ``` ### app/dev/page.tsx ```typescript "use client"; import { useEffect, useRef, useState, useCallback } from "react"; interface DevLogEntry { id: number; timestamp: number; category: string; level: string; title: string; data?: Record; durationMs?: number; } const CATEGORY_COLORS: Record = { stagehand: "#a78bfa", llm: "#00e5ff", navigation: "#00ff6a", orchestrator: "#ffbe0b", navigator: "#d4ff00", error: "#f87171", }; const LEVEL_ICONS: Record = { info: "INFO", warn: "WARN", error: "ERR!", debug: "DBG ", }; function formatTime(ts: number) { return new Date(ts).toLocaleTimeString("en-US", { hour12: false, hour: "2-digit", minute: "2-digit", second: "2-digit", fractionalSecondDigits: 3, } as Intl.DateTimeFormatOptions); } export default function DevPage() { const [logs, setLogs] = useState([]); const [filter, setFilter] = useState("all"); const [autoScroll, setAutoScroll] = useState(true); const [expandedIds, setExpandedIds] = useState>(new Set()); const [connected, setConnected] = useState(false); const bottomRef = useRef(null); const lastIdRef = useRef(0); // Poll for new logs every second const poll = useCallback(async () => { try { const res = await fetch(`/api/dev-logs?since=${lastIdRef.current}`, { cache: "no-store", }); if (!res.ok) return; const newLogs: DevLogEntry[] = await res.json(); if (newLogs.length > 0) { lastIdRef.current = newLogs[newLogs.length - 1].id; setLogs((prev) => [...prev, ...newLogs].slice(-500)); setConnected(true); } } catch { setConnected(false); } }, []); useEffect(() => { // Initial fetch of all logs poll(); const interval = setInterval(poll, 1000); return () => clearInterval(interval); }, [poll]); useEffect(() => { if (autoScroll && bottomRef.current) { bottomRef.current.scrollIntoView({ behavior: "smooth" }); } }, [logs, autoScroll]); const filtered = filter === "all" ? logs : logs.filter((l) => l.category === filter); const categories = ["all", "stagehand", "llm", "navigation", "orchestrator", "navigator", "error"]; const toggleExpand = (id: number) => { setExpandedIds((prev) => { const next = new Set(prev); if (next.has(id)) next.delete(id); else next.add(id); return next; }); }; return (
{/* Header */}
ReSight DEV | {/* Category filters */} {categories.map((cat) => ( ))}
{filtered.length} entries