# Project export: KlarifAi This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: AI Math tutor that lives on your computer and see what you're seeing to analyze and offer various types of solutions and explanations to help you not only solve but understand the math problems. - Devpost: https://devpost.com/software/klarifai - GitHub: https://github.com/DMgaming00/ScreenTutor_AI - Video: https://www.youtube.com/embed/40AstKKiV0U?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) β€” Cursor (4 commits), Andre Agle (3 commits) ## Devpost submission (written by the team) ### Inspiration The idea for KlarifAi came from a problem we experienced ourselves while studying for standardized exams like the SAT. We frequently used ChatGPT to help solve math problems, but we found ourselves spending a lot of time manually describing equations, diagrams, graphs, and problem statements. For visual questions in particular, explaining the context accurately was often more work than solving the problem itself. We started asking a simple question: why should students have to translate what they see into text for an AI to understand it? What if an AI could see the problem directly and explain it the same way a tutor sitting next to you would? That question became the foundation for KlarifAi. ### What it does KlarifAi is a desktop AI assistant that analyzes what's on your screen and provides visual, step-by-step explanations for math problems in real time. Instead of copying and pasting questions or manually describing diagrams, users can simply have the problem visible on their screen. KlarifAi identifies the relevant mathematical content, understands the context, and generates guided explanations that break the solution into manageable steps. The goal is not just to provide answers, but to help students understand the reasoning behind them through visual and interactive guidance. ### How we built it At the core of KlarifAi is a multi-step AI workflow designed to replicate how a human tutor approaches a problem. First, the system captures and analyzes the user's screen, identifying the relevant mathematical content and surrounding context. That information is stored and managed through a memory layer that preserves important details throughout the interaction. Next, KlarifAi classifies the user's intentβ€”whether they need a full solution, a hint, conceptual clarification, or a step-by-step walkthrough. Based on that intent, the system performs targeted research and gathers additional context that may help explain the underlying concepts. Finally, all of this informationβ€”the screen analysis, conversation memory, user intent, research results, and visual contextβ€”is combined into a single reasoning request. The final AI call synthesizes everything into a structured explanation that is then rendered visually on screen for the user. Rather than relying on a single prompt, KlarifAi uses an orchestrated pipeline that separates observation, memory management, research, reasoning, and presentation into a unified workflow. ### Challenges we ran into One of our biggest challenges was reliably understanding complex mathematical content from screenshots. SAT-style problems often include graphs, geometric diagrams, tables, and formatting that can be difficult for AI systems to interpret correctly. Another challenge was balancing accuracy with usability. We wanted explanations that were detailed enough to teach concepts without overwhelming students with unnecessary information. We also spent significant time optimizing the user experience so that assistance felt immediate and seamless rather than requiring multiple manual steps. ### Accomplishments we're proud of We're proud that we transformed a frustration we personally experienced into a working product that feels natural to use. Instead of forcing students to rewrite questions for an AI, KlarifAi allows them to get help directly from what they already have on screen. We're especially proud of the quality of the visual step-by-step explanations and how quickly users can move from confusion to understanding. Most importantly, we built a tool that focuses on learning and comprehension rather than simply generating answers. ### What we learned Building KlarifAi taught us that the biggest barriers to learning are often not the concepts themselves, but the friction involved in accessing help. We learned how important visual context is for educational AI applications and how much information can be lost when students are forced to convert diagrams and equations into plain text. On the technical side, we gained experience integrating computer vision, multimodal AI systems, desktop application development, and user-centered design into a single product. ### What's next Our vision is to expand KlarifAi into a universal learning companion that can assist students across a wide range of subjects, not just mathematics. Future plans include: Support for science, physics, and engineering problems. More interactive visual explanations and tutoring experiences. Personalized learning paths based on student strengths and weaknesses. Improved real-time screen understanding and context awareness. Collaboration features for study groups and classrooms. Ultimately, we want KlarifAi to make high-quality, personalized tutoring accessible to anyone with a computer, turning every screen into an opportunity to learn. ## README (from the GitHub repository) # πŸŽ“ ScreenTutor AI **A cursor-native AI tutor that lives near your cursor, captures the screen on demand, and draws overlays directly on top of your windows.** ![Mock AI Mode](https://img.shields.io/badge/Mock_AI-Included-blueviolet) ![Redis Optional](https://img.shields.io/badge/Redis-Optional-green) ![Privacy First](https://img.shields.io/badge/Privacy-First-blue) --- ## What It Does - **Cursor-Native Buddy**: Buddy lives near your cursor, follows you around in a compact bubble state, and expands to a full chat panel on request. When explaining concepts, Buddy smoothly animates (flies) to coordinates on your screen to point at items. - **Captures Screen on Demand**: Trigger screen capture explicitly using `Ctrl+Shift+E` or the floating microphone button. Never records in the background or polls your screen silently. - **Draws Overlays on Top of Apps**: Draws circles, arrows, highlighters, step badges, and text labels directly on a transparent SVG window layered over your active screen content. - **Remembers Weak Concepts**: Uses Redis (or in-memory fallback) to log weak concepts, track question patterns, and adapt explanation styles dynamically. - **Speech Interactivity**: Push-to-talk microphone input records audio chunk sessions safely. Text-to-speech engine speaks answers out loud while stripping coordinate tags. --- ## Quick Start ### Prerequisites - **Node.js 20+** - **pnpm 9+** - **Docker** (optional β€” only needed for Redis container) ### Install & Run ```bash # 1. Clone the repository git clone cd 2nd_brain # 2. Install dependencies pnpm install # 3. Create env file cp .env.example .env # 4. Start Redis (optional β€” falls back to in-memory) docker compose up -d redis # 5. Start development environment (orchestrator + desktop concurrently) powershell -ExecutionPolicy Bypass -Command "pnpm dev" ``` --- ## Configuration & Setup ScreenTutor AI works **out of the box** in **Mock Mode** using local templates for lessons. To configure real integrations, set the following environment variables in `.env`: ### 1. Mock Mode (Default) Runs entirely locally without external API connections or paid accounts. ```env AI_PROVIDER=mock MEMORY_ENABLED=true ``` ### 2. Redis Memory To persist profile records, weak concepts, and user sessions across app restarts. ```env REDIS_URL=redis://localhost:6379 MEMORY_ENABLED=true ``` *Note: If Redis is unavailable or fails to connect, the orchestrator automatically logs a warning and falls back to `InMemoryService` in RAM.* ### 3. Claude (Anthropic AI) For real-time multi-modal screen capture analysis and chat responses. ```env AI_PROVIDER=claude ANTHROPIC_API_KEY=sk-ant-your-key-here CLAUDE_MODEL=claude-3-5-sonnet-20241022 ``` ### 4. Browserbase (Research Mode) Allows the tutor to spawn headless browser sessions to search Wikipedia/web resources for tutoring material. ```env BROWSERBASE_ENABLED=true BROWSERBASE_API_KEY=your-api-key BROWSERBASE_PROJECT_ID=your-project-id ``` ### 5. Simular (Co-drive Actions) Proposes desktop actions (clicks, keypresses) that the user can review and approve. ```env SIMULAR_ENABLED=true SIMULAR_API_KEY=your-simular-key ENABLE_SAFE_ACTIONS=true ``` ### 6. Fetch AI / Agentverse (Specialist Agent Routing) Registers and routes tutoring tasks to local or remote Fetch-compatible agents. ```env FETCH_AGENTVERSE_ENABLED=true FETCH_AGENTVERSE_API_KEY=your-agentverse-key FETCH_AGENTVERSE_AGENT_ADDRESS=agent-address-here ``` ### 7. Arize / Phoenix (Observability Tracing) Tracks telemetry spans, latency metrics, and LLM evaluations. ```env ARIZE_ENABLED=true ARIZE_API_KEY=your-arize-key PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006/v1/traces ``` ### 8. Sentry (Error Tracking) Captures backend runtime errors and desktop crashes. ```env SENTRY_ENABLED=true SENTRY_DSN=https://your-dsn-key@o0.ingest.sentry.io/0 ``` --- ## Privacy Model & Redaction - **On-Demand Capture Only**: Captures the screen only when the hotkey is triggered. No background polling or recordings. - **Sensitive Window Denylist**: Analysis is immediately blocked if any of the 16 sensitive window title terms (e.g. `password`, `keychain`, `bank`, `email`, `stripe`) are active. - **Data Redaction Layer**: A shared `RedactionService` automatically strips passwords, API keys, emails, credit cards, and query parameters from all logs, Sentry events, Arize telemetry, and UI debug views. - **Developer Redaction Override**: Available as a toggle in the developer panel, allowing developers to see raw data for debugging. It is turned off by default. - **No Screenshot Storage**: Screenshots are processed in memory and never written to disk or sent to observability endpoints. Only a secure MD5 hash is stored. --- ## Bundled Demo Lessons ScreenTutor includes six bundled offline lessons accessible by clicking the **Graduate Cap (πŸŽ“)** icon in the header: 1. **Algebra: distributive property** 2. **Algebra: common mistake** 3. **Calculus: chain rule** 4. **Calculus: u-substitution** 5. **Graph interpretation: vertex/slope** 6. **Coding error: off-by-one IndexError** Selecting a lesson displays a static lesson preview and allows you to test the hotkey, drawing overlays, and voice synthesis without external network calls. --- ## Troubleshooting ### `Error: connect ECONNREFUSED 127.0.0.1:6379` - **Cause**: Redis container is not running. - **Fix**: Redis is completely optional. The app falls back to memory storage. To enable Redis, start the container: `docker compose up -d redis`. ### Hotkey `Ctrl+Shift+E` Not Working - **macOS**: Ensure Electron has **Accessibility** permissions granted in System Preferences β†’ Privacy & Security β†’ Accessibility. - **Windows**: Run as Administrator if hotkeys are intercepted by other fullscreen windows. ### Screen Capture Black / Empty - **macOS**: Ensure Electron has **Screen Recording** permissions granted in System Preferences β†’ Privacy & Security β†’ Screen Recording. ### `listen EADDRINUSE :::4317` - **Cause**: Port 4317 is already in use by another process. - **Fix**: Change `ORCHESTRATOR_PORT` in your `.env` file to a free port. ## Detected evidence (automated analysis) Indexed codebase: 147 recognized source files, 709 KB. - Anthropic (technology) β€” detected in the code - CSS (language) β€” detected in the code - HTML (language) β€” detected in the code - JavaScript (language) β€” detected in the code - React (technology) β€” detected in the code - TypeScript (language) β€” detected in the code - Redis (technology) β€” claimed on Devpost, not found in the code - AI coding agent: Cursor β€” evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 159) ``` .env.example .gitignore .prettierrc apps/desktop/electron.vite.config.ts apps/desktop/package.json apps/desktop/src/main/hold-key-poll.win32.ts apps/desktop/src/main/index.ts apps/desktop/src/main/privacy.ts apps/desktop/src/main/screenshot.ts apps/desktop/src/main/sentry.ts apps/desktop/src/main/shortcuts.ts apps/desktop/src/preload/index.ts apps/desktop/src/renderer/api/client.ts apps/desktop/src/renderer/api/transcription.ts apps/desktop/src/renderer/api/tts.ts apps/desktop/src/renderer/App.css apps/desktop/src/renderer/App.tsx apps/desktop/src/renderer/components/AnnotationOverlay.css apps/desktop/src/renderer/components/AnnotationOverlay.tsx apps/desktop/src/renderer/components/ChatPanel.css apps/desktop/src/renderer/components/ChatPanel.tsx apps/desktop/src/renderer/components/CompactBuddy.css apps/desktop/src/renderer/components/CompactBuddy.tsx apps/desktop/src/renderer/components/DemoLessonsPanel.css apps/desktop/src/renderer/components/DemoLessonsPanel.tsx apps/desktop/src/renderer/components/DictationGlow.css apps/desktop/src/renderer/components/DictationGlow.tsx apps/desktop/src/renderer/components/ExplanationBubbleText.tsx apps/desktop/src/renderer/components/FloatingButton.css apps/desktop/src/renderer/components/FloatingButton.tsx apps/desktop/src/renderer/components/FloatingLauncher.css apps/desktop/src/renderer/components/FloatingLauncher.tsx apps/desktop/src/renderer/components/MemoryPanel.css apps/desktop/src/renderer/components/MemoryPanel.tsx apps/desktop/src/renderer/components/MockBadge.css apps/desktop/src/renderer/components/MockBadge.tsx apps/desktop/src/renderer/components/ModeTabs.css apps/desktop/src/renderer/components/ModeTabs.tsx apps/desktop/src/renderer/components/overlay-canvas.ts apps/desktop/src/renderer/components/PrivacyIndicator.css apps/desktop/src/renderer/components/PrivacyIndicator.tsx apps/desktop/src/renderer/components/RegionSelectOverlay.css apps/desktop/src/renderer/components/RegionSelectOverlay.tsx apps/desktop/src/renderer/components/SidebarRail.css apps/desktop/src/renderer/components/SidebarRail.tsx apps/desktop/src/renderer/components/SponsorStatusPanel.css apps/desktop/src/renderer/components/SponsorStatusPanel.tsx apps/desktop/src/renderer/components/TutorCursorIcon.tsx apps/desktop/src/renderer/demo/demo-math.html apps/desktop/src/renderer/demo/sample-math.html apps/desktop/src/renderer/env.d.ts apps/desktop/src/renderer/hooks/useChatHistory.ts apps/desktop/src/renderer/hooks/useScreenCapture.ts apps/desktop/src/renderer/hooks/useTutor.ts apps/desktop/src/renderer/index.html apps/desktop/src/renderer/main.tsx apps/desktop/src/renderer/overlay.html apps/desktop/src/renderer/overlay.tsx apps/desktop/src/renderer/panelLayout.ts apps/desktop/src/renderer/sentry.ts apps/desktop/src/renderer/types/explanation.ts apps/desktop/tsconfig.json apps/desktop/tsconfig.node.json apps/desktop/tsconfig.web.json CHANGES.md docker-compose.yml docs/architecture.md docs/current-state-audit.md docs/demo-script.md docs/fetch-agentverse-plan.md docs/final-acceptance-report.md docs/privacy-and-safety.md docs/sponsor-integrations.md implementation_plan.md package.json packages/shared/package.json packages/shared/src/capture-region.ts packages/shared/src/coordinates.ts packages/shared/src/demo-lessons.ts packages/shared/src/draw-shape-sanitize.ts packages/shared/src/explanation-format.ts packages/shared/src/index.ts packages/shared/src/narration-steps.ts packages/shared/src/overlay-refine.ts packages/shared/src/overlay-text.ts packages/shared/src/point-parser.ts packages/shared/src/privacy.ts packages/shared/src/redaction.ts packages/shared/src/schemas/action.ts packages/shared/src/schemas/drawing.ts packages/shared/src/schemas/memory.ts packages/shared/src/schemas/research.ts packages/shared/src/schemas/tutor.ts packages/shared/src/sponsors.ts packages/shared/src/stream-tag-parser.ts packages/shared/tsconfig.json pnpm-workspace.yaml README.md services/orchestrator/package.json services/orchestrator/src/__tests__/action-safety.test.ts services/orchestrator/src/__tests__/agents.test.ts services/orchestrator/src/__tests__/capture-region.test.ts services/orchestrator/src/__tests__/coordinate-mapping.test.ts services/orchestrator/src/__tests__/draw-shape-sanitize.test.ts services/orchestrator/src/__tests__/drawing-coords.test.ts services/orchestrator/src/__tests__/gemini-provider.test.ts services/orchestrator/src/__tests__/memory.test.ts services/orchestrator/src/__tests__/mock-provider.test.ts services/orchestrator/src/__tests__/narration-steps.test.ts services/orchestrator/src/__tests__/observability.test.ts services/orchestrator/src/__tests__/orchestrator-recent-turns.test.ts services/orchestrator/src/__tests__/orchestrator.test.ts services/orchestrator/src/__tests__/overlay-refine.test.ts services/orchestrator/src/__tests__/overlay-text.test.ts services/orchestrator/src/__tests__/point-tag-parser.test.ts services/orchestrator/src/__tests__/privacy-guard.test.ts services/orchestrator/src/__tests__/recent-turns-prompt.test.ts services/orchestrator/src/__tests__/redaction.test.ts services/orchestrator/src/__tests__/schemas.test.ts services/orchestrator/src/__tests__/semantic-cache.test.ts [39 more files omitted for size] ``` ### Dependencies - apps/desktop/package.json: @electron-toolkit/preload@^3.0.0, @electron-toolkit/utils@^4.0.0, @screentutor/shared@workspace:*, @sentry/node@^10.58.0, @sentry/react@^10.58.0, @types/react@^19.0.0, @types/react-dom@^19.0.0, @vitejs/plugin-react@^6.0.2, electron@^42.4.1, electron-vite@^5.0.0, koffi@^3.0.2, react@^19.2.7, react-dom@^19.2.7, typescript@^5.8.0 - package.json: concurrently@^9.1.0, prettier@^3.5.0, typescript@^5.8.0 - packages/shared/package.json: typescript@^5.8.0, zod@^4.4.3 - services/orchestrator/package.json: @anthropic-ai/sdk@^0.104.2, @fastify/cors@^11.0.0, @screentutor/shared@workspace:*, @sentry/node@^10.58.0, @types/node@^22.0.0, @types/uuid@^10.0.0, dotenv@^16.5.0, fastify@^5.8.5, ioredis@^5.11.1, tsx@^4.19.0, typescript@^5.8.0, uuid@^11.1.0, vitest@^4.1.9 ### Recent commits (newest first) - feat: add screen region selection and fix overlay interactions - feat: interface visible again - feat: use logo for widget/floater - Add hold-to-dictate hotkey, Siri glow UI, and Redis semantic cache. - Improve overlay diagram annotations so shapes align with narration. - feat: use recent logs from redis to inform claude api and output more conversation aware answers - Fix coordinate mapping so overlay pointers align with captured display. - Initial commit: ScreenTutor AI ## Key source files (fetched from GitHub, selected and truncated for size) ### CHANGES.md ```markdown # ScreenTutor AI Optimization & Polish Achievements This document summarizes the performance, architecture, and polish changes implemented across the ScreenTutor AI codebase. --- ## 1. Streaming Response Engine (Anthropic SSE) - **Problem**: Large model completions blocked the interface until the entire response was fully generated. - **Solution**: - Refactored `AnthropicProvider` in [anthropic-provider.ts](file:///c:/Users/User/Desktop/2nd_brain/services/orchestrator/src/providers/anthropic-provider.ts) to utilize Anthropic's streaming API with `stream: true`. - Modified the main `/ask` orchestrator endpoint to return a Server-Sent Events (SSE) stream (`text/event-stream`). - Updated [useTutor.ts](file:///c:/Users/User/Desktop/2nd_brain/apps/desktop/src/renderer/hooks/useTutor.ts) to parse the SSE stream chunk-by-chunk using `ReadableStream`. - Implemented a smooth, CSS-animated `.blinking-cursor` element in the renderer chat windows to provide immediate feedback during generation. ## 2. Ultra-Fluid Fullscreen Overlay & Cursor Follow - **Problem**: The app container previously relied on heavy static window resizing (`setBounds`) on mouse movements, causing visual stuttering and grey background flickers. - **Solution**: - Reconfigured the Electron windows in [index.ts](file:///c:/Users/User/Desktop/2nd_brain/apps/desktop/src/main/index.ts) to run fullscreen, transparent, and default to ignore-mouse-events (click-through). - Designed a high-frequency (60fps) mouse coordinates IPC broadcaster. - Implemented CSS custom variables (`--bx` and `--by`) on the renderer to position widgets via hardware-accelerated `transform: translate3d(var(--bx), var(--by), 0)` to achieve butter-smooth 60fps movement. - Set up dynamic mouse-events ignore toggling on the panel hover bounds. ## 3. Redis Memory Pipeline & Local Caching - **Problem**: Concurrent, repetitive round-trips to Redis slowed down request cycle processing times. - **Solution**: - Implemented a 5-minute local cache (`profileCache`) in [redis-memory.ts](file:///c:/Users/User/Desktop/2nd_brain/services/orchestrator/src/services/redis-memory.ts) to avoid redundant retrievals. - Refactored profile fetches, session turn additions, and memory resets to run using Redis pipelines (`redis.pipeline()`), consolidating commands into single network transactions. ## 4. Browserbase Session Re-use & Research Cache - **Problem**: Launching fresh browser sessions for every web search created high latency. - **Solution**: - Enhanced the browser service in [browserbase.ts](file:///c:/Users/User/Desktop/2nd_brain/services/orchestrator/src/services/browserbase.ts) to fetch active browser sessions via the API (`GET /v1/sessions`) and re-use running sessions. - Added a 60-second in-memory cache for research queries to prevent hitting Browserbase or external APIs for identical questions. ## 5. Non-Blocking Telemetry & Arize Queueing - **Problem**: Sending telemetry payloads to Arize (OTLP) dur [truncated β€” 2811 more characters] ``` ### implementation_plan.md ```markdown # ScreenTutor AI - Implementation Plan ## 1. Product Goal ScreenTutor AI is a cursor-native AI tutor that lives near the user’s cursor, activates with a hotkey or floating button, and captures the screen only after explicit user action. It understands the visible learning material to explain it briefly using Claude or mock-Claude, points/draws directly on the screen using a transparent SVG overlay, and teaches through hints, progressive explanation ladders, quizzes, and misconception detection rather than immediately providing the final solution. The assistant remembers weak concepts using Redis, executes web research via Browserbase, proposes safe GUI actions through Simular/SimuLang, routes to specialist agents inspired by Fetch AI/Agentverse, and instruments the AI pipeline and app telemetry using Arize and Sentry. --- ## 2. Sponsor Architecture We strictly include and integrate only the following sponsor technologies: - **Claude**: Anthropic Vision API for visual screen understanding and text replies. - **Fetch AI / Agentverse**: Routing model matching specialist local or remote tutor agents. - **Redis**: Persistent learner profiles, session logs, recent questions, and concept mappings. - **Browserbase**: Headless browser automation for live concept scraping and simpler research web summaries. - **Simular / Sai / SimuLang**: Safe desktop automation, calculating coordinates and proposing clicks/keys with approval cards. - **Arize**: Tracing of LLM inputs, completions, tokens, and latency measurements. - **Sentry**: Capturing main process, renderer, and orchestrator errors and performance spans. ### Explicitly Excluded Runtime Integrations: - QNX - Zoox - Cognition / Devin - Annapurna Labs / AWS - Poke / Interaction Co No code or active runtime integration will use or mention these excluded sponsors. --- ## 3. Target Repo Structure We align with the existing monorepo structure, extending components to enforce clarity and sponsor segregation: ``` /apps/desktop β”œβ”€β”€ src/main β”‚ β”œβ”€β”€ index.ts # Electron main process, transparent shell management β”‚ β”œβ”€β”€ screenshot.ts # Screen capture via desktopCapturer β”‚ β”œβ”€β”€ shortcuts.ts # Global hotkey registration (Ctrl+Shift+E) β”‚ └── privacy.ts # Privacy filter & window title blocklist └── src/renderer β”œβ”€β”€ App.tsx # Main entry, toggling collapsed/expanded state └── components β”œβ”€β”€ ChatPanel.tsx # Chat interface, prompt inputs, text displays β”œβ”€β”€ AnnotationOverlay.tsx # Transparent SVG canvas overlay drawing layer └── MemoryPanel.tsx # Memory state visualization (concepts, history) /services/orchestrator β”œβ”€β”€ src/providers β”‚ β”œβ”€β”€ ai-provider.ts # AI Interface β”‚ β”œβ”€β”€ anthropic-provider.ts # Claude Vision API implementation β”‚ └── mock-provider.ts # Mock-Claude matching core demo rules β”œβ”€β”€ src/integrations β”‚ β”œβ”€β”€ browserbase.ts # Headless research brow [truncated β€” 6010 more characters] ``` ### docker-compose.yml ```yaml version: '3.8' services: redis: image: redis:7-alpine container_name: screentutor-redis ports: - '6379:6379' volumes: - redis-data:/data command: redis-server --appendonly yes restart: unless-stopped volumes: redis-data: ``` ### package.json ``` { "name": "screentutor-ai", "version": "0.1.0", "private": true, "description": "ScreenTutor AI β€” an always-available desktop AI tutor that draws on your screen", "scripts": { "dev": "concurrently \"pnpm dev:orchestrator\" \"pnpm dev:desktop\"", "dev:orchestrator": "pnpm --filter @screentutor/orchestrator dev", "dev:desktop": "pnpm --filter @screentutor/desktop dev", "build": "pnpm --filter @screentutor/shared build && pnpm --filter @screentutor/orchestrator build && pnpm --filter @screentutor/desktop build", "test": "pnpm --filter @screentutor/orchestrator test", "lint": "eslint . --ext .ts,.tsx", "typecheck": "pnpm -r typecheck", "format": "prettier --write \"**/*.{ts,tsx,json,md,css,html}\"" }, "devDependencies": { "concurrently": "^9.1.0", "typescript": "^5.8.0", "prettier": "^3.5.0" }, "engines": { "node": ">=20.19.0", "pnpm": ">=9.0.0" }, "pnpm": { "onlyBuiltDependencies": [ "electron", "koffi" ] } } ``` ### packages/shared/package.json ``` { "name": "@screentutor/shared", "version": "0.1.0", "private": true, "type": "module", "main": "./src/index.ts", "types": "./src/index.ts", "exports": { ".": "./src/index.ts" }, "scripts": { "build": "echo 'No build step required'", "typecheck": "tsc --noEmit" }, "dependencies": { "zod": "^4.4.3" }, "devDependencies": { "typescript": "^5.8.0" } } ``` ### services/orchestrator/package.json ``` { "name": "@screentutor/orchestrator", "version": "0.1.0", "private": true, "type": "module", "scripts": { "dev": "tsx watch src/index.ts", "build": "tsc", "start": "node dist/index.js", "test": "vitest run", "test:watch": "vitest", "typecheck": "tsc --noEmit" }, "dependencies": { "@anthropic-ai/sdk": "^0.104.2", "@fastify/cors": "^11.0.0", "@screentutor/shared": "workspace:*", "@sentry/node": "^10.58.0", "dotenv": "^16.5.0", "fastify": "^5.8.5", "ioredis": "^5.11.1", "uuid": "^11.1.0" }, "devDependencies": { "@types/node": "^22.0.0", "@types/uuid": "^10.0.0", "tsx": "^4.19.0", "typescript": "^5.8.0", "vitest": "^4.1.9" } } ``` ### apps/desktop/package.json ``` { "name": "@screentutor/desktop", "version": "0.1.0", "private": true, "main": "./out/main/index.js", "scripts": { "dev": "electron-vite dev", "build": "electron-vite build", "preview": "electron-vite preview", "typecheck": "tsc --noEmit -p tsconfig.node.json && tsc --noEmit -p tsconfig.web.json" }, "dependencies": { "@screentutor/shared": "workspace:*", "@sentry/node": "^10.58.0", "@sentry/react": "^10.58.0", "koffi": "^3.0.2", "react": "^19.2.7", "react-dom": "^19.2.7" }, "devDependencies": { "@electron-toolkit/preload": "^3.0.0", "@electron-toolkit/utils": "^4.0.0", "@types/react": "^19.0.0", "@types/react-dom": "^19.0.0", "@vitejs/plugin-react": "^6.0.2", "electron": "^42.4.1", "electron-vite": "^5.0.0", "typescript": "^5.8.0" } } ``` ### packages/shared/src/index.ts ```typescript // @screentutor/shared β€” central type and schema barrel export // Schemas export * from './schemas/drawing.js'; export * from './schemas/tutor.js'; export * from './schemas/memory.js'; export * from './schemas/action.js'; export * from './schemas/research.js'; export * from './sponsors.js'; export * from './point-parser.js'; export * from './overlay-refine.js'; export * from './draw-shape-sanitize.js'; export * from './stream-tag-parser.js'; export * from './overlay-text.js'; export * from './narration-steps.js'; export * from './explanation-format.js'; // Privacy export * from './privacy.js'; // Demo lessons (offline/mock mode) export * from './demo-lessons.js'; // Redaction & Observability Safety export * from './redaction.js'; // Coordinate Mapping export * from './coordinates.js'; export * from './capture-region.js'; ``` ### services/orchestrator/src/index.ts ```typescript import { patchConsole } from '@screentutor/shared'; patchConsole(); import Fastify from 'fastify'; import cors from '@fastify/cors'; import { config } from './config.js'; import type { AIProvider } from './providers/ai-provider.js'; import type { BrowserbaseIntegration } from './integrations/browserbase.js'; import type { SimularIntegration } from './integrations/simular.js'; import type { ElevenLabsIntegration } from './integrations/elevenlabs.js'; import type { ObservabilityService } from './observability/observability-service.js'; import type { TutorOrchestrator } from './orchestrator/tutor-orchestrator.js'; import * as Sentry from '@sentry/node'; import { initOrchestratorSentry } from './observability/sentry.js'; // Initialize Sentry error tracing initOrchestratorSentry(); import { healthRoutes } from './routes/health.js'; import { tutorRoutes } from './routes/tutor.js'; import { memoryRoutes } from './routes/memory.js'; import { researchRoutes } from './routes/research.js'; import { actionRoutes } from './routes/actions.js'; import { sponsorRoutes } from './routes/sponsors.js'; import { ttsRoutes } from './routes/tts.js'; // Setup fastify instance const fastify = Fastify({ logger: true }); // Decorate fastify so routes can access services declare module 'fastify' { interface FastifyInstance { getOrchestrator(): Promise; getMemory(): Promise; getBrowserbase(): Promise; getSimular(): Promise; getElevenLabs(): Promise; getObservability(): Promise; } } let aiProviderInstance: AIProvider | null = null; async function getAIProvider(): Promise { if (!aiProviderInstance) { if (!config.ANTHROPIC_API_KEY) { throw new Error("ANTHROPIC_API_KEY is missing! Please provide it in the .env file."); } const { ClaudeProvider } = await import('./providers/anthropic-provider.js'); aiProviderInstance = new ClaudeProvider(config.ANTHROPIC_API_KEY); } return aiProviderInstance; } let memoryInstance: any = null; async function getMemory(): Promise { if (!memoryInstance) { const { RedisMemoryService } = await import('./memory/redis-memory.js'); memoryInstance = await RedisMemoryService.create(config.REDIS_URL); } return memoryInstance; } let browserbaseInstance: BrowserbaseIntegration | null = null; async function getBrowserbase(): Promise { if (!browserbaseInstance) { const { BrowserbaseService, MockBrowserbaseService } = await import('./integrations/browserbase.js'); if (config.BROWSERBASE_ENABLED && config.BROWSERBASE_API_KEY) { const mem = await getMemory(); browserbaseInstance = new BrowserbaseService(config.BROWSERBASE_API_KEY, config.BROWSERBASE_PROJECT_ID, mem); } else { browserbaseInstance = new MockBrowserbaseService(); } } return browserbaseInstance; } let simularInstance: SimularIntegration | null = null; async function getSimular(): Promise { if (!simularInstance) { const { RealSimularService, MockSimularService } = await import('./integrations/simular.js'); simularInstance = (config.SIMULAR_ENABLED && config.SIMULAR_API_KEY) ? new RealSimularService(config.SIMULAR_API_KEY, config.ENABLE_SAFE_ACTIONS) : new MockSimularService(); } return simularInstance; } let elevenLabsInstance: ElevenLabsIntegration | null = null; async function getElevenLabs(): Promise { if (!elevenLabsInstance) { const { ElevenLabsService, MockElevenLabsService } = await import('./integrations/elevenlabs.js'); elevenLabsInstance = (config.ELEVENLABS_ENABLED && config.ELEVENLABS_API_KEY && config.ELEVENLABS_VOICE_ID) ? new ElevenLabsService(config.ELEVENLABS_API_KEY, config.ELEVENLABS_VOICE_ID, config.ELEVENLABS_MODEL_ID) : new MockElevenLabsService(); } return elevenLabsInstance; } let observabilityInstance: ObservabilityService | null = null; async function getObservability(): Promise { if (!observabilityInstance) { const { ArizePhoenixObservabilityService, NoopObservabilityService } = await import('./observability/observability-service.js'); observabilityInstance = (config.ARIZE_ENABLED && config.PHOENIX_COLLECTOR_ENDPOINT) ? new ArizePhoenixObservabilityService(config.ARIZE_API_KEY, config.PHOENIX_COLLECTOR_ENDPOINT) : new NoopObservabilityService(); } return observabilityInstance; } let orchestratorInstance: TutorOrchestrator | null = null; async function getOrchestrator(): Promise { if (!orchestratorInstance) { const aiProv = await getAIProvider(); const mem = await getMemory(); const bb = await getBrowserbase(); const sim = await getSimular(); const obs = await getObservability(); const { AgentRouter } = await import('./agents/agent-router.js'); const { TutorOrchestrator } = await import('./orchestrator/tutor-orchestrator.js'); const agentRouter = new AgentRouter(aiProv, mem, bb, obs); orchestratorInstance = new TutorOrchestrator(aiProv, mem, agentRouter, bb, sim, obs); } return orchestratorInstance; } fastify.decorate('getOrchestrator', getOrchestrator); fastify.decorate('getMemory', getMemory); fastify.decorate('getBrowserbase', getBrowserbase); fastify.decorate('getSimular', getSimular); fastify.decorate('getElevenLabs', getElevenLabs); fastify.decorate('getObservability', getObservability); import { RedactionService } from '@screentutor/shared'; // Global error handler to redact error responses fastify.setErrorHandler((error: any, request, reply) => { fastify.log.error(error); const safeError = RedactionService.redactError(error); reply.status(error.statusCode || 500).send({ statusCode: error.statusCode || 500, error: safeError.name || 'Internal Server Error', message: safeError.message || 'An error occurred' }); }); / [truncated β€” 1103 more characters] ``` ### apps/desktop/src/renderer/main.tsx ```typescript import { patchConsole } from '@screentutor/shared'; patchConsole(); import React from 'react'; import { createRoot } from 'react-dom/client'; import App from './App'; import './App.css'; import { initRendererSentry } from './sentry'; initRendererSentry(); const rootElement = document.getElementById('root'); if (rootElement) { createRoot(rootElement).render( ); } ``` [139 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]