# Project export: Ferb AI - Friendly Education & Reasoning Bots This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Your AI tutor that thinks on the board, not just in the chat. - Devpost: https://devpost.com/software/ferb-ai-friendly-education-reasoning-bots - GitHub: https://github.com/maunguyengit/FerbAI - Video: https://www.youtube.com/embed/Rst3uLkklW0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Henry Qi (9 commits), maucodemau (7 commits), Claude Opus 4.8 (2 commits) ## Devpost submission (written by the team) ### Inspiration Most learning tech feels passive. You watch a video or scroll through text while an AI explains things at you. Great teachers do something different. They stand at a whiteboard, work through each step by hand, explain their thinking out loud, and slow down the moment a student looks confused. Ferb AI was built to feel more like that kind of teaching. The goal was to create an AI tutor that teaches on a whiteboard instead of inside a chat box, while still letting students replay lessons, jump back to specific moments, and ask questions in the middle of a recording. ### What it does Ferb AI is an AI tutor built around a live infinite whiteboard. A student can sketch a problem, and the tutor writes the next step by hand in a chalk-style font while explaining it out loud. The audio is synced to the writing so it feels like a real teacher working at the board. The system reads the current whiteboard using vision, so it can respond to the actual problem on screen instead of a separate text prompt. It can switch into a graphing window to plot 2D and 3D functions, then move into a Learn tab that generates interactive visualizations students can drag through and explore step by step. Students can talk to it using hands-free voice input through Deepgram or type normally. Once signed in, every lesson is saved to the user's account, including the recording, audio, transcript, and auto-generated chapters. A lesson can be shared through a link, and only signed-in students with access can open it. During playback, students get auto-chapters, live captions, transcript search, and full scrubbing. There is also an "ask the recording" feature. A student can pause a replay, ask a question by voice, and the AI responds out loud while annotating the frozen board in blue. When the question is done, those annotations fade and the lesson can continue. Every AI call is traced and evaluated with Arize so teaching quality can be monitored instead of guessed. How it was built Frontend The frontend uses React, TypeScript, and Vite. It includes a custom infinite whiteboard with pan and zoom support, hand-drawn strokes, text, and shapes. It also includes a Plotly graphing window powered by mathjs for 2D and 3D plotting, plus a sandboxed iframe renderer that safely runs AI-generated interactive widgets. Backend The backend is a Node and Express proxy. It streams Anthropic Claude responses with server-sent events and handles voice, data, and authentication services so secrets never reach the browser. AI system Claude acts as the core reasoning model. It reads a snapshot of the whiteboard and returns structured action blocks that the app can render directly. Those actions may tell the app what to write on the board, what equations to graph, or what interactive experience to build. Voice Deepgram handles both speech-to-text and text-to-speech. Live microphone input is streamed through a server-side WebSocket relay, and text-to-speech uses the aura-asteria-en voice. A custom math-to-speech normalizer was added so expressions sound natural, like reading x^2 as "x squared" instead of "x caret two." Auth, database, and storage Supabase turns the project into a real product instead of a one-off demo. Email authentication is protected with Row Level Security so each user only sees their own recordings. Recordings are stored in Postgres as JSONB, including the event stream, scene snapshots, transcript, and chapters, so lessons can be replayed accurately. Audio is stored in a private Supabase Storage bucket, and the backend creates short-lived signed URLs so private audio can still be streamed securely. Shared recordings are rechecked on the server for ownership and sharing permissions before audio is served. A fallback save path keeps a local copy of a lesson if a cloud save fails, which helps prevent data loss. Recording and playback The replay engine uses a scene model that stores board elements, graph equations, visualization specs, and the active view as incremental events plus periodic snapshots. A sceneAt(t) reconstruction model restores any point in time, while a requestAnimationFrame loop driven by the audio clock keeps the visual playback tightly synced. Observability Arize AX is used for observability and evaluation. OpenTelemetry tracing exports spans to Arize for lesson generation, chapter generation, and recording Q and A. A second evaluator agent scores the tutor on engagement, scaffolding, tone, goal alignment, and grounding to what is actually on the board. Challenges Several parts of the build were harder than expected. Deepgram browser token limits caused live transcription failures, so the speech-to-text pipeline had to be redesigned around a server-side WebSocket audio relay. Deepgram text-to-speech does not provide word-level timestamps, so syncing speech with handwriting had no obvious timing source. The solution was to split narration into beats, measure each audio clip's real duration, and let the voice timing control the writing speed. Keeping narration synchronized while switching between the whiteboard, Plotly graphs, and interactive demos required careful state and timing logic. Supabase introduced real production concerns like email rate limits, Row Level Security policies, private audio access, signed URL streaming, and reliable cloud saves. Arize integration also took significant work because tracing design, evaluator spans, and automated scoring had to be meaningful rather than just technically connected. A lot of the process involved debugging undocumented API limits, rendering edge cases, and synchronization issues that took many iterations to solve. Accomplishments A few parts of Ferb AI stand out. The handwriting and voice sync feels genuinely close to a teacher explaining at a board, and the pacing is driven by the spoken audio itself. The full account, record, replay, and ask-the-recording loop works end to end with authentication, persistent recordings, private audio, sharing, captions, chapters, and voice follow-up questions. The system does not just generate tutoring sessions. It also evaluates them with Arize using an automated rubric for engagement, scaffolding, tone, and goal alignment. The product combines three teaching surfaces, the board, graphing tools, and interactives, into one continuous experience. What was learned This project taught a few clear lessons. Real APIs often force architecture changes, not just quick configuration fixes. Using voice as the timing source is a clean and effective way to synchronize speech with animation. Shipping an actual product means solving authentication, privacy, persistence, and sharing, not just building a clever interface. Observability matters for AI products because teaching quality should be measured and improved over time. Prompting an AI that acts by writing, graphing, and building is a different design problem from prompting one that only chats. ### What's next Ferb AI started in education because teaching on a live whiteboard is especially useful there, but the underlying system can go much further. Expand into subjects like math, physics, computer science, and chemistry with richer topic-specific interactive tools. Build a team of specialized agents where one lesson director coordinates graphing and visualization specialists in parallel. Add deeper Arize dashboards and regression alerts to track teaching quality over time and across domains. Support classrooms and teams with progress tracking, assignments, teacher dashboards, and shared content libraries. Make the experience more mobile-friendly so lessons can happen on any device. The broader vision is not only an education tool. It is a reasoning system for any field where an expert would normally explain something visually, such as medicine, finance, engineering, legal analysis, policy, or employee training. ## README (from the GitHub repository) # FerbAI — Brutalist Whiteboard + AI Tutor A whiteboard on the left (draw, like Excalidraw), an AI study companion on the right that **sees your board** and guides you to the next step. Built with React + TypeScript + Vite. Brutalist theme: thick black rules, hard offset shadows, electric-yellow accent. ## Run it ```bash npm install cp .env.example .env # then paste whatever keys you have (all optional) npm run dev # starts the Vite frontend AND the backend proxy together ``` Open http://localhost:5173. `npm run dev` runs both processes via `concurrently` (`web` = Vite on 5173, `api` = proxy on 8787). Vite forwards `/api/*` to the proxy, so there's nothing else to configure. You don't strictly need a `.env` — you can paste keys into the in-app **Settings (⚙)** panel instead (see below). ## Use it 1. **Draw** your problem on the left — pen, eraser, lasso (select + drag + `Delete`), text, box, oval. Pick colour + stroke width. `Ctrl+Z` / `Ctrl+Shift+Z` undo/redo. `⤓ PNG` downloads the board. 2. **Pick a model** in the dropdown (top-right). Vision-capable models can read the board snapshot — look for the `👁 sees board` chip. 3. **Add your API key.** Two ways, your choice: - **Backend `.env`** (recommended) — keys never touch the browser. The status chip shows `● key set` automatically. - **Settings (⚙)** — paste a key in the UI. It's stored in your browser's `localStorage` and forwarded to the local proxy per request, overriding the `.env` key. Leave a field blank to fall back to the server key. 4. **Ask →**. With *attach board snapshot* on, the AI gets a PNG of your drawing and nudges you toward the next step (Socratic — not the full answer). ## Models The chat runs on **Claude Code** (Anthropic). Pick the model in the top-bar dropdown — **Claude Sonnet 4.6** (default) · Claude Opus 4.8 · Claude Haiku 4.5. The Base URL is editable in Settings. Models live in [`src/lib/providers.ts`](src/lib/providers.ts). ## Record & replay a lesson (▶ Replay) Hit **● Record** (top bar) to capture a lesson, then **Stop**. Everything you do on the board — strokes, text, shapes, erases, undo/redo, and the AI's annotations — is logged as timestamped events; the microphone is captured in parallel via `MediaRecorder` (optional — if there's no mic, the lesson records silently). Open the **▶ Replay** tab to watch it back. **The engine** ([`src/lib/recording/`](src/lib/recording/)): a recording is an incremental **event log** (for smooth playback) + periodic **snapshots** (full board state every 30s, for instant seeking). Playback is a `requestAnimationFrame` loop driven by the audio clock (`audio.currentTime`) — or a virtual clock when silent — applying events as their timestamps pass. **Seeking** to time *T* finds the nearest snapshot ≤ *T* and fast-forwards events from there, so scrubbing is instant regardless of lesson length. The live board and the playback canvas share one renderer ([`src/lib/render.ts`](src/lib/render.ts)) so replays are pixel-identical. > Recordings are in-memory for the session (audio as an object URL). Persistence > (IndexedDB) and Deepgram live transcription are later phases. ## Learn window — interactive lessons the AI builds (not videos) Toggle the left panel to **◆ Learn**. Ask the tutor to teach a concept and it builds something you **step through, edit, and explore** — active learning, not a wall of text. - *"Teach me binary search trees"* → an interactive BST: insert / find / remove / traverse, stepping through every comparison, editable values. - *"Teach me selection sort with 9, 3, 7, 1, 5"* → an animated sorter with play / step / speed and an editable array. - *"Teach me how a stack works"* → a custom push/pop visualizer. **The architecture (fighting AI generation error).** The AI does **not** generate interactive code from scratch — that's where LLMs break. Instead: 1. **Tier 1 — reusable widgets.** A registry of pre-built, tested, fully interactive widgets ([`src/components/viz/`](src/components/viz/)) owns *all* the hard parts (stepping, animation, drag, editing). The AI emits only a tiny **spec** — which widget + initial data + narration — via a `ferbai-viz` JSON block. Near-zero generation risk. The widget catalog ([`src/lib/viz/registry.ts`](src/lib/viz/registry.ts)) is fed to the AI's prompt so it knows what to reuse. 2. **Tier 2 — sandboxed custom.** When no widget fits, the AI emits self-contained HTML in a *separate* `ferbai-html` block (raw HTML in its own fence — no JSON escaping to corrupt), rendered in a **sandboxed iframe** (`allow-scripts`, no same-origin) so it can't touch the app. Adding a new widget is one registry entry + one component — the AI can use it immediately. Starter chips let you explore the built-ins without asking the AI. ## Graph window (Desmos-style 2D + 3D) the AI can plot on Toggle the left panel between **✎ Board** and **∿ Graph** (top bar). The graph window takes typed equations and the AI can plot onto it too. - **Type equations** in the side rail: explicit `y = x^2 + 9`, surfaces `z = x^2 - y^2`, or implicit relations `x^2 + y^2 + z^2 = 9`. Anything using `z` renders in **3D** (drag to rotate, scroll to zoom); otherwise **2D**. Click a color dot to show/hide, double-click to recolor, `×` to delete. - **The AI plots too.** In the graph view, ask things like *"graph the derivative and integral of y = x³ + 3x²"* or *"plot a cubic that intersects this parabola"*. The model **does the calculus itself** and emits a `ferbai-graph` block of equations, which the app plots (marked `✦` as AI-added). It sees the current graph (snapshot + equation list) so it builds on what's there. Built on [Plotly.js](https://plotly.com/javascript/) (2D lines/contours, 3D surfaces/isosurfaces) + [mathjs](https://mathjs.org/) (parsing/evaluation). The equation engine lives in [`src/lib/graph.ts`](src/lib/graph.ts); the panel in [`GraphView.tsx`](src/components/GraphView.tsx). > Note: bundling Plotly makes the production JS large (~5.7 MB / 1.7 MB gzip). > Fine for local use; code-split it (lazy-load the graph) before shipping wide. ## The AI writes ON the board (not just chat) FerbAI's tutor doesn't only talk in the sidebar — it **writes the next step onto the whiteboard**, in the empty space, like a teacher at a board. Draw `3x = 12`, ask "solve for x", and it writes `3x / 3 = 12 / 3` in blue directly below your work. How it stays accurate (instead of guessing pixel positions blind): 1. **The app tells the model the geometry.** Every request includes the board size and the bounding box of your existing strokes ([`getBoardMeta`](src/components/Whiteboard.tsx)), so the model knows exactly where the empty space is. 2. **The model returns structured draw commands** in a fenced `ferbai-draw` JSON block — `text`, `arrow`, `line`, `rect`, `ellipse` (circle the answer), and `highlight` (mark a mistake on your work). The block is parsed out of the chat text and rendered as real, **undoable** board elements in AI-blue, with a fade-in ([`drawblock.ts`](src/lib/drawblock.ts), `applyAIActions`). 3. **The "✎ AI draws on board" toggle** (on by default) injects a firm directive so the model draws whenever you want it — turn it off for pure chat. Use a **vision** model (look for `👁 sees board`) so it can read your actual handwriting; the geometry hints help non-vision models place text too. Anything the AI draws is just board elements — undo it, erase it, or move it like your own. ## How the backend helps All model calls go through a thin local proxy ([`server/index.js`](server/index.js)) instead of straight from the browser. That buys you three things: - **No CORS pain.** The browser only ever calls same-origin `/api/*`. The proxy makes the real provider call server-side, where CORS doesn't apply. - **Keys can stay off the client.** Put them in `.env` and the browser never sees them. - [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 79 recognized source files, 401 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Redis (technology) — detected in the code - SQL (language) — detected in the code - Supabase (technology) — detected in the code - TypeScript (language) — detected in the code - Anthropic (technology) — claimed on Devpost, not found in the code - Node.js (technology) — claimed on Devpost, not found in the code - PostgreSQL (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (92 of 92) ``` .agents/skills/redis-semantic-cache/.cursor-plugin/plugin.json .agents/skills/redis-semantic-cache/references/best-practices.md .agents/skills/redis-semantic-cache/references/langcache-usage.md .agents/skills/redis-semantic-cache/SKILL.md .agents/skills/redis-vector-search/.cursor-plugin/plugin.json .agents/skills/redis-vector-search/references/algorithm-choice.md .agents/skills/redis-vector-search/references/hybrid-search.md .agents/skills/redis-vector-search/references/index-creation.md .agents/skills/redis-vector-search/references/rag-pattern.md .agents/skills/redis-vector-search/SKILL.md .claude/launch.json .gitignore docs/architecture.md index.html package.json README.md requirements.txt server/agent2_mcp.js server/deepgram.js server/e2e_embeddings.mjs server/embedding_worker.py server/index.js server/instrumentation.mjs server/memory.js server/memory.test.js server/providers.js server/redisvl_embed.py server/supabase.js server/tutorReview.js server/wait_for_api.mjs skills-lock.json src/App.css src/App.tsx src/components/auth/AuthGate.css src/components/auth/AuthGate.tsx src/components/ChatPanel.css src/components/ChatPanel.tsx src/components/GraphView.css src/components/GraphView.tsx src/components/ModelSelector.css src/components/ModelSelector.tsx src/components/RecordControl.tsx src/components/replay/ChapterThumb.tsx src/components/replay/PlaybackCanvas.tsx src/components/replay/PlaybackGraph.tsx src/components/replay/PlaybackStage.tsx src/components/replay/PlaybackViz.tsx src/components/replay/ReplayView.css src/components/replay/ReplayView.tsx src/components/SettingsModal.css src/components/SettingsModal.tsx src/components/Toolbar.css src/components/Toolbar.tsx src/components/viz/BstWidget.tsx src/components/viz/CustomWidget.tsx src/components/viz/SortWidget.tsx src/components/viz/VisualizationView.css src/components/viz/VisualizationView.tsx src/components/viz/widgets.css src/components/Whiteboard.css src/components/Whiteboard.tsx src/lib/ai.ts src/lib/ask.ts src/lib/deepgram/live.ts src/lib/deepgram/useVoiceDictation.ts src/lib/drawblock.ts src/lib/graph.ts src/lib/providers.ts src/lib/recording/chapters.ts src/lib/recording/demoRecording.ts src/lib/recording/snapshot.ts src/lib/recording/store.ts src/lib/recording/types.ts src/lib/recording/useRecorder.ts src/lib/render.ts src/lib/storage.ts src/lib/supabase/client.ts src/lib/supabase/useAuth.ts src/lib/types.ts src/lib/viz/registry.ts src/lib/viz/types.ts src/main.tsx src/styles/global.css src/types/plotly.d.ts src/vite-env.d.ts supabase/schema_v2.sql supabase/schema.sql tokens.css tsconfig.json tsconfig.node.json tsconfig.node.tsbuildinfo vite.config.ts ``` ### Dependencies - package.json: @deepgram/sdk@^3.13.0, @opentelemetry/api@^1.9.1, @opentelemetry/exporter-trace-otlp-http@^0.219.0, @opentelemetry/resources@^2.8.0, @opentelemetry/sdk-trace-node@^2.8.0, @supabase/supabase-js@^2.108.2, @types/react@^18.3.12, @types/react-dom@^18.3.1, @vitejs/plugin-react@^4.3.4, concurrently@^9.1.0, cors@^2.8.5, dotenv@^16.4.7, express@^4.21.2, mathjs@^15.2.0, plotly.js-dist-min@^3.6.0, react@^18.3.1, react-dom@^18.3.1, redis@^6.0.0, typescript@^5.6.3, vite@^5.4.11, ws@^8.21.0 - requirements.txt: redisvl[sentence-transformers], sentence-transformers ### Recent commits (newest first) - Update architecture.md - Add architecture diagram and dev tracing workflow - Added Arize AI features - agent2 is not manually triggered - updated the README - managed to wire in Redis - Merge branch 'main' of https://github.com/maunguyengit/FerbAI - ready to merge - Merge 07-ask-recording: Students talk to the recording - ask: Talk button, Stop voice, natural math TTS, scrollable panel - checkpoint - ui: toggle the Tutor chat panel on/off (default on) - 07: students ask the recording (pause, ask, AI answers + draws in blue, resume) - Merge 06-deepgram: voice input, recording transcription, auto-chapters, playback chapters/search/captions - cleanup: Claude Code only, default Claude Sonnet 4.6 - deepgram: relay audio through the server instead of minting browser tokens - 06: Deepgram — voice input, recording transcription, auto-chapters, playback chapters/search/captions - Merge 05-auth-persistence: Supabase auth, recordings persistence, sharing, multi-window recording - Merge 04-recording-playback: recording & playback engine - recording: capture all content windows (board, graph, learn), not just board ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/architecture.md ```markdown # FerbAI Architecture ```mermaid flowchart LR User["Student / User"] <--> App["ChalkAI Web App
Whiteboard, Chat, Replay"] App <--> API["Local Backend API
Coordinates tutoring, memory, voice, tracing"] API <--> Agent1["Agent 1: AI Tutor
Explains, answers, draws, graphs"] Agent1 <--> Models["AI Models
Anthropic / OpenAI"] API --> Agent2["Agent 2: Evaluator
Reviews tutor quality"] Agent2 --> Memory["Redis Memory
Session history + feedback"] API <--> Memory Memory --> Agent1 API <--> Voice["Deepgram
Speech + transcription"] API <--> Storage["Supabase
Login + saved recordings"] API --> Arize["Arize
Observability + traces"] Agent1 --> App ``` ChalkAI has two AI roles: Agent 1 tutors the student, while Agent 2 reviews how well Agent 1 taught. Agent 2's feedback is saved in Redis memory, so future tutoring responses can improve based on past sessions. Most product paths are bidirectional because the app sends user/session data to the backend and receives streamed responses, recordings, transcripts, or saved state back. Arize is mostly one-way: the backend exports traces so the team can inspect what the AI is doing. ``` ### .agents/skills/redis-semantic-cache/SKILL.md ```markdown --- name: redis-semantic-cache description: Redis LangCache guidance for semantic caching of LLM responses on Redis Cloud — calling search/set via the SDK or REST API, tuning the similarity threshold, separating caches per task type, and filtering with custom attributes. Use when caching LLM completions or RAG answers to cut API cost and latency, building a cache-aside layer in front of OpenAI / Anthropic / etc., tuning hit rate vs precision, or splitting one app's LLM workloads into multiple LangCache caches. license: MIT metadata: author: Redis, Inc. version: "0.1.0" --- # Redis Semantic Cache Semantic caching for LLM responses with Redis Cloud's LangCache service. Stores prompts as embeddings; subsequent semantically-similar prompts return the cached response without re-calling the model. > LangCache is currently in **preview** on Redis Cloud. Features and behavior may change. ## When to apply - Wrapping an LLM call (OpenAI, Anthropic, etc.) with a cache layer to cut cost and latency. - Caching RAG answers, classification outputs, or any deterministic LLM workload. - Tuning the precision/hit-rate trade-off for a semantic cache. - Splitting one application's LLM workloads across multiple cache instances. ## 1. The cache-aside flow LangCache fits in front of any LLM call as a standard cache-aside pattern: 1. Send the user's prompt to LangCache's `search`. 2. **Cache hit** — return the stored response directly. 3. **Cache miss** — call the LLM, then `set` the response so future similar prompts hit. ```python from langcache import LangCache import os lang_cache = LangCache( server_url=f"https://{os.getenv('HOST')}", cache_id=os.getenv("CACHE_ID"), api_key=os.getenv("API_KEY"), ) result = lang_cache.search(prompt="What is Redis?", similarity_threshold=0.9) if result: response = result[0]["response"] else: response = llm.generate("What is Redis?") lang_cache.set(prompt="What is Redis?", response=response) ``` The same operations are available via REST (`POST /v1/caches/{cacheId}/entries/search` and `POST /v1/caches/{cacheId}/entries`) when an SDK isn't an option. See [references/langcache-usage.md](references/langcache-usage.md) for full SDK + REST samples and attribute-based storage. ## 2. Tune the similarity threshold The threshold controls how close (in embedding cosine distance) a new prompt must be to a cached one to count as a hit. Higher = stricter match, fewer false positives. Lower = more hits, more risk of returning an off-topic answer. | Threshold | Behavior | Use when | |---|---|---| | 0.95+ | Near-exact match required | Customer-facing answers where wrong responses are costly | | 0.9 | Balanced default | Most workloads — start here | | 0.8 | Loose semantic match | Internal tools, exploratory queries, FAQ deduplication | ```python # Stricter — fewer false positives result = lang_cache.search(prompt="What is Redis?", similarity_threshold=0.95) # Looser — higher hit rate result = lang_cache.sea [truncated — 1086 more characters] ``` ### requirements.txt ``` redisvl[sentence-transformers] sentence-transformers ``` ### package.json ``` { "name": "ferbai-whiteboard", "private": true, "version": "0.1.0", "type": "module", "scripts": { "dev": "concurrently --kill-others -n api,web -c blue,yellow \"node server/index.js\" \"node server/wait_for_api.mjs && vite\"", "dev:web": "vite", "dev:api": "node server/index.js", "build": "tsc -b && vite build", "embeddings:health": "python server/redisvl_embed.py --health", "mcp:agent2": "node server/agent2_mcp.js", "test:memory": "node server/memory.test.js", "preview": "vite preview", "start": "node server/index.js" }, "dependencies": { "@deepgram/sdk": "^3.13.0", "@opentelemetry/api": "^1.9.1", "@opentelemetry/exporter-trace-otlp-http": "^0.219.0", "@opentelemetry/resources": "^2.8.0", "@opentelemetry/sdk-trace-node": "^2.8.0", "@supabase/supabase-js": "^2.108.2", "cors": "^2.8.5", "dotenv": "^16.4.7", "express": "^4.21.2", "mathjs": "^15.2.0", "plotly.js-dist-min": "^3.6.0", "react": "^18.3.1", "react-dom": "^18.3.1", "redis": "^6.0.0", "ws": "^8.21.0" }, "devDependencies": { "@types/react": "^18.3.12", "@types/react-dom": "^18.3.1", "@vitejs/plugin-react": "^4.3.4", "concurrently": "^9.1.0", "typescript": "^5.6.3", "vite": "^5.4.11" } } ``` ### src/main.tsx ```typescript import { StrictMode } from 'react' import { createRoot } from 'react-dom/client' import App from './App' import './styles/global.css' createRoot(document.getElementById('root')!).render( , ) ``` ### src/App.tsx ```typescript import { useEffect, useRef, useState } from 'react' import Whiteboard from './components/Whiteboard' import Toolbar from './components/Toolbar' import GraphView from './components/GraphView' import VisualizationView from './components/viz/VisualizationView' import ReplayView from './components/replay/ReplayView' import RecordControl from './components/RecordControl' import ChatPanel from './components/ChatPanel' import ModelSelector from './components/ModelSelector' import SettingsModal from './components/SettingsModal' import AuthGate from './components/auth/AuthGate' import { useAuth } from './lib/supabase/useAuth' import { fetchProviderStatus } from './lib/ai' import { DEFAULT_SELECTION } from './lib/providers' import { catalogForPrompt } from './lib/viz/registry' import { useRecorder } from './lib/recording/useRecorder' import { DEMO_RECORDING } from './lib/recording/demoRecording' import { deleteRemote, fetchAudioUrl, getById, listMine, parseShareLink, saveRecording, setShared, shareLinkFor } from './lib/recording/store' import { generateChapters } from './lib/recording/chapters' import type { GraphEqSnap, Recording, Scene } from './lib/recording/types' import { getSelection, setSelection as persistSelection } from './lib/storage' import type { AIAction, AIGraphEquation, ChatContext, GraphHandle, Tool, View, VizHandle, VizSpec, WhiteboardHandle } from './lib/types' import './App.css' export default function App() { const wbRef = useRef(null) const graphRef = useRef(null) const vizRef = useRef(null) const [view, setView] = useState('board') const [tool, setTool] = useState('pen') const [color, setColor] = useState('oklch(27% 0.008 70)') const [width, setWidth] = useState(3) const [canUndo, setCanUndo] = useState(false) const [canRedo, setCanRedo] = useState(false) const [selection, setSelection] = useState(() => getSelection(DEFAULT_SELECTION)) const [settingsOpen, setSettingsOpen] = useState(false) const [keysVersion, setKeysVersion] = useState(0) const [ready, setReady] = useState(false) const [chatOpen, setChatOpen] = useState(true) const auth = useAuth() const recorder = useRecorder() useEffect(() => { if (!import.meta.env.DEV || import.meta.env.VITE_DEV_SHUTDOWN_ON_CLOSE === 'false') return const devApiTarget = import.meta.env.VITE_API_TARGET || 'http://localhost:8787' const shutdownUrl = `${devApiTarget}/api/dev/shutdown` const cancelUrl = `${devApiTarget}/api/dev/shutdown/cancel` const cancelPendingShutdown = () => { fetch(cancelUrl, { method: 'POST', keepalive: true }).catch(() => {}) } const cancelTimers = [ window.setTimeout(cancelPendingShutdown, 1000), window.setTimeout(cancelPendingShutdown, 3000), ] const scheduleShutdown = () => { const body = JSON.stringify({ delayMs: 5000 }) const blob = new Blob([body], { type: 'application/json' }) if (!navigator.sendBeacon(shutdownUrl, blob)) { fetch(shutdownUrl, { method: 'POST', body, headers: { 'content-type': 'application/json' }, keepalive: true, }).catch(() => {}) } } window.addEventListener('pagehide', scheduleShutdown) return () => { window.removeEventListener('pagehide', scheduleShutdown) cancelTimers.forEach((timer) => window.clearTimeout(timer)) } }, []) // cloud (DB) recordings for the signed-in user + any shared ones opened by link const [cloud, setCloud] = useState([]) const [opened, setOpened] = useState([]) const [selectId, setSelectId] = useState(null) // recording to auto-select in Replay // live content of each window, kept in refs so the recorder can snapshot the // whole scene (board + graph + learn) at any moment. const viewRef = useRef('board') const graphEqsRef = useRef([]) const vizSpecRef = useRef(null) viewRef.current = view const getScene = (): Scene => ({ view: viewRef.current, elements: wbRef.current?.getElements() ?? [], equations: graphEqsRef.current, viz: vizSpecRef.current, }) // record window switches + each window's content changes (while recording) const recordEvent = recorder.recordEvent // stable (useCallback) useEffect(() => { recordEvent({ type: 'view', view }) }, [view, recordEvent]) const onGraphEquations = (eqs: GraphEqSnap[]) => { graphEqsRef.current = eqs; recordEvent({ type: 'graph', equations: eqs }) } const onVizSpec = (spec: VizSpec | null) => { vizSpecRef.current = spec; recordEvent({ type: 'viz', spec }) } // load the user's saved recordings on sign-in useEffect(() => { if (auth.user) listMine(auth.user.id).then((recs) => { setCloud(recs); if (recs.length) setSelectId(recs[0].id) }) else { setCloud([]); setOpened([]) } }, [auth.user?.id, auth.user]) const startRecording = () => { recorder.start({ title: `Lesson ${(auth.user ? cloud.length : recorder.recordings.length) + 1}`, getScene }) } const [saveNotice, setSaveNotice] = useState(null) const stopRecording = async () => { const rec = await recorder.stop() // already added to recorder.recordings (shown immediately) setView('replay') if (rec?.transcript?.length) { rec.chapters = await generateChapters(rec.transcript) // auto-chapter from the transcript } if (rec && auth.user) { setSaveNotice(null) const saved = await saveRecording(rec, auth.user.id) if (saved) { setCloud((c) => [saved, ...c]) recorder.remove(rec.id) // drop the local copy now that the saved one is shown setSelectId(saved.id) } else { setSaveNotice("Couldn't save to your account — it's kept locally for now. Did you run the SQL migration in Supabase?") setSelectId(rec.id) } } else if (rec) { setSelectId(rec.id) } } // the [truncated — 8480 more characters] ``` ### server/index.js ```javascript import './instrumentation.mjs' import express from 'express' import cors from 'cors' import { createServer } from 'node:http' import { PROVIDERS, SYSTEM_PROMPT, ASK_SYSTEM_PROMPT, envKeyFor } from './providers.js' import { WebSocketServer } from 'ws' import { LiveTranscriptionEvents } from '@deepgram/sdk' import { supabaseAdmin, supabaseEnabled, AUDIO_BUCKET } from './supabase.js' import { deepgramEnabled, dgClient, LIVE_OPTIONS } from './deepgram.js' import { formatTutorReview, messagesToTranscript, scoreTutorTranscript } from './tutorReview.js' import { appendMemoryEvent, finalizeAgent1Session, getAgentMemoryPacket, getSessionReviewContext, warmEmbeddingCache, writeAgent2ReviewMemory, } from './memory.js' import { getTracingStatus, setSpanAttributes, shutdownTracing, truncate, withSpan } from './instrumentation.mjs' const app = express() const PORT = process.env.PORT || 8787 const CHAT_MEMORY_PACKET_TIMEOUT_MS = Number(process.env.CHAT_MEMORY_PACKET_TIMEOUT_MS || 1200) const DEV_SHUTDOWN_ENABLED = process.env.NODE_ENV !== 'production' && process.env.DEV_SHUTDOWN_ON_CLOSE !== 'false' let devShutdownTimer = null app.use(cors()) app.use(express.json({ limit: '12mb' })) // board PNGs can be large if (process.env.MEMORY_WARM_EMBEDDINGS_ON_START !== 'false') { warmEmbeddingCache() .then((ready) => { if (ready) console.info('[memory] local embedding worker warmed.') }) .catch((err) => console.warn(`[memory] local embedding warmup skipped: ${err?.message || err}`)) } // ---- Deepgram: configured check (the audio relay is a WebSocket, see below) ---- app.get('/api/deepgram/status', (_req, res) => res.json({ configured: deepgramEnabled })) // ---- Ask the recording: student paused a lesson + asked a question ---- app.post('/api/ask', async (req, res) => { const { image, transcriptWindow, question, sessionId, recordingId, activeView } = req.body || {} const apiKey = envKeyFor('claude-code') if (!apiKey) return res.status(503).json({ error: 'No Claude key configured.' }) const content = [] if (image) { try { const { mediaType, base64 } = splitDataUrl(image); content.push({ type: 'image', source: { type: 'base64', media_type: mediaType, data: base64 } }) } catch { /* skip image */ } } content.push({ type: 'text', text: askInputText(transcriptWindow, question), }) res.set({ 'content-type': 'text/event-stream', 'cache-control': 'no-cache', connection: 'keep-alive' }) res.flushHeaders?.() const abort = new AbortController() res.on('close', () => { if (!res.writableEnded) abort.abort() }) try { const streamed = await withSpan('agent1.ask_recording', { 'openinference.span.kind': 'LLM', 'attributes.openinference.span.kind': 'LLM', 'llm.model_name': 'claude-sonnet-4-6', 'attributes.llm.model_name': 'claude-sonnet-4-6', 'input.value': truncate(content.find((item) => item.type === 'text')?.text), 'attributes.input.value': truncate(content.find((item) => item.type === 'text')?.text), 'session.id': sessionId, 'recording.id': recordingId, 'board.active_view': activeView, 'board.used': !!image, 'board.image_attached': !!image, }, async (span) => { const upstream = await fetch('https://api.anthropic.com/v1/messages', { method: 'POST', signal: abort.signal, headers: { 'content-type': 'application/json', 'x-api-key': apiKey, 'anthropic-version': '2023-06-01' }, body: JSON.stringify({ model: 'claude-sonnet-4-6', max_tokens: 1024, system: ASK_SYSTEM_PROMPT, stream: true, messages: [{ role: 'user', content }] }), }) setSpanAttributes(span, { 'http.response.status_code': upstream.status }) if (!upstream.ok || !upstream.body) { const detail = await readError(upstream) setSpanAttributes(span, { 'output.value': `Claude ${upstream.status}: ${detail}`, 'attributes.output.value': `Claude ${upstream.status}: ${detail}`, }) res.write(`data: ${JSON.stringify({ error: `Claude ${upstream.status}: ${detail}` })}\n\n`) res.end() return false } let answerText = '' await pumpSSE(upstream.body, (data) => { if (data === '[DONE]') return try { const evt = JSON.parse(data) if (evt.type === 'content_block_delta' && evt.delta?.type === 'text_delta') { answerText += evt.delta.text res.write(`data: ${JSON.stringify({ t: evt.delta.text })}\n\n`) } else if (evt.type === 'error') { const message = evt.error?.message || 'stream error' setSpanAttributes(span, { 'llm.stream.error': message }) res.write(`data: ${JSON.stringify({ error: message })}\n\n`) } } catch { /* */ } }) setSpanAttributes(span, { 'output.value': truncate(answerText), 'attributes.output.value': truncate(answerText), }) return true }) if (!streamed) return res.write('data: [DONE]\n\n') res.end() } catch (e) { if (abort.signal.aborted) return res.end() res.write(`data: ${JSON.stringify({ error: e?.message || 'ask error' })}\n\n`) res.end() } }) // ---- Text-to-speech (Deepgram Speak) for the AI's spoken answer ---- app.post('/api/tts', async (req, res) => { if (!deepgramEnabled) return res.status(503).json({ error: 'TTS not configured.' }) const text = (req.body?.text || '').toString().slice(0, 1800).trim() if (!text) return res.status(400).json({ error: 'No text.' }) try { const r = await fetch('https://api.deepgram.com/v1/speak?model=aura-asteria-en&encoding=mp3', { method: 'POST', headers: { authorization: `Token ${process.env.DEEPGRAM_API_KEY}`, 'content-type': 'application/json' }, body: JSON.stringify({ text }), }) if (!r.ok || !r.body) { const t = await r.text().catch(() => ''); return res.status(500).json({ error: t.slice(0, 200) || 'tts failed' }) } res.s [truncated — 28592 more characters] ``` ### vite.config.ts ```typescript import { defineConfig } from 'vite' import react from '@vitejs/plugin-react' // https://vite.dev/config/ export default defineConfig({ plugins: [react()], server: { port: 5173, open: true, proxy: { // forward API calls (HTTP + the Deepgram audio WebSocket) to the backend '/api': { target: process.env.VITE_API_TARGET || 'http://localhost:8787', ws: true }, }, }, }) ``` ### index.html ```html ChalkAI · draw · learn
``` ### tokens.css ```css /* Hallmark · theme: Chalk (soft neo-brutalism) · tone: warm-tactile · anchor hue: terracotta ~42° * Warm cream paper, thin dark rules, rounded corners, clay + sage accents. * "Brutalism, but not too brutal": structure stays (visible borders, mono labels), * harshness softens (radius, thin rules, near-no hard shadows). * pre-emit critique: P5 H4 E5 S5 R5 V5 */ :root { /* ---- Paper / ink (warm cream) ---- */ --color-paper: oklch(91% 0.013 88); /* app field — warm cream */ --color-paper-2: oklch(95.5% 0.008 88); /* canvas + cards — lighter cream */ --color-paper-3: oklch(88.5% 0.014 88); /* hovers / sunk wells */ --color-ink: oklch(27% 0.008 70); /* warm near-black — text + borders */ --color-ink-soft: oklch(50% 0.012 72); /* secondary text / captions */ /* ---- Accents ---- */ --color-accent: oklch(61% 0.115 42); /* terracotta / clay — primary */ --color-accent-soft: oklch(75% 0.075 45); /* lighter clay for rings/hovers */ --color-accent-ink: oklch(97% 0.008 88); /* text on terracotta */ --color-sage: oklch(66% 0.05 150); /* sage green — secondary */ --color-red: oklch(56% 0.15 33); /* warm red — destructive */ --color-yellow: oklch(80% 0.11 90); --color-brown: oklch(54% 0.07 55); --color-navy: oklch(42% 0.06 260); --color-green: oklch(66% 0.05 150); /* alias → sage (status ok) */ /* ---- Functional ---- */ --color-focus: oklch(61% 0.115 42); --color-grid: oklch(86% 0.012 88); /* faint board grid */ /* ---- Type ---- */ --font-display: 'Bricolage Grotesque', system-ui, sans-serif; /* friendly grotesque */ --font-body: 'Inter', system-ui, sans-serif; --font-mono: 'JetBrains Mono', ui-monospace, monospace; --font-hand: 'Caveat', 'Bricolage Grotesque', cursive; /* chalk / AI writing */ --text-xs: 0.72rem; --text-sm: 0.84rem; --text-base: 0.95rem; --text-lg: 1.1rem; --text-xl: 1.4rem; --text-2xl: 1.85rem; --text-3xl: 2.5rem; /* ---- 4-pt spacing scale ---- */ --space-2xs: 4px; --space-xs: 8px; --space-sm: 12px; --space-md: 16px; --space-lg: 24px; --space-xl: 36px; --space-2xl: 56px; /* ---- Soft structure (the "not too brutal" part) ---- */ --rule: 2px; --rule-thick: 3px; --radius-sm: 8px; --radius: 12px; --radius-lg: 16px; --radius-pill: 999px; --shadow-soft: 0 1px 2px oklch(27% 0.008 70 / 0.10); --shadow-card: 0 2px 10px oklch(27% 0.008 70 / 0.07); --shadow-lift: 0 6px 22px oklch(27% 0.008 70 / 0.13); /* ---- Motion ---- */ --dur-fast: 110ms; --dur-base: 180ms; --ease-out: cubic-bezier(0.2, 0.7, 0.3, 1); --ease-in: cubic-bezier(0.6, 0, 0.8, 0.3); --ease-in-out: cubic-bezier(0.5, 0, 0.2, 1); } ``` [70 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]