FerbAI — Brutalist Whiteboard + AI Tutor

A whiteboard on the left (draw, like Excalidraw), an AI study companion on the right that sees your board and guides you to the next step. Built with React + TypeScript + Vite. Brutalist theme: thick black rules, hard offset shadows, electric-yellow accent.

Run it

npm install
cp .env.example .env     # then paste whatever keys you have (all optional)
npm run dev              # starts the Vite frontend AND the backend proxy together

Open http://localhost:5173. npm run dev runs both processes via concurrently (web = Vite on 5173, api = proxy on 8787). Vite forwards /api/* to the proxy, so there's nothing else to configure.

You don't strictly need a .env — you can paste keys into the in-app Settings (⚙) panel instead (see below).

Use it

1. Draw your problem on the left — pen, eraser, lasso (select + drag + Delete), text, box, oval. Pick colour + stroke width. Ctrl+Z / Ctrl+Shift+Z undo/redo. ⤓ PNG downloads the board.
2. Pick a model in the dropdown (top-right). Vision-capable models can read the board snapshot — look for the 👁 sees board chip.
3. Add your API key. Two ways, your choice:
   • Backend .env (recommended) — keys never touch the browser. The status chip shows ● key set automatically.
   • Settings (⚙) — paste a key in the UI. It's stored in your browser's localStorage and forwarded to the local proxy per request, overriding the .env key. Leave a field blank to fall back to the server key.
4. Ask →. With attach board snapshot on, the AI gets a PNG of your drawing and nudges you toward the next step (Socratic — not the full answer).

Models

The chat runs on Claude Code (Anthropic). Pick the model in the top-bar dropdown — Claude Sonnet 4.6 (default) · Claude Opus 4.8 · Claude Haiku 4.5. The Base URL is editable in Settings. Models live in src/lib/providers.ts.

Record & replay a lesson (▶ Replay)

Hit ● Record (top bar) to capture a lesson, then Stop. Everything you do on the board — strokes, text, shapes, erases, undo/redo, and the AI's annotations — is logged as timestamped events; the microphone is captured in parallel via MediaRecorder (optional — if there's no mic, the lesson records silently). Open the ▶ Replay tab to watch it back.

The engine (src/lib/recording/): a recording is an incremental event log (for smooth playback) + periodic snapshots (full board state every 30s, for instant seeking). Playback is a requestAnimationFrame loop driven by the audio clock (audio.currentTime) — or a virtual clock when silent — applying events as their timestamps pass. Seeking to time T finds the nearest snapshot ≤ T and fast-forwards events from there, so scrubbing is instant regardless of lesson length. The live board and the playback canvas share one renderer (src/lib/render.ts) so replays are pixel-identical.

Recordings are in-memory for the session (audio as an object URL). Persistence (IndexedDB) and Deepgram live transcription are later phases.

Learn window — interactive lessons the AI builds (not videos)

Toggle the left panel to ◆ Learn. Ask the tutor to teach a concept and it builds something you step through, edit, and explore — active learning, not a wall of text.

• "Teach me binary search trees" → an interactive BST: insert / find / remove / traverse, stepping through every comparison, editable values.
• "Teach me selection sort with 9, 3, 7, 1, 5" → an animated sorter with play / step / speed and an editable array.
• "Teach me how a stack works" → a custom push/pop visualizer.

The architecture (fighting AI generation error). The AI does not generate interactive code from scratch — that's where LLMs break. Instead:

1. Tier 1 — reusable widgets. A registry of pre-built, tested, fully interactive widgets (src/components/viz/) owns all the hard parts (stepping, animation, drag, editing). The AI emits only a tiny spec — which widget + initial data + narration — via a ferbai-viz JSON block. Near-zero generation risk. The widget catalog (src/lib/viz/registry.ts) is fed to the AI's prompt so it knows what to reuse.
2. Tier 2 — sandboxed custom. When no widget fits, the AI emits self-contained HTML in a separate ferbai-html block (raw HTML in its own fence — no JSON escaping to corrupt), rendered in a sandboxed iframe (allow-scripts, no same-origin) so it can't touch the app.

Adding a new widget is one registry entry + one component — the AI can use it immediately. Starter chips let you explore the built-ins without asking the AI.

Graph window (Desmos-style 2D + 3D) the AI can plot on

Toggle the left panel between ✎ Board and ∿ Graph (top bar). The graph window takes typed equations and the AI can plot onto it too.

• Type equations in the side rail: explicit y = x^2 + 9, surfaces z = x^2 - y^2, or implicit relations x^2 + y^2 + z^2 = 9. Anything using z renders in 3D (drag to rotate, scroll to zoom); otherwise 2D. Click a color dot to show/hide, double-click to recolor, × to delete.
• The AI plots too. In the graph view, ask things like "graph the derivative and integral of y = x³ + 3x²" or "plot a cubic that intersects this parabola". The model does the calculus itself and emits a ferbai-graph block of equations, which the app plots (marked ✦ as AI-added). It sees the current graph (snapshot + equation list) so it builds on what's there.

Built on Plotly.js (2D lines/contours, 3D surfaces/isosurfaces) + mathjs (parsing/evaluation). The equation engine lives in src/lib/graph.ts; the panel in GraphView.tsx.

Note: bundling Plotly makes the production JS large (~5.7 MB / 1.7 MB gzip). Fine for local use; code-split it (lazy-load the graph) before shipping wide.

The AI writes ON the board (not just chat)

FerbAI's tutor doesn't only talk in the sidebar — it writes the next step onto the whiteboard, in the empty space, like a teacher at a board. Draw 3x = 12, ask "solve for x", and it writes 3x / 3 = 12 / 3 in blue directly below your work.

How it stays accurate (instead of guessing pixel positions blind):

1. The app tells the model the geometry. Every request includes the board size and the bounding box of your existing strokes (getBoardMeta), so the model knows exactly where the empty space is.
2. The model returns structured draw commands in a fenced ferbai-draw JSON block — text, arrow, line, rect, ellipse (circle the answer), and highlight (mark a mistake on your work). The block is parsed out of the chat text and rendered as real, undoable board elements in AI-blue, with a fade-in (drawblock.ts, applyAIActions).
3. The "✎ AI draws on board" toggle (on by default) injects a firm directive so the model draws whenever you want it — turn it off for pure chat.

Use a vision model (look for 👁 sees board) so it can read your actual handwriting; the geometry hints help non-vision models place text too. Anything the AI draws is just board elements — undo it, erase it, or move it like your own.

How the backend helps

All model calls go through a thin local proxy (server/index.js) instead of straight from the browser. That buys you three things:

• No CORS pain. The browser only ever calls same-origin /api/*. The proxy makes the real provider call server-side, where CORS doesn't apply.
• Keys can stay off the client. Put them in .env and the browser never sees them.
• One normalized stream. Anthropic and OpenAI shapes are unified into a single SSE format (data: {"t": "…"}), so the frontend has one tiny parser.

The proxy does not persist API keys — .env keys are read at start, UI-pasted keys are used for that one request and discarded. Tutoring memory is persisted separately when Redis is configured, as described below.

Two-agent memory and review loop

FerbAI now keeps a lightweight two-agent tutoring loop:

1. Agent 1 — the tutor. This is the chat model that sees the board/graph/learn context, answers the student, and can emit ferbai-draw, ferbai-graph, or ferbai-viz blocks for the app to render. Every turn is written as a memory event (user_message_received, assistant_response_started, assistant_response_completed, tool results, and failures).
2. Agent 2 — the reviewer. The Review button runs a local heuristic tutor evaluator (server/tutorReview.js) over the transcript, board state, and lesson goal. It scores engagement, diagnosis, scaffolding, board grounding, tone, and goal alignment, then writes its risks and recommendations back into memory as Agent 2 review summaries. Weak rubric dimensions are always surfaced as risks, even when the average score is high. End Session also triggers Agent 2 on the backend when the session contains any substantive student turn; greetings and closings like "hi" or "thanks" are skipped so empty sessions do not produce noisy reviews. The backend awaits the Agent 2 memory write before finalizing the session.

Before each Agent 1 response, the backend builds a memory packet for the current session (getAgentMemoryPacket): recent events, the current session summary, Agent 2 guidance, and semantically similar prior sessions for that same user. Agent 2 risks and recommendations are injected into the latest Agent 1 prompt as advisory coaching context, so the tutor can adapt without exposing the memory block to the student. Live chat has a small semantic-memory time budget; if embeddings/vector search are cold, Agent 1 falls back to fast session memory instead of waiting on the spinner.

Agent 2 MCP

Agent 2 is also available as a local stdio MCP server:

npm run mcp:agent2

It exposes three tools:

• agent2.review_transcript — score an explicit transcript and optionally write the review into FerbAI memory.
• agent2.review_agent1_session — load Agent 1 events for a FerbAI sessionId, run the Agent 2 review, and persist the guidance.
• arize_evaluator.agent1_session — prepare or trigger an Arize evaluator task for Agent 1 session traces. This pairs with FerbAI's backend OpenTelemetry / OpenInference tracing so Agent 1 session spans can be evaluated in Arize.

Arize AX tracing

The backend initializes tracing from server/instrumentation.mjs, which is imported first by server/index.js. It uses OpenTelemetry with a BatchSpanProcessor and exports OTLP traces to Arize AX when credentials are present.

Set these in .env to export traces:

ARIZE_SPACE_ID=...
ARIZE_API_KEY=...
ARIZE_PROJECT_NAME=ferbai-tutor
# optional override; defaults to Arize's OTLP traces endpoint
ARIZE_COLLECTOR_ENDPOINT=https://otlp.arize.com/v1/traces

If ARIZE_SPACE_ID or ARIZE_API_KEY is missing, the backend logs that the exporter is disabled and keeps running normally.

Current span model:

• agent1.chat — LLM span around the main /api/chat tutor response, with capped input.value, output.value, provider/model, session/user/request IDs, active view, board/image metadata, memory context status, and HTTP status.
• agent1.ask_recording — LLM span around the Anthropic paused-recording Q&A call in /api/ask, with capped input.value, output.value, model name, session/recording IDs, and board metadata.
• agent1.generate_chapters — LLM span around /api/chapters, with capped transcript prompt and parsed chapter output.
• agent2.tutor_review — CHAIN span around scoreTutorTranscript, with verdict, average score, per-dimension rubric scores/labels, evidence counts, and board grounding metadata.
• tutoring_session — CHAIN span when a session is ended through /api/memory/session/:sessionId/end, with session ID, recording ID, lesson goal, Agent 2 auto-review status, and memory summary output. When Agent 2 is triggered by session end, agent2.tutor_review is a child span of tutoring_session.

Look for traces under the ARIZE_PROJECT_NAME project, defaulting to ferbai-tutor.

You can check the no-secret tracing configuration at /api/tracing/status. The backend flushes the OpenTelemetry provider on SIGINT/SIGTERM so recent batch-processed spans are not dropped during local shutdown.

In local dev, closing the browser tab schedules /api/dev/shutdown with a short delay; a refresh cancels it on the next page load. The shutdown endpoint is local-only, disabled when NODE_ENV=production, and can be disabled in dev with DEV_SHUTDOWN_ON_CLOSE=false or VITE_DEV_SHUTDOWN_ON_CLOSE=false.

Redis memory, vector search, and embedding cache

FerbAI can recall similar past tutoring sessions by storing session events, session state, review summaries, and session-summary embeddings in Redis. With Redis Stack enabled, summary embeddings are also written to a vector index for semantic retrieval. The default local setup uses RedisVL with Hugging Face Sentence Transformers, so there are no embedding API costs:

pip install -r requirements.txt

Set these in .env:

MEMORY_EMBEDDINGS_BACKEND=redisvl-hf
MEMORY_HF_MODEL=sentence-transformers/all-MiniLM-L6-v2
MEMORY_EMBEDDING_TIMEOUT_MS=120000
MEMORY_EMBEDDING_WORKER_START_TIMEOUT_MS=120000
MEMORY_EMBEDDING_CACHE_TTL_SECONDS=604800
MEMORY_EMBEDDING_MEMORY_CACHE_LIMIT=500
MEMORY_WARM_EMBEDDINGS_ON_START=true
CHAT_MEMORY_PACKET_TIMEOUT_MS=1200
PYTHON_BIN=python

The first embedding call downloads the model to your machine and can be slow on CPU. After that, the backend keeps a warm Python embedding worker alive so session finalization and memory lookup reuse the loaded model instead of spawning Python for every request. Embeddings are cached by text hash in-process and, when Redis is configured, under a TTL-backed Redis key before being stored as 384-dim FLOAT32 cosine vectors in Redis Stack, e.g. ferbai_summary_vector_idx_384.

Redis keys are scoped by session and user:

• ferbai:memory:session:<sessionId>:events — recent raw memory events.
• ferbai:memory:session:<sessionId>:state — current session summary, status, preferences, entities, decisions, and Agent 2 reviews.
• ferbai:memory:user:<userId>:summary_embeddings — per-user semantic summary records used for fallback cosine search.
• ferbai:vector:summary:<dims>:<recordId> — Redis Stack vector documents, filtered by userId before vector KNN search.
• ferbai:memory:embedding_cache:<sha256> — cached embeddings keyed by backend, model, and normalized text.

Generated tool blocks are scrubbed before memory summaries are created. That keeps ferbai-draw, ferbai-graph, ferbai-viz, raw ferbai-html, <script> fragments, and generated DOM code out of "Recent focus" and semantic summary embeddings. Closing turns like "I'm good thank you" are not used as the next recommended step.

If local embeddings fail, FerbAI still saves normal session events and summaries. You can optionally configure EMBEDDINGS_BASE_URL, EMBEDDINGS_API_KEY, and EMBEDDINGS_MODEL as an OpenAI-compatible fallback.

Verification

Useful local checks:

npm run embeddings:health   # verifies the local HF embedding worker/model
npm run test:memory         # exercises session memory and Agent 2 guidance
npm run build               # TypeScript + production Vite build

The current live E2E path has been verified with Redis Stack and the warm local HF worker enabled:

• MEMORY_EMBEDDINGS_BACKEND=redisvl-hf
• MEMORY_VECTOR_BACKEND=redis-stack
• Redis responded with PONG.
• Redis Search had ferbai_summary_vector_idx_384.
• The HF worker loaded sentence-transformers/all-MiniLM-L6-v2 and returned 384-dimensional embeddings.
• Session finalization wrote Redis Stack vector docs.
• A later session packet retrieved a prior semantically similar session through user-filtered vector search.
• Session end auto-triggered Agent 2 for substantive sessions, wrote Agent 2 guidance to memory, and exported both tutoring_session and agent2.tutor_review spans to Arize in the same trace.
• A prompt-capture test verified Agent 2 actually affects Agent 1: the next Agent 1 model request included both Agent 2 risks and recommendations in the injected memory context.
• A chat speed check verified that cold semantic memory lookup falls back to fast session memory instead of blocking the first streamed tutor response.
• A multi-user sequence user1 -> user2 -> user1 -> user3 -> user2 retrieved only each user's own prior session:
  • user1's second query matched user1's first session.
  • user2's second query matched user2's first session.
  • user1 did not retrieve user2 memory, and user2 did not retrieve user1 memory.

CORS / providers

The OpenAI-compatible groups (DeepSeek, OpenCode Go) have editable Base URLs in Settings, so you can point them at any gateway (OpenRouter, a self-hosted proxy, the real OpenCode Go endpoint). Adjust the model IDs in server/providers.js and src/lib/providers.ts to match your gateway.

Structure

server/
  index.js                 Express API: chat, recording Q&A, chapters, memory, review, health
  instrumentation.mjs      OpenTelemetry / Arize AX tracing setup
  memory.js                session memory, Agent 2 guidance, Redis vector retrieval, embedding cache
  tutorReview.js           local Agent 2 tutor rubric scorer
  agent2_mcp.js            stdio MCP server for Agent 2 review/evaluator tools
  embedding_worker.py      warm local HF embedding worker
  redisvl_embed.py         RedisVL/HF embedding health helper
  providers.js             server-side model catalog + system prompt + env-key mapping
src/
  App.tsx                  layout: board (left) + chat (right)
  components/
    Whiteboard.tsx         canvas drawing engine + tools + undo/redo + PNG export
    Toolbar.tsx            tools · colours · widths · undo/redo/clear/download
    ChatPanel.tsx          streaming chat, model selector, vision toggle, clear
    ModelSelector.tsx      grouped provider → model dropdown
    SettingsModal.tsx      per-provider API key + base URL (optional override)
  lib/
    providers.ts           provider + model catalog (UI)
    ai.ts                  talks to the proxy, parses the normalized stream
    storage.ts             localStorage for optional keys / urls / selection
    types.ts               shared types
tokens.css                 portable Hallmark Brutal design tokens
.env.example               backend keys template → copy to .env