# Project export: Mimic This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Every browser agent pays an LLM to re-reason the task on every run. Mimic learns it once, then replays deterministically. Same job: $1.55 vs $0.06. Teach once, run for pennies. - Devpost: https://devpost.com/software/mimic-show-don-t-code - GitHub: https://github.com/ish-cs/ai-hackathon - Video: https://www.youtube.com/embed/MY-9uJDWCEg?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — ish-cs (40 commits), Duan, Cheng Kai (15 commits), SJ Janolkar (11 commits) ## Devpost submission (written by the team) ### Overview Mimic is browser automation you build by showing it, not coding it. Do a task once, and Mimic does it forever, for pennies. Today's AI browser agents are capable, but they start every run from scratch. Ask one to repeat a task and it re-reads the page and re-reasons each step as if it had never seen the job before, paying an LLM to think it through again on every run. Do the job a thousand times and you pay to figure it out a thousand times. Mimic watches you do it once, learns the workflow, and then replays it deterministically, with zero LLM calls. The only time it stops to think again is when the website actually changes underneath it, so the cost stays flat whether you run it once or a million times. We proved it by racing Mimic against a state-of-the-art agent on the same outreach job. The other agent spent \$1.55. Mimic spent \$0.06, about 25x cheaper, and faster too. We are upfront about the limit, though: sending a thousand messages still means a thousand clicks, so the total time still grows with the work. We cannot beat physics. What collapses is the thinking cost, not the doing. And when a page changes and a step breaks, Mimic does not fall over. It works out what the missing element was meant to do, finds it again, fixes itself, and remembers the fix so it never breaks there twice. ### Inspiration We are Berkeley students, and like most students, we lose hours of every week to tedious web busywork: cold emails, LinkedIn outreach, pasting the same details into form after form to chase internships, recruiters, and the occasional reply. It is repetitive, it is nearly identical every time, and it is exactly the kind of work a computer should be doing for us. So why isn't it? Because every existing fix is flawed in its own way: Traditional automation (UiPath, Zapier) needs a developer and weeks of setup, then breaks the moment a button moves. The new AI browser agents will do the task, but they improvise it from scratch on every run. That makes them both unreliable and surprisingly expensive: you are paying an LLM to relearn the same job over and over. Nobody builds the obvious thing: working out a repetitive task should be a one-time cost, not a tax you pay on every run. Learn it once by watching a human, then run it forever for almost nothing, with no code and no config, and it should not collapse the first time a site redesigns a page. We wanted to put that in the hands of people who have neither a budget nor a CS degree, starting with ourselves. To test it safely, without spamming real people or risking banned accounts, we rebuilt the task we actually dread as a sandbox: LinkedUp, a stand-in for LinkedIn we could automate freely. ### What it does The idea in one line: you do a task once, and Mimic learns to do it for you. Concretely, you show it the job inside Mimic's browser: read a lead from a spreadsheet, open the messaging site, type a personalized note, hit send. Mimic records every move. Then Anthropic's Claude (Opus 4.8) reads that recording and turns it into a clean, reusable workflow, working out which parts are the variables that change every time (the person's name, their role) and which are the fixed steps that never do (click Send). Press run, and it works through a whole spreadsheet of new people on its own. The Cost Race is the demo. A claim like "it's cheaper" means nothing without proof, so we made the savings impossible to miss. We put Mimic head to head with Stagehand (a top-tier AI agent from Browserbase) on the identical job: go down a list of leads, message each one on LinkedUp, come back, repeat. A live meter between them counts real tokens and real dollars, pulled from each side's actual API usage, nothing estimated and nothing faked. Stagehand re-reads and re-reasons every step for every lead, and its meter climbs the entire time. Mimic replays what it already learned: zero LLM calls, zero tokens on a clean run. Same three-lead job: \$1.55 versus \$0.06 (about 25x cheaper), and 132 seconds versus 32. Then we break it, live. Mid-race, we rename the Send button to Send to, the kind of small change that breaks normal automation. A brittle script would fail immediately. Stagehand handles it, but only because it is already re-reading the entire page on every step, which is exactly why it is so expensive. Mimic hits the broken button and, for the first and only time in the run, stops to think: Anthropic's Claude looks at the page, asks which element is the button that sends this message, finds it by meaning instead of the old broken selector, patches the step, finishes the lead, and writes the fix back so it never breaks there again. Cheap because it is deterministic, trustworthy because it heals, and you watch all of it happen live, on one screen. Design and UX A demo can be technically strong and still fail to land if nobody can see what makes it strong. The parts that matter most, the cost gap, the deterministic replay, the self-healing, all happen invisibly, in milliseconds, inside a backend. So we treated the interface as the argument, not a footnote to it. It all fits on one screen. Two agents run side by side, each driving a real cloud browser you can watch, with a live cost-and-token meter and a per-lane timer between them. You do not read a claim about savings, you watch one counter race ahead while the other barely moves. There is no code anywhere. Teaching Mimic is just using a browser. Someone who has never written a line of code records a workflow by doing the task once: no scripts, no selector editors, no settings to configure. The heal is the centerpiece. The moment we break the page, the UI tells the story in real time: the failure, Mimic working out the fix, and the step running again. A recovery that would normally be a buried error log becomes the high point of the demo. None of it is pre-recorded. Every number on screen is streamed live, step by step, over WebSockets, straight from the real run. ### How we built it Under the hood it is a simple loop, record, structure, replay, heal, with a memory layer holding it together. Record. Playwright captures every click, keystroke, and page change, and snapshots the context around each one (the DOM, the element's attributes, what it semantically is), even across multiple browser tabs. Structure. Anthropic's Claude (Opus 4.8, with structured JSON output) reads that raw recording and turns it into a clean, parameterized workflow, separating the variables from the fixed actions. Store. The workflow and its full version history live in Redis as genuine agent memory: not a cache, but a permanent, auditable record the healer reads from and writes back to. Replay. The runtime runs the saved workflow against new data, unattended, streaming every step to the UI over WebSockets, inside a real cloud browser (Browserbase). Heal. When a step fails, Anthropic's Claude re-grounds the element from its stored intent, patches it, retries, and saves the upgraded workflow back to Redis as a new version. Sentry captures the full failure-and-recovery moment, because failure is our product, so we instrument it closely. Why the cost collapses. A normal agent re-reasons every step on every run, so its token cost grows in step with how many times you run it: a thousand runs, a thousand bills. Mimic pays the thinking cost once, up front, to learn the task, then replays it with no LLM involved at all. The only time it spends a token again is when the website itself changes and a step needs healing, and that depends on how often the site changes, not on how many times you run. So the cost stays roughly flat no matter the volume, and the cost per run trends toward zero. The honest caveat, which we care about: a thousand tasks is still a thousand rounds of clicking and typing, so the total time still grows with the work. We cannot beat physics. What flattens is the thinking, not the doing. In short, a normal agent re-interprets the task from scratch every run, while Mimic compiles it once and runs the result. We split the build across one clean contract: the Brain (Anthropic's Claude handling intent-extraction, structuring, and healing, pure logic that owns the Redis schema) and the Hands and Face (the browser engine, record and replay, the Cost Race, and the live heal visualization). A single shared types file was the only seam the three of us had to agree on, which is the main reason we did not trip over each other. ### Challenges we ran into The healer is not allowed to guess. A confident wrong heal is worse than no heal at all: it clicks the wrong button and submits bad data. So we hardened the prompt and wrote a dedicated anti-hallucination test: when nothing on the page matches what the step is trying to do, the healer must return "could not heal" rather than invent a plausible-looking selector. It passes. The meter had to be honest. A made-up cost number would have undermined the whole pitch. So we estimate nothing: we read each side's actual token usage (Mimic's heal tokens from Anthropic's reported usage, Stagehand's from its SDK's per-step totals) and price both at list rate. The \$0.06 really is near-zero, and the \$1.55 really is what the agent spent. The demo could only work once. In our first version, the healing lane wrote its fix back to shared memory, so on the next run the control lane read the already-healed workflow and survived too, erasing the contrast. The fix: both lanes always start from the pristine original. The heal still accrues in the memory trail, but the race stays repeatable. We had to keep our own selector broken on purpose. The structuring step kept "helpfully" upgrading our deliberately brittle selector into a robust one, which meant our staged break would not break anything, and nothing would heal. We had to force structuring to preserve the recorded selector exactly as-is. The healer is the only part of the system allowed to improve it. ### Accomplishments we're proud of The whole loop is real, end to end. A real recorded trace goes to Anthropic, into Redis, hits a real break, gets a real heal, and the fix is written back, all verified against the live Anthropic API and a live cloud Redis, running in a real cloud browser via Browserbase. Nothing is stubbed. The cost claim is measured, not asserted. \$1.55 versus \$0.06 on an identical job, taken from both sides' real API usage. We made an invisible system legible. A live side-by-side cost-and-token meter, two real cloud browsers running in parallel, and a real-time heal visualization, every value streamed step by step over WebSockets rather than pre-baked. The healer is accurate and cautious. It re-grounds a renamed or moved control by intent with high confidence, and it correctly refuses when there is nothing valid to click. Four sponsor technologies are load-bearing, not bolted on. Anthropic (Claude Opus 4.8) is the brain, Redis is the agent memory, Browserbase runs the live cloud browsers for both lanes (and makes Stagehand, the agent we benchmark against), and Sentry catches the exact failure that triggers a heal. ### What we learned Self-healing is only as trustworthy as its willingness to fail loudly. The "refuse to guess" behavior mattered more than the healing itself; a heal you cannot trust is a liability, not a feature. The win is on cost, not runtime, and that distinction is the whole pitch. Caching the reasoning stops your per-run token cost from growing with the number of runs; it does not make the actual clicking and typing any less work. Knowing exactly which axis you are improving keeps the claim honest. For a systems demo, the UI is the argument. The cost gap and the self-heal are real in the backend, but they only land because a judge can watch them happen live, on one screen, in real browsers. "Agent memory" should mean an auditable, versioned trail of what changed and why, not a glorified key-value cache. Redis modeled that cleanly. ### What's next More kinds of healing: not just renamed or moved buttons, but reordered flows, multi-page tasks, and login walls. Scheduling, so a workflow can run on its own, unattended. A savings dashboard showing the cumulative dollars each workflow has saved versus a reasoning agent. A shared library where non-coders can publish and fork each other's workflows. Confidence thresholds, so when the healer is unsure, it hands off to a human instead of guessing. ## README (from the GitHub repository) # Mimic — Show, Don't Code Teach-by-demonstration web automation with a self-healing core. Do a tedious web task once; it learns the workflow, replays it on new data, and **repairs itself when the site changes** instead of breaking. > Product name `Mimic` is a placeholder — swap freely. Strategy: [DOCS/PROJECT.md](./DOCS/PROJECT.md). Data contract: [DOCS/CONTRACT.md](./DOCS/CONTRACT.md). Full docs in [DOCS/](./DOCS). ## Locked design decisions (hour-1 calls, already made) 1. **Transport: single Node process.** Brain/Runtime/Web are modules; the runtime serves the UI and streams events over WebSocket. (Band rooms are a later swap — same JSON shapes, see CONTRACT.md.) 2. **Engine: local Playwright Chromium.** The user demonstrates in a real local browser they can click. Browserbase is an optional replay-only stretch (banks that prize) — left out of the critical path because a cloud browser adds live-demo network risk. 3. **Replay is dataset-driven.** Tab A is the human's teach prop; replay fills the target form from provided `DataRow`s. No source-scraping, so no `extract` action — healing only concerns the write side, which matches the demo. 4. **The "dead" control agent = our own replay with healing disabled** (`heal: false`). More reliable on stage than a third-party agent, and honest: "the same automation without our healing layer." 5. **Model: `claude-opus-4-8`** (one constant in `brain/anthropic.ts` — swap to Haiku if heal latency hurts the live demo). ## Layout ``` shared/ types.ts (the CONTRACT made code) + fixtures brain/ structure() · heal() · Redis store · Anthropic client ← Ishaan runtime/ recorder (Playwright capture) · player (replay+heal) · Express+WS server ← ck (Hands) web/ zero-build UI: live event feed + split-screen heal-vs-crash ← ck (Face) ``` ## Setup ```bash npm install npm run browsers # one-time: installs Playwright Chromium cp .env.example .env # fill ANTHROPIC_API_KEY + REDIS_URL (sponsor starter packs) npm run typecheck # should pass clean npm run dev # runtime + UI on http://localhost:3000 ``` Local Redis for dev: `docker run -p 6379:6379 redis` (or use the Redis Cloud URL from the sponsor pack). ## The seam Everything crossing a module boundary is a type in `shared/types.ts`. The flow: ``` recorder → RawTrace → structure() → Workflow → Redis │ POST /replay → player.replay(Workflow, DataRow, {heal}) │ per step → StepResult (→ WS → web) on failure → heal(HealRequest) → HealResult → retry + write back ``` ## Status Scaffold: the seam, stubs, and wiring compile and run end-to-end in shape. The bodies marked `TODO(brain)` / `TODO(hands)` are where the real logic goes — start with the hour-0–4 crude end-to-end (record → replay one task), then make the heal bulletproof. See PROJECT.md build order. ## Detected evidence (automated analysis) Indexed codebase: 84 recognized source files, 583 KB. - Anthropic (technology) — detected in the code - Express (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Redis (technology) — detected in the code - TypeScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code - React (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (93 of 93) ``` .env.example .gitignore brain-test.ts brain/anthropic.ts brain/heal.ts brain/store.ts brain/structure.ts browserbase-cdp-capture-test.ts browserbase-cdp-probe.ts browserbase-cleanup.ts browserbase-e2e-test.ts browserbase-e2e-v2-test.ts browserbase-record-test.ts browserbase-record-v2-test.ts browserbase-run-test.ts browserbase-smoke.ts browserbase-verify-url.ts browserbase-verify-v2.ts capture-trace.ts disable-vercel-protection.ts DOCS/AI Hackathon 2026 Hacker Guide.md DOCS/BROWSERBASE.md DOCS/CHECKPOINTS.md DOCS/CK-plan.md DOCS/CONTRACT.md DOCS/DEMO-V2.md DOCS/DESIGN.md DOCS/DEVPOST.md DOCS/ISHGOON-plan.md DOCS/plans/2026-06-20-cost-race-ishaan-brief.md DOCS/plans/2026-06-20-cp3-heal-viz.md DOCS/plans/2026-06-21-landing-cost-race-integration.md DOCS/PROJECT.md DOCS/specs/2026-06-20-cost-race-design.md DOCS/specs/2026-06-20-cp3-heal-viz-design.md DOCS/STAGE.md fullchain-test.ts login-burner.ts mimic-break-verify.ts mock-public/.gitignore mock-public/index.html mock-public/leads.js mock-public/leadsheet.html mock-public/linkedin.html multitab-test.ts negative-test.ts package.json race-verify.ts README.md record-parody.ts record-sheet.ts redis-cloud-test.ts repeat-test.ts runtime/breaker.ts runtime/browser.ts runtime/metrics.ts runtime/mimic-lane.ts runtime/multitab.ts runtime/player.ts runtime/recorder-capture.js runtime/recorder.ts runtime/sentry.ts runtime/server.ts runtime/stagehand-lane.ts seam-test.ts seed-demo.ts shared/fixtures/real-trace.json shared/fixtures/sample-trace.json shared/fixtures/sample-workflow.json shared/types.ts smoke.ts stagehand-lane-smoke.ts stagehand-live-test.ts stagehand-multitab-test.ts stagehand-resume-test.ts stagehand-spike.ts store-invariant-test.ts submit-code.ts tools/extract-design.mjs tools/memory-inspect.ts tsconfig.json verify-leads.ts verify-login.ts verify-windows.ts warm-contexts.ts web/app.js web/classic.html web/index.html web/live.js web/memory.html web/mock.html web/race.html web/race.js ``` ### Dependencies - package.json: @anthropic-ai/sdk@^0.105.0, @browserbasehq/sdk@^2.14.1, @browserbasehq/stagehand@^3.6.0, @sentry/node@^10.59.0, @types/express@^5.0.6, @types/node@^26.0.0, @types/ws@^8.18.1, express@^5.2.1, playwright@^1.61.0, redis@^6.0.0, tsx@^4.22.4, typescript@^6.0.3, ws@^8.21.0, zod@^4.4.3 ### Recent commits (newest first) - Landing: lead with token-cost story, de-AI copy, contain hero video - Cost Race: fix break round on Stagehand + persist break/heal through round 4 - Cost Race: operator-stepped demo on LinkedUp, break + heal on both lanes - race.html: add per-lane stopwatch markup so Start button works - Integrate ishgoon cost-race backend into the landing page (Tasks 1-2) - Sheet-driven multi-tab Cost Race: both lanes work the LeadSheet → LinkedUp - Landing: link "The demo" nav + add cost-race CTA to race.html - Add DESIGN.md: landing design system handoff for the Cost Race demo - Merge remote-tracking branch 'origin/ishgoon' into frontend-landing-page - Merge remote-tracking branch 'origin/ishgoon' into ishgoon - Merge ck's LinkedUp People-search rebuild into Cost Race wiring - Wire Cost Race to LinkedUp parody + scripted recorder/verify harness - Scale LinkedUp + LeadSheet to 100 leads from a shared dataset - Add Status column to LeadSheet lead tracker - Rebuild LinkedUp mock as People search-results list; fix v2 smoke selectors - Merge ishgoon: cost-race engine (MimicLane/StagehandLane/breaker) + Browserbase recorder - Landing page redesign: problem/how/demo/get nav, 3-step How-it-works section, drop dead style-hover attrs - Add SJ hand-off brief: integrate cost race into landing page - Merge remote-tracking branch 'origin/ishgoon' into ishgoon - Add live LinkedIn proof for Stagehand lane (warmed context) ## Key source files (fetched from GitHub, selected and truncated for size) ### DOCS/DESIGN.md ```markdown # Mimic Design System — Handoff for the Cost Race demo This is the design language of the landing page (`web/index.html`). Apply it to `web/race.html` so the demo reads as the same product. Pull tokens from here; do not guess. The mock target sites (`mock-public/*`) are **out of scope** — leave them looking like real third-party sites. ## 1. Tokens ### Color | Role | Value | |---|---| | Base background | `#050504` | | Page gradient (optional) | `radial-gradient(150% 90% at 50% 0%,#16140f 0%,#0a0a08 42%,#050504 100%) fixed` | | Demo/section background | `#000` | | Primary ink (brand cream) | `#E1E0CC` | | Dim ink | `#cfcbb2`, `#9b988b`, `#6f6e63` | | Hairline / borders | `rgba(225,224,204,.10)` (also `.08`, `.18`) | | Muted text | `rgba(225,224,204,.45)` / `.35` / `.6` / `.78` | | Accent — alert/danger | `#ff6b6b` (lighter `#ff9b9b`) | | Accent — success | `#7fd3a0` | | Surfaces / cards | `#0c0c0a`, `#1c1c19` | ### Typography - **Body / UI / labels:** `'Almarai', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif` — weights 300 / 400 / 700. - **Display / headlines:** `'Instrument Serif', serif`, usually `font-style:italic`, color `#cfcbb2` or `#E1E0CC` for emphasis spans. - **Data / numbers (live cost counters):** `ui-monospace, Menlo, monospace`. - Fonts are self-hosted as `.woff2` in `web/assets/` (UUID filenames). Easiest path for the demo: load `Almarai` (300,400,700) + `Instrument Serif` from Google Fonts; the landing already `preconnect`s to `fonts.googleapis.com` / `fonts.gstatic.com`. - Responsive sizing uses `clamp()`, e.g. headline `clamp(28px,4.4vw,58px)`, body `clamp(14px,1.2vw,17px)`. ### Shape & motion - Pill buttons: `border-radius:999px`. - Signature easing (use everywhere for reveals/transitions): `cubic-bezier(.16,1,.3,1)`. - Reveal pattern: start `opacity:0;transform:translateY(20px)`, transition `opacity .9s` + `transform .9s` with the easing above. ## 2. Components ### CTA / primary button ```html Label → ``` On `:hover`, widen the gap (e.g. `gap:14px`) using **real CSS `:hover`** — the landing's `style-hover` attributes are dead/inert; do not copy them. ### Section heading Cream label kicker (uppercase, letter-spaced, 11px) above an `Instrument Serif` italic headline. Center-aligned, generous vertical padding `clamp(60px,9vh,120px)`. ## 3. Voice / feel Minimal, warm-on-near-black, editorial. Serif italic carries emotional lines; sans carries function; mono carries numbers. Lots of negative space. ## 4. race.html → landing mapping Swap the demo's current dev vars for landing tokens. **Keep `--stage` and `--mimic` as functional lane colors** — they encode the two competitors and must stay distinct from each other and from the brand red [truncated — 615 more characters] ``` ### DOCS/DEMO-V2.md ```markdown # Demo v2 — multi-tab LinkedIn outreach (impressive demo upgrade) > Replace the single self-coded form with a **two-app, tab-switching** workflow that looks like real > SaaS: read a lead from a Sheets-like page → switch tab → message them on a LinkedIn-like page → > Send. Break the Send button → control dies, Mimic heals. Runs over N rows = N personalized messages. > Approved 2026-06-20. ## Why Current demo = one homemade form. Underwhelming. This shows the agent **navigating between two apps like a human** (the real pain: data trapped between systems that don't talk) while keeping the **self-heal kill-shot** — the differentiator. Maps to Toolbox (workflow automation) + Anthropic (economic opportunity: outreach for recruiters/sales). ## HARD REQUIREMENT (gates everything): tab-switching The demo is impossible without it, and it does not exist today. Build it first. **Contract (`shared/types.ts`)** - Add `"switchTab"` to `ActionType`. - Every `WorkflowStep` (and `RawTrace` action) carries `tab: number` (which tab it runs on, default `0`). - A `switchTab` step's target = the destination tab (by index; carry its `url` too so replay can open it if missing). **Recorder (`runtime/recorder.ts`)** - Subscribe to `context.on("page")` to detect new tabs/popups; keep an ordered tab list + the active tab. - Emit a `switchTab` action whenever focus moves to a different tab (or a click opens one). - Tag every captured action with its `tab` index. - The injected capture script stays per-page (it already re-injects post-goto for Browserbase CDP) — attach it to each new page too. **Player (`runtime/player.ts`)** - Hold `pages: Page[]` (or a map by tab index), not a single `page`. - Route each step to its `step.tab`. `switchTab` → bring that tab to front (open it via `context.newPage()` + goto if it doesn't exist yet). - Heal runs on the **active** tab (the heal logic itself shouldn't need changes — it re-grounds against whatever DOM is live). **Browserbase (`runtime/browser.ts`)** - Each tab has its own live-view: `bb.sessions.debug(id).pages[i].debuggerFullscreenUrl`. On a `switchTab`, emit a fresh `liveview` event for the active tab so the iframe follows the active tab. **Backward compatibility** - Default `tab: 0` everywhere → the **current single-page demo and saved workflows still replay unchanged.** This is required (the old demo is our fallback). ## The two clone pages (`mock-public/`, deployed to Vercel, public + breakable) **Page A — "LeadSheet" (Sheets-like):** a polished table of leads, columns `name · role · company`, 3–5 rows. Clean, stable selectors (`data-cell="name"` etc.). Looks like Google Sheets. **Page B — "LinkedIn-like":** search bar → person profile → **Message** button → compose textarea → **Send** button. Polished to read as LinkedIn. Clean selectors (`#li-search`, `#li-message-btn`, `#li-compose`, `#li-send`). - **Breakable:** `?break=1` renames/re-ids the **Send** button (e.g. `#li-send` → `#li-submit`, label "Send" → "S [truncated — 2738 more characters] ``` ### package.json ``` { "name": "mimic", "version": "0.1.0", "private": true, "type": "module", "description": "Show, Don't Code — teach-by-demonstration web automation with a self-healing core.", "scripts": { "dev": "tsx watch --env-file=.env runtime/server.ts", "start": "tsx --env-file=.env runtime/server.ts", "typecheck": "tsc --noEmit", "browsers": "playwright install chromium", "memory": "tsx --env-file=.env tools/memory-inspect.ts" }, "dependencies": { "@anthropic-ai/sdk": "^0.105.0", "@browserbasehq/sdk": "^2.14.1", "@browserbasehq/stagehand": "^3.6.0", "@sentry/node": "^10.59.0", "express": "^5.2.1", "playwright": "^1.61.0", "redis": "^6.0.0", "ws": "^8.21.0", "zod": "^4.4.3" }, "devDependencies": { "@types/express": "^5.0.6", "@types/node": "^26.0.0", "@types/ws": "^8.18.1", "tsx": "^4.22.4", "typescript": "^6.0.3" } } ``` ### web/app.js ```javascript // Zero-build UI. Consumes RunEvent (shared/types.ts) over WebSocket and drives the split-screen. const $ = (id) => document.getElementById(id); const log = (m) => { $("log").textContent += m + "\n"; $("log").scrollTop = 1e9; }; const ws = new WebSocket(`ws://${location.host}`); ws.onopen = () => log("[ws] connected"); ws.onmessage = (e) => handle(JSON.parse(e.data)); function laneEls(lane) { const key = lane === "control" ? "control" : "healing"; return { steps: $(`${key}-steps`), status: $(`${key}-status`) }; } const lastFailed = {}; // lane -> { stepId, selector } — the selector that missed, for the heal card function ring(confidence) { const pct = Math.round(confidence * 100); const R = 16, C = 2 * Math.PI * R; const NS = "http://www.w3.org/2000/svg"; const svg = document.createElementNS(NS, "svg"); svg.setAttribute("viewBox", "0 0 40 40"); svg.classList.add("ring"); svg.innerHTML = `` + `` + `${pct}%`; svg.dataset.offset = String(C * (1 - confidence)); return svg; } function healCard(result, oldSelector) { const card = document.createElement("div"); card.className = result.healed ? "heal-card" : "heal-card refused"; const head = document.createElement("div"); head.className = "hc-head"; head.textContent = result.healed ? "⚡ RE-GROUNDED by intent" : "HELD BACK — refused to guess"; card.appendChild(head); if (result.healed) { if (oldSelector) { const oldEl = document.createElement("div"); oldEl.className = "hc-old"; oldEl.textContent = oldSelector; card.appendChild(oldEl); const arrow = document.createElement("div"); arrow.className = "hc-arrow"; arrow.textContent = "↓"; card.appendChild(arrow); } const newEl = document.createElement("div"); newEl.className = "hc-new"; newEl.textContent = result.newSelector; card.appendChild(newEl); card.appendChild(ring(result.confidence)); } const reason = document.createElement("div"); reason.className = "hc-reason"; reason.textContent = result.reasoning; card.appendChild(reason); requestAnimationFrame(() => { card.classList.add("reveal"); const fg = card.querySelector(".ring-fg"); const svg = card.querySelector(".ring"); if (fg && svg) fg.style.strokeDashoffset = svg.dataset.offset; }); return card; } function setLaneState(lane, state) { const key = lane === "control" ? "control" : "healing"; const laneEl = document.querySelector(`.lane.${key}`); const status = $(`${key}-status`); laneEl.classList.remove("crashed", "completed"); laneEl.classList.add(state); status.textContent = state === "completed" ? "✓ COMPLETED" : "💀 CRASHED"; status.className = `status big ${state}`; } function handle(ev) { log(`[${ev.lane ?? "-"}] ${ev.kind} ${ev.result ? JSON.stringify(ev.result).slice(0, 120) : ""}`); if (ev.kind === "run_start") { const rf = $("record-frame"); if (rf) { rf.innerHTML = ""; rf.style.display = "none"; } // teach frame done const { steps, status } = laneEls(ev.lane); steps.innerHTML = ""; status.textContent = `running — row ${JSON.stringify(ev.row)}`; delete lastFailed[ev.lane]; const laneEl = document.querySelector(`.lane.${ev.lane === "control" ? "control" : "healing"}`); laneEl.classList.remove("crashed", "completed"); status.className = "status"; } else if (ev.kind === "step") { const { steps } = laneEls(ev.lane); const r = ev.result; const row = document.createElement("div"); row.className = `step ${r.status}`; const left = document.createElement("span"); left.textContent = `${r.stepId} · ${r.status.toUpperCase()}`; const right = document.createElement("span"); right.className = "s"; right.textContent = r.attemptedSelector + (r.error ? " — " + r.error.slice(0, 40) : ""); row.append(left, right); steps.appendChild(row); if (r.status === "failed") lastFailed[ev.lane] = { stepId: r.stepId, selector: r.attemptedSelector }; } else if (ev.kind === "heal") { const { steps } = laneEls(ev.lane); const old = lastFailed[ev.lane] ? lastFailed[ev.lane].selector : null; steps.appendChild(healCard(ev.result, old)); } else if (ev.kind === "run_done") { setLaneState(ev.lane, ev.ok ? "completed" : "crashed"); } else if (ev.kind === "liveview") { renderLiveView(ev.lane, ev.url); } } function makeFrame(url, interactive) { const f = document.createElement("iframe"); f.className = "liveview"; f.src = url; f.setAttribute("sandbox", "allow-same-origin allow-scripts"); f.setAttribute("allow", "clipboard-read; clipboard-write"); if (!interactive) f.style.pointerEvents = "none"; // replay lanes read-only; record is read/write return f; } // ENGINE=browserbase only: embed each cloud session's live view. control→left lane, healing→right // lane (read-only); record→centered frame (read/write, so the user can teach inside it). function renderLiveView(lane, url) { if (lane === "record") { const host = $("record-frame"); host.innerHTML = '
● Recording — demonstrate the task in the live browser below, then “Stop & build”.
'; host.appendChild(makeFrame(url, true)); host.style.display = "block"; return; } const laneEl = document.querySelector(`.lane.${lane === "control" ? "control" : "healing"}`); laneEl.querySelector(".liveview")?.remove(); // replace any prior run's frame laneEl.querySelector("h2").insertAdjacentElement("afterend", makeFrame(url, false)); } // Browserbase posts this when a session ends — note it; it is NOT a heal failure. window.addEventListener("message", (e) => { if (e.data === "browserbase-disconnected") log("[liveview] a Browserbas [truncated — 2790 more characters] ``` ### runtime/server.ts ```typescript import { createServer } from "node:http"; import { fileURLToPath } from "node:url"; import { dirname, join } from "node:path"; import express from "express"; import { WebSocketServer, WebSocket } from "ws"; import type { DataRow, RawTrace, RunEvent, WorkflowStep } from "../shared/types"; import { structure } from "../brain/structure"; import { lastUsage } from "../brain/anthropic"; import { saveWorkflow, getWorkflow, listWorkflows, getHistory, saveTrace } from "../brain/store"; import { Recorder } from "./recorder"; import { replay, closeLiveBrowsers } from "./player"; import { stripSwitchTabs, applyTabs, breakForDemo, mergeHeal } from "./multitab"; import { StagehandLane } from "./stagehand-lane"; import { MimicLane } from "./mimic-lane"; import { cost } from "./metrics"; import { initSentry } from "./sentry"; initSentry(); const app = express(); app.use(express.json({ limit: "5mb" })); const __dirname = dirname(fileURLToPath(import.meta.url)); const webDir = join(__dirname, "..", "web"); app.use(express.static(webDir)); app.get("/mock", (_req, res) => res.sendFile(join(webDir, "mock.html"))); const recorder = new Recorder(); // ---- WebSocket event bus (Runtime → Web live feed) ---- const server = createServer(app); const wss = new WebSocketServer({ server }); const clients = new Set(); wss.on("connection", (ws) => { clients.add(ws); ws.on("close", () => clients.delete(ws)); }); function broadcast(event: RunEvent): void { const msg = JSON.stringify(event); for (const ws of clients) if (ws.readyState === WebSocket.OPEN) ws.send(msg); } // ---- Record ---- app.post("/api/record/start", async (req, res) => { try { // Record against the URL the UI asks for (e.g. the LeadSheet), falling back to MOCK_URL then the // local mock. Recording is local, so any public/local URL is reachable. const url = req.body.url ?? process.env.MOCK_URL ?? "http://localhost:3000/mock"; const { liveViewUrl } = await recorder.start(url); // Browserbase: push the record live-view to the UI so the user can teach inside the iframe. // Local: liveViewUrl is undefined → no event → the OS record window is used as before. if (liveViewUrl) broadcast({ kind: "liveview", lane: "record", url: liveViewUrl }); res.json({ recording: true, liveViewUrl }); } catch (e) { res.status(500).json({ error: (e as Error).message }); } }); app.post("/api/record/stop", async (req, res) => { try { const trace = await recorder.stop(req.body.task ?? "untitled task"); await saveTrace(trace); res.json(trace); } catch (e) { res.status(500).json({ error: (e as Error).message }); } }); // ---- Structure + store ---- app.post("/api/workflows", async (req, res) => { try { const trace = req.body as RawTrace; // Structure WITHOUT the switchTab actions (brain stays untouched), then stamp per-step tabs and // re-insert the switchTab steps in the runtime so the player can route a multi-tab replay. const wf = await structure(stripSwitchTabs(trace)); // Capture the structure() call's REAL token usage = Mimic's one-time "teaching" cost, and stamp it // on the workflow so /api/race can show it as run 0 (the meter's elevated starting point). lastUsage // is the most-recent completeJSON usage; structure() is the only model call between here and now. const tabbed = applyTabs(wf, trace) as typeof wf & { teachingTokensIn?: number; teachingTokensOut?: number }; tabbed.teachingTokensIn = lastUsage.tokensIn; tabbed.teachingTokensOut = lastUsage.tokensOut; await saveWorkflow(tabbed); res.json(tabbed); } catch (e) { res.status(500).json({ error: (e as Error).message }); } }); app.get("/api/workflows", async (_req, res) => { res.json(await listWorkflows()); }); app.get("/api/workflows/:id", async (req, res) => { const wf = await getWorkflow(req.params.id); if (!wf) return res.status(404).json({ error: "not found" }); res.json(wf); }); // Agent-memory trail: every version ever saved to Redis (workflow:{id}:history). Feeds the memory panel. app.get("/api/workflows/:id/history", async (req, res) => { res.json(await getHistory(req.params.id)); }); // ---- Replay: runs BOTH lanes (control dies | healing survives) for the split-screen kill shot ---- // Both lanes start from the PRISTINE original (version 1, from history), deep-cloned, EVERY run. // Why this matters: the healing lane writes its re-grounded selector back to Redis (the agent-memory // trail). If the control lane re-read the LIVE workflow, then after the first heal it would inherit // that cured selector and stop crashing — the split-screen would only work once. Reading v1 keeps // control brittle so the kill shot is repeatable; heal write-backs still accrue in history. app.post("/api/replay", async (req, res) => { const { workflowId, row, breakSite } = req.body as { workflowId: string; row: DataRow; breakSite?: boolean }; const history = await getHistory(workflowId); const pristine = history.length ? history[0].wf : await getWorkflow(workflowId); if (!pristine) return res.status(404).json({ error: "workflow not found" }); await closeLiveBrowsers(); // reap the PREVIOUS run's lingering result windows before opening this run's let control = structuredClone(pristine); // never healed — always hits the brittle selector let healing = structuredClone(pristine); // re-grounds live every run; write-back logs to history healing.version = history.length; // onHeal bumps this → monotonic v2, v3, … in the memory trail // ENGINE=browserbase: a SINGLE-PAGE workflow recorded against the operator's localhost can't be // reached by the cloud browser, so swap a localhost startUrl for the public MOCK_URL. A multi-tab // workflow's startUrl is its own page (e.g. the public LeadSheet) — never replace it with the mock. const isMultiTab = pristine.steps.some((s) => s.action === "switchTab"); const isLocalHost = /^https?:\/\/(localhost|127\.0\.0\. [truncated — 11543 more characters] ``` ### browserbase-cleanup.ts ```typescript // Releases all RUNNING Browserbase sessions — frees the concurrency budget when lingering demo // sessions (or leaked test sessions) pile up against the cap. Safe to run anytime between demos. // Run: tsx --env-file=.env browserbase-cleanup.ts import Browserbase from "@browserbasehq/sdk"; const apiKey = process.env.BROWSERBASE_API_KEY; if (!apiKey) throw new Error("BROWSERBASE_API_KEY unset"); const bb = new Browserbase({ apiKey }); const running = await bb.sessions.list({ status: "RUNNING" }); console.log(`running sessions: ${running.length}`); for (const s of running) { try { await bb.sessions.update(s.id, { projectId: s.projectId, status: "REQUEST_RELEASE" }); console.log(` released ${s.id}`); } catch (e) { console.log(` release failed ${s.id}: ${(e as Error).message}`); } } console.log("done"); process.exit(0); ``` ### redis-cloud-test.ts ```typescript // Cloud Redis reachability check — proves the shared sponsor DB works before we both point at it. import { readFileSync } from "node:fs"; import { createClient } from "redis"; for (const line of readFileSync(new URL("./.env", import.meta.url), "utf8").split("\n")) { const m = line.match(/^\s*([A-Z_]+)\s*=\s*(.*?)\s*$/); if (m && !process.env[m[1]]) process.env[m[1]] = m[2]; } const url = process.env.REDIS_URL ?? ""; console.log(`\n── connecting to ${url.replace(/:[^:@]+@/, ":****@")} ──`); const client = createClient({ url }); client.on("error", (e) => console.error(" redis error:", e.message)); const t0 = Date.now(); await client.connect(); console.log(` ✓ connected in ${Date.now() - t0}ms`); console.log(` PING → ${await client.ping()}`); await client.set("mimic:smoke", "alive"); console.log(` SET/GET round-trip → ${await client.get("mimic:smoke")}`); await client.del("mimic:smoke"); await client.quit(); console.log(`\n══ CLOUD REDIS: PASS ✅ — shared DB reachable, both teammates can point here ══\n`); ``` ### verify-windows.ts ```typescript // Visual check for the demo windows: launch the two lanes at their real positions, inject the // REAL label + verdict helpers from player.ts, and screenshot each PAGE (captures injected DOM // regardless of how the OS stacks windows). control → LEFT + red FAILED, healing → RIGHT + green // SUCCEEDED. Server must be running so /mock loads. Run: tsx --env-file=.env verify-windows.ts import { chromium } from "playwright"; import { injectLaneLabel, markLaneResult } from "./runtime/player"; const URL = "http://localhost:3000/mock"; async function shot(lane: "control" | "healing", x: number, ok: boolean, file: string): Promise { const browser = await chromium.launch({ headless: false, args: [`--window-position=${x},40`, "--window-size=780,960"], }); const page = await browser.newPage({ viewport: null }); await page.goto(URL, { timeout: 10000 }); await injectLaneLabel(page, lane); await markLaneResult(page, ok); // simulate the end-of-run verdict await page.screenshot({ path: file }); console.log(`${lane}: window @ x=${x}, verdict=${ok ? "SUCCEEDED" : "FAILED"} → ${file}`); await browser.close(); } await shot("control", 24, false, "/tmp/lane-control.png"); // LEFT, red await shot("healing", 824, true, "/tmp/lane-healing.png"); // RIGHT, green console.log("done"); process.exit(0); ``` ### seed-demo.ts ```typescript // Seeds the shared cloud DB with ONE clean demo workflow whose startUrl is machine-independent // (http://localhost:3000/mock, served by each teammate's own server) so EITHER machine can replay // it — unlike a file:// path. Adds one real heal so the memory panel has a trail to show. // Run: tsx --env-file=.env seed-demo.ts (server must be running so /mock is reachable) import { readFileSync } from "node:fs"; import { structure } from "./brain/structure"; import { saveWorkflow } from "./brain/store"; import { replay } from "./runtime/player"; import type { RawTrace, DataRow, RunEvent } from "./shared/types"; const trace: RawTrace = JSON.parse(readFileSync(new URL("./shared/fixtures/real-trace.json", import.meta.url), "utf8")); const url = "http://localhost:3000/mock"; const wf = await structure(trace); wf.startUrl = url; await saveWorkflow(wf); console.log(`seeded ${wf.workflowId} startUrl=${wf.startUrl}`); // One heal on the broken page → memory trail = v1 original + healed v2. const row: DataRow = { customerName: "Globex Corporation", customerEmail: "ap@globex.com" }; const events: RunEvent[] = []; const ok = await replay({ ...wf, startUrl: url + "?break=1" }, row, { heal: true, lane: "healing", emit: (e) => events.push(e), onHeal: async (w) => { w.version += 1; await saveWorkflow(w); }, }); console.log(`healing replay ok=${ok} heals=${events.filter((e) => e.kind === "heal").length}`); process.exit(0); ``` ### stagehand-lane-smoke.ts ```typescript // Smoke: StagehandLane class end-to-end (open → runRound → metrics event → close) on a benign page, // before /api/race wires it for real. Run: npx tsx --env-file=.env stagehand-lane-smoke.ts import { StagehandLane } from "./runtime/stagehand-lane"; import type { RunEvent } from "./shared/types"; async function main(): Promise { const lane = new StagehandLane({ startUrl: "https://news.ycombinator.com/", instruction: "Tell me the title of the top story on this page.", stealth: false, // benign page → no stealth/proxy/Context needed for the smoke proxies: false, maxSteps: 6, }); const events: RunEvent[] = []; const emit = (e: RunEvent): void => { events.push(e); console.log(` emit ${e.kind}` + (e.kind === "metrics" ? ` in=${e.tokensIn} out=${e.tokensOut} $${e.costUsd.toFixed(4)}` : "")); }; const { liveViewUrl } = await lane.open(); console.log(`[smoke] open OK · live=${liveViewUrl ? "yes" : "no"}`); const r = await lane.runRound({ name: "Ada" }, 1, emit); await lane.close(); const gotMetrics = events.some((e) => e.kind === "metrics"); const pass = r.ok && r.tokensIn > 0 && gotMetrics; console.log(`[smoke] ok=${r.ok} in=${r.tokensIn} out=${r.tokensOut} ms=${r.ms}`); console.log(pass ? "LANE ✅ class works: open→runRound→metrics→close, real tokens" : "LANE ❌ inspect above"); process.exit(pass ? 0 : 1); } main().catch((e) => { console.error("[smoke] CRASHED:", e); process.exit(1); }); ``` [72 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]