# Project export: Shepherd

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: UC Berkeley AI Hackathon 2026
- Tagline: Mission control for AI agents
- Devpost: https://devpost.com/software/the-shepherd
- GitHub: https://github.com/KarthikSubramanian07/shepherd
- Video: https://www.youtube.com/embed/RQPCj6lHS98?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 7 GitHub contributor(s) — Leon Liu (102 commits), Devin AI (86 commits), Karthik Subramanian (73 commits), Claude Sonnet 4.6 (69 commits), Ben Huang (43 commits), Rohan Kumar (14 commits), Cursor (2 commits)

## Devpost submission (written by the team)

### Overview

Mission control for AI agents. The agent is the part you cannot trust. Shepherd is the system that lets you run it anyway. An AI agent is clicking through real software right now. Shepherd is the reason you can let it: it watches every move, halts the dangerous one before it happens, and signs a tamper-proof record of the rest. 💡

### Inspiration

We built an agent that could drive a desktop. Then we watched it do something we never asked it to, and we froze. Not because of the mistake. Because of the silence after it. There was no rewind, no record, no way to answer the only question that mattered: what did it just do, and why? The cursor had moved on its own, and the most honest thing we could say was a shrug. That shrug is the entire reason agents are stuck in demos. The capability curve went vertical and the trust curve never moved. Every month a model drives a computer better, and every month the gap widens between "the agent can do this" and "I would let it do this unwatched." The thing keeping agents out of the rooms where they would matter most, a clinic, a finance back office, a benefits desk, was never capability. It is that a black box touching a real machine is a liability nobody will sign. So we stopped trying to build a smarter sheep and built the shepherd. Every wave of computing earned a control plane the moment it touched things that mattered: networks got the firewall, the cloud got IAM, code got review. Agents that click around real machines are the next wave, and they have nothing standing watch. A shepherd does not cage the flock or walk every step beside it. It knows where the cliffs are, runs out front, and steps in at the exact moment before something goes over. That is the whole job, and nobody had built it. 🎛️

### What it does

Shepherd is local mission control for AI desktop agents. You hand it work two ways: show it once and the demonstration becomes a reusable routine, or just say the goal out loud and let it figure the rest out. Then it drives the real desktop to get it done while the same watch stays on the entire time. We did not polish one scripted trick. We ran it across a deliberately messy spread to prove it generalizes: filling a job application and stopping cold at the credential field, drafting a candidate decision email and refusing to send it to an outside address with a secret in the body, taking a Photo Booth selfie hands-free, opening a YouTube video and putting on lo-fi, hunting down the cheapest flight to NYC and starting the booking, reading and pulling structured data off a live web page, and running an agent on a second machine across the network that we steered mid-task. A macro recorder cannot do that. The proof that this is an agent and not a script is that the same watch falls on a job nobody wrote down as it does on one we demonstrated. Open-ended capability, still on a leash. The watch is the part that matters, and it tightens as the stakes rise. A rule engine reads every screen in under a millisecond. The risky ones get a second, independent opinion from a separate model. The genuinely fraught ones go to a room full of specialists who argue it out. And anything still uncertain stops and asks a human, out loud, before it touches anything it cannot take back. Every move, including every stop, becomes a link in a tamper-evident chain you can verify in one click. Nothing the agent does is ever a shrug again. 🎬 Watch it happen You say "send the candidate decision email." Your voice becomes text, the intent resolves, and the agent opens the composer and starts typing on the real screen. The draft is thin, so a durable research worker quietly looks the candidate up on the live web and fills in the body. The agent reaches for Send. In under a millisecond the rule engine sees it: external recipient, secret in the body. HALT, a beat before the irreversible click. The Control Hub blooms with the milestone graph, the council's votes, the signed intent token, and the live screen, and the agent speaks the danger out loud and waits. You say "stop." It stops, exactly at the boundary, and the halt itself is signed into the chain. Then you change one line of policy.yaml, run it again, and the behavior changes instantly. That last part is the difference between a demo and a product. 🧱 The anatomy of the watch Shepherd is one system, not a coat of logos. Every layer below is load-bearing: pull one and a real capability goes dark. We built each in its intended shape and exercised every one live, never stubbed, never a screenshot of a logo. And the watch is not a black box itself. The crystallized task graph is a measured object: we treat the runs of a task as a Markov process over milestones, recover the single most-likely path with Viterbi, report each fork's Shannon entropy in bits so you can see exactly where judgment lives versus rote, score the structure with McCabe cyclomatic complexity, and lay it all out with the Sugiyama algorithm so a graph with real shape reads like one. That is the difference between a tool that says "trust me" and one that shows its work. 🔧

### How we built it

The trick that made all of this fit together is one rule we refused to break: the click path is sacred. Nothing networked, async, or model-driven runs inside a step sequence, so a slow API or a stalled model can never strand the mouse mid-action. Everything else hangs off a single event bus, and every layer subscribes to it at the boundaries between steps. That one decision is why we could bolt a dozen systems onto a live agent without ever making it slower or shakier, and why the whole thing still runs offline with nothing configured: each layer degrades to a clean no-op when its service is gone. FastAPI underneath, a Next.js 16 Control Hub streaming live over WebSocket on top, and the sacred path in the middle. Observability & Reliability — Arize Phoenix + Sentry A desktop agent that clicks real buttons on your real machine is only trustworthy if you can see what it did and catch it when it goes wrong. We treat observability as a first-class feature, not an afterthought — and we used Arize Phoenix and Sentry as the two halves of that story: Phoenix to understand every decision, Sentry to catch every failure. Arize Phoenix — the agent's glass box Every run is a trace, and every step nests inside it, so we can replay an entire autonomous run decision-by-decision. We capture three layers for each turn: the planning step that records the prompt, the model's reasoning, and the action it chose — so you can see why the agent clicked where it did; the grounding step that turns a described target into precise screen coordinates; and the actuation step that fires the real click or keystroke. Because the grounding decisions are traced, we can pinpoint exactly which screenshot the model misread when a click lands in the wrong place. We also run evals on the agent's oversight verdicts, scoring its self-reported success against what the screen actually shows — turning "it said done" into a measurable signal. Sentry — reliability for an agent, not just an app We mapped Sentry's app-monitoring primitives onto agent behavior: Runs are performance transactions, with a child span per milestone, so latency regressions and slow steps surface like any production workload. Halts and human interventions become structured, queryable issues — tagged with the decision, trigger, and verifier verdict, with the halt screenshot attached. Silent failures get caught: a run that "completes" but actually stalled is reported as a dedicated event, so the failures that usually stay invisible become a ticket. Our favorite touch — Agent Session Replay. Every agent turn records the screenshot it saw, its reasoning, and the coordinates it clicked into a rolling buffer. The moment a run fails or is halted, Sentry doesn't just get a stack trace — it gets a frame-by-frame visual filmstrip of what the agent saw and where it clicked leading up to the failure, plus a manifest stitching reasoning to coordinates to outcome. It's Session Replay, but for a screen-driving agent — something Sentry was never designed for, mapped onto it perfectly. The two, connected Every Sentry issue carries a clickable link straight to the matching Phoenix trace. So the workflow is: Sentry pages you that a run halted → one click jumps to the full Phoenix trace → you replay every reasoning step that led there. Sentry turns the mystery into an alert; Phoenix turns the alert into a root cause. 🧗

### Challenges we ran into

The platforms fought us, and that was the fun part. Several behaved nothing like their docs (an auth header that had to be X-API-Key not Bearer, a token field secretly named jwt_token, a session API shaped differently than advertised), and we verified every integration against the live service instead of trusting a README. A second package root quietly broke the Next.js build until we traced it to dueling lockfiles. And in the most on-brand bug of the weekend, a late review caught our own fleet view reporting a halted agent as "completed." A safety product silently passing off a stopped agent as a success is precisely the failure we exist to prevent, so we made the status fail safe and wrote the regression test that should have existed. Four humans and a coding agent, branches landing fast, main staying green on every push because CI, the audit log, and a pre-ship review pass kept us honest under the clock. 🌟

### Accomplishments we're proud of

Twelve technologies, every one running live and pulling weight, composed into a single product instead of a pile of demos. A genuinely closed loop where the system's own evaluations change what gets watched next run. A council that actually deliberates. A tamper-evident ledger sitting next to a task graph with real math on it. A voice loop where the agent asks permission out loud before it acts. And all of it shipped clean under a hackathon clock: roughly 39,000 lines, 237 passing tests, green CI on every push, a security and code-review pass, and a dependency CVE cleared before we called it done. 📚

### What we learned

Oversight is an architecture, not a feature you sprinkle on at the end. The event-bus seam is the whole reason watching, durability, memory, and observability could attach without coupling to the agent or slowing it down. Building every layer to fail gracefully turned out to be a feature, not a chore: it is what makes the thing both robust and demoable anywhere. A council of specialists that disagree catches what a lone verifier misses. And the deepest lesson is the simplest: the cursor moving is the hook, but the audit trail beside it is the product. Trust is the thing a clinic or a bank would actually pay for. 🚀

### What's next

for The Shepherd There is going to be a layer like this, the same way there was always going to be a firewall. Every agent that touches a real machine will eventually run behind one, and we intend for it to be this one. Next we take Shepherd from hackathon to pilot in the rooms where trust is the blocker, ship a policy marketplace so teams share governance the way they share lint configs, build audit export for regulators, and extend the one watch across new surfaces: mobile, full browser fleets, whole organizations of agents. The agents will keep getting the headlines. The layer that lets them into production is the business. Built by four people and a Devin teammate at the UC Berkeley AI Hackathon. Local-first. Model-agnostic. Honest enough to hand the audit log to a regulator. The cursor moving is the hook. The audit trail beside it is the product.

## README (from the GitHub repository)

<div align="center">

<img src="frontend/public/shepherd-mark.png" alt="Shepherd" width="104" />

# Shepherd

### Mission control for AI agents. The agent is the part you cannot trust. Shepherd is the system that lets you run it anyway.

[![CI](https://github.com/KarthikSubramanian07/shepherd/actions/workflows/ci.yml/badge.svg)](https://github.com/KarthikSubramanian07/shepherd/actions/workflows/ci.yml)

![Python](https://img.shields.io/badge/Python-3.11+-2c6e60?logo=python&logoColor=white)
![Next.js](https://img.shields.io/badge/Next.js-16-223b3a?logo=nextdotjs&logoColor=white)
![Redis](https://img.shields.io/badge/Redis-8_vector_sets-cf6a43?logo=redis&logoColor=white)
![OpenTelemetry](https://img.shields.io/badge/OTel-Arize_Phoenix-2c6e60?logo=opentelemetry&logoColor=white)
![Claude](https://img.shields.io/badge/oversight-Claude-cf6a43?logo=anthropic&logoColor=white)
![License](https://img.shields.io/badge/license-MIT-223b3a)
![Sponsor tech](https://img.shields.io/badge/sponsor_tech-10_integrated_live-cf6a43)

🐑 &nbsp;**Local-first** &nbsp;·&nbsp; 🛰️ **Operate any machine** &nbsp;·&nbsp; 🔗 **Tamper-evident audit** &nbsp;·&nbsp; 🗣️ **Voice in, voice out** &nbsp;·&nbsp; ⚡ **Sub-ms policy gate**

</div>

We built an agent that could drive a desktop. Then we watched it do something we
never asked it to, and the scary part was not the mistake. It was the silence
after: no rewind, no record, no way to answer the only question that mattered,
*what did it just do, and why?* The cursor had moved on its own, and the most
honest answer we had was a shrug.

Teams are shipping agents like that into production and quietly hoping for the
best. Hoping is not a control.

**Shepherd is local mission control for AI desktop agents.** A shepherd does not
cage the flock or walk every step for it. It knows the dangerous ground, runs out
front, and steps in at exactly the right moment. That is the job: you teach a task
by demonstrating it once, Shepherd drives the real desktop to carry it out, governs
every high-stakes step in real time, catches it the instant it strays, replays
precisely what it did while you were away, and runs it on a machine across the
country. The sub-millisecond policy gate, the independent AI verifier, the human
decision point, and the tamper-evident audit trail are one system, not features
bolted on.

Shepherd is not a safety wrapper around someone else's agent. The agent is the
engine inside it. Everything that turns raw capability into something you can
actually deploy, the teaching, the governance, the memory, the audit, the remote
control, is Shepherd.

---

## ⚡ Why now

The agent-capability curve went vertical and the trust curve did not move. Every
month a new model can drive a computer better, and every month the gap widens
between "the agent can do this" and "I would let it do this unattended." The
blocker to deploying agents in anything that matters (a clinic, a finance back
office, a benefits desk) is not capability. It is that a black box touching a
real machine is a liability nobody can sign off on. Shepherd is the missing
control plane: the thing that turns "impressive demo" into "approved in prod."

There is going to be a layer like this. Every agent that touches a real machine
will eventually run behind one, the same way every packet eventually ran behind a
firewall. We are building that layer to be the obvious one: local-first, model-
agnostic, fast enough to be invisible, and honest enough that you can hand the
audit log to a regulator. The agents get the headlines. The layer that lets them
into production is the business.

---

## ⏱️ The demo in 90 seconds

```mermaid
flowchart TD
    V["🗣️ You say: send the candidate decision email"] --> STT["Deepgram transcribes · router resolves the intent"]
    STT --> AS["Agent S opens the mail composer<br/>Simular drives the real desktop"]
    AS --> RES["Missing context, so a durable Agentspan agent<br/>researches the candidate on the live web Browserbase<br/>and drafts the body"]
    RES --> SEND(["The agent moves to Send"])
    SEND --> POL["⚡ Policy engine, under 1ms:<br/>external recipient + secret in the body"]
    POL -->|"Verdict: HALT"| GATE["🛑 Stopped before a single irreversible click"]
    GATE --> HUB["🎛️ Control Hub lights up<br/>milestone graph · verifier second opinion · live screen"]
    HUB --> HUMAN{"You approve,<br/>or say stop out loud"}
    HUMAN -->|approve| GO["▶️ It continues"]
    HUMAN -->|stop| STOP["🛑 Halts at the boundary"]
    GO --> AUDIT["🔗 Every action, including the halt,<br/>signed into a SHA-256 hash chain"]
    STOP --> AUDIT

    classDef danger fill:#bb4a3a,stroke:#bb4a3a,color:#fff;
    classDef gate fill:#cf6a43,stroke:#cf6a43,color:#fff;
    classDef proof fill:#223b3a,stroke:#223b3a,color:#fff;
    class GATE,STOP danger;
    class POL,HUMAN gate;
    class AUDIT proof;
```

Change one rule in `data/policy.yaml`, re-run, and the behavior changes
immediately. That is the difference between a demo and a product.

---

## 🎛️ What you get (the Control Hub)

A single console, not a pile of scripts. Every tab is live over WebSocket.

- **Command Center**: dispatch a task by voice or text, watch the live execution
  graph light up milestone by milestone, approve or halt at the gate.
- **Remote**: operate another machine. Its live screen (WebRTC peer-to-peer)
  beside the workflow graph it builds as it goes. Steer or teach it inline.
- **Routines**: the recorded "tools," each a demonstration plus per-step notes.
- **Task Graph**: the crystallized milestone DAG a task has learned across runs,
  with the modal execution path, branch entropy, and transition odds drawn on it.
- **Workflows**: the dispatchable, versioned workflows, including the judgment
  calls baked in from your steers.
- **Runs**: full scrubbable replay of any past run, step by step.
- **Interventions**: the queue of moments that needed a human, and how they
  resolved.
- **Audit**: the tamper-evident hash chain, verifiable in one click.
- **Policy**: the live governance rules.
- **Voice Lab**: the Deepgram transcription surface.

---

## ⚙️ How it works

```mermaid
flowchart TD
    IN["🗣️ Voice / typed intent"] --> STT["Deepgram STT · boundary only"]
    STT --> ROUTER["Intent router<br/>Redis vector search BAAI/bge + keyword fallback"]
    ROUTER --> ENGINE["Execution engine<br/>autonomous plan, or routine replay vision/deterministic<br/>Agent S drives the desktop"]
    ENGINE -->|"at high-stakes boundaries only, never mid-click"| STACK

    subgraph STACK["🛡️ 3-layer oversight stack, fastest first"]
        direction TB
        L1["1 · Policy engine<br/>rule-based, always on, under 1ms · data/policy.yaml"]
        L2["2 · Verifier<br/>independent Claude, or a Band council"]
        L3["3 · Human gate<br/>approve / halt / steer · spoken stop fires the same halt"]
        L1 --> L2 --> L3
    end

    STACK --> AUDIT["🔗 Tamper-evident audit log<br/>SHA-256 hash chain"]
    STACK --> TEL["📊 Telemetry<br/>Arize Phoenix spans"]
    STACK --> MEM["🧠 Memory + crystallization<br/>Redis recall · runs coalesce into workflows"]
    AUDIT --> HUB["🎛️ Control Hub Next.js<br/>+ Remote Command Center"]
    TEL --> HUB
    MEM --> HUB

    classDef gate fill:#cf6a43,stroke:#cf6a43,color:#fff;
    classDef proof fill:#223b3a,stroke:#223b3a,color:#fff;
    class STACK gate;
    class AUDIT proof;
```

A single rule runs through the whole codebase: **the click path is sacred.**
Nothing async, networked, or ML-based runs inside a routine's step sequence.
Every integration is invoked between steps, at boundaries, so a network hiccup or
a model stall can never strand the mouse mid-action.

---

## 🏆 The five things that make it a product, not a demo

### 1. 🎬 Two ways to hand it a task: show it once, or just say it

For a task you repeat, **demonstrate it once** and the demonstration becomes the
routine. No node graph, no Zapier-style flowchart. The agent works in the same
workspace you do

[README truncated for size]

## Detected evidence (automated analysis)

Indexed codebase: 232 recognized source files, 1780 KB.
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Redis (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code
- SQL (language) — claimed on Devpost, not found in the code
- AI coding agent: Claude Code — evidence: commit authorship or trailers
- AI coding agent: Cursor — evidence: commit authorship or trailers

## Codebase structure (from repository index)

### Files (120 of 257)

```
.env.example
.github/workflows/ci.yml
.gitignore
.python-version
.vscode/settings.json
agent_config.yaml.example
compat.py
config.py
coordinator/__init__.py
coordinator/__main__.py
coordinator/catalog_store.py
coordinator/Dockerfile
coordinator/fly.toml
coordinator/server.py
coordinator/title_gen.py
dashboard/__init__.py
dashboard/deepgram_routes.py
dashboard/events.py
dashboard/fleet_trace.py
dashboard/forwarder.py
dashboard/server.py
data/audit.jsonl
data/coords.demo.json
data/demo_form.html
data/demo_mail.html
data/live_job_app.html
data/live_projects.html
data/policy.yaml
data/routines.json
data/task_graphs.json.bak
data/workflows.json.bak
data/workflows.json.bak2
DEMO.md
DESIGN.md
docs/AGENT_S_GRID.md
docs/design-mid-run-steering-v2.md
docs/design-mid-run-steering.md
docs/DEVIN_TESTING.md
docs/MULTI_AGENT.md
docs/PEERING.md
docs/pitch_scripts.md
docs/PROTOCOL.md
docs/SETUP.md
docs/setup/multi-machine.md
docs/setup/README.md
docs/setup/single-machine.md
docs/setup/tailscale.md
docs/setup/tunnel.md
docs/workflow-engine.md
engine/__init__.py
engine/_anthropic_compat.py
engine/agent_s_adapter.py
engine/agent_s_grounding.py
engine/approvals.py
engine/browserbase_driver.py
engine/coalescer.py
engine/coords.py
engine/engine.py
engine/generalize.py
engine/llm.py
engine/milestones.py
engine/permissions.py
engine/recorder.py
engine/routine_planner.py
engine/routines.py
engine/run_summary.py
engine/task_graph.py
engine/text_input.py
engine/trace_journal.py
engine/workflow_control.py
engine/workflow_describe.py
engine/workflow_edit.py
engine/workflow_executor.py
engine/workflow_promote.py
engine/workflow_store.py
frontend/.env.example
frontend/.gitignore
frontend/index.html
frontend/next.config.mjs
frontend/package.json
frontend/postcss.config.mjs
frontend/src/app/api/agents/[id]/route.ts
frontend/src/app/api/agents/route.ts
frontend/src/app/api/deepgram/status/route.ts
frontend/src/app/api/deepgram/transcribe/route.ts
frontend/src/app/api/interventions/[id]/route.ts
frontend/src/app/api/interventions/route.ts
frontend/src/app/api/routines/[id]/route.ts
frontend/src/app/api/routines/route.ts
frontend/src/app/api/runs/[id]/route.ts
frontend/src/app/api/runs/route.ts
frontend/src/app/audit/page.tsx
frontend/src/app/command-center/page.tsx
frontend/src/app/fleet/page.tsx
frontend/src/app/globals.css
frontend/src/app/interventions/page.tsx
frontend/src/app/kit/page.tsx
frontend/src/app/layout.tsx
frontend/src/app/page.tsx
frontend/src/app/policy/page.tsx
frontend/src/app/remote/page.tsx
frontend/src/app/routines/[id]/page.tsx
frontend/src/app/routines/page.tsx
frontend/src/app/runs/[id]/page.tsx
frontend/src/app/runs/page.tsx
frontend/src/app/task-graph/page.tsx
frontend/src/app/voice-lab/page.tsx
frontend/src/app/workflows/[id]/page.tsx
frontend/src/app/workflows/page.tsx
frontend/src/components/agents/AgentCard.tsx
frontend/src/components/experimental/MicTranscriber.tsx
frontend/src/components/graph/node-kit.tsx
frontend/src/components/graph/nodes.tsx
frontend/src/components/graph/RoutineGraph.tsx
frontend/src/components/graph/StepNode.tsx
frontend/src/components/graph/TaskGraphView.tsx
frontend/src/components/graph/TraceGraph.tsx
frontend/src/components/graph/WorkflowGraph.tsx
frontend/src/components/IntegrationsPanel.tsx
frontend/src/components/layout/PageHeader.tsx
[137 more files omitted for size]
```

### Dependencies

- frontend/package.json: @types/node@^20.16.11, @types/react@^18.3.11, @types/react-dom@^18.3.0, @xyflow/react@^12.3.5, autoprefixer@^10.4.20, clsx@^2.1.1, eslint-config-next@^16.2.9, lucide-react@^0.454.0, next@^16.2.9, postcss@^8.5.15, react@^18.3.1, react-dom@^18.3.1, tailwind-merge@^2.5.4, tailwindcss@^3.4.14, typescript@^5.6.3
- overlay/package.json: electron@^39.8.5
- package.json: @simular-ai/simulang-js@^8.0.0, tsx@^4.19.2
- pyproject.toml: agentspan@>=0.1.10, aiortc@>=1.9.0, arize-phoenix@>=8.0.0, arize-phoenix-otel@>=0.9.0, armoriq-sdk@>=0.3.7, armoriq-sdk@>=0.1, band-sdk[anthropic]@>=1.0.0, browserbase@==0.3.0, deepgram-sdk@>=3.2.7, fastapi@>=0.115.0, fastembed@>=0.7.3, gui-agents@>=0.3, gui-agents, httpx@>=0.27.0, opentelemetry-exporter-otlp@>=1.24.0, opentelemetry-sdk@>=1.24.0, Pillow@>=12.2.0, playwright@==1.44.0, protobuf@>=5.29.6, pyaudio@==0.2.14, pyautogui@==0.9.54, pydantic-settings@>=2.2.0, pynput@==1.7.6, pytesseract@==0.3.10, python-dotenv@>=1.2.2, python-multipart@>=0.0.18, pyyaml@>=6.0.3, redis@>=5.0.4, sentry-sdk@>=2.8.0, stagehand@>=3.0.0, starlette@>=0.47.2, uvicorn[standard]@>=0.29.0, websockets@>=12.0

### Recent commits (newest first)

- chore: gitignore generated data/simulang/ artifacts
- chore(deps-dev): bump electron (#63)
- feat: agent-initiated help signal — Agent S proactively suspends when blocked (#62)
- feat(fleet): non-blocking execution graphs for fleet agents (#58)
- fix: make milestone segmenter intervention-aware for stable teaching loop (#61)
- fix: restore steps_done counter from SuspendedTask on resume (#59)
- Fix CI red: guard _first_click_coords against non-string code
- stop tracking overlay/node_modules
- add screenshots to sentry
- rem wf_autonomous
- remove files
- switch to duckduckgo
- adding curated pitches and accessibility error logging
- adding evals in phoenix
- Merge remote-tracking branch 'origin/main'
- Query-aware run response, Fleet response UI, browser parallelism
- Fix CI red from run-summary: unused import + WorkflowStepRecord crash
- Merge remote-tracking branch 'origin/main'
- Per-request text response + browser-agent robustness + log hygiene
- README: founding-moment narrative + reframe integrations as core anatomy

## Key source files (fetched from GitHub, selected and truncated for size)

### PRODUCT.md

```markdown
# Product

## Register

product

## Users

Developers and small engineering teams who have built or are evaluating an **AI
desktop agent** and don't yet trust it enough to run unattended in production.
Their context: the agent already works, but it's a black box — when it misclicks
or does something unexpected there's no rewind, no audit trail, no way to prove
what it did. Secondary user: anyone who stepped away while the agent ran and
needs to know exactly what happened. In the hackathon demo, the immediate
audience is judges absorbing the trust thesis in one 5-minute sitting.

The job to be done: **author a task by demonstrating it once, watch the agent run
it live, catch it when it strays, and replay exactly what it did.** On any given
screen the primary task is oversight — see the current state, decide on a flagged
step, or review a past run.

## Product Purpose

The Shepherd is a **local oversight and governance layer for AI desktop agents**.
The agent is the part you can't trust; the Shepherd is the layer that makes it
trustworthy — a configurable monitor, a tamper-evident audit trail, and a human
decision gate between the agent's intent and the machine. Success is a developer
deploying an agent they previously would have babysat, because they can now see
every action, halt it on command, and prove what it did.

## Brand Personality

Watchful, calm, trustworthy — a shepherd keeping a night-and-day watch over a
flock. Three words: **vigilant, grounded, reassuring.** The voice is plain and
direct, never alarmist; it states what happened and what to do. The emotional
goal is *earned calm* — the feeling of handing something risky to a steady pair
of hands. The metaphor is expressive (a guiding lantern, waypoints along a path,
a wool-and-earth warmth) but always credible and enterprise-legible.

## Anti-references

- **Generic SaaS dashboard** — no Linear/Vercel/Stripe-clone purple gradients,
  endless identical cards, or default-Inter-plus-blue.
- **Cream/beige editorial** — no 2026 warm-neutral "parchment" body background
  with a serif display and terracotta. Warmth is carried by accent + type, not a
  beige page.
- **Toy / cartoon pastoral** — no literal cartoon sheep, clip-art crooks, or
  childish farm whimsy. The metaphor stays grown-up.
- **Loud / neon** — no high-saturation neon, heavy glow, or dark-mode-gamer
  energy. This is calm oversight.

## Design Principles

- **The audit trail is the product, the cursor is the hook.** Live automation
  earns attention; what wins trust is being able to see, halt, and prove. Design
  the oversight surfaces (live graph, intervention, replay) as first-class, not
  as chrome around the automation.
- **The lantern lights the danger.** A single warm accent is reserved for the
  moment that needs a human — the flagged step, the active node. Calm everywhere
  else so that one signal reads instantly.
- **Show the watch, don't claim it.** Surface real state (which milestone is
  running, what was recalled
[truncated — 809 more characters]
```

### DESIGN.md

```markdown
# Design

## Theme

**Daybreak Watch** — a light, pastoral-but-credible oversight console. The mood
is a shepherd's morning watch: a wool off-white ground, earthy browns for
structure, a warm orange "lantern" that lights up exactly where a human is
needed, and a meadow green for what's safe. Light, calm, grounded — warmth comes
from the accents and type, never from a beige page.

## Color

Light theme. Off-white wool ground kept near-neutral (a whisper of warmth, not
parchment); warmth carried by the orange lantern + earthy browns.

| Role | Hex | Use |
|---|---|---|
| `ground` | `#f7f5f1` | App background — wool off-white, near-neutral |
| `surface` | `#fffefb` | Raised cards / panels (slightly lifted off ground) |
| `surface-2` | `#f0ece4` | Inset / secondary fills, rails |
| `edge` | `#e2dcd0` | Hairlines, borders |
| `ink` | `#2a231d` | Primary text — warm charcoal |
| `muted` | `#7c7064` | Secondary text — warm taupe (≥4.5:1 on ground) |
| `accent` (lantern) | `#dd6a1f` | Identity + the attention/flag moment — orange |
| `accent-ink` | `#b4520f` | Orange text on light (contrast-safe) |
| `bark` | `#7a5c44` | Earthy brown — structure, brand glyph, secondary |
| `ok` | `#1f8a5b` | Safe / completed — meadow green |
| `flag` | `#dd6a1f` | Needs attention — same lantern orange (unified) |
| `halt` | `#cf3b34` | Stopped / dangerous — clay red |
| `idle` | `#9a8f81` | Pending / inactive — warm gray |

Accent is the **only** saturated color used for identity, and it is reserved for
the one thing that matters: the milestone that's running and the step that needs
a human. Green/red are functional status; brown is structural warmth.

## Typography

One-family-in-weights for the UI (no two-similar-sans pairing), a monospace for
data/IDs/traces (true to an oversight console).

- **UI / display**: a humanist grotesque system stack — `ui-sans-serif, system-ui,
  "Segoe UI", Roboto, "Helvetica Neue", Arial`. Headings use weight 600–700,
  letter-spacing −0.01em on large sizes (never tighter than −0.04em).
- **Data / mono**: `ui-monospace, "SF Mono", "JetBrains Mono", Menlo, Consolas`
  for run IDs, timings, hashes, similarity scores, step counts.
- **Eyebrows used sparingly** — a deliberate brand cadence, not above every
  section. `text-wrap: balance` on headings; body capped ~70ch.

## Brand glyph

A minimal **shepherd's-crook + waypoint** mark (a hooked stroke ending in a
filled dot), bark-brown with an orange dot — the lantern. Lives in the sidebar
lockup and favicons. No literal sheep.

## Components

- **Cards/panels**: `surface` on `ground`, 1px `edge` border, generous radius
  (`0.75rem`), soft low shadow for lift (no glassmorphism, no side-stripes).
- **Buttons**: primary = solid orange lantern, white text; outline = `edge`
  border on surface; danger = clay red. Visible focus ring (orange, 2px).
- **Status**: always hue + icon/label (never color alone). Running = orange +
  pulse; done = green + check; flagged = orange + alert; halted = red + octa
[truncated — 945 more characters]
```

### package.json

```
{
  "name": "shepherd-simulang-runtime",
  "private": true,
  "description": "Runtime for replaying Shepherd-compiled SimuLang workflows (data/simulang/*.ts). The Python engine shells out to `npx tsx <script>`; these deps make that resolve.",
  "dependencies": {
    "@simular-ai/simulang-js": "^8.0.0"
  },
  "devDependencies": {
    "tsx": "^4.19.2"
  }
}

```

### pyproject.toml

```
[project]
name = "shepherd"
version = "0.1.0"
description = "Local oversight and governance layer for AI desktop agents"
readme = "README.md"
requires-python = ">=3.11"
license = { text = "MIT" }
dependencies = [
    "pyautogui==0.9.54",
    "Pillow>=12.2.0",
    "pynput==1.7.6",
    "fastapi>=0.115.0",
    "uvicorn[standard]>=0.29.0",
    "websockets>=12.0",
    "arize-phoenix-otel>=0.9.0",
    "opentelemetry-sdk>=1.24.0",
    "opentelemetry-exporter-otlp>=1.24.0",
    "sentry-sdk>=2.8.0",
    "redis>=5.0.4",
    "deepgram-sdk>=3.2.7",
    "pydantic-settings>=2.2.0",
    "browserbase==0.3.0",
    "playwright==1.44.0",
    "httpx>=0.27.0",
    "python-multipart>=0.0.18",
    "pytesseract==0.3.10",
    "python-dotenv>=1.2.2",
    "gui-agents>=0.3",
    "fastembed>=0.7.3",
    "pyyaml>=6.0.3",
    "protobuf>=5.29.6",
    "starlette>=0.47.2",
    "arize-phoenix>=8.0.0",
    "agentspan>=0.1.10",
    "armoriq-sdk>=0.3.7",
]

[project.optional-dependencies]
voice   = ["pyaudio==0.2.14"]
agent_s = ["gui-agents"]
webrtc  = ["aiortc>=1.9.0"]
# Band multi-agent oversight. Only the verifier *peer* process needs the SDK;
# the engine side talks to Band's free Agent API over httpx (already a core dep).
band    = ["band-sdk[anthropic]>=1.0.0"]
# ArmorIQ intent-intelligence authorization (ships the armoriq CLI too).
armoriq = ["armoriq-sdk>=0.1"]
# Stagehand: NL web actions (act/extract/observe) on a Browserbase cloud browser.
web     = ["stagehand>=3.0.0"]

[tool.uv]
package = false

# gui-agents pulls in paddlepaddle + paddleocr, but those are only used by its
# (unused) s1 OCR server — we run the s3 agent with LLM/UI-TARS grounding and the
# monitor uses pytesseract. paddlepaddle also has no wheel for Intel macOS, which
# breaks `uv sync` there. Drop both via an always-false marker so they're never
# installed on any platform.
override-dependencies = [
    "paddlepaddle; sys_platform == 'never'",
    "paddleocr; sys_platform == 'never'",
    # fastembed (vector router) pulls onnxruntime, which has no wheel for Intel
    # macOS. Vector routing is optional (falls back to keyword routing), so drop
    # both ONLY on Intel macOS; keep them everywhere else.
    "fastembed; sys_platform != 'darwin' or platform_machine != 'x86_64'",
    "onnxruntime; sys_platform != 'darwin' or platform_machine != 'x86_64'",
]

[dependency-groups]
dev = [
    "pytest>=9.1.1",
    "ruff>=0.6",
]

[tool.pytest.ini_options]
# Put the repo root on sys.path so `from services import ...` resolves under the
# `pytest` console script (CI), not just `python -m pytest` (which adds cwd itself).
pythonpath = ["."]
testpaths = ["tests"]

[tool.ruff]
line-length = 100
extend-exclude = [".venv", "node_modules", "frontend", "data"]

[tool.ruff.lint]
# Enforce the real-bug rules (pyflakes F, most pycodestyle E/W); ignore patterns
# that are intentional in this codebase: E402 (lazy imports gate optional deps for
# graceful degradation), E731 (small lambdas), E741 (a few short loop vars),
# E702 (compact one-liners in scripts).
ignore = ["E402", "E731", "E741", "E702"]

[tool.ruff.lint.per-file-ignores]
# Tests and demo scripts keep deliberately-unused locals (fixtures, kept refs).
"tests/**" = ["F841"]
"scripts/**" = ["F841"]

```

### overlay/package.json

```
{
  "name": "shepherd-overlay",
  "version": "1.0.0",
  "private": true,
  "description": "Translucent, always-on-top, content-protected HUD that streams Shepherd's live agent activity over it screen-share-invisible.",
  "main": "main.js",
  "scripts": {
    "start": "electron ."
  },
  "devDependencies": {
    "electron": "^39.8.5"
  }
}

```

### frontend/package.json

```
{
  "name": "shepherd-web",
  "version": "0.1.0",
  "private": true,
  "description": "Shepherd dashboard — record a task, let an agent run it, watch it traverse the routine graph. Mock API for now.",
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint",
    "typecheck": "tsc --noEmit"
  },
  "dependencies": {
    "@xyflow/react": "^12.3.5",
    "clsx": "^2.1.1",
    "eslint-config-next": "^16.2.9",
    "lucide-react": "^0.454.0",
    "next": "^16.2.9",
    "react": "^18.3.1",
    "react-dom": "^18.3.1",
    "tailwind-merge": "^2.5.4"
  },
  "devDependencies": {
    "@types/node": "^20.16.11",
    "@types/react": "^18.3.11",
    "@types/react-dom": "^18.3.0",
    "autoprefixer": "^10.4.20",
    "postcss": "^8.5.15",
    "tailwindcss": "^3.4.14",
    "typescript": "^5.6.3"
  },
  "overrides": {
    "postcss": "^8.5.15"
  }
}

```

### coordinator/Dockerfile

```
# Shepherd Coordinator — slim relay-only image.
#
# Runs ONLY the coordinator relay (coordinator/server.py). Does NOT require
# Agent S, Playwright, pyautogui, GUI deps, or any of the heavy ML/vision
# packages. Works on any container host: a VPS with Docker, Railway, Render,
# Fly.io, or even a Raspberry Pi.
#
# Build:  docker build -f coordinator/Dockerfile -t shepherd-coordinator .
# Run:    docker run -p 8770:8770 -e COORDINATOR_TOKEN=secret shepherd-coordinator

FROM python:3.12-slim

WORKDIR /app

# Install only the coordinator's runtime deps (no GUI, no ML, no browser).
RUN pip install --no-cache-dir \
    "fastapi>=0.115.0" \
    "uvicorn[standard]>=0.29.0" \
    "pydantic-settings>=2.2.0" \
    "python-dotenv>=1.2.2"

# Copy only the files the coordinator needs at runtime.
COPY compat.py /app/compat.py
COPY config.py /app/config.py
COPY coordinator/ /app/coordinator/

# The coordinator optionally imports dashboard.deepgram_routes; create an empty
# package so the import doesn't fail (the try/except in server.py handles it).
RUN mkdir -p /app/dashboard && touch /app/dashboard/__init__.py

# COORDINATOR_TOKEN is set at runtime (docker run -e), not baked into the image.
ENV COORDINATOR_PORT=8770
ENV RELAY_FPS=3.0
ENV RELAY_FRAME_WIDTH=1024
ENV RELAY_FRAME_QUALITY=55

EXPOSE 8770

HEALTHCHECK --interval=10s --timeout=3s --start-period=5s \
  CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8770/api/health')"

CMD ["python", "-m", "coordinator"]

```

### overlay/main.js

```javascript
// Shepherd Overlay — Electron main process.
//
// A translucent, frameless, always-on-top window that floats over everything
// (including a fullscreen app being driven) and renders Shepherd's live agent
// activity as a compact terminal HUD. `setContentProtection(true)` keeps it
// INVISIBLE to screen capture / share / recording, so an audience sees the Mac
// being driven but not this control panel.
//
// It is a thin shell: the HUD page (index.html) talks directly to the Shepherd
// dashboard backend (WebSocket /ws for events, POST /api/intent + /api/halt for
// control), so no Shepherd backend coupling lives here beyond the URL.

const {
  app, BrowserWindow, ipcMain, globalShortcut, screen, Tray, Menu, nativeImage,
} = require("electron");
const path = require("node:path");

// Keep the renderer's timers/WebSocket alive even when the HUD isn't focused.
app.commandLine.appendSwitch("disable-background-timer-throttling");

// Where the Shepherd dashboard backend lives. Override with SHEPHERD_URL.
const SHEPHERD_URL = process.env.SHEPHERD_URL || "http://localhost:8765";

// ── Visibility toggles ──────────────────────────────────────────────────────
// Transparent windows can fail to composite on some macOS/GPU setups (the whole
// window renders invisible). OVERLAY_DEBUG launches a plain, opaque, framed,
// centered window with devtools so we can confirm rendering. OVERLAY_NO_PROTECT
// turns OFF content protection so the HUD shows in screen captures too.
const DEBUG = !!process.env.OVERLAY_DEBUG;
const TRANSPARENT = !DEBUG && process.env.OVERLAY_OPAQUE !== "1";
const NO_PROTECT = DEBUG || process.env.OVERLAY_NO_PROTECT === "1";

let win = null;
let tray = null;
let clickThrough = false;

function createWindow() {
  const primary = screen.getPrimaryDisplay();
  const { width: sw } = primary.workAreaSize;

  win = new BrowserWindow({
    width: 420,
    height: 560,
    // Top-right corner by default — out of the way of whatever is being driven.
    x: DEBUG ? undefined : sw - 440,
    y: DEBUG ? undefined : 24,
    center: DEBUG,
    minWidth: 300,
    minHeight: 220,
    // Show immediately. Gating on `ready-to-show` is unreliable for transparent
    // windows on macOS (the event can fail to fire → the window stays hidden).
    show: true,
    frame: DEBUG ? true : false,
    transparent: TRANSPARENT,
    hasShadow: DEBUG ? true : false,
    resizable: true,
    movable: true,
    alwaysOnTop: true,
    fullscreenable: false,
    skipTaskbar: true,
    backgroundColor: TRANSPARENT ? "#00000000" : "#0b0e14",
    webPreferences: {
      preload: path.join(__dirname, "preload.js"),
      contextIsolation: true,
      nodeIntegration: false,
    },
  });

  // Hide from screen capture / share / recording — the whole point of the HUD.
  // Skipped when NO_PROTECT so it shows in captures (and during debugging).
  if (!NO_PROTECT) win.setContentProtection(true);

  if (process.platform === "darwin") {
    win.setVisibleOnAllWorkspaces(true, { visibleOnFullScreen: true });
    win.setAlwaysOnTop(true, "floating");
    if (win.setHiddenInMissionControl) win.setHiddenInMissionControl(true);
  } else {
    win.setAlwaysOnTop(true);
  }

  win.loadFile(path.join(__dirname, "index.html"), {
    query: { shepherd: SHEPHERD_URL },
  });

  // Belt-and-suspenders: force the window visible and to the front once the page
  // has loaded, regardless of whether `ready-to-show` fired.
  const reveal = () => {
    if (!win || win.isDestroyed()) return;
    win.show();
    win.moveTop();
    win.setAlwaysOnTop(true, "screen-saver");
    const b = win.getBounds();
    console.log(
      `[overlay] reveal: visible=${win.isVisible()} bounds=${b.x},${b.y} ${b.width}x${b.height} ` +
      `transparent=${TRANSPARENT} protect=${!NO_PROTECT}`
    );
    if (DEBUG) win.webContents.openDevTools({ mode: "detach" });
  };
  win.webContents.on("did-finish-load", reveal);
  win.once("ready-to-show", reveal);
  setTimeout(reveal, 1200);

  win.webContents.on("did-fail-load", (_e, code, desc) => {
    console.error("[overlay] failed to load HUD:", code, desc);
  });
  win.on("closed", () => {
    win = null;
  });
}

function applyClickThrough(enabled) {
  clickThrough = enabled;
  if (win && !win.isDestroyed()) {
    win.setIgnoreMouseEvents(enabled, { forward: true });
  }
  return clickThrough;
}

// Force the HUD into view: recreate if needed, move it onto the display under
// the cursor, place it top-right there, then show + focus + assert top level.
// This is the reliable "I can't find the window" escape hatch — it works even
// when the window opened on another Space or display.
function summon() {
  if (!win || win.isDestroyed()) {
    createWindow();
    return;
  }
  const cursor = screen.getCursorScreenPoint();
  const disp = screen.getDisplayNearestPoint(cursor);
  const wa = disp.workArea;
  const b = win.getBounds();
  win.setBounds({
    x: wa.x + wa.width - b.width - 16,
    y: wa.y + 16,
    width: b.width,
    height: b.height,
  });
  if (process.platform === "darwin") {
    win.setVisibleOnAllWorkspaces(true, { visibleOnFullScreen: true });
  }
  win.show();
  win.focus();
  win.moveTop();
  win.setAlwaysOnTop(true, "screen-saver");
}

function createTray() {
  if (tray) return;
  try {
    tray = new Tray(nativeImage.createEmpty());
    if (process.platform === "darwin") tray.setTitle(" 🐑 Shepherd");
    tray.setToolTip("Shepherd Overlay — ⌘⇧Space to summon");
    tray.setContextMenu(
      Menu.buildFromTemplate([
        { label: "Show / Summon (⌘⇧Space)", click: summon },
        {
          label: "Toggle visibility (⌘⇧H)",
          click: () => {
            if (win && win.isVisible()) win.hide();
            else summon();
          },
        },
        {
          label: "Toggle click-through (⌘⇧C)",
          click: () => {
            applyClickThrough(!clickThrough);
            if (win) win.webContents.send("clickthrough-changed", clickThrough);
          },
        },
        { type:
[truncated — 1572 more characters]
```

### main.py

```python
#!/usr/bin/env python3
"""
The Shepherd — main entry loop.
Voice/typed intent → router → engine → telemetry + memory + dashboard.

Front door is controlled by USE_ROUTER (config / .env):
  USE_ROUTER=false (default)  free-form autonomous Agent S goals (no routing)
  USE_ROUTER=true             match a saved workflow/routine first, autonomous on no match
  ROUTINE_REPLAY=vision|deterministic  how a matched routine is driven (LIVE|LOCKED)
  MATCH_WORKFLOWS / MATCH_ROUTINES  enable each routing source separately (both on by default)

Usage:
  python main.py
  python main.py --mode LOCKED      # one-off override: force deterministic replay
  python main.py --mode AUTONOMOUS  # one-off override: free-form Agent S goals
"""
import os
import queue
import sys
import time
import threading
from types import SimpleNamespace

from config import (
    FEATURES, EXECUTION_MODE, DASHBOARD_PORT, USE_ROUTER, ROUTINE_REPLAY,
    AUTONOMOUS_ON_UNMATCHED, EXIT_WHEN_DONE, BACKEND_URL, CONSOLE_LOG,
    MATCH_WORKFLOWS, MATCH_ROUTINES,
)
from shepherd_types import Intent, ResolvedRoutine
from router.router import ShepherdIntentRouter
from engine.engine import ShepherdExecutionEngine
from engine.coords import load_coords
from engine.routines import load_routines
from telemetry.telemetry import ShepherdTelemetry
from telemetry.sentry_init import (
    init_sentry,
    capture as sentry_capture,
    capture_message as sentry_capture_message,
)
from telemetry.memory import ExecutionMemory
from telemetry.evolution import RoutineEvolution
from dashboard.events import event_bus


def _stdin_producer(
    engine: ShepherdExecutionEngine,
    remote_intents: "queue.Queue[str]",
    idle: "threading.Event",
) -> None:
    """
    Read typed (or spoken) goals from the command line and feed them into the
    SHARED intent queue — the same queue the frontend / coordinator / poller feed.
    Runs in a background thread so the CLI and the frontend can both drive the agent
    at the same time. Exits quietly if there's no interactive stdin (headless).

    Only prompts while the agent is idle (`idle` is set), so the "Intent ->" prompt
    never interleaves with a run's log output.
    """
    while True:
        idle.wait()   # hold the prompt until the current run finishes
        if FEATURES["deepgram"]:
            try:
                from services.deepgram_input import listen_and_transcribe, listen_for_stop_command
                listen_for_stop_command(halt_callback=engine.request_halt)
                transcript = listen_and_transcribe()
                if transcript:
                    idle.clear()
                    remote_intents.put(transcript)
                    continue
            except Exception as e:
                print(f"[deepgram] {e} — using typed input.")
        try:
            line = input("Intent → ").strip()
        except EOFError:
            print("[shepherd] No interactive stdin — taking goals from the frontend only.")
            return
        if line:
            idle.clear()   # a run is about to start; don't reprompt until it's done
            remote_intents.put(line)


def _record_mode(routine_id: str) -> None:
    """
    Record a human demonstration for a routine and save it to data/routines.json.
    Usage: python main.py --record ROUTINE_FORM_FILL
    Controls: Cmd+Shift+M = mark step boundary  |  Cmd+Shift+Q = stop
    """
    import json
    from engine.recorder import DemonstrationRecorder

    print(f"\n[record] Demonstration mode — routine: {routine_id}")
    print("[record] Cmd+Shift+M = mark step  |  Cmd+Shift+Q = stop\n")

    narration_fn = None
    if FEATURES["deepgram"]:
        try:
            from services.deepgram_input import listen_and_transcribe
            narration_fn = lambda: listen_and_transcribe(4.0)
            print("[record] Deepgram active — speak step instructions after each Cmd+Shift+M\n")
        except Exception as e:
            print(f"[record] Deepgram unavailable ({e}) — no narration this session\n")

    recorder = DemonstrationRecorder(get_narration_fn=narration_fn)
    recorder.start()

    try:
        while recorder._running:
            time.sleep(0.5)
    except KeyboardInterrupt:
        pass

    steps = recorder.stop()
    if not steps:
        print("[record] No steps recorded — exiting.")
        return

    print(f"\n[record] {len(steps)} steps captured. Saving to routines.json…")

    steps_data = [
        {
            "index":           s.index,
            "action":          s.action,
            "target":          s.target,
            "text":            s.text,
            "timestamp":       s.timestamp,
            "instruction":     s.instruction,
            "screenshot_path": s.screenshot_path,
        }
        for s in steps
    ]

    routines_path = os.path.join(os.path.dirname(__file__), "data", "routines.json")
    with open(routines_path) as f:
        routines = json.load(f)

    matched = False
    for r in routines:
        if r["routine_id"] == routine_id:
            r["demonstration"] = steps_data
            matched = True
            break

    if not matched:
        print(f"[record] Routine '{routine_id}' not found. Available: "
              f"{[r['routine_id'] for r in routines]}")
        return

    with open(routines_path, "w") as f:
        json.dump(routines, f, indent=2)

    print(f"[record] ✓ Saved {len(steps)} steps → {routine_id}.demonstration")
    print("[record]   Screenshots: data/screenshots/step_NNN.png")
    print("[record]   Run 'python main.py' to execute with Agent S against this recording.\n")


def main() -> None:
    # ── Parse args ────────────────────────────────────────────────────────────
    if "--record" in sys.argv:
        idx = sys.argv.index("--record")
        rid = sys.argv[idx + 1] if idx + 1 < len(sys.argv) else "ROUTINE_FORM_FILL"
        _record_mode(rid)
        sys.exit(0)

    mode = EXECUTION_MODE
    if "--mode" in sys.argv:
        idx = sys.argv.index("--m
[truncated — 24585 more characters]
```

### coordinator/server.py

```python
"""
Shepherd Coordinator — the central relay for remote orchestration / governance /
observability.

Topology (star, not p2p): every operated machine runs Shepherd plus an *outbound*
relay client (`services/relay_client.py`) that dials INTO this coordinator. The
remote Command Center (the Next.js app) also connects here. The coordinator is
therefore the only component that needs a public URL — agents never expose an
inbound port.

Two socket roles:
  /agent  — an operated machine. Streams events + screen frames up; receives
            commands (intent / approve / halt / override / mode) down.
  /ui     — a Command Center browser. Receives the live agent roster, every
            agent's event stream, and the watched agent's screen frames;
            sends commands targeted at a specific agent.

Everything here is observability + control plumbing. It holds no automation
logic and actuates nothing itself.
"""
from __future__ import annotations

import json
import time
from collections import deque
from dataclasses import dataclass, field
from typing import Optional

import uvicorn
from fastapi import FastAPI, Request, WebSocket, WebSocketDisconnect
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import HTMLResponse, JSONResponse

from config import COORDINATOR_PORT, COORDINATOR_TOKEN, PROTOCOL_VERSION
from coordinator.catalog_store import load_catalog, load_catalog_version, save_catalog
from coordinator.title_gen import generate_title_async

# Reuse the agent's Deepgram transcription surface so the Command Center can turn
# a spoken command into an intent without a backend of its own.
try:
    from dashboard.deepgram_routes import router as deepgram_router
except Exception:  # pragma: no cover - optional
    deepgram_router = None

_AGENT_EVENT_HISTORY = 200


# ── Agent state ───────────────────────────────────────────────────────────────


@dataclass
class AgentConn:
    agent_id: str
    name: str
    host: str
    ws: WebSocket
    code: str = ""               # session / pairing code this agent belongs to
    online: bool = True
    status: str = "idle"          # idle | running | blocked | completed | failed
    mode: str = "LIVE"
    routine_id: Optional[str] = None
    run_id: Optional[str] = None
    step_index: Optional[int] = None
    total_steps: Optional[int] = None
    block: Optional[dict] = None   # populated while status == "blocked"
    last_activity: float = field(default_factory=time.time)
    last_frame: Optional[str] = None        # base64 JPEG
    last_frame_ts: float = 0.0
    catalog: Optional[dict] = None         # cached routines/workflows/task-graphs
    catalog_version: int = 0               # incremented on each catalog push
    history: deque = field(default_factory=lambda: deque(maxlen=_AGENT_EVENT_HISTORY))
    # Live workflow traversal state, built on the fly from workflow.* events so the
    # Command Center can render the milestone graph for this agent.
    workflow: Optional[dict] = None
    # Most recent ad-hoc dispatch routing decision (intent → workflow / autonomous),
    # surfaced so the operator can see what the vector router matched.
    routing: Optional[dict] = None
    # ── Fleet summary fields (issue #21) ──────────────────────────────────────
    # Async-generated human-readable title of what the agent is doing this run.
    title: Optional[str] = None
    # Whether title generation has already been triggered for this run.
    _title_requested: bool = field(default=False, repr=False)
    # The raw goal/intent text used as fallback if LLM title gen fails.
    _goal_text: Optional[str] = field(default=None, repr=False)
    # Recent step descriptions (last 3) for the "peek" in the fleet list.
    recent_steps: list = field(default_factory=list)
    # Live execution-trace graph, built on the fly from step.* events for runs that
    # are NOT following a saved workflow (autonomous goals / routines). This is the
    # granular "what is the agent actually doing right now" view; for a brand-new
    # task it is the trace being crystallized into a future workflow.
    trace: Optional[dict] = None

    def snapshot(self) -> dict:
        return {
            "id":               self.agent_id,
            "name":             self.name,
            "host":             self.host,
            "code":             self.code,
            "online":           self.online,
            "status":           self.status,
            "mode":             self.mode,
            "routineId":        self.routine_id,
            "runId":            self.run_id,
            "currentStepIndex": self.step_index,
            "totalSteps":       self.total_steps,
            "progress":         self._progress(),
            "block":            self.block,
            "lastActivityAt":   _iso(self.last_activity),
            "hasFrame":         self.last_frame is not None,
            "workflow":         self._workflow_view(),
            "routing":          self.routing,
            "title":            self.title,
            "recentSteps":      list(self.recent_steps),
            "trace":            self._trace_view(),
        }

    def _workflow_view(self) -> Optional[dict]:
        """Roster-safe view of the live workflow graph (no frames; the UI captures
        per-node screenshots client-side from the frame stream)."""
        if not self.workflow:
            return None
        wf = self.workflow
        return {
            "id":        wf.get("id"),
            "name":      wf.get("name"),
            "current":   wf.get("current"),
            "awaiting":  wf.get("awaiting", False),
            "nodes":     [wf["nodes"][k] for k in wf.get("order", []) if k in wf["nodes"]],
            "edges":     wf.get("edges", []),
            "status":    wf.get("status"),
            "baked":     wf.get("baked"),
            "finalize":  wf.get("finalize"),
            "finalized": wf.get("finalized"),
        }

    def _trace_view(self) -> Optio
[truncated — 31407 more characters]
```

[216 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]