Project Info

Shepherd

Mission control for AI agents. The agent is the part you cannot trust. Shepherd is the system that lets you run it anyway. An AI agent is clicking through real software right now. Shepherd is the reason you can let it: it watches every move, halts the dangerous one before it happens, and signs a tamper-proof record of the rest. 💡

Inspiration

We built an agent that could drive a desktop. Then we watched it do something we never asked it to, and we froze. Not because of the mistake. Because of the silence after it. There was no rewind, no record, no way to answer the only question that mattered: what did it just do, and why? The cursor had moved on its own, and the most honest thing we could say was a shrug. That shrug is the entire reason agents are stuck in demos. The capability curve went vertical and the trust curve never moved. Every month a model drives a computer better, and every month the gap widens between "the agent can do this" and "I would let it do this unwatched." The thing keeping agents out of the rooms where they would matter most, a clinic, a finance back office, a benefits desk, was never capability. It is that a black box touching a real machine is a liability nobody will sign. So we stopped trying to build a smarter sheep and built the shepherd. Every wave of computing earned a control plane the moment it touched things that mattered: networks got the firewall, the cloud got IAM, code got review. Agents that click around real machines are the next wave, and they have nothing standing watch. A shepherd does not cage the flock or walk every step beside it. It knows where the cliffs are, runs out front, and steps in at the exact moment before something goes over. That is the whole job, and nobody had built it. 🎛️

What it does

Shepherd is local mission control for AI desktop agents. You hand it work two ways: show it once and the demonstration becomes a reusable routine, or just say the goal out loud and let it figure the rest out. Then it drives the real desktop to get it done while the same watch stays on the entire time. We did not polish one scripted trick. We ran it across a deliberately messy spread to prove it generalizes: filling a job application and stopping cold at the credential field, drafting a candidate decision email and refusing to send it to an outside address with a secret in the body, taking a Photo Booth selfie hands-free, opening a YouTube video and putting on lo-fi, hunting down the cheapest flight to NYC and starting the booking, reading and pulling structured data off a live web page, and running an agent on a second machine across the network that we steered mid-task. A macro recorder cannot do that. The proof that this is an agent and not a script is that the same watch falls on a job nobody wrote down as it does on one we demonstrated. Open-ended capability, still on a leash. The watch is the part that matters, and it tightens as the stakes rise. A rule engine reads every screen in under a millisecond. The risky ones get a second, independent opinion from a separate model. The genuinely fraught ones go to a room full of specialists who argue it out. And anything still uncertain stops and asks a human, out loud, before it touches anything it cannot take back. Every move, including every stop, becomes a link in a tamper-evident chain you can verify in one click. Nothing the agent does is ever a shrug again. 🎬 Watch it happen You say "send the candidate decision email." Your voice becomes text, the intent resolves, and the agent opens the composer and starts typing on the real screen. The draft is thin, so a durable research worker quietly looks the candidate up on the live web and fills in the body. The agent reaches for Send. In under a millisecond the rule engine sees it: external recipient, secret in the body. HALT, a beat before the irreversible click. The Control Hub blooms with the milestone graph, the council's votes, the signed intent token, and the live screen, and the agent speaks the danger out loud and waits. You say "stop." It stops, exactly at the boundary, and the halt itself is signed into the chain. Then you change one line of policy.yaml, run it again, and the behavior changes instantly. That last part is the difference between a demo and a product. 🧱 The anatomy of the watch Shepherd is one system, not a coat of logos. Every layer below is load-bearing: pull one and a real capability goes dark. We built each in its intended shape and exercised every one live, never stubbed, never a screenshot of a logo. And the watch is not a black box itself. The crystallized task graph is a measured object: we treat the runs of a task as a Markov process over milestones, recover the single most-likely path with Viterbi, report each fork's Shannon entropy in bits so you can see exactly where judgment lives versus rote, score the structure with McCabe cyclomatic complexity, and lay it all out with the Sugiyama algorithm so a graph with real shape reads like one. That is the difference between a tool that says "trust me" and one that shows its work. 🔧

How we built it

The trick that made all of this fit together is one rule we refused to break: the click path is sacred. Nothing networked, async, or model-driven runs inside a step sequence, so a slow API or a stalled model can never strand the mouse mid-action. Everything else hangs off a single event bus, and every layer subscribes to it at the boundaries between steps. That one decision is why we could bolt a dozen systems onto a live agent without ever making it slower or shakier, and why the whole thing still runs offline with nothing configured: each layer degrades to a clean no-op when its service is gone. FastAPI underneath, a Next.js 16 Control Hub streaming live over WebSocket on top, and the sacred path in the middle. Observability & Reliability — Arize Phoenix + Sentry A desktop agent that clicks real buttons on your real machine is only trustworthy if you can see what it did and catch it when it goes wrong. We treat observability as a first-class feature, not an afterthought — and we used Arize Phoenix and Sentry as the two halves of that story: Phoenix to understand every decision, Sentry to catch every failure. Arize Phoenix — the agent's glass box Every run is a trace, and every step nests inside it, so we can replay an entire autonomous run decision-by-decision. We capture three layers for each turn: the planning step that records the prompt, the model's reasoning, and the action it chose — so you can see why the agent clicked where it did; the grounding step that turns a described target into precise screen coordinates; and the actuation step that fires the real click or keystroke. Because the grounding decisions are traced, we can pinpoint exactly which screenshot the model misread when a click lands in the wrong place. We also run evals on the agent's oversight verdicts, scoring its self-reported success against what the screen actually shows — turning "it said done" into a measurable signal. Sentry — reliability for an agent, not just an app We mapped Sentry's app-monitoring primitives onto agent behavior: Runs are performance transactions, with a child span per milestone, so latency regressions and slow steps surface like any production workload. Halts and human interventions become structured, queryable issues — tagged with the decision, trigger, and verifier verdict, with the halt screenshot attached. Silent failures get caught: a run that "completes" but actually stalled is reported as a dedicated event, so the failures that usually stay invisible become a ticket. Our favorite touch — Agent Session Replay. Every agent turn records the screenshot it saw, its reasoning, and the coordinates it clicked into a rolling buffer. The moment a run fails or is halted, Sentry doesn't just get a stack trace — it gets a frame-by-frame visual filmstrip of what the agent saw and where it clicked leading up to the failure, plus a manifest stitching reasoning to coordinates to outcome. It's Session Replay, but for a screen-driving agent — something Sentry was never designed for, mapped onto it perfectly. The two, connected Every Sentry issue carries a clickable link straight to the matching Phoenix trace. So the workflow is: Sentry pages you that a run halted → one click jumps to the full Phoenix trace → you replay every reasoning step that led there. Sentry turns the mystery into an alert; Phoenix turns the alert into a root cause. 🧗

Challenges we ran into

The platforms fought us, and that was the fun part. Several behaved nothing like their docs (an auth header that had to be X-API-Key not Bearer, a token field secretly named jwt_token, a session API shaped differently than advertised), and we verified every integration against the live service instead of trusting a README. A second package root quietly broke the Next.js build until we traced it to dueling lockfiles. And in the most on-brand bug of the weekend, a late review caught our own fleet view reporting a halted agent as "completed." A safety product silently passing off a stopped agent as a success is precisely the failure we exist to prevent, so we made the status fail safe and wrote the regression test that should have existed. Four humans and a coding agent, branches landing fast, main staying green on every push because CI, the audit log, and a pre-ship review pass kept us honest under the clock. 🌟

Accomplishments we're proud of

Twelve technologies, every one running live and pulling weight, composed into a single product instead of a pile of demos. A genuinely closed loop where the system's own evaluations change what gets watched next run. A council that actually deliberates. A tamper-evident ledger sitting next to a task graph with real math on it. A voice loop where the agent asks permission out loud before it acts. And all of it shipped clean under a hackathon clock: roughly 39,000 lines, 237 passing tests, green CI on every push, a security and code-review pass, and a dependency CVE cleared before we called it done. 📚

What we learned

Oversight is an architecture, not a feature you sprinkle on at the end. The event-bus seam is the whole reason watching, durability, memory, and observability could attach without coupling to the agent or slowing it down. Building every layer to fail gracefully turned out to be a feature, not a chore: it is what makes the thing both robust and demoable anywhere. A council of specialists that disagree catches what a lone verifier misses. And the deepest lesson is the simplest: the cursor moving is the hook, but the audit trail beside it is the product. Trust is the thing a clinic or a bank would actually pay for. 🚀

What's next

for The Shepherd There is going to be a layer like this, the same way there was always going to be a firewall. Every agent that touches a real machine will eventually run behind one, and we intend for it to be this one. Next we take Shepherd from hackathon to pilot in the rooms where trust is the blocker, ship a policy marketplace so teams share governance the way they share lint configs, build audit export for regulators, and extend the one watch across new surfaces: mobile, full browser fleets, whole organizations of agents. The agents will keep getting the headlines. The layer that lets them into production is the business. Built by four people and a Devin teammate at the UC Berkeley AI Hackathon. Local-first. Model-agnostic. Honest enough to hand the audit log to a regulator. The cursor moving is the hook. The audit trail beside it is the product.

Shepherd

Mission control for AI agents. The agent is the part you cannot trust. Shepherd is the system that lets you run it anyway.

🐑 Local-first · 🛰️ Operate any machine · 🔗 Tamper-evident audit · 🗣️ Voice in, voice out · ⚡ Sub-ms policy gate

We built an agent that could drive a desktop. Then we watched it do something we never asked it to, and the scary part was not the mistake. It was the silence after: no rewind, no record, no way to answer the only question that mattered, what did it just do, and why? The cursor had moved on its own, and the most honest answer we had was a shrug.

Teams are shipping agents like that into production and quietly hoping for the best. Hoping is not a control.

Shepherd is local mission control for AI desktop agents. A shepherd does not cage the flock or walk every step for it. It knows the dangerous ground, runs out front, and steps in at exactly the right moment. That is the job: you teach a task by demonstrating it once, Shepherd drives the real desktop to carry it out, governs every high-stakes step in real time, catches it the instant it strays, replays precisely what it did while you were away, and runs it on a machine across the country. The sub-millisecond policy gate, the independent AI verifier, the human decision point, and the tamper-evident audit trail are one system, not features bolted on.

Shepherd is not a safety wrapper around someone else's agent. The agent is the engine inside it. Everything that turns raw capability into something you can actually deploy, the teaching, the governance, the memory, the audit, the remote control, is Shepherd.

⚡ Why now

The agent-capability curve went vertical and the trust curve did not move. Every month a new model can drive a computer better, and every month the gap widens between "the agent can do this" and "I would let it do this unattended." The blocker to deploying agents in anything that matters (a clinic, a finance back office, a benefits desk) is not capability. It is that a black box touching a real machine is a liability nobody can sign off on. Shepherd is the missing control plane: the thing that turns "impressive demo" into "approved in prod."

There is going to be a layer like this. Every agent that touches a real machine will eventually run behind one, the same way every packet eventually ran behind a firewall. We are building that layer to be the obvious one: local-first, model- agnostic, fast enough to be invisible, and honest enough that you can hand the audit log to a regulator. The agents get the headlines. The layer that lets them into production is the business.

⏱️ The demo in 90 seconds

flowchart TD
    V["🗣️ You say: send the candidate decision email"] --> STT["Deepgram transcribes · router resolves the intent"]
    STT --> AS["Agent S opens the mail composer<br/>Simular drives the real desktop"]
    AS --> RES["Missing context, so a durable Agentspan agent<br/>researches the candidate on the live web Browserbase<br/>and drafts the body"]
    RES --> SEND(["The agent moves to Send"])
    SEND --> POL["⚡ Policy engine, under 1ms:<br/>external recipient + secret in the body"]
    POL -->|"Verdict: HALT"| GATE["🛑 Stopped before a single irreversible click"]
    GATE --> HUB["🎛️ Control Hub lights up<br/>milestone graph · verifier second opinion · live screen"]
    HUB --> HUMAN{"You approve,<br/>or say stop out loud"}
    HUMAN -->|approve| GO["▶️ It continues"]
    HUMAN -->|stop| STOP["🛑 Halts at the boundary"]
    GO --> AUDIT["🔗 Every action, including the halt,<br/>signed into a SHA-256 hash chain"]
    STOP --> AUDIT

    classDef danger fill:#bb4a3a,stroke:#bb4a3a,color:#fff;
    classDef gate fill:#cf6a43,stroke:#cf6a43,color:#fff;
    classDef proof fill:#223b3a,stroke:#223b3a,color:#fff;
    class GATE,STOP danger;
    class POL,HUMAN gate;
    class AUDIT proof;

Change one rule in data/policy.yaml, re-run, and the behavior changes immediately. That is the difference between a demo and a product.

🎛️ What you get (the Control Hub)

A single console, not a pile of scripts. Every tab is live over WebSocket.

Command Center: dispatch a task by voice or text, watch the live execution graph light up milestone by milestone, approve or halt at the gate.
Remote: operate another machine. Its live screen (WebRTC peer-to-peer) beside the workflow graph it builds as it goes. Steer or teach it inline.
Routines: the recorded "tools," each a demonstration plus per-step notes.
Task Graph: the crystallized milestone DAG a task has learned across runs, with the modal execution path, branch entropy, and transition odds drawn on it.
Workflows: the dispatchable, versioned workflows, including the judgment calls baked in from your steers.
Runs: full scrubbable replay of any past run, step by step.
Interventions: the queue of moments that needed a human, and how they resolved.
Audit: the tamper-evident hash chain, verifiable in one click.
Policy: the live governance rules.
Voice Lab: the Deepgram transcription surface.

⚙️ How it works

flowchart TD
    IN["🗣️ Voice / typed intent"] --> STT["Deepgram STT · boundary only"]
    STT --> ROUTER["Intent router<br/>Redis vector search BAAI/bge + keyword fallback"]
    ROUTER --> ENGINE["Execution engine<br/>autonomous plan, or routine replay vision/deterministic<br/>Agent S drives the desktop"]
    ENGINE -->|"at high-stakes boundaries only, never mid-click"| STACK

    subgraph STACK["🛡️ 3-layer oversight stack, fastest first"]
        direction TB
        L1["1 · Policy engine<br/>rule-based, always on, under 1ms · data/policy.yaml"]
        L2["2 · Verifier<br/>independent Claude, or a Band council"]
        L3["3 · Human gate<br/>approve / halt / steer · spoken stop fires the same halt"]
        L1 --> L2 --> L3
    end

    STACK --> AUDIT["🔗 Tamper-evident audit log<br/>SHA-256 hash chain"]
    STACK --> TEL["📊 Telemetry<br/>Arize Phoenix spans"]
    STACK --> MEM["🧠 Memory + crystallization<br/>Redis recall · runs coalesce into workflows"]
    AUDIT --> HUB["🎛️ Control Hub Next.js<br/>+ Remote Command Center"]
    TEL --> HUB
    MEM --> HUB

    classDef gate fill:#cf6a43,stroke:#cf6a43,color:#fff;
    classDef proof fill:#223b3a,stroke:#223b3a,color:#fff;
    class STACK gate;
    class AUDIT proof;

A single rule runs through the whole codebase: the click path is sacred. Nothing async, networked, or ML-based runs inside a routine's step sequence. Every integration is invoked between steps, at boundaries, so a network hiccup or a model stall can never strand the mouse mid-action.

🏆 The five things that make it a product, not a demo

1. 🎬 Two ways to hand it a task: show it once, or just say it

For a task you repeat, demonstrate it once and the demonstration becomes the routine. No node graph, no Zapier-style flowchart. The agent works in the same workspace you do, so doing the task once is the authoring step, and each spoken instruction attaches to the step it was said at ("when you reach the credential field, stop and ask me").

python main.py --record ROUTINE_FORM_FILL   # Cmd+Shift+M marks a step, Cmd+Shift+Q stops

For anything else, switch to AUTONOMOUS and just say the goal. There is no recorded routine and no script: the planner drafts a plan from your words and Agent S drives whatever app the goal needs. "Take my selfie from Photo Booth." "Play lo-fi on YouTube." "Find the cheapest flight to NYC and start the booking." It opens the app, reads the screen, and goes, recording a per-goal milestone graph of the run (which it can recall to plan faster next time when AUTONOMOUS_USE_MEMORY is on). This is the difference between a macro recorder and an actual agent: it generalizes to tasks nobody scripted. And the part that matters most, the same oversight stack watches an autonomous run exactly as it watches a recorded one. Open-ended capability, still on a leash.

2. 🛡️ The oversight stack catches it before it costs you

Defense in depth, fastest first, each layer independent so a miss in one is caught by the next.

Pre-flight, before the first click: cryptographic intent authorization. When ArmorIQ is on, the resolved plan is captured at the run boundary and ArmorIQ issues a cryptographically-signed intent token gated by an allow/deny policy derived from the same containment rules. The plan is authorized as a whole before a single action runs, and the signed token is intent-level proof that sits beside the audit chain. A tenant-policy denial halts the run before it starts.

Per high-stakes screen, under a millisecond: the rule-based policy engine. It evaluates credential fields, captchas, phishing, and outbound sends, enforces app and domain containment, and rate-limits the run, all from a hot-reloaded data/policy.yaml.

On a flag, a genuinely independent second opinion. A separate Claude verifier re-examines the screen and can upgrade a flag to a halt or stand down. This is not just an in-process call: when Band is on, the second opinion is a real two-agent collaboration over Band's agentic mesh: the engine posts the flagged action into a shared room, an independent shepherd-verifier agent reasons and replies with its verdict, and the engine reads it back (live round-trip verified). If Band is offline it degrades to the identical in-process check.

Anything still flagged waits for a human, who can approve, halt, or steer with a natural-language override. Saying "stop" out loud fires the same halt path. Two demo flows ship today: a job application that halts at the credential field, and an email that halts before sending to an external recipient with a secret in the body. Both are governance moments, not form-fillers.

3. 🧠 It learns: runs crystallize into reusable workflows

Most agent tools forget a run the second it ends. Shepherd remembers, at the altitude a person thinks at. Off the hot path, a finished run is coalesced into a crystallized workflow: hundreds of fine clicks collapse into the handful of milestones a human would narrate (open, research, fill, submit), and the graph thickens across runs, so the second time you ask, the agent already knows the road. When you steer a run, you can remember it: the steer is baked in as a conditional, and a finalize gate at run end lets you persist it, save it as a new workflow, or discard it. The judgment calls future agents inherit are an explicit decision, not a silent side effect.

That crystallized graph is not a doodle, it is a measured object. We treat the runs as a Markov process over milestones: every edge carries its transition probability from how often the agent took it, and Viterbi (max-product dynamic programming) recovers the single most-likely path through the whole workflow, which the UI highlights so you can see at a glance what "normally" happens. Each decision point reports its Shannon entropy in bits, the honest measure of how branchy a choice really is (0 bits is deterministic, 1 bit is a true coin flip), and branch and merge points fall out of the in/out degree. The layout itself is the Sugiyama algorithm: nodes are assigned to layers by longest-path rank, then ordered with iterated median / barycenter sweeps to minimize edge crossings, so a graph with real structure reads as one. The whole analysis is in frontend/src/lib/graph-analysis.ts. The point is not the math for its own sake: it is that you can look at a learned task and immediately tell the rote part from the part where judgment lives.

The oversight learns too. Every step accumulates success / failure / halt / deviation stats across runs, surfaced as per-step confidence in the UI, and a step that keeps proving risky is auto-promoted into the monitored set, so the safety net tightens exactly where reality showed it should. And in LIVE mode, deviation detection compares what Agent S actually did against the demonstration and flags drift the moment the agent goes off-script, before it compounds.

4. 🛰️ Operate any machine, with no inbound ports

Shepherd runs an agent on a machine across the country as easily as on your own. The operated agent dials out to a coordinator relay (one outbound connection, no inbound ports, no VPN), and a remote Command Center watches its live screen over WebRTC peer-to-peer beside the workflow graph it builds in real time. You dispatch ad-hoc tasks and see exactly how the vector router resolved them, and you are never a spectator: steer, suspend, and resume an autonomous run mid-flight, inject a new instruction between steps, pause it cold, then let it pick up where it left off, or teach it a better move that bakes into the workflow. Deploy the relay anywhere with a one-command Cloudflare Tunnel. The full remote-operation and theoretical peering model is in docs/PEERING.md.

5. 🧩 Built on real agent infrastructure, not glue

The execution engine is Simular's Agent S planning against your demonstration. The research digression is a genuine Agentspan (Orkes) agent that compiles into a durable workflow on a self-hosted server, reasons, and calls a tool, leaving a queryable execution behind. The oversight verifier can be a separate Claude agent on Band's mesh. Run authorization is a real ArmorIQ intent token. Three things run on Redis: vector intent routing (BGE embeddings over a Redis 8 vector set), agent replay memory, and a semantic LLM cache that hits by meaning, not by key so a paraphrased goal reuses a prior milestone segmentation. Observability is real OpenTelemetry into Arize Phoenix. Every one of these was exercised live during the build, not stubbed: none of it is a screenshot of a logo.

🧱 The anatomy of the watch

Shepherd is one system, not a coat of logos. Every layer below is load-bearing: pull one and a real capability goes dark. We built each in its intended shape and ran every one live, never stubbed, never a screenshot of a logo. The Status column is honest about what each takes to light up (Core always on, On by default, Key-gated needs a credential, Off by default, or Build-time), and every one degrades gracefully when its service is gone.

Layer	Status	What it is, and why the product needs it
Simular (Agent S + SimuLang)	Core	The execution engine, the only code that actuates. Real `gui-agents` AgentS3 (`engine/agent_s_adapter.py`): it plans each LIVE and autonomous action from a screenshot and drives the desktop via pyautogui. The cursor moving on its own is Agent S. Two Simular products composed in their intended shape: once a task is learned, Shepherd graduates it into a deterministic SimuLang script (`services/simulang_runner.py`) that replays off the accessibility tree with zero LLM tokens per run (compiled `.ts`, run with `npx tsx` against the real `@simular-ai/simulang-js` native runtime), with Agent S vision as the explorer and the fallback. Agent S learns; SimuLang replays cheaply and auditably. Nothing else here clicks.
Anthropic / Claude	Core	The cognitive layer. Claude is the independent verifier (`services/verifier.py`) and the autonomous routine planner (`engine/routine_planner.py`), and the model behind the Agentspan researcher. It can also drive milestone segmentation and the Agent S planner, but those are provider-configurable (Gemini is the default segmenter to conserve budget; the Agent S provider is set per-config). The deployability thesis (agents in health, public services, finance) rests on this oversight.
Arize Phoenix	On by default	Not just observability, a closed loop that improves the agent. The base layer is real OpenTelemetry: spans on every run, plan, action, and workflow node (`routine.run → agent_s.plan → action.N`) with OpenInference I/O on LLM/TOOL spans. On top, `services/phoenix_evals.py` runs an LLM-as-judge over each oversight decision, writes the score back onto the Phoenix span (the Annotations panel), and feeds it into adaptive risk: steps the judge repeatedly calls genuine risks get auto-promoted into the monitored set, so Phoenix evaluation data literally changes which steps require human approval next run. `scripts/phoenix_experiment.py` produces the before/after "oversight precision" table. Off the click path; degrades to no-op spans if Phoenix is down. `./scripts/serve_phoenix.sh` → http://localhost:6006
Redis	On if running	Redis is the agent's memory, well beyond caching, all on Redis 8 vector sets (`VADD`/`VSIM`): (1) cross-run semantic recall (`services/run_memory.py`) embeds every finished run and, when a new goal arrives, recalls the most similar successful prior run by meaning so a reworded goal ("submit my Acme application" vs "apply to the Acme job") reuses a proven milestone chain instead of planning from scratch; (2) vector search for intent routing; (3) a semantic LLM cache that skips repeat milestone segmentation by meaning; plus agent replay memory and the adaptive-risk signal. Three distinct AI uses of the same primitive. Off the click path; every one degrades gracefully (keyword routing, fresh planning, heuristics) so the system runs fine without Redis.
Deepgram	Key-gated	A full hands-free voice oversight loop, not voice-as-input bolted on. STT three ways (speak the intent, narrate per-step instructions while recording a demonstration, say "stop" to halt mid-run) AND Aura TTS talking back: at a high-stakes gate the agent speaks the flagged action out loud ("Heads up. About to send an email to an external address containing a secret. Approve, or stop?") and takes your spoken approve/stop, racing alongside the on-screen gate (halt words win, safety-first). Voice is the control plane for supervising a desktop agent eyes-free. Real `deepgram-sdk`; falls back to the on-screen gate when `DEEPGRAM_API_KEY` or audio is unavailable.
Browserbase	Key-gated	The agent's hands on the open web, powered by Stagehand. `services/stagehand_web.py` drives a real Browserbase cloud Chrome by natural language (`act` / `extract` / `observe`) instead of brittle selectors (verified live: extracted a page heading in ~8s), with the raw-CDP + Playwright path as the deterministic fallback. Every URL it touches passes the same containment/SSRF guard as the desktop. And the live cloud browser is embedded, interactive, in the Control Hub (`debuggerFullscreenUrl`): you watch the agent work the web in real time and, on a halt, click in and take control before approving. The web is a second body under the one oversight stack. Degrades to a local value when offline/unconfigured.
Orkes / Agentspan	Server-gated	Durability, two ways. (1) The research digression is a real Agentspan agent: `shepherd-researcher` compiles into a durable workflow on a self-hosted Agentspan server, reasons, and calls a `fetch_page` tool (Browserbase under the hood), so a flaky research step retries/replays instead of failing the run. (2) Every run is durably checkpointed milestone by milestone (`services/agentspan_durable.py`, off the click path): if the process is killed mid-task, the next boot detects the orphaned run and re-dispatches it instead of silently abandoning the work, with a durable record of how far it got. Reuses the Anthropic key; degrades to a direct page read when the server is unreachable and to a no-op without Redis.
ArmorIQ	Key-gated	Intent authorization in front of the click path. At the run boundary Shepherd captures the resolved plan and ArmorIQ issues a cryptographically-signed intent token gated by an allow/deny policy derived from `data/policy.yaml`; a denial halts the run before the first action. Verified live: at the boundary ArmorIQ issues a real signed JWT intent token (carrying the plan hash and step proofs), confirmed end to end with a live key. No-ops cleanly when `ARMORIQ_API_KEY` is unset.
Sentry	DSN-gated	The production reliability backbone, not just error capture. Every run is a Sentry performance transaction with a child span per milestone (so p95 step latency and slow-step flame views come for free), and every halt / intervention is a structured, queryable issue: tagged by decision / trigger / verdict, with the screenshot attached and a clickable `phoenix.trace_url`. An operator can open a dashboard filtered to `op:shepherd.run` and see halt-rate, failure-rate, and slow steps. Driven off the event stream (off the click path); a daemon-thread-safe scope and `traces_sample_rate=1.0`. No-op without `SENTRY_DSN`.
Band (band.ai)	Off by default	A live multi-agent oversight council, verified end to end. On an uncertain high-stakes flag the engine (as `shepherd-monitor`, the chair) posts the action into a Band room and a panel of independent specialist agents on Band's mesh, `shepherd-security`, `shepherd-privacy`, `shepherd-destructive`, each judge it from their own lane and reply with a VOTE, even @mentioning each other to escalate. The chair tallies the votes (any halt wins) into the human gate. It is Band's own DevSquad / Drafter-Reviewer pattern mapped onto Shepherd's oversight: genuine N-agent deliberation, not a single call. Boundary-only, roughly 5-second round-trip against the real Agent API. Off by default: with the council unconfigured it gracefully collapses to one `shepherd-verifier`, and with Band off entirely the identical verdict comes from the in-process Claude verifier.
Cognition / Devin	Build-time	A coding agent used during development, not a runtime dependency, no Devin code runs in production. `devin-ai-integration[bot]` authored or co-authored several merged feature branches (WebRTC P2P remote, the live execution-trace graph, fleet session summaries), reviewed via Devin Review and resolved in-branch before merge.

Every layer is feature-flagged and degrades to a clean no-op when its service is absent, so with everything off the core automation and Control Hub still run fully offline. The only two you cannot remove are the hands (Agent S) and the judgment (a planner/verifier model).

📋 Governance policy

data/policy.yaml is the whole oversight contract, hot-reloaded on every evaluation. No code change, no restart.

screen_rules:
  - name: no-credentials
    match_text: ["password", "api key", "secret key", "ssn"]
    action: halt
    reason: "Credential or sensitive data field detected"

triggers:            # deterministic, planted demo triggers (no OCR needed)
  credential: halt
  external_send: halt
  captcha: halt
  payment: flag

containment:
  allowed_apps: ["Google Chrome", "Safari", "TextEdit"]
  allowed_domains: ["localhost", "workbridge.com", "github.com"]
  max_actions_per_minute: 60
  max_steps_per_run: 100

🔗 Audit log

Every action is appended to a SHA-256 hash chain. Change one byte anywhere and verification pinpoints the break.

curl localhost:8765/api/audit/verify
# {"valid": true, "entries": 47, "tampered_at": null, "reason": "chain intact"}

🎚️ Execution modes

Two un-bundled knobs decide how an intent is handled (USE_ROUTER / ROUTINE_REPLAY):

USE_ROUTER=false (default), skip routing; every intent runs as a free-form autonomous Agent S goal (an LLM drafts a plan from the raw words, then Agent S executes it screenshot-by-screenshot). Prior memory is not consulted unless AUTONOMOUS_USE_MEMORY=true, off by default, so each run plans fresh.
USE_ROUTER=true, match a saved workflow/routine first, falling back to autonomous on no match. ROUTINE_REPLAY picks how a matched routine is driven: vision (Agent S plans against the demonstration, was LIVE) or deterministic (verbatim coordinate replay, the offline floor, was LOCKED).

These derive the legacy LIVE/LOCKED/AUTONOMOUS enum internally. Switch at runtime with POST /api/mode/<MODE> or the Control Hub sidebar (no restart), a runtime override wins for the live process until changed.

🚀 Quick start

# 1. Install (Python via uv, Node for the Control Hub)
uv sync
cd frontend && npm install && cd ..

# 2. Configure: copy .env.example to .env. ANTHROPIC_API_KEY is the only
#    must-have. Every other key degrades gracefully if absent.
cp .env.example .env

# 3. Supporting services (all optional, all degrade gracefully)
redis-server                 # vector routing, memory, semantic cache
./scripts/serve_phoenix.sh   # developer traces at http://localhost:6006
agentspan server start       # durable research agent (open-source, keyless)
# Optional: SENTRY_DSN in .env → errors cross-linked to Phoenix traces

Then run it, either way:

# A) One command, everything (backend + agent + frontend)
./scripts/dev.sh

# B) Persistent backend, agents come and go (recommended for a long session)
uv run python -m dashboard.server                 # http://localhost:8765
cd frontend && npm run dev                         # http://localhost:3000
BACKEND_URL=http://localhost:8765 uv run python main.py

Open http://localhost:3000 and speak or type an intent.

📊 Observability (Phoenix + Sentry)

Both are optional and off the click path.

# Phoenix: live OTel traces (no API key for local)
./scripts/serve_phoenix.sh          # Terminal 1 → http://localhost:6006
uv run python main.py               # Terminal 2

# Sentry: add to .env, then failed runs link back to Phoenix
SENTRY_DSN=https://xxx@oXXX.ingest.sentry.io/XXX

Sentry events include Contexts → phoenix → trace_url (clickable) and tag phoenix.trace_id. The project slug in trace URLs is resolved automatically via Phoenix GraphQL (getProjectByName).

Implementation: telemetry/telemetry.py, telemetry/agent_trace.py, telemetry/phoenix_client.py, telemetry/sentry_init.py.

🗂️ Project layout

main.py            Entry loop: intent -> router -> engine -> telemetry + memory
config.py          Typed settings + feature flags (all from .env)
router/            Intent router: Redis vector search + deterministic keyword fallback
engine/            Execution core (the only code that actuates): recorder, task graph,
                   coalescer, milestones, routine planner, workflow executor/store/edit,
                   trace journal, Agent S adapter + grounding
services/          Boundary integrations: monitor, policy, verifier, deepgram,
                   browserbase, band, agentspan research agent, coordinator relay client
telemetry/         Phoenix OTel spans, Sentry capture, hash-chain audit, Redis replay
dashboard/         FastAPI: REST + WebSocket event stream
coordinator/       Remote relay + WebRTC signaling so a Command Center can watch + steer
frontend/          Next.js Control Hub
docs/              PEERING.md, PROTOCOL.md, workflow-engine.md
data/              routines.json, policy.yaml, workflows, demo target pages

🎯 The bet

Every wave of computing got a control plane once it touched things that mattered. Networks got the firewall. Cloud got IAM. Code got CI and code review. Agents that click around real machines are the next wave, and right now they have nothing: no gate, no record, no proof, just a cursor moving and a shrug. That absence is the single biggest thing keeping capable agents out of the rooms where they would be worth the most.

Shepherd is the control plane for that wave. We did not build a safer agent. We built the layer that makes any agent safe enough to deploy, and we built it local-first, model-agnostic, and fast enough to disappear, so it wins on the one axis that compounds: the more agents the world ships, the more they all need this. Every piece in this repo runs today. We exercised every integration live, signed every action into a hash chain, and put the whole thing behind one console a non-engineer could operate. This is not a prototype of the idea. It is the idea, working.

The cursor moving is the hook. The audit trail beside it is the product.

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CSSIn code
FastAPIIn code
HTMLIn code
JavaScriptIn code
Next.jsIn code
PythonIn code
ReactIn code
RedisIn code
Tailwind CSSIn code
TypeScriptIn code
Google GeminiClaimed
SQLClaimed

10 of 12 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

Claude CodeCommits
CursorCommits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

1.7 MB

Source files

232

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Project Info

Shepherd

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments we're proud of

What we learned

What's next

Shepherd

Mission control for AI agents. The agent is the part you cannot trust. Shepherd is the system that lets you run it anyway.

⚡ Why now

⏱️ The demo in 90 seconds

🎛️ What you get (the Control Hub)

⚙️ How it works

🏆 The five things that make it a product, not a demo

1. 🎬 Two ways to hand it a task: show it once, or just say it

2. 🛡️ The oversight stack catches it before it costs you

3. 🧠 It learns: runs crystallize into reusable workflows

4. 🛰️ Operate any machine, with no inbound ports

5. 🧩 Built on real agent infrastructure, not glue

🧱 The anatomy of the watch

📋 Governance policy

🔗 Audit log

🎚️ Execution modes

🚀 Quick start

📊 Observability (Phoenix + Sentry)

🗂️ Project layout

🎯 The bet

Analysis

Technology

AI coding agents

Codebase size

Structure

Interface

API & routing

Application logic

Languages

Dependencies