# Project export: Clara This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: We put agents to work. - Devpost: https://devpost.com/software/clara-mkjrny - GitHub: https://github.com/Preetam3620/clara.git - Demo: https://drive.google.com/file/d/1WGdoknMpcSWPTncsLxUkBLSgwIzAZR6b/view?usp=drivesdk - Video: https://www.youtube.com/embed/abuw00WAiMY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 0 GitHub contributor(s) — ## Devpost submission (written by the team) ### Overview Elevator pitch Clara is the agent that lives on your machine and gives every other AI agent a body. ### Inspiration There's a swarm of AI agents out there now, each one good at one narrow thing. But none of them actually live anywhere. We wanted an agent that sits on your own computer, has context and control over your machine, and can pull in any of those external agents and put them to work for you. ### What it does Clara gives you full access to your laptop through iMessage (via Poke). You text it like you'd text a friend: order groceries on Amazon, book a table on Yelp, whatever. Behind the scenes, a classifier figures out what you actually want, builds a plan, and hands it off to a specialist agent in the swarm. For Amazon orders, Clara builds the cart and shows it to you in chat before doing anything. You approve or reject, then it executes. For restaurants, you get a list of options right in iMessage, pick one, and Clara finishes the reservation on your desktop. Under the hood, this is a real swarm of agents running on Fetch.ai's Agentverse, not one bot pretending to be many. ### How we built it Three pieces: An Express + MCP middleware pipeline that talks to Poke A Fetch.ai uAgents orchestrator that handles intent classification and routing An Electron app on the desktop that handles automation and cowork Intent classification runs through ASI:One's LLM. There's a dashboard with access to the Agentverse marketplace, so you can connect any agent you want into the swarm, plus a dummy fallback agent so the demo never dies if ASI is down. The orchestrator routes to downstream specialist agents over the Agent Chat Protocol. Redis backs state and trust scoring, with a disk fallback, so the system always has context on what's happening on your desktop. MCP runs over StreamableHTTP so Poke can talk to it directly. ### Challenges we ran into Bridging a text thread with a desktop. iMessage is built for sending words back and forth, not for telling a laptop what to do. Getting Poke to talk to the Electron app cleanly, so a casual text actually triggers something real on your machine, took more wrangling than it looked like it should. Giving the AI enough context about your desktop without giving it everything. The agent needs to know what's going on on your machine to act on your behalf, but dumping your whole desktop at it is messy and a little unnerving. We had to figure out what context actually mattered and feed it just that, without losing the thread of what you're doing. Keeping multiple agents talking without crossing wires. Once you've got a swarm instead of one bot, you have to make sure a reply from the Amazon agent doesn't get matched to a restaurant request that came in five seconds later. Sounds simple until two requests are in flight at once. Making sure a demo never just dies. Any one piece, ASI, Redis, the executor, could go down mid-demo. We built the whole thing to fail soft instead of fail loud, so even if something upstream broke, Clara still looked like it worked. ### Accomplishments we're proud of Plug in any agent from the Agentverse marketplace, not just the ones we built. Clara isn't locked to a fixed set of skills. Open the dashboard, connect an agent from the marketplace, and it's part of your swarm. The system was built to grow past Amazon and Yelp from day one. Hands free control of your desktop with voice. You can talk to your machine like you'd talk to a person and have it actually do something, no keyboard, no clicking through menus. That's the part that made people in the room go quiet for a second. A purchase never happens without you seeing it first. Even with agents acting on your behalf, nothing gets bought or booked blind. Clara shows you the cart, you say yes or no, and only then does it execute. We wanted autonomy without giving up control. ### What we learned On working with Fetch.ai/uAgents and ASI:One: The agent protocol doesn't hand you anything like a request ID to match a reply against. We assumed agent-to-agent communication would feel like calling an API and getting a response back. It's closer to mailing a letter and hoping it comes back in the order you sent it. ASI:One's classification itself worked well out of the box. The surprise was how much hand-built plumbing it took to make agent replies land back in the right place. On giving an AI control over your desktop: Most software adds a button. This adds judgment. The hard part wasn't teaching Clara to act on your machine, it was deciding what she should never be allowed to do without asking first. Control turned out to be a design problem, not an engineering one. On the word "agentic" itself: Building an actual swarm of agents made us a lot more skeptical of how that word gets used in pitch decks. The real work is unglamorous: matching replies to requests, building fallbacks for when a service goes down, deciding who owns state when three agents are talking at once. None of that is exciting to demo, but it's most of what makes the thing actually work. ### What's next More agents, more use cases. Amazon and Yelp were the proof of concept. The marketplace model means the next agent someone plugs in could be anything, travel booking, calendar management, your own weird side project. The sky's the limit once the swarm is the product instead of the two skills we shipped. Open sourcing the connector. We want anyone to be able to build their own specialized agent and wire it straight into Clara, instead of waiting on us to add it. That's the actual unlock here. Clara stops being our project and becomes whatever the community plugs into it. Beyond iMessage, and real payments. Texting was the fastest way to prove the idea worked. Long term, more channels and an actual payment flow, instead of a cart you just approve and someone else checks out, turns this from a demo into something you'd trust with real money. ## README (from the GitHub repository) # Clara [Demo conversation](https://asi1.ai/shared-chat/a814d20a-7e35-4621-9072-c37c3a6c6d72) An agentic desktop assistant stack — a 3D avatar that controls your computer, a phone-based approval gate, a Fetch.ai intent router, and a live control dashboard. ## Components | Folder | Name | What it does | |--------|------|--------------| | `eletron_app/` | **Shadow** | Electron app + Python sidecar. Type a task; an animated 3D avatar drives your Mac via Claude computer-use, then shows you a screenshot and narrates what it did. | | `poke_middleware/` | **Deadbolt** | TypeScript middleware bridging your phone (Poke), the Fetch.ai classifier, and Shadow. Every action requires your phone approval before it executes. | | `fetch_agents/` | **Orchestrator** | Fetch.ai (uAgents) service. Classifies natural-language intent with ASI:One LLM and routes to downstream specialist agents (e.g. Amazon grocery). | | `dashboard/` | **Dashboard** | Vite + React control plane. Browse the Agentverse marketplace, inspect wired routes, and watch the live intent feed. | ## How it fits together ``` Phone (Poke) → Deadbolt middleware → Orchestrator (/classify) → downstream agents ↓ approval gate (Fetch.ai / Amazon …) Shadow (executor) ↓ 3D avatar + screenshot ``` Deadbolt captures the intent, asks the orchestrator to plan it, sends the plan to your phone for approval, then forwards approved instructions to Shadow. Shadow executes them step-by-step and returns a verification screenshot. ## Quick start Each component has its own README with full setup instructions. ```bash # 1. Shadow (Electron desktop app) cd eletron_app && npm install && npm run dev # 2. Orchestrator (Fetch.ai agent) cd fetch_agents && pip install -r requirements.txt && python -m orchestrator.agent # 3. Deadbolt middleware cd poke_middleware && npm install && npm run up # 4. Dashboard cd dashboard && npm install && npm run dev # http://localhost:5273 ``` ## Requirements - macOS, Node 20+, Python 3.10+ - `ANTHROPIC_API_KEY` — Shadow (computer-use) - `ASI_ONE_API_KEY` — Orchestrator (intent classification) - `POKE_API_KEY` + `REDIS_URL` — Deadbolt (phone gate + state) ## Detected evidence (automated analysis) Indexed codebase: 85 recognized source files, 380 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - Express (technology) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Redis (technology) — detected in the code - TypeScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (110 of 110) ``` dashboard/.gitignore dashboard/index.html dashboard/package.json dashboard/README.md dashboard/src/api.ts dashboard/src/App.tsx dashboard/src/main.tsx dashboard/src/styles.css dashboard/src/types.ts dashboard/tsconfig.json dashboard/vite.config.ts eletron_app/.env.example eletron_app/.gitignore eletron_app/agent/actions.py eletron_app/agent/agent_runner.py eletron_app/agent/anthropic_patch.py eletron_app/agent/config.py eletron_app/agent/converse.py eletron_app/agent/grounding_check.py eletron_app/agent/http_server.py eletron_app/agent/main.py eletron_app/agent/native_engine.py eletron_app/agent/requirements.txt eletron_app/agent/screenshot.py eletron_app/agent/smoke_test.py eletron_app/agent/verify.py eletron_app/CLAUDE.md eletron_app/electron.vite.config.ts eletron_app/package.json eletron_app/README.md eletron_app/scripts/tunnel.sh eletron_app/src/main/index.ts eletron_app/src/preload/index.d.ts eletron_app/src/preload/index.ts eletron_app/src/renderer/index.html eletron_app/src/renderer/src/App.tsx eletron_app/src/renderer/src/components/Chat.tsx eletron_app/src/renderer/src/components/Composer.tsx eletron_app/src/renderer/src/components/EmotionGallery.tsx eletron_app/src/renderer/src/components/HelpDial.tsx eletron_app/src/renderer/src/components/Sunny.tsx eletron_app/src/renderer/src/components/TaskCard.tsx eletron_app/src/renderer/src/components/VoiceToggle.tsx eletron_app/src/renderer/src/hooks/useChat.ts eletron_app/src/renderer/src/hooks/useHelpMode.ts eletron_app/src/renderer/src/index.css eletron_app/src/renderer/src/ipc.ts eletron_app/src/renderer/src/main.tsx eletron_app/src/renderer/src/vite-env.d.ts eletron_app/src/renderer/src/voice/useVoice.ts eletron_app/tsconfig.json eletron_app/tsconfig.node.json eletron_app/tsconfig.web.json fetch_agents/.env.example fetch_agents/.gitignore fetch_agents/FETCHAI.md fetch_agents/orchestrator/__init__.py fetch_agents/orchestrator/agent.py fetch_agents/orchestrator/chat_utils.py fetch_agents/orchestrator/intent.py fetch_agents/orchestrator/models.py fetch_agents/orchestrator/routing.py fetch_agents/orchestrator/session.py fetch_agents/README.md fetch_agents/requirements.txt fetch_agents/restaurant_agent/__init__.py fetch_agents/restaurant_agent/agent.py fetch_agents/restaurant_agent/chat_utils.py fetch_agents/restaurant_agent/search.py fetch_agents/search_agent/__init__.py fetch_agents/search_agent/agent.py fetch_agents/search_agent/chat_utils.py fetch_agents/search_agent/README.md fetch_agents/search_agent/search.py poke_middleware/.env poke_middleware/.env.example poke_middleware/data/intents.json poke_middleware/ENDPOINTS.md poke_middleware/middleware_pipline.md poke_middleware/package.json poke_middleware/README.md poke_middleware/recipe/SETUP.md poke_middleware/scripts/dummy-agent.mjs poke_middleware/scripts/expose.mjs poke_middleware/scripts/poke-send.mjs poke_middleware/scripts/simulate-flow.mjs poke_middleware/scripts/test-capture.mjs poke_middleware/scripts/tunnel.mjs poke_middleware/scripts/up.mjs poke_middleware/src/asi/classify.ts poke_middleware/src/asi/dummy-planner.ts poke_middleware/src/asi/instructions-executor.ts poke_middleware/src/asi/invoke-agent.ts poke_middleware/src/index.ts poke_middleware/src/mcp.ts poke_middleware/src/pipeline/process-intent.ts poke_middleware/src/poke/notify.ts poke_middleware/src/redis/clients.ts poke_middleware/src/redis/context.ts poke_middleware/src/redis/domain.ts poke_middleware/src/redis/index.ts poke_middleware/src/redis/intents.ts poke_middleware/src/redis/memory.ts poke_middleware/src/redis/resilience.ts poke_middleware/src/redis/timeout.ts poke_middleware/src/redis/trust.ts poke_middleware/src/store.ts poke_middleware/src/types.ts poke_middleware/tsconfig.json README.md ``` ### Dependencies - dashboard/package.json: @types/react@^18.3.12, @types/react-dom@^18.3.1, @vitejs/plugin-react@^4.7.0, react@^18.3.1, react-dom@^18.3.1, typescript@^5.6.3, vite@^5.4.21 - eletron_app/agent/requirements.txt: flask@==3.1.3, gui-agents@==0.1.3, pillow@==12.2.0, pyautogui@==0.9.54, python-dotenv@==1.2.2 - eletron_app/package.json: @deepgram/sdk@^5.4.0, @types/node@^22.10.0, @types/react@^19.0.0, @types/react-dom@^19.0.0, @vitejs/plugin-react@^4.3.4, electron@^33.2.0, electron-vite@^2.3.0, react@^19.0.0, react-dom@^19.0.0, typescript@^5.6.3, vite@^5.4.11 - fetch_agents/requirements.txt: anthropic@>=0.40.0, google-genai@>=1.0.0, openai@>=1.40.0, python-dotenv@>=1.0.0, uagents@>=0.22.0, uagents-core@>=0.3.0 - poke_middleware/package.json: @modelcontextprotocol/sdk@^1.12.1, @redis-iris/agent-memory@^0.1.0, @types/express@^4.17.21, @types/node@^22.13.10, dotenv@^16.4.7, express@^4.21.2, poke@^0.4.2, redis@^6.0.0, tsx@^4.19.3, typescript@^5.8.2, zod@^3.24.2 ### Recent commits (newest first) - working code ## Key source files (fetched from GitHub, selected and truncated for size) ### eletron_app/CLAUDE.md ```markdown # CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Setup ```bash pip install -r requirements.txt cp .env.example .env # fill in ASI_ONE_API_KEY, AGENT_SEED, AMAZON_AGENT_ADDRESS ``` ## Run ```bash python -m orchestrator.agent ``` ## Test endpoints ```bash # Health check curl localhost:8000/health # Classify intent and route to downstream curl -X POST localhost:8000/classify \ -H 'content-type: application/json' \ -d '{"query":"order 2 dozen eggs and milk from amazon"}' ``` ## Architecture This is a Fetch.ai (uAgents) **orchestrator agent** that bridges two inbound channels to downstream specialist agents: | Channel | How it works | |---|---| | **REST** (`POST /classify`) | Synchronous: REST handler awaits an `asyncio.Future` resolved when the downstream reply arrives | | **ASI:One chat** (`ChatMessage`) | Asynchronous: reply is sent back to the original chat sender when the downstream responds | **Request flow:** 1. Query arrives (REST or chat) 2. `intent.py:classify_intent()` calls the ASI:One LLM (`asi1` model, OpenAI-compatible endpoint) to classify to one of the `INTENTS` dict entries 3. `routing.py:forward_to_downstream()` looks up the intent in `ROUTES`, registers a `PendingEntry` in `session.py`, and sends a `ChatMessage` to the downstream agent 4. When the downstream agent replies, `handle_chat` in `agent.py` correlates the reply back to the pending session via `session.py:pop_for_sender()` (FIFO per downstream address) and resolves the future or sends a chat reply **Key modules:** - `agent.py` — Agent definition, REST handlers, chat protocol handlers - `intent.py` — `INTENTS` registry and LLM-based classification; add new intent names here - `routing.py` — `ROUTES` mapping intent names to downstream agent addresses; add new routes here - `session.py` — In-memory async correlation bridge (`PENDING` map + `EXPECT` FIFO queues per downstream address) - `chat_utils.py` — Chat protocol message construction helpers and product parsing from downstream replies - `models.py` — Pydantic models for REST request/response (`OrchestrateRequest`, `OrchestrateResponse`, `Product`) ## Adding a new downstream skill 1. Add an intent name + description to `INTENTS` in `intent.py` 2. Map it to a downstream agent address in `ROUTES` in `routing.py` Classification and routing pick it up automatically — no changes to `agent.py` or `session.py` needed. ## Key design constraints - **Session correlation is FIFO per downstream address.** Replies from downstream agents do not echo the originating `msg_id`, so `session.py` matches replies to pending sessions in order. If a downstream agent later echoes a reference id, switch to exact-id matching in `handle_chat`. - **In-memory only.** `PENDING` and `EXPECT` are process-local dicts; restarts lose in-flight sessions. - **`AgentPaymentProtocol`** is out of scope — `RequestPayment` messages are not handled. ``` ### poke_middleware/middleware_pipline.md ```markdown # CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Commands ```bash npm run dev # start with tsx watch (hot reload) npm run up # dev server + poke tunnel together npm run typecheck # tsc --noEmit (no test suite) npm run dummy-agent # start remote executor stub on :8790 npm run expose # cloudflare tunnel → https URL for MCP npm run tunnel # poke tunnel (alternative to expose) npm run simulate # run a simulated end-to-end flow locally ``` One-time setup: ```bash npm install cp .env.example .env # fill POKE_API_KEY, REDIS_URL, etc. npm run poke:login # authenticate poke CLI ``` ## Architecture **Deadbolt** is an Express + MCP middleware (port 8787) that orchestrates a multi-step intent pipeline between a phone (Poke), an ASI classifier, and a remote executor agent. ### Pipeline (4 steps) ``` 1. Phone/Poke → POST /intents or MCP submit_intent 2. Middleware → ASI classify (POST ASI_CLASSIFY_URL) OR local dummy planner 3. Executor → POST AGENT_WEBHOOK_URL/intent (or EXECUTOR_INSTRUCTIONS_URL) → agent POSTs back to POST /agent/callback (verified plan) 4. Middleware → Poke chat shows cart, user sends APPROVE/REJECT → POST /intents/:id/poke-approval → middleware notifies executor → POST /agent/execution-result ``` ### Intent lifecycle `captured → planning → awaiting_agent → awaiting_poke_approval → approved → executing → completed | failed` `rejected` and `failed` are also terminal. The `blocked` sub-state reopens `awaiting_poke_approval` without going terminal (checkout blocker). ### Key files | File | Role | |------|------| | `src/index.ts` | Express routes + MCP session manager (StreamableHTTP) | | `src/types.ts` | All shared types: `CapturedIntent`, `IntentPlan`, `IntentStatus`, etc. | | `src/store.ts` | In-memory `Map` — single source of truth at runtime | | `src/mcp.ts` | MCP tools exposed to Poke: `submit_intent`, `submit_approval`, `submit_message_intention`, debug tools | | `src/pipeline/process-intent.ts` | Pipeline steps: `processIntent`, `handleAgentCallback`, `handlePokeApproval`, `handleAgentExecutionResult` | | `src/redis/` | Redis clients, `events:intents` stream, trust scores, Agent Memory, context building | | `src/asi/` | ASI classify, dummy planner, executor invocation (`invoke-agent.ts`, `instructions-executor.ts`) | | `src/poke/notify.ts` | Poke chat notifications (cart preview, final result) | ### Storage layers - **L1**: in-memory `Map` in `store.ts` — always authoritative at runtime - **L2**: Redis (`REDIS_URL` env) — `intent:{uuid}` hashes, `events:intents` stream, trust, profile; hydrated into L1 on startup - **Disk fallback**: `data/intents.json` — used only when Redis is not configured **Redis invariant**: only the middleware writes Redis and Agent Memory. The executor reads derived context via `GET /context` — never connects to Redis [truncated — 1308 more characters] ``` ### fetch_agents/requirements.txt ``` uagents>=0.22.0 uagents-core>=0.3.0 openai>=1.40.0 anthropic>=0.40.0 google-genai>=1.0.0 python-dotenv>=1.0.0 ``` ### dashboard/package.json ``` { "name": "deadbolt-dashboard", "private": true, "version": "0.1.0", "type": "module", "description": "Deadbolt control plane — live Fetch.ai marketplace + orchestrator routes + intent feed.", "scripts": { "dev": "vite", "build": "vite build", "preview": "vite preview" }, "dependencies": { "react": "^18.3.1", "react-dom": "^18.3.1" }, "devDependencies": { "@types/react": "^18.3.12", "@types/react-dom": "^18.3.1", "@vitejs/plugin-react": "^4.7.0", "typescript": "^5.6.3", "vite": "^5.4.21" } } ``` ### eletron_app/package.json ``` { "name": "shadow", "version": "0.1.0", "description": "Sunny — a voice-first AI companion that operates the computer", "main": "./out/main/index.js", "author": "", "license": "MIT", "scripts": { "dev": "electron-vite dev", "build": "electron-vite build", "preview": "electron-vite preview", "tunnel": "bash scripts/tunnel.sh", "typecheck": "tsc --noEmit -p tsconfig.web.json && tsc --noEmit -p tsconfig.node.json" }, "dependencies": { "@deepgram/sdk": "^5.4.0", "react": "^19.0.0", "react-dom": "^19.0.0" }, "devDependencies": { "@types/node": "^22.10.0", "@types/react": "^19.0.0", "@types/react-dom": "^19.0.0", "@vitejs/plugin-react": "^4.3.4", "electron": "^33.2.0", "electron-vite": "^2.3.0", "typescript": "^5.6.3", "vite": "^5.4.11" } } ``` ### poke_middleware/package.json ``` { "name": "deadbolt-middleware", "version": "0.1.0", "private": true, "type": "module", "scripts": { "dev": "tsx watch src/index.ts", "start": "tsx src/index.ts", "up": "node scripts/up.mjs", "poke:login": "poke login", "tunnel": "node scripts/tunnel.mjs", "tunnel:recipe": "node scripts/tunnel.mjs --recipe", "expose": "node scripts/expose.mjs", "poke:send": "node scripts/poke-send.mjs", "test:capture": "node scripts/test-capture.mjs", "simulate": "node scripts/simulate-flow.mjs", "dummy-agent": "node scripts/dummy-agent.mjs", "typecheck": "tsc --noEmit" }, "dependencies": { "@modelcontextprotocol/sdk": "^1.12.1", "@redis-iris/agent-memory": "^0.1.0", "dotenv": "^16.4.7", "express": "^4.21.2", "poke": "^0.4.2", "redis": "^6.0.0", "zod": "^3.24.2" }, "devDependencies": { "@types/express": "^4.17.21", "@types/node": "^22.13.10", "tsx": "^4.19.3", "typescript": "^5.8.2" }, "engines": { "node": ">=18" } } ``` ### eletron_app/agent/requirements.txt ``` # Shadow agent sidecar dependencies. # anthropic / openai are pulled in transitively by gui-agents; pinned for reproducibility. gui-agents==0.1.3 pyautogui==0.9.54 python-dotenv==1.2.2 pillow==12.2.0 flask==3.1.3 ``` ### dashboard/src/main.tsx ```typescript import React from 'react' import { createRoot } from 'react-dom/client' import App from './App' import './styles.css' createRoot(document.getElementById('root')!).render( , ) ``` ### eletron_app/agent/main.py ```python """Shadow Python sidecar — bridges Electron (stdio) and middleware (HTTP) to Agent-S. Commands arrive two ways and feed one sequential task queue: - stdin JSON: {"type":"run_task","id":"...","instruction":"..."} / {"type":"cancel"} - HTTP POST /instructions: {"instructions": [...]} (see http_server.py) Events stream on stdout, one JSON object per line: ready / queued / status / step / screenshot / done / error (each carries the task id) stdout is reserved for JSON only — Agent-S and its deps print to stdout, so we redirect their output to stderr up front. """ import json import os import queue import sys import threading import uuid # Reserve real stdout for JSON; send everything else (library prints, logs) to stderr. _OUT = sys.stdout sys.stdout = sys.stderr sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) from config import Config # noqa: E402 from http_server import start_http # noqa: E402 def build_runner(cfg: Config): """Pick the automation engine: native computer-use (default) or Agent-S.""" if cfg.engine == "agent-s": from agent_runner import AgentRunner return AgentRunner(cfg) from native_engine import NativeRunner return NativeRunner(cfg) class Sidecar: def __init__(self): self._queue: queue.Queue = queue.Queue() self._cancel = threading.Event() self._runner = None # built lazily on first task self._current_id = None self._cfg = None # loaded once in loop() self._send_lock = threading.Lock() # worker + converse threads share stdout self._results: dict = {} # task_id -> terminal event (sync callers only) self._events: dict = {} # task_id -> threading.Event (sync callers only) def send(self, event: dict): # Serialize writes: a converse reply can race a running task's events. line = json.dumps(event) + "\n" with self._send_lock: _OUT.write(line) _OUT.flush() def enqueue(self, instruction: str, source: str, mode: str = "hands-on") -> str: task_id = str(uuid.uuid4()) self._queue.put((task_id, instruction, mode)) self.send({"type": "queued", "id": task_id, "instruction": instruction, "source": source}) return task_id def run_sync(self, instruction: str, timeout: float = 300.0) -> dict: """Enqueue and block until the task finishes; return its verdict.""" task_id = str(uuid.uuid4()) done = threading.Event() self._events[task_id] = done self._queue.put((task_id, instruction, "hands-on")) self.send({"type": "queued", "id": task_id, "instruction": instruction, "source": "api"}) finished = done.wait(timeout) self._events.pop(task_id, None) result = self._results.pop(task_id, None) if not finished or result is None: return {"id": task_id, "instruction": instruction, "verdict": "rejected", "reason": "timed out", "summary": ""} if result.get("type") == "error": return {"id": task_id, "instruction": instruction, "verdict": "rejected", "reason": result.get("message", ""), "summary": ""} if result.get("type") == "cancelled": return {"id": task_id, "instruction": instruction, "verdict": "rejected", "reason": "cancelled", "summary": ""} return {"id": task_id, "instruction": instruction, "verdict": result.get("verdict", "approved"), "reason": result.get("reason", ""), "summary": result.get("summary", "")} def handle_stdin(self, cmd: dict): ctype = cmd.get("type") if ctype == "run_task": task_id = cmd.get("id") or str(uuid.uuid4()) self._queue.put((task_id, cmd.get("instruction", ""), cmd.get("mode", "hands-on"))) self.send({"type": "queued", "id": task_id, "instruction": cmd.get("instruction", ""), "source": "ui"}) elif ctype == "converse": # Fast conversational turn: reply + decide task vs. chat. Off the main # queue and on its own thread so it never waits behind a running task. cid = cmd.get("id") or str(uuid.uuid4()) threading.Thread(target=self._converse, daemon=True, args=(cid, cmd.get("text", ""), cmd.get("mode", "hands-on"))).start() elif ctype == "cancel": self._drain() self._cancel.set() def _converse(self, cid: str, text: str, mode: str): from converse import converse cfg = self._cfg or Config.load() try: result = converse(cfg.anthropic_api_key, cfg.chat_model, text, mode) except Exception as exc: # converse() already fails safe, but guard the thread result = {"intent": "task", "say": "On it.", "task": text, "error": str(exc)} self.send({"type": "reply", "id": cid, **result}) def _drain(self): """Discard pending tasks (used on cancel).""" try: while True: self._queue.get_nowait() self._queue.task_done() except queue.Empty: pass def _worker(self): while True: task_id, instruction, mode = self._queue.get() self._current_id = task_id self._cancel.clear() def emit(ev: dict): self.send({**ev, "id": task_id}) # Hand the terminal result to any sync caller waiting on this task. if ev.get("type") in ("done", "error", "cancelled"): waiter = self._events.get(task_id) if waiter: self._results[task_id] = ev waiter.set() try: if self._runner is None: self._runner = build_runner(Config.load()) self._runner.run(instruction, [truncated — 920 more characters] ``` ### poke_middleware/src/index.ts ```typescript import { randomUUID } from "node:crypto"; import "dotenv/config"; import express from "express"; import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js"; import { isInitializeRequest } from "@modelcontextprotocol/sdk/types.js"; import { z } from "zod"; import { createMcpServer } from "./mcp.js"; import { enqueueProcessIntent, handleAgentCallback, handleAgentExecutionResult, handlePokeApproval, } from "./pipeline/process-intent.js"; import { isAsiClassifyEnabled } from "./asi/classify.js"; import { captureIntent, formatIntentQueueEntry, formatIntentSummary, getIntent, hydrateStoreFromRedis, listActiveIntents, listIntentQueue, listIntents } from "./store.js"; import { buildGatewayContext, getTrust, initGatewayRedis, isRedisEnabled, redisReady, replayEvents, } from "./redis/index.js"; import { inferDomain } from "./redis/domain.js"; import type { IntentPlan } from "./types.js"; const HOST = process.env.HOST ?? "127.0.0.1"; const PORT = Number(process.env.PORT ?? 8787); const MCP_API_KEY = process.env.MCP_API_KEY?.trim(); const app = express(); app.use(express.json()); const mcpTransports = new Map(); function requireMcpAuth(req: express.Request, res: express.Response): boolean { if (!MCP_API_KEY) return true; const header = req.headers.authorization; if (header === `Bearer ${MCP_API_KEY}`) return true; res.status(401).json({ error: "Unauthorized MCP request" }); return false; } app.get("/health", (_req, res) => { res.json({ ok: true, service: "deadbolt-middleware", phase: "capture-plan-approve", mcpTransport: "streamable-http", activeMcpSessions: mcpTransports.size, pendingIntents: listActiveIntents().filter((i) => i.status === "captured").length, activeQueue: listActiveIntents().length, awaitingAgent: listIntents().filter((i) => i.status === "awaiting_agent").length, awaitingPokeApproval: listIntents().filter((i) => i.status === "awaiting_poke_approval").length, executing: listIntents().filter((i) => i.status === "executing").length, redis: redisReady(), asiClassify: isAsiClassifyEnabled(), executorMock: process.env.EXECUTOR_MOCK?.trim().toLowerCase() === "true", }); }); /** Executor reads this before planning — similar decisions, profile, confidence (read-only) */ app.get("/context", async (req, res) => { const intentText = typeof req.query.intent === "string" ? req.query.intent : typeof req.query.text === "string" ? req.query.text : undefined; if (!intentText?.trim()) { res.status(400).json({ error: "Query param intent= or text= required" }); return; } const intentId = typeof req.query.intentId === "string" ? req.query.intentId : undefined; const ctx = await buildGatewayContext(intentText.trim(), intentId); res.json({ similar_decisions: ctx.similar_decisions, profile: ctx.profile, confidence: ctx.confidence, confidence_rationale: ctx.confidence_rationale, domain: ctx.domain, trust: ctx.trust, }); }); /** Trust scores per domain (demo dashboard) */ app.get("/trust", async (req, res) => { const domain = typeof req.query.domain === "string" ? req.query.domain : "groceries"; const trust = await getTrust(domain); res.json({ domain, ...trust }); }); /** Replay events:intents stream (session proof artifact) */ app.get("/events/replay", async (req, res) => { const intentId = typeof req.query.intentId === "string" ? req.query.intentId : undefined; const events = await replayEvents(intentId); res.json({ intentId, events }); }); const intentBodySchema = z.object({ intent: z.string().min(1), source: z.string().optional(), metadata: z.record(z.unknown()).optional(), }); app.post("/intents", (req, res) => { const parsed = intentBodySchema.safeParse(req.body); if (!parsed.success) { res.status(400).json({ error: "Invalid body", details: parsed.error.flatten() }); return; } const result = captureIntent({ text: parsed.data.intent, source: parsed.data.source, metadata: parsed.data.metadata, }); enqueueProcessIntent(result.intentId); res.status(202).json(result); }); /** Active queue — one row per in-flight session (pipeline steps, not full cart) */ app.get("/intents", (_req, res) => { const queue = listIntentQueue(); res.json({ queue: queue.map(formatIntentQueueEntry), count: queue.length, note: "Full cart + items: GET /intents/:id · All history: GET /intents/history", }); }); /** Full history including completed intents (debug only) */ app.get("/intents/history", (_req, res) => { res.json({ intents: listIntents() }); }); const agentCallbackSchema = z.object({ intentId: z.string().uuid(), agentId: z.string().optional(), note: z.string().optional(), plan: z.object({ provider: z.string(), summary: z.string(), items: z.array( z.object({ name: z.string(), url: z.string().url(), price: z.string(), inStock: z.boolean(), added: z.boolean().optional(), }), ), subtotal: z.string(), agentNote: z.string().optional(), plannedAt: z.string(), }), }); /** Executor agent on other machine POSTs cart/plan here */ app.post("/agent/callback", async (req, res) => { const parsed = agentCallbackSchema.safeParse(req.body); if (!parsed.success) { res.status(400).json({ error: "Invalid body", details: parsed.error.flatten() }); return; } try { await handleAgentCallback({ intentId: parsed.data.intentId, plan: parsed.data.plan as IntentPlan, agentId: parsed.data.agentId, note: parsed.data.note, }); const intent = getIntent(parsed.data.intentId); res.json({ ok: true, intent }); } catch (err) { res.status(400).json({ error: err instanceof Error ? err.message : "Callback failed" }); } }); const agentExecutionSchema = z.object({ intentId: z.string().uu [truncated — 6748 more characters] ``` [73 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]