Project Info
Elevator pitch Clara is the agent that lives on your machine and gives every other AI agent a body.
Inspiration
There's a swarm of AI agents out there now, each one good at one narrow thing. But none of them actually live anywhere. We wanted an agent that sits on your own computer, has context and control over your machine, and can pull in any of those external agents and put them to work for you.
What it does
Clara gives you full access to your laptop through iMessage (via Poke). You text it like you'd text a friend: order groceries on Amazon, book a table on Yelp, whatever. Behind the scenes, a classifier figures out what you actually want, builds a plan, and hands it off to a specialist agent in the swarm. For Amazon orders, Clara builds the cart and shows it to you in chat before doing anything. You approve or reject, then it executes. For restaurants, you get a list of options right in iMessage, pick one, and Clara finishes the reservation on your desktop. Under the hood, this is a real swarm of agents running on Fetch.ai's Agentverse, not one bot pretending to be many.
How we built it
Three pieces: An Express + MCP middleware pipeline that talks to Poke A Fetch.ai uAgents orchestrator that handles intent classification and routing An Electron app on the desktop that handles automation and cowork Intent classification runs through ASI:One's LLM. There's a dashboard with access to the Agentverse marketplace, so you can connect any agent you want into the swarm, plus a dummy fallback agent so the demo never dies if ASI is down. The orchestrator routes to downstream specialist agents over the Agent Chat Protocol. Redis backs state and trust scoring, with a disk fallback, so the system always has context on what's happening on your desktop. MCP runs over StreamableHTTP so Poke can talk to it directly.
Challenges we ran into
Bridging a text thread with a desktop. iMessage is built for sending words back and forth, not for telling a laptop what to do. Getting Poke to talk to the Electron app cleanly, so a casual text actually triggers something real on your machine, took more wrangling than it looked like it should. Giving the AI enough context about your desktop without giving it everything. The agent needs to know what's going on on your machine to act on your behalf, but dumping your whole desktop at it is messy and a little unnerving. We had to figure out what context actually mattered and feed it just that, without losing the thread of what you're doing. Keeping multiple agents talking without crossing wires. Once you've got a swarm instead of one bot, you have to make sure a reply from the Amazon agent doesn't get matched to a restaurant request that came in five seconds later. Sounds simple until two requests are in flight at once. Making sure a demo never just dies. Any one piece, ASI, Redis, the executor, could go down mid-demo. We built the whole thing to fail soft instead of fail loud, so even if something upstream broke, Clara still looked like it worked.
Accomplishments we're proud of
Plug in any agent from the Agentverse marketplace, not just the ones we built. Clara isn't locked to a fixed set of skills. Open the dashboard, connect an agent from the marketplace, and it's part of your swarm. The system was built to grow past Amazon and Yelp from day one. Hands free control of your desktop with voice. You can talk to your machine like you'd talk to a person and have it actually do something, no keyboard, no clicking through menus. That's the part that made people in the room go quiet for a second. A purchase never happens without you seeing it first. Even with agents acting on your behalf, nothing gets bought or booked blind. Clara shows you the cart, you say yes or no, and only then does it execute. We wanted autonomy without giving up control.
What we learned
On working with Fetch.ai/uAgents and ASI:One: The agent protocol doesn't hand you anything like a request ID to match a reply against. We assumed agent-to-agent communication would feel like calling an API and getting a response back. It's closer to mailing a letter and hoping it comes back in the order you sent it. ASI:One's classification itself worked well out of the box. The surprise was how much hand-built plumbing it took to make agent replies land back in the right place. On giving an AI control over your desktop: Most software adds a button. This adds judgment. The hard part wasn't teaching Clara to act on your machine, it was deciding what she should never be allowed to do without asking first. Control turned out to be a design problem, not an engineering one. On the word "agentic" itself: Building an actual swarm of agents made us a lot more skeptical of how that word gets used in pitch decks. The real work is unglamorous: matching replies to requests, building fallbacks for when a service goes down, deciding who owns state when three agents are talking at once. None of that is exciting to demo, but it's most of what makes the thing actually work.
What's next
More agents, more use cases. Amazon and Yelp were the proof of concept. The marketplace model means the next agent someone plugs in could be anything, travel booking, calendar management, your own weird side project. The sky's the limit once the swarm is the product instead of the two skills we shipped. Open sourcing the connector. We want anyone to be able to build their own specialized agent and wire it straight into Clara, instead of waiting on us to add it. That's the actual unlock here. Clara stops being our project and becomes whatever the community plugs into it. Beyond iMessage, and real payments. Texting was the fastest way to prove the idea worked. Long term, more channels and an actual payment flow, instead of a cart you just approve and someone else checks out, turns this from a demo into something you'd trust with real money.
Clara
An agentic desktop assistant stack — a 3D avatar that controls your computer, a phone-based approval gate, a Fetch.ai intent router, and a live control dashboard.
Components
| Folder | Name | What it does |
|---|---|---|
eletron_app/ | Shadow | Electron app + Python sidecar. Type a task; an animated 3D avatar drives your Mac via Claude computer-use, then shows you a screenshot and narrates what it did. |
poke_middleware/ | Deadbolt | TypeScript middleware bridging your phone (Poke), the Fetch.ai classifier, and Shadow. Every action requires your phone approval before it executes. |
fetch_agents/ | Orchestrator | Fetch.ai (uAgents) service. Classifies natural-language intent with ASI:One LLM and routes to downstream specialist agents (e.g. Amazon grocery). |
dashboard/ | Dashboard | Vite + React control plane. Browse the Agentverse marketplace, inspect wired routes, and watch the live intent feed. |
How it fits together
Phone (Poke)
→ Deadbolt middleware → Orchestrator (/classify) → downstream agents
↓ approval gate (Fetch.ai / Amazon …)
Shadow (executor)
↓
3D avatar + screenshot
Deadbolt captures the intent, asks the orchestrator to plan it, sends the plan to your phone for approval, then forwards approved instructions to Shadow. Shadow executes them step-by-step and returns a verification screenshot.
Quick start
Each component has its own README with full setup instructions.
# 1. Shadow (Electron desktop app)
cd eletron_app && npm install && npm run dev
# 2. Orchestrator (Fetch.ai agent)
cd fetch_agents && pip install -r requirements.txt && python -m orchestrator.agent
# 3. Deadbolt middleware
cd poke_middleware && npm install && npm run up
# 4. Dashboard
cd dashboard && npm install && npm run dev # http://localhost:5273
Requirements
- macOS, Node 20+, Python 3.10+
ANTHROPIC_API_KEY— Shadow (computer-use)ASI_ONE_API_KEY— Orchestrator (intent classification)POKE_API_KEY+REDIS_URL— Deadbolt (phone gate + state)
Analysis
View
Metric
No commits on this project resolved to a GitHub account.
Technology
- AnthropicIn code
- CSSIn code
- ExpressIn code
- FlaskIn code
- HTMLIn code
- OpenAIIn code
- PythonIn code
- ReactIn code
- RedisIn code
- TypeScriptIn code
- Node.jsClaimed
10 of 11 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
- Claude CodeConfig
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
380 KB
Source files
85
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
Preetam3620/clara
113 files · 726 KB · @ b24ffd1
Structure
Interface
8 files · 7%Screens, components and styles rendered to the user.
Application logic
66 files · 58%Domain rules, services and shared utilities.
+3 more
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- TypeScript54%
- Python26%
- Markdown11%
- CSS8%
- Shell0%
- HTML0%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
eletron_app/package.json
npm · 11- @deepgram/sdk
- react
- react-dom
- +8 more
poke_middleware/package.json
npm · 11- @modelcontextprotocol/sdk
- @redis-iris/agent-memory
- dotenv
- express
- poke
- redis
- zod
- +4 more
dashboard/package.json
npm · 7- react
- react-dom
- +5 more
fetch_agents/requirements.txt
pypi · 6- anthropic
- google-genai
- openai
- python-dotenv
- uagents
- uagents-core
eletron_app/agent/requirements.txt
pypi · 5- flask
- gui-agents
- pillow
- pyautogui
- python-dotenv
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
Feature verification
3D avatar drives desktop via Claude computer-use / Agent-SVerified
Shadow: an animated 3D avatar drives your Mac via Claude computer-use, then shows you a screenshot and narrates what it did
Claimed on readmehigh confidenceeletron_app/agent/agent_runner.py:73— AgentRunner builds gui_agents AgentS3 (Claude-driven planning + grounding) and executes actions via pyautoguieletron_app/agent/verify.py:21— verify() sends the final screenshot to Claude to judge task success, returned to the caller as narration
Amazon cart preview with explicit approve/reject before checkoutVerified
For Amazon orders, Clara builds the cart and shows it in chat before doing anything; you approve or reject, then it executes
Claimed on Devposthigh confidencepoke_middleware/src/asi/dummy-planner.ts:144— formatPlanForPoke renders the cart with 'Nothing checks out until YOU approve' and approve/reject instructionspoke_middleware/src/pipeline/process-intent.ts:245— handlePokeApproval only calls the executor agent after an 'approved' decision is recorded
Dummy fallback agent keeps the demo alive if ASI is downVerified
a dummy fallback agent so the demo never dies if ASI is down
Claimed on Devposthigh confidencepoke_middleware/src/asi/dummy-planner.ts:17— buildLocalDummyPlan returns a hardcoded demo cartpoke_middleware/src/pipeline/process-intent.ts:115— runProcessIntent falls back to buildLocalDummyPlan when isAsiClassifyEnabled() is false, and to an offline draft plan when the executor is unreachable
Electron app + Python sidecar architecture (Shadow)Verified
eletron_app/ (Shadow): Electron app + Python sidecar
Claimed on readmemedium confidenceeletron_app/agent/http_server.py:21— Flask HTTP server exposes an /instructions endpoint, acting as the Python sidecar the Electron main process talks to
Hands-free voice control of the desktopVerified
Hands free control of your desktop with voice, no keyboard, no clicking through menus
Claimed on Devposthigh confidenceeletron_app/src/renderer/src/voice/useVoice.ts:50— useVoice streams mic audio to Deepgram STT (Nova-3) and speaks replies via Deepgram TTS (Aura-2), with barge-in handling
Intent classifier builds a plan and routes to a specialist agentVerified
A classifier figures out what you want, builds a plan, and hands it to a specialist agent in the swarm
Claimed on Devposthigh confidencepoke_middleware/src/asi/classify.ts:187— classifyAndBuildPlan posts to ASI classify endpoint and resolves an amazon vs restaurant flowfetch_agents/orchestrator/agent.py:134— classify_intent() picks an intent and forward_to_downstream() routes to the matching specialist agent
MCP over StreamableHTTP for PokeVerified
MCP runs over StreamableHTTP so Poke can talk to it directly
Claimed on Devposthigh confidencepoke_middleware/src/index.ts:4— imports and instantiates StreamableHTTPServerTransport from the MCP SDK to serve the MCP endpoint
Purchases never happen without user approval (approval gate)Verified
A purchase never happens without you seeing it first; nothing gets bought or booked blind
Claimed on Devposthigh confidencepoke_middleware/src/pipeline/process-intent.ts:261— notifyAgentOfPokeApproval (which triggers execution) is only invoked when approval.decision === 'approved'
Real multi-agent swarm on Fetch.ai Agentverse via Agent Chat ProtocolVerified
This is a real swarm of agents running on Fetch.ai's Agentverse, not one bot pretending to be many
Claimed on Devposthigh confidencefetch_agents/orchestrator/agent.py:22— uses uagents.Agent/Protocol and uagents_core chat_protocol_spec, with mailbox=True for Agentverse registrationfetch_agents/orchestrator/routing.py:29— forward_to_downstream sends ChatMessage to separate downstream agent processes (amazon/restaurant)
Redis-backed state and trust scoring with disk fallbackVerified
Redis backs state and trust scoring, with a disk fallback, so the system always has context
Claimed on Devposthigh confidencepoke_middleware/src/redis/trust.ts:12— getTrust/bumpTrust read/write a trust score in Redis, degrading gracefully via warnRedisDegraded when Redis is unavailablepoke_middleware/src/store.ts:64— intents are persisted to a local intents.json file, loaded on startup as the disk fallbackpoke_middleware/src/redis/clients.ts:34— initGatewayRedis logs 'REDIS_URL not set — intent store uses local disk' when Redis isn't configured
Restaurant options list with pick-to-book flowVerified
For restaurants, you get a list of options right in iMessage, pick one, and Clara finishes the reservation on your desktop
Claimed on Devposthigh confidencepoke_middleware/src/asi/dummy-planner.ts:129— formatRestaurantOptionsForPoke lists numbered options and asks the user to select onepoke_middleware/src/pipeline/process-intent.ts:295— bookRestaurantSelection books the picked item via the executor agent, treating selection itself as approval
Text-based control of desktop via iMessage/PokeVerified
Clara gives you full access to your laptop through iMessage (via Poke); you text it like a friend
Claimed on Devposthigh confidencepoke_middleware/src/mcp.ts:202— MCP server exposes submit_intent/submit_approval tools that Poke calls with the user's text and returns cart/status replies verbatim to the chatpoke_middleware/src/poke/notify.ts:20— sendPokeChatMessage (proactive push to Poke inbox) is stubbed to a no-op; real replies flow only through synchronous MCP tool results, a narrower mechanism than full async messaging
Dashboard: browse and connect Agentverse marketplace agentsClaimed only
There's a dashboard with access to the Agentverse marketplace, so you can connect any agent you want into the swarm
Claimed on Devposthigh confidence
An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.