# Project export: remark. This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: “Making life more convenient one [re]mark at a time.” - Devpost: https://devpost.com/software/remark-ow4myn - GitHub: https://github.com/cadencheng888/remark. - Video: https://www.youtube.com/embed/L0OMEbHEUpw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 5 GitHub contributor(s) — MICH3LL3D (13 commits), davdwan21 (10 commits), Caden Cheng (9 commits), Claude Opus 4.8 (1M context) (5 commits), juj021 (3 commits) ## Devpost submission (written by the team) ### Overview David Wan, Caden Cheng, Michelle Dong, Julia Jin About Our Project & Significance remark — stylized as "remark." — is a multi-agent operating system that marks down information based on remarks made in conversation. remark detects plans to grab dinner between two friends and automatically adds the event to your calendar. It keeps track of grocery lists, takes notes, sends texts, searches the web, and so much more — all without you ever touching a screen. remark selects and utilizes the optimal AI agent to carry out your everyday needs, making your life more convenient by helping you anywhere and everywhere. It works in the background, almost unnoticed, but there is nothing unremarkable about the benefits it brings. remark facilitates convenience without the risk of forgetting. But remark's significance goes beyond convenience. Today, the AI agent ecosystem is vast and rapidly growing — there are specialized agents for shopping, scheduling, communication, research, navigation, and nearly every domain of everyday life. The problem is that accessing them requires knowing they exist, finding the right one, and navigating interfaces that were built for technically fluent users. The average person never touches an AI agent. That gap is what remark closes. By living in a pair of glasses and listening to natural speech, remark becomes the universal entry point to the entire agent ecosystem — no dashboards, no prompts, no learning curve. A grandmother making plans over the phone benefits from the same AI infrastructure as a software engineer. A child saying "I want to get pizza tonight" gets the same outcome as someone who knows how to write a system prompt. The interface is just talking, something every person already knows how to do. Future Expansions & Applications remark already runs hands-free on Meta Ray-Ban glasses (and works from a laptop mic with no app needed), listening for intent in ordinary conversation and turning it into real actions. Today, that means automatically creating, rescheduling, and canceling calendar events when you discuss, commit to, or back out of plans — plus routing shopping, messages, reminders, music, directions, and web lookups to an autonomous agent that actually carries them out. You don't have to do anything; remark marks it down for you. Here's where we're taking it next: Smarter planning. Beyond detecting that you've made plans, remark will factor in travel time and overlapping events, and proactively notify you when you're at risk of being late or double-booked. Live translation. remark is English-only today. We plan to have it bridge the language gap in real time: when it hears someone speaking a language you don't understand, it will transcribe and translate what they say — capturing intent, not just the literal words — and read it back to you, muting the other speaker while it translates so you only hear the translated speech. Everyday assistance. remark can already take notes and run shopping and errand tasks through its web agent. We want to grow this into richer, longer-lived helpers — a running grocery or shopping list, and, once on-glasses vision is wired in, live situational coaching such as real-time baking advice ("mix more — it doesn't look light and fluffy yet"). Reduce latency. Currently, there’s a bit of latency when communicating between agents; we want to see if we can find a way to reduce that. _Add video. remark will be expanded upon with video and an object-detection only pipeline. What Makes Us Unique In this era of constant new hardware and innovation, privacy remains a major source of anxiety — and it's where remark is deliberately different. Plenty of existing technology does pieces of what remark's features do, but we combine them into one easy-to-use, multi-agent system: a tiered router that hands each request to the agent best suited to handle it — a specialized Fetch.ai agent, the calendar, or an autonomous web agent — which keeps the platform extendable and adaptable to unusual, even one-off situations it hasn't seen before. Just as importantly, remark is built to minimize what it keeps. By default, nothing is persisted: the live transcript stays in memory and auto-clears seconds after the moment passes, and the only lasting record is the calendar event you actually wanted. We don't record or stream video — visual context is handled entirely on-device, where a face-presence check (running locally, with no frames ever leaving your device) gates passive listening. And if a feature ever needs to remember something longer — say, holding onto a grocery list for as long as you want it tracked — we ask first, so anything persistent is permission-based. remark is designed from the ground up to shrink the privacy surface, not expand it. Our Process We started our project from the frustration that AI agents are everywhere and incredibly capable, but using one means knowing it exists, finding it, and learning to navigate a dashboard that may not be intuitive for those unfamiliar with them. Our project proposes a solution by asking: what's the most natural, always-on way to talk? Smart glasses. The process of voice in, real-world action out seeks to make AI agents accessible to everyone. Step 1 — Hearing (audio → text) We built the ears first. transcribe.py takes the Meta Ray-Ban glasses microphone and streams raw PCM16 audio to Deepgram's nova-3 model over a WebSocket, getting back live interim and final transcripts. We turned on Deepgram's entity detection early because we knew we'd later need to resolve pronouns like "buy them." We made it reconnect automatically with exponential backoff so a long listening session survives network blips. Step 2 — Understanding (text → intent) Next, the brain: agent.py. We gave Claude Haiku 4.5 a tool-calling schema and a carefully tuned prompt so it could read messy, real conversational speech and decide what's actually actionable — distinguishing a real plan from chit-chat, an active command from passive admiration ("play this" vs. "I love this song"). Our first concrete action was calendar events, executed through the Google Calendar API with a mock mode so the demo works even before you connect Google. Step 3 — Seeing it (the HUD) + privacy We built a React + Tailwind "glasses HUD" (server.py + web/) that streams live captions and action cards over WebSocket, with the calendar embedded so events appear instantly. Then we added the features that make ambient listening acceptable: an on-device face-presence gate using the laptop webcam (only listens when someone's actually talking to you — camera frames never leave the machine) and an ephemeral transcript buffer that auto-wipes itself 20 seconds after the moment passes. Step 4 — Turning intent into action The hardest part: making "add this to my cart" or "text Sarah I'm running late" actually happen, not just appear as a card on screen. When Claude identifies a non-calendar action, it hands a natural-language intent string to our TypeScript agentic router (src/router.ts) running on a background thread — the optimistic card appears on the HUD immediately while the real work happens in parallel. The router tries three tiers in order. First, it queries the Fetch.ai Agentverse marketplace to find a specialized agent for the task — searching for candidates by capability, then using Claude to judge the best match. Second, if the intent is calendar-shaped, it hits the Google Calendar API directly. Third, and most powerfully, it falls through to Browserbase + Stagehand: a cloud-hosted browser driven by Claude Sonnet 4.6 that can navigate and interact with essentially any website on the internet. The wearer's current location (resolved via IP geolocation) is baked into every intent string so "near me" and directions work correctly. The router streams its reasoning back to the HUD as live "thinking" lines — so you can watch the agent decide, step by step, before the final result card lands. Privacy + Ethical Use of AI The primary privacy concern related to our project is that the Meta Ray Ban microphone listens to whatever is going on, 24/7. However, we do not retain raw audio, video, images, or full transcripts. The system converts each interaction into a short, topic-level memory summary, stores only the minimum needed context, and deletes the source content after processing. Raw audio, video, images, and transcripts never leave the user’s device. They are processed locally, compressed into short topic-level summaries, and then deleted. Only the minimum necessary summary or task-specific instruction is shared with an external agent, and only when the user approves or triggers an action. ## README (from the GitHub repository)

remark. Logo
Michelle Dong, Caden Cheng, Julia Jin, David Wan

## About Our Project & Significance remark — stylized as "remark." — is a multi-agent operating system that marks down information based on remarks made in conversation. remark detects plans to grab dinner between two friends and automatically adds the event to your calendar. It keeps track of grocery lists, takes notes, sends texts, searches the web, and so much more — all without you ever touching a screen. remark selects and utilizes the optimal AI agent to carry out your everyday needs, making your life more convenient by helping you anywhere and everywhere. It works in the background, almost unnoticed, but there is nothing unremarkable about the benefits it brings. remark facilitates convenience without the risk of forgetting. But remark's significance goes beyond convenience. Today, the AI agent ecosystem is vast and rapidly growing — there are specialized agents for shopping, scheduling, communication, research, navigation, and nearly every domain of everyday life. The problem is that accessing them requires knowing they exist, finding the right one, and navigating interfaces that were built for technically fluent users. The average person never touches an AI agent. That gap is what remark closes. By living in a pair of glasses and listening to natural speech, remark becomes the universal entry point to the entire agent ecosystem — no dashboards, no prompts, no learning curve. A grandmother making plans over the phone benefits from the same AI infrastructure as a software engineer. A child saying "I want to get pizza tonight" gets the same outcome as someone who knows how to write a system prompt. The interface is just talking, something every person already knows how to do. ## Future Expansions & Applications remark already runs hands-free on Meta Ray-Ban glasses (and works from a laptop mic with no app needed), listening for intent in ordinary conversation and turning it into real actions. Today, that means automatically creating, rescheduling, and canceling calendar events when you discuss, commit to, or back out of plans — plus routing shopping, messages, reminders, music, directions, and web lookups to an autonomous agent that actually carries them out. You don't have to do anything; remark marks it down for you. Here's where we're taking it next: - Smarter planning. Beyond detecting that you've made plans, remark will factor in travel time and overlapping events, and proactively notify you when you're at risk of being late or double-booked. - Live translation. remark is English-only today. We plan to have it bridge the language gap in real time: when it hears someone speaking a language you don't understand, it will transcribe and translate what they say — capturing intent, not just the literal words — and read it back to you, muting the other speaker while it translates so you only hear the translated speech. - Everyday assistance. remark can already take notes and run shopping and errand tasks through its web agent. We want to grow this into richer, longer-lived helpers — a running grocery or shopping list, and, once on-glasses vision is wired in, live situational coaching such as real-time baking advice ("mix more — it doesn't look light and fluffy yet"). - Somehow find a way to reduce latency when communicating between agents. - Add video and implement our object-detection only pipeline. ## What Makes Us Unique In this era of constant new hardware and innovation, privacy remains a major source of anxiety — and it's where remark is deliberately different. Plenty of existing technology does pieces of what remark's features do, but we combine them into one easy-to-use, multi-agent system: a tiered router that hands each request to the agent best suited to handle it — a specialized Fetch.ai agent, the calendar, or an autonomous web agent — which keeps the platform extendable and adaptable to unusual, even one-off situations it hasn't seen before. Just as importantly, remark is built to minimize what it keeps. By default, nothing is persisted: the live transcript stays in memory and auto-clears seconds after the moment passes, and the only lasting record is the calendar event you actually wanted. We don't record or stream video — visual context is handled entirely on-device, where a face-presence check (running locally, with no frames ever leaving your device) gates passive listening. And if a feature ever needs to remember something longer — say, holding onto a grocery list for as long as you want it tracked — we ask first, so anything persistent is permission-based. remark is designed from the ground up to shrink the privacy surface, not expand it. ## Our Process We started our project from the frustration that AI agents are everywhere and incredibly capable, but using one means knowing it exists, finding it, and learning to navigate a dashboard that may not be intuitive for those unfamiliar with them. Our project proposes a solution by asking: what's the most natural, always-on way to talk? Smart glasses. The process of voice in, real-world action out seeks to make AI agents accessible to everyone. Step 1 — Hearing (audio → text) We built the ears first. transcribe.py takes the Meta Ray-Ban glasses microphone and streams raw PCM16 audio to Deepgram's nova-3 model over a WebSocket, getting back live interim and final transcripts. We turned on Deepgram's entity detection early because we knew we'd later need to resolve pronouns like "buy them." We made it reconnect automatically with exponential backoff so a long listening session survives network blips. Step 2 — Understanding (text → intent) Next, the brain: agent.py. We gave Claude Haiku 4.5 a tool-calling schema and a carefully tuned prompt so it could read messy, real conversational speech and decide what's actually actionable — distinguishing a real plan from chit-chat, an active command from passive admiration ("play this" vs. "I love this song"). Our first concrete action was calendar events, executed through the Google Calendar API with a mock mode so the demo works even before you connect Google. Step 3 — Seeing it (the HUD) + privacy We built a React + Tailwind "glasses HUD" (server.py + web/) that streams live captions and action cards over WebSocket, with the calendar embedded so events appear instantly. Then we added the features that make ambient listening acceptable: an on-device face-presence gate using the laptop webcam (only listens when someone's actually talking to you — camera frames never leave the machine) and an ephemeral transcript buffer that auto-wipes itself 20 seconds after the moment passes. Step 4 — Turning intent into action The hardest part: making "add this to my cart" or "text Sarah I'm running late" actually happen, not just appear as a card on screen. When Claude identifies a non-calendar action, it hands a natural-language intent string to our TypeScript agentic router (src/router.ts) running on a background thread — the optimistic card appears on the HUD immediately while the real work happens in parallel. The router tries three tiers in order. First, it queries the Fetch.ai Agentverse marketplace to find a specialized agent for the task — searching for candidates by capability, then using Claude to judge the best match. Second, if the intent is calendar-shaped, it hits the Google Calendar API directly. Third, and most powerfully, it falls through to Browserbase + Stagehand: a cloud-hosted browser driven by Claude Sonnet 4.6 that can navigate and interact with essentially any website on the internet. The wearer's current location (resolved via IP geolocation) is baked into every intent string so "near me" and directions work correctly. The router streams its reasoning back to the HUD as live "thinking" lines — so you can watch the agent decide, step by step, before the [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 51 recognized source files, 305 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (69 of 69) ``` .env.example .gitignore agent.py ambient_awareness.py archived/find_cams.py archived/schedule_meeting.py archived/scrape_deals.py archived/test_browserbase.py archived/visual_context.py authorize_calendar.py calendar_service/app.py calendar_service/calendar_tool.py calendar_service/requirements.txt calendar_tool.py calendar_tool2.py calendar_uagent/calendar_agent.py calendar_uagent/calendar_tool.py calendar_uagent/requirements.txt demo.py design/DESIGN_BRIEF.md design/PROMPTS.md face_gate.py geo.py listener.sh package.json README.md requirements.txt router_client.py run.sh server.py src/agentverse_search.ts src/browserbase.ts src/calendar_action.ts src/cli.ts src/gate.ts src/llm.ts src/router.ts src/server.ts src/types.ts transcribe.py tsconfig.json web/index.html web/package.json web/src/agent.js web/src/App.jsx web/src/CameraMesh.jsx web/src/index.css web/src/main.jsx web/src/useAgentSocket.js web/vite.config.js WeMightBeCooked/server.py WeMightBeCooked/WeMightBeCooked.xcodeproj/project.pbxproj WeMightBeCooked/WeMightBeCooked.xcodeproj/project.xcworkspace/contents.xcworkspacedata WeMightBeCooked/WeMightBeCooked.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved WeMightBeCooked/WeMightBeCooked.xcodeproj/project.xcworkspace/xcuserdata/michelledong.xcuserdatad/UserInterfaceState.xcuserstate WeMightBeCooked/WeMightBeCooked.xcodeproj/xcuserdata/michelledong.xcuserdatad/xcschemes/xcschememanagement.plist WeMightBeCooked/WeMightBeCooked/Assets.xcassets/AccentColor.colorset/Contents.json WeMightBeCooked/WeMightBeCooked/Assets.xcassets/AppIcon.appiconset/Contents.json WeMightBeCooked/WeMightBeCooked/Assets.xcassets/Contents.json WeMightBeCooked/WeMightBeCooked/ContentView.swift WeMightBeCooked/WeMightBeCooked/Info.plist WeMightBeCooked/WeMightBeCooked/Item.swift WeMightBeCooked/WeMightBeCooked/RayBanCaptureManager.swift WeMightBeCooked/WeMightBeCooked/StreamingDemoView.swift WeMightBeCooked/WeMightBeCooked/WeMightBeCooked.entitlements WeMightBeCooked/WeMightBeCooked/WeMightBeCookedApp.swift WeMightBeCooked/WeMightBeCookedTests/WeMightBeCookedTests.swift WeMightBeCooked/WeMightBeCookedUITests/WeMightBeCookedUITests.swift WeMightBeCooked/WeMightBeCookedUITests/WeMightBeCookedUITestsLaunchTests.swift ``` ### Dependencies - calendar_service/requirements.txt: fastapi, google-api-python-client, google-auth, google-auth-oauthlib, uvicorn - calendar_uagent/requirements.txt: google-api-python-client, google-auth, google-auth-oauthlib, uagents, uagents-core - package.json: @browserbasehq/stagehand@^3.0.0, @types/node@^22.0.0, tsx@^4.19.0, typescript@^5.6.0, zod@^3.23.8 - requirements.txt: anthropic@>=0.40.0, fastapi@>=0.110.0, google-api-python-client@>=2.130.0, google-auth-httplib2@>=0.2.0, google-auth-oauthlib@>=1.2.0, numpy@>=1.24.0, opencv-python@>=4.8.0, python-dotenv@>=1.0.0, sounddevice@>=0.4.7, uvicorn[standard]@>=0.29.0, websockets@>=13.0 - web/package.json: @tailwindcss/vite@^4.0.0, @vitejs/plugin-react@^4.3.4, react@^18.3.1, react-dom@^18.3.1, tailwindcss@^4.0.0, vite@^6.0.7 ### Recent commits (newest first) - Revise README with project details and future plans - Update README.md - Rename IMG_2653.jpeg to remark_logo.jpeg - Add files via upload - i think everything works. i'm so tired - so some stuff is done, like output audio and talking to mark - lowkey no idea what I'm even adding anymore - Merge branch 'main' of github.com:cadencheng888/omgWeRnotscrewed - intent, gotta fix some stuff, it's too late - calendar server - fuck me - Merge branch 'main' of https://github.com/cadencheng888/omgWeRnotscrewed - server for calendar api - end to end piipeline for ev parking is DONEEEE - Update server.py - connected meta ai to deepgram - update - Merge branch 'main' of https://github.com/cadencheng888/omgWeRnotscrewed - Wire audio pipeline into agentic router + location-aware intents - Add files via upload ## Key source files (fetched from GitHub, selected and truncated for size) ### design/DESIGN_BRIEF.md ```markdown # Design Brief — Glasses Agent > Attach or paste this whole file as context when prompting Claude to generate UI. > It is the single source of truth for *what the product is* and *how it should look*. ## 1. What it is (the pitch) Smart glasses that **passively listen to your real-world conversations and quietly put your life on your calendar.** You say "let's grab dinner at 6 tonight" out loud to a friend — a few seconds later it's a Google Calendar event. No phone, no typing, no "hey assistant." It just hears the plan and handles it. One-liner for the hero: **"Your calendar, handled by your glasses."** ## 2. How it works (the pipeline — this IS the demo) ``` 🎙️ Mic (Ray-Ban Meta / laptop) │ raw audio ▼ 🌊 Deepgram (nova-3) → live speech-to-text │ transcript text ▼ 🧠 Claude (haiku-4-5) → decides: is this an action? which tool? │ tool call ▼ 📅 Google Calendar / Tasks → event created / cancelled / rescheduled ``` Claude has 4 "tools" it can choose: **create event, cancel event, reschedule event, create task.** It also decides to do *nothing* for vague talk ("we should hang sometime"). The magic to visualize: **the gap between hearing and acting is automatic and invisible.** The UI should make that pipeline feel alive and intelligent. ## 3. Who's watching Hackathon judges + a live demo on a projector. So: legible from across a room, looks impressive in motion, tells the pipeline story in one glance. ## 4. The screens to design 1. **Live Demo Dashboard (HERO — build this first).** Real-time view of the pipeline: listening state, live transcript streaming in, Claude's decision appearing as a card, the calendar event materializing. This is what runs on screen during the demo. 2. **Today / Agenda view.** The calendar the agent has been quietly filling — clean, glanceable, shows events with the little detail Claude inferred (title, time, people). 3. **Landing / hero splash.** One screen for the Devpost & intro slide. The pitch line, a glasses visual, the 3-step pipeline. 4. *(stretch)* **Glasses HUD overlay.** First-person AR-style view: what the wearer "sees" — a subtle confirmation toast floating in the corner when an event is captured. ## 5. Visual direction **Mood:** futuristic, calm, premium hardware. Apple x Linear x a16z. NOT a busy SaaS dashboard. Lots of negative space. Motion is the personality. **Theme:** dark mode primary. Near-black background, not pure black. **Palette (suggested — refine freely):** - Background: `#0A0A0F` → `#0E0E16` (subtle vertical gradient) - Surface / cards: `#16161F` with 1px `#262633` border, soft glassmorphism blur - Primary accent: electric indigo→cyan gradient `#6366F1 → #22D3EE` - Success (event created): mint `#34D399` - Warning (cancel): amber `#FBBF24` - Text: `#F5F5F7` primary, `#8A8A99` secondary - "Listening" pulse: cyan glow **Typography:** - UI / headings: Inter or Geist (tight tracking on big headings) - Transcript + [truncated — 1784 more characters] ``` ### design/PROMPTS.md ```markdown # Copy-paste prompts for Claude (artifacts / design) How to use: **first attach `DESIGN_BRIEF.md`** (or paste it), then send one prompt below. Build them in order — Screen 1 is the hero and sets the visual system the rest reuse. After the first artifact looks right, prompt 5 locks the system so the others match. --- ## Prompt 1 — Live Demo Dashboard (the hero) ``` Using the attached DESIGN_BRIEF.md, design the Live Demo Dashboard as a single self-contained React + Tailwind artifact. This is a real-time view of a smart-glasses agent that hears conversations and creates calendar events. Lay it out as a 3-stage horizontal pipeline with a visible flowing connection between stages: [ LISTENING ] → [ CLAUDE THINKING ] → [ CALENDAR ] - LISTENING (left): a live, reacting audio waveform/pulse in cyan, a "Listening…" label, and a stream of transcript lines appearing in monospace as they're "heard." - CLAUDE THINKING (center): when a transcript implies a plan, show a card that reveals Claude's decision — the tool name (e.g. create_calendar_event) and the extracted fields (title, time, participants) in monospace, like structured output forming. - CALENDAR (right): finished event cards materialize with a satisfying pop + mint glow. Show "Dinner with Alex · Tomorrow 7:00 PM" style cards stacking newest-on-top. Animate the whole flow on a loop using hardcoded data + setTimeout so it plays like a live demo: hear → think → create, staggered, eased, ~400ms transitions. Include at least one CREATE, one CANCEL (event greys out + strikethrough), one TASK (checklist item), and one "nothing actionable" line that softly dismisses. Dark mode, glassmorphism cards, indigo→cyan accent, lots of negative space, premium and calm. Make motion the personality. It must look impressive on a projector from across a room. ``` --- ## Prompt 2 — Today / Agenda view ``` Using the same visual system as the dashboard, design a "Today" agenda screen: the calendar the glasses agent has been quietly filling. A clean vertical timeline of the day with event cards the agent created. Each card shows title, time, duration, participants, and a tiny "captured by glasses 🎙️" tag with a relative timestamp ("2m ago"). Mix event types: Coffee, Study Session, Dinner with Alex, a Call, plus one Google Task (a to-do without a time, shown distinctly with a checkbox). Show one event in a "cancelled" state (struck through, dimmed) and one freshly-created event still glowing. Dark mode, glassmorphism, indigo→cyan, generous spacing. Glanceable and premium — Linear/Apple energy, not a busy SaaS calendar. ``` --- ## Prompt 3 — Landing / hero splash ``` Using the same visual system, design a single landing/hero screen for this project, suitable for a Devpost header and the intro slide of a live demo. Center the headline "Your calendar, handled by your glasses." with a one-line subhead explaining it passively listens to conversations and creates calendar events automatically. Incl [truncated — 1845 more characters] ``` ### package.json ``` { "name": "intent-router", "version": "0.1.0", "private": true, "type": "module", "scripts": { "dev": "tsx --env-file=.env src/cli.ts", "serve": "tsx --env-file=.env src/server.ts", "build": "tsc" }, "dependencies": { "@browserbasehq/stagehand": "^3.0.0", "zod": "^3.23.8" }, "devDependencies": { "tsx": "^4.19.0", "typescript": "^5.6.0", "@types/node": "^22.0.0" } } ``` ### requirements.txt ``` # Numerical / array support (also pulled in by opencv, but declared explicitly) numpy>=1.24.0 # Audio capture + Deepgram streaming sounddevice>=0.4.7 websockets>=13.0 # Claude (intent extraction + tool use) anthropic>=0.40.0 # Google Calendar / Tasks google-api-python-client>=2.130.0 google-auth-oauthlib>=1.2.0 google-auth-httplib2>=0.2.0 # Config python-dotenv>=1.0.0 # Web HUD (demo dashboard) fastapi>=0.110.0 uvicorn[standard]>=0.29.0 # Face presence gate (on-device, Conversation mode) opencv-python>=4.8.0 ``` ### calendar_service/requirements.txt ``` fastapi uvicorn google-auth google-auth-oauthlib google-api-python-client ``` ### calendar_uagent/requirements.txt ``` uagents uagents-core google-auth google-auth-oauthlib google-api-python-client ``` ### web/package.json ``` { "name": "hearsay-hud", "private": true, "version": "0.1.0", "type": "module", "scripts": { "dev": "vite", "build": "vite build", "preview": "vite preview" }, "dependencies": { "react": "^18.3.1", "react-dom": "^18.3.1" }, "devDependencies": { "@tailwindcss/vite": "^4.0.0", "@vitejs/plugin-react": "^4.3.4", "tailwindcss": "^4.0.0", "vite": "^6.0.7" } } ``` ### calendar_service/app.py ```python """ Minimal local HTTP wrapper around calendar_tool.py so the TypeScript router can call it with plain fetch — no Python/Node interop needed. Run: uvicorn calendar_service.app:app --port 8787 Then: POST http://localhost:8787/events (create) GET http://localhost:8787/events (list) PATCH http://localhost:8787/events/{event_id} (update) DELETE http://localhost:8787/events/{event_id}(delete) """ from fastapi import FastAPI, HTTPException from pydantic import BaseModel from typing import Optional import calendar_tool as cal app = FastAPI(title="calendar-action-service") class CreateEventBody(BaseModel): title: str start_iso: str duration_minutes: int = 60 location: Optional[str] = None notes: Optional[str] = None class UpdateEventBody(BaseModel): title: Optional[str] = None start_iso: Optional[str] = None duration_minutes: Optional[int] = None location: Optional[str] = None notes: Optional[str] = None @app.get("/health") def health(): return {"ok": True} @app.post("/events") def create_event(body: CreateEventBody): try: return cal.create_event( title=body.title, start_iso=body.start_iso, duration_minutes=body.duration_minutes, location=body.location, notes=body.notes, ) except Exception as e: raise HTTPException(status_code=500, detail=str(e)) @app.get("/events") def list_events(time_min_iso: Optional[str] = None, time_max_iso: Optional[str] = None): try: return cal.list_events(time_min_iso=time_min_iso, time_max_iso=time_max_iso) except Exception as e: raise HTTPException(status_code=500, detail=str(e)) @app.patch("/events/{event_id}") def update_event(event_id: str, body: UpdateEventBody): try: return cal.update_event( event_id=event_id, title=body.title, start_iso=body.start_iso, duration_minutes=body.duration_minutes, location=body.location, notes=body.notes, ) except Exception as e: raise HTTPException(status_code=500, detail=str(e)) @app.delete("/events/{event_id}") def delete_event(event_id: str): try: message = cal.delete_event(event_id) return {"message": message} except Exception as e: raise HTTPException(status_code=500, detail=str(e)) ``` ### src/server.ts ```typescript // HTTP wrapper around route() so the Python audio pipeline can hand off an // intent string and get back the router's structured result. // // npm run serve (tsx --env-file=.env src/server.ts) // POST /route { "intent": "add AirPods to my cart" } -> RouteResult JSON // GET /health // // This is the missing bridge between the perception half (Python: mic -> // Deepgram -> Claude -> intent string) and the execution half (this TS router: // Agentverse / Calendar / Browserbase). Any intent is accepted — calendar is // just one tier inside route(). import { createServer, type ServerResponse } from "node:http"; import { route } from "./router"; const PORT = Number(process.env.ROUTER_PORT) || 8788; function send(res: ServerResponse, code: number, obj: unknown) { res.writeHead(code, { "Content-Type": "application/json" }); res.end(JSON.stringify(obj)); } const server = createServer((req, res) => { if (req.method === "GET" && req.url === "/health") { return send(res, 200, { ok: true }); } if (req.method !== "POST" || req.url !== "/route") { return send(res, 404, { error: "use POST /route or GET /health" }); } let body = ""; req.on("data", (chunk) => (body += chunk)); req.on("end", async () => { let intent = ""; try { intent = String(JSON.parse(body).intent ?? "").trim(); } catch { return send(res, 400, { error: "body must be JSON: { intent: string }" }); } if (!intent) return send(res, 400, { error: "missing 'intent'" }); console.log(`\n→ route("${intent}")`); // Stream newline-delimited JSON: one {type:"trace"} per reasoning line as it // happens, then a final {type:"result"}. Lets the HUD show the agent think // live during a 30-90s Browserbase run instead of a silent wait. res.writeHead(200, { "Content-Type": "application/x-ndjson", "Cache-Control": "no-cache", }); const write = (obj: unknown) => res.write(JSON.stringify(obj) + "\n"); try { const result = await route(intent, (line) => write({ type: "trace", line })); console.log(`← ${result.source}/${result.status}`); write({ type: "result", result }); } catch (e) { console.error("route failed:", e); write({ type: "result", result: { source: "none", status: "failed", payload: { error: (e as Error).message }, trace: [], }, }); } res.end(); }); }); server.listen(PORT, () => { console.log(`intent router listening → http://localhost:${PORT}/route`); }); ``` ### src/cli.ts ```typescript import { route } from "./router"; import { RouteResult } from "./types"; const DIV = "─".repeat(64); function header(text: string) { console.log("\n" + DIV); console.log(" " + text); console.log(DIV); } function labeledSource(source: RouteResult["source"]): string { switch (source) { case "agentverse": return "Fetch.ai Agentverse (specialized agent)"; case "calendar": return "Calendar Action (Google Calendar API)"; case "browserbase": return "Browserbase (autonomous web agent)"; default: return source; } } function renderTrace(trace: string[]) { header("ROUTING"); for (const line of trace) { const idx = line.indexOf(":"); if (idx === -1) { console.log(" " + line); continue; } const tier = line.slice(0, idx).trim(); const msg = line.slice(idx + 1).trim(); // Gate lines carry a verdict — flag it visibly so the one match (if any) // doesn't read identically to the rejections around it. let marker = ""; if (tier === "gate") { if (msg.startsWith("selected ")) { marker = " [MATCH] "; } else if (msg.includes("capable=true")) { marker = " [PASS] "; } else if (msg.includes("capable=false")) { marker = " [reject] "; } } console.log(" " + tier.padEnd(12) + marker + msg); } } function renderResult(result: RouteResult) { header("RESULT"); console.log(" Handled by " + labeledSource(result.source)); console.log(" Status " + result.status.toUpperCase()); const p = result.payload as Record | null; const selected = p?.agentverseSelected as | { name: string; address: string; confidence: number } | undefined; if (selected) { console.log(""); console.log(" Agentverse selected: " + selected.name); console.log(" Confidence: " + selected.confidence.toFixed(2)); console.log(" Address: " + selected.address); console.log( " (invocation not yet wired — task completed by " + labeledSource(result.source).trim() + ")" ); } console.log(""); if (!p) { console.log(" (no payload)"); return; } const preferred = [ "outcome", "details", "actionTaken", "stoppedBecause", "message", ]; for (const key of preferred) { if (p[key] != null && p[key] !== "") { console.log(" " + key.padEnd(15) + String(p[key])); } } for (const [key, val] of Object.entries(p)) { if (preferred.includes(key) || key === "agentverseSelected") continue; if (val == null || val === "") continue; const rendered = typeof val === "object" ? JSON.stringify(val) : String(val); console.log(" " + key.padEnd(15) + rendered); } } async function main() { const intent = process.argv.slice(2).join(" ").trim(); if (!intent) { console.error('Usage: npm run dev -- "the user wants to do X"'); process.exit(1); } header("INTENT"); console.log(" " + intent); const result = await route(intent); renderTrace(result.trace); renderResult(result); console.log(""); } main().catch((e) => { console.error(e); process.exit(1); }); ``` [45 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]