# Project export: MoneyPenny
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: UC Berkeley AI Hackathon 2026
- Tagline: Super Secured Secretary - Nothing happens without you
- Devpost: https://devpost.com/software/moneypenny-7a0rfu
- GitHub: https://github.com/DuongAnh1201/sss
- Video: https://www.youtube.com/embed/KW9GjIJ5nEM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 4 GitHub contributor(s) — DuongAnh1201 (43 commits), MynameisKoi (12 commits), Claude Sonnet 4.6 (8 commits), Cursor (7 commits)
## Devpost submission (written by the team)
### Overview
Ethical AI, Privacy, and Human Oversight MoneyPenny was built around a simple belief: the future of AI should not be measured solely by how autonomous an agent can become, but by how trustworthy, transparent, and accountable it remains while acting on behalf of people. From the beginning, we made a deliberate decision not to pursue fully autonomous agent architectures. While modern agent ecosystems continue to push the boundaries of autonomous execution and self-directed behavior, they also introduce significant concerns around security, privacy, accountability, and the broader societal impact of increasingly powerful AI systems. Rather than maximizing autonomy, we chose to maximize human control. This philosophy shaped every technical decision in MoneyPenny. Our system is built as a multi-agent architecture with clearly defined responsibilities, strict delegation rules, and controlled communication channels between agents. Each agent specializes in a specific domain—email, scheduling, memory, communication, knowledge retrieval, or coordination—while a central orchestrator routes tasks and enforces system-wide safety policies. By limiting agent authority and clearly defining operational boundaries, we reduce the risks associated with unrestricted autonomous behavior. At the core of the system is PydanticAI, which powers both the orchestrator and specialist agents. We chose PydanticAI because of its strong support for typed, structured outputs and deterministic tool execution. Rather than allowing language models to directly invoke external tools, agents emit validated ProposedAction objects that are subsequently reviewed and classified by code-defined policies. In this architecture, the model proposes actions, but it never decides whether those actions are allowed to execute. To support collaboration beyond a single user, MoneyPenny is deployed on Fetch.ai Agentverse and is accessible through ASI:ONE. Through Fetch.ai's agent ecosystem, MoneyPenny agents can discover, communicate with, and coordinate with other agents using standardized protocols. This enables scenarios such as two personal assistants negotiating meeting times on behalf of their owners or hiring specialized agents from an open marketplace to perform tasks outside their own capabilities. However, regardless of where a request originates or which agent generates a response, every interaction is routed through our Consent and Governance Layer. This layer acts as a bidirectional safety mechanism that continuously evaluates communication between users and agents. On the input side, the layer analyzes user requests for unsafe, malicious, or potentially harmful use of AI capabilities. On the output side, it reviews agent-generated responses and proposed actions against predefined safety, privacy, and compliance policies. Every consequential action—sending emails, modifying calendars, sharing files, contacting other agents, or performing transactions—must pass through this validation process before execution. The enforcement mechanism is implemented through a hard-coded consent gate. Every side-effecting action requires explicit approval from the user and receives a single-use consent token before execution. Even if an agent attempts to bypass the approval process, execution wrappers verify the presence and validity of the consent token before any external action can occur. If validation fails, the system fails closed and no action is performed. To make consent auditable rather than merely procedural, every approval, denial, revision, and execution outcome is recorded in an append-only consent ledger backed by Redis. This ledger provides a verifiable history of user decisions and enables continuous auditing of agent behavior. MoneyPenny evaluates its own execution traces against this ledger to ensure that no action was performed without corresponding user approval. Our design philosophy can be summarized as keeping humans over the loop, not merely in the loop. Humans retain ultimate authority over what agents are allowed to do, how they collaborate, and when real-world actions occur. Agents can negotiate, coordinate, and reason autonomously, but they cannot independently commit users to consequential actions. By combining structured multi-agent orchestration through PydanticAI, open agent communication through Fetch.ai, and a comprehensive consent and governance framework, MoneyPenny demonstrates that advanced AI systems can be both powerful and responsible. Privacy, security, transparency, and ethical AI are not features added after the fact; they are foundational principles embedded directly into the architecture of the system.
## README (from the GitHub repository)
# MoneyPenny — The Assistant That Asks First
> *Your trusted right hand. It does real things in the world on your behalf — and never without your word.*
**Built at the UC Berkeley AI Hackathon.**
---
## The Problem
Today's AI can write you a beautiful email. It cannot send it. It can suggest three times for a meeting. It cannot actually find the one that works for both you *and* the person you're meeting. The moment a task touches the real world — your inbox, your calendar, your files, another human being — the AI taps out and hands the work back to you.
The few assistants that *can* act have the opposite problem: they act too freely. They'll run a command, send a message, or change something on your behalf without ever stopping to ask. Convenient — until it does something you didn't want, and you find out after.
**MoneyPenny is built on one principle: an assistant should be able to do real things in the world — but never without your permission.** A great assistant doesn't just do what you say — it knows what to handle, what to check, and what never to send without your word. MoneyPenny guards the line between *what you asked for* and *what actually happens*.
---
## Meet MoneyPenny
MoneyPenny is a voice-driven personal assistant that actually *acts* on your behalf. You speak to it the way you'd ask a capable, trustworthy person:
- *"Email my team that standup moves to 10."*
- *"Save these notes to my Drive."*
- *"What did I tell you about the Henderson project last week?"*
- *"Set up coffee with Sam sometime next week."*
It understands you, figures out what needs to happen, and does it — **but every action with real consequences pauses for your approval first.** You see exactly what it's about to do and say "send it," "cancel," or "change the time" — out loud. Nothing leaves your hands without your word.
That's the whole personality of MoneyPenny: **capable, but never presumptuous.**
---
## The Big Idea: Your MoneyPenny Talks to Mine
This is where MoneyPenny goes somewhere new.
Most assistants live on an island. They can act for *you*, but they can't reach anyone else's assistant. So the hardest, most annoying coordination problems — *"when are we both free?"*, *"can your side handle this part?"* — still land back on two humans emailing each other.
**MoneyPenny agents can find and talk to one another.**
When you ask MoneyPenny to set up coffee with Sam, your MoneyPenny doesn't email Sam. It finds **Sam's MoneyPenny**, and the two assistants negotiate directly — comparing calendars, proposing times, ruling out conflicts — then come back to each of you with a single answer to approve. Two assistants did the back-and-forth. Two humans just said "yes."
And it isn't limited to people you know. MoneyPenny can also reach across an open network of agents to **hire a specialist** — a restaurant-booking agent, a flight-finder, a research agent — for jobs your own MoneyPenny can't do alone.
The principle holds the whole way down: **agents negotiate, humans decide.** Even when my MoneyPenny is talking to yours, neither of us can be committed to anything until each owner approves it. Each MoneyPenny looks after its own boss's side of the deal. Consent isn't a feature bolted on top — it's the rule the entire network runs by.
---
## How It Works — A Day With MoneyPenny
**Morning.** You sit down, tap the power button, and MoneyPenny greets you by name. It remembers you — your preferences, your contacts, what you worked on yesterday.
**A quick email.** *"Email Priya that the deck is ready."* MoneyPenny drafts it and shows you a review card. You glance at it: *"Make it a little more casual."* It rewrites. *"Send it."* Gone. A confirmation appears, and the action is quietly recorded in your consent log — proof of exactly what you approved.
**Coordinating with another human.** *"Find a time for a 30-minute sync with Marcus this week."* Marcus also uses MoneyPenny. Behind the scenes, your assistant and his trade proposals against both calendars and land on Thursday at 2. Each of you gets one clean question: *"Thursday at 2pm work?"* You both say yes. Booked. Neither of you sent a single "does this work for you?" message.
**Reaching beyond your circle.** *"Book us a table somewhere good near the office for four on Friday."* Your MoneyPenny doesn't know restaurants — so it hires an agent that does, out on the open network. It comes back with options, you pick one, you approve the booking. The specialist agent is paid automatically for its help.
**Throughout, you're in control.** Every consequential step — the email, the meeting, the reservation — waited for your "yes." And every one of them is traceable: you can see what MoneyPenny did, why, and that nothing happened without you.
---
## Key Features
- **Voice-first.** Talk to MoneyPenny naturally. It listens, thinks, and answers out loud, in real time.
- **It actually does things.** Sends email, manages your calendar, searches the web, messages and calls people, and saves files to Google Drive.
- **Consent gate on every real action.** Anything with consequences pauses for your spoken approval — approve, cancel, or revise.
- **Agent-to-agent coordination.** Your MoneyPenny can talk to other people's MoneyPenny agents to handle two-sided tasks like scheduling.
- **An open agent network.** MoneyPenny can discover and hire specialist agents for jobs it can't do alone.
- **It remembers you.** Preferences, contacts, and context carry across sessions — it gets more useful the more you use it.
- **A consent ledger.** Every approval and denial is logged, so there's always a clear record of what MoneyPenny did on your behalf.
- **Provable trust.** MoneyPenny continuously checks its own behavior to confirm that no action ever bypassed your approval — and can show you the proof.
---
## The Consent Principle
Most agentic AI optimizes for *seamlessness* — fewer interruptions, more autonomy, get out of the user's way. MoneyPenny deliberately does the opposite where it counts.
We believe the assistants that earn a real place in people's lives won't be the ones that do the most on their own — they'll be the ones people **trust** to do things on their own. And trust isn't a vibe; it's a guarantee you can verify.
So MoneyPenny makes consent a structural property, not a polite habit:
1. **Every consequential action stops for approval.** Sending, booking, sharing, spending — all of it waits for you.
2. **Every decision is recorded.** The consent ledger is an honest, reviewable history of what you approved.
3. **The guarantee is checked, not just claimed.** MoneyPenny evaluates its own traces against that ledger to confirm nothing slipped through. If an action ever fired without approval, we'd know — and so would you.
*Other assistants ask you to trust that they did the right thing. MoneyPenny lets you check. Like the assistant it's named for, it's the desk everything passes through — and nothing reaches the outside world without going through you.*
---
## How MoneyPenny Is Different
The pieces exist separately in 2026 — but the combination doesn't. Agent-to-agent protocols are built to remove humans from the loop. Consumer assistants that take actions optimize for seamlessness, not consent. AI schedulers negotiate with the *other person*, not with their assistant.
| Capability | Agent-to-agent protocols | Big-tech assistants | AI schedulers | **MoneyPenny** |
|---|---|---|---|---|
| Agents discover & talk to each other | ✅ | ❌ | ❌ | ✅ |
| Takes real-world actions | enterprise | ✅ | scheduling only | ✅ |
| Per-action consent gate (by voice) | ❌ | ⚠️ minimal | ❌ | ✅ core |
| **Two-sided human approval** in agent-to-agent | ❌ | ❌ | ❌ | ✅ |
| Personal / peer-to-peer | ❌ | ✅ | ✅ | ✅ |
| Provable, auditable trust | ❌ | ❌ | ⚠️ | ✅ |
Everyone else is racing to take humans *out* of the loop. MoneyPenny deliberately keeps them in — and makes that verifiable.
---
## Architecture
```
[README truncated for size]
## Detected evidence (automated analysis)
Indexed codebase: 155 recognized source files, 590 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Redis (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- AI coding agent: Claude Code — evidence: commit authorship or trailers
- AI coding agent: Cursor — evidence: commit authorship or trailers
## Codebase structure (from repository index)
### Files (120 of 171)
```
.consent/ledger.jsonl
.env.example
.gitignore
.python-version
ai/__init__.py
ai/agents/__init__.py
ai/agents/agent1.py
ai/agents/agent2.py
ai/agents/agent3.py
ai/agents/agent4.py
ai/agents/agent5.py
ai/agents/agent6.py
ai/agents/agent7.py
ai/agents/agent8.py
ai/agents/consent_token.py
ai/agents/consent.py
ai/agents/deps.py
ai/agents/orchestrator.py
ai/prompts/__init__.py
ai/prompts/agentverse_agent.md
ai/prompts/calendar_agent.md
ai/prompts/communication_agent.md
ai/prompts/drive_agent.md
ai/prompts/email_agent.md
ai/prompts/gmail_agent.md
ai/prompts/knowledge_base_agent.md
ai/prompts/orchestrator.md
ai/prompts/search_agent.md
ai/prompts/tombio.md
ai/session/__init__.py
ai/session/clock.py
ai/session/deps_factory.py
ai/transport/__init__.py
ai/transport/bridge.py
ai/transport/fetch_wrapper.py
backend/__init__.py
backend/approval.py
backend/protocol.py
backend/session.py
backend/voice_pipeline.py
config.py
doc/__init__.py
doc/IMPLEMENTATION_PLAN.md
doc/PHASE_1.md
doc/SOUL.md
Dockerfile
Dockerfile.fetch
Dockerfile.server
docs/consent-architecture/01-consent-gate.md
docs/consent-architecture/02-fallback-matrix.md
docs/consent-architecture/03-observability-and-ledger.md
docs/consent-architecture/04-trust-tiers.md
docs/consent-architecture/README.md
docs/multi-agent-system.md
docs/observability/phoenix.md
docs/workspace-integration/README.md
frontend/.dockerignore
frontend/Dockerfile
frontend/index.html
frontend/package.json
frontend/public/audio-processor.js
frontend/src/App.tsx
frontend/src/component/RobotFace.tsx
frontend/src/component/ToolStatus.tsx
frontend/src/features/voice-agent/components/ApprovalCard.tsx
frontend/src/features/voice-agent/components/CapabilityCard.tsx
frontend/src/features/voice-agent/components/CapabilityDetailViewer.tsx
frontend/src/features/voice-agent/components/CapabilityPanel.tsx
frontend/src/features/voice-agent/components/ConversationPanel.tsx
frontend/src/features/voice-agent/components/ExecutionTimeline.tsx
frontend/src/features/voice-agent/components/ExecutionTimelineItem.tsx
frontend/src/features/voice-agent/components/SystemStatusBadge.tsx
frontend/src/features/voice-agent/components/VoiceAgentHeader.tsx
frontend/src/features/voice-agent/components/VoiceAgentIcons.tsx
frontend/src/features/voice-agent/components/VoiceAgentOrb.tsx
frontend/src/features/voice-agent/components/VoiceAgentOverlay.tsx
frontend/src/features/voice-agent/components/VoiceCommandBar.tsx
frontend/src/features/voice-agent/data/mockVoiceFlows.ts
frontend/src/features/voice-agent/hooks/useVoiceAgentUIState.ts
frontend/src/features/voice-agent/index.ts
frontend/src/features/voice-agent/types/voiceAgent.types.ts
frontend/src/features/voice-agent/utils/approvalState.js
frontend/src/features/voice-agent/utils/approvalTimeline.js
frontend/src/features/voice-agent/utils/executionPresentation.ts
frontend/src/features/voice-agent/utils/voiceAgentEventAdapter.ts
frontend/src/features/voice-agent/utils/voiceAgentLayoutTokens.ts
frontend/src/index.css
frontend/src/main.tsx
frontend/src/service/audioService.ts
frontend/src/types.ts
frontend/tests/approval-state.test.mjs
frontend/tests/approval-timeline.test.mjs
frontend/vite.config.ts
knowledge_base/Calendar.md
knowledge_base/Mandatory Class.md
knowledge_base/School Tasks.md
LICENSE
main.py
memory/__init__.py
memory/execution_log.py
memory/graph.py
observability/__init__.py
observability/consent_trace.py
observability/evaluator.py
observability/kill_switch.py
observability/phoenix_config.py
observability/phoenix.py
observability/spans.py
pyproject.toml
railway.toml
README.md
run_text.py
schemas/__init__.py
schemas/agent1.py
schemas/agent2.py
schemas/agent3.py
schemas/agent4.py
schemas/agent5.py
schemas/agent6.py
schemas/agent7.py
[51 more files omitted for size]
```
### Dependencies
- frontend/package.json: @tailwindcss/vite@^4.2.1, @types/react@^19.2.14, @types/react-dom@^19.2.3, @vitejs/plugin-react@^5.1.4, clsx@^2.1.1, lucide-react@^0.577.0, motion@^12.35.0, react@^19.0.0, react-dom@^19.0.0, tailwind-merge@^3.5.0, tailwindcss@^4.2.1, typescript@^5.9.3, vite@^7.3.1
- pyproject.toml: aiosqlite, arize-phoenix@>=8.0.0, asyncpg@>=0.30.0, datetime@>=6.0, deepgram-sdk@>=7.3.1, email-validator, fastapi[standard]@>=0.135.2, google-api-python-client@>=2.192.0, google-auth-httplib2@>=0.3.0, google-auth-oauthlib@>=1.3.0, httpx@>=0.28.0, jose@>=1.0.0, numpy@>=2.0.0, openai@>=1.75.0, openinference-instrumentation-pydantic-ai@>=0.1.16, opentelemetry-exporter-otlp-proto-http@>=1.30.0, opentelemetry-sdk@>=1.30.0, passlib[bcrypt], pydantic-ai@>=1.67.0, pydantic-settings@>=2.13.1, pytest@>=9.0.2, python-dotenv@>=1.2.2, python-jose[cryptography]@>=3.5.0, redis@>=8.0.0, resen@>=2021.1.0, sounddevice@>=0.5.1, sqlalchemy[asyncio], uagents@>=0.25.2, uvicorn, watchfiles@>=0.22.0, websockets@>=13.0
### Recent commits (newest first)
- Merge origin/main: keep MoneyPenny branding
- final update
- Rename Désir branding to MoneyPenny across UI, agents, and config.
- update(final of final)
- update date and time
- Merge branch 'main' of https://github.com/DuongAnh1201/sss
- upload adapting soul
- update: log
- resolve the conflict between ledge and timestamp by removing datetime and timezone
- update agent 8
- update to resolve mailbox connection
- update orchestrator schemas
- update(agentverse)
- update(agent 8): update protocol A2A using agentverse
- update to mvp
- fix: correct Dockerfile CMD syntax — remove invalid line continuation in JSON array
- fix: unified Dockerfile — SERVICE env var selects server vs fetch agent
- feat: wire Deepgram STT+TTS voice pipeline into WebSocket session
- feat: auto-register as chat agent on Agentverse at startup
- ci: trigger Railway rebuild
## Key source files (fetched from GitHub, selected and truncated for size)
### knowledge_base/Mandatory Class.md
```markdown
Mandatory class on June 22, 2026, at 10 AM.
```
### knowledge_base/School Tasks.md
```markdown
Tasks related to school assignments and responsibilities.
```
### Dockerfile
```
FROM python:3.13-slim
WORKDIR /app
RUN pip install uv --no-cache-dir
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev
COPY . .
# Set SERVICE=server in Railway env vars for the WebSocket server service.
# Leave unset (or set to anything else) for the Fetch.ai uAgent service.
CMD ["sh", "-c", "if [ \"$SERVICE\" = 'server' ]; then uv run python server.py; else uv run python -m ai.transport.fetch_wrapper; fi"]
```
### pyproject.toml
```
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "moneypenny"
version = "0.1.0"
description = "A command-line tool"
requires-python = ">=3.13"
license = "MIT"
authors = [
{ name = "KhoiNguyen", email = "khoiduong2913@gmail.com" },
]
dependencies = [
"google-api-python-client>=2.192.0",
"google-auth-httplib2>=0.3.0",
"google-auth-oauthlib>=1.3.0",
"arize-phoenix>=8.0.0",
"openinference-instrumentation-pydantic-ai>=0.1.16",
"opentelemetry-sdk>=1.30.0",
"opentelemetry-exporter-otlp-proto-http>=1.30.0",
"openai>=2.26.0",
"pydantic-ai>=1.67.0",
"pydantic-settings>=2.13.1",
"pytest>=9.0.2",
"python-dotenv>=1.2.2",
"sounddevice>=0.5.5",
"sounddevice>=0.5.1",
"numpy>=2.0.0",
"websockets>=13.0",
"httpx>=0.28.0",
"openai>=1.75.0",
"resen>=2021.1.0",
"watchfiles>=0.22.0",
"fastapi[standard]>=0.135.2",
"jose>=1.0.0",
"datetime>=6.0",
"asyncpg>=0.30.0",
"passlib[bcrypt]",
"sqlalchemy[asyncio]",
"aiosqlite",
"uvicorn",
"email-validator",
"python-jose[cryptography]>=3.5.0",
"deepgram-sdk>=7.3.1",
"redis>=8.0.0",
"uagents>=0.25.2",
]
[project.scripts]
moneypenny = "moneypenny:main"
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["."]
markers = [
"integration: live tests that hit real Google APIs (opt-in via RUN_GOOGLE_INTEGRATION=1)",
"phoenix_integration: live OTLP export to Phoenix Cloud (requires PHOENIX_COLLECTOR_ENDPOINT + PHOENIX_API_KEY)",
]
[tool.hatch.build.targets.wheel]
packages = ["src/moneypenny"]
```
### frontend/Dockerfile
```
FROM node:22-alpine
WORKDIR /app
COPY package.json package-lock.json* ./
RUN npm install
COPY . ./
EXPOSE 5173
CMD ["npm", "run", "dev", "--", "--host", "0.0.0.0"]
```
### frontend/package.json
```
{
"name": "frontend",
"version": "1.0.0",
"description": "",
"main": "index.js",
"scripts": {
"dev": "vite",
"build": "tsc && vite build",
"preview": "vite preview",
"test": "echo \"Error: no test specified\" && exit 1"
},
"keywords": [],
"author": "",
"license": "ISC",
"type": "module",
"dependencies": {
"clsx": "^2.1.1",
"lucide-react": "^0.577.0",
"motion": "^12.35.0",
"react": "^19.0.0",
"react-dom": "^19.0.0",
"tailwind-merge": "^3.5.0"
},
"devDependencies": {
"@tailwindcss/vite": "^4.2.1",
"@types/react": "^19.2.14",
"@types/react-dom": "^19.2.3",
"@vitejs/plugin-react": "^5.1.4",
"tailwindcss": "^4.2.1",
"typescript": "^5.9.3",
"vite": "^7.3.1"
}
}
```
### main.py
```python
def main():
print("Hello from warden!")
if __name__ == "__main__":
main()
```
### server.py
```python
"""MoneyPenny backend server — Phase 2.
FastAPI + WebSocket bridge between the browser and the Pydantic AI orchestrator.
Run:
uv run python server.py
# WebSocket: ws://localhost:8765/ws
# Health: http://localhost:8765/health
# Ledger: http://localhost:8765/api/ledger
"""
from __future__ import annotations
import logging
import os
import uvicorn
from fastapi import FastAPI, Request, WebSocket
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import HTMLResponse, RedirectResponse
from backend.protocol import serialize_ledger_entry
from backend.session import AgentSession
from observability.phoenix import setup_observability
from tools.ledger import get_ledger
logger = logging.getLogger(__name__)
HOST = os.getenv("SERVER_HOST", "0.0.0.0")
PORT = int(os.getenv("PORT", os.getenv("SERVER_PORT", "8765")))
def create_app() -> FastAPI:
setup_observability()
app = FastAPI(title="MoneyPenny", version="0.1.0")
app.add_middleware(
CORSMiddleware,
allow_origins=os.getenv("CORS_ORIGINS", "*").split(","),
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@app.get("/health")
async def health() -> dict[str, str]:
return {"status": "ok", "service": "moneypenny"}
@app.get("/api/ledger")
async def ledger_history(limit: int = 20) -> dict:
entries = await get_ledger().history(limit=min(limit, 100))
return {"entries": [serialize_ledger_entry(e) for e in entries]}
# ── Google Workspace OAuth ────────────────────────────────────────────────
@app.get("/api/workspace/status")
async def workspace_status() -> dict:
from tools.google_auth import granted_scopes, SCOPE_CATALOG
scopes = granted_scopes()
connected: dict[str, str] = {}
for surface, spec in SCOPE_CATALOG.items():
for level, info in spec["levels"].items():
if level == "off":
continue
if all(s in scopes for s in info["scopes"]) and info["scopes"]:
connected[surface] = level
break
else:
connected[surface] = "off"
return {"connected": bool(scopes), "surfaces": connected, "scopes": scopes}
@app.get("/api/workspace/connect")
async def workspace_connect(
drive: str = "file",
gmail: str = "read",
calendar: str = "manage",
):
"""Start the Google OAuth flow. Redirects the browser to Google's consent screen."""
from config import settings
from tools.google_auth import resolve_scopes
if not settings.google_client_id or not settings.google_client_secret:
return {"error": "Google OAuth not configured (GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET missing)"}
selection = {"drive": drive, "gmail": gmail, "calendar": calendar}
try:
scopes = resolve_scopes(selection)
except ValueError as e:
return {"error": str(e)}
if not scopes:
return {"error": "All surfaces set to off — nothing to connect."}
from google_auth_oauthlib.flow import Flow
client_config = {
"web": {
"client_id": settings.google_client_id,
"client_secret": settings.google_client_secret,
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"redirect_uris": [settings.google_redirect_uri],
}
}
flow = Flow.from_client_config(client_config, scopes=scopes)
flow.redirect_uri = settings.google_redirect_uri
auth_url, _ = flow.authorization_url(
access_type="offline",
include_granted_scopes="true",
prompt="consent",
)
return RedirectResponse(auth_url)
@app.get("/oauth2callback")
async def workspace_callback(request: Request):
"""Google redirects here after the user approves. Saves the token and closes."""
from config import settings
from tools.google_auth import save_token
code = request.query_params.get("code")
error = request.query_params.get("error")
if error:
return HTMLResponse(f"OAuth error: {error}
Close this tab and try again.
")
if not code:
return HTMLResponse("Missing code
No authorization code received.
")
from google_auth_oauthlib.flow import Flow
client_config = {
"web": {
"client_id": settings.google_client_id,
"client_secret": settings.google_client_secret,
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"redirect_uris": [settings.google_redirect_uri],
}
}
# Re-derive scopes from the state — use broadest set; granted_scopes() will reflect actual
from tools.google_auth import SCOPE_CATALOG
all_scopes = [s for spec in SCOPE_CATALOG.values()
for lvl in spec["levels"].values() for s in lvl["scopes"]]
all_scopes = sorted(set(all_scopes))
flow = Flow.from_client_config(client_config, scopes=all_scopes)
flow.redirect_uri = settings.google_redirect_uri
try:
flow.fetch_token(code=code)
except Exception as exc:
logger.exception("OAuth token exchange failed")
return HTMLResponse(f"Token exchange failed
{exc}")
creds = flow.credentials
save_token({
"token": creds.token,
"refresh_token": creds.refresh_token,
"token_uri": creds.token_uri,
"client_id": creds.client_id,
"client_secret": creds.client_secret,
"scopes": list(creds.scopes or a
[truncated — 1094 more characters]
```
### frontend/src/main.tsx
```typescript
import {StrictMode} from 'react';
import {createRoot} from 'react-dom/client';
import App from './App.tsx';
import './index.css';
createRoot(document.getElementById('root')!).render(
,
);
```
### frontend/src/App.tsx
```typescript
import { useEffect, useRef, useState } from "react";
import {
VoiceAgentOverlay,
mapRealtimeMessageToAgentEvents,
useVoiceAgentUIState,
} from "./features/voice-agent";
import { AudioRecorder, AudioStreamer } from "./service/audioService";
const WS_URL =
import.meta.env.VITE_WS_URL ||
`ws://${window.location.hostname}:8765/ws`;
interface RuntimeState {
error: string | null;
themeColor: string;
tasks: string[];
}
const INITIAL_RUNTIME_STATE: RuntimeState = {
error: null,
themeColor: "#c8a45c",
tasks: [],
};
function isRecord(value: unknown): value is Record {
return typeof value === "object" && value !== null;
}
function getMicrophoneErrorMessage(error: { name?: string }) {
if (
error.name === "NotAllowedError" ||
error.name === "PermissionDeniedError"
) {
return "Microphone access is blocked. Please allow microphone access in your browser and refresh.";
}
if (error.name === "NotFoundError") {
return "No microphone detected. Please connect a microphone and try again.";
}
if (error.name === "NotSupportedError") {
return "Microphone not supported. Please use Chrome or Edge over HTTPS.";
}
return "Microphone access denied.";
}
export default function App() {
const [runtimeState, setRuntimeState] = useState(INITIAL_RUNTIME_STATE);
const [isPowerOn, setIsPowerOn] = useState(false);
const audioRecorderRef = useRef(null);
const audioStreamerRef = useRef(null);
const wsRef = useRef(null);
const {
state: voiceAgentState,
beginListening,
markSessionConnected,
stopSession,
dispatchEvent,
toggleCapabilityDetail,
} = useVoiceAgentUIState();
useEffect(() => {
document.documentElement.style.setProperty(
"--theme-color",
runtimeState.themeColor,
);
}, [runtimeState.themeColor]);
const sendJson = (payload: Record) => {
if (wsRef.current?.readyState === WebSocket.OPEN) {
wsRef.current.send(JSON.stringify(payload));
}
};
const sendApprovalDecision = (
actionId: string,
decision: "approve" | "cancel",
) => {
sendJson({ type: "approval_decision", action_id: actionId, decision });
};
const sendTextPrompt = (text: string) => {
sendJson({ type: "text", text });
};
const handleApprove = () => {
const actionId = voiceAgentState.approvalRequest?.id;
if (actionId) {
sendApprovalDecision(actionId, "approve");
}
};
const handleCancel = () => {
const actionId = voiceAgentState.approvalRequest?.id;
if (actionId) {
sendApprovalDecision(actionId, "cancel");
}
};
const teardownAssistant = (fromSocket = false) => {
audioRecorderRef.current?.stop();
audioRecorderRef.current = null;
audioStreamerRef.current?.stop();
audioStreamerRef.current = null;
if (!fromSocket && wsRef.current) {
const socket = wsRef.current;
wsRef.current = null;
socket.onclose = null;
socket.close();
} else {
wsRef.current = null;
}
setIsPowerOn(false);
};
const stopAssistant = (fromSocket = false) => {
teardownAssistant(fromSocket);
setRuntimeState(INITIAL_RUNTIME_STATE);
stopSession();
};
const handleFrontendTool = (
callId: string,
name: string,
args: unknown,
ws: WebSocket,
) => {
let result = "done";
if (
name === "changeThemeColor" &&
isRecord(args) &&
typeof args.color === "string"
) {
setRuntimeState((previous) => ({
...previous,
themeColor: args.color,
}));
result = `Theme color updated to ${args.color}`;
} else if (
name === "update_daily_tasks" &&
isRecord(args) &&
Array.isArray(args.tasks)
) {
setRuntimeState((previous) => ({
...previous,
tasks: args.tasks.filter(
(task): task is string => typeof task === "string",
),
}));
result = "Tasks updated.";
}
ws.send(JSON.stringify({ type: "tool_result", call_id: callId, result }));
};
const handleRealtimeMessage = (message: unknown, ws: WebSocket) => {
const agentEvents = mapRealtimeMessageToAgentEvents(message);
if (
isRecord(message) &&
message.type === "audio" &&
typeof message.data === "string"
) {
audioStreamerRef.current?.playChunk(message.data);
}
if (isRecord(message) && message.type === "tool_call") {
if (
typeof message.call_id === "string" &&
typeof message.name === "string"
) {
handleFrontendTool(message.call_id, message.name, message.args, ws);
}
}
if (
isRecord(message) &&
message.type === "error" &&
typeof message.message === "string"
) {
setRuntimeState((previous) => ({
...previous,
error: message.message,
}));
}
if (isRecord(message) && message.type === "transcript") {
setRuntimeState((previous) => ({
...previous,
error: null,
}));
}
agentEvents.forEach((event) => {
dispatchEvent(event);
});
};
const startAssistant = async () => {
if (wsRef.current) {
return;
}
beginListening();
setRuntimeState((previous) => ({
...previous,
error: null,
}));
try {
audioStreamerRef.current = new AudioStreamer();
audioRecorderRef.current = new AudioRecorder((base64) => {
sendJson({ type: "audio", data: base64 });
});
await audioRecorderRef.current.start();
} catch (error) {
audioRecorderRef.current = null;
audioStreamerRef.current = null;
const errorMessage = getMicrophoneErrorMessage(
error as { name?: string },
);
setRuntimeState((previous) => ({
...previous,
error: `${errorMessage} Text input is still available.`,
}));
}
const ws = new WebSocket(WS_URL);
wsRef.current =
[truncated — 1855 more characters]
```
[135 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]