# Project export: Daisy This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Voice-driven, accessibility-first phone assistant that lets blind users operate mobile apps. - Devpost: https://devpost.com/software/aaa-8xq2io - GitHub: https://github.com/iandang312/calhacks-ai - Team: 5 GitHub contributor(s) — vien.nguyen (34 commits), iandang312 (14 commits), Ngan Nguyen (7 commits), Cursor (2 commits), Alec Huang (1 commits) ## Devpost submission (written by the team) ### Inspiration Screen readers like TalkBack let blind and low-vision (BLV) users hear what's on screen, but operating an app still means swiping through it one element at a time. A single everyday task like ordering food, booking a ride, replying to a message, or changing a setting can take dozens of sequential swipes and taps, which is slow, fatiguing, and easy to lose track of. Mainstream voice assistants don't fix this. They answer questions and fire off a few built-in commands, but they can't actually drive third-party apps. We wanted an assistant where a BLV user states a goal once, in plain language, and the agent does the navigating while narrating every step and confirming before anything important, so the user stays fully in control without needing to see the screen. ### What it does Daisy is an accessibility-first Android agent that sits between a user's voice and their phone. After activation (a wake word), the user says what they want. Daisy interprets the intent, explains its plan aloud, asks for confirmation, and then uses Android automation to carry out the multi-step task inside a real app, describing what's on screen as it goes and pausing before any meaningful action. The core goal isn't just app automation. It's independence: letting BLV users complete multi-step mobile tasks by voice, without grinding through linear screen-reader navigation, and without ever losing visibility into what the agent is doing. ### How we built it Daisy combines speech, LLM reasoning, user memory, and Android app control: Deepgram — speech-to-text and text-to-speech, powering the audio-first interface (the primary channel for our users). Claude — interprets the user's goal, plans the multi-step task, and reasons over the current screen to decide the next action. Android Accessibility Service — captures screen state (the same accessibility tree TalkBack uses) and performs taps, swipes, and text entry. Evaluation tooling — tracks whether the agent completed the task the user actually intended. Architecture Rather than one monolithic chatbot, Daisy splits the work across agents with genuinely distinct jobs — a divide-and-conquer design where each stage does one thing well: Wake — a wake word activates the assistant; a persistent audio/visual cue signals it's listening. Listen & transcribe — Deepgram converts the spoken request to text. Planner — Claude reconstructs the user's intent and drafts a step-by-step roadmap for the task. Perceiver — reads the current screen via the Accessibility Service and identifies the relevant elements (search box, buttons, list items). Verifier / guard — narrates what happened, checks the action succeeded, and gates any risky or irreversible step behind explicit spoken confirmation. Evaluation — each interaction is scored to measure intent accuracy and task success. Steps 4–6 loop (read the screen, act, narrate) until the task is complete. What makes it different Execution, not just narration Existing screen readers describe the screen and leave the work to the user. Daisy does the multi-step task across real apps, turning dozens of manual swipes into a single spoken request. Built for non-visual confirmation Because our users can't glance at the screen to double-check, transparency is the product, not a feature. Daisy explains its interpreted intent before acting and reads back consequential details (what, where, how much) before committing. Divide-and-conquer agents Separate planner, perceiver, executor, and verifier roles — each with a different job — rather than one model wearing many labels. This makes the system more reliable and easier to reason about. Personalized memory Daisy learns a specific user over time: their go-to apps, their usual orders, how much narration detail they prefer. Evaluation pipeline We built Daisy to be measured. Each interaction can be scored by comparing the agent's interpreted intent and completed task against the user's confirmed goal. Metrics tracked: Intent match accuracy Task completion rate Steps / turns to completion Correction rate Improvement across repeated interactions Safety For users who can't visually verify what's happening, safety and trust are central. Daisy is designed to: Explain the interpreted intent before acting Ask for confirmation before any meaningful action Never take irreversible actions (placing orders, payments) without explicit approval Narrate continuously so the user always knows what the agent is doing and seeing Summarize sensitive steps before committing to them ### Challenges we ran into Reading arbitrary app screens reliably is hard because not every app labels its elements well in the accessibility tree, so the perceiver has to handle messy or incomplete information. Mapping a single high-level voice goal to a robust multi-step plan, then executing it against a live, changing screen, is much harder than a one-shot command. Designing an audio-only UX that keeps a non-visual user oriented and in control — informed enough to trust the agent, but not buried in narration — took as much care as the models did. For accessibility, the fastest action isn't the best action; the system has to be clear, confirmable, and trustworthy. Accomplishments We built an end-to-end prototype that connects voice input, intent understanding, planning, live screen reading, real app execution, spoken confirmation, personalized memory, and evaluation focusing on a problem where AI agents create real accessibility value: helping blind and low-vision users complete multi-step phone tasks independently by voice. ### What we learned Accessibility-first agents need more than strong models. They need confirmation loops, careful narration design, personalized memory, evaluation metrics, and deliberate audio UX. And the hardest part isn't clicking through an app but reliably understanding the user's goal and keeping a non-visual user in control the whole way through. ### What's next A companion/caregiver view for reviewing task history and progress over time Support for more apps and more complex, multi-app workflows Multimodal confirmation (haptics, simple yes/no gestures) alongside voice Partnerships with BLV organizations and accessibility researchers On-device processing for stronger privacy Research & references We grounded Daisy's design in prior work on mobile UI agents, screen understanding, and accessibility. Papers we read: Smartphones-Based Assistive Technology: Accessibility Features and Apps for People with Visual Impairment, and its Usage, Challenges, and Usability Testing https://pmc.ncbi.nlm.nih.gov/articles/PMC8636846/ mHealth Technology Experiences of Middle-Aged and Older Individuals With Visual Impairments: Cross-Sectional Interview Study https://pubmed.ncbi.nlm.nih.gov/38145472/ Challenges and Enablers for Smartphone Use by Persons With Vision Loss During the COVID-19 Pandemic: A Report of Two Case Studies https://pubmed.ncbi.nlm.nih.gov/35875007/ Blind people and mobile touch-based text-entry: acknowledging the need for different flavors https://dl.acm.org/doi/abs/10.1145/2049536.2049569 ## README (from the GitHub repository) # Daisy 🌼 ### An accessibility-first phone agent for blind and low-vision users > Screen readers tell you what's on screen. Daisy actually does the task — by voice, one request at a time. ![Deepgram](https://img.shields.io/badge/Deepgram-STT%2FTTS-blue) ![Claude](https://img.shields.io/badge/Anthropic-Claude-orange) ![Android](https://img.shields.io/badge/Android-Accessibility%20Service-green) ![Python](https://img.shields.io/badge/Python-3.11%2B-blue) ![FastAPI](https://img.shields.io/badge/FastAPI-services-009688) --- ## Inspiration Screen readers like TalkBack let blind and low-vision (BLV) users hear what's on screen, but operating an app still means swiping through it one element at a time. A single everyday task — ordering food, booking a ride, replying to a message, changing a setting — can take dozens of sequential swipes and taps. It's slow, fatiguing, and easy to lose track of. Mainstream voice assistants don't fix this. They answer questions and fire off a few built-in commands, but they can't actually drive third-party apps. We wanted an assistant where a BLV user states a goal once, in plain language, and the agent does the navigating — narrating every step and confirming before anything important — so the user stays fully in control without needing to see the screen. --- ## What It Does Daisy is an accessibility-first Android agent that sits between a user's voice and their phone. After a wake word, the user says what they want. Daisy interprets the intent, explains its plan aloud, asks for confirmation, then uses Android automation to carry out the multi-step task inside a real app — describing what's on screen as it goes and pausing before any meaningful action. The core goal isn't just app automation. It's **independence**: letting BLV users complete multi-step mobile tasks by voice, without grinding through linear screen-reader navigation, and without ever losing visibility into what the agent is doing. --- ## How We Built It Daisy combines speech, LLM reasoning, user memory, and Android app control: - **Deepgram** — speech-to-text and text-to-speech, powering the audio-first interface (the primary channel for our users). - **Claude** — interprets the user's goal, plans the multi-step task, and reasons over the current screen to decide the next action. - **Android Accessibility Service** — captures screen state (the same accessibility tree TalkBack uses) and performs taps, swipes, and text entry. - **Evaluation tooling** — tracks whether the agent completed the task the user actually intended. --- ## Architecture Rather than one monolithic chatbot, Daisy splits the work across agents with genuinely distinct jobs — a divide-and-conquer design where each stage does one thing well: ```mermaid flowchart TD Wake["🔔 Wake
wake word + persistent listening cue"] --> Listen Listen["🎙️ Listen & Transcribe
Deepgram STT"] --> Planner Planner["🧭 Planner (Claude)
reconstruct intent → step-by-step roadmap"] --> Perceiver Perceiver["👁️ Perceiver
read screen via Accessibility tree"] --> Verifier Verifier["🛡️ Verifier / Guard
narrate, check success, gate risky steps"] --> Done{Task complete?} Done -- "no" --> Perceiver Done -- "yes" --> Eval["📊 Evaluation
score intent + task success"] ## Detected evidence (automated analysis) Indexed codebase: 80 recognized source files, 230 KB. - Anthropic (technology) — detected in the code - FastAPI (technology) — detected in the code - Java (language) — detected in the code - Kotlin (language) — detected in the code - Python (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (109 of 109) ``` .env.example .gitignore .vscode/settings.json agent/__init__.py agent/anthropic_loop.py agent/bm25_rank.py agent/node.py agent/prompt.md agent/repl.py agent/run.py agent/server.py agent/tools.py backend/intent_service/.gitignore backend/intent_service/api/__init__.py backend/intent_service/api/health.py backend/intent_service/api/infer.py backend/intent_service/api/schemas.py backend/intent_service/intent.py backend/intent_service/main.py backend/intent_service/README.md backend/intent_service/requirements.txt CLAUDE.md daisy/SandBox/.gitignore daisy/SandBox/app/.gitignore daisy/SandBox/app/build.gradle.kts daisy/SandBox/app/proguard-rules.pro daisy/SandBox/app/src/androidTest/java/com/example/showgraphs/ExampleInstrumentedTest.java daisy/SandBox/app/src/main/AndroidManifest.xml daisy/SandBox/app/src/main/java/com/example/showgraphs/CommandInterpreter.kt daisy/SandBox/app/src/main/java/com/example/showgraphs/ConversationEngine.kt daisy/SandBox/app/src/main/java/com/example/showgraphs/DaisyAccessibilityService.kt daisy/SandBox/app/src/main/java/com/example/showgraphs/DaisyOrbView.kt daisy/SandBox/app/src/main/java/com/example/showgraphs/DaisyService.kt daisy/SandBox/app/src/main/java/com/example/showgraphs/DaisyState.kt daisy/SandBox/app/src/main/java/com/example/showgraphs/IntentServiceClient.kt daisy/SandBox/app/src/main/java/com/example/showgraphs/MainActivity.kt daisy/SandBox/app/src/main/java/com/example/showgraphs/OrbStyles.kt daisy/SandBox/app/src/main/java/com/example/showgraphs/voice/stt/AudioStreamer.java daisy/SandBox/app/src/main/java/com/example/showgraphs/voice/stt/DeepgramSttClient.java daisy/SandBox/app/src/main/java/com/example/showgraphs/voice/stt/SttCallback.java daisy/SandBox/app/src/main/java/com/example/showgraphs/voice/tts/DeepgramTts.kt daisy/SandBox/app/src/main/java/com/example/showgraphs/VoiceAssistant.kt daisy/SandBox/app/src/main/res/drawable/ic_launcher_background.xml daisy/SandBox/app/src/main/res/drawable/ic_launcher_foreground.xml daisy/SandBox/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml daisy/SandBox/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml daisy/SandBox/app/src/main/res/values-night/themes.xml daisy/SandBox/app/src/main/res/values/colors.xml daisy/SandBox/app/src/main/res/values/strings.xml daisy/SandBox/app/src/main/res/values/themes.xml daisy/SandBox/app/src/main/res/xml/backup_rules.xml daisy/SandBox/app/src/main/res/xml/daisy_accessibility_config.xml daisy/SandBox/app/src/main/res/xml/data_extraction_rules.xml daisy/SandBox/app/src/main/res/xml/network_security_config.xml daisy/SandBox/app/src/test/java/com/example/showgraphs/ExampleUnitTest.java daisy/SandBox/build.gradle.kts daisy/SandBox/gradle.properties daisy/SandBox/gradle/libs.versions.toml daisy/SandBox/gradle/wrapper/gradle-wrapper.properties daisy/SandBox/gradlew daisy/SandBox/gradlew.bat daisy/SandBox/local.properties.example daisy/SandBox/settings.gradle.kts env/__init__.py env/device.py frontend/.gitignore frontend/app/build.gradle frontend/app/proguard-rules.pro frontend/app/src/main/AndroidManifest.xml frontend/app/src/main/java/com/calhacks/ai/ui/MainActivity.java frontend/app/src/main/java/com/calhacks/ai/voice/stt/AudioStreamer.java frontend/app/src/main/java/com/calhacks/ai/voice/stt/DeepgramSttClient.java frontend/app/src/main/java/com/calhacks/ai/voice/stt/SttCallback.java frontend/app/src/main/res/layout/activity_main.xml frontend/app/src/main/res/values/strings.xml frontend/app/src/main/res/values/themes.xml frontend/build.gradle frontend/gradle.properties frontend/gradle/wrapper/gradle-wrapper.properties frontend/gradlew frontend/gradlew.bat frontend/local.properties.example frontend/README.md frontend/settings.gradle logs/session_20260621_045718.log logs/session_20260621_051959.log logs/session_20260621_052807.log logs/session_20260621_055301.log README.md requirements.txt scripts/build_phone_index.py scripts/start_server.sh services/__init__.py services/pii_protection/__init__.py services/pii_protection/recognizers.py services/pii_protection/redactor.py services/tts.py tests/__init__.py tests/manual_run.py tests/services/__init__.py tests/services/test_pii_redactor.py tests/services/test_tts.py tests/test_agentspan_guard.py tests/test_bm25_rank.py tests/test_compress.py tests/test_device_auto.py tests/test_phone_index.py tests/test_tool_fidelity.py tests/test_xml_filter.py ``` ### Dependencies - backend/intent_service/requirements.txt: anthropic@>=0.69.0, fastapi@>=0.115.0, pydantic@>=2.7.0, python-dotenv@>=1.0.0, uvicorn[standard]@>=0.30.0 - requirements.txt: agentspan, anthropic, deepgram-sdk, fastapi, presidio-analyzer, presidio-anonymizer, pytest, python-dotenv, rank-bm25, spacy, uiautomator2, uvicorn[standard] ### Recent commits (newest first) - Revise README for clarity and updated features - Merge pull request #25 from iandang312/tts - Merge pull request #24 from iandang312/tts - Merge pull request #23 from iandang312/fix-agent-execution - Merge pull request #22 from iandang312/tts - feat: add TTS speak tool via Deepgram async client - feat: add TTS speak tool via Deepgram async client - feat: add TTS speak tool via Deepgram async client - Merge pull request #21 from iandang312/retry-timeout - log info for backend infer service - Updated README to align with current project state - Merge pull request #20 from iandang312/retry-timeout - android <-> llm api connection wired - retry timeout - fix the prompt - fix image encode - fix test manual - fix - switch agent loop to direct Anthropic SDK with per-turn screenshots - Merge pull request #19 from iandang312/fix-agent-execution ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown # calhacks-ai — Android UI Agent (MVP) Base agent that drives a Samsung Android emulator via `uiautomator2`, orchestrated with `agentspan`. Action set: tap, tap_text, long_press, swipe, drag, type_text, press_key, open_app, dump_ui, screenshot, finish. ## Prerequisites 1. Android Studio + a running AVD (Samsung-style system image). See . 2. `adb devices` shows the emulator. 3. `python -m uiautomator2 init` (one-time per device — installs atx-agent). 4. Python 3.11+, `pip install -r requirements.txt`. 5. `cp .env.example .env` and set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`. ## Run CLI (local): ``` python -m agent.run "open the settings app" ``` HTTP service (for the Java side to call): ``` uvicorn agent.server:app --host 0.0.0.0 --port 8000 # POST http://localhost:8000/agent/run {"task": "open the settings app"} # GET http://localhost:8000/health ``` ## Test ``` RUN_DEVICE_TESTS=1 pytest -q ``` Device tests are integration-only; they hit the live emulator (no mocks). Skipped unless the env flag is set. ## Module map - `env/device.py` — `Device` class wrapping `uiautomator2`. Reusable. - `agent/tools.py` — JSON-schema tool definitions + `(device, **args) -> str` handlers. `finish` is a sentinel. - `agent/prompt.md` — system prompt: loop contract, do/don't, two few-shots. - `agent/node.py` — `build_graph(device, max_steps, model_call)` returns a `run(task) -> Trajectory`. `model_call` is the seam for the agentspan Agent node — wire it to the SDK when integrating the LLM. - `agent/run.py` — CLI entrypoint. - `agent/server.py` — FastAPI wrapper. `POST /agent/run` and `GET /health`. This is the boundary the Java side calls. ## Architecture decision: Python service, Java client The Deepgram transcription and simulation/orchestration logic live in Java. The agent loop lives in Python because `uiautomator2` (and the agentspan Python SDK) are the most mature options for driving an Android device. Rather than rewrite either side, the two languages are split across an HTTP boundary: - **Java owns the system flow**: speech → text (Deepgram), task framing, multi-step orchestration, user-facing UI. - **Python owns one narrow capability**: "given a task string, drive the emulator and return a trajectory." Exposed as a single FastAPI service (`agent/server.py`). Why HTTP and not an in-process bridge (JPype/Py4J): - Decouples deployment — each side restarts independently. - Debuggable in isolation (curl the endpoint, run the Java side without the agent up). - Cross-call rate is low (one call per user task, not per UI action), so HTTP latency is irrelevant. If finer-grained control is needed later (Java wants to send individual actions instead of full tasks), expand the HTTP surface — do not collapse the boundary. ## Conventions - All device actions go through `Device` — never call `uiautomator2` directly from agent code. - New tools: add schema + handler in `agent/tools.py`, wire an [truncated — 416 more characters] ``` ### agent/prompt.md ```markdown # Android UI Agent You control a Samsung Android phone via tool calls. You receive a fresh screenshot of the current screen attached to every user message — use it to perceive UI state. No XML dump is available. ## Step 1 — Plan (required, before any tool call) Before calling any tool, write a numbered plan in plain text: ``` Plan: 1. Open the Chrome browser 2. Tap the address bar 3. Type "flowers" and press enter 4. Tap the first search result ``` This is the only place you may write text. After the plan, switch to tool calls only. ## Step 2 — Execute Follow the loop contract: 1. Look at the most recent screenshot to read current UI state. 2. Choose ONE tool per turn. Wait for its result and the next screenshot before the next step. 3. When the goal is fully achieved and visible in the screenshot, call `finish(success=true, note=...)`. 4. When the goal is unreachable, call `finish(success=false, note=...)` using the failure format below. ## Tool selection rules - Prefer `tap_text` over `tap(x,y)` whenever the target element has visible text. - For `tap`, `long_press`, `swipe`, `drag`: read pixel coordinates from the screenshot. The viewport is typically 1080×2340 — top-left is (0,0), bottom-right is (1080,2340). - Use `open_app(package)` for known launchers (`com.android.settings`, `com.android.chrome`, `com.google.android.dialer`, etc.) instead of navigating from the home screen. `open_app` blocks until the app is in the foreground. - After typing into a field, press `enter` via `press_key` if the form requires submission. - If `tap_text` returns "miss", the text was not visible. Look at the screenshot for the actual visible text and retry with the correct string, or fall back to `tap(x,y)`. - If a dialog or popup appears unexpectedly, dismiss with `press_key("back")` before retrying. ## Voice narration The user cannot see the screen. Use `speak(text)` to narrate what you are doing: - Call `speak` before every `open_app` call, e.g. `speak("Opening Settings")` - Call `speak` before navigating back or home, e.g. `speak("Going back to home screen")` - Call `speak` with a summary just before `finish`, e.g. `speak("Done. I found 3 unread messages from John.")` - Do NOT call `speak` before every tap or swipe — only at meaningful transitions. ## Do NOT - Do not call any tool before writing your plan. - Do not emit reasoning or narration after the plan — only tool calls. - Do not call the same tool with the same arguments more than twice in a row. - Do not invent coordinates without reading them from the screenshot. ## Failure format When calling `finish(success=false, ...)`, the note must follow this format: ``` Completed: [list what succeeded, or "nothing"]. Failed at: [step N — description]. Reason: [why it failed]. Suggestion: [what the user can provide or try]. ``` Example: ``` Completed: opened Chrome, tapped address bar. Failed at: step 3 — typing search query. Reason: keyboard did not appear after tapping the address bar. Sugge [truncated — 1010 more characters] ``` ### requirements.txt ``` deepgram-sdk agentspan anthropic uiautomator2 python-dotenv fastapi uvicorn[standard] pytest presidio-analyzer presidio-anonymizer spacy rank-bm25 ``` ### backend/intent_service/requirements.txt ``` # Pin floors that include output_config structured outputs + the claude-opus-4-8 model. # Run `pip install -U -r requirements.txt` to pull the latest compatible releases. anthropic>=0.69.0 fastapi>=0.115.0 uvicorn[standard]>=0.30.0 pydantic>=2.7.0 python-dotenv>=1.0.0 ``` ### agent/server.py ```python """FastAPI wrapper around the agent loop. Boundary between the Java side (Deepgram + simulation/orchestration) and the Python side (uiautomator2 + agent loop). Java POSTs a task string; this service drives the device and returns the trajectory. Run: `uvicorn agent.server:app --host 0.0.0.0 --port 8000` """ from __future__ import annotations from contextlib import asynccontextmanager from typing import Any from dotenv import load_dotenv from fastapi import FastAPI, HTTPException from pydantic import BaseModel from env.device import Device from agent.node import build_graph, llm_env_key class RunRequest(BaseModel): task: str max_steps: int = 25 class StepDTO(BaseModel): tool: str args: dict[str, Any] observation: str class RunResponse(BaseModel): success: bool note: str steps: list[StepDTO] _state: dict[str, Any] = {} @asynccontextmanager async def lifespan(app: FastAPI): load_dotenv() if not llm_env_key(): raise RuntimeError("no ANTHROPIC_API_KEY or OPENAI_API_KEY in .env") _state["device"] = Device() yield _state.clear() app = FastAPI(title="Android UI Agent", lifespan=lifespan) @app.get("/health") def health() -> dict[str, Any]: d: Device | None = _state.get("device") return {"ok": d is not None} @app.post("/agent/run", response_model=RunResponse) def run_agent(req: RunRequest) -> RunResponse: device: Device | None = _state.get("device") if device is None: raise HTTPException(status_code=503, detail="device not initialized") # NOTE: pass a real `model_call` here once the agentspan node is wired. run = build_graph(device, max_steps=req.max_steps) traj = run(req.task) return RunResponse( success=traj.success, note=traj.note, steps=[StepDTO(tool=s.tool, args=s.args, observation=s.observation) for s in traj.steps], ) ``` ### backend/intent_service/main.py ```python """ FastAPI service that turns a noisy Deepgram speech-to-text transcript into a first-person plan of Android UI steps. Run locally: pip install -r requirements.txt cp .env.example .env # then put your ANTHROPIC_API_KEY in it uvicorn main:app --reload --host 0.0.0.0 --port 8000 The Android app POSTs the final Deepgram transcript to /infer and gets back a `plan` to speak via TTS and feed to the mobilerun UI framework. Routes live in the `api` package; this module only wires the app together. """ import logging from dotenv import load_dotenv # Must run before importing `api` (which imports `intent`, building the Anthropic # client at module load and reading ANTHROPIC_API_KEY from the environment). load_dotenv() from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from api import api_router logging.basicConfig(level=logging.INFO) app = FastAPI(title="Intent Inference Service", version="0.1.0") # Permissive CORS so the endpoint is easy to hit from a browser or web test page. # (Native Android clients don't enforce CORS, so this only matters for web callers.) app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"], ) app.include_router(api_router) ``` ### agent/run.py ```python """CLI entrypoint: `python -m agent.run "open the settings app"`.""" from __future__ import annotations import os import sys from dotenv import load_dotenv from env.device import Device from agent.anthropic_loop import load_prompt, run_anthropic def main(argv: list[str]) -> int: load_dotenv() if not os.environ.get("ANTHROPIC_API_KEY"): print("ERROR: no ANTHROPIC_API_KEY in .env", file=sys.stderr) return 2 if len(argv) < 2: print("usage: python -m agent.run \"\"", file=sys.stderr) return 2 task = argv[1] device = Device() system = load_prompt() traj = run_anthropic(device, system, task, max_steps=25) for i, s in enumerate(traj.steps, 1): print(f"[{i}] {s.tool}({s.args}) -> {s.observation}") print(f"\nDONE success={traj.success} note={traj.note}") return 0 if traj.success else 1 if __name__ == "__main__": sys.exit(main(sys.argv)) ``` ### services/tts.py ```python import asyncio import os import subprocess import sys import tempfile from deepgram import AsyncDeepgramClient _MODEL = "aura-2-thalia-en" _MAX_CHARS = 500 async def _speak_async(text: str) -> None: client = AsyncDeepgramClient() audio_bytes = b"" async for chunk in client.speak.v1.audio.generate(text=text, model=_MODEL): audio_bytes += chunk with tempfile.NamedTemporaryFile(suffix=".mp3", delete=False) as f: f.write(audio_bytes) tmp_path = f.name subprocess.run(["afplay", tmp_path], check=False) def speak(text: str) -> None: if not text or not text.strip(): return text = text[:_MAX_CHARS] api_key = os.environ.get("DEEPGRAM_API_KEY") if not api_key: print("WARNING: DEEPGRAM_API_KEY not set, skipping TTS", file=sys.stderr) return try: <<<<<<< HEAD asyncio.run(_speak_async(text, api_key)) ======= asyncio.run(_speak_async(text)) >>>>>>> 5d6e069 (feat: add TTS speak tool via Deepgram async client) except Exception as e: print(f"WARNING: TTS failed: {e}", file=sys.stderr) ``` ### scripts/start_server.sh ```shell #!/usr/bin/env bash set -euo pipefail ENV_FILE="${1:-.env}" if [[ ! -f "$ENV_FILE" ]]; then echo "ERROR: $ENV_FILE not found" >&2 exit 1 fi echo "Stopping any existing agentspan server..." agentspan server stop 2>/dev/null || true echo "Starting agentspan server..." agentspan server start echo "Waiting for server to be ready..." for i in $(seq 1 20); do if curl -sf http://localhost:6767/health > /dev/null 2>&1; then break fi sleep 1 done if ! curl -sf http://localhost:6767/health > /dev/null 2>&1; then echo "ERROR: server did not become ready in time" >&2 exit 1 fi echo "Setting agentspan credentials from $ENV_FILE..." while IFS= read -r line || [[ -n "$line" ]]; do # Skip blank lines and comments [[ -z "$line" || "$line" == \#* ]] && continue key="${line%%=*}" value="${line#*=}" # Skip entries with no value or keys not meant for the credential store [[ -z "$value" || "$key" == "MODEL" ]] && continue echo " agentspan credentials set $key ..." agentspan credentials set "$key" "$value" done < "$ENV_FILE" echo "Done. Server running at http://localhost:6767" ``` ### agent/repl.py ```python """Interactive REPL — keep the device connected, run task after task. Usage: source .venv/bin/activate python -m agent.repl Connection setup mirrors tests/manual_run.py: load env first, then Device(). """ from __future__ import annotations import os import sys import traceback from dotenv import load_dotenv load_dotenv() from env.device import Device from agent.anthropic_loop import load_prompt, run_anthropic def main() -> int: if not os.environ.get("ANTHROPIC_API_KEY"): print("ERROR: no ANTHROPIC_API_KEY in .env", file=sys.stderr) return 2 print("connecting to device...") device = Device() system = load_prompt() print(f"connected. viewport={device.viewport}") print("type a task and press enter. type 'quit' to exit.\n") while True: try: task = input("> ").strip() except (EOFError, KeyboardInterrupt): print() return 0 if not task: continue if task.lower() in {"quit", "exit", "q"}: return 0 if task.lower() == "home": device.press_key("home") print("(pressed home)") continue try: traj = run_anthropic(device, system, task, max_steps=25) for i, s in enumerate(traj.steps, 1): print(f"[{i}] {s.tool}({s.args}) -> {s.observation}") print(f"DONE success={traj.success} note={traj.note}\n") except Exception: traceback.print_exc() print() if __name__ == "__main__": sys.exit(main()) ``` [64 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]