# Project export: yooni This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Say it, and it's done. yooni is personal intelligence that navigates your mobile phone and works for you. - Devpost: https://devpost.com/software/yooni - GitHub: https://github.com/nolawiyonas1/yooni - Demo: http://yooni.lovable.app/ - Video: https://www.youtube.com/embed/G00_yOa5kzk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Troy Gunawardene (13 commits), Nolawi Teklehaimanot (8 commits), Claude Sonnet 4.5 (4 commits), Ifeoluwa Oyetimehin (2 commits) ## Devpost submission (written by the team) ### Overview For a product overview video, visit link ### Inspiration We asked Siri to call us an uber. It replied with "Hey Uber". This got us thinking, even well funded AI voice assistants have quite a long way to go. We want something that can actually get things done. We wanted an agent that can control our devices and do our boring tasks like booking our ubers, ordering us food and spam texting our ex so we didn't have to. We saw the potential of "Blue" (YC S25) but not only are they not yet on the market after almost a year, we also realized a closed ecosystem limits innovation and privacy. We wanted to build an open-source, privacy-first alternative that gives users full control over their own devices using affordable, off-the-shelf hardware. ### What it does Yooni is an open-source voice agent that controls your mobile device to complete real-world tasks, not just productivity tools. Yooni can buy you concert tickets, it can buy your favorite meal on instacart, it can get you a lyft, it can do anything you can do on your phone. Natural Voice Interaction Speak naturally to Yooni (e.g., "Order my usual from DoorDash" or "Text mom I'll be there in 10") Real-time speech-to-text using Whisper and ultra-low latency response Conversational memory to handle follow-up questions and refinements On-Device Control Android: Uses accessibility services and programmatic control to view your screen, tap buttons, swipe, and type text directly in your apps Agent Logic: Understands app layouts and navigates through complex flows (like finding a specific email or changing a setting) without needing special API integrations iOS: Planned support via mobile-use Privacy & Safety Transparent Execution: Yooni explains what it's about to do before taking action Human-in-the-Loop: Asks for explicit confirmation before sensitive actions (like sending money or messages) Open Source: No black box—the agent logic is fully auditable and extendable by the community Local Inference: Compatible with self-hosted voice and multimodal LLMs Intelligent Planning Breaks down vague requests into precise, actionable steps Verifies screen state before and after actions to ensure success Handles errors gracefully by retrying or asking for clarification ### How we built it We built Yooni as a distributed system to handle the heavy lifting of agentic reasoning while keeping the mobile app lightweight. Android App (Kotlin & Jetpack Compose): The frontend is a native Android app that handles voice input (OpenAI Whisper), speech synthesis (TTS), and the user interface. It captures the user's intent and displays the agent's thought process. Brain (Python & Gemini): The core intelligence runs on a backend (prototyped on a Raspberry Pi/local server) using Python. In production, this can run on significantly smaller edge devices, making it convenient for users to handle. We utilize Google's Gemini 3 Pro Preview for the high-level reasoning and planning, transforming vague voice commands into precise, step-by-step navigation instructions. Mobile Control (Mobile-Use): We improved and integrated mobile-use, an open-source framework that allows our agent to interface with the Android operating system, enabling it to "see" the screen hierarchy and simulate touch events. Networking: The Android app and the Python brain communicate via HTTP/WebSockets to stream audio and commands in real-time. Hardware Portability: We cross-compile from an NVIDIA/ASUS Ascent GX10 to support older, widely available hardware—so the community can run Yooni without expensive devices. ### Challenges we ran into Latency vs. Accuracy: Balancing the speed of voice response with the time it takes for the agent to analyze a screen and decide on a tap was tough. We had to optimize our prompt engineering to get faster, reliable actions. Android Permissions: Gaining the necessary accessibility permissions to control other apps programmatically is (rightfully) difficult on Android. We spent a lot of time navigating the security model to allow Yooni to act on the user's behalf safely. Audio Handling: Implementing a robust "wake word" style experience and handling raw audio streams between Kotlin and Python required debugging low-level byte streams and format conversions (PCM to WAV). Old Hardware: Working with an 11-year-old Raspberry Pi to run state-of-the-art mobile agents was a challenge - we used an NVIDIA-provided ASUS Ascent GX110 to cross-compile binaries for it. ### Accomplishments we're proud of End-to-End Voice to Action: We successfully demoed a flow where a simple voice command triggers a real, physical interaction in a third-party app on the phone. Open Source Foundation: We built this on top of open standards, meaning anyone can fork Yooni and add support for their favorite apps or custom workflows. Sleek UI: We built a modern, responsive UI in Jetpack Compose with custom animations (breathing agent circle) that makes the AI feel alive and responsive. Privacy-First Architecture: By design, Yooni is transparent. It doesn't act in a "black box"; the user sees the plan and approves critical steps. ### What we learned Agentic Workflows are Hard: "Planning" is easy for LLMs, but "executing" reliably in a dynamic environment like a smartphone OS is incredibly complex. Screen states change, popups appear, and loading times vary. Voice UI requires Trust: Users need constant feedback. We learned that visual cues (like the breathing animation and text logs) are essential to let the user know the agent is "thinking" or "working," otherwise they think it froze. The Power of Accessibility Services: Android's accessibility layer is incredibly powerful for automation, far beyond just screen reading. ### What's next for Yooni Voice Authentication: Built-in speaker verification ensures only your voice can command Yooni, preventing unauthorized access even if someone else has your phone. On-Device Processing: Moving the LLM inference entirely to the device (using models like Gemini Nano or Llama 3 quantized) for offline capability and ultimate privacy. Visual Understanding: Improving the screen parsing with vision-language models (VLMs) to understand custom UI elements that standard accessibility services miss (like game menus). Proactive Help: Yooni learning your habits and suggesting tasks (e.g., "It's 6 PM, should I order dinner?"). ## README (from the GitHub repository) # yooni Open-source AI phone agent. Voice in, phone actions out. yooni is an open-source, privacy-first voice assistant that takes natural language commands and executes them on your Android device. ## Architecture ``` Android App (Kotlin) Raspberry Pi - Wake word detection ("Hey yooni") - mobile-use (controls phone via ADB) - Records speech - Whisper STT (openai-kotlin) - LLM formats action (openai-kotlin) - TTS speaks back for confirmation - User confirms/refines (voice loop) - Sends confirmed command ------> websocket/HTTP ------> executes on phone ``` ### Two layers **Voice Layer (Android App)** - Native Kotlin app that runs on the phone. Listens for the "Hey yooni" wake word using [Porcupine](https://picovoice.ai/platform/porcupine/) (on-device, no network). Once triggered, records speech, transcribes via OpenAI Whisper, uses an LLM to format the command into a clean action preview, and speaks it back via OpenAI TTS for confirmation. User can refine until satisfied, then confirms to execute. **Phone Control Layer (Raspberry Pi)** - A Raspberry Pi connected to the Android phone via USB. Runs [mobile-use](https://github.com/minitap-ai/mobile-use) which receives the confirmed command and handles all device interaction over ADB. Reads the screen, decides what to tap/type/swipe, and executes autonomously. ## Design [Figma](https://www.figma.com/make/r1MwLN4N2Aw0EqVVmqyRjl/Untitled?p=f) ## Example flow > **User:** "Hey yooni, text Mom that I'll be there in 10 mins" > > **yooni:** "I'll send this to Mom: 'Hey! I'll be there in about 10 minutes.' Sound good?" > > **User:** "Add 'do you need anything?'" > > **yooni:** "Got it: 'Hey! I'll be there in about 10 minutes. Do you need anything?' Ready to send?" > > **User:** "Yes" > > **yooni:** *sends to Pi, mobile-use executes* "Sent!" ## File structure ``` yooni/ ├── android/ # Native Android app (Kotlin) │ ├── app/ │ │ ├── build.gradle.kts # App dependencies (openai-kotlin, ktor, porcupine) │ │ └── src/main/ │ │ ├── AndroidManifest.xml │ │ └── java/com/example/yooni/ │ │ ├── MainActivity.kt # App entry point │ │ ├── WakeWordService.kt # Porcupine "Hey yooni" listener (planned) │ │ ├── VoiceManager.kt # Recording, Whisper STT, TTS (planned) │ │ ├── ActionFormatter.kt # LLM formats command (planned) │ │ ├── ConfirmationLoop.kt # Confirm/refine loop (planned) │ │ ├── PiClient.kt # Sends commands to Pi (planned) │ │ └── ui/theme/ │ │ ├── Color.kt │ │ ├── Theme.kt │ │ └── Type.kt │ ├── build.gradle.kts # Root Gradle config │ └── settings.gradle.kts ├── pi/ # Raspberry Pi server (planned) │ ├── server.py # Receives commands from Android app │ └── executor.py # Passes commands to mobile-use └── README.md ``` ## Detected evidence (automated analysis) Indexed codebase: 45 recognized source files, 198 KB. - FastAPI (technology) — detected in the code - Kotlin (language) — detected in the code - Python (language) — detected in the code - Docker (technology) — claimed on Devpost, not found in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (61 of 61) ``` .gitignore .idea/.gitignore .idea/caches/deviceStreaming.xml .idea/markdown.xml .idea/misc.xml .idea/modules.xml .idea/vcs.xml .idea/yooni.iml android/.gitignore android/.idea/.gitignore android/.idea/.name android/.idea/AndroidProjectSystem.xml android/.idea/codeStyles/codeStyleConfig.xml android/.idea/codeStyles/Project.xml android/.idea/compiler.xml android/.idea/deploymentTargetSelector.xml android/.idea/deviceManager.xml android/.idea/gradle.xml android/.idea/inspectionProfiles/Project_Default.xml android/.idea/kotlinc.xml android/.idea/migrations.xml android/.idea/misc.xml android/.idea/other.xml android/.idea/runConfigurations.xml android/.idea/studiobot.xml android/.idea/vcs.xml android/app/.gitignore android/app/build.gradle.kts android/app/proguard-rules.pro android/app/src/androidTest/java/com/example/yooni/ExampleInstrumentedTest.kt android/app/src/main/AndroidManifest.xml android/app/src/main/assets/hey_yooni.ppn android/app/src/main/java/com/example/yooni/ActionFormatter.kt android/app/src/main/java/com/example/yooni/MainActivity.kt android/app/src/main/java/com/example/yooni/PiClient.kt android/app/src/main/java/com/example/yooni/ui/theme/Color.kt android/app/src/main/java/com/example/yooni/ui/theme/Theme.kt android/app/src/main/java/com/example/yooni/ui/theme/Type.kt android/app/src/main/java/com/example/yooni/VoiceManager.kt android/app/src/main/res/drawable/ic_launcher_background.xml android/app/src/main/res/drawable/ic_launcher_foreground.xml android/app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml android/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml android/app/src/main/res/values/colors.xml android/app/src/main/res/values/strings.xml android/app/src/main/res/values/themes.xml android/app/src/main/res/xml/backup_rules.xml android/app/src/main/res/xml/data_extraction_rules.xml android/app/src/test/java/com/example/yooni/ExampleUnitTest.kt android/build.gradle.kts android/gradle.properties android/gradle/libs.versions.toml android/gradle/wrapper/gradle-wrapper.properties android/gradlew android/gradlew.bat android/settings.gradle.kts pi/.gitignore pi/executor.py pi/requirements.txt pi/server.py README.md ``` ### Dependencies - pi/requirements.txt: fastapi@==0.115.0, pydantic@==2.9.2, uvicorn[standard]@==0.32.0 ### Recent commits (newest first) - feat: replace absolute timeout with idle timeout in executor - Merge branch 'main' of https://github.com/nolawiyonas1/yooni - add logging - Merge branch 'main' of https://github.com/nolawiyonas1/yooni - style: Hey yooni! - fix unicode issue - update venv path - weofwehfo - add venv activation back - fix folder path - changes - refactor: simplify executor to use python3 directly - switch to fastapi - feat: stream mobile-use subprocess output to server terminal - feat: use mobile-use venv Python when executing commands - feat: add hands-free wake word and auto-stop recording - Merge pull request #2 from nolawiyonas1/feature/frontend-redesign - fix: MediaPlayer crash and add robust TTS file deletion error handling and file cleanup - Update App UI - style: UI updates ## Key source files (fetched from GitHub, selected and truncated for size) ### pi/requirements.txt ``` fastapi==0.115.0 uvicorn[standard]==0.32.0 pydantic==2.9.2 ``` ### pi/server.py ```python """ HTTP server that receives commands from the Android app and passes them to mobile-use. Run: uvicorn server:app --host 0.0.0.0 --port 8080 Then POST to http://:8080/execute with JSON {"command": "..."} """ import logging import os from contextlib import asynccontextmanager from logging.handlers import RotatingFileHandler from fastapi import FastAPI, HTTPException, Request, status from fastapi.responses import JSONResponse from pydantic import BaseModel, Field from executor import execute # Configure logging to both console and file log_dir = os.path.join(os.path.dirname(__file__), "logs") os.makedirs(log_dir, exist_ok=True) log_file = os.path.join(log_dir, "yooni-pi.log") # Create formatters and handlers formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") # Console handler console_handler = logging.StreamHandler() console_handler.setFormatter(formatter) # File handler with rotation (10MB max, keep 5 backups) file_handler = RotatingFileHandler(log_file, maxBytes=10*1024*1024, backupCount=5) file_handler.setFormatter(formatter) # Configure root logger logging.basicConfig( level=logging.INFO, handlers=[console_handler, file_handler] ) logger = logging.getLogger(__name__) @asynccontextmanager async def lifespan(app: FastAPI): """Lifespan context manager for startup/shutdown events.""" logger.info("Yooni Pi server starting up on 0.0.0.0:8080") yield logger.info("Yooni Pi server shutting down") app = FastAPI( title="Yooni Pi Server", description="Receives commands from Android app and passes them to mobile-use", version="1.0.0", lifespan=lifespan, ) class CommandRequest(BaseModel): """Request model for command execution.""" command: str = Field(..., min_length=1, description="Natural language command to execute") class CommandResponse(BaseModel): """Response model for command execution.""" success: bool message: str @app.exception_handler(Exception) async def global_exception_handler(request: Request, exc: Exception): """Global exception handler that logs all unhandled exceptions with traces.""" logger.exception("Unhandled exception while processing request to %s", request.url.path) return JSONResponse( status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, content={"error": "Internal server error"} ) @app.post("/execute", response_model=CommandResponse) async def execute_command(request: CommandRequest): """ Execute a natural language command via mobile-use. Args: request: CommandRequest containing the command to execute Returns: CommandResponse with success status and message Raises: HTTPException: 500 if command execution fails """ command = request.command.strip() logger.info("Executing: %s", command[:80] + ("..." if len(command) > 80 else "")) try: success, message = execute(command) logger.info("Result: success=%s, message=%s", success, message[:100] if len(message) > 100 else message) if not success: raise HTTPException( status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=message ) return CommandResponse(success=True, message=message) except HTTPException: raise except Exception as e: logger.exception("Error executing command: %s", command[:80]) raise HTTPException( status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=f"Command execution failed: {str(e)}" ) @app.get("/health") async def health_check(): """Health check endpoint.""" return {"status": "healthy"} if __name__ == "__main__": import uvicorn uvicorn.run(app, host="0.0.0.0", port=8080) ``` ### .idea/vcs.xml ```xml ``` ### .idea/misc.xml ```xml ``` ### android/build.gradle.kts ```kotlin // Top-level build file where you can add configuration options common to all sub-projects/modules. plugins { alias(libs.plugins.android.application) apply false alias(libs.plugins.jetbrains.kotlin.android) apply false } ``` ### .idea/modules.xml ```xml ``` ### .idea/markdown.xml ```xml ``` ### android/settings.gradle.kts ```kotlin pluginManagement { repositories { google { content { includeGroupByRegex("com\\.android.*") includeGroupByRegex("com\\.google.*") includeGroupByRegex("androidx.*") } } mavenCentral() gradlePluginPortal() } } dependencyResolutionManagement { repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS) repositories { google() mavenCentral() } } rootProject.name = "Yooni" include(":app") ``` ### pi/executor.py ```python """ Passes confirmed commands from the Android app to mobile-use. Assumes mobile-use is installed and configured on the Pi. """ import logging import os import subprocess import time from threading import Thread logger = logging.getLogger(__name__) def execute(command: str) -> tuple[bool, str]: """ Run mobile-use with the given natural language command. Args: command: The confirmed action to execute (e.g. "Text Mom: I'll be there in 10 mins") Returns: (success, message) - success is True if mobile-use exited 0, else False """ if not command or not command.strip(): return False, "Empty command" try: cwd = os.path.expanduser("~/Documents/mobile-use") venv_path = os.path.join(cwd, ".venv") # Build command that activates venv and runs mobile-use # Using bash -c to activate venv and run command in same shell bash_cmd = f'source "{venv_path}/bin/activate" && python -m minitap.mobile_use.main "{command.strip()}"' logger.info(f"Launching mobile-use:") logger.info(f" Venv: {venv_path}") logger.info(f" Command: {bash_cmd}") logger.info(f" Working directory: {cwd}") # Set up environment with UTF-8 encoding to handle emojis env = os.environ.copy() env["PYTHONIOENCODING"] = "utf-8" # Start the process process = subprocess.Popen( ["bash", "-c", bash_cmd], cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, encoding="utf-8", errors="replace", bufsize=1 # Line buffered ) # Track output and last activity time output_lines = [] last_activity = time.time() idle_timeout = 300 # 5 minutes of no output logger.info("mobile-use output:") # Read output line by line try: while True: line = process.stdout.readline() if line: # We got output, update activity time last_activity = time.time() output_lines.append(line) # Log the line if line.strip(): logger.info(f" {line.rstrip()}") # Check if process has finished if process.poll() is not None: # Process finished, read any remaining output remaining = process.stdout.read() if remaining: output_lines.append(remaining) for line in remaining.splitlines(): if line.strip(): logger.info(f" {line}") break # Check for idle timeout (only if no line was read) if not line: time.sleep(0.1) # Small sleep to avoid busy waiting if time.time() - last_activity > idle_timeout: logger.warning("Process idle for 5 minutes, terminating...") process.terminate() try: process.wait(timeout=5) except subprocess.TimeoutExpired: process.kill() process.wait() return False, "Task timed out due to inactivity (no output for 5 minutes)" finally: process.stdout.close() returncode = process.returncode if returncode == 0: return True, "Command completed successfully" return False, f"Command failed with exit code {returncode}" except FileNotFoundError: return False, "mobile-use not found (pip install mobile-use)" except Exception as e: logger.exception("Error executing command") return False, str(e) ``` ### android/.idea/compiler.xml ```xml ``` [33 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]