# Project export: Pilot This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Automate your phone. - Devpost: https://devpost.com/software/pilot-glb9kd - GitHub: https://github.com/arihanv/pilot.git - Video: https://www.youtube.com/embed/09EtyRMHils?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Ethan Goodhart (20 commits), arihanv (10 commits), Cursor (4 commits), James Zhang (1 commits) ## Devpost submission (written by the team) ### Inspiration : On the first night of Treehacks, we struggled to come up with an idea. At one point, a teammate was about to order The Melt on DoorDash when someone else suggested ordering directly from The Melt website instead. That sparked a conversation. We realized that while many businesses encourage direct ordering, their websites often provide a clunky user experience, even if they outsource delivery to platforms like DoorDash or Uber Eats. As a group, we started thinking about how useful it would be to have a tool that quickly checks across platforms and tells you the cheapest option. From there, we expanded the idea. What if this worked not just for food delivery, but also for ride sharing, hotel bookings, and other services where pricing varies across platforms? That led us to a broader vision: a computer use agent for iPhone. Instead of building separate integrations for every service, an agent could navigate apps and websites directly on your behalf. Price comparison would be just one use case. The same agent could create and send a Partiful invite to your contacts, transfer money through Venmo, or handle other everyday digital tasks automatically. ### What it does : Pilot is the hardware-augmented computer-use agent that operates your smartphone the same way a human would: by seeing the screen, reasoning about context, and executing physical input on the interface. It functions as a closed-loop autonomous system: voice intent becomes real on-screen action, results are verified through vision, and the agent adapts until the task is complete. Hardware Bluetooth HID controller (ESP32) – Emulates a keyboard and mouse to perform real taps, keystrokes, swipes, scrolling, and system shortcuts anywhere on the device. Dual connectivity architecture – Maintains stable BLE input while communicating with a cloud relay server over WiFi for low-latency command delivery. Precision input execution – Calibrated keystroke and gesture timing ensures reliable interaction across dynamic UI states, including lock screen and system-level input. Custom 3D-printed phone case housing – Securely integrates the ESP32 into a practical, daily-use form factor with seamless attachment and detachment. AI-powered software Always-on speech recognition (Apple Speech + VPIO) – Converts natural voice into structured intent with hardware echo cancellation for true full-duplex interaction. System-level screen perception (ReplayKit Broadcast Extension) – Captures the entire live device display in real time, across all apps and system screens, without jailbreak. Multimodal reasoning + tool orchestration (Gemini 3 Flash) – Interprets voice intent and visual screen context together, generates multi-step actions, invokes control tools, and adapts dynamically based on real-time outcomes. Visual grounding (Moondream API) – Identifies UI elements from screenshots and computes pixel-accurate coordinates for precise, reliable interaction. Cloud relay infrastructure (FastAPI + Cloudflare Tunnel) – Routes command sequences to the ESP32 through a persistent endpoint. Pilot replaces brittle app-specific integrations with universal screen understanding and real physical control. It works across arbitrary apps, shifting interfaces, and multi-step workflows—anywhere a human can tap, type, or scroll. ### How we built it : Pilot is a computer-use agent that runs on your iPhone. It sees your screen, reasons about it with a vision-language model, and acts via an ESP32 that acts as a BLE HID keyboard and mouse. There’s no per-app API integration: the agent uses the same interface you do, screens and input. We built an iOS app in Swift. It uses ReplayKit to capture the screen and shares the latest frame when the agent needs a screenshot. Voice input uses on-device speech recognition; we send the transcript to OpenRouter (Gemini 3 Flash), which can take a screenshot or send HID commands. We use Cartesia for voice replies. An ESP32 pairs with the iPhone as a Bluetooth keyboard and mouse, joins the same WiFi, and connects to our relay over WebSocket. The app sends commands (e.g. type, tap, swipe) via HTTP; the relay forwards them to the ESP32, which turns them into keypresses and taps. The agent controls the phone through this bridge. The relay runs in the cloud so the app and ESP32 can both reach it. A /plan endpoint takes a task, uses OpenRouter with app-specific prompts (Uber, DoorDash, etc.), and returns a step-by-step plan. We keep short text files that describe each app’s typical UI flow. BLE HID on iOS needed tuning for pairing, reconnection, and keypress timing. We used one relay so the app and ESP32 don’t need hardcoded IPs, and we had to coordinate the broadcast extension with the main app and handle failed tool chains carefully. ### Challenges we ran into Building Pilot required solving tightly coupled hardware, iOS, and infrastructure constraints. BLE cannot transmit rich data and HID payloads are size-constrained, so we separated reasoning from execution. All inference runs server-side, forwarding only compact command strings to the ESP32 over a persistent WebSocket. Full-duplex audio introduced feedback loops where the speech recognizer transcribed Pilot's own voice. We resolved this by routing recognition and TTS through a single shared AVAudioEngine, keeping the session alive across cycles so Pilot can be interrupted mid-sentence without tearing down the audio pipeline. Screen capture on non-jailbroken iOS required a ReplayKit Broadcast Extension running in a separate process with no direct memory access to the main app. We bridged this with a shared App Group filesystem where frames are compressed, written atomically, and consumed on demand by the vision pipeline. Action reliability was the hardest challenge. BLE HID caps cursor movement at 127 pixels per packet, iOS drops keystrokes under aggressive timing, and UI states vary unpredictably. We built a recursive tool-calling loop that executes, captures a screenshot, verifies the result, and adapts if the screen doesn't match expectations. ### Accomplishments we're proud of Built a complete hardware + AI phone agent from scratch in one weekend. Full-duplex voice interaction — Pilot listens, speaks, and controls the phone simultaneously. End-to-end autonomous loop: voice → AI reasoning → screen capture → HID execution → visual verification. Integrated Gemini, Cartesia TTS, ReplayKit, and ESP32 BLE into a unified real-time pipeline on a physical iPhone. Pilot is a fully operational, voice-controlled phone agent that works end-to-end on a real device. ### What we learned Embedded systems + BLE — Managing dual-radio coexistence on ESP32 and WebSocket keepalive over unstable connections. iOS platform constraints — Screen capture requires a separate system process with filesystem-based IPC, and iOS blocks all forms of absolute mouse positioning from external devices. Full-duplex audio — Routing TTS playback alongside live speech recognition on a shared audio engine without feedback loops. Autonomous agent design — Building recursive tool-calling loops where AI reasons, acts, verifies, and adapts. ### What's next Our vision for Pilot’s technical evolution focuses on two key areas: expanding autonomous capabilities and broadening platform reach. We are developing real-time intent prediction using patterns in a user’s daily routines, calendar events, and usage history to proactively suggest and execute actions before they are even requested. Imagine Pilot recognizing you have a flight in two hours and automatically opening your boarding pass, checking for gate changes, and notifying your ride with your updated arrival time, all without a prompt. On the platform side, we are architecting Pilot to be truly cross-platform, extending the same core architecture to Android, macOS, and Windows. The relay server and AI reasoning layer are already device agnostic. Any platform capable of screen capture and Bluetooth HID input can integrate into the same execution pipeline. We are also advancing visual grounding models that identify and localize UI elements directly from screenshots, eliminating reliance on prewritten app descriptions and enabling autonomous navigation across unfamiliar interfaces. We are particularly excited about enabling multi-app workflows end to end, chaining actions across Messages, Calendar, Maps, and other applications within a single natural language request. This brings Pilot closer to a universal, device-level agent that operates seamlessly across every screen a person uses. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 17 recognized source files, 149 KB. - Python (language) — detected in the code - Swift (language) — detected in the code - C++ (language) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (73 of 73) ``` .gitignore cad/Amazing Migelo-Hango.stl esp32/esp32-pilot.ino esp32/mouse_control.py ios/.DS_Store ios/app_prompts/airbnb.txt ios/app_prompts/airline.txt ios/app_prompts/amazon.txt ios/app_prompts/canvas.txt ios/app_prompts/coinbase.txt ios/app_prompts/discord.txt ios/app_prompts/doordash.txt ios/app_prompts/gmail.txt ios/app_prompts/googlemaps.txt ios/app_prompts/imessage.txt ios/app_prompts/instagram.txt ios/app_prompts/lyft.txt ios/app_prompts/outlook.txt ios/app_prompts/reddit.txt ios/app_prompts/robinhood.txt ios/app_prompts/safari.txt ios/app_prompts/settings.txt ios/app_prompts/slack.txt ios/app_prompts/snapchat.txt ios/app_prompts/spotify.txt ios/app_prompts/telegram.txt ios/app_prompts/tiktok.txt ios/app_prompts/twitter.txt ios/app_prompts/uber.txt ios/app_prompts/ubereats.txt ios/app_prompts/venmo.txt ios/app_prompts/whatsapp.txt ios/app_prompts/youtube.txt ios/app_prompts/zoom.txt ios/Pilot/.DS_Store ios/Pilot/BroadcastExtension/BroadcastExtension.entitlements ios/Pilot/BroadcastExtension/Info.plist ios/Pilot/BroadcastExtension/SampleHandler.swift ios/Pilot/BroadcastExtensionSetupUI/BroadcastSetupViewController.swift ios/Pilot/BroadcastExtensionSetupUI/Info.plist ios/Pilot/Pilot.xcodeproj/project.pbxproj ios/Pilot/Pilot.xcodeproj/project.xcworkspace/contents.xcworkspacedata ios/Pilot/Pilot.xcodeproj/project.xcworkspace/xcuserdata/arihanvaranasi.xcuserdatad/UserInterfaceState.xcuserstate ios/Pilot/Pilot.xcodeproj/project.xcworkspace/xcuserdata/ethangoodhart.xcuserdatad/UserInterfaceState.xcuserstate ios/Pilot/Pilot.xcodeproj/xcshareddata/xcschemes/Pilot.xcscheme ios/Pilot/Pilot.xcodeproj/xcuserdata/arihanvaranasi.xcuserdatad/xcschemes/xcschememanagement.plist ios/Pilot/Pilot.xcodeproj/xcuserdata/ethangoodhart.xcuserdatad/xcschemes/xcschememanagement.plist ios/Pilot/Pilot/.DS_Store ios/Pilot/Pilot/Assets.xcassets/.DS_Store ios/Pilot/Pilot/Assets.xcassets/AccentColor.colorset/Contents.json ios/Pilot/Pilot/Assets.xcassets/AppIcon 1.appiconset/Contents.json ios/Pilot/Pilot/Assets.xcassets/AppIcon 2.appiconset/.DS_Store ios/Pilot/Pilot/Assets.xcassets/AppIcon 2.appiconset/Contents.json ios/Pilot/Pilot/Assets.xcassets/AppIcon 3.appiconset/Contents.json ios/Pilot/Pilot/Assets.xcassets/AppIcon.appiconset/Contents.json ios/Pilot/Pilot/Assets.xcassets/Contents.json ios/Pilot/Pilot/ContentView.swift ios/Pilot/Pilot/DeviceDetector.swift ios/Pilot/Pilot/Info.plist ios/Pilot/Pilot/LiveModeManager.swift ios/Pilot/Pilot/MoondreamService.swift ios/Pilot/Pilot/OpenRouterService.swift ios/Pilot/Pilot/Pilot.entitlements ios/Pilot/Pilot/PilotActivity.swift ios/Pilot/Pilot/PilotApp.swift ios/Pilot/Pilot/PilotShortcuts.swift ios/Pilot/Pilot/ScreenCaptureManager.swift ios/Pilot/Pilot/SettingsView.swift ios/Pilot/Pilot/SpeechManager.swift ios/Pilot/Pilot/SuggestionService.swift ios/Pilot/VoiceActivityWidget/Info.plist ios/Pilot/VoiceActivityWidget/VoiceActivityAttributes.swift ios/Pilot/VoiceActivityWidget/VoiceActivityWidget.swift ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - chore: add Config.swift to .gitignore and remove from tracking - chore: update OpenRouter API key - chore: add .gitignore for build artifacts and .DS_Store - feat: add BLE status LED, move esp32 sketch, iOS UI and BLE improvements - Merge branch 'main' of https://github.com/arihanv/claire - Merge esp-digitizer into main: batched actions, Dynamic Island, BLE fixes - feat: batched action execution, Dynamic Island live activity, BLE timeout fix - continued - App v2! - continued - major updates to ios app, abs working esp32 implemented! - Save current state for esp-digitizer branch - added print file - Merge branch 'revert-method' of https://github.com/arihanv/pilot into revert-method - fix: removed wifi now everything runs via bluetooth in app and esp32. absolute positioning is not working right now. - fix: entitle - Merge branch 'revert-method' of https://github.com/arihanv/pilot into revert-method - fix: action button intent support - update: improve - update: stuff ## Key source files (fetched from GitHub, selected and truncated for size) ### esp32/mouse_control.py ```python #!/usr/bin/env python3 """Interactive controller for ESP32 BLE Mouse + Keyboard + Media. Mouse (x,y are percentages 0-100): click x y move to (x,y) and click move x y move cursor to (x,y) press x y press and hold at (x,y) release release button Keyboard: type hello type a string key enter named key (enter, space, tab, esc, backspace, delete, up, down, left, right, home, end, pageup, pagedown, capslock, insert, f1-f24) key cmd+space with modifiers (ctrl, shift, alt, gui/cmd, chainable) Media: play / pause play/pause toggle next / prev next/previous track stop stop playback volup [n] volume up (optional repeat count) voldown [n] volume down mute mute toggle brightup brightness up brightdown brightness down lock lock screen eject / power / sleep iOS shortcuts: spotlight open Spotlight search home go to home screen appswitcher open app switcher screenshot take screenshot undo / redo undo/redo copy / paste / cut clipboard selectall select all find find in page/app newtab / closetab browser tabs refresh refresh page dock toggle dock (iPadOS) keyboard dismiss keyboard Other: demo run mouse demo status check BLE connection help show commands on ESP quit exit """ import serial import sys import threading import time PORT = "/dev/cu.usbmodem2101" BAUD = 115200 def reader(ser, stop_event): while not stop_event.is_set(): try: line = ser.readline() if line: text = line.decode("utf-8", errors="replace").rstrip() if text: print(f" <- {text}") except Exception: break def main(): port = sys.argv[1] if len(sys.argv) > 1 else PORT print(f"Connecting to {port}...") ser = serial.Serial(port, BAUD, timeout=0.5) time.sleep(0.5) while ser.in_waiting: ser.readline() stop = threading.Event() t = threading.Thread(target=reader, args=(ser, stop), daemon=True) t.start() print(__doc__) try: while True: try: cmd = input("> ").strip() except EOFError: break if not cmd: continue if cmd in ("quit", "exit", "q"): break ser.write((cmd + "\n").encode()) except KeyboardInterrupt: pass finally: stop.set() ser.close() print("\nDone.") if __name__ == "__main__": main() ``` ### ios/Pilot/VoiceActivityWidget/VoiceActivityAttributes.swift ```swift import ActivityKit struct VoiceActivityAttributes: ActivityAttributes { struct ContentState: Codable & Hashable { var phase: Phase var statusText: String /// Incremented on every update so iOS never deduplicates the state. var nonce: Int = 0 enum Phase: String, Codable, Hashable { case thinking // LLM is deciding next action case executing // running actions on phone case detecting // Moondream vision detection case waiting // explicit wait case listening // PTT recording case speaking // TTS playback } } } ``` ### ios/Pilot/Pilot/DeviceDetector.swift ```swift import Foundation /// Tracks available ESP32 BLE devices and persists the user's selection. @MainActor @Observable final class DeviceDetector { var availableDevices: [String] = [] var selectedDevice: String? { didSet { UserDefaults.standard.set(selectedDevice, forKey: "selectedESP32Device") } } init() { selectedDevice = UserDefaults.standard.string(forKey: "selectedESP32Device") } func refresh() async { } func updateFromBLE(names: [String], connected: String?) { availableDevices = names if let connected { selectedDevice = connected return } if let selected = selectedDevice, names.contains(selected) { return } if names.count == 1 { selectedDevice = names[0] } else if names.isEmpty { selectedDevice = nil } } } ``` ### ios/Pilot/BroadcastExtensionSetupUI/BroadcastSetupViewController.swift ```swift // // BroadcastSetupViewController.swift // BroadcastExtensionSetupUI // // Created by Ethan Goodhart on 2/14/26. // import ReplayKit class BroadcastSetupViewController: UIViewController { // Call this method when the user has finished interacting with the view controller and a broadcast stream can start func userDidFinishSetup() { // URL of the resource where broadcast can be viewed that will be returned to the application let broadcastURL = URL(string:"http://apple.com/broadcast/streamID") // Dictionary with setup information that will be provided to broadcast extension when broadcast is started let setupInfo: [String : NSCoding & NSObjectProtocol] = ["broadcastName": "example" as NSCoding & NSObjectProtocol] // Tell ReplayKit that the extension is finished setting up and can begin broadcasting self.extensionContext?.completeRequest(withBroadcast: broadcastURL!, setupInfo: setupInfo) } func userDidCancelSetup() { let error = NSError(domain: "YouAppDomain", code: -1, userInfo: nil) // Tell ReplayKit that the extension was cancelled by the user self.extensionContext?.cancelRequest(withError: error) } } ``` ### ios/Pilot/Pilot/ScreenCaptureManager.swift ```swift import Foundation /// Reads the latest full-device screenshot from the App Group shared container, /// written by the BroadcastExtension's SampleHandler. class ScreenCaptureManager { static let appGroupID = "group.dev.ethan.Pilot" private var sharedContainerURL: URL? { FileManager.default.containerURL( forSecurityApplicationGroupIdentifier: Self.appGroupID ) } private var screenshotURL: URL? { sharedContainerURL?.appendingPathComponent("latest_screenshot.jpg") } /// Whether the broadcast extension is currently running. var isBroadcastActive: Bool { guard let containerURL = sharedContainerURL else { return false } let flagURL = containerURL.appendingPathComponent("broadcast_active") return FileManager.default.fileExists(atPath: flagURL.path) } /// Remove the broadcast_active flag file to clear stale state. func clearBroadcastFlag() { guard let containerURL = sharedContainerURL else { return } let flagURL = containerURL.appendingPathComponent("broadcast_active") try? FileManager.default.removeItem(at: flagURL) print("[ScreenCapture] Cleared broadcast_active flag") } /// Returns the latest JPEG screenshot from the broadcast, or nil. func takeScreenshot() -> Data? { guard let url = screenshotURL, FileManager.default.fileExists(atPath: url.path) else { print("[ScreenCapture] No screenshot file available") return nil } do { let data = try Data(contentsOf: url) print("[ScreenCapture] Read screenshot: \(data.count) bytes") return data } catch { print("[ScreenCapture] Failed to read screenshot: \(error)") return nil } } } ``` ### ios/Pilot/Pilot/PilotShortcuts.swift ```swift import AppIntents import Foundation private enum ShortcutBridge { static let appGroup = "group.dev.ethan.Pilot" static let queueKey = "shortcut_prompt_queue" static let dictateKey = "shortcut_dictate_requested" } // MARK: - Send Prompt struct SendPromptToPilotIntent: AppIntent { static var title: LocalizedStringResource = "Send Prompt to Pilot" static var description = IntentDescription("Sends a text prompt to Pilot so it can run without dictation.") static var openAppWhenRun = true @Parameter(title: "Prompt") var prompt: String static var parameterSummary: some ParameterSummary { Summary("Send \(\.$prompt) to Pilot") } @MainActor func perform() async throws -> some IntentResult & ProvidesDialog { if let defaults = UserDefaults(suiteName: ShortcutBridge.appGroup) { var queue = defaults.stringArray(forKey: ShortcutBridge.queueKey) ?? [] queue.append(prompt) defaults.set(queue, forKey: ShortcutBridge.queueKey) } return .result(dialog: "Sent your prompt to Pilot.") } } // MARK: - Dictate struct PilotDictateIntent: AppIntent { static var title: LocalizedStringResource = "Pilot Dictate" static var description = IntentDescription("Opens Pilot and starts listening for your voice command.") static var openAppWhenRun = true @MainActor func perform() async throws -> some IntentResult & ProvidesDialog { if let defaults = UserDefaults(suiteName: ShortcutBridge.appGroup) { defaults.set(true, forKey: ShortcutBridge.dictateKey) } return .result(dialog: "Pilot is listening.") } } // MARK: - App Shortcuts struct PilotAppShortcuts: AppShortcutsProvider { static var appShortcuts: [AppShortcut] { AppShortcut( intent: PilotDictateIntent(), phrases: [ "Dictate to \(.applicationName)", "\(.applicationName) dictate", "Start \(.applicationName)" ], shortTitle: "Dictate", systemImageName: "mic" ) AppShortcut( intent: SendPromptToPilotIntent(), phrases: [ "Ask \(.applicationName)", "Send to \(.applicationName)" ], shortTitle: "Send Prompt", systemImageName: "text.bubble" ) } } ``` ### ios/Pilot/Pilot/PilotActivity.swift ```swift import ActivityKit import Foundation // Duplicate of VoiceActivityAttributes for the main app target. // Must stay in sync with VoiceActivityWidget/VoiceActivityAttributes.swift. struct VoiceActivityAttributes: ActivityAttributes { struct ContentState: Codable & Hashable { var phase: Phase var statusText: String var nonce: Int = 0 enum Phase: String, Codable, Hashable { case thinking case executing case detecting case waiting case listening case speaking } } } /// Manages the Live Activity lifecycle for the CUA loop Dynamic Island. @MainActor final class PilotActivityManager { static let shared = PilotActivityManager() private var activity: Activity? private var nonce = 0 private init() {} func start(phase: VoiceActivityAttributes.ContentState.Phase, status: String = "") { // End any existing activity first endSync() let state = VoiceActivityAttributes.ContentState(phase: phase, statusText: status, nonce: nonce) do { activity = try Activity.request( attributes: VoiceActivityAttributes(), content: .init(state: state, staleDate: nil), pushType: nil ) print("[Activity] Started: \(phase.rawValue)") } catch { print("[Activity] Failed to start: \(error)") } } func update(phase: VoiceActivityAttributes.ContentState.Phase, status: String = "") { guard let activity else { return } nonce += 1 let state = VoiceActivityAttributes.ContentState(phase: phase, statusText: status, nonce: nonce) Task { await activity.update(.init(state: state, staleDate: nil)) } } func end() { guard let activity else { return } nonce = 0 let finalState = VoiceActivityAttributes.ContentState(phase: .thinking, statusText: "Done", nonce: -1) Task { await activity.end(.init(state: finalState, staleDate: nil), dismissalPolicy: .immediate) self.activity = nil print("[Activity] Ended") } } /// Synchronous end for use before starting a new activity. private func endSync() { guard let activity else { return } nonce = 0 let finalState = VoiceActivityAttributes.ContentState(phase: .thinking, statusText: "", nonce: -1) Task { await activity.end(.init(state: finalState, staleDate: nil), dismissalPolicy: .immediate) } self.activity = nil } } ``` ### ios/Pilot/Pilot/PilotApp.swift ```swift // // PilotApp.swift // Pilot // // Created by Ethan Goodhart on 2/14/26. // import SwiftUI import Foundation private enum ShortcutBridge { static let appGroup = "group.dev.ethan.Pilot" static let queueKey = "shortcut_prompt_queue" static let dictateKey = "shortcut_dictate_requested" } @main struct PilotApp: App { @State private var manager = LiveModeManager(apiKey: Config.openRouterAPIKey) @Environment(\.scenePhase) private var scenePhase private func dequeueShortcutPrompts() -> [String] { guard let defaults = UserDefaults(suiteName: ShortcutBridge.appGroup) else { return [] } let queue = defaults.stringArray(forKey: ShortcutBridge.queueKey) ?? [] if !queue.isEmpty { defaults.removeObject(forKey: ShortcutBridge.queueKey) } return queue } private func processQueuedShortcutPrompts() { let prompts = dequeueShortcutPrompts() for prompt in prompts { let text = prompt.trimmingCharacters(in: .whitespacesAndNewlines) guard !text.isEmpty else { continue } print("[ShortcutBridge] Received queued prompt: \(text.prefix(80))") manager.submitShortcutPrompt(text) } } private func processDictateRequest() { guard let defaults = UserDefaults(suiteName: ShortcutBridge.appGroup), defaults.bool(forKey: ShortcutBridge.dictateKey) else { return } defaults.removeObject(forKey: ShortcutBridge.dictateKey) print("[ShortcutBridge] Dictate requested") Task { if !manager.isActive { await manager.startLiveMode() } manager.beginRecording() } } var body: some Scene { WindowGroup { ContentView(manager: manager) .onAppear { processQueuedShortcutPrompts() processDictateRequest() } .onChange(of: scenePhase) { _, newPhase in if newPhase == .active { processQueuedShortcutPrompts() processDictateRequest() } } .onOpenURL { url in guard url.scheme?.lowercased() == "pilot" else { return } print("[ShortcutBridge] Opened via URL: \(url.absoluteString)") // Prefer App Group queue for reliable delivery across process boundaries. processQueuedShortcutPrompts() // Backward-compatible fallback if a text query item is present. guard let components = URLComponents(url: url, resolvingAgainstBaseURL: false) else { return } let text = components.queryItems?.first(where: { $0.name == "text" })?.value ?? "" if !text.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { manager.submitShortcutPrompt(text) } } } } } ``` ### ios/Pilot/BroadcastExtension/SampleHandler.swift ```swift // // SampleHandler.swift // BroadcastExtension // // Created by Ethan Goodhart on 2/14/26. // import ReplayKit import CoreMedia import UIKit import CoreImage /// Broadcast Upload Extension handler. Receives full-device screen frames and /// writes the latest one as a JPEG to the App Group shared container so the /// main Pilot app can read it on demand. class SampleHandler: RPBroadcastSampleHandler { static let appGroupID = "group.dev.ethan.Pilot" private let ciContext = CIContext() private var frameCount = 0 /// Process every Nth video frame to save CPU (~2 fps at 30 fps input). private let captureInterval = 15 private var sharedContainerURL: URL? { FileManager.default.containerURL( forSecurityApplicationGroupIdentifier: Self.appGroupID ) } // MARK: - Lifecycle override func broadcastStarted(withSetupInfo setupInfo: [String: NSObject]?) { guard let containerURL = sharedContainerURL else { return } // Write a flag so the main app knows the broadcast is live. let flagURL = containerURL.appendingPathComponent("broadcast_active") try? Data().write(to: flagURL) // Clear stale screenshot. let screenshotURL = containerURL.appendingPathComponent("latest_screenshot.jpg") try? FileManager.default.removeItem(at: screenshotURL) } override func broadcastFinished() { guard let containerURL = sharedContainerURL else { return } let flagURL = containerURL.appendingPathComponent("broadcast_active") try? FileManager.default.removeItem(at: flagURL) } // MARK: - Frame processing override func processSampleBuffer( _ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType ) { guard sampleBufferType == .video else { return } frameCount += 1 guard frameCount % captureInterval == 0 else { return } guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer), let containerURL = sharedContainerURL else { return } let screenshotURL = containerURL.appendingPathComponent("latest_screenshot.jpg") let ciImage = CIImage(cvPixelBuffer: imageBuffer) guard let cgImage = ciContext.createCGImage(ciImage, from: ciImage.extent) else { return } let uiImage = UIImage(cgImage: cgImage) // Scale down for API efficiency (max 1024px on longest side). let maxDim: CGFloat = 2048 let scale = min(maxDim / uiImage.size.width, maxDim / uiImage.size.height, 1.0) let newSize = CGSize( width: floor(uiImage.size.width * scale), height: floor(uiImage.size.height * scale) ) let fmt = UIGraphicsImageRendererFormat() fmt.scale = 1.0 let renderer = UIGraphicsImageRenderer(size: newSize, format: fmt) let scaled = renderer.image { _ in uiImage.draw(in: CGRect(origin: .zero, size: newSize)) } if let jpegData = scaled.jpegData(compressionQuality: 0.7) { try? jpegData.write(to: screenshotURL, options: .atomic) } } } ``` ### ios/Pilot/VoiceActivityWidget/VoiceActivityWidget.swift ```swift import ActivityKit import SwiftUI import WidgetKit @main struct VoiceActivityWidgetBundle: WidgetBundle { var body: some Widget { VoiceActivityLiveActivity() } } struct VoiceActivityLiveActivity: Widget { var body: some WidgetConfiguration { ActivityConfiguration(for: VoiceActivityAttributes.self) { context in lockScreenView(context.state) } dynamicIsland: { context in DynamicIsland { DynamicIslandExpandedRegion(.leading) { phaseIcon(context.state.phase, size: 16) .frame(width: 28, height: 28) } DynamicIslandExpandedRegion(.center) { Text("Pilot") .font(.system(size: 13, weight: .semibold, design: .rounded)) .foregroundStyle(.white.opacity(0.6)) } DynamicIslandExpandedRegion(.bottom) { Text(context.state.statusText.isEmpty ? phaseLabel(context.state.phase) : context.state.statusText) .font(.system(size: 15, weight: .medium, design: .rounded)) .lineLimit(2) .frame(maxWidth: .infinity, alignment: .leading) .padding(.top, 4) } } compactLeading: { phaseIcon(context.state.phase, size: 12) } compactTrailing: { Text(context.state.statusText.isEmpty ? phaseLabel(context.state.phase) : context.state.statusText) .font(.system(size: 12, weight: .medium, design: .rounded)) .lineLimit(1) .truncationMode(.tail) .frame(maxWidth: 120) } minimal: { phaseIcon(context.state.phase, size: 12) } } } @ViewBuilder private func phaseIcon(_ phase: VoiceActivityAttributes.ContentState.Phase, size: CGFloat) -> some View { let config = phaseConfig(phase) Image(systemName: config.icon) .font(.system(size: size, weight: .bold)) .foregroundStyle(config.color) .symbolEffect(.variableColor.iterative, isActive: phase == .thinking || phase == .executing || phase == .waiting || phase == .detecting) .contentTransition(.symbolEffect(.replace)) } private struct PhaseConfig { let icon: String let color: Color } private func phaseConfig(_ phase: VoiceActivityAttributes.ContentState.Phase) -> PhaseConfig { switch phase { case .thinking: return PhaseConfig(icon: "sparkles", color: .white) case .executing, .waiting: return PhaseConfig(icon: "bolt.fill", color: .white) case .detecting: return PhaseConfig(icon: "moon.fill", color: .white) case .listening: return PhaseConfig(icon: "ear.fill", color: .white) case .speaking: return PhaseConfig(icon: "waveform", color: .white) } } private func phaseLabel(_ phase: VoiceActivityAttributes.ContentState.Phase) -> String { switch phase { case .thinking: return "Thinking…" case .executing: return "Executing…" case .detecting: return "Detecting…" case .waiting: return "Waiting…" case .listening: return "Listening…" case .speaking: return "Speaking…" } } private func lockScreenView(_ state: VoiceActivityAttributes.ContentState) -> some View { let config = phaseConfig(state.phase) return VStack(alignment: .leading, spacing: 8) { HStack(spacing: 8) { Image(systemName: config.icon) .font(.system(size: 14, weight: .semibold)) .foregroundStyle(config.color) .symbolEffect(.variableColor.iterative, isActive: state.phase == .thinking || state.phase == .executing) .contentTransition(.symbolEffect(.replace)) Text("Pilot") .font(.system(size: 12, weight: .semibold, design: .rounded)) .foregroundStyle(.secondary) Spacer() } Text(state.statusText.isEmpty ? phaseLabel(state.phase) : state.statusText) .font(.system(size: 16, weight: .medium, design: .rounded)) .lineLimit(2) .frame(maxWidth: .infinity, alignment: .leading) } .padding(16) .activityBackgroundTint(.black.opacity(0.75)) } } ``` [6 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]