# Project export: Skyheart This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Medicinal drone delivery with facial confirmation - Devpost: https://devpost.com/software/skyheart - GitHub: https://github.com/Mootbing/SkyHeart - Video: https://www.youtube.com/embed/DcMdC08CqSM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Claude Opus 4.6 (21 commits) ## Devpost submission (written by the team) ### Inspiration This project started as a real problem I faced in the last few weeks. Several friends at Duke got sick with pneumonia, and while they were recovering, something as simple as picking up medication became a major burden. Many were tired and weak, and often couldn’t move around easily, so they had to depend on others for basic delivery logistics. Every pickup meant delays, coordination risk, and stress for both the patients and the people trying to help them. That felt wrong—especially since timely medication can directly affect recovery. Skyheart was built to reduce that friction: an autonomous system that can travel to a destination, verify the correct person, and complete a safe handoff with minimal manual control. ### What it does Skyheart is an end-to-end autonomous drone medication delivery system. A dispatcher opens the app, enters a patient’s address, and uploads a reference photo. The drone is then routed to that exact location using turn-by-turn road directions (not direct-flight shortcuts), which is safer in dense urban environments. When the drone arrives, it enters identification mode: onboard vision segments and detects people, then verifies identity by comparing the detected face against the reference photo using AWS Rekognition. The drone only descends and releases the medication after positive identification. The control app is built in React Native with an Uber-style interface: dispatchers can select destinations via autocomplete, view route and live progress, and monitor mission state. Since the drone manufacturer’s app is closed-source and offers no public SDK, Skyheart injects touch gestures via Android Accessibility API to issue equivalent joystick swipes and button presses. The phone streams the drone camera over USB (its Wi-Fi is tied up with the drone connection), while all CV and navigation logic runs on a Python backend. A live dashboard shows segmentation overlays, detection boxes, GPS telemetry, and mission state in real time. ### How we built it We implemented this as a modular system: React Native + Kotlin Module app for operator control and phone-to-backend communication. Python backend (FastAPI/Uvicorn) for all mission logic and computer vision. USB transport via ADB reverse port forwarding, with the phone connected to the drone over Wi-Fi and to the backend over USB. A 7-state mission pipeline: INPUT → NAVIGATION → IDENTIFICATION → APPROACH → DELIVERY → DONE/HOVER. A modular CV stack with configurable backends and fallbacks for person detection, segmentation, and face matching. A browser live dashboard for monitoring with real-time overlays and telemetry. A proxy routing layer for geocoding, reverse geocoding, routing, and map tiles to support mapping features in the phone environment. ### Challenges we ran into No SDK / closed-source drone app required building a control interface through Android Accessibility gesture injection instead of official APIs. Inference speed and latency from face/person detection running too frequently at high resolution. No phone internet access while connected to the drone via Wi-Fi, forcing USB- based architecture and additional proxy constraints. End-to-end latency from drone → phone → backend created about a 1-second reaction delay in practice. Inconsistent cheap drone GPS, which made precision navigation harder after a few waypoints. Real-world data variability (signal drops, unstable frame quality, variable subject scale, inconsistent lighting). Hardware/software instability, including ADB failures that required phone resets and driver reinstallation. ### Accomplishments we're proud of Built a working autonomous delivery loop: enter an address, route safely, identify the correct person, and complete delivery. Replaced an SDK dependency with a robust Accessibility-based control path and action recorder for consistent cross-device tap mapping. Designed a truly swappable AI pipeline where person detection, face matching, segmentation, and obstacle logic can be swapped without rewiring the system. Delivered real-time situational awareness through a live dashboard with overlays, telemetry, and mission state in one place. ### What we learned In robotics, reliability is often about control flow, timing, and graceful fallback paths—not just model accuracy. End-to-end latency tuning (throttling, scheduling, and frame strategy) is as important as per-frame precision. Deployments benefit from strong defaults plus optional upgrades for different hardware profiles (GPU laptops vs CPU-only machines). Well-defined typed protocol contracts are essential to coordinate frontend, backend, and control layers as complexity grows. ### What's next Upgrade to a production-grade drone with a real SDK and stronger onboard control reliability. Add edge/on-device inference to reduce cloud dependency and latency. Improve temporal tracking (object IDs, smoothing, and memory) to reduce flicker and jitter. Add adaptive scheduling for frame rate, resolution, and inference budget based on mission state and network/battery conditions. Expand dashboard tooling with session replay and stage-level precision/latency metrics for faster iteration and tuning. ## README (from the GitHub repository) # SkyHeart — Drone Delivery & Identification System Autonomous drone delivery system that navigates street-level routes, identifies a target person using computer vision, and delivers a message. A phone captures the drone manufacturer's app screen and streams frames to a PC server over USB for processing. The server runs YOLOv8 person detection and AWS Rekognition face matching, then sends movement commands back to the phone, which injects touch gestures into the drone app via Android's Accessibility Service. All network traffic between the phone and server runs over USB via `adb reverse` port forwarding — the phone's WiFi stays connected to the drone. --- ## Architecture Overview ``` +----------------------------------------------------------------------+ | PHONE (Android) | | | | +---------------+ +------------------+ +-------------------+ | | | React Native | | MediaProjection | | Accessibility | | | | App UI |--->| Screen Capture | | Service (Touch) | | | | (SkyHeart) | | (10fps, 2400x1080)| | Gesture Inject | | | +-------+-------+ +--------+---------+ +-------^-----------+ | | | | | | | | base64 JPEG frames swipe gestures | | | | | | | +----------+----------+ | | | | WebSocket (JSON) via USB | | | v | | +---------------------+----------------------------------+--------------+ | USB (adb reverse) | | ws://localhost:8765/ws | | | +---------------------+----------------------------------+--------------+ | v PC SERVER | | | +----------------------------------------------------+--+ | | | FastAPI + WebSocket | | | | | | | | +-----------+ +--------------+ +----------+ | | | | | State | | YOLOv8 | | Face | | | | | | Machine | | Nano | | Matcher | | | | | | | | (~6MB) | | (Rekog.) | | | | | +-----------+ +--------------+ +----------+ | | | | | | | | +-----------+ +--------------+ +----------+ | | | | | HTTP Proxy| | Dashboard | | Approach | | | | | | (Geocode, | | (detect | | Control | | | | | | Route) | | toggle) | | | | | | | +-----------+ +--------------+ +----------+ | | | | | | | | movement commands (JSON) -------------------+ | | +----------------------------------------------------+ | +----------------------------------------------------------------------+ ``` ### Data Flow 1. **Phone WiFi** -> Drone (flight control) 2. **Phone USB** -> PC Server (frames, commands, geocoding, maps) 3. `adb reverse tcp:8765 tcp:8765` tunnels server to `localhost:8765` on phone 4. `adb reverse tcp:8081 tcp:8081` tunnels Metro bundler for dev --- ## Phone App (React Native + Android Native) ### Screens | Screen | Purpose | |--------|---------| | `InputScreen` | Uber-style booking: From (GPS) / To (address search), route map with waypoints, turn-by-turn directions, reference photo, delivery message | | `SettingsScreen` | Server WebSocket URL, connection status, reference photo (persisted + auto-sent to server), drone app picker, accessibility service, action recorder, test streaming | | `WatchScreen` | Streaming screen — starts capture, sends reference photo to server, black screen with "Streaming live via USB" status | | `ActionRecorderScreen` | Fullscreen grid for recording tap positions (takeoff/landing) on the drone app | | `DeliveryScreen` | Displays delivery message, confirm button | ### Features - **Uber-style booking UI** — From/To card with green/red dots, reverse-geocoded current location - **Address autocomplete** — debounced Nominatim search through server proxy - **Route map** — Leaflet + OSM tiles rendered in WebView, waypoint markers at each turn - **Turn-by-turn waypoints** — scrollable list with coordinates, tappable to highlight on map - **Reference photo** — pick from gallery or camera, persisted across restarts, sent to server immediately on upload - **Drone app picker** — select which drone manufacturer app to control - **Action recorder** — record tap positions for takeoff/landing automation - **GPS retry** — 3 attempts with high/low accuracy fallback - **Live dashboard** — browser UI at `http://localhost:8765/dashboard` with live stream, detections, GPS, mission state - **Test mode** — starts streaming without mission for detection testing ### Native Modules (Kotlin) **Screen Capture** (`ScreenCaptureModule` + `ScreenCaptureService`): - Uses Android **MediaProjection API** to capture the drone manufacturer's app screen - Runs as a foreground service with `mediaProjection` foreground service type - Captures at 2400x1080 (native resolution), JPEG quality 85, ~10 fps - Emits `onFrameCaptured` events with base64 JPEG data **Touch Injection** (`DroneAccessibilityService` + `TouchInjectorModule`): - Uses Android **Accessibility Service** with `GestureDescription` API - Maps directional commands to swipe gestures on configurable joystick positions - Right joystick: forward/back/left/right (pitch & roll) - Left joystick: up/down (throttle), rotate_cw/rotate_ccw (yaw) - Intensity (0.0-1.0) scales swipe distance from joystick center **App Launcher** (`AppLauncher`): - Lists installed apps, launches selected drone app by package name ### Server Proxy Endpoints Since the phone's WiFi is connected to the drone, all HTTP requests go through the server via USB: | Endpoint | Purpose | |----------|---------| | `GET /geocode?q=...` | Nominatim address search | | `GET /reverse-geocode?lat=...&lon=...` | Nominatim reverse geocoding | | `GET /route?from_lat=...&from_lng=...&to_lat=...&to_lng=...` | OSRM driving route (with steps) | | `GET /tile/{z}/{x}/{y}.png` | OpenStreetMap tile proxy | | `GET /health` | Server health check | | `GET /dashboard` | Live web dashboard (stream, detections, GPS, state) | | `WS /ws` | Phone WebSocket (frames, commands, mission data) | | `WS /ws/dashboard` | Dashboard WebSocket (binary JPEG frames + JSON metadata, detect toggle) | --- ## How It Works ### 1. Mission Input The user opens the phone app and sees an Uber-style booking screen: - **From** — current GPS location, automatically reverse-geocoded to a street address - **To** — search via Nominatim autocomplete, select destination - Route map appears with waypoint markers at each turn - Scrollable turn-by-turn directions with coordinates (tappable to highlight on map) - **Reference photo** of the target person (from camera or gallery, persisted, sent to server on upload) - **Delivery message** (default: "moo") On "Book Delivery", the phone sends all turn-by-turn waypoint coordinates to the server. ### 2. Navigation The drone follows the planned waypoints (turn-by-turn coordinates from OSRM) sequentially. For each frame received from the phone (~10 fps): 1. **GPS comparison**: The server compares the drone's current GPS to the next waypoint using haversine distance 2. **Heading computation**: Bearing from current position to targe [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 57 recognized source files, 238 KB. - FastAPI (technology) — detected in the code - JavaScript (language) — detected in the code - Kotlin (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (71 of 71) ``` .gitignore adb-tunnel.sh IMPLEMENTATION.md phone/android/app/build.gradle phone/android/app/debug.keystore phone/android/app/proguard-rules.pro phone/android/app/src/main/AndroidManifest.xml phone/android/app/src/main/java/com/dronecontrol/accessibility/DroneAccessibilityService.kt phone/android/app/src/main/java/com/dronecontrol/accessibility/TouchInjectorModule.kt phone/android/app/src/main/java/com/dronecontrol/accessibility/TouchInjectorPackage.kt phone/android/app/src/main/java/com/dronecontrol/applauncher/AppLauncherModule.kt phone/android/app/src/main/java/com/dronecontrol/applauncher/AppLauncherPackage.kt phone/android/app/src/main/java/com/dronecontrol/MainActivity.kt phone/android/app/src/main/java/com/dronecontrol/MainApplication.kt phone/android/app/src/main/java/com/dronecontrol/screencapture/ScreenCaptureModule.kt phone/android/app/src/main/java/com/dronecontrol/screencapture/ScreenCapturePackage.kt phone/android/app/src/main/java/com/dronecontrol/screencapture/ScreenCaptureService.kt phone/android/app/src/main/res/values/strings.xml phone/android/app/src/main/res/values/styles.xml phone/android/app/src/main/res/xml/accessibility_service_config.xml phone/android/build.gradle phone/android/gradle.properties phone/android/gradle/wrapper/gradle-wrapper.properties phone/android/gradlew phone/android/local.properties phone/android/settings.gradle phone/app.json phone/App.tsx phone/index.js phone/metro.config.js phone/package.json phone/src/components/ManualControl.tsx phone/src/components/NavigationOverlay.tsx phone/src/components/StatusBar.tsx phone/src/screens/ActionRecorderScreen.tsx phone/src/screens/DeliveryScreen.tsx phone/src/screens/InputScreen.tsx phone/src/screens/SettingsScreen.tsx phone/src/screens/WatchScreen.tsx phone/src/services/DroneControl.ts phone/src/services/ScreenCapture.ts phone/src/services/WebSocketService.ts phone/src/types/protocol.ts phone/tsconfig.json README.md server/config.py server/identification/__init__.py server/identification/approach.py server/identification/face_matcher.py server/identification/person_detector.py server/main.py server/models/__init__.py server/navigation/__init__.py server/navigation/commander.py server/navigation/geo_utils.py server/navigation/geocoder.py server/navigation/obstacle_avoidance.py server/navigation/router.py server/requirements.txt server/state_machine.py server/tests/__init__.py server/tests/test_approach.py server/tests/test_commander.py server/tests/test_geo_utils.py server/tests/test_health.py server/tests/test_obstacle_avoidance.py server/tests/test_person_detector.py server/tests/test_state_machine.py server/tests/test_ws_handler.py server/ws_handler.py SETUP.md ``` ### Dependencies - phone/package.json: @react-native-async-storage/async-storage@^2.2.0, @react-native-community/geolocation@^3.4.0, @react-native/eslint-config@^0.73.2, @react-native/gradle-plugin@^0.73.4, @react-native/metro-config@^0.84.0, @react-navigation/native@^6.1.9, @react-navigation/native-stack@^6.9.17, @types/react@^18.2.45, react@18.2.0, react-native@0.73.4, react-native-image-picker@^7.1.0, react-native-maps@^1.14.0, react-native-permissions@^4.1.4, react-native-safe-area-context@^4.8.2, react-native-screens@3.29.0, react-native-webview@^13.16.0, typescript@^5.3.3 - server/requirements.txt: boto3@>=1.29.0, fastapi@>=0.104.0, googlemaps@>=4.10.0, httpx@>=0.25.0, numpy@>=1.24.0, opencv-python@>=4.8.0, Pillow@>=10.0.0, python-dotenv@>=1.0.0, ultralytics@>=8.0.0, uvicorn@>=0.24.0, websockets@>=12.0 ### Recent commits (newest first) - Pre-load YOLOv8 at startup with dashboard loading bar - Use producer/consumer queue to keep WS alive during detection - Fix detection breaking WebSocket tunnel by non-blocking frame processing - Add adb tunnel keepalive script and increase WS reconnect resilience - Update all docs to reflect SkyHeart name and YOLOv8 + Rekognition stack - Send reference photo to server immediately on upload - Add AWS Rekognition face matching to dashboard detect mode - Add frame-skipping to dashboard detection (every 10th frame) - Add SETUP.md with WSL2 + adb.exe deployment workflow - Replace SAM with YOLOv8 nano for person detection, add dashboard detect toggle - Reorder settings, fix screen capture flow, add accessibility check, improve action recorder - Add action recorder, drone app picker, and app launcher native module - Add AI inference integration guide (IMPLEMENTATION.md) and update README - Uber-style UI, settings page, waypoint navigation, persistent photo - Update README to reflect current state: native res capture, dashboard, no map picker - Increase capture to native resolution (2400x1080) at 10fps with quality 85 - Boost capture to 60fps, simplify watch screen, optimize dashboard streaming - Add phone app UI, live dashboard, screen capture fix, and server proxy endpoints - Add unit tests, pluggable obstacle detection, README, and bug fixes - Fix all issues from senior code review (CRITICAL, HIGH, MEDIUM) ## Key source files (fetched from GitHub, selected and truncated for size) ### SETUP.md ```markdown # SkyHeart Setup Guide ## Prerequisites - WSL2 (Ubuntu) on Windows - Android phone with USB debugging enabled - Node.js 18+, JDK 17, Android SDK - Python 3.12+ ## Important: ADB on WSL2 WSL2 cannot see USB devices natively. You **must** use the Windows `adb.exe` instead of the Linux `adb` package. - **Do NOT install** the Linux `adb` package (`apt install adb`) -- it will never detect your phone - Use `adb.exe` which is available from the Windows PATH inside WSL2 - If you have multiple devices (USB + wireless), specify the serial: `adb.exe -s ` ### Deploying the APK from WSL2 Since `adb.exe` can't read WSL2 UNC paths directly, copy the APK to a Windows path first: ```bash # 1. Build the APK cd phone/android && ./gradlew assembleDebug # 2. Copy to a Windows-accessible location mkdir -p /mnt/c/temp cp phone/android/app/build/outputs/apk/debug/app-debug.apk /mnt/c/temp/app-debug.apk # 3. Install via Windows adb adb.exe install -r 'C:\temp\app-debug.apk' # If multiple devices are connected, specify the serial: adb.exe devices # list devices adb.exe -s install -r 'C:\temp\app-debug.apk' ``` ### Port Forwarding The phone's WiFi is connected to the drone, so all server communication goes over USB: ```bash adb.exe reverse tcp:8765 tcp:8765 # Python server adb.exe reverse tcp:8081 tcp:8081 # Metro bundler (dev) ``` USB tunnels drop frequently. Use the keepalive script to auto-re-establish every 3 seconds: ```bash ./adb-tunnel.sh # runs in foreground, Ctrl+C to stop ``` Or run it in the background: `./adb-tunnel.sh &` ### Restarting the App ```bash adb.exe shell am force-stop com.dronecontrol adb.exe shell am start -n com.dronecontrol/.MainActivity ``` ## Server Setup ```bash cd server python3 -m venv venv && source venv/bin/activate pip install -r requirements.txt # installs ultralytics (YOLOv8), fastapi, boto3, httpx, etc. python3 main.py # starts on 0.0.0.0:8765 ``` YOLOv8 nano model (~6MB) pre-loads at server startup with a warmup inference, so first detection is instant. The dashboard shows a loading bar until the model is ready. ### AWS Rekognition Credentials Face matching requires AWS credentials: 1. Go to [IAM Console](https://console.aws.amazon.com/iam/) -> Users -> Create user 2. Attach the `AmazonRekognitionFullAccess` policy 3. Security credentials tab -> Create access key -> "Application running outside AWS" 4. Copy the Access Key ID and Secret Access Key into `server/.env`: ```env AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID=AKIA... AWS_SECRET_ACCESS_KEY=... REKOGNITION_SIMILARITY_THRESHOLD=90.0 ``` ### Restarting the Server ```bash lsof -ti:8765 | xargs kill -9 2>/dev/null sleep 1 cd server && python3 main.py ``` ### Verify ```bash curl http://localhost:8765/health # -> {"status":"ok","state":"input","model_ready":true,"model_loading":false} ``` ## Phone App Setup ```bash cd phone npm install cd android && ./gradlew assembleDebug # Then deplo [truncated — 1245 more characters] ``` ### IMPLEMENTATION.md ```markdown # AI Inference Integration Guide This document explains how the AI components work and how to swap them for custom models. Each component has a clean interface — replace the implementation without changing the pipeline. --- ## Architecture The frame processing pipeline runs in `server/ws_handler.py`. Each frame flows through different processing stages depending on the current state: ``` Phone frame (JPEG) -> decode -> state router | +-------------+------------------+ v v v NAVIGATION IDENTIFICATION APPROACH | | | GPS waypoint YOLOv8 detect YOLOv8 detect following -> face_matcher -> face_matcher | | -> approach_command v v | movement cmd match found? v -> APPROACH bbox tracking or "arrived" (any state with detect toggle ON) -> YOLOv8 detect -> face_matcher (if reference photo uploaded) ``` ### Key files | File | Role | |------|------| | `server/ws_handler.py` | Frame pipeline — calls each component | | `server/identification/person_detector.py` | YOLOv8 nano person detection | | `server/identification/face_matcher.py` | AWS Rekognition face comparison | | `server/identification/approach.py` | Bbox -> movement commands | | `server/navigation/obstacle_avoidance.py` | Obstacle detection stub (placeholder) | | `server/config.py` | All tuneable parameters (.env) | --- ## 1. Person Detection (YOLOv8) **File:** `server/identification/person_detector.py` Uses YOLOv8 nano for fast, accurate person detection. The model auto-downloads (~6MB) on first use. ### Interface ```python class PersonDetector: def detect(self, frame: np.ndarray) -> List[Dict]: """ Args: frame: BGR numpy array (H, W, 3) Returns: List of person detections: - 'bbox': [x1, y1, x2, y2] pixel coordinates (int) - 'confidence': float (0-1) """ ``` ### Current implementation - **Model:** YOLOv8n (`yolov8n.pt`, ~6MB, auto-downloads) - **Class filter:** COCO class 0 (person) only - **Confidence threshold:** `PERSON_CONFIDENCE_THRESHOLD` (default 0.4) - **Speed:** ~20-50ms on CPU, ~5-10ms on GPU - **Lazy loading:** Model loaded on first `detect()` call ### How to replace To use a different detector (e.g., YOLOv8s for better accuracy, or a custom model): ```python class PersonDetector: def _load(self): from ultralytics import YOLO self._model = YOLO("yolov8s.pt") # or your custom model def detect(self, frame: np.ndarray) -> List[Dict]: # Just keep the same return format: [{"bbox": [...], "confidence": f [truncated — 7225 more characters] ``` ### server/requirements.txt ``` fastapi>=0.104.0 uvicorn>=0.24.0 websockets>=12.0 httpx>=0.25.0 ultralytics>=8.0.0 opencv-python>=4.8.0 Pillow>=10.0.0 boto3>=1.29.0 googlemaps>=4.10.0 numpy>=1.24.0 python-dotenv>=1.0.0 ``` ### phone/package.json ``` { "name": "DroneControl", "version": "1.0.0", "private": true, "scripts": { "android": "react-native run-android", "start": "react-native start", "lint": "eslint ." }, "dependencies": { "@react-native-async-storage/async-storage": "^2.2.0", "@react-native-community/geolocation": "^3.4.0", "@react-native/gradle-plugin": "^0.73.4", "@react-native/metro-config": "^0.84.0", "@react-navigation/native": "^6.1.9", "@react-navigation/native-stack": "^6.9.17", "react": "18.2.0", "react-native": "0.73.4", "react-native-image-picker": "^7.1.0", "react-native-maps": "^1.14.0", "react-native-permissions": "^4.1.4", "react-native-safe-area-context": "^4.8.2", "react-native-screens": "3.29.0", "react-native-webview": "^13.16.0" }, "devDependencies": { "@react-native/eslint-config": "^0.73.2", "@types/react": "^18.2.45", "typescript": "^5.3.3" } } ``` ### phone/index.js ```javascript import { AppRegistry } from 'react-native'; import App from './App'; import { name as appName } from './app.json'; AppRegistry.registerComponent(appName, () => App); ``` ### phone/App.tsx ```typescript /** * Drone Control — Root component with navigation setup. */ import React, { useEffect } from 'react'; import { TouchableOpacity, Text } from 'react-native'; import { NavigationContainer } from '@react-navigation/native'; import { createNativeStackNavigator } from '@react-navigation/native-stack'; import InputScreen from './src/screens/InputScreen'; import WatchScreen from './src/screens/WatchScreen'; import DeliveryScreen from './src/screens/DeliveryScreen'; import SettingsScreen, { loadReferencePhoto } from './src/screens/SettingsScreen'; import ActionRecorderScreen, { loadActionPoints } from './src/screens/ActionRecorderScreen'; const Stack = createNativeStackNavigator(); export default function App() { useEffect(() => { loadReferencePhoto(); loadActionPoints(); }, []); return ( ({ title: 'SkyHeart', headerTitleStyle: { color: '#fff', fontSize: 18, fontWeight: 'bold' }, headerRight: () => ( navigation.navigate('Settings')} style={{ paddingHorizontal: 4, paddingVertical: 6 }} > {'\u2699'} ), })} /> ); } ``` ### server/main.py ```python """FastAPI entry point for the drone control server.""" import asyncio import logging from contextlib import asynccontextmanager import httpx import uvicorn from fastapi import FastAPI, WebSocket, WebSocketDisconnect, Query from fastapi.responses import HTMLResponse, Response from config import WS_HOST, WS_PORT from ws_handler import ConnectionManager logging.basicConfig( level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s", ) logger = logging.getLogger(__name__) manager = ConnectionManager() @asynccontextmanager async def lifespan(app: FastAPI): logger.info("Drone Control Server starting on %s:%d", WS_HOST, WS_PORT) # Pre-load YOLO in a background thread so it doesn't block the event loop loop = asyncio.get_event_loop() loop.run_in_executor(None, manager.person_detector.load) yield app = FastAPI(title="Drone Control Server", lifespan=lifespan) http_client = httpx.AsyncClient(timeout=15.0) @app.get("/health") async def health(): det = manager.person_detector return { "status": "ok", "state": manager.sm.state.value, "model_ready": det.model_ready, "model_loading": det.model_loading, } @app.get("/geocode") async def geocode(q: str = Query(...)): """Proxy Nominatim geocoding for phone (no internet on phone).""" url = f"https://nominatim.openstreetmap.org/search?format=json&q={q}&limit=5&addressdetails=1" resp = await http_client.get(url, headers={"User-Agent": "DroneControl/1.0"}) return Response(content=resp.content, media_type="application/json") @app.get("/route") async def route( from_lat: float = Query(...), from_lng: float = Query(...), to_lat: float = Query(...), to_lng: float = Query(...), ): """Proxy OSRM routing for phone.""" url = f"https://router.project-osrm.org/route/v1/driving/{from_lng},{from_lat};{to_lng},{to_lat}?overview=full&geometries=geojson&steps=true" resp = await http_client.get(url) return Response(content=resp.content, media_type="application/json") @app.get("/reverse-geocode") async def reverse_geocode(lat: float = Query(...), lon: float = Query(...)): """Proxy Nominatim reverse geocoding for phone.""" url = f"https://nominatim.openstreetmap.org/reverse?format=json&lat={lat}&lon={lon}" resp = await http_client.get(url, headers={"User-Agent": "DroneControl/1.0"}) return Response(content=resp.content, media_type="application/json") @app.get("/tile/{z}/{x}/{y}.png") async def map_tile(z: int, x: int, y: int): """Proxy OSM map tiles for phone.""" url = f"https://tile.openstreetmap.org/{z}/{x}/{y}.png" resp = await http_client.get(url, headers={"User-Agent": "DroneControl/1.0"}) return Response(content=resp.content, media_type="image/png") @app.websocket("/ws") async def websocket_endpoint(websocket: WebSocket): import asyncio connected = await manager.connect(websocket) if not connected: return queue: asyncio.Queue = asyncio.Queue(maxsize=2) async def reader(): """Read messages as fast as possible so the WS stays healthy.""" try: while True: data = await websocket.receive_text() # Drop old frames if consumer is slow — keep only latest if queue.full(): try: queue.get_nowait() except asyncio.QueueEmpty: pass await queue.put(data) except WebSocketDisconnect: await queue.put(None) except Exception: await queue.put(None) async def processor(): """Process messages from queue — heavy work happens here.""" while True: data = await queue.get() if data is None: break await manager.handle_message(data) reader_task = asyncio.create_task(reader()) try: await processor() finally: reader_task.cancel() await manager.disconnect() @app.websocket("/ws/dashboard") async def dashboard_ws(websocket: WebSocket): """WebSocket for dashboard viewers — receives live frames + detections.""" await manager.dashboard_connect(websocket) try: while True: data = await websocket.receive_text() await manager.handle_dashboard_message(data) except WebSocketDisconnect: await manager.dashboard_disconnect(websocket) except Exception: await manager.dashboard_disconnect(websocket) @app.get("/dashboard", response_class=HTMLResponse) async def dashboard(): """Live dashboard showing phone stream, detections, and mission state.""" return DASHBOARD_HTML DASHBOARD_HTML = """ Drone Control Dashboard