# Project export: Mira This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: AI eldercare assistant that reconstructs 3D scenes, localizes lost objects, and alerts caregivers—all through voice and vision. - Devpost: https://devpost.com/software/mira-w65b0a - GitHub: https://github.com/nathanjzhao/treehacks2026 - Video: https://www.youtube.com/embed/UdgzCapZGrc?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Most Impactful; [OpenAI] Artificial Intelligence Track ([1st] Lunch with OpenAI engineers at the office + 1 year of ChatGPT Pro [2nd] 1 year of ChatGPT Pro [3rd] OpenAI swag)) - Team: 5 GitHub contributor(s) — nathanjzhao (32 commits), Victor Chen (20 commits), Madhuhaas Gottimukkala (14 commits), Claude Opus 4.6 (7 commits), Antonio Alonso-Stepanov (6 commits) ## Devpost submission (written by the team) ### Inspiration Three of us have grandparents with Alzheimer's or dementia. If you've been around it, you know the loop: "Have you seen my pills?" five times in an afternoon. It's not just forgetting. It's the loss of autonomy, the slow erosion of someone's confidence that they can manage their own life. Several of us have backgrounds in robotics, simulation infrastructure, manipulation policies, and 3D perception. Across the team we'd been exploring XR, lab automation, and healthcare. We considered other directions: smart glasses for blind navigation, lab automation assistants. But we kept coming back to this: we all have moments where we'd kill for a recording of our lives. Where did I put my keys? Did I already take that pill? Looking at the tech available right now (real-time 3D reconstruction, visual foundation models, tool-using LLMs, consumer AR glasses), we realized this is the most futuristic thing we could actually build in a weekend. Many dementia patients refuse traditional care, insisting family handle everything, which puts enormous strain on people who can't be present 24/7. Mira is designed for this gap: asynchronous monitoring without pulling out a phone, assistance at 3 AM, and a caregiver dashboard that keeps family informed. For someone with Alzheimer's, a searchable spatial memory isn't a convenience. It's dignity. So we built Mira: an AI companion on smart glasses that understands your home in 3D, and when you say "where are my pills?", actually finds them. What It Does Mira is a voice-first AI assistant for dementia patients that combines 3D scene understanding with medical knowledge and caregiver alerts. It runs on Ray-Ban Meta glasses. For the patient: "Where's my pill bottle?" Mira searches a 3D reconstruction of the room (built from a short video walkthrough), locates the object, and flies the 3D viewer to it. "What medications do I take and when?" Pulls from the patient's health record, cross-references clinical guidelines, and returns evidence-graded answers with every claim tagged to its source (PubMed, NIH, clinical databases). "Help, I've fallen." The caregiver gets an email within seconds with full context: the event, actions leading up to it, and the patient's recent activity timeline. For the caregiver: Real-time dashboard showing health summaries, a live event timeline, and emergency alerts. Updates are instant with zero polling, so the caregiver always sees the latest state without refreshing. Email escalation for emergencies so they don't need to be actively watching the dashboard. When something urgent happens, they get full context in their inbox: what the patient said, what Mira did, and the recent activity leading up to it. Every interaction logged as an immutable event for audit. Questions asked, tools called, alerts triggered, all timestamped and searchable. No menus, no apps, no screens to read. The patient just talks. How We Built It 3D Scene. A video walkthrough gets reconstructed into a dense 3D point cloud via structure-from-motion on Modal A100 GPUs. Grounding DINO detects objects per frame, and each detection gets backprojected into 3D using depth maps and camera poses. Detections of the same object across different views are merged by geometric overlap and CLIP similarity, producing a 3D Scene Object Graph mapping object labels to real-world coordinates. Live Pose Tracking. The glasses' video stream feeds DPVO for continuous 6DoF pose tracking. DPVO is fast but drifts over time, so after a few minutes your estimated position can be meters off. HLoc re-localization periodically anchors the pose against the pre-built 3D map, correcting drift. Voice Pipeline. Ray-Ban Meta SDK only exposes camera and mic to Android, so glasses connect to an Android phone first. Bluetooth audio to phone, JPEG frames over hotspot to laptop, RTMP to media server, Tailscale for tunneling. An absurd number of hops for "listen to a sentence." It works. Agent. LLM agent (OpenRouter) with function-calling tools: spatial navigation (3D scene graph + live pose), patient info (OpenEvidence/FHIR), medication lookup (Perplexity Sonar with evidence grading). Whisper for STT, OpenAI TTS for response. Escalation. Emergencies trigger an email to the caregiver with full context. Works alongside the real-time dashboard. Real-time. Supabase Postgres with Realtime subscriptions. Every interaction is an immutable event. Dashboard updates via postgres_changes, zero polling. Privacy. Records are based off of FHIR R4 and are de-identified before reaching any LLM. Age ranges and condition names only, never raw identifiers. Challenges We Ran Into 3D Orientation. Point clouds don't know which way is up. A room can render upside-down and the geometry is equally valid. We built a Gemini-based detector that renders three candidate rotations and asks which looks like an actual room. Inelegant. Works. Stanford WiFi. Stanford's network blocks a lot of traffic. Our streaming pipeline kept dying because the university WiFi was dropping connections. Tailscale mesh VPN was the only thing that reliably punched through. Browser-side ML. Running YOLOv8n via ONNX Runtime Web for the patient experience required WASM webpack config and render loop optimization. Smooth real-time detection overlays in a browser are harder than they sound. Modal at Hackathon Speed. Multiple heavy models (Grounding DINO, DPVO, depth estimation, 3D scene reconstruction) on Modal A100 GPUs. Every new model or dependency change meant rebuilding containers with complex dependency trees. A lot of pure engineering to get latency down and orchestrate everything under time pressure. Accomplishments We're Proud Of End-to-end voice to 3D object localization in under ten seconds. Complete 3D scene reconstruction from casual phone video on Modal GPUs. The point clouds came out detailed enough to identify individual objects on shelves. Seeing the first reconstruction was one of those moments where the whole team gathered around a screen. Real-time event propagation to the caregiver dashboard plus email escalation. Zero polling, instant updates, every interaction logged. Evidence-graded medical citations where every answer is tagged with source and confidence level. Most AI assistants just give you an answer. Mira tells you why you should or shouldn't trust it. Patient records de-identified before reaching any LLM. The model never sees a name, date of birth, or identifying information. What We Learned Don't ask a foundation model to do geometry. We tried getting Gemini to predict 3D coordinates directly. Models hallucinate spatial information. Let the model do perception in 2D, handle the 3D math yourself with projection matrices and known geometry. Composing specialist models beats monoliths. Our pipeline chains Whisper, Gemini, Grounding DINO, depth estimation, visual odometry, and more. Same argument behind Grounded SAM: composition is more flexible than one unified model. Add a capability, plug in a specialist. Nothing retrains. Voice-first design changes everything. We prototyped with a touchscreen. Then we thought about using it with low vision, tremor, and cognitive impairment. Scrolling, tapping a 44px button, parsing a settings menu, all become walls. When your user is 80, the only interface that works is speech. Consumer AR hardware isn't there yet. Ray-Ban Metas have no IMU data exposed to developers, no depth sensor, SDK is Android-only. Meta seems to be deliberately restricting developer access because they want to ship their own products on these features first. No display control, no raw sensor data. We hit the ceiling quickly. What's Next for Mira Continuous scene updates. The 3D model is currently from a one-time walkthrough. Objects move. The system should update incrementally as the resident goes about their day. Predictive object tracking. If the event log shows Mrs. Chen leaves her glasses in the fridge every Tuesday, the system should learn that and check there first. Dedicated sensor hardware for fall detection. Ray-Bans don't expose IMU data, so fall detection needs external hardware. A wearable with accelerometer and gyroscope, combined with 3D scene context, could distinguish a real fall from sitting down. Remote monitoring and health device integration. Cameras in common areas, Apple Watch vitals, other health peripherals feeding into the same event stream. Scaling point clouds. One room works. A full facility with hallways, common areas, and dozens of rooms requires procedural rendering and level-of-detail management for very large point clouds, similar to what Foxglove does for robotics data. ## README (from the GitHub repository) # Mira — TreeHacks 2026 **An AI companion on smart glasses that understands your home in 3D.** When a dementia patient says "where are my pills?", Mira actually finds them. Voice-first assisted living platform combining 3D scene understanding, medical knowledge, and real-time caregiver alerts. Runs on Ray-Ban Meta glasses. **Tech Stack:** Next.js · React · Tailwind CSS · Supabase (Postgres + Realtime) · OpenRouter · Whisper STT · OpenAI TTS · Modal (A100 GPU) · Viser · ONNX Runtime Web · Python · FastAPI ## System Architecture ## Inspiration Three of us have grandparents with Alzheimer's or dementia. If you've been around it, you know the loop: "Have you seen my pills?" five times in an afternoon. It's not just forgetting — it's the loss of autonomy, the slow erosion of someone's confidence that they can manage their own life. Many dementia patients refuse traditional care, insisting family handle everything, which puts enormous strain on people who can't be present 24/7. Mira is designed for this gap: asynchronous monitoring without pulling out a phone, assistance at 3 AM, and a caregiver dashboard that keeps family informed. For someone with Alzheimer's, a searchable spatial memory isn't a convenience. It's dignity. ## Modules ### 3D Reconstruction | Module | What it does | |--------|-------------| | **[scenegraph/](scenegraph/)** | Video → 3D scene graph. End-to-end pipeline on Modal (A100): structure-from-motion, Grounding DINO object detection per frame, backprojection into 3D via depth maps + camera poses, geometric overlap + CLIP similarity merging into a Scene Object Graph. See [scenegraph/README.md](scenegraph/README.md). | | **[reconstruction/](reconstruction/)** | Dense 3D reconstruction — video → dense depth, camera poses, point clouds. | ### Camera Localization | Module | What it does | |--------|-------------| | **[hloc_localization/](hloc_localization/)** | Visual localization via SuperPoint + LightGlue + PnP. Builds an SfM reference map from video (Modal GPU), then localizes new frames against it in 6DoF. Live pose tracking uses DPVO for continuous 6DoF with periodic HLoc re-localization to correct drift. | ### Object Localization | Module | What it does | |--------|-------------| | **[segmentation/](segmentation/)** | Image + language query → object segmentation. Open-vocabulary detection and mask generation. | | **[depthanything/](depthanything/)** | Monocular depth estimation per frame. Used to get depth of segmented objects — median/trimmed mean since mask edges bleed into background. | ### Apps & Services | Module | What it does | |--------|-------------| | **[explorer/](explorer/)** | Interactive 3D point cloud viewer (Viser + FastAPI). Natural language query → Gemini multi-view detection → raycast into point cloud → animated camera fly-to. Auto-detects scene orientation via Gemini rotation voting. | | **[mira-chat/](mira-chat/)** | Full-stack Next.js app — resident voice/text chat (LLM + function calling for spatial nav, medical Q&A, medication lookup), supervisor dashboard with real-time event timeline, caregiver escalation via email. Supabase Realtime for zero-polling updates. See [mira-chat/README.md](mira-chat/README.md). | | **[android/](android/)** | Android client app — receives Bluetooth audio + JPEG frames from Ray-Ban Meta glasses, bridges to backend via Tailscale. | | **[securitycam/](securitycam/)** | Security camera streaming — mediamtx config for RTMP/RTSP/HLS/WebRTC ingestion. | | **[vic-backend/](vic-backend/)** | Backend services. | ## Team | Member | Contributions | |--------|--------------| | **Nathan** | ML pipeline & GPU compute on Modal — object localization, 3D scene reconstruction, camera pose tracking with visual odometry. Security cam integration, point cloud viewer. | | **Victor** | Streaming video and audio from the Ray-Ban Meta SDK, linking to frontend, delivering audio feedback back to the user. | | **Madhuhaas** | Agentic system and tool-calling pipeline — pulling patient info, web search for medication lookup. Frontend development. | | **Antonio** | Online object localization, animated camera fly-to in the 3D viewer, high-level pitch direction, demo video editing. | ## What's Next - **Continuous scene updates.** The 3D model is currently from a one-time walkthrough. Objects move. The system should update incrementally as the resident goes about their day. - **Predictive object tracking.** If the event log shows someone leaves their glasses in the fridge every Tuesday, the system should learn that and check there first. - **Dedicated sensor hardware.** Ray-Bans don't expose IMU data, so fall detection needs external hardware — a wearable with accelerometer and gyroscope combined with 3D scene context. - **Health device integration.** Apple Watch vitals, cameras in common areas, other health peripherals feeding into the same event stream. - **Scaling point clouds.** One room works. A full facility requires procedural rendering and level-of-detail management for very large point clouds. ## Detected evidence (automated analysis) Indexed codebase: 164 recognized source files, 1114 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Kotlin (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - SQL (language) — detected in the code - Supabase (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code - PostgreSQL (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 192) ``` .gitignore android/.gitignore android/app/.gitignore android/app/build.gradle.kts android/app/proguard-rules.pro android/app/sample.keystore android/app/src/androidTest/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/InstrumentationTest.kt android/app/src/main/AndroidManifest.xml android/app/src/main/assets/Hey-Mira_en_android_v4_0_0.ppn android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/audio/AudioCaptureState.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/audio/AudioPreferencesDataStore.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/audio/AudioRecordingManager.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/audio/AudioRecordingViewModel.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/audio/AudioStreamDestination.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/audio/BluetoothScoManager.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/audio/MiraChatService.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/audio/WhisperTranscriptionService.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/MainActivity.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/mockdevicekit/MockDeviceKitUiState.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/mockdevicekit/MockDeviceKitViewModel.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/stream/StreamUiState.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/stream/StreamViewModel.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/streaming/CloudStreamDestination.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/streaming/ComputerStreamDestination.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/streaming/StreamingConfiguration.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/streaming/StreamingLogger.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/streaming/StreamingPreferencesDataStore.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/streaming/VideoStreamingManager.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/AppColor.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/AudioSettingsScreen.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/CameraAccessScaffold.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/CircleButton.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/DebugConsoleScreen.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/HomeScreen.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/MockDeviceKitScreen.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/NonStreamScreen.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/SharePhotoDialog.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/StreamingSettingsScreen.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/StreamScreen.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/ui/SwitchButton.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/wearables/WearablesUiState.kt android/app/src/main/java/com/meta/wearable/dat/externalsampleapps/cameraaccess/wearables/WearablesViewModel.kt android/app/src/main/res/drawable/camera_access_icon.xml android/app/src/main/res/drawable/camera_icon.xml android/app/src/main/res/drawable/hourglass_icon.xml android/app/src/main/res/drawable/smart_glasses_icon.xml android/app/src/main/res/drawable/sound_icon.xml android/app/src/main/res/drawable/tap_icon.xml android/app/src/main/res/drawable/timer_icon.xml android/app/src/main/res/drawable/video_icon.xml android/app/src/main/res/drawable/walking_icon.xml android/app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml android/app/src/main/res/values/strings.xml android/app/src/main/res/values/themes.xml android/app/src/main/res/xml/file_paths.xml android/app/src/main/res/xml/network_security_config.xml android/build.gradle.kts android/gradle.properties android/gradle/libs.versions.toml android/gradle/wrapper/gradle-wrapper.properties android/gradlew android/gradlew.bat android/IMPLEMENTATION_SUMMARY.md android/QUICKSTART.md android/README.md android/settings.gradle.kts android/STREAMING_README.md android/test_receiver.py depthanything/.modalignore depthanything/app.py depthanything/README.md depthanything/video_app.py depthanything/viewer.py devpost.md explorer/.gitignore explorer/camera_utils.py explorer/gemini_client.py explorer/object_finder.py explorer/openrouter_client.py explorer/requirements.txt explorer/templates/index.html explorer/viewer.py hloc_localization/__init__.py hloc_localization/backend/__init__.py hloc_localization/backend/app.py hloc_localization/backend/benchmark_dpvo.py hloc_localization/backend/benchmark.py hloc_localization/backend/debug_localize.py hloc_localization/backend/dpvo_app.py hloc_localization/backend/dpvo_server.py hloc_localization/backend/plot_trajectory.py hloc_localization/backend/run_batch.py hloc_localization/backend/server.py hloc_localization/backend/visualize_poses.py hloc_localization/frontend/__init__.py hloc_localization/frontend/dpvo_viewer.py hloc_localization/frontend/requirements.txt hloc_localization/frontend/templates/dpvo_index.html hloc_localization/frontend/templates/index.html hloc_localization/frontend/view_dpvo.py hloc_localization/frontend/view_trajectory.py hloc_localization/frontend/viewer.py hloc_localization/pose_viewer.py hloc_localization/README.md mira-chat/.gitignore mira-chat/app/api/chat/route.ts mira-chat/app/api/escalate/route.ts mira-chat/app/api/events/route.ts mira-chat/app/api/objects/request/route.ts mira-chat/app/api/objects/update/route.ts mira-chat/app/api/patients/route.ts mira-chat/app/api/stream/frame/jpeg/route.ts mira-chat/app/api/stream/frame/route.ts mira-chat/app/api/stream/localize/route.ts mira-chat/app/api/stream/mjpeg/route.ts mira-chat/app/api/stream/ws/route.ts mira-chat/app/api/voice/transcribe/route.ts mira-chat/app/api/voice/tts/route.ts mira-chat/app/components/CitationCard.tsx mira-chat/app/dashboard/components/AlertBanner.tsx [72 more files omitted for size] ``` ### Dependencies - explorer/requirements.txt: fastapi, google-genai, httpx, jinja2, numpy, Pillow, trimesh, uvicorn[standard], viser - hloc_localization/frontend/requirements.txt: fastapi, httpx, jinja2, numpy, trimesh, uvicorn[standard], viser - mira-chat/package.json: @supabase/supabase-js@^2.95.3, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, @types/ws@^8.18.1, dotenv@^17.3.1, eslint@^9, eslint-config-next@16.1.6, next@16.1.6, onnxruntime-web@^1.24.1, react@19.2.3, react-dom@19.2.3, streamdown@^2.2.0, tailwindcss@^4, typescript@^5, ws@^8.19.0 - reconstruction/requirements.txt: fastapi, jinja2, numpy, trimesh, uvicorn[standard], viser@==0.2.23 - scenegraph/requirements.txt: fastapi, jinja2, numpy, open3d, trimesh, uvicorn[standard], viser - vic-backend/requirements.txt: fastapi@==0.110.0, google-cloud-firestore@==2.16.0, google-cloud-storage@==2.14.0, google-cloud-texttospeech@==2.16.0, google-generativeai@==0.3.2, httpx@==0.27.0, python-multipart@==0.0.9, uvicorn[standard]@==0.27.1 ### Recent commits (newest first) - cleanup - . - viewers + readme - just live viz update so doesn't have cat update - latest - again - readme - devpost again - Update architecture video with showcase + data flow - Update system architecture video in README - devpost new - remove sam3 - Rename modules: openfungraph→scenegraph, groundedsam2→segmentation, mapanything→reconstruction - Stream video to two IPs - Fix timing of showing the tool calls - fix YOLO: add model, support img element for MJPEG stream - Final changes to main - Revert "Final Improvements to Video Streaming" - Merge branch 'main' of https://github.com/nathanjzhao/treehacks2026 - switch TTS to gpt-4o-mini-tts marin voice ## Key source files (fetched from GitHub, selected and truncated for size) ### VIEWERS.md ```markdown # Viewers All viewers require the conda base environment: ```bash source ~/miniforge3/etc/profile.d/conda.sh && conda activate base ``` ## Viewer Summary | Port | Viewer | Description | |------|--------|-------------| | 8890 | Locate | 3D object finder — GLB point cloud + raycasted object positions | | 8891 | Segmentation | Side-by-side source / tracking / masked depth video | | 8892 | Depth | Side-by-side source / depth video | | 8893 | MapAnything | 3D point cloud browser (viser) | | 8895 | OpenFunGraph | 3D scene graph with objects, parts, edges (viser) | --- ## 1. MapAnything Viewer (port 8893) **What it shows:** Interactive 3D point cloud reconstructions with camera frustums. ### Generate data ```bash modal run reconstruction/app.py --video-path data/IMG_4717.MOV ``` **Outputs:** `data/IMG_4717.glb` (move to `data/mapanything/`) ### Run viewer ```bash python3 reconstruction/viewer.py data/mapanything/ --downsample 80 --port 8893 --viser-port 8894 ``` **Inputs:** - Directory of `.glb` files (dropdown to switch between them) - `--downsample`: point skip factor (higher = fewer points, faster load) --- ## 2. Depth Viewer (port 8892) **What it shows:** Side-by-side original video and depth estimation. ### Generate data (per-frame) ```bash modal run depthanything/app.py --video-path data/IMG_4718.MOV ``` **Outputs:** `depthanything/examples/{stem}_depth.mp4` ### Generate data (temporally consistent) ```bash modal run depthanything/video_app.py --video-path data/IMG_4717.MOV ``` **Outputs:** `depthanything/examples/{stem}_depth.mp4` (same format, replaces per-frame) ### Run viewer ```bash python3 depthanything/viewer.py depthanything/examples/ --source-dir data/ --port 8892 ``` **Inputs:** - Directory of `*_depth.mp4` files (dropdown to switch) - `--source-dir`: where to find matching source videos (e.g. `data/`) - Without `--source-dir`, both panels show depth --- ## 3. Segmentation Viewer (port 8891) **What it shows:** Three panels — original source, SAM2 tracking overlay, masked depth. ### Generate tracking data ```bash modal run segmentation/app.py --video-path data/IMG_4723.MOV --text-prompt "person. chair." ``` **Outputs** to `data/segmentation/`: - `{stem}_tracked.mp4` — video with tracking overlay - `{stem}_detections.json` — per-frame bounding boxes ### Generate depth + segmentation composite ```bash modal run segmentation/depth_app.py --video-path data/IMG_4723.MOV --text-prompt "person. chair." ``` **Outputs** to `data/segmentation/`: - `{stem}_masked_depth.mp4` — depth only on segmented objects - `{stem}_composite.mp4` — dimmed full depth + bright object depth + outlines - `{stem}_seg_depth.json` — per-frame detections with depth ### Run viewer ```bash python3 segmentation/seg_viewer.py data/segmentation/ --source-dir data/ --port 8891 ``` **Inputs:** - Directory containing `*_tracked.mp4`, `*_masked_depth.mp4`, `*_composite.mp4` - `--source-dir`: where to find original videos - Defaults to IMG_4723 [truncated — 2558 more characters] ``` ### devpost.md ```markdown # Mira ## Inspiration Three of us have grandparents with Alzheimer's or dementia. If you've been around it, you know the loop: "Have you seen my pills?" five times in an afternoon. It's not just forgetting. It's the loss of autonomy, the slow erosion of someone's confidence that they can manage their own life. - Several of us have backgrounds in robotics, simulation infrastructure, manipulation policies, and 3D perception. Across the team we'd been exploring XR, lab automation, and healthcare. - We considered other directions: smart glasses for blind navigation, lab automation assistants. But we kept coming back to this: we all have moments where we'd kill for a recording of our lives. Where did I put my keys? Did I already take that pill? - Looking at the tech available right now (real-time 3D reconstruction, visual foundation models, tool-using LLMs, consumer AR glasses), we realized this is the most futuristic thing we could actually build in a weekend. - Many dementia patients refuse traditional care, insisting family handle everything, which puts enormous strain on people who can't be present 24/7. Mira is designed for this gap: asynchronous monitoring without pulling out a phone, assistance at 3 AM, and a caregiver dashboard that keeps family informed. For someone with Alzheimer's, a searchable spatial memory isn't a convenience. It's dignity. So we built Mira: an AI companion on smart glasses that understands your home in 3D, and when you say "where are my pills?", actually finds them. ## What It Does Mira is a voice-first AI assistant for dementia patients that combines 3D scene understanding with medical knowledge and caregiver alerts. It runs on Ray-Ban Meta glasses. **For the patient:** - *"Where's my pill bottle?"* Mira searches a 3D reconstruction of the room (built from a short video walkthrough), locates the object, and flies the 3D viewer to it. - *"What medications do I take and when?"* Pulls from the patient's health record, cross-references clinical guidelines, and returns evidence-graded answers with every claim tagged to its source (PubMed, NIH, clinical databases). - *"Help, I've fallen."* The caregiver gets an email within seconds with full context: the event, actions leading up to it, and the patient's recent activity timeline. **For the caregiver:** - Real-time dashboard showing health summaries, a live event timeline, and emergency alerts. Updates are instant with zero polling, so the caregiver always sees the latest state without refreshing. - Email escalation for emergencies so they don't need to be actively watching the dashboard. When something urgent happens, they get full context in their inbox: what the patient said, what Mira did, and the recent activity leading up to it. - Every interaction logged as an immutable event for audit. Questions asked, tools called, alerts triggered, all timestamped and searchable. No menus, no apps, no screens to read. The patient just talks. ## How We Built It - **3D Scen [truncated — 6172 more characters] ``` ### scenegraph/requirements.txt ``` numpy trimesh viser fastapi uvicorn[standard] jinja2 open3d ``` ### reconstruction/requirements.txt ``` numpy trimesh viser==0.2.23 fastapi uvicorn[standard] jinja2 ``` ### explorer/requirements.txt ``` numpy trimesh viser fastapi uvicorn[standard] jinja2 httpx Pillow google-genai ``` ### vic-backend/requirements.txt ``` fastapi==0.110.0 uvicorn[standard]==0.27.1 google-cloud-firestore==2.16.0 google-cloud-storage==2.14.0 google-cloud-texttospeech==2.16.0 google-generativeai==0.3.2 python-multipart==0.0.9 httpx==0.27.0 ``` ### vic-backend/Dockerfile ``` FROM python:3.11-slim ENV PYTHONDONTWRITEBYTECODE=1 \ PYTHONUNBUFFERED=1 WORKDIR /app COPY requirements.txt /app/requirements.txt RUN pip install --no-cache-dir -r /app/requirements.txt COPY main.py /app/main.py ENV PORT=8080 EXPOSE 8080 CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"] ``` ### mira-chat/package.json ``` { "name": "mira", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --webpack", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@supabase/supabase-js": "^2.95.3", "dotenv": "^17.3.1", "next": "16.1.6", "onnxruntime-web": "^1.24.1", "react": "19.2.3", "react-dom": "19.2.3", "streamdown": "^2.2.0", "ws": "^8.19.0" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "@types/ws": "^8.18.1", "eslint": "^9", "eslint-config-next": "16.1.6", "tailwindcss": "^4", "typescript": "^5" } } ``` ### hloc_localization/frontend/requirements.txt ``` numpy trimesh viser fastapi uvicorn[standard] jinja2 httpx ``` ### depthanything/app.py ```python """ Depth Anything V2 on Modal — Monocular depth estimation. Runs per-frame depth estimation on video using an A100. Returns a colorized depth video and raw 16-bit depth frames packed as bytes. Deploy: modal deploy depthanything/app.py Dev: modal serve depthanything/app.py Video: modal run depthanything/app.py --video-path ~/video.mp4 """ import pathlib import modal app = modal.App("depthanything") cuda_version = "12.4.0" flavor = "devel" os_version = "ubuntu22.04" tag = f"{cuda_version}-{flavor}-{os_version}" ENCODER = "vitl" MODEL_CONFIGS = { "vits": {"encoder": "vits", "features": 64, "out_channels": [48, 96, 192, 384]}, "vitb": {"encoder": "vitb", "features": 128, "out_channels": [96, 192, 384, 768]}, "vitl": {"encoder": "vitl", "features": 256, "out_channels": [256, 512, 1024, 1024]}, } HF_WEIGHT_URL = ( "https://huggingface.co/depth-anything/Depth-Anything-V2-Large" "/resolve/main/depth_anything_v2_vitl.pth" ) depthanything_image = ( modal.Image.from_registry(f"nvidia/cuda:{tag}", add_python="3.11") .apt_install("git", "ffmpeg", "libgl1", "libglib2.0-0") .pip_install( "torch==2.3.1", "torchvision", extra_index_url="https://download.pytorch.org/whl/cu124", ) .pip_install( "numpy<2", "Pillow", "huggingface_hub", "opencv-python", "tqdm", "scipy", "matplotlib", ) .run_commands( "git clone https://github.com/DepthAnything/Depth-Anything-V2.git /opt/DepthAnythingV2", ) .env({ "HF_HOME": "/opt/hf_cache", "TORCH_HOME": "/opt/torch_cache", }) # Pre-download model weights into the image .run_commands( "mkdir -p /opt/DepthAnythingV2/checkpoints && " f"python -c \"" f"from huggingface_hub import hf_hub_download; " f"hf_hub_download(" f" repo_id='depth-anything/Depth-Anything-V2-Large'," f" filename='depth_anything_v2_vitl.pth'," f" local_dir='/opt/DepthAnythingV2/checkpoints'" f")\"", ) # Warm up model on GPU to validate it loads .run_commands( "python -c \"" "import sys; sys.path.insert(0, '/opt/DepthAnythingV2'); " "import torch; " "from depth_anything_v2.dpt import DepthAnythingV2; " "model = DepthAnythingV2(encoder='vitl', features=256, out_channels=[256, 512, 1024, 1024]); " "model.load_state_dict(torch.load('/opt/DepthAnythingV2/checkpoints/depth_anything_v2_vitl.pth', map_location='cpu')); " "model = model.to('cuda').eval(); " "print('Depth Anything V2 vitl loaded OK')\"", gpu="any", ) ) with depthanything_image.imports(): import os import sys # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- def _extract_frames(video_bytes: bytes, tmpdir: str, target_fps: int): """Extract frames from video bytes at target FPS. Returns (image_paths, source_fps, frame_w, frame_h).""" import cv2 video_path = os.path.join(tmpdir, "input.mp4") with open(video_path, "wb") as f: f.write(video_bytes) images_dir = os.path.join(tmpdir, "images") os.makedirs(images_dir) vs = cv2.VideoCapture(video_path) source_fps = vs.get(cv2.CAP_PROP_FPS) frame_w = int(vs.get(cv2.CAP_PROP_FRAME_WIDTH)) frame_h = int(vs.get(cv2.CAP_PROP_FRAME_HEIGHT)) frame_interval = max(1, int(source_fps / target_fps)) image_paths = [] count = 0 frame_num = 0 while True: gotit, frame = vs.read() if not gotit: break count += 1 if count % frame_interval == 0: path = os.path.join(images_dir, f"{frame_num:06d}.png") cv2.imwrite(path, frame) image_paths.append(path) frame_num += 1 vs.release() print(f"Extracted {len(image_paths)} frames from {count} total (interval={frame_interval})") return image_paths, source_fps, frame_w, frame_h def _load_model(encoder: str = ENCODER): """Load the Depth Anything V2 model (cached in the container image).""" import torch sys.path.insert(0, "/opt/DepthAnythingV2") from depth_anything_v2.dpt import DepthAnythingV2 cfg = MODEL_CONFIGS[encoder] model = DepthAnythingV2(**cfg) ckpt = f"/opt/DepthAnythingV2/checkpoints/depth_anything_v2_{encoder}.pth" model.load_state_dict(torch.load(ckpt, map_location="cpu")) model = model.to("cuda").eval() print(f"Loaded Depth Anything V2 ({encoder})") return model # --------------------------------------------------------------------------- # Batch video depth estimation # --------------------------------------------------------------------------- @app.function( image=depthanything_image, gpu="A100", timeout=1800, memory=32768, ) def predict_video( video_bytes: bytes, target_fps: int = 6, input_size: int = 518, grayscale: bool = False, ) -> dict: """ Run Depth Anything V2 on a video. Returns colorized depth MP4 and raw 16-bit depth frames as a packed numpy array. Args: video_bytes: Raw video file content. target_fps: Frames to extract per second. input_size: Inference resolution (higher = finer detail, slower). grayscale: If True, output grayscale depth instead of colormap. """ import tempfile import cv2 import numpy as np import torch model = _load_model() with tempfile.TemporaryDirectory() as tmpdir: image_paths, source_fps, frame_w, frame_h = _extract_frames( video_bytes, tmpdir, target_fps ) if len(image_paths) == 0: raise ValueError("No frames extracted from video") out_fps = min(target_fps, source_fps) # Output video out_path = os.path.join(tmpdir, "depth_vis.mp4") fourcc = cv2.VideoWriter_fourcc(*"mp4 [truncated — 5909 more characters] ``` [143 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]