# Project export: Miru This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Indoor navigation for the visually impaired, made easy. - Devpost: https://devpost.com/software/mapdash - GitHub: https://github.com/flatypus/miru - Video: https://www.youtube.com/embed/kjCsRmlAYiY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Vespa.ai: Best Hack Using a Vision Language Model ($500 Amazon gift card per team member)) - Team: 2 GitHub contributor(s) — flatypus [mac] (12 commits), Michael Yu (6 commits) ## Devpost submission (written by the team) ### Overview The ### Inspiration More than 20 million Americans experience visual impairment, with 1 million classified as legally blind. For many, navigating indoor spaces independently is a significant challenge, limiting daily activities and self-sufficiency. While outdoor navigation solutions—powered by GPS and mapping services like Google Maps—are widely available, indoor navigation remains a relatively unsolved problem due to the lack of reliable spatial data and GPS limitations. Given that people spend most of their time indoors, the status quo is the reverse of what we need. We set out to create an intuitive, lightweight, and accessible indoor navigation aid for the visually impaired. With the rise of advanced video capture technology—such as the Apple Vision Pro and Meta Ray-Ban glasses—we saw an opportunity to harness computer vision and wearable technology to make indoor spaces more navigable for those with visual impairments. What Miru Does Miru is an indoor navigation tool for the blind, effectively serving as an indoor Google Maps. Miru provides real-time, step-by-step guidance to help visually impaired users navigate complex indoor environments with ease. A picture of the floor plan of the building—readily available on fire escape plans near elevators or fire extinguishers—is uploaded to Miru, which automatically generates waypoints and pathfinding routes. The user wears a pair of Meta Ray-Ban smart glasses for real-time visual input and localization, and a servo-fitted belt provides gentle haptic feedback to guide movement, ensuring a non-intrusive and intuitive experience. By integrating wearable technology with intelligent navigation, Miru empowers visually impaired individuals to move confidently and independently in indoor spaces. How We Built It To convert fire escape plans into functional map data, we implemented Canny edge detection, extracting key architectural features such as walls and doors. Analyzing the map, we then generated waypoints, followed by A* pathfinding to compute optimal navigation routes. Live video footage was acquired through Meta’s Ray-Ban glasses. We tried to use streaming through Instagram to acquire the footage, but realized there was around 30 seconds of latency which wouldn't work. We found that streaming footage to WhatsApp provided data much faster, with less than 1 second of latency. We also had the challenge of extending sensory capabilities to the visually impaired; in particular, a system that could reliably provide direction. We ended up deciding on creating a wearable belt, created using servos as a tactile feedback mechanism, and it has five servos capable of generating directional instructions (left, right, forward, left-forward, right-forward). As the user turns around, we use our orientation mechanisms to provide feedback in the correct direction. Localization—identifying the user's real-time position indoors—was one of our most challenging tasks. Specifically, because it is impossible to access GPS, no indoor navigation system that relies on GPS will work. Our innovative approach involved two key steps: First, we generated a reference dataset of approximately 1,500 geotagged images in the building and embedded them using CLIP. We stored the vectors in an database using Vespa AI. We then embedded the live footage from the Ray-Ban and queried for similar images in the Vespa vector database. Then through the most similar images we averaged the coordinate metadata that we got from those images. Most importantly was filtering, after gathering these coordinates we compared it to the data we already had and would throw out any values that were greater than 5.0 meters away from the current pose. As much as we tried to reduce noise, there will always be noise in a system and filtering is something we can always work to improve. Besides localization, determining orientation was also crucial for our project’s success. We built our own iOS compass app that could interact with our websocket system using the iOS compass data to determine orientation and integrated it with a wearable utility built to guide user interaction. We found a bug in the iOS compass app, where the compass would drift randomly for no apparent reason. This resulted in even more pain as it meant that we had one less source of truth to trust. Challenges We Ran Into We immediately ran into latency issues with Meta’s Ray-Ban glasses. Previous projects we’d looked at involving Meta Ray-Ban glasses streamed the Ray-Ban footage to Instagram Live, because Meta Ray-Ban glasses are incapable of natively streaming to a laptop. Instagram Live, however, had a stream delay of ~30 seconds. We resolved this issue by streaming to WhatsApp and mirroring the phone screen onto a laptop, providing latency of <1 second. Parsing through vector embeddings via OpenAI’s vision model meant high latency, which was undesirable for real-time use. As a result, we opted to use Vespa AI’s API for efficient updates and queries. The most difficult challenge we ran into was localization. Determining where a user was proved extremely challenging, and we opted to use computer vision and similarity detection to determine the location of the user. Orientation was also an issue. Initially, we tried using tri-sensor IMU fusion, but sensor drift rendered IMUs inaccurate. We managed to pull iOS compass data as an effective and simple method of determining user orientation. Accomplishments We’re Proud Of We’re extremely proud of the progress we were able to make in just 36 hours. Prior to this hackathon, none of us had extensive experience with remote sensing, and we’re proud that we were able to develop a real-time, GPS-free indoor navigation system that required only a floor plan and live video feed. This serves as a simple alternative to beacon-based positioning. We’re also happy that we were able to create a functional and intuitive haptic feedback system that allows users to “see” through feel. We think that localization is one of the most difficult challenges within robotics and we think that the new age of AI has so much potential and ability to help robots better understand the world around us. What We Learned We learned how to work extensively with computer vision, vector embeddings, basic hardware, and software integration, all of which were extremely challenging but rewarding to work with. We also learned how to integrate software with hardware for an intuitive and powerful experience. What’s Next for Miru A few ideas we hope to implement: Multi-Floor Path Finding: It’d be cool to direct users to stairwells and elevators and enable travel across more than just one floor. SLAM-Powered Real-Time Mapping: If done effectively, using Simultaneous Localization and Mapping (SLAM) would allow us to dynamically build and update indoor maps without the need for reference images—saving a lot of time. Scaling to Large-Scale Public Spaces: Expanding to airports, malls, hospitals, and transit hubs would enable us to provide invaluable services to the visually impaired in more than just small buildings. We hope that while we had to create our own dataset to query from for the Huang building, often-visited locations like train stations and museums have publically available floor plans and Google maps data that our system can easily apply into. ## README (from the GitHub repository) # 🏠 Miru: Indoor Navigation for the Visually Impaired ## 🏆 Treehacks 2025 - Best Use of VLMs ![miru](https://github.com/user-attachments/assets/b4222674-a848-459b-96d1-9c1edadeba85) ## 🚀 Inspiration Over **20 million Americans** experience visual impairment, with **1 million classified as legally blind**. While GPS-based navigation tools exist for outdoor spaces, indoor navigation remains a major challenge due to **GPS limitations and lack of reliable spatial data**. With the rise of **smart glasses** like Meta Ray-Ban and Apple Vision Pro, we saw an opportunity to **harness wearable technology and computer vision** to empower visually impaired individuals with **independent indoor mobility**. --- ## How It Works **Miru** is an **indoor navigation system** for the visually impaired, providing **real-time, step-by-step guidance** using a combination of **floor plans, wearable smart glasses, and haptic feedback**. 1. **Upload a Floor Plan** - Extracts architectural features using **Canny edge detection**. - Generates **waypoints and navigation paths** using **A\* pathfinding**. 2. **Wear Meta Ray-Ban Smart Glasses** - Streams live video for **real-time localization**. - Uses **WhatsApp streaming** to minimize latency (<1s). 3. **Receive Haptic Feedback via a Servo Belt** - A wearable **servo motor belt** provides tactile feedback to guide movement. - Supports five directional instructions: **left, right, forward, left-forward, right-forward**. --- ## 🛠️ How We Built It ### 🗺 Indoor Mapping & Navigation - **Converted fire escape plans** into navigable maps using **computer vision**. - **Pathfinding** performed with **A* algorithm** for optimal routes. ### 📍 Real-Time Localization - Captured **1,500+ geotagged images** and embedded them using **CLIP**. - Stored in **Vespa AI vector database** for efficient querying. - Filtered out noisy location data (>5m deviation). ### 🎯 Orientation & Feedback - Developed an **iOS compass app** to determine direction. - Integrated a **haptic belt** with servo motors for **non-visual navigation**. --- ## 🚧 Challenges We Faced - **High Latency in Ray-Ban Streaming** - Meta Ray-Ban glasses do **not support native laptop streaming**. - **Solution**: Streamed via **WhatsApp** instead of **Instagram Live**, reducing latency from **30s to <1s**. - **Localization Without GPS** - GPS **does not work indoors**, making navigation extremely difficult. - **Solution**: Used **computer vision and vector search** instead of traditional GPS. - **IMU Sensor Drift for Orientation** - **Solution**: Used iOS **compass data** for more accurate direction sensing. --- ## 🎉 Accomplishments We're Proud Of ✅ Built a **real-time, GPS-free** indoor navigation system in **just 36 hours**. ✅ Created an **intuitive haptic feedback belt** for non-visual navigation. ✅ Successfully integrated **computer vision, vector embeddings, and hardware** into a seamless user experience. --- ## 🔮 What’s Next? 🔹 **Multi-Floor Navigation** – Guide users across **stairwells & elevators**. 🔹 **SLAM-Based Real-Time Mapping** – Dynamically update maps without reference images. 🔹 **Scaling to Public Spaces** – Expand to **airports, malls, hospitals, and transit hubs** using public floor plans. --- ## Devpost: https://devpost.com/software/mapdash ## Detected evidence (automated analysis) Indexed codebase: 43 recognized source files, 87 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (79 of 79) ``` .DS_Store .gitignore .idea/.gitignore .idea/git_toolbox_blame.xml .idea/inspectionProfiles/Project_Default.xml .idea/misc.xml .idea/modules.xml .idea/treehacks.iml .idea/vcs.xml .vscode/settings.json api/.gitignore api/.python-version api/compass.html api/edges.json api/hello.py api/helper.py api/nodes.json api/pyproject.toml api/README.md api/server.py api/treehacks/schemas/doc.sd api/treehacks/search/query-profiles/default.xml api/treehacks/search/query-profiles/types/root.xml api/treehacks/services.xml api/treehackszoning/schemas/doc.sd api/treehackszoning/search/query-profiles/default.xml api/treehackszoning/search/query-profiles/types/root.xml api/treehackszoning/services.xml api/uv.lock compass/compass.xcodeproj/project.pbxproj compass/compass.xcodeproj/project.xcworkspace/contents.xcworkspacedata compass/compass.xcodeproj/project.xcworkspace/xcuserdata/michaelyu.xcuserdatad/UserInterfaceState.xcuserstate compass/compass.xcodeproj/xcuserdata/michaelyu.xcuserdatad/xcschemes/xcschememanagement.plist compass/compass/Assets.xcassets/AccentColor.colorset/Contents.json compass/compass/Assets.xcassets/AppIcon.appiconset/Contents.json compass/compass/Assets.xcassets/Contents.json compass/compass/compassApp.swift compass/compass/ContentView.swift compass/compass/Info.plist compass/compass/Preview Content/Preview Assets.xcassets/Contents.json compass/compassTests/compassTests.swift compass/compassUITests/compassUITests.swift compass/compassUITests/compassUITestsLaunchTests.swift embedded/.DS_Store embedded/embedded.ino README.md requirements.txt site/.gitignore site/app/globals.css site/app/layout.tsx site/app/page.tsx site/bun.lock site/components/LocationMarkers.tsx site/components/Map.tsx site/components/ScaledImageOverlay.tsx site/components/SearchPanel.tsx site/eslint.config.mjs site/next.config.ts site/package.json site/postcss.config.mjs site/README.md site/tailwind.config.ts site/tsconfig.json testing.py TreeHacks-Voice/.gitignore TreeHacks-Voice/.idea/.gitignore TreeHacks-Voice/.idea/git_toolbox_blame.xml TreeHacks-Voice/.idea/inspectionProfiles/profiles_settings.xml TreeHacks-Voice/.idea/misc.xml TreeHacks-Voice/.idea/modules.xml TreeHacks-Voice/.idea/TreeHacks-Voice.iml TreeHacks-Voice/.idea/vcs.xml TreeHacks-Voice/locations.json TreeHacks-Voice/main.py TreeHacks-Voice/stt.py TreeHacks-Voice/system_prompt.txt TreeHacks-Voice/test.py TreeHacks-Voice/tools.py TreeHacks-Voice/tts.py ``` ### Dependencies - api/pyproject.toml: fastapi, pyserial@>=3.5, uvicorn[standard] - requirements.txt: elevenlabs@~=1.51.0, groq@~=0.18.0, keyboard@~=0.13.5, mss@~=10.0.0, numpy@~=2.0.2, openai@~=1.63.0, pillow@~=11.1.0, PyAudio@~=0.2.14, python-dotenv@~=1.0.1, torch@~=2.6.0, transformers@~=4.48.3 - site/package.json: @eslint/eslintrc@^3, @maptiler/leaflet-maptilersdk@^4.0.2, @types/leaflet@^1.9.16, @types/node@^20, @types/react@^19, @types/react-dom@^19, @uidotdev/usehooks@^2.4.1, eslint@^9, eslint-config-next@15.1.7, leaflet@^1.9.4, next@15.1.7, postcss@^8, react@^19.0.0, react-dom@^19.0.0, react-leaflet@^5.0.0, tailwindcss@^3.4.1, typescript@^5 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - done - Update README.md - fixed ignoreg - added compass - done - fix: git ignore - ok - Merge remote-tracking branch 'origin/hi' - hi - final tweaks - added tts - chabnges - Merge branch 'hinson/test' - Vespra (#1) ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` numpy~=2.0.2 mss~=10.0.0 openai~=1.63.0 keyboard~=0.13.5 python-dotenv~=1.0.1 transformers~=4.48.3 pillow~=11.1.0 torch~=2.6.0 elevenlabs~=1.51.0 groq~=0.18.0 PyAudio~=0.2.14 ``` ### api/pyproject.toml ``` [project] name = "api" version = "0.1.0" description = "Add your description here" readme = "README.md" requires-python = ">=3.13" dependencies = [ "fastapi", "pyserial>=3.5", "uvicorn[standard]", ] ``` ### site/package.json ``` { "name": "site", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@maptiler/leaflet-maptilersdk": "^4.0.2", "@uidotdev/usehooks": "^2.4.1", "leaflet": "^1.9.4", "next": "15.1.7", "react": "^19.0.0", "react-dom": "^19.0.0", "react-leaflet": "^5.0.0" }, "devDependencies": { "@eslint/eslintrc": "^3", "@types/leaflet": "^1.9.16", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "15.1.7", "postcss": "^8", "tailwindcss": "^3.4.1", "typescript": "^5" } } ``` ### TreeHacks-Voice/main.py ```python import time import os import json from groq import Groq from stt import SpeechRecognizer from tts import TextToSpeech from tools import find_destination # Load API keys with open("api_keys.txt", "r") as f: keys = {line.split('=')[0].strip(): line.split('=')[1].strip() for line in f.readlines()} groq_api_key = keys.get("GROQ_API_KEY") elevenlabs_api_key = keys.get("ELEVENLABS_API_KEY") # Load system prompt template with open("system_prompt.txt", "r") as f: system_prompt_template = f.read().strip() # Load locations with open("locations.json", "r") as f: locations = json.load(f)["locations"] # Debugging flag DEBUG_MODE = True # Initialize variables current_location = "Terman Library" # Example default location destination = None # Initialize Groq client client = Groq(api_key=groq_api_key) # Initialize STT and TTS recognizer = SpeechRecognizer(api_key=groq_api_key) tts_engine = TextToSpeech(api_key=elevenlabs_api_key) def log_debug(message): """Print debug messages if DEBUG_MODE is enabled.""" if DEBUG_MODE: print(f"[DEBUG] {message}") def format_system_prompt(): """Format the system prompt with the current location and destination.""" dest_text = destination if destination else "still asking user for destination" formatted_prompt = system_prompt_template.format(current_location, dest_text) log_debug(f"Formatted System Prompt:\n{formatted_prompt}") return formatted_prompt def chat_loop(): """Main loop for continuous voice interaction.""" global destination system_prompt = format_system_prompt() conversation_history = [ {"role": "system", "content": system_prompt}, {"role": "assistant", "content": "Hey Michael! Where do you want to go?"} ] tts_engine.speak("Hey Michael! Where do you want to go?") while True: print("Listening for user input...") user_text = recognizer.listen_once() if not user_text: continue log_debug(f"User said: {user_text}") conversation_history.append({"role": "user", "content": user_text}) # Check for destination if destination is None: detected_destination = find_destination(user_text) if detected_destination: destination = detected_destination log_debug(f"Destination set to: {destination}") system_prompt = format_system_prompt() conversation_history[0] = {"role": "system", "content": system_prompt} # Speak fixed messages tts_engine.speak("Alright, can you look around for a few seconds so I can get a sense of our location?") time.sleep(2.5) tts_engine.speak("I think I've found the location. I've sent the instructions to your belt.") continue # Go back to listening log_debug(f"Current Location: {current_location}") log_debug(f"Destination: {destination if destination else 'None'}") # Get AI response completion = client.chat.completions.create( model="llama-3.3-70b-versatile", messages=conversation_history, temperature=1, max_completion_tokens=1024, top_p=1, stream=False ) ai_response = completion.choices[0].message.content.strip() log_debug(f"AI Response: {ai_response}") conversation_history.append({"role": "assistant", "content": ai_response}) # Convert AI response to speech tts_engine.speak(ai_response) if __name__ == "__main__": chat_loop() ``` ### api/server.py ```python from fastapi import FastAPI, WebSocket from fastapi.concurrency import asynccontextmanager from fastapi.middleware.cors import CORSMiddleware from fastapi.websockets import WebSocketState import asyncio import serial degrees: float = 0 last_degrees: float = None ser = serial.Serial('/dev/cu.usbmodem2101', 9600, timeout=1) print(f"Connected to {ser.name}") ser.write(b'Hello, serial port!') async def send_zero(): while True: v = open("value.txt", "r").read() print(f"Sending {v} to serial") ser.write(str(v).encode()) await asyncio.sleep(3) @asynccontextmanager async def lifespan(app: FastAPI): asyncio.create_task(send_zero()) yield ser.close() app = FastAPI(lifespan=lifespan) # Add CORS middleware app.add_middleware( CORSMiddleware, allow_origins=["*"], # Allows all origins allow_credentials=True, allow_methods=["*"], # Allows all methods allow_headers=["*"], # Allows all headers ) # Global variables coords = [37.4280207092758, -122.17424679547551] last_coords = [0, 0] degrees: float = 0 last_degrees: float = None # ser = serial.Serial('/dev/cu.usbmodem1101', 9600, timeout=1) # print(f"Connected to {ser.name}") # ser.write(b'Hello, serial port!') @app.get("/") async def root(): return {"message": "lmfao"} @app.websocket("/ws-for-ios") async def websocket_endpoint_for_ios(websocket: WebSocket): global degrees global coords await websocket.accept() print("IOS connection accepted") try: while True: degrees = await websocket.receive_json() with open("degrees.txt", "w") as f: f.write(str(degrees)) except Exception as e: if websocket.client_state == WebSocketState.CONNECTED: await websocket.close() print(e) print("IOS connection closed") @app.websocket("/ws-for-frontend") async def websocket_endpoint(websocket: WebSocket): global last_degrees global last_coords await websocket.accept() await websocket.send_json({"data": { "degrees": degrees, "coordinates": [0, 0] }}) print("Frontend connection accepted") try: while True: coords = open("./coordinates.txt", "r").read().split() coords = list(map(float, coords)) if coords != last_coords: await websocket.send_json({"data": { "degrees": degrees, "coordinates": coords }}) last_coords = coords if degrees != last_degrees: await websocket.send_json({"data": { "degrees": degrees, "coordinates": coords }}) last_degrees = degrees await asyncio.sleep(0.1) except Exception as e: if websocket.client_state == WebSocketState.CONNECTED: await websocket.close() print(e) print("Frontend connection closed") @app.websocket("/ws-for-buttons") async def websocket_endpoint_buttons(websocket: WebSocket): await websocket.accept() print("Buttons connection accepted") try: while True: message = await websocket.receive_json() print(f"Received message: {message}") with open("value.txt", "w") as f: f.write(str(message["index"])) except Exception as e: if websocket.client_state == WebSocketState.CONNECTED: await websocket.close() print("Buttons connection closed", e) if __name__ == "__main__": import uvicorn uvicorn.run(app, host="0.0.0.0", port=4000) ``` ### site/app/page.tsx ```typescript "use client"; import "leaflet/dist/leaflet.css"; import Map from "@/components/Map"; import L from "leaflet"; const STANFORD = { latitude: 37.42796, longitude: -122.17436, }; export default function Home() { const center = new L.LatLng(STANFORD.latitude, STANFORD.longitude); return ; } ``` ### site/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "Navigation", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### testing.py ```python import cv2 import torch import matplotlib.pyplot as plt cap = cv2.VideoCapture(0) if not cap.isOpened(): print("Error: Could not open camera.") exit() # MiDaS v3 - Large (highest accuracy, slowest inference speed) # model_type = "DPT_Large" # model_type = "DPT_Hybrid" # MiDaS v3 - Hybrid (medium accuracy, medium inference speed) # MiDaS v2.1 - Small (lowest accuracy, highest inference speed) # model_type = "MiDaS_small" model_type = "DPT_Hybrid" midas = torch.hub.load("intel-isl/MiDaS", model_type) device = torch.device("mps") midas.to(device) midas.eval() midas_transforms = torch.hub.load("intel-isl/MiDaS", "transforms") transform = midas_transforms.dpt_transform if model_type == "DPT_Large" or model_type == "DPT_Hybrid" else midas_transforms.small_transform with torch.no_grad(): try: while True: ret, frame = cap.read() if not ret: print("Error: Can't receive frame") break frame = cv2.resize(frame, (640, 480)) img = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) input_batch = transform(img).to(device) prediction = midas(input_batch) prediction = torch.nn.functional.interpolate( prediction.unsqueeze(1), size=img.shape[:2], mode="bicubic", align_corners=False, ).squeeze() output = prediction.cpu().numpy() output_normalized = cv2.normalize( output, None, 0, 255, cv2.NORM_MINMAX, dtype=cv2.CV_8U) output_colored = cv2.applyColorMap( output_normalized, cv2.COLORMAP_PLASMA) # Blend the original frame with the depth map alpha = 0.7 # Adjust this value between 0 and 1 to change blend strength blended = cv2.addWeighted(frame, 1-alpha, output_colored, alpha, 0) cv2.imshow('Camera Feed', blended) if cv2.waitKey(1) & 0xFF == ord('q'): break finally: cap.release() cv2.destroyAllWindows() ``` ### api/hello.py ```python def main(): print("Hello from api!") if __name__ == "__main__": main() ``` ### site/next.config.ts ```typescript import type { NextConfig } from "next"; const nextConfig: NextConfig = { /* config options here */ }; export default nextConfig; ``` [33 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]