# Project export: EchoRoute This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: CruzHacks 2025 - Tagline: An AI-powered navigation system that detects obstacles, generates voice instructions, and visualizes the safest path—bringing real-time spatial awareness to the visually impaired. - Devpost: https://devpost.com/software/echoroute - GitHub: https://github.com/smjanson/cruzhacks2025.git - Video: https://www.youtube.com/embed/1d70KLC1yh8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Wildest Idea) - Team: 2 GitHub contributor(s) — Sierra Janson (8 commits), smjanson (2 commits) ## Devpost submission (written by the team) ### Inspiration We asked ourselves: "What if AI could see the world and speak it for someone who can’t?" Millions of blind and visually impaired people face uncertainty every time they navigate public spaces. Most assistive tools rely on GPS or limited object detection — but they don't provide the kind of real-time, spatially-aware guidance a sighted person takes for granted. We wanted to change that. We set out to build a tool that acts like a co-pilot — one that sees, thinks, and speaks. ### What it does EchoRoute is a smart, real-time navigation assistant designed for the visually impaired. It combines computer vision, AI-generated natural language, and audio feedback to help users move through complex environments safely. The system captures live video from a webcam, detects obstacles and walkable areas using segmentation, computes a "safe path" visually, and delivers spoken instructions like "veer slightly left to avoid a chair." Every few seconds, the camera feed is analyzed, a dynamic path is generated, and a fresh voice instruction is spoken aloud. Users can also interact with EchoRoute using voice input to describe their destination, which is transcribed using speech-to-text and used to inform future movement decisions. How We Built It We combined multiple technologies to make EchoRoute work: YOLOv8 (segmentation mode) for real-time object and floor detection, used to identify obstacles and walkable regions from the webcam feed. Custom masking and pathfinding logic to interpret YOLO segmentation masks and draw a visual "safe route" through the environment. Google Gemini 2.0 Flash to analyze both the raw image and structured scene data and produce natural language instructions on how to safely move forward. Text-to-Speech (TTS) using gTTS, enabling real-time verbal feedback with clear, actionable guidance. Speech-to-Text (STT) using the speech_recognition library, allowing users to describe their destination through voice input. Streamlit for rapid UI prototyping, live video display, and real-time audio/image interaction. Every few seconds, a frame is analyzed, safe zones are highlighted, and Gemini generates a fresh voice instruction — creating a continuous feedback loop between vision, language, and action. What We Learned We dove deep into: Real-time computer vision with YOLO and OpenCV Multimodal prompting with image and structured data inputs Converting raw mask data into intuitive navigation paths Voice interface engineering using both TTS and STT Prompt engineering and instruction clarity for large language models Managing real-time UI in Streamlit without breaking the video loop We also learned how much precision and timing matter when safety is involved. It’s one thing to detect a chair — it’s another to describe how to move around it in plain English, in real time. Challenges We Faced Latency vs. Quality tradeoffs: We wanted our visual and auditory feedback to be as quick as possible to maximize how useful it is to the user. Balancing frame rate, model inference time, and speech synthesis without GPU acceleration was a recurring challenge. Webcam handling in Streamlit: Streamlit re-renders on every interaction, which doesn't play nicely with persistent video capture. We implemented throttled inference and session state management to keep the stream responsive and stable. Segmentation-based pathfinding: Translating segmentation masks into dynamic paths required a lot of tweaking — including grid-based area detection and arrow-based path rendering. Real-time TTS and STT integration: Ensuring that speech input and audio output didn’t block UI responsiveness took some careful orchestration of temp files, audio buffers, and threading. ### What's next AR or Smart Glasses Hardware: Our immediate next step would be porting the system to smart glasses for real world use (with a los res wifi camera centered in front of the glasses frame, and bluetooth/wifi speakers on the side) Map building: Integrating SLAM or depth estimation to build a local floor map as the user walks Accessibility testing: Partnering with organizations for the visually impaired to collect feedback and iterate Danger assessments: Expanding our model to recognize threats such as incoming cars or bicyclists, and according actions to take Connecting to GPS: integrating our project with location to have large-scale direction Our goals is that EchoRoute will be more than a project, and more a step toward a world in which AI actively amplifies independence, mobility, and safety for everyone. ## README (from the GitHub repository) ## Inspiration We asked ourselves: "What if AI could see the world and speak it for someone who can’t?" Millions of blind and visually impaired people face uncertainty every time they navigate public spaces. Most assistive tools rely on GPS or limited object detection — but they don't provide the kind of real-time, spatially-aware guidance a sighted person takes for granted. We wanted to change that. We set out to build a tool that acts like a co-pilot — one that sees, thinks, and speaks. ## What it does EchoRoute is a smart, real-time navigation assistant designed for the visually impaired. It combines computer vision, AI-generated natural language, and audio feedback to help users move through complex environments safely. The system captures live video from a webcam, detects obstacles and walkable areas using segmentation, computes a "safe path" visually, and delivers spoken instructions like "veer slightly left to avoid a chair." Every few seconds, the camera feed is analyzed, a dynamic path is generated, and a fresh voice instruction is spoken aloud. Users can also interact with EchoRoute using voice input to describe their destination, which is transcribed using speech-to-text and used to inform future movement decisions. ## How We Built It We combined multiple technologies to make EchoRoute work: * YOLOv8 (segmentation mode) for real-time object and floor detection, used to identify obstacles and walkable regions from the webcam feed. * Custom masking and pathfinding logic to interpret YOLO segmentation masks and draw a visual "safe route" through the environment. * Google Gemini 2.0 Flash to analyze both the raw image and structured scene data and produce natural language instructions on how to safely move forward. * Text-to-Speech (TTS) using gTTS, enabling real-time verbal feedback with clear, actionable guidance. * Speech-to-Text (STT) using the speech_recognition library, allowing users to describe their destination through voice input. * Streamlit for rapid UI prototyping, live video display, and real-time audio/image interaction. Every few seconds, a frame is analyzed, safe zones are highlighted, and Gemini generates a fresh voice instruction — creating a continuous feedback loop between vision, language, and action. ## What We Learned We dove deep into: * Real-time computer vision with YOLO and OpenCV * Multimodal prompting with image and structured data inputs * Converting raw mask data into intuitive navigation paths * Voice interface engineering using both TTS and STT * Prompt engineering and instruction clarity for large language models * Managing real-time UI in Streamlit without breaking the video loop We also learned how much precision and timing matter when safety is involved. It’s one thing to detect a chair — it’s another to describe how to move around it in plain English, in real time. ## Challenges We Faced * Latency vs. Quality tradeoffs: We wanted our visual and auditory feedback to be as quick as possible to maximize how useful it is to the user. Balancing frame rate, model inference time, and speech synthesis without GPU acceleration was a recurring challenge. * Webcam handling in Streamlit: Streamlit re-renders on every interaction, which doesn't play nicely with persistent video capture. We implemented throttled inference and session state management to keep the stream responsive and stable. * Segmentation-based pathfinding: Translating segmentation masks into dynamic paths required a lot of tweaking — including grid-based area detection and arrow-based path rendering. * Real-time TTS and STT integration: Ensuring that speech input and audio output didn’t block UI responsiveness took some careful orchestration of temp files, audio buffers, and threading. ## What's next for EchoRoute * AR or Smart Glasses Hardware: Our immediate next step would be porting the system to smart glasses for real world use (with a los res wifi camera centered in front of the glasses frame, and bluetooth/wifi speakers on the side) * Map building: Integrating SLAM or depth estimation to build a local floor map as the user walks * Accessibility testing: Partnering with organizations for the visually impaired to collect feedback and iterate * Danger assessments: Expanding our model to recognize threats such as incoming cars or bicyclists, and according actions to take * Connecting to GPS: integrating our project with location to have large-scale direction Our goals is that EchoRoute will be more than a project, and more a step toward a world in which AI actively amplifies independence, mobility, and safety for everyone. ## How to run EchoRoute **Clone the repository** ```bash $ git clone git@github.com:smjanson/cruzhacks2025.git $ cd cruzhacks2025 ``` **Instal Streamlit** ``` bash $ pip install streamlit ``` **Run the app** ```bash $ streamlit run app.py ``` ## Detected evidence (automated analysis) Indexed codebase: 2 recognized source files, 25 KB. - Python (language) — detected in the code - Streamlit (technology) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - PyTorch (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (5 of 5) ``` app.py packages.txt README.md requirements.txt yolov8n-seg.pt ``` ### Dependencies - requirements.txt: numpy, opencv-python-headless, streamlit, ultralytics ### Recent commits (newest first) - Update README.md - Update README.md - update readme - fully integrated visual audio version - . - tts from sankie integrated - speech to text working - browser based camera - with path finding - updating requirements - . - Add files via upload ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` streamlit ultralytics numpy opencv-python-headless ``` ### app.py ```python import streamlit as st import cv2 import numpy as np from ultralytics import YOLO import tempfile import time from gtts import gTTS from io import BytesIO from PIL import Image from google import genai # import google.generativeai as genai import os import json import tempfile import speech_recognition as sr # Page configuration st.set_page_config(page_title="Navigation Assistant", layout="wide") st.title("Smart Navigation Guidance") # Initialize Gemini client try: client = genai.Client(api_key="INSERT GEMINI API KEY") except Exception as e: st.error(f"Failed to initialize Gemini client: {e}") client = None # Initialize session state variables if 'audio_feedback' not in st.session_state: st.session_state.audio_feedback = True if "last_instruction_time" not in st.session_state: st.session_state.last_instruction_time = 0 if "latest_instruction" not in st.session_state: st.session_state.latest_instruction = "Waiting for instruction..." if "run_camera" not in st.session_state: st.session_state.run_camera = False if "scene_context" not in st.session_state: st.session_state.scene_context = { "previous_instructions": [], "environment_description": "", "detected_objects": [] } audio_value = st.audio_input("") if audio_value: # st.audio(audio_value) with tempfile.NamedTemporaryFile(delete=False, suffix=".wav") as tmp: tmp.write(audio_value.getvalue()) tmp_path = tmp.name # Load the recognizer recognizer = sr.Recognizer() # # Load the audio file with sr.AudioFile(tmp_path) as source: audio_data = recognizer.record(source) # Read the entire file try: text = recognizer.recognize_whisper(audio_data) st.success("Transcription:") st.write(text) prompt = f"Can you tell me what location the user wants to go to from their transcribe: {text}" res = client.models.generate_content( model="gemini-2.0-flash", contents=[prompt] ) except sr.UnknownValueError: st.error("Could not understand audio") except sr.RequestError as e: st.error(f"Could not request results; {e}") # Load YOLO model @st.cache_resource def load_yolo_model(): try: return YOLO("yolov8n-seg.pt") except Exception as e: st.error(f"Failed to load YOLO model: {e}") return None model = load_yolo_model() # Text-to-Speech function def text_to_speech(text, language="en", slow=False): if not text or not st.session_state.audio_feedback: return None try: tts = gTTS(text=text, lang=language, slow=slow) audio_file = BytesIO() tts.write_to_fp(audio_file) audio_file.seek(0) return audio_file except Exception as e: st.error(f"TTS Error: {e}") return None # Function to extract scene information from YOLO results def extract_scene_info(results, frame_shape): scene_info = { "detected_objects": [], "obstacles": [], "walkable_areas": {}, "frame_width": frame_shape[1], "frame_height": frame_shape[0] } # Extract object information if available if hasattr(results, 'boxes') and results.boxes is not None: for i, box in enumerate(results.boxes): if hasattr(results, 'names') and results.names is not None: class_id = int(box.cls.item()) class_name = results.names[class_id] # Get confidence score confidence = float(box.conf.item()) if hasattr(box, 'conf') else 0.0 # Get box coordinates (normalized format) if hasattr(box, 'xyxy'): x1, y1, x2, y2 = box.xyxy.cpu().numpy()[0] # Calculate position in the frame (relative to center) center_x = (x1 + x2) / 2 center_y = (y1 + y2) / 2 # Determine horizontal position relative_x = center_x / frame_shape[1] horizontal_pos = "left" if relative_x < 0.33 else "center" if relative_x < 0.66 else "right" # Determine vertical position (lower y value is higher in the frame) relative_y = center_y / frame_shape[0] vertical_pos = "top" if relative_y < 0.33 else "middle" if relative_y < 0.66 else "bottom" # Object size estimation area = (x2 - x1) * (y2 - y1) area_percentage = area / (frame_shape[0] * frame_shape[1]) size = "small" if area_percentage < 0.1 else "medium" if area_percentage < 0.3 else "large" # Add to detected objects obj_info = { "class": class_name, "confidence": confidence, "position": { "horizontal": horizontal_pos, "vertical": vertical_pos }, "size": size } scene_info["detected_objects"].append(obj_info) # Treat most objects as obstacles scene_info["obstacles"].append(obj_info) # Extract walkable area information if masks are available if hasattr(results, 'masks') and results.masks is not None: h, w = frame_shape[:2] # Analyze walkable areas by dividing the image into a grid grid_size = 3 # 3x3 grid for x_section in range(grid_size): for y_section in range(grid_size): # Define section boundaries x_start = int(x_section * w / grid_size) [truncated — 14464 more characters] ```