# Project export: VisionMate This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Empowering the visually-impaired with AI-driven speech, vision, and navigation assistance for safer traveling. - Devpost: https://devpost.com/software/hachi - GitHub: https://github.com/lisere5/blindNavigation - Demo: https://app.flutterflow.io/project/blind-navigation-8fbmw1 - Video: https://www.youtube.com/embed/McMcTzVFlZU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Serena Li (32 commits), aidanlee09 (11 commits), ecc2205@columbia.edu (6 commits) ## Devpost submission (written by the team) ### Inspiration We came together as a team over our shared passion to create high-impact products and solutions to contribute to the healthcare sector. A member shared how she came across visually-impaired influencers on social media detailing the hardships they faced on a daily basis—just how difficult it was to complete simple tasks such as navigating around an obstacle. Watching her suggested videos, we echoed her empathy and decided that we wanted to build a product specifically targeted at improving ease of mobility for the visually impaired. Hopefully, starting with a smartphone app, we can make walking outside a safer, more practical task. ### What it does Walking around within a bustling surrounding is typically dangerous for a person that is visually impaired, sometimes even for those that are not. But with this innovative mobile application, users can take photos of their surroundings with their phone, which are then processed with image recognition and depth estimation algorithms to identify obstacles and provide personalized and quantified safety suggestions from LLMs for blind users to travel within their surrounding environment. ### How we built it This fullstack project is built through a seamless integration of a robust backend and innovative frontend, with hidden features that enhance the user experience. For the backend, we implemented FastAPI for its speed and flexibility. After processing the image taken by the user, through OpenAI API calls and carefully engineered prompting using techniques such as Chain-of-Thought, we verbalize surrounding of the visually impaired. When an obstacle is detected, we use the Segment Anything Model (SAM) and Depth Anything V2 to estimate its distance from the user. Since no highly accurate measuring AI model exists, we combined Depth Anything V2 for depth mapping with SAM for object segmentation, creating a precise obstacle mask to improve distance estimation. On the frontend, we utilized FlutterFlow, which allowed us to send surrounding photos to the backend through Firebase. Additionally, we integrated special features, including Eleven Labs for text-to-speech, the Whisper API for speech-to-text, LangChain for advanced language processing, and depth estimation frameworks to provide an extra degree of surrounding information for those in need. ### Challenges we ran into Some of the biggest challenges we ran into were our indecisiveness in mapping out the project benchmarks, individual technical challenges which prompted us to reach out to mentors and other hackers, and arguably the largest of them all, scrambling to beat the deadline through all our attempts to perfect our project. ### Accomplishments we're proud of Having two new hackers on the team, along with our decision to explore a variety of new apps and methods we had never encountered before, has been our biggest challenge. However, we’re all really proud of how well we collaborated and pushed our limits. We’re also especially proud that our work resulted in an app that contributes to accessibility and addresses important social issues, making a meaningful impact beyond just the technical side. ### What we learned Through this process, we learned the value of time and it being the most valuable resource especially in hackathon situations. This involved improving methods of time management, finding new ways to brainstorm ideas, and having a high sense of adaptability. Additionally, we learned to familiarize ourselves with novel technologies and use them in ways to solve problems which meant a lot to us. ### What's next for Navigating the Unknown The next couple challenges that we want to tackle with the app are fine-tuning the LLM image recognition model to evoke more precise responses that can be better quantified to provide the user with pin-point information, and implementing a translation text to speech feature to further simplify the user's surroundings in a foreign environment. There are several directions with which this project can be advanced, but we feel that the combination of the wide variety of technologies that we used exhibited the power of fusing cutting-edge resources together to create impactful developments to the field of healthcare. ## README (from the GitHub repository) --- title: "Blind Navigation" emoji: 🏃‍♂️ colorFrom: purple colorTo: indigo sdk: docker app_file: Dockerfile pinned: false --- # blindNavigation ## Backend ### Setup cd to project directory (../blindNavigation) #### ENV SETUP ``` conda create --name blindnav python=3.11 -y conda activate blindnav ``` #### INSTALL DEPENDENCIES ``` conda install pytorch torchvision torchaudio -c pytorch-nightly pip install fastapi uvicorn opencv-python numpy pillow pip install langchain langchain-openai openai pip install elevenlabs pip install python-dotenv pip install python-multipart pip install pillow pillow-heif pip install sounddevice pip install tensorflow tensorflow-hub pip install tensorflow-macos tensorflow-metal tensorflow-hub pip install scipy pip install timm pip install --upgrade langchain langchain-community langchain-openai pip install git+https://github.com/facebookresearch/segment-anything.git ``` #### VERIFY (in terminal) ``` python import torch import openai import langchain import elevenlabs from dotenv import load_dotenv print("MPS (Apple GPU) Available:", torch.backends.mps.is_available()) print("MPS Backend Built:", torch.backends.mps.is_built()) print("PyTorch Version:", torch.__version__) print("OpenAI Installed:", openai.__version__) print("LangChain Installed:", langchain.__version__) print("ElevenLabs Installed:", hasattr(elevenlabs, "generate")) ``` #### check if everything returns the right string #### DEACTIVATE ``` conda deactivate ``` ### Start Fast API Server ``` uvicorn main:app --reload ``` Once the server starts, visit: Swagger UI (API Docs): http://127.0.0.1:8000/docs JSON Response (Basic Test): http://127.0.0.1:8000 ### Exit control + c ## Citation @article{depth_anything_v2, title={Depth Anything V2}, author={Yang, Lihe and Kang, Bingyi and Huang, Zilong and Zhao, Zhen and Xu, Xiaogang and Feng, Jiashi and Zhao, Hengshuang}, journal={arXiv:2406.09414}, year={2024} } @inproceedings{depth_anything_v1, title={Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data}, author={Yang, Lihe and Kang, Bingyi and Huang, Zilong and Xu, Xiaogang and Feng, Jiashi and Zhao, Hengshuang}, booktitle={CVPR}, year={2024} } @article{kirillov2023segany, title={Segment Anything}, author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{\'a}r, Piotr and Girshick, Ross}, journal={arXiv:2304.02643}, year={2023} } ## Detected evidence (automated analysis) Indexed codebase: 53 recognized source files, 188 KB. - Dart (language) — detected in the code - FastAPI (technology) — detected in the code - LangChain (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - Firebase (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (61 of 61) ``` .gitattributes .gitignore app.py blindNavigationFF/app_state.dart blindNavigationFF/backend/api_requests/api_calls.dart blindNavigationFF/backend/api_requests/api_manager.dart blindNavigationFF/backend/api_requests/get_streamed_response.dart blindNavigationFF/backend/backend.dart blindNavigationFF/backend/firebase/firebase_config.dart blindNavigationFF/backend/schema/images_record.dart blindNavigationFF/backend/schema/index.dart blindNavigationFF/backend/schema/util/firestore_util.dart blindNavigationFF/backend/schema/util/schema_util.dart blindNavigationFF/camera/camera_model.dart blindNavigationFF/camera/camera_widget.dart blindNavigationFF/custom_code/widgets/camera_photo.dart blindNavigationFF/custom_code/widgets/index.dart blindNavigationFF/flutter_flow/custom_functions.dart blindNavigationFF/flutter_flow/flutter_flow_model.dart blindNavigationFF/flutter_flow/flutter_flow_theme.dart blindNavigationFF/flutter_flow/flutter_flow_util.dart blindNavigationFF/flutter_flow/flutter_flow_widgets.dart blindNavigationFF/flutter_flow/lat_lng.dart blindNavigationFF/flutter_flow/nav/nav.dart blindNavigationFF/flutter_flow/nav/serialization_util.dart blindNavigationFF/flutter_flow/place.dart blindNavigationFF/flutter_flow/upload_data.dart blindNavigationFF/flutter_flow/uploaded_file.dart blindNavigationFF/home_page/home_page_model.dart blindNavigationFF/home_page/home_page_widget.dart blindNavigationFF/index.dart blindNavigationFF/main.dart checkpoints/depth_anything_v2_metric_hypersim_vitb.pth checkpoints/sam_vit_b_01ec64.pth depth_anything_v2/__init__.py depth_anything_v2/dinov2_layers/__init__.py depth_anything_v2/dinov2_layers/attention.py depth_anything_v2/dinov2_layers/block.py depth_anything_v2/dinov2_layers/drop_path.py depth_anything_v2/dinov2_layers/layer_scale.py depth_anything_v2/dinov2_layers/mlp.py depth_anything_v2/dinov2_layers/patch_embed.py depth_anything_v2/dinov2_layers/swiglu_ffn.py depth_anything_v2/dinov2.py depth_anything_v2/dpt.py depth_anything_v2/util/__init__.py depth_anything_v2/util/blocks.py depth_anything_v2/util/transform.py depth.py Dockerfile image_processor.py llm_converse.py prompts/obstacle.txt query_manager.py README.md requirements.txt sam.py speechtotext.py start.sh tss.py vercel.json ``` ### Dependencies - requirements.txt: aiofiles, elevenlabs, fastapi, firebase-admin, gradio@==4.29.0, gradio_imageslider, langchain, langchain-community, langchain-openai, matplotlib, numpy, opencv-python, Pillow, pillow-heif, pydantic, python-dotenv, requests, scipy, segment_anything@@ git+https://github.com/facebookresearch/segment-anything.git@dca509fe793f601edb92606367a655c15ac00fdf, sounddevice, starlette, tensorflow-hub, torch, torchvision ### Recent commits (newest first) - change - pls - back to github - pls - Trigger redeploy - fix - fix - firebase - return audio - fix - fix - Delete flutter_flow directory - Delete home_page directory - Delete app_state.dart - Delete index.dart - Delete main.dart - Delete custom_code/widgets directory - Delete camera directory - Delete backend directory - committing all flutterflow code ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` fastapi pydantic requests aiofiles starlette python-dotenv elevenlabs langchain langchain-openai sounddevice numpy tensorflow-hub torch Pillow pillow-heif scipy segment_anything @ git+https://github.com/facebookresearch/segment-anything.git@dca509fe793f601edb92606367a655c15ac00fdf gradio_imageslider gradio==4.29.0 matplotlib opencv-python torchvision langchain-community firebase-admin ``` ### Dockerfile ``` # Use official Python image FROM python:3.11 # Install OpenGL and other necessary libraries RUN apt-get update && apt-get install -y \ libgl1-mesa-glx \ libglib2.0-0 # Set working directory WORKDIR /app # Copy dependencies file and install dependencies COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Copy all app files into the container COPY . . # Expose the FastAPI port EXPOSE 8080 # Command to run the FastAPI server CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"] ``` ### app.py ```python import imghdr from PIL import Image from fastapi import FastAPI, UploadFile, File from query_manager import QueryManager from tss import text_to_speech from fastapi import HTTPException from starlette.responses import FileResponse import firebase_admin from firebase_admin import credentials, storage from starlette.responses import Response import os import io # # Initialize Firebase Admin SDK cred = credentials.Certificate("blind-navigation-8fbmw1-firebase-adminsdk-fbsvc-92f43ecbbb.json") if not firebase_admin._apps: firebase_admin.initialize_app(cred, {"storageBucket": "blind-navigation-8fbmw1.firebasestorage.app"}) def get_storage_bucket(): return storage.bucket() app = FastAPI() query_manager = QueryManager() @app.post("/test/") async def test(testing: str): return testing @app.get("/retrieve_image") async def retrieve_image(): bucket = get_storage_bucket() blobs = list(bucket.list_blobs(prefix="photos/")) # List only images in "photos/" folder # Filter out directories and ensure only image files are considered image_blobs = [blob for blob in blobs if not blob.name.endswith("/")] if not image_blobs: raise HTTPException(status_code=404, detail="No images found in Firebase Storage 'photos/' folder.") # Get the latest image (sorted by time created) latest_blob = max(image_blobs, key=lambda blob: blob.time_created) # Download the image as bytes image_bytes = latest_blob.download_as_bytes() # Detect file type file_type = imghdr.what(None, h=image_bytes) # Convert to JPEG if necessary if file_type != "jpeg": image = Image.open(io.BytesIO(image_bytes)) image = image.convert("RGB") # Ensure no transparency issues img_io = io.BytesIO() image.save(img_io, format="JPEG", quality=95) # Save as JPEG with high quality img_io.seek(0) image_bytes = img_io.read() result = query_manager.save_image(image_bytes, "image/jpeg") image_id = result['image_id'] response = query_manager.default_ask(image_id) text_to_speech(response) filepath = f"audio/output.mp3" try: return FileResponse(filepath, media_type="audio/mpeg", filename=filepath) except FileNotFoundError: raise HTTPException(status_code=404, detail="File not found") # return result # return Response(content=image_bytes, media_type="image/jpeg") # # @app.post("/upload-image/") # async def upload_image(file: UploadFile = File(...)): # file_bytes = await file.read() # result = query_manager.save_image(file_bytes, file.content_type) # return result # # @app.post("/detect-image/") # async def detect_image(image_id: str): # response = query_manager.default_ask(image_id) # text_to_speech(response) # filepath = f"audio/output.mp3" # try: # return FileResponse(filepath, media_type="audio/mpeg", filename=filepath) # except FileNotFoundError: # raise HTTPException(status_code=404, detail="File not found") # # return {"description": response} # @app.post("/follow-up/") # async def follow_up(question: str): # followup_response = query_manager.ask_gpt(question) # query_manager.text_to_speech(followup_response) # return {"followup_response": followup_response} if __name__ == "__main__": import uvicorn uvicorn.run(app, host="0.0.0.0", port=8000, log_level="debug") ``` ### start.sh ```shell #!/bin/bash pip install -r requirements.txt uvicorn main:app --host 0.0.0.0 --port 8000 ``` ### sam.py ```python from segment_anything import SamAutomaticMaskGenerator, sam_model_registry import numpy as np def segment_anything(compressed_img): sam = sam_model_registry["vit_b"](checkpoint="checkpoints/sam_vit_b_01ec64.pth") mask_generator = SamAutomaticMaskGenerator(sam) masks = mask_generator.generate(compressed_img) final_mask = np.zeros(compressed_img.shape[:2], dtype=np.uint8) # Combine all masks into one binary mask for mask in masks: final_mask |= mask["segmentation"].astype(np.uint8) return final_mask ``` ### llm_converse.py ```python from langchain_community.chat_models import ChatOpenAI from langchain.schema import HumanMessage import os def converse(prompt, image, llm_type): api_key = os.getenv("OPENAI_API_KEY") if not api_key: raise ValueError("OPENAI_API_KEY not set in environment variables!") llm = ChatOpenAI(model=llm_type, openai_api_key=api_key) messages = [ HumanMessage( content=[ {"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image}"}} ] ) ] response = llm.invoke(messages) return response.content ``` ### tss.py ```python from fastapi import HTTPException from starlette.responses import FileResponse import os import requests import time def text_to_speech(text): api_key = os.getenv("ELEVENLABS_API_KEY") API_URL = "https://api.elevenlabs.io/v1/text-to-speech/9BWtsMINqrJLrRacOk9x" HEADERS = { "xi-api-key": api_key } body = { "text": text, "model_id": "eleven_monolingual_v1", "voice_settings": {"stability": 0.5, "similarity_boost": 0.5}, "optimize_streaming_latency": 3, "output_format": "mp3_44100_128" } response = requests.post(API_URL, headers=HEADERS, json=body) if response.status_code != 200: raise HTTPException(status_code=500, detail="API request failed") # Ensure output directory exists os.makedirs("audio", exist_ok=True) timestamp = int(time.time()) filename = f"audio/output.mp3" # Write the binary response content to a file with open(filename, "wb") as f: f.write(response.content) ``` ### depth.py ```python import torch import numpy as np from scipy import stats from depth_anything_v2.dpt import DepthAnythingV2 from sam import segment_anything def depth_calculation(compressed_img): model_configs = { 'vits': {'encoder': 'vits', 'features': 64, 'out_channels': [48, 96, 192, 384]}, 'vitb': {'encoder': 'vitb', 'features': 128, 'out_channels': [96, 192, 384, 768]}, 'vitl': {'encoder': 'vitl', 'features': 256, 'out_channels': [256, 512, 1024, 1024]} } encoder = 'vitb' # or 'vits', 'vitb' dataset = 'hypersim' # 'hypersim' for indoor model, 'vkitti' for outdoor model max_depth = 20 # 20 for indoor model, 80 for outdoor model model = DepthAnythingV2(**{**model_configs[encoder], 'max_depth': max_depth}) device = torch.device("mps" if torch.backends.mps.is_available() else "cpu") model = model.to(device) model.load_state_dict( torch.load(f'checkpoints/depth_anything_v2_metric_{dataset}_{encoder}.pth', map_location='cpu')) model.eval() depth = model.infer_image(compressed_img) # HxW depth map in meters in numpy rounded_depth = np.round(depth, 2) # Compute statistics average = np.mean(rounded_depth) # Mean (Average) mode_result = stats.mode(rounded_depth, axis=None) # Flatten array and find mode mode = mode_result.mode if np.isscalar(mode_result.mode) else mode_result.mode[0] threshold = max(average, mode) sam_mask = segment_anything(compressed_img) mask = np.where(depth < threshold, depth, 0) * sam_mask non_zero_values = mask[mask != 0] # Extract non-zero values distance_average = np.mean(non_zero_values) if non_zero_values.size > 0 else 0 # Avoid division by zero return distance_average ``` ### speechtotext.py ```python import sounddevice as sd import wavio import tempfile import os from openai import OpenAI from openai import APIConnectionError, APIError def record_audio(filename, duration=5, fs=44100, channels=1): """Records audio from the microphone and saves it as a WAV file.""" print(f"Recording audio for {duration} seconds...") try: recording = sd.rec(int(duration * fs), samplerate=fs, channels=channels) sd.wait() wavio.write(filename, recording, fs, sampwidth=2) print(f"Recording saved to {filename}") except Exception as e: print(f"Recording error: {str(e)}") raise def transcribe_audio(client, file_path): """Sends the audio file to OpenAI for transcription.""" try: with open(file_path, "rb") as audio_file: response = client.audio.transcriptions.create( model="whisper-1", file=audio_file, response_format="text" ) return response except APIConnectionError as e: print(f"Connection error: {e.__cause__}") raise except APIError as e: print(f"API error: {e}") raise except Exception as e: print(f"Transcription error: {str(e)}") raise def main(): client = OpenAI(api_key="OPENAI_KEY") try: # Create a temporary file with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp: temp_filename = tmp.name print(f"Temporary file created: {temp_filename}") # Record audio record_audio(temp_filename, duration=5) # Verify the file exists and has content if os.path.getsize(temp_filename) == 0: raise ValueError("Recorded file is empty") # Transcribe audio transcription = transcribe_audio(client, temp_filename) print("\nTranscription Result:") print(transcription if transcription else "No transcription returned") except Exception as e: print(f"Main error: {str(e)}") finally: # Clean up the temporary file if os.path.exists(temp_filename): os.remove(temp_filename) print(f"Temporary file {temp_filename} removed") if __name__ == "__main__": main() ``` ### image_processor.py ```python from PIL import Image import io import base64 from pillow_heif import open_heif def resize_image(image, max_size=(256, 256)): """Resize image to reduce resolution and maintain aspect ratio.""" img = image.copy() img.thumbnail(max_size) # Resize while keeping aspect ratio return img def compress_image(image, quality=30): """Compress image to reduce size while maintaining readability.""" if image.mode == "RGBA": image = image.convert("RGB") buffer = io.BytesIO() image.save(buffer, format="JPEG", quality=quality) return buffer.getvalue() def image_to_base64(image): """Convert a PIL image to a base64 string, ensuring it's a valid JPEG.""" if image.mode in ("RGBA", "P"): # Convert RGBA/Palette images to RGB image = image.convert("RGB") buffer = io.BytesIO() image.save(buffer, format="JPEG") # Ensure JPEG format base64_str = base64.b64encode(buffer.getvalue()).decode("utf-8") return base64_str class ImageProcessor: def __init__(self): self.image_info = {} def reset_memory(self): """Optional method to clear image history if needed.""" self.image_history = {} def save_image(self, file_bytes, content_type, image_id): """Process and save image with optimizations for reduced token usage.""" self.reset_memory() try: if content_type in ["image/heic", "image/heif"]: # Convert HEIC to PNG heif_image = open_heif(io.BytesIO(file_bytes)) image = Image.frombytes( heif_image.mode, heif_image.size, heif_image.data, "raw", heif_image.mode ) else: # Handle regular images (JPEG, PNG, etc.) image = Image.open(io.BytesIO(file_bytes)) image_format = image.format # **Apply optimizations** image = resize_image(image) # Resize to 256x256 compressed_image_bytes = compress_image(image, quality=30) # Compress with 30% quality compressed_image = Image.open(io.BytesIO(compressed_image_bytes)) base64_image = image_to_base64(compressed_image) if not isinstance(base64_image, str): raise ValueError("Base64 encoding failed. Expected a string but got: " + str(type(base64_image))) self.image_info = { "image_id": image_id, "original_format": content_type, "converted_format": image_format, "compressed_image": compressed_image, "base64_image": base64_image } return self.image_info except Exception as e: return {"error": f"Failed to process image: {str(e)}"} # Handle errors gracefully ``` [42 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]