# Project export: Vera This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Bringing visually impaired one step closer to full independence - Devpost: https://devpost.com/software/vera-xsbhj0 - GitHub: https://github.com/eden-byte/TreeHacks - Video: https://www.youtube.com/embed/3ic2jEOOD18?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 0 GitHub contributor(s) — ## Devpost submission (written by the team) ### Overview Problem People with impaired vision struggle with independent daily living because today’s tools do not give support for understanding their surroundings, recognizing people and objects, reading text, and avoiding collisions in dynamic environments. As a result, everyday tasks like moving through crowded spaces, shopping, taking medications safely, and recognizing acquaintances become stressful and dependent on sighted assistance. Existing aids like white canes, basic screen readers, and single‑purpose apps either provide only partial information (obstacle detection without context), require manual interaction that breaks flow (stopping to use a phone app), or lack memory and personalization over time. There is a clear need for a wearable system that can see, remember, and guide continuously using voice and haptic feedback. Our solution Vera is a smart wearable system that helps people with impaired vision understand their surroundings, recognize people and objects, read text, and navigate safely using live AI processing and feedback. It has two main parts: Smart glasses Live environmental understanding by capturing video and audio, processing them with cloud AI, then giving spoken guidance to any questions the user might have, including scene descriptions, text reading, medication information, barcode scanning, expiration dates, and any web or deep research searches powered by Perplexity. Voice-activated assistance allows users to ask questions naturally using wake word detection ("Hey Vera") and receive spoken answers. Memory and personalization through remembering registered faces, tracking daily encounters, maintaining conversation context for follow-ups, and providing context-aware responses across sessions. Vibrating necklace Collision Prevention in a necklace with 5 vibration motors placed around the neck. The motors vibrate in the direction where the obstacle is located to tell the user which way to turn. The vibrations also get stronger as the object gets closer, indicating a higher collision risk. ### How we built it Software: On the backend, we use OpenAI's GPT-4o-mini vision model for scene description, text reading, and object recognition, with Perplexity API (sonar and sonar-deep-research models) answering web queries and complex background research. YOLOv5 performs real-time object detection for obstacle avoidance and collision prevention. We built a RAG memory system using ChromaDB vector database with OpenAI text-embedding-3-small to store user interactions, face registrations, and conversation context across sessions. Our Python-based system uses a multi-threaded architecture: one thread captures webcam frames, another handles voice input via Google Speech Recognition API, and a third manages text-to-speech output through OpenAI TTS (nova voice). Hardware: At the beginning, we worked on breadboarding that was eventually transitioned onto computer boards. Using 5 vibration motors in the necklace connected to a raspberry pi that works in conjunction with a justin nano to run our custom architecture. The glasses hardware has a camera on it that facilitates object detection and other features. There is also an attachment for the raspberry pi and jenson nano to clip onto the waist band. ### Challenges we ran into Throughout development, we bumped into several key obstacles that actually drove our innovation further. The NVIDIA API was tough to integrate and required a lot of troubleshooting to get working properly. Hardware limitations meant we had to find creative ways to optimize performance and make the most of what we had. Getting our different code components to work together smoothly after working separately for most of the time - connecting the vision processing, voice interface, and hardware - took significant effort. However, these constraints actually helped us grow. They forced us to experiment with different solutions and build a more flexible system than we might have if everything had been ideal from the start. ### Accomplishments we're proud of Taking on this project was extremely ambitious, building a fully functional wearable assistive system with both smart glasses and a haptic necklace in just 36 hours was an uncertain task, and at many points we weren't sure if we could integrate all the components in time. We're incredibly proud that we not only created working hardware but also implemented a voice-controlled interface that requires no technical knowledge to use. Most importantly, we spoke with visually impaired people who confirmed the critical need for this type of integrated assistive technology. Their feedback validated that the problems we're solving are real and that a system like Vera could make a genuine difference in people's lives. ### What we learned We learned how to integrate hardware components with software systems, working extensively with API integration and managing Python libraries to ensure compatibility. We got experience combining different parts of the system (voice control, vision processing, memory) into a unified codebase. We discovered how to think about object detection from a mathematical perspective, and how to work in development sandboxes to test integrations before deploying to the final system. We also learned the importance of splitting roles and having a whiteboard to visualize the whole idea. ### What's next Vibrating bracelets for lost object detection - We were originally thinking of having both bracelets and necklaces, but ran out of time. The system would detect where a misplaced object is located and vibrate specific bracelets to guide the user's hand directly toward it, making it easy to find keys, phones, or other everyday items without searching. GPS integration for turn-by-turn navigation - Adding GPS to glasses will let Vera provide spoken directions for walking routes, helping users navigate unfamiliar areas independently with live guidance. Road sign recognition - Expanding given visual recognition to identify crosswalk signals, street signs, and pavement textures will help users walk more safely by detecting curbs, crosswalks, and other important navigation cues. ## README (from the GitHub repository) # AI-Native Assistive Vision System An AI-powered navigation and assistance system for people who are blind or visually impaired. Built at TreeHacks 2026 ## Overview 285 million people worldwide are blind or visually impaired. Current assistive technologies are either expensive ($30,000 for a guide dog) or limited in capability (traditional white canes). We built a comprehensive AI-powered system that combines real-time vision understanding, haptic feedback, and conversational AI to provide independence and safety. ## What We Built An AI-native assistive vision system that provides: - **AI Vision Intelligence**: Identifies currency, colors, text, and obstacles in real-time - **Multi-Zone Haptic Feedback**: 5-direction vibration necklace for spatial awareness - **Conversational AI Assistant**: Voice interface with scene understanding - **Medical Integration**: Medication safety checking and vitals monitoring - **Cloud-Native Scale**: Scalable architecture using Modal sandboxes ## Architecture ### System Overview ``` ┌─────────────────────────────────────────────────────────┐ │ USER HARDWARE │ │ ┌──────────────────┐ ┌────────────────────────┐ │ │ │ Meta Glasses │ │ Vibration Necklace │ │ │ │ (Camera Input) │ │ (5-Zone Haptic Output)│ │ │ └────────┬─────────┘ └──────────┬─────────────┘ │ └───────────┼────────────────────────────┼────────────────┘ │ │ ▼ ▼ ┌─────────────────────────────────────────────────────────┐ │ JETSON ORIN NANO (Edge AI) │ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Obstacle │ │ Object │ │ Haptic │ │ │ │ Detection │ │ Recognition │ │ Controller │ │ │ │ (Real-time) │ │ (YOLOv5) │ │ (GPIO) │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ └───────────┬─────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ MODAL SANDBOXES (Cloud Intelligence) │ │ │ │ ┌────────────────────────────────────────────────┐ │ │ │ Per-User Isolated Sandboxes (GPU-Accelerated)│ │ │ │ │ │ │ │ ┌──────────────┐ ┌────────────────────────┐ │ │ │ │ │ Vision AI │ │ Conversational Agent │ │ │ │ │ │ (GPT-4V) │ │ (ChatGPT + RAG) │ │ │ │ │ └──────────────┘ └────────────────────────┘ │ │ │ │ │ │ │ │ ┌──────────────┐ ┌────────────────────────┐ │ │ │ │ │ Medical AI │ │ Emergency Monitor │ │ │ │ │ │ (Drug Check) │ │ (Fall Detection) │ │ │ │ │ └──────────────┘ └────────────────────────┘ │ │ │ └────────────────────────────────────────────────┘ │ │ │ │ Auto-scales: 1 user → 50,000 users │ └─────────────────────────────────────────────────────────┘ ``` ### Technology Stack **Edge Computing:** - NVIDIA Jetson Orin Nano (Edge AI processor) - YOLOv5 (Real-time object detection) - OpenCV (Computer vision pipeline) - Custom GPIO haptic controller **AI & Cloud:** - OpenAI GPT-4 Vision (Scene understanding) - ChatGPT API (Conversational intelligence) - Modal (Serverless GPU sandboxes) - Text-to-Speech (Audio feedback) **Hardware:** - Custom glasses with camera - Custom 5-zone vibration necklace - Arduino/ESP32 (Haptic motor control) ## Key Capabilities ### AI Vision Intelligence Real-time object and scene understanding using computer vision and AI. ```python # User holds up object to glasses camera → AI Vision Processing → "This is a twenty dollar bill" → "Prescription bottle: Ibuprofen 200mg" → "Red shirt, medium size" ``` ### Multi-Zone Haptic Navigation 5-zone vibration necklace provides directional obstacle awareness. ``` Vibration Zones: [Left] [Center-Left] [Center] [Center-Right] [Right] Obstacle at 2 o'clock → Center-Right motor vibrates Person approaching left → Left motor pulses Clear path ahead → All motors off ``` ### Conversational AI Assistant Natural voice interaction for questions and guidance. ``` User: "What am I looking at?" AI: "You're in front of a CVS pharmacy. The entrance is 15 feet ahead." User: "What color is this?" AI: "Navy blue" User: "Can I take this medicine?" AI: "This is ibuprofen. Checking your medical history... WARNING: You have a documented allergy to NSAIDs." ``` ### Medical Integration Continuous health monitoring and medication safety. - Medication Safety: OCR prescription labels with drug interaction checking - Vitals Monitoring: Heart rate and movement patterns from necklace sensors - Fall Detection: Automatic emergency alerts - Provider Dashboard: Healthcare professionals can view safety events and location logs ### Scalable Cloud Architecture Modal sandboxes provide personalized AI for each user. ```python # Each user gets isolated AI sandbox @app.function(gpu="T4", secrets=[...]) def user_vision_sandbox(user_id: str, image: bytes): """ Personalized AI processing: - Learns user preferences - Stores medical context - Adapts to environment """ return personalized_results ``` ## Impact This system addresses the needs of 285 million visually impaired people worldwide by providing: - Independent navigation in unfamiliar environments - Real-time object and obstacle identification - Medication safety and health monitoring - Emergency detection and response - Affordable, scalable access to AI assistance ($5-10/month vs $30,000 for guide dogs) ## Getting Started ### Prerequisites Hardware: - NVIDIA Jetson Orin Nano - Meta Ray-Ban Smart Glasses (or USB webcam) - 5x vibration motors + motor driver - Arduino/ESP32 for haptic control Software: - Python 3.8+ - PyTorch - OpenCV - Modal account (for cloud deployment) ### Installation ```bash # Clone repository git clone https://github.com/yourusername/sunu-ai-native.git cd sunu-ai-native # Install dependencies pip install -r requirements.txt --break-system-packages # Configure environment cp .env.example .env # Add your API keys: OPENAI_KEY, MODAL_TOKEN # Run on Jetson python main.py # Deploy to Modal (optional) modal deploy cloud_deploy.py ``` ### Quick Test ```bash # Test camera + object detection python test_vision.py # Test haptic feedback python test_haptics.py # Test full system python main.py --demo ``` ## License MIT License - see LICENSE file for details ## Detected evidence (automated analysis) Indexed codebase: 1 recognized source files, 8 KB. - OpenAI (technology) — claimed on Devpost, not found in the code - Python (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (1 of 1) ``` README.md ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Update hardware description in README - Remove Key Capabilities section - Revise README - Revise README - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) No suitable source files found.