# Project export: LookOut: AI Guidance for Blind Navigation This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: “LookOut” empowers blind and low-vision users with real-time voice guidance, turning obstacles into safe paths and restoring freedom, independence, and confidence in every step. - Devpost: https://devpost.com/software/lookout-ai-guidance-for-blind-navigation - GitHub: https://github.com/AidanGu/LookOut - Video: https://www.youtube.com/embed/FQbiLSn3Yac?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Aidan Gu (8 commits) ## Devpost submission (written by the team) ### Inspiration Over 285 million people globally live with vision impairment, out of which 39 million are completely blind. Every single day, they face obstacles most of us never notice, like a misplaced chair, a stairwell with no railing, a silent car approaching from the side, and more. Our goal was to deliver a solution that delivers real-time, actionable guidance, translating vision into safe and clear next steps. By combining the phone’s camera, depth sensing, and AI voice interaction, it helps people move safely with confidence, empowering Accessibility for ALL. ### What it does LookOut listens to the user’s voice input, scans the surroundings through the phone camera, and uses an AI model to detect obstacles in real time. It then responds with short, clear directions on where to go, using depth and clock-based cues to describe obstacles and safe paths. The system is designed for low latency so guidance feels instant and natural, giving blind and low-vision users the confidence to move safely. ### How we built it We decided to try LiveKit for low latency and first implemented and tested our setup in the Agents Playground. From there, we leveraged a Python agent on LiveKit Agents to receive video streams, run reasoning with the Gemini Live API, and send back speech within the same session. We also used LiveKit’s data channel for basic controls such as start and stop listening and switching between front and rear cameras. ### Challenges we ran into At first we built a setup using Omi hardware for voice and a separate camera integration on web side using Next.js, React, and TypeScript, but connecting the two with low latency proved difficult and often left the user waiting. When we switched fully to LiveKit , the system became faster, more stable, and far more reliable. One of our hardest challenges was keeping the camera feed and spoken responses in sync, especially when the network was weak. After carefully tuning the audio and video pipeline we finally achieved a smooth, natural flow that felt responsive instead of robotic. Just as critical, we realized that long or cluttered instructions could overwhelm the user. The model had to speak in short, precise, and accurate phrases that guided the way without confusion. Through this process we also learned how refining the system prompt itself improved accuracy. By teaching the model to focus on depth and clock-based directions, it began to describe obstacles and safe paths in a way that blind and low-vision users could trust instantly. ### Accomplishments we're proud of We are proud that we turned an ambitious idea into a working prototype within the hackathon. Our biggest accomplishment was achieving real-time guidance with low latency, where the camera feed, AI reasoning, and spoken responses stayed in sync. We successfully integrated LiveKit with the Gemini Live API and refined the system prompt so the model could deliver short, accurate, and natural instructions instead of overwhelming users. Another accomplishment was designing the interaction to feel intuitive: the app listens to the user, scans the environment, and replies instantly with depth and clock-based cues. Most importantly, we built something that has the potential to replace hesitation with confidence for blind and low-vision users! ### What we learned We learned how challenging and important it is to keep audio, video, and model responses perfectly in sync. At first, small delays made the interaction feel unreliable, but through testing and fine-tuning we discovered how to optimize LiveKit for low latency, so guidance feels instant and natural. We also learned how much the quality of the system prompt affects the model’s performance. By refining the prompt to focus on short, accurate, and context-aware directions, we improved both the clarity and reliability of the responses. Together, these lessons taught us not only technical skills in streaming and real-time AI but also how critical it is to design for user trust and confidence. ### What's next for Lookout: AI Guidance for Blind Navigation The next step for LookOut is integrating maps and destination-based navigation. Right now, the system guides users safely through their immediate surroundings, but we envision combining that with turn-by-turn directions so a user can choose a destination and receive both obstacle avoidance and route guidance in one experience. This will allow LookOut to not only prevent collisions but also help blind and low-vision users reach where they need to go with independence and confidence. Video- https://www.facebook.com/61582589294855/videos/1139565257807161/ ## README (from the GitHub repository) # [LookOut](https://olookout.tech/) ## oLookOut.tech **Contributors:** [Gauri Jain](https://github.com/gaurijain21) · [Samanyu Kumar](https://github.com/samanyuk) · [Victor Cruz](https://github.com/Victor20120) · [Aidan Gu](https://github.com/AidanGu) > Voice-powered guidance for blind and low-vision users, combining real-time computer vision with turn-taking voice and Google Maps navigation. --- ## Setup ### Prerequisites - Node.js ≥ 18 and npm (or pnpm) - Python ≥ 3.11 - LiveKit Cloud project credentials - Google API keys (Gemini + Maps) --- ## Detected evidence (automated analysis) Indexed codebase: 72 recognized source files, 221 KB. - CSS (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - JavaScript (language) — claimed on Devpost, not found in the code - Node.js (technology) — claimed on Devpost, not found in the code - Swift (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (79 of 79) ``` agent/main.py agent/requirements.txt agent/ruff.toml app/actions/livekit.ts app/api/directions/route.ts app/globals.css app/layout.tsx app/page.tsx components.json components/theme-provider.tsx components/ui/accordion.tsx components/ui/alert-dialog.tsx components/ui/alert.tsx components/ui/aspect-ratio.tsx components/ui/avatar.tsx components/ui/badge.tsx components/ui/breadcrumb.tsx components/ui/button-group.tsx components/ui/button.tsx components/ui/calendar.tsx components/ui/card.tsx components/ui/carousel.tsx components/ui/chart.tsx components/ui/checkbox.tsx components/ui/collapsible.tsx components/ui/command.tsx components/ui/context-menu.tsx components/ui/dialog.tsx components/ui/drawer.tsx components/ui/dropdown-menu.tsx components/ui/empty.tsx components/ui/field.tsx components/ui/form.tsx components/ui/hover-card.tsx components/ui/input-group.tsx components/ui/input-otp.tsx components/ui/input.tsx components/ui/item.tsx components/ui/kbd.tsx components/ui/label.tsx components/ui/menubar.tsx components/ui/navigation-menu.tsx components/ui/pagination.tsx components/ui/popover.tsx components/ui/progress.tsx components/ui/radio-group.tsx components/ui/resizable.tsx components/ui/scroll-area.tsx components/ui/select.tsx components/ui/separator.tsx components/ui/sheet.tsx components/ui/sidebar.tsx components/ui/skeleton.tsx components/ui/slider.tsx components/ui/sonner.tsx components/ui/spinner.tsx components/ui/switch.tsx components/ui/table.tsx components/ui/tabs.tsx components/ui/textarea.tsx components/ui/toast.tsx components/ui/toaster.tsx components/ui/toggle-group.tsx components/ui/toggle.tsx components/ui/tooltip.tsx components/ui/use-mobile.tsx components/ui/use-toast.ts components/voice-interface.tsx DEPLOYMENT.md hooks/use-mobile.ts hooks/use-toast.ts lib/utils.ts LICENSE next.config.mjs package.json postcss.config.mjs README.md styles/globals.css tsconfig.json ``` ### Dependencies - agent/requirements.txt: googlemaps@>=4.10.0, livekit-agents[google,images,turn-detector]@~=1.2,>=1.2.0, livekit-plugins-noise-cancellation@~=0.2, python-dotenv - package.json: @tailwindcss/postcss@^4.1.9, @types/dom-mediacapture-record@^1.0.22, @types/node@^22, @types/react@^19, @types/react-dom@^19, @vercel/analytics@latest, class-variance-authority@^0.7.1, clsx@^2.1.1, jose@latest, livekit-client@latest, livekit-server-sdk@2.14.0, lucide-react@^0.454.0, next@^15.1.6, postcss@^8.5, react@^19.0.0, react-dom@^19.0.0, server-only@latest, tailwind-merge@^2.5.5, tailwindcss@^4.1.9, tailwindcss-animate@^1.0.7, tw-animate-css@latest, typescript@^5 ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - Add files via upload - Add files via upload ## Key source files (fetched from GitHub, selected and truncated for size) ### DEPLOYMENT.md ```markdown # Deployment Guide ## Deploying to Vercel ### Web App 1. Push your code to GitHub 2. Import your repository in Vercel 3. Add environment variables in Vercel dashboard: - `NEXT_PUBLIC_LIVEKIT_URL` - `LIVEKIT_API_KEY` - `LIVEKIT_API_SECRET` - `GOOGLE_API_KEY` - `GOOGLE_MAPS_API` 4. Deploy ### Python Agent The Python agent needs to run on a server with GPU/CPU resources. #### Option 1: Deploy to Railway 1. Create a new project on Railway 2. Connect your GitHub repository 3. Set the root directory to `agent` 4. Add environment variables 5. Deploy #### Option 2: Deploy to Fly.io 1. Install flyctl CLI 2. Navigate to agent directory: \`\`\`bash cd agent \`\`\` 3. Create a `fly.toml`: \`\`\`toml app = "lookout-agent" [build] builder = "paketobuildpacks/builder:base" [env] PORT = "8080" [[services]] internal_port = 8080 protocol = "tcp" [[services.ports]] port = 80 handlers = ["http"] [[services.ports]] port = 443 handlers = ["tls", "http"] \`\`\` 4. Deploy: \`\`\`bash fly launch fly secrets set LIVEKIT_URL=your_url fly secrets set LIVEKIT_API_KEY=your_key fly secrets set LIVEKIT_API_SECRET=your_secret fly secrets set GOOGLE_API_KEY=your_key fly secrets set GOOGLE_MAPS_API=your_key fly deploy \`\`\` ## Mobile Optimization The web app is optimized for mobile browsers and can be added to the home screen: ### iOS (Safari) 1. Open the app in Safari 2. Tap the Share button 3. Tap "Add to Home Screen" 4. The app will now launch in full-screen mode with camera access ### Android (Chrome) 1. Open the app in Chrome 2. Tap the menu (three dots) 3. Tap "Add to Home Screen" 4. The app will now launch as a PWA ## Camera Permissions The app requires back camera access. On first launch: 1. Browser will prompt for camera permission 2. Select "Allow" to enable the back camera 3. The camera feed will appear full-screen 4. If denied, the app will show an error message ## Performance Tips - Use compute-optimized instances (AWS c6i/c7i) for the agent - Enable CDN caching for static assets - Use edge functions for API routes when possible - Monitor LiveKit connection quality - Ensure good lighting for optimal camera performance ## Security - Keep API keys secure in environment variables - Use HTTPS for all connections - Implement rate limiting on API routes - Regular security audits of dependencies - Camera feed is processed locally, not stored - Context Gathering - Use Parallel Tool Calls Where Possible - Don't stop at first match - When searching finds multiple files or components, examine ALL of them to ensure you find the right variant/version - Understand the full system before changes - Check existing patterns, parent components, utilities, schemas, and architecture to see if solutions already exist - Search systematically - Use a broad → specific → verify relationships approach to fully understand how your changes fit into the broader codebase archi [truncated — 58 more characters] ``` ### package.json ``` { "name": "lookout-web", "version": "0.1.0", "private": true, "scripts": { "build": "next build", "dev": "next dev", "lint": "next lint", "start": "next start" }, "dependencies": { "@vercel/analytics": "latest", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "jose": "latest", "livekit-client": "latest", "livekit-server-sdk": "2.14.0", "lucide-react": "^0.454.0", "next": "^15.1.6", "react": "^19.0.0", "react-dom": "^19.0.0", "server-only": "latest", "tailwind-merge": "^2.5.5", "tailwindcss-animate": "^1.0.7", "tw-animate-css": "latest" }, "devDependencies": { "@tailwindcss/postcss": "^4.1.9", "@types/dom-mediacapture-record": "^1.0.22", "@types/node": "^22", "@types/react": "^19", "@types/react-dom": "^19", "postcss": "^8.5", "tailwindcss": "^4.1.9", "typescript": "^5" } } ``` ### agent/requirements.txt ``` livekit-agents[google,images,turn-detector]~=1.2,>=1.2.0 livekit-plugins-noise-cancellation~=0.2 python-dotenv googlemaps>=4.10.0 ``` ### app/page.tsx ```typescript "use client" import { VoiceInterface } from "@/components/voice-interface" export default function Home() { return (
) } ``` ### app/layout.tsx ```typescript import type React from "react" import type { Metadata } from "next" import { Geist, Geist_Mono } from "next/font/google" import { Analytics } from "@vercel/analytics/next" import "./globals.css" const _geist = Geist({ subsets: ["latin"] }) const _geistMono = Geist_Mono({ subsets: ["latin"] }) export const metadata: Metadata = { title: "LookOut - AI Navigation Assistant", description: "Voice-powered navigation assistant for the visually impaired", viewport: "width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no", themeColor: "#000000", appleWebApp: { capable: true, statusBarStyle: "black-translucent", title: "LookOut", }, generator: 'v0.app' } export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode }>) { return ( {children} ) } ``` ### agent/main.py ```python import logging import asyncio import base64 import os from dotenv import load_dotenv import googlemaps from livekit.agents import ( Agent, AgentSession, JobContext, RoomInputOptions, WorkerOptions, cli, get_job_context, inference, ) from livekit.agents.llm import ImageContent, FunctionContext from livekit.plugins import google, noise_cancellation from livekit.plugins.turn_detector.english import EnglishModel logger = logging.getLogger("vision-assistant") load_dotenv() gmaps = googlemaps.Client(key=os.getenv("GOOGLE_MAPS_API")) class VisionAssistant(Agent): def __init__(self) -> None: self._tasks = [] self.user_location = None super().__init__( instructions=""" Role You are LookOut, a safety-first, camera-aware voice assistant for blind and low-vision users. Translate vision into brief, actionable guidance and answer spoken questions with one concise sentence. You can also provide navigation assistance using Google Maps. When users ask for directions, you can calculate routes and provide turn-by-turn guidance. Prime Directives Actionable > descriptive. Structure every reply as Action → Reason → Suggestion (compressed to one sentence). One sentence only. No filler, no meta-talk, no questions, no requests for user input or confirmation. Privacy by default. Do not store/surface identities, personal details, or raw images. No speculation about people. Be candid about uncertainty. State uncertainty and choose the safer alternative. Tone: Calm, direct, non-judgmental. Navigation Commands When user asks for directions (e.g., "How do I get to [place]?"), use the get_directions function to provide: - Distance to destination - Estimated time - Initial direction to start walking Use simple, clear language: "Head north for 500 meters, about 6 minutes." Runtime Loop (Scan Cadence) Speak only on material change or detected hazard. Heartbeat ≤ 9 words. Suppress output if the user is mid-utterance or nothing meaningful changed. Spatial Language Use clock-face directions and meters/steps. Round meters to 1 decimal. Examples: "Person at 10 o'clock, 2 m; step right toward 3 o'clock for 3 steps." / "Wall at 12; turn left to 11." Safety & Ethics No identity inference (age, gender, health), no surveillance. Do not mention that the user is blind in responses. Avoid advising movement on stairs/uneven ground unless highly confident; otherwise pause and reassess. Never instruct running, jumping, or rapid moves. For vehicles or moving hazards, default to "Unsure—please pause." unless extremely confident of clearance. Response Template Canonical (then compress to one sentence with commas/";"): . . . Examples: "No, table 1.2 m ahead; veer right toward 3 o'clock for 3 steps." "Yes, clear about 2 m; go forward for 2 steps." "Unsure—please pause, low light and occlusion." "Head north 500 meters, about 6 minutes to Starbucks." Navigation Heuristics "Step right and keep moving" for predictable, avoidable obstacles (standing person, chair, trash can) with adequate clearance. Sample lines: "Person at 10 o'clock, 2 m; step right, keep walking." "Chair 1.0 m at 12; turn left slightly, continue." """, llm=google.realtime.RealtimeModel( voice="Puck", temperature=0.8, ), fnc_ctx=FunctionContext(), ) self.fnc_ctx.ai_callable()(self.get_directions) async def on_enter(self): def _image_received_handler(reader, participant_identity): task = asyncio.create_task( self._image_received(reader, participant_identity) ) self._tasks.append(task) task.add_done_callback(lambda t: self._tasks.remove(t)) get_job_context().room.register_byte_stream_handler("test", _image_received_handler) self.session.generate_reply( instructions="Greet the user by saying: 'Hi, my name is LookOut. I'm here to look out for you. Where would you like to go today?'" ) async def _image_received(self, reader, participant_identity): logger.info("Received image from %s: '%s'", participant_identity, reader.info.name) try: image_bytes = bytes() async for chunk in reader: image_bytes += chunk chat_ctx = self.chat_ctx.copy() chat_ctx.add_message( role="user", content=[ ImageContent( image=f"data:image/png;base64,{base64.b64encode(image_bytes).decode('utf-8')}" ) ], ) await self.update_chat_ctx(chat_ctx) logger.info("Image processed and added to chat context") except Exception as e: logger.error("Error processing image: %s", e) async def get_directions( self, origin: str, destination: str, ) -> str: """ Get walking directions from origin to destination using Google Maps API. Args: origin: Starting location (address or "current location") destination: Destination address or place name Returns: A brief summary of the route with distance and duration """ try: if origin.lower() in ["current location", "here", "my location"]: if self.user_location: origin = f"{self.user_location['lat']},{self.user_location['lng']}" else: return "I don't have your current location yet." directions_result = gmaps.directions( origin, destination, mode="walking", units="metric" ) if directions_result: route = directions_result[0] [truncated — 1081 more characters] ``` ### app/api/directions/route.ts ```typescript import { type NextRequest, NextResponse } from "next/server" export async function POST(request: NextRequest) { try { const { origin, destination } = await request.json() // This will be called by the agent, not directly by the client // The agent will use the Google Maps API to get directions return NextResponse.json({ distance: "Calculating...", duration: "Calculating...", }) } catch (error) { console.error("Error getting directions:", error) return NextResponse.json({ error: "Failed to get directions" }, { status: 500 }) } } ``` ### lib/utils.ts ```typescript import { clsx, type ClassValue } from 'clsx' import { twMerge } from 'tailwind-merge' export function cn(...inputs: ClassValue[]) { return twMerge(clsx(inputs)) } ``` ### components/theme-provider.tsx ```typescript 'use client' import * as React from 'react' import { ThemeProvider as NextThemesProvider, type ThemeProviderProps, } from 'next-themes' export function ThemeProvider({ children, ...props }: ThemeProviderProps) { return {children} } ``` ### hooks/use-mobile.ts ```typescript import * as React from 'react' const MOBILE_BREAKPOINT = 768 export function useIsMobile() { const [isMobile, setIsMobile] = React.useState(undefined) React.useEffect(() => { const mql = window.matchMedia(`(max-width: ${MOBILE_BREAKPOINT - 1}px)`) const onChange = () => { setIsMobile(window.innerWidth < MOBILE_BREAKPOINT) } mql.addEventListener('change', onChange) setIsMobile(window.innerWidth < MOBILE_BREAKPOINT) return () => mql.removeEventListener('change', onChange) }, []) return !!isMobile } ``` [62 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]