# Project export: SignVision This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Helping the visually impaired navigate pedestrian traffic - Devpost: https://devpost.com/software/signvision-xrgcev - GitHub: https://github.com/Sanjith0/signvision-ar - Video: https://www.youtube.com/embed/Wc2ocYFzyRI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) โ€” Sanjith0 (29 commits), Aditya-Yan (1 commits) ## Devpost submission (written by the team) ### Inspiration Helping ## README (from the GitHub repository) # SignVision Hybrid AR ๐Ÿš€โšก **Real-time hybrid AR object detection** - Fast local YOLO + Accurate Gemini refinement! ## ๐ŸŽฏ Hybrid Architecture ``` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ COCO-SSD (Local) โ†’ Instant AR Overlays โ”‚ โ”‚ โšก 10-30 FPS โœ… Always visible โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ†“ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ Gemini (Backend) โ†’ Label Refinement โ”‚ โ”‚ ๐Ÿง  0.5 FPS โœจ Better accuracy โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` **Best of both worlds:** - โšก **Fast**: COCO-SSD gives instant visual feedback - ๐ŸŽฏ **Accurate**: Gemini refines labels in background - ๐ŸŽจ **AR**: Labels stick smoothly to objects in 3D space --- ## ๐ŸŒŸ Features - โœ… **Instant Detection** - COCO-SSD shows AR overlays immediately - โœจ **Smart Refinement** - Gemini upgrades labels for accuracy - ๐ŸŽฏ **Advanced AR Tracking** - Labels stick to objects (Google Lens style) - ๐Ÿ“ฑ **Mobile Optimized** - Works on iOS and Android - ๐Ÿ”Š **Voice Feedback** - Audio alerts for important signs - ๐Ÿ“น **Dashcam Mode** - Record video with detections - ๐ŸŒ **PWA** - Install as native app - ๐Ÿšถ **Fall Detection** - Emergency pause and recording --- ## ๐Ÿš€ Quick Start ### 1. Clone & Setup ```bash git clone https://github.com/YOUR_USERNAME/SignVision-AR.git cd SignVision-AR # Install Python dependencies pip install -r requirements.txt # Set up Gemini API key cp .env.example .env # Edit .env and add your GEMINI_API_KEY ``` ### 2. Get Gemini API Key 1. Go to [Google AI Studio](https://aistudio.google.com/apikey) 2. Click "Create API Key" 3. Copy and paste into `.env` file ### 3. Run the App ```bash # Start backend (Gemini refinement) python server.py # In a new terminal, serve frontend python -m http.server 8080 # Open http://localhost:8080 ``` --- ## ๐ŸŽฎ How It Works ### Detection Flow 1. **Camera captures frame** (1920x1080) 2. **COCO-SSD detects objects** (50-150ms) - Shows AR overlays immediately - Labels: Traffic lights, stop signs, vehicles, pedestrians 3. **Gemini refines labels** (background, every 2 seconds) - More accurate classification - Detects walk/no walk signals - Identifies specific sign types - Upgrades COCO labels with โœจ sparkle 4. **AR tracking keeps labels stuck** (Google Lens style) - IoU matching - Motion prediction - Camera motion compensation - Exponential smoothing ### Visual Indicators - **Regular box (3px)**: COCO-SSD detection - **Thick box (4px) + โœจ**: Gemini-refined label - **Dashed box**: Predicted position (object not currently detected) - **Glow effect**: Active detection --- ## ๐ŸŽฏ What It Detects ### COCO-SSD (Instant) - ๐Ÿšฆ Traffic lights - ๐Ÿ›‘ Stop signs - ๐Ÿš— Vehicles (cars, trucks, buses) - ๐Ÿšถ Pedestrians ### Gemini (Refined) - ๐Ÿšฆ Walk/Don't Walk signals - ๐Ÿ›‘ All traffic signs (stop, yield, speed limit, etc.) - โš ๏ธ Road hazards - ๐Ÿšง Construction zones - More accurate labels --- ## ๐Ÿ“ฑ Deployment ### Local Development ```bash python server.py # Backend on :8000 python -m http.server 8080 # Frontend on :8080 ``` ### Production (Split Deployment) **Option 1: Backend on Render + Frontend on Vercel** 1. **Deploy Backend** (Render/Railway/Heroku): ```bash # Push to GitHub git push origin main # On Render.com: # - New Web Service # - Connect repo # - Build: pip install -r requirements.txt # - Start: python server.py # - Add environment variable: GEMINI_API_KEY ``` 2. **Deploy Frontend** (Vercel/Netlify): ```bash # Update script.js config.apiEndpoint to your backend URL # Then deploy to Vercel vercel ``` **Option 2: Single Server** - Deploy entire app to one server - Backend serves API + static files - Simpler but less scalable --- ## โš™๏ธ Configuration Edit `script.js`: ```javascript config: { apiEndpoint: 'https://your-backend.onrender.com/analyze', processingInterval: 100, // COCO-SSD speed (10 FPS) geminiInterval: 2000, // Gemini frequency (0.5 FPS) minConfidence: 0.3 // Detection threshold } ``` Adjust for your needs: - **Faster COCO**: Lower `processingInterval` (more CPU) - **More Gemini**: Lower `geminiInterval` (more API calls) - **Less noise**: Increase `minConfidence` --- ## ๐ŸŽจ Visual Guide ``` โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ”‚ ๐Ÿ“ฑ Camera View โ”‚ โ”‚ โ”‚ โ”‚ โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”‚ โ”‚ โ”ƒ ๐Ÿšฆ Traffic โ”ƒ โ† COCO-SSD โ”‚ โ”‚ โ”ƒ Signal โ”ƒ โ”‚ โ”‚ โ”—โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”› โ”‚ โ”‚ โ”‚ โ”‚ โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”‚ โ”‚ โ”ƒ โœจ Walk Signal โ”ƒ โ† Gemini refined โ”‚ โ”‚ โ”ƒ - Green โ”ƒ (thicker glow) โ”‚ โ”‚ โ”—โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”› โ”‚ โ”‚ โ”‚ โ”‚ โ” โ”„ โ”„ โ”„ โ”„ โ”„ โ”„ โ”“ โ”‚ โ”‚ โ”† ๐Ÿ›‘ Stop Sign โ”† โ† Predicted โ”‚ โ”‚ โ”— โ”„ โ”„ โ”„ โ”„ โ”„ โ”„ โ”› (dashed) โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` --- ## ๐Ÿ“Š Performance | Metric | Value | | -------------------- | ---------------------- | | **COCO-SSD Latency** | 50-150ms | | **COCO-SSD FPS** | 10-30 FPS | | **Gemini Latency** | 500-2000ms | | **Gemini Frequency** | 0.5 FPS (every 2s) | | **AR Tracking** | Smooth 60 FPS | | **Total Model Size** | ~13 MB (COCO-SSD only) | --- ## ๐Ÿ’ฐ Cost Estimate **Gemini API** (Free tier): - 15 requests per minute - 1,500 requests per day - ~$0.01 per 100 requests after free tier **Usage**: - 0.5 requests/second = 30 requests/minute - ~1,800 requests/hour - Should stay within free tier for testing! --- ## ๐Ÿ› Troubleshooting ### COCO-SSD works but no Gemini refinement - Check backend is running (`python server.py`) - Verify `GEMINI_API_KEY` in `.env` - Check browser console for API errors - Confirm `config.apiEndpoint` is correct ### Slow performance - Increase `processingInterval` (lower FPS) - Increase `geminiInterval` (less refinement) - Use better device/browser ### Labels not sticking - Enable device motion sensors in settings - Keep device steady during initial detection - Check AR tracking parameters in code --- ## ๐ŸŽฏ Architecture Benefits | Aspect | Pure COCO-SSD | Pure Gemini | Hybrid (This!) | | -------- | ------------- | ------------------ | -------------------- | | Speed | โšก Instant | ๐Ÿข Slow | โšก Instant | | Accuracy | โœ… Good (70%) | ๐ŸŽฏ Excellent (95%) | ๐ŸŽฏ Excellent (95%) | | Offline | โœ… Yes | โŒ No | โš ๏ธ Partial | | Cost | ๐Ÿ’š Free | ๐Ÿ’ฐ Paid | ๐Ÿ’š Mostly Free | | UX | โšก Instant | โฐ Laggy | โšก Instant + Refined | --- ## ๐Ÿ“š Tech Stack - **Frontend**: Vanilla JS (PWA) - **Fast Detection**: TensorFlow.js + COCO-SSD - **Accurate Refinement**: Google Gemini 2.0 Flash - **Backend**: FastAPI (Python) - **AR Tracking**: Custom (IoU, Kalman-like prediction) - **Camera**: WebRTC getUserMedia - **Audio**: Web Speech Synthesis - **Storage**: IndexedDB --- ## ๐Ÿ”’ Privacy - **COCO-SSD**: 100% local, no data sent - **Gemini**: Images sent to Google for refinement (optional) - **No Tracking**: No analytics, no user data collection - **Works Offline**: COCO-SSD continues without internet --- ## ๐Ÿ› ๏ธ Development ```bash # Install dependencies pip install -r requirements.txt # Run with auto-reload uvicorn server:app --reload --port 8000 # Run frontend python -m http.server 8080 ``` --- ## ๐ŸŽ‰ Credits Built with: - [TensorFlow.js](https://www.tensorflow.org/js) - [COCO-SSD Model](https://github.com/tensorflow/tfjs-models/tree/master/coco-ssd) - [Google Gemini API](https://ai.google.dev/) - [FastAPI](https://fastapi.tiangolo.com/) --- **Made for accessibility.** Empowering visually impaired users with real-time hybrid AR detection. ๐ŸŒŸ **Star this repo if you find it useful!** ## Detected evidence (automated analysis) Indexed codebase: 14 recognized source files, 112 KB. - CSS (language) โ€” detected in the code - FastAPI (technology) โ€” detected in the code - HTML (language) โ€” detected in the code - JavaScript (language) โ€” detected in the code - Python (language) โ€” detected in the code ## Codebase structure (from repository index) ### Files (18 of 18) ``` .env.example .gitignore DEPLOY.md FIX_AND_START.sh HYBRID_GUIDE.md index.html INSTALL.md manifest.json QUICK_DEPLOY.md README.md requirements.txt script.js server.py START_BACKEND.sh style.css sw.js test_api_key.py WHAT_CHANGED.md ``` ### Dependencies - requirements.txt: fastapi@>=0.115.0, google-generativeai@>=0.8.0, httpx@>=0.28.1, Pillow@>=11.0.0, python-dotenv@>=1.0.0, python-multipart@>=0.0.20, uvicorn[standard]@>=0.32.0 ### Recent commits (newest first) - voices and street sign names - Wire up audio toggle button event listener - Add audio toggle button to control panel - Add audio toggle button to fix browser autoplay policy - Add extensive debug logging for audio announcements - Initialize audio announcement properties - Add audio announcements for Gemini-detected signs - Debug: Add logging and lower thresholds for generic detector - Fix: Rebalance generic detector for yellow and white signs - Fix: Make generic detector much more conservative - Update README with 3-layer detection architecture - Add generic sign detector for instant AR highlighting - Improve Gemini prompt with detailed bbox format instructions - Fix: Validate and clamp bounding box coordinates - Add debug logging for detection pipeline - Improve error handling - graceful fallback on Gemini errors - Fix: Indentation error in server.py else statement - Frontend: Add ALLOWED_CLASSES whitelist for strict filtering - STRICT: Traffic signs ONLY - block all other detections - Remove pedestrian detection - SIGNS ONLY ## Key source files (fetched from GitHub, selected and truncated for size) ### INSTALL.md ```markdown # ๐Ÿš€ Installation Guide ## The Issue You installed `google-genai` but the code needs `google-generativeai` (different package!). ## Fix Run this in your terminal (NOT in sandbox): ```bash cd /Users/sanjith/Downloads/SignVision1/SignVision-AR # Uninstall wrong package pip uninstall google-genai -y # Install correct package pip install google-generativeai --user ``` Or install all dependencies fresh: ```bash pip install -r requirements.txt --user ``` ## Setup Gemini API Key ```bash # Create .env file echo "GEMINI_API_KEY=your_actual_key_here" > .env ``` Get your key: https://aistudio.google.com/apikey ## Run ```bash # Terminal 1: Backend python server.py # Terminal 2: Frontend (already running) # Open http://localhost:8080 ``` ## Verify You should see: ``` INFO: Started server process INFO: Uvicorn running on http://0.0.0.0:8000 ``` Then in browser console: ``` โšก COCO-SSD: 75ms, Objects: 2 ๐Ÿง  Gemini refined 2 labels in 850ms ``` If you see the first line but not the second, backend isn't connected. ``` ### DEPLOY.md ```markdown # ๐Ÿš€ Deployment Guide SignVision AR is a **pure static website** - deploy it anywhere that serves HTML/CSS/JS! ## โœ… Zero Configuration Deployment No build process, no environment variables, no backend - just upload and go! --- ## ๐ŸŒ Deployment Options ### 1. Vercel (Easiest - Recommended!) **Method A: Web UI (No CLI)** 1. Go to [vercel.com/new](https://vercel.com/new) 2. Import your Git repository 3. Click "Deploy" (no configuration needed!) 4. Done! ๐ŸŽ‰ **Method B: CLI** ```bash npm install -g vercel cd SignVision-AR vercel ``` That's it! Your app is live at `https://your-app.vercel.app` --- ### 2. Netlify **Method A: Drag & Drop** 1. Go to [app.netlify.com/drop](https://app.netlify.com/drop) 2. Drag the `SignVision-AR` folder onto the page 3. Done! ๐ŸŽ‰ **Method B: CLI** ```bash npm install -g netlify-cli cd SignVision-AR netlify deploy --prod ``` --- ### 3. GitHub Pages (Free) ```bash # 1. Push to GitHub git add . git commit -m "Deploy SignVision AR" git push origin main # 2. Enable GitHub Pages # Go to: Settings โ†’ Pages โ†’ Source โ†’ main branch โ†’ Save ``` Your app will be live at: `https://YOUR_USERNAME.github.io/SignVision-AR/` --- ### 4. Cloudflare Pages 1. Go to [pages.cloudflare.com](https://pages.cloudflare.com/) 2. Connect your GitHub repo 3. Build settings: **None needed!** 4. Deploy! --- ### 5. Surge.sh (Super Fast) ```bash npm install -g surge cd SignVision-AR surge ``` Choose a subdomain and deploy in seconds! --- ### 6. Firebase Hosting ```bash npm install -g firebase-tools firebase login firebase init hosting # Select your project, use current directory, single-page app: No firebase deploy ``` --- ### 7. AWS S3 + CloudFront ```bash # Upload to S3 aws s3 sync . s3://your-bucket-name --acl public-read # Enable static website hosting in S3 console ``` --- ### 8. Any Web Server **Apache/Nginx/IIS** - Just copy files to web root! ```bash # Copy files cp -r SignVision-AR /var/www/html/signvision # Or use FTP/SFTP to upload ``` --- ## ๐Ÿ”’ Important: HTTPS Required Modern browsers require **HTTPS** for camera access! All deployment platforms above provide free HTTPS automatically. If self-hosting, use: - [Let's Encrypt](https://letsencrypt.org/) (free SSL) - [Cloudflare](https://www.cloudflare.com/) (free CDN + SSL) --- ## ๐Ÿ“ฑ Testing Your Deployment 1. Open your deployed URL 2. Allow camera permissions 3. Wait 2-3 seconds for AI model to load 4. Click "Start" and point camera at objects! --- ## ๐Ÿ› Common Issues ### Camera not working - **Cause**: Not using HTTPS - **Fix**: Deploy to a platform with HTTPS (all options above) ### Model not loading - **Cause**: CDN blocked or slow network - **Fix**: Check console for errors, try different browser ### Slow performance - **Cause**: Heavy device load - **Fix**: Close other apps, use modern browser (Chrome/Safari) --- ## ๐ŸŽฏ Performance Optimization Already optimized! But if you want to go further: ### 1. Enable Compression Most platforms do this au [truncated โ€” 1355 more characters] ``` ### requirements.txt ``` fastapi>=0.115.0 uvicorn[standard]>=0.32.0 python-dotenv>=1.0.0 google-generativeai>=0.8.0 httpx>=0.28.1 python-multipart>=0.0.20 Pillow>=11.0.0 ``` ### server.py ```python """ SignVision Gemini Backend - Label Refinement Server Provides accurate sign classification for COCO-SSD detections """ from fastapi import FastAPI, HTTPException from fastapi.middleware.cors import CORSMiddleware from fastapi.staticfiles import StaticFiles from fastapi.responses import FileResponse from pydantic import BaseModel import google.generativeai as genai import os import base64 import logging from typing import List, Dict from dotenv import load_dotenv # Load environment variables from .env file load_dotenv() # Configure logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) # Initialize FastAPI app = FastAPI(title="SignVision Gemini Refinement API") # Enable CORS for frontend access app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Initialize Gemini GEMINI_API_KEY = os.getenv("GEMINI_API_KEY") if not GEMINI_API_KEY: logger.error("GEMINI_API_KEY not found in environment") logger.error("Make sure .env file exists with: GEMINI_API_KEY=your_key") else: genai.configure(api_key=GEMINI_API_KEY) logger.info("โœ… Gemini API configured successfully") # Request model class AnalyzeRequest(BaseModel): image: str content_type: str = "image/jpeg" # Detection response model class Detection(BaseModel): label: str bbox: List[float] # [x, y, width, height] normalized 0-1 color: str confidence: float class AnalyzeResponse(BaseModel): detections: List[Detection] processing_time_ms: float @app.get("/") async def root(): return FileResponse('index.html') @app.get("/api/status") async def api_status(): return {"status": "SignVision Gemini Refinement API", "version": "2.0"} @app.post("/analyze", response_model=AnalyzeResponse) async def analyze_image(request: AnalyzeRequest): """ Analyze image with Gemini for accurate sign/hazard detection Returns refined labels for COCO-SSD detections """ import time start_time = time.time() try: # Decode base64 image image_data = base64.b64decode(request.image) # Initialize Gemini model model = genai.GenerativeModel('gemini-2.0-flash-exp') # Prepare image for Gemini image_parts = [{ "mime_type": request.content_type, "data": image_data }] # Prompt for traffic sign and hazard detection with OCR prompt = """TRAFFIC SIGNS AND ROAD SIGNS ONLY. Ignore everything else. DO NOT DETECT: โŒ People, pedestrians, humans โŒ Vehicles (cars, trucks, bikes) โŒ Objects (phones, bags, etc.) โŒ Animals ONLY DETECT THESE ROAD SIGNS: 1. **Street Name Signs** (Street Signs): - Green street signs with white text (most common) โ†’ "Street: [READ THE TEXT]" - Blue street signs with white text โ†’ "Street: [READ THE TEXT]" - White street signs with black text โ†’ "Street: [READ THE TEXT]" - Example: If you see "MAIN ST" โ†’ "Street: Main Street" 2. **Pedestrian Signs** (physical signs on poles): - White square sign with person + red circle/diagonal line โ†’ "No Walk Sign" - Yellow diamond sign with walking person symbol โ†’ "Pedestrian Crossing" 3. **Pedestrian Signals** (traffic light type): - Walk signal (green hand/person) โ†’ "Walk Signal - Green" - Don't Walk signal (red hand/person) โ†’ "Don't Walk - Red" 4. **Traffic Control Signs**: - Stop sign (red octagon) โ†’ "Stop Sign" - Yield sign โ†’ "Yield Sign" - Speed limit signs โ†’ "Speed Limit [number]" - One way, no entry, etc. 5. **Traffic Lights**: - "Traffic Light - Red" - "Traffic Light - Yellow" - "Traffic Light - Green" 6. **Warning Signs**: - Construction signs - Curve warning - Merge warning STRICT RULES: - ONLY detect mounted signs and signals - Ignore all people, even if near signs - Ignore all vehicles - Ignore all handheld objects - If you see a person silhouette ON a sign, that's a SIGN not a person - **FOR STREET SIGNS: READ AND INCLUDE THE ACTUAL TEXT** - Extract street names using OCR - Use format "Street: [exact street name text]" in the label - Example: Green sign showing "PARK AVE" โ†’ label should be "Street: Park Avenue" Response format (JSON only): [ { "label": "Street: Main Street", "bbox": [x, y, width, height], "color": "green", "confidence": 95 } ] BBOX FORMAT: - [x, y, width, height] as percentages (0-100) - x: horizontal position from LEFT edge (0 = left, 100 = right) - y: vertical position from TOP edge (0 = top, 100 = bottom) - width: horizontal size (typically 10-30) - height: vertical size (typically 10-30) - ALL VALUES MUST BE BETWEEN 0 AND 100! Example: Sign at 25% from left, 30% from top, 15% wide, 20% tall: "bbox": [25, 30, 15, 20] Colors: red (danger), yellow (caution), green (safe/street signs), blue (info), orange (construction) If no SIGNS, return: []""" # Generate content response = model.generate_content([prompt, image_parts[0]]) # Parse response try: # Extract JSON from response text = response.text.strip() # Log raw response for debugging logger.info(f"Gemini response (first 200 chars): {text[:200]}") # Remove markdown code blocks if present if text.startswith("```json"): text = text[7:] elif text.startswith("```"): text = text[3:] if text.endswith("```"): text = text[:-3] text = text.strip() # Handle empty or non-JSON responses if not text or text == "[]": logger.info("Gemini returned no detections") return AnalyzeResponse( detections=[], processing_time_ms=(time.time() - start_time) * 1000 ) import [truncated โ€” 4045 more characters] ``` ### FIX_AND_START.sh ```shell #!/bin/bash echo "๐Ÿ”ง Fixing SignVision Backend..." echo "" # Kill any process on port 8000 echo "๐Ÿ”ด Killing old process on port 8000..." lsof -ti:8000 | xargs kill -9 2>/dev/null sleep 1 # Check if .env exists if [ ! -f .env ]; then echo "โŒ .env file not found!" echo " Run: echo 'GEMINI_API_KEY=your_key_here' > .env" exit 1 fi echo "โœ… Port 8000 cleared" echo "โœ… .env file found" echo "" echo "๐Ÿš€ Starting backend on http://localhost:8000" echo " Press Ctrl+C to stop" echo "" # Start the server python3 server.py ``` ### START_BACKEND.sh ```shell #!/bin/bash echo "๐Ÿš€ Starting SignVision Backend..." echo "" # Check if .env exists if [ ! -f .env ]; then echo "โŒ .env file not found!" echo " Create it with: echo 'GEMINI_API_KEY=your_key' > .env" exit 1 fi echo "โœ… .env file found" # Check if dependencies are installed python3 -c "import google.generativeai" 2>/dev/null if [ $? -ne 0 ]; then echo "โš ๏ธ google-generativeai not installed" echo " Installing now..." pip3 install google-generativeai --user fi echo "โœ… Dependencies ready" echo "" echo "๐Ÿ”ฅ Starting backend on http://localhost:8000" echo " Press Ctrl+C to stop" echo "" # Start the server python3 server.py ``` ### test_api_key.py ```python #!/usr/bin/env python3 """ Quick test to verify Gemini API key works """ import os from dotenv import load_dotenv print("๐Ÿ” Testing Gemini API Key...\n") # Load .env file load_dotenv() # Check if key exists api_key = os.getenv("GEMINI_API_KEY") if not api_key: print("โŒ GEMINI_API_KEY not found!") print(" Create .env file with: GEMINI_API_KEY=your_key_here") exit(1) print(f"โœ… API key found: {api_key[:8]}...{api_key[-4:]}") print(f" Length: {len(api_key)} characters") # Test if key works with Gemini try: import google.generativeai as genai print("\nโœ… google.generativeai package installed") genai.configure(api_key=api_key) print("โœ… API key configured") # Try to list models (quick API test) print("\n๐Ÿงช Testing API connection...") models = genai.list_models() model_names = [m.name for m in models if 'gemini' in m.name.lower()][:3] print(f"โœ… API KEY WORKS! Connected to Gemini.") print(f" Available models: {len(model_names)}") for name in model_names: print(f" - {name}") print("\nโœจ Everything is set up correctly!") print(" You can now run: python server.py") except ImportError as e: print(f"\nโŒ Package not installed: {e}") print(" Run: pip install google-generativeai --user") except Exception as e: print(f"\nโŒ API key test failed: {e}") print(" Check if your API key is valid at:") print(" https://aistudio.google.com/apikey") ``` ### sw.js ```javascript /** * SignVision Service Worker * Handles offline caching and PWA functionality */ const CACHE_NAME = 'signvision-v1'; const STATIC_CACHE_URLS = [ '/', '/index.html', '/style.css', '/script.js', '/manifest.json', ]; // Install event - cache static resources self.addEventListener('install', (event) => { console.log('Service Worker installing...'); event.waitUntil( caches.open(CACHE_NAME) .then((cache) => { console.log('Caching static resources'); return cache.addAll(STATIC_CACHE_URLS); }) .catch((error) => { console.error('Cache failed:', error); }) ); // Force the waiting service worker to become the active service worker self.skipWaiting(); }); // Activate event - clean up old caches self.addEventListener('activate', (event) => { console.log('Service Worker activating...'); event.waitUntil( caches.keys() .then((cacheNames) => { return Promise.all( cacheNames.map((cacheName) => { if (cacheName !== CACHE_NAME) { console.log('Deleting old cache:', cacheName); return caches.delete(cacheName); } }) ); }) ); // Take control of all pages immediately event.waitUntil(clients.claim()); }); // Fetch event - serve from cache, fallback to network self.addEventListener('fetch', (event) => { const { request } = event; // Skip non-GET requests if (request.method !== 'GET') { return; } // Skip API requests and external resources const url = new URL(request.url); if (url.origin !== self.location.origin && url.hostname !== 'localhost') { return; } event.respondWith( caches.match(request) .then((cachedResponse) => { if (cachedResponse) { // Return cached version return cachedResponse; } // Fetch from network return fetch(request) .then((response) => { // Don't cache if not a valid response if (!response || response.status !== 200 || response.type !== 'basic') { return response; } // Clone the response for caching const responseToCache = response.clone(); caches.open(CACHE_NAME) .then((cache) => { cache.put(request, responseToCache); }); return response; }) .catch((error) => { console.error('Fetch failed:', error); // Return offline page if available if (request.mode === 'navigate') { return caches.match('/index.html'); } }); }) ); }); // Handle messages from the main thread self.addEventListener('message', (event) => { console.log('Service Worker received message:', event.data); if (event.data && event.data.type === 'SKIP_WAITING') { self.skipWaiting(); } if (event.data && event.data.type === 'CLEAR_CACHE') { event.waitUntil( caches.delete(CACHE_NAME) .then(() => { console.log('Cache cleared'); return self.clients.matchAll(); }) .then((clients) => { clients.forEach((client) => { client.postMessage({ type: 'CACHE_CLEARED' }); }); }) ); } }); // Background sync for recording uploads (if needed) self.addEventListener('sync', (event) => { console.log('Background sync:', event.tag); if (event.tag === 'upload-recordings') { event.waitUntil(uploadRecordings()); } }); async function uploadRecordings() { // Implementation for uploading recordings console.log('Uploading recordings...'); } ``` ### index.html ```html SignVision - Road Sign Detection
โ— ๐Ÿ“ท

SignVision

๐Ÿ”‡ Audio: Ready
``` ### style.css ```css /** * SignVision CSS - Mobile-friendly responsive design * Optimized for iPhone Safari with PWA support */ /* Reset and Base Styles */ * { margin: 0; padding: 0; box-sizing: border-box; -webkit-tap-highlight-color: transparent; } html, body { width: 100%; height: 100%; overflow: hidden; font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI, Roboto, Oxygen, Ubuntu, Cantarell, Fira Sans, Droid Sans, Helvetica Neue', sans-serif; background: #000; color: #fff; padding: 0; margin: 0; } /* App Container */ #app { width: 100%; height: 100vh; display: flex; flex-direction: column; position: relative; overflow: hidden; } /* Status Bar Header */ .status-bar { display: flex; justify-content: space-between; align-items: center; padding: 8px 15px; padding-top: calc(8px + env(safe-area-inset-top, 0px)); background: rgba(0, 0, 0, 0.95); backdrop-filter: blur(10px); z-index: 100; position: fixed; top: 0; left: 0; right: 0; box-shadow: 0 2px 10px rgba(0, 0, 0, 0.3); } .status-indicators { display: flex; gap: 8px; align-items: center; } .status-indicator { font-size: 12px; width: 20px; height: 20px; border-radius: 50%; display: flex; align-items: center; justify-content: center; } .status-bar h1 { font-size: 18px; font-weight: 600; flex: 1; text-align: center; } .icon-btn { background: none; border: none; font-size: 20px; cursor: pointer; padding: 5px; color: #fff; } /* Camera Container */ .camera-container { flex: 1; position: relative; width: 100%; overflow: hidden; background: #000; padding-top: 60px; } #video { width: 100%; height: 100%; object-fit: cover; position: absolute; top: 0; left: 0; /* No mirroring - display normally */ } #overlay { position: absolute; top: 0; left: 0; width: 100vw; height: 100%; pointer-events: none; /* No mirroring - matches video */ } #capture { display: none; } /* Loading Overlay */ .loading-overlay { position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); text-align: center; display: none; } .loading-overlay.visible { display: block; } .spinner { border: 4px solid rgba(255, 255, 255, 0.1); border-top: 4px solid #fff; border-radius: 50%; width: 40px; height: 40px; animation: spin 1s linear infinite; margin: 0 auto 10px; } @keyframes spin { 0% { transform: rotate(0deg); } 100% { transform: rotate(360deg); } } /* Control Panel */ .control-panel { display: flex; gap: 12px; padding: 15px 12px; padding-bottom: calc(15px + env(safe-area-inset-bottom, 0px)); background: rgba(0, 0, 0, 0.95); justify-content: center; flex-wrap: nowrap; position: fixed; bottom: 0; left: 0; right: 0; z-index: 200; box-shadow: 0 -4px 20px rgba(0, 0, 0, 0.5); } .control-btn { flex: 1; min-width: 85px; max-width: 140px; padding: 14px 12px; border: none; border-radius: 25px; font-size: 14px; font-weight: 600; cursor: pointer; display: flex; align-items: center; justify-content: center; gap: 6px; transition: all 0.2s; color: #fff; touch-action: manipulation; -webkit-tap-highlight-color: transparent; line-height: 1.2; } .control-btn span { white-space: nowrap; } .control-btn.primary { background: linear-gradient(135deg, #667eea 0%, #764ba2 100%); } .control-btn.secondary { background: rgba(255, 255, 255, 0.2); backdrop-filter: blur(10px); } .control-btn:active { transform: scale(0.95); } .control-btn:disabled { opacity: 0.5; cursor: not-allowed; } .control-btn.recording { background: linear-gradient(135deg, #f44336 0%, #d32f2f 100%); animation: pulse 2s infinite; } @keyframes pulse { 0%, 100% { opacity: 1; } 50% { opacity: 0.8; } } .control-btn .icon { font-size: 16px; flex-shrink: 0; } /* Detection Panel */ .detection-panel { position: fixed; bottom: 120px; left: 10px; right: 10px; background: rgba(0, 0, 0, 0.9); backdrop-filter: blur(10px); border-radius: 15px; padding: 12px; max-height: 180px; overflow-y: auto; display: none; z-index: 100; box-shadow: 0 4px 20px rgba(0, 0, 0, 0.5); } .detection-panel.visible { display: block; } .detection-panel h3 { font-size: 14px; margin-bottom: 8px; opacity: 0.7; } #detection-list { display: flex; flex-direction: column; gap: 5px; } .detection-item { padding: 8px; background: rgba(255, 255, 255, 0.1); border-radius: 8px; font-size: 12px; display: flex; justify-content: space-between; align-items: center; } .detection-item .label { font-weight: 600; } .detection-item .confidence { opacity: 0.6; font-size: 11px; } /* Audio Status */ .audio-status { position: fixed; bottom: 100px; left: 50%; transform: translateX(-50%); background: rgba(0, 0, 0, 0.9); padding: 10px 20px; border-radius: 25px; font-size: 13px; z-index: 150; display: none; box-shadow: 0 4px 15px rgba(0, 0, 0, 0.5); } .audio-status.visible { display: block; } /* Settings Modal */ .modal { display: none; position: fixed; top: 0; left: 0; width: 100%; height: 100%; background: rgba(0, 0, 0, 0.8); z-index: 1000; align-items: center; justify-content: center; } .modal.visible { display: flex; } .modal-content { background: #1a1a1a; border-radius: 20px; max-width: 90%; max-height: 80vh; overflow-y: auto; width: 400px; } .modal-header { display: flex; justify-content: space-between; align-items: center; padding: 20px; border-bottom: 1px solid rgba(255, 255, 255, 0.1); } .modal-h [truncated โ€” 2603 more characters] ``` [3 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]