# Project export: OmniScope This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: How many times today have you pulled out your phone to identify a landmark, check air quality, or see your next turn? OmniScope removes this, bringing information directly into your line of sight! - Devpost: https://devpost.com/software/omniscope - GitHub: https://github.com/Jbae04/OmniScope - Video: https://www.youtube.com/embed/8iPYce6jj0o?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 5 GitHub contributor(s) — Joseph bae (39 commits), Michael Thompson (28 commits), EnjunL (15 commits), Claude (5 commits), suvse7en (3 commits) ## Devpost submission (written by the team) ### Inspiration How many times a day do you pull out your phone to identify a landmark, check the air quality, or find your next turn? We wanted to eliminate that friction. OmniScope was born from the idea that contextual, AI-powered information should live in your line of sight, not behind a screen. We envisioned a world where exploring your surroundings feels effortless, immersive, and intelligent. ### What it does OmniScope is a wearable AI assistant that brings the digital and physical worlds together. Built with Xreal AR glasses and a Raspberry Pi, it processes real-time camera feeds, GPS data, and environmental sensor inputs to deliver instant, contextual insights. Whether you're identifying a monument, checking air quality, or navigating city streets, OmniScope overlays this information seamlessly through AR, keeping your hands free and your eyes on the world. ### How we built it We built OmniScope as a fully integrated hardware-software system. Our setup combines: Xreal AR glasses for immersive, hands-free visual display Raspberry Pi + camera module for on-device processing Claude Sonnet 4 (Vision) for real-time image understanding and contextual analysis iOS companion app for control, data syncing, and Google Maps integration The workflow is simple: point your camera, tap “Capture” on the app, and OmniScope analyzes what it sees, cross-referencing GPS data to provide detailed historical or environmental context. We also implemented turn-by-turn navigation over AR, synced wirelessly from the iOS app, and a custom interface for air quality and weather visualization. ### Challenges we ran into We faced: Accurate navigation updates on both iOS app and AR display. Accessing Weather and AQI properly, making sure they're accurate. Bypassing iOS' hotspot client isolation problem. ### Accomplishments we're proud of We achieved full hardware integration, a seamless AR user interface, and end-to-end AI scene understanding. Seeing the system identify landmarks and overlay relevant context in real time was a true “wow” moment for the team. ### What we learned We learned how critical context-aware design is for wearable tech; balancing information richness with unobtrusive presentation. We also deepened our understanding of multimodal AI (vision + text), embedded system optimization, and the importance of efficient cross-device communication. Above all, we learned that the future of computing is not in our pockets, it’s in our perception. ### What's next Next, we plan to refine OmniScope into a consumer-ready product by: Miniaturizing the hardware into a sleeker, all-in-one module Expanding context recognition with offline AI models for privacy Enhancing AR UX with dynamic overlays and gesture/voice controls Adding more functionalities, such as real-time public transit navigation Activity tracking ## README (from the GitHub repository) # OmniScope **Augmented Reality Information Display System for Smart Glasses** OmniScope is an integrated AR system that provides real-time environmental data, AI-powered scene analysis, and turn-by-turn navigation through smart glasses. The system combines an iOS companion app with a Raspberry Pi backend to deliver contextual information directly to your field of view. ## Team - **Michael Thompson** - **Joseph Bae** - **Eugenio Ramirez** - **Enjun Li** ## Overview OmniScope transforms consumer AR glasses (Xreal Air 2 Pro) into a powerful heads-up display by combining: - **iOS Companion App**: Captures GPS location, fetches weather data via Open-Meteo API, retrieves air quality from Google Cloud AQI API, and provides turn-by-turn navigation using Google Maps API - **Raspberry Pi Backend**: Processes camera images through Claude AI for scene analysis and serves as the central data hub - **AR Display**: Renders three simultaneous information panels on smart glasses with a frosted glass aesthetic ## Core Features ### 1. Environmental Monitoring - Real-time temperature and weather conditions via Open-Meteo API (free, unlimited) - Air Quality Index (AQI) with color-coded health levels via Google Cloud AQI API - GPS coordinates display - Humidity and wind speed data ### 2. AI-Powered Scene Analysis - Captures images using Raspberry Pi camera module - Analyzes architectural and historical context via Claude Sonnet 4 API - Displays landmark identification, architectural style, and historical facts - Sophisticated prompting for scholarly depth and cultural sensitivity - Auto-expires after 45 seconds to minimize visual clutter ### 3. Turn-by-Turn Navigation - Google Maps integration with real-time directions - Visual arrow indicators for maneuvers (turn, merge, straight, etc.) - Distance tracking in feet/miles with real-time GPS updates - Step-by-step instructions displayed in top-center panel - Automatic route following with current position tracking ### 4. Data Persistence - Search history tracking for quick re-navigation - Historical capture records stored locally on iPhone - Recent searches accessible from navigation interface ## System Architecture ``` ┌─────────────────────────────────────────────────┐ │ iPhone App │ │ - GPS Tracking │ │ - Open-Meteo API (weather) │ │ - Google Cloud AQI API (air quality) │ │ - Google Maps API (navigation) │ │ - User Interface │ └───────────────────┬─────────────────────────────┘ │ HTTP/JSON v ┌──────────────────┐ ┌──────────────────┐ │ Raspberry Pi │ │ Claude API │ │ - Flask Server │<------->│ Vision Analysis │ │ - Camera Module │ └──────────────────┘ │ - Data Hub │ └────────┬─────────┘ │ HDMI v ┌──────────────────┐ │ AR Glasses │ │ - Pygame Display│ │ - 3 Info Panels │ │ - Frosted Glass │ └──────────────────┘ ``` ## Technical Stack ### Hardware - **Raspberry Pi 4** (4GB+ RAM recommended) - **Pi Camera Module v2/v3** (for scene capture) - **Xreal Air 2 Pro Glasses** (AR display) - **Micro-HDMI to USB-C Adapter** (with DP Alt Mode support) - **iPhone** (iOS 16+) ### Software - **iOS App**: Swift, SwiftUI, CoreLocation, MapKit - **Backend**: Python 3.9+, Flask, Picamera2, Anthropic SDK - **Display**: Pygame, NumPy - **APIs**: Open-Meteo (weather), Google Cloud (AQI + Maps), Claude Sonnet 4 (AI) ### Key Dependencies ``` Flask 3.0.0 pygame 2.5.2 anthropic 0.39.0 python-dotenv 1.0.0 requests 2.31.0 picamera2 (system package) ``` ## Quick Start ### 1. Raspberry Pi Setup ```bash # Clone repository git clone https://github.com/Jbae04/OmniScope.git cd OmniScope # Install system dependencies sudo apt update && sudo apt upgrade -y sudo apt install -y python3-pip python3-venv python3-picamera2 sudo apt install -y libsdl2-dev libsdl2-image-dev # Create environment python3 -m venv venv source venv/bin/activate pip install -r requirements.txt # Configure API keys cp .env.example .env nano .env # Add your Claude API key # Enable camera sudo raspi-config # Navigate to: Interface Options -> Camera -> Enable ``` ### 2. Start Backend Services ```bash # Terminal 1: Start Flask server cd OmniScope source venv/bin/activate python3 src/server.py # Terminal 2: Start AR display cd OmniScope source venv/bin/activate export DISPLAY=:0 # Adjust if glasses on different display python3 src/display/ar_display.py ``` ### 3. iOS App Setup ```bash # Open Xcode project open OmniScopeApp/OmniScope.xcodeproj # Required capabilities: # - Location (Always) # - Background Modes: Location updates # Configure API keys in Config/Secrets.swift: # - Google Maps API key (for navigation and AQI) # - Raspberry Pi IP address # Build and run on iPhone ``` ### 4. Connect and Use 1. Ensure iPhone and Raspberry Pi are on the same WiFi network 2. Launch iOS app and grant location permissions 3. In Settings tab, enter Raspberry Pi IP address 4. App will automatically start sending weather data 5. Use Capture button to trigger AI scene analysis 6. Use Navigation tab to search and start turn-by-turn directions ## Display Layout The AR glasses show three panels: ``` ┌──────────────────────────────────────────────────┐ │ ┌──────────────┐ ┌───────────┐ │ │ │ AI ANALYSIS │ ┌──────────┐ │ WEATHER │ │ │ │ Landmark Info│ │NAVIGATION│ │ AQI │ │ │ │ Historical │ │ Arrows │ │ GPS │ │ │ │ Facts (45s) │ │ Distance │ └───────────┘ │ │ └──────────────┘ └──────────┘ │ │ [Transparent AR View of Real World] │ └──────────────────────────────────────────────────┘ ``` - **Top-Left**: AI analysis (appears on capture, disappears after 45s) - **Top-Center**: Navigation arrows and instructions (active during navigation) - **Top-Right**: Weather, AQI, and GPS (always visible) ## Configuration ### Environment Variables (.env) ```bash # Required CLAUDE_API_KEY=sk-ant-xxxxx # Optional (defaults provided) GPS_SERVER_HOST=0.0.0.0 GPS_SERVER_PORT=5000 DISPLAY_WIDTH=1920 DISPLAY_HEIGHT=1080 FONT_SIZE=24 CAMERA_WIDTH=1920 CAMERA_HEIGHT=1080 ``` ### iOS App Settings - **Pi IP Address**: Set in Settings tab - **Location Permissions**: Required for weather and navigation - **Background Location**: Enables continuous tracking during navigation - **Google Maps API Key**: Required for directions (configured in code) ## API Keys Required 1. **Claude API Key** (Anthropic) - Sign up at https://console.anthropic.com - Used for AI scene analysis - Pay-as-you-go pricing 2. **Google Maps API Key** (Google Cloud) - Create project at https://console.cloud.google.com - Enable: Maps SDK, Directions API, Places API, Air Quality API - Used for navigation, location search, and AQI data - Free tier: $200 credit per month 3. **Open-Meteo API** (No key required) - Completely free weather API - No registration needed - Unlimited requests ## Documentation Detailed guides available in the `docs/` directory: - **QUICKSTART.md** - Fastest path to running system - **RASPBERRY_PI_SETUP.md** - Complete Pi configuration - **NAVIGATION_SETUP.md** - Google Maps and AQI setup - **AR_DISPLAY_README.md** - Display customization - **DEBUGGING_GUIDE.md** - Troubleshooting common issues ## Project Structure ``` OmniScope/ ├── OmniScopeApp/ # iOS companion app │ └── OmniScope/ │ ├── Views/ # SwiftUI views │ ├── Services/ # Location, Weather, Network │ ├── Models/ # Data structures │ └── Config/ # API keys ├── src/ │ ├── server.py # Flask backend server │ ├── camera/ │ │ └── vision_service.py # Claude AI integration │ └── display/ │ └── ar_display.py # Pygame AR renderer ├── scripts/ # Deploymen [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 31 recognized source files, 272 KB. - Anthropic (technology) — detected in the code - Flask (technology) — detected in the code - Python (language) — detected in the code - Swift (language) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (40 of 40) ``` .gitignore docs/AR_DISPLAY_README.md docs/ARCHITECTURE.md docs/DEBUGGING_GUIDE.md docs/NAVIGATION_SETUP.md docs/QUICKSTART.md docs/RASPBERRY_PI_SETUP.md docs/SETUP.md docs/WEATHERKIT_SETUP.md OmniScopeApp/OmniScope.xcodeproj/project.pbxproj OmniScopeApp/OmniScope.xcodeproj/project.xcworkspace/contents.xcworkspacedata OmniScopeApp/OmniScope.xcodeproj/project.xcworkspace/xcuserdata/michaelthompson.xcuserdatad/UserInterfaceState.xcuserstate OmniScopeApp/OmniScope.xcodeproj/xcuserdata/michaelthompson.xcuserdatad/xcschemes/xcschememanagement.plist OmniScopeApp/OmniScope/Config/README.md OmniScopeApp/OmniScope/Info.plist OmniScopeApp/OmniScope/Info.plist.backup OmniScopeApp/OmniScope/Models/CaptureResult.swift OmniScopeApp/OmniScope/OmniScope.entitlements OmniScopeApp/OmniScope/OmniScopeApp.swift OmniScopeApp/OmniScope/Services/GoogleMapsService.swift OmniScopeApp/OmniScope/Services/LocationManager.swift OmniScopeApp/OmniScope/Services/NetworkManager.swift OmniScopeApp/OmniScope/Services/PersistenceManager.swift OmniScopeApp/OmniScope/Services/WeatherKitService.swift OmniScopeApp/OmniScope/Views/ContentView.swift OmniScopeApp/OmniScope/Views/DetailView.swift OmniScopeApp/OmniScope/Views/HistoryView.swift OmniScopeApp/OmniScope/Views/MainView.swift OmniScopeApp/OmniScope/Views/NavigationView.swift OmniScopeApp/OmniScope/Views/SettingsView.swift README.md requirements.txt scripts/debug_server.sh scripts/install_deps.sh scripts/setup_pi.sh scripts/start_omniscope.sh src/camera/test_camera.py src/camera/vision_service.py src/display/ar_display.py src/server.py ``` ### Dependencies - requirements.txt: anthropic@==0.39.0, flask@==3.0.0, httpx@==0.27.0, Pillow@==10.1.0, pygame@==2.5.2, pytest@==7.4.3, python-dateutil@==2.8.2, python-dotenv@==1.0.0, requests@==2.31.0 ### Recent commits (newest first) - Updated README - removed file - Merge branch 'main' of https://github.com/Jbae04/OmniScope - removed file - cleaning up - made restrictions stricter - Revert "Changed UI" - Changed UI - Merge branch 'main' of https://github.com/Jbae04/OmniScope - fixed text overflow and non-disapearing ui clutter - fixed some UI stuff on app - Cleaned up scripts folder - Merge branch 'main' of https://github.com/Jbae04/OmniScope - added recent searches, and fixed some pathing logic - Merge branch 'main' of https://github.com/Jbae04/OmniScope - Changed length of image analysis responses added timer - fixed build error with app - Merge branch 'main' of https://github.com/Jbae04/OmniScope - Changed vague responses from capture responses - working on location stuff ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/QUICKSTART.md ```markdown # OmniScope - Quick Start Guide ## The Fastest Way to Get Running ### On Raspberry Pi (Recommended) #### 1. Copy files to Pi ```bash # From your Mac scp -r /Users/michaelthompson/Desktop/Repos/OmniScope pi@YOUR_PI_IP:~/ ``` #### 2. Install dependencies ```bash # SSH into Pi ssh pi@YOUR_PI_IP # Install packages pip3 install flask pygame requests python-dotenv ``` #### 3. Run it! ```bash cd ~/OmniScope/scripts chmod +x start_omniscope.sh ./start_omniscope.sh ``` #### 4. Connect iOS app - Open OmniScope app on iPhone - Settings → Set Pi IP address - Enable location tracking - Done! Weather appears on glasses --- ### Testing on Mac (Before Deploying) #### Terminal 1: Start Server ```bash cd scripts python3 server.py ``` #### Terminal 2: Start Display ```bash cd scripts python3 ar_weather_display.py ``` #### Terminal 3: Test with iOS app - Launch iOS app in Xcode - Set Pi IP to `localhost` or your Mac's IP - Enable location tracking --- ## System Overview ``` iPhone → WeatherKit (FREE!) → Flask Server → AR Glasses ↓ ↓ ↓ ↓ GPS Weather/AQI Raspberry Pi Display ``` ## What You Need **Hardware:** - Raspberry Pi 4 (4GB+) - Xreal Air 2 Pro glasses - Micro-HDMI to USB-C adapter - iPhone with iOS app **Software:** - Python 3 - Flask, Pygame, Requests - iOS app with WeatherKit ## Detailed Guides - **Full Pi Setup**: See `RASPBERRY_PI_SETUP.md` - **WeatherKit Setup**: See `WEATHERKIT_SETUP.md` - **AR Display Details**: See `scripts/AR_DISPLAY_README.md` ## Quick Troubleshooting **iOS app can't connect to Pi:** - Check both on same WiFi - Verify Pi IP: `hostname -I` - Test: `curl http://PI_IP:5000/status` **No display on glasses:** - Check HDMI connection - Run: `export DISPLAY=:0` - Check: `tvservice -s` **No weather data:** - Enable location in iOS app - Check Xcode: WeatherKit capability added - Check iOS console for errors ## Auto-Start on Boot ```bash # On Pi sudo systemctl enable omniscope-server.service sudo systemctl enable omniscope-display.service sudo systemctl start omniscope-server.service sudo systemctl start omniscope-display.service ``` See `RASPBERRY_PI_SETUP.md` Part 6 for setup instructions. ## Checking Status ```bash # On Pi sudo systemctl status omniscope-server.service sudo systemctl status omniscope-display.service # View logs sudo journalctl -u omniscope-server.service -f sudo journalctl -u omniscope-display.service -f ``` ## Cost **$0** - Everything is FREE! - Apple WeatherKit: 500,000 calls/month free - No API keys needed - No subscriptions ## You're Ready! That's it! Put on your glasses and enjoy your AR weather display powered by Apple WeatherKit. ``` ### docs/WEATHERKIT_SETUP.md ```markdown # WeatherKit Integration - Quick Setup Guide Your OmniScope system now uses **Apple WeatherKit** for weather and AQI data - completely FREE with 500,000 API calls/month! ## What Changed ### Before - Required OpenWeatherMap API key (limited to 1,000 calls/day) - Required IQAir API key (limited to 10,000 calls/month) - AR display fetched data directly from APIs - Raspberry Pi needed internet for API calls ### After - Uses Apple WeatherKit (500,000 calls/month FREE) - No external API keys needed - iOS app fetches weather and sends to Pi - AR display reads from local server (faster!) - Better privacy and reliability ## Architecture ``` iOS App (iPhone) └─> Gets GPS location └─> Fetches weather/AQI from WeatherKit (Apple servers) └─> Sends to Raspberry Pi server via HTTP └─> AR display reads from server └─> Renders to Xreal glasses ``` ## Setup Steps ### 1. Enable WeatherKit in Xcode 1. Open: `OmniScopeApp/OmniScopeApp/OmniScope/OmniScope.xcodeproj` 2. Select your project target 3. Go to "Signing & Capabilities" tab 4. Click "+ Capability" button 5. Search for and add "WeatherKit" 6. Save and build the project **Note:** You'll need an Apple Developer account (free tier works!) ### 2. Add WeatherKitService to Xcode The file `WeatherKitService.swift` has been created at: `OmniScopeApp/OmniScopeApp/OmniScope/OmniScope/Services/WeatherKitService.swift` Add it to your Xcode project: 1. Right-click "Services" folder in Xcode 2. Choose "Add Files to OmniScope..." 3. Select `WeatherKitService.swift` 4. Make sure "Copy items if needed" is checked 5. Click "Add" ### 3. Files Modified The following files have been automatically updated: **iOS App:** - `Models/CaptureResult.swift` - Added weather/AQI to location updates - `Services/WeatherKitService.swift` - NEW: Fetches from Apple WeatherKit - `Services/NetworkManager.swift` - Now sends weather/AQI with location **Raspberry Pi:** - `scripts/server.py` - Stores weather/AQI, added `/get_current_weather` endpoint - `scripts/ar_weather_display.py` - Fetches from server instead of APIs - `.env.example` - Removed API key requirements **Documentation:** - `scripts/AR_DISPLAY_README.md` - Updated with WeatherKit instructions - `WEATHERKIT_SETUP.md` - This file! ### 4. Test the System #### Step 1: Start the server ```bash cd scripts python server.py ``` Should see: ``` Starting OmniScope server... Running on http://0.0.0.0:5000 ``` #### Step 2: Start the AR display ```bash # In a new terminal cd scripts python ar_weather_display.py ``` Should see: ``` ============================================================ OmniScope AR Weather & AQI Display Powered by Apple WeatherKit (via iOS app) ============================================================ ``` #### Step 3: Run the iOS app 1. Build and run in Xcode 2. Go to Settings tab 3. Set Pi IP address (e.g., `192.168.1.100`) 4. Test connection (should see green checkmark) 5. Enable location tracking 6. Go to Ma [truncated — 2658 more characters] ``` ### requirements.txt ``` # Core anthropic==0.39.0 httpx==0.27.0 python-dotenv==1.0.0 # Web server flask==3.0.0 # Display pygame==2.5.2 # Image processing Pillow==10.1.0 # HTTP requests requests==2.31.0 # Testing pytest==7.4.3 # Utilities python-dateutil==2.8.2 ``` ### src/server.py ```python from flask import Flask, request, jsonify import json from datetime import datetime from camera import vision_service # <--- IMPORT YOUR NEW SERVICE # Initialize the Flask app app = Flask(__name__) # Global state to store latest data current_state = { "latitude": None, "longitude": None, "weather": None, "aqi": None, "analysis": None, # <--- ADDED for AI data "directions": None, # <--- ADDED for navigation data "current_step": None, # <--- ADDED for real-time navigation "last_update": None, "last_analysis_update": None, # <--- ADDED "last_directions_update": None, # <--- ADDED for directions "last_step_update": None # <--- ADDED for current step } # --- API Endpoints for your OmniScope App --- @app.route("/status", methods=["GET"]) def get_status(): """Endpoint for the iOS app to check if the Pi is online.""" print("iOS app checked status. Responding 'online'.") return jsonify({"status": "online"}) @app.route("/update_location", methods=["POST"]) def update_location(): """Endpoint for the iOS app to send continuous GPS updates with weather/AQI.""" data = request.json print("\n" + "="*60) print("RECEIVED LOCATION DATA FROM iOS APP") print(json.dumps(data, indent=2)) # Store in global state for AR display to access current_state["latitude"] = data.get("latitude") current_state["longitude"] = data.get("longitude") current_state["weather"] = data.get("weather") current_state["aqi"] = data.get("aqi") current_state["last_update"] = datetime.now().isoformat() print("Location/Weather data stored in server state") print("="*60 + "\n") return jsonify({"status": "location_received"}), 200 @app.route("/get_current_weather", methods=["GET"]) def get_current_weather(): """Endpoint for the AR display to fetch the latest weather/AQI data.""" print("\n AR Display requested current weather") if current_state["weather"] is None: print("No weather data available yet - waiting for iOS app") return jsonify({"status": "no_data"}), 404 print("Sending weather data to AR display") return jsonify({ "status": "success", "latitude": current_state["latitude"], "longitude": current_state["longitude"], "weather": current_state["weather"], "aqi": current_state["aqi"], "last_update": current_state["last_update"] }), 200 # --- NEW ENDPOINT FOR AR DISPLAY --- @app.route("/get_current_analysis", methods=["GET"]) def get_current_analysis(): """ Endpoint for the AR display to fetch the latest AI analysis data. """ print("\n AR Display requested current analysis") if current_state["analysis"] is None: print("No analysis data available yet - waiting for capture trigger") return jsonify({"status": "no_data"}), 404 print("Sending analysis data to AR display") return jsonify({ "status": "success", "analysis": current_state["analysis"], "last_analysis_update": current_state["last_analysis_update"] }), 200 # --- END NEW ENDPOINT --- # --- NAVIGATION/DIRECTIONS ENDPOINTS --- @app.route("/update_directions", methods=["POST"]) def update_directions(): """ Endpoint for the iOS app to send navigation directions data. Stores the directions in the global state for the AR display to access. """ data = request.json print("\n" + "="*60) print(" RECEIVED DIRECTIONS DATA FROM iOS APP") print(json.dumps(data, indent=2)) # Store in global state for AR display to access current_state["directions"] = { "destination": data.get("destination"), "address": data.get("address"), "distance": data.get("distance"), # total distance in meters "duration": data.get("duration"), # total duration in seconds "steps": data.get("steps"), # array of turn-by-turn steps "timestamp": data.get("timestamp") } current_state["last_directions_update"] = datetime.now().isoformat() print(f" Navigation to: {data.get('destination')}") print(f"📏 Distance: {data.get('distance')} meters") print(f"⏱️ Duration: {data.get('duration')} seconds") print(f" Steps: {len(data.get('steps', []))} turn-by-turn instructions") print("Directions data stored in server state") print("="*60 + "\n") return jsonify({"status": "directions_received"}), 200 @app.route("/get_current_directions", methods=["GET"]) def get_current_directions(): """ Endpoint for the AR display to fetch the latest navigation directions. Returns the full directions data including turn-by-turn steps. """ print("\n AR Display requested current directions") if current_state["directions"] is None: print("No directions data available yet - waiting for iOS app") return jsonify({"status": "no_data"}), 404 print("Sending directions data to AR display") return jsonify({ "status": "success", "directions": current_state["directions"], "last_directions_update": current_state["last_directions_update"] }), 200 @app.route("/update_current_step", methods=["POST"]) def update_current_step(): """ Endpoint for the iOS app to send real-time current step updates during navigation. Updates the current step index and distance information. Now includes GPS coordinates for real-time distance calculation on display. """ data = request.json print("\n" + "="*60) print(" RECEIVED CURRENT STEP UPDATE FROM iOS APP") print(json.dumps(data, indent=2)) # Store current step information with GPS coordinates current_state["current_step"] = { "step_index": data.get("step_index"), "total_steps": data.get("total_steps"), "instruction": data.get("instruction"), "distance_to_step": data.get("distance_to_step"), "step_distance": data.get("step_distance"), "step_duration": d [truncated — 5298 more characters] ``` ### scripts/debug_server.sh ```shell #!/bin/bash # # Debug Server - Shows all iOS app data in real-time # Use this to see exactly what data is being sent from the iOS app # echo "============================================================" echo "🔍 OmniScope Debug Server" echo "============================================================" echo "" # Get the directory where this script is located SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" cd "$SCRIPT_DIR" # Get IP address IP_ADDRESS=$(hostname -I | awk '{print $1}' 2>/dev/null || echo "localhost") echo "📡 Server will listen on: http://$IP_ADDRESS:5000" echo " or http://0.0.0.0:5000 (all interfaces)" echo "" echo "💡 This debug mode will show:" echo " ✅ Full JSON payload from iOS app" echo " ✅ Parsed location coordinates" echo " ✅ Weather data (if available)" echo " ✅ AQI data (if available)" echo " ✅ Requests from AR display" echo "" echo "📱 How to test:" echo " 1. Make sure iOS app is configured with Pi IP: $IP_ADDRESS" echo " 2. Enable location tracking in iOS app" echo " 3. Watch the data appear below in real-time!" echo "" echo "🛑 Press Ctrl+C to stop" echo "============================================================" echo "" # Run the server in foreground with all output visible python3 server.py ``` ### scripts/install_deps.sh ```shell #!/bin/bash # # OmniScope Dependencies Installer # Installs all required packages for OmniScope on Raspberry Pi # set -e # Exit on error echo "============================================================" echo "=' OmniScope Dependencies Installer" echo "============================================================" echo "" # Check if running on Raspberry Pi if ! command -v raspi-config &> /dev/null; then echo "� Warning: This script is designed for Raspberry Pi" echo " Continuing anyway, but some features may not work" echo "" fi # Update system packages echo "=� Updating system packages..." sudo apt update echo " System packages updated" echo "" # Install Python 3 and pip if not already installed echo "= Installing Python 3 and pip..." sudo apt install -y python3 python3-pip python3-venv echo " Python 3 installed" echo "" # Install system dependencies for Python packages echo "=� Installing system dependencies..." sudo apt install -y \ python3-dev \ build-essential \ libssl-dev \ libffi-dev \ libjpeg-dev \ zlib1g-dev \ libfreetype6-dev \ liblcms2-dev \ libopenjp2-7 \ libtiff5 echo " System dependencies installed" echo "" # Install Pygame dependencies echo "<� Installing Pygame dependencies..." sudo apt install -y \ libsdl2-dev \ libsdl2-image-dev \ libsdl2-mixer-dev \ libsdl2-ttf-dev \ libportmidi-dev echo " Pygame dependencies installed" echo "" # Install PiCamera2 (Raspberry Pi specific) echo "=� Installing PiCamera2..." sudo apt install -y python3-picamera2 echo " PiCamera2 installed" echo "" # Get project root directory SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" PROJECT_ROOT="$( cd "$SCRIPT_DIR/.." && pwd )" cd "$PROJECT_ROOT" # Upgrade pip echo " Upgrading pip..." python3 -m pip install --upgrade pip echo " pip upgraded" echo "" # Install Python packages from requirements.txt echo "=� Installing Python packages from requirements.txt..." if [ -f "requirements.txt" ]; then python3 -m pip install -r requirements.txt echo " Python packages installed" else echo "L Error: requirements.txt not found" exit 1 fi echo "" # Verify installations echo "= Verifying installations..." python3 -c "import flask; print('  Flask:', flask.__version__)" python3 -c "import pygame; print('  Pygame:', pygame.__version__)" python3 -c "import anthropic; print('  Anthropic SDK:', anthropic.__version__)" python3 -c "import requests; print('  Requests:', requests.__version__)" python3 -c "import dotenv; print('  python-dotenv: OK')" python3 -c "from PIL import Image; print('  Pillow: OK')" # Check PiCamera2 (may not be available on non-Pi systems) if python3 -c "from picamera2 import Picamera2" 2>/dev/null; then echo "  PiCamera2: OK" else echo " � PiCamera2: Not available (OK if not on Raspberry Pi)" fi echo "" echo "============================================================" echo " All dependencies installed successfully!" echo "============================================================" echo "" echo "Next steps:" echo " 1. Run setup_pi.sh to configure OmniScope" echo " 2. Add your API keys to .env file" echo " 3. Start OmniScope with: ./scripts/start_omniscope.sh" echo "" ``` ### scripts/start_omniscope.sh ```shell #!/bin/bash # # OmniScope Quick Start Script for Raspberry Pi # This script starts both the server and AR display # echo "============================================================" echo "OmniScope Unified AR Display - Quick Start" echo "============================================================" echo "" # Check if we're on a Raspberry Pi if ! command -v raspi-config &> /dev/null; then echo "⚠️ Warning: This doesn't look like a Raspberry Pi" echo " You can still run for testing, but display may not work properly" echo "" fi # Get the project root directory (one level up from scripts) SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" PROJECT_ROOT="$( cd "$SCRIPT_DIR/.." && pwd )" cd "$PROJECT_ROOT" # Check if Python 3 is installed if ! command -v python3 &> /dev/null; then echo "❌ Error: Python 3 is not installed" echo " Install it with: sudo apt install python3" exit 1 fi # Check if required packages are installed echo "🔍 Checking dependencies..." python3 -c "import flask, pygame, requests" 2>/dev/null if [ $? -ne 0 ]; then echo "❌ Error: Missing Python packages" echo " Install them with: pip3 install flask pygame requests python-dotenv" exit 1 fi echo "✅ Dependencies OK" echo "" # Get Pi's IP address IP_ADDRESS=$(hostname -I | awk '{print $1}') echo "📡 Your Pi's IP address: $IP_ADDRESS" echo " Configure this in your iOS app Settings" echo "" # Check if DISPLAY is set if [ -z "$DISPLAY" ]; then export DISPLAY=:0 echo "📺 Set DISPLAY=:0 for AR glasses" fi echo "" # Function to cleanup on exit cleanup() { echo "" echo "🛑 Shutting down OmniScope..." kill $SERVER_PID 2>/dev/null kill $DISPLAY_PID 2>/dev/null echo "✅ Stopped" exit 0 } trap cleanup SIGINT SIGTERM # Start the server echo "🚀 Starting OmniScope server..." python3 src/server.py > server.log 2>&1 & SERVER_PID=$! # Wait a moment for server to start sleep 2 # Check if server started successfully if ! kill -0 $SERVER_PID 2>/dev/null; then echo "❌ Server failed to start. Check server.log for errors" exit 1 fi echo "✅ Server running (PID: $SERVER_PID)" echo " Listening on http://0.0.0.0:5000" echo "" # Start the unified AR display echo "🎮 Starting unified AR display..." python3 src/display/ar_display.py > display.log 2>&1 & DISPLAY_PID=$! # Wait a moment for display to start sleep 2 # Check if display started successfully if ! kill -0 $DISPLAY_PID 2>/dev/null; then echo "❌ Display failed to start. Check display.log for errors" kill $SERVER_PID 2>/dev/null exit 1 fi echo "✅ AR display running (PID: $DISPLAY_PID)" echo "" echo "============================================================" echo "✅ OmniScope is running!" echo "============================================================" echo "" echo "Next steps:" echo "1. Open iOS app on your iPhone" echo "2. Go to Settings tab" echo "3. Set Pi IP to: $IP_ADDRESS" echo "4. Test connection (should see green ✓)" echo "5. Enable location tracking" echo "6. AR glasses will show:" echo " • Weather & AQI (top-right)" echo " • AI Landmark Analysis (top-left)" echo " • Navigation Directions (top-middle, when active)" echo "" echo "Logs:" echo " Server: tail -f $PROJECT_ROOT/server.log" echo " Display: tail -f $PROJECT_ROOT/display.log" echo "" echo "Press Ctrl+C to stop" echo "" # Wait for both processes wait $SERVER_PID $DISPLAY_PID ``` ### scripts/setup_pi.sh ```shell #!/bin/bash # # OmniScope Setup Script for Raspberry Pi # Configures the Raspberry Pi for OmniScope # set -e # Exit on error echo "============================================================" echo "� OmniScope Raspberry Pi Setup" echo "============================================================" echo "" # Check if running on Raspberry Pi if ! command -v raspi-config &> /dev/null; then echo "� Warning: This script is designed for Raspberry Pi" echo " Some configurations may not apply" echo "" fi # Get project root directory SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" PROJECT_ROOT="$( cd "$SCRIPT_DIR/.." && pwd )" cd "$PROJECT_ROOT" # Make all scripts executable echo "=' Making scripts executable..." chmod +x scripts/*.sh echo " Scripts are now executable" echo "" # Check if .env file exists, if not create template if [ ! -f ".env" ]; then echo "=� Creating .env template file..." cat > .env << 'EOF' # OmniScope Configuration # Replace with your actual API keys # Anthropic Claude API Key (Required for vision analysis) CLAUDE_API_KEY=your_anthropic_api_key_here # Google Maps API Key (Required for navigation and AQI) GOOGLE_MAPS_API_KEY=your_google_maps_api_key_here # Optional: Server configuration # PI_IP_ADDRESS=192.168.1.x # SERVER_PORT=5000 EOF echo " Created .env template" echo "� IMPORTANT: Edit .env file and add your API keys!" echo "" else echo " .env file already exists" echo "" fi # Check if API keys are configured if grep -q "your_anthropic_api_key_here" .env 2>/dev/null; then echo "� WARNING: .env file still has placeholder values" echo " You need to edit .env and add your API keys" echo "" fi # Get Pi's IP address IP_ADDRESS=$(hostname -I | awk '{print $1}' 2>/dev/null || echo "unknown") echo "=� Raspberry Pi Network Information:" echo " IP Address: $IP_ADDRESS" echo " Hostname: $(hostname)" echo "" echo " Configure this IP in your iOS app Settings" echo "" # Enable camera if on Raspberry Pi if command -v raspi-config &> /dev/null; then echo "=� Checking camera configuration..." # Check if camera is already enabled if ! vcgencmd get_camera | grep -q "detected=1"; then echo "� Camera not detected. Enabling camera..." echo " You may need to run: sudo raspi-config" echo " Then: Interface Options � Camera � Enable" echo "" else echo " Camera is enabled" echo "" fi fi # Set up display environment echo "=� Configuring display settings..." if [ -z "$DISPLAY" ]; then echo "export DISPLAY=:0" >> ~/.bashrc export DISPLAY=:0 echo " DISPLAY environment variable set to :0" else echo " DISPLAY already configured: $DISPLAY" fi echo "" # Create log directory if needed if [ ! -d "logs" ]; then mkdir -p logs echo " Created logs directory" else echo " Logs directory exists" fi echo "" # Test Python imports echo "= Testing Python environment..." if python3 -c "import flask, pygame, anthropic, requests, dotenv" 2>/dev/null; then echo " All required Python packages are installed" else echo "L Some Python packages are missing" echo " Run: ./scripts/install_deps.sh" exit 1 fi echo "" # Summary echo "============================================================" echo " OmniScope setup complete!" echo "============================================================" echo "" echo "Configuration Summary:" echo " =� Pi IP Address: $IP_ADDRESS" echo " =� Project Root: $PROJECT_ROOT" echo " =� Config File: $PROJECT_ROOT/.env" echo "" echo "Next steps:" echo " 1. Edit .env file and add your API keys:" echo " nano .env" echo "" echo " 2. Configure iOS app with Pi IP: $IP_ADDRESS" echo "" echo " 3. Start OmniScope:" echo " ./scripts/start_omniscope.sh" echo "" echo " 4. Or run debug mode to see all data:" echo " ./scripts/debug_server.sh" echo "" echo "For help, see: README.md" echo "" ``` ### OmniScopeApp/OmniScope/OmniScopeApp.swift ```swift // // OmniScopeApp.swift // OmniScope // // Main app entry point // import SwiftUI @main struct OmniScopeApp: App { // Initialize shared managers @StateObject private var networkManager = NetworkManager.shared @StateObject private var locationManager = LocationManager.shared @StateObject private var persistenceManager = PersistenceManager.shared var body: some Scene { WindowGroup { ContentView() .onAppear { // Request initial location permissions locationManager.requestPermissions() // Check Pi connection status Task { await networkManager.checkStatus() } } } } } ``` ### src/camera/test_camera.py ```python #!/usr/bin/env python3 """ Simple camera test script for Pi-Sight AR glasses Tests camera capture and displays preview on Xreal glasses Press SPACE to capture, ESC to quit """ import pygame import io from picamera2 import Picamera2 from PIL import Image import time def main(): print("Initializing camera test...") # Initialize Pygame display pygame.init() # Set fullscreen for AR glasses screen = pygame.display.set_mode((1920, 1080), pygame.FULLSCREEN) pygame.display.set_caption("Camera Test") # Initialize camera print("Starting camera...") camera = Picamera2() # Configure camera for preview (lower res for speed) preview_config = camera.create_preview_configuration( main={"size": (1920, 1080), "format": "RGB888"} ) camera.configure(preview_config) # CRITICAL: Disable IR/NoIR mode and set proper controls camera.set_controls({ "AeEnable": True, # Auto exposure "AwbEnable": True, # Auto white balance "AwbMode": 1, # Auto white balance mode (daylight) "Brightness": 0.0, # Normal brightness "Contrast": 1.0, # Normal contrast "Saturation": 1.0 # Normal saturation }) camera.start() # Wait for camera to warm up time.sleep(2) print("Camera ready!") print("\nControls:") print(" SPACE - Capture and freeze frame") print(" ESC - Quit") clock = pygame.time.Clock() running = True frozen_frame = None try: while running: # Handle events for event in pygame.event.get(): if event.type == pygame.QUIT: running = False elif event.type == pygame.KEYDOWN: if event.key == pygame.K_ESCAPE: running = False elif event.key == pygame.K_SPACE: # Capture and freeze frame print("Capturing frame...") frozen_frame = camera.capture_array() print("Frame captured and frozen!") # Display either frozen frame or live feed if frozen_frame is not None: # Show frozen frame frame = frozen_frame else: # Show live feed frame = camera.capture_array() # Convert numpy array to pygame surface # Rotate if needed (depends on camera orientation) surface = pygame.surfarray.make_surface(frame.swapaxes(0, 1)) # Scale to fit screen if needed surface = pygame.transform.scale(surface, (1920, 1080)) # Draw to screen screen.blit(surface, (0, 0)) # Add status text font = pygame.font.Font(None, 36) if frozen_frame is not None: status_text = font.render("FROZEN - Press SPACE to unfreeze", True, (0, 255, 0)) else: status_text = font.render("LIVE FEED - Press SPACE to capture", True, (255, 255, 255)) # Add background for text readability text_rect = status_text.get_rect() text_rect.topleft = (10, 10) pygame.draw.rect(screen, (0, 0, 0), text_rect.inflate(20, 10)) screen.blit(status_text, (10, 10)) pygame.display.flip() clock.tick(30) # 30 FPS except KeyboardInterrupt: print("\nTest interrupted by user") except Exception as e: print(f"\nError: {e}") import traceback traceback.print_exc() finally: print("\nCleaning up...") camera.stop() pygame.quit() print("Test complete!") if __name__ == "__main__": main() ``` [18 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]