OmniScope

Augmented Reality Information Display System for Smart Glasses

OmniScope is an integrated AR system that provides real-time environmental data, AI-powered scene analysis, and turn-by-turn navigation through smart glasses. The system combines an iOS companion app with a Raspberry Pi backend to deliver contextual information directly to your field of view.

Team

• Michael Thompson
• Joseph Bae
• Eugenio Ramirez
• Enjun Li

Overview

OmniScope transforms consumer AR glasses (Xreal Air 2 Pro) into a powerful heads-up display by combining:

• iOS Companion App: Captures GPS location, fetches weather data via Open-Meteo API, retrieves air quality from Google Cloud AQI API, and provides turn-by-turn navigation using Google Maps API
• Raspberry Pi Backend: Processes camera images through Claude AI for scene analysis and serves as the central data hub
• AR Display: Renders three simultaneous information panels on smart glasses with a frosted glass aesthetic

Core Features

1. Environmental Monitoring

• Real-time temperature and weather conditions via Open-Meteo API (free, unlimited)
• Air Quality Index (AQI) with color-coded health levels via Google Cloud AQI API
• GPS coordinates display
• Humidity and wind speed data

2. AI-Powered Scene Analysis

• Captures images using Raspberry Pi camera module
• Analyzes architectural and historical context via Claude Sonnet 4 API
• Displays landmark identification, architectural style, and historical facts
• Sophisticated prompting for scholarly depth and cultural sensitivity
• Auto-expires after 45 seconds to minimize visual clutter

3. Turn-by-Turn Navigation

• Google Maps integration with real-time directions
• Visual arrow indicators for maneuvers (turn, merge, straight, etc.)
• Distance tracking in feet/miles with real-time GPS updates
• Step-by-step instructions displayed in top-center panel
• Automatic route following with current position tracking

4. Data Persistence

• Search history tracking for quick re-navigation
• Historical capture records stored locally on iPhone
• Recent searches accessible from navigation interface

System Architecture

┌─────────────────────────────────────────────────┐
│              iPhone App                         │
│  - GPS Tracking                                 │
│  - Open-Meteo API (weather)                     │
│  - Google Cloud AQI API (air quality)           │
│  - Google Maps API (navigation)                 │
│  - User Interface                               │
└───────────────────┬─────────────────────────────┘
                    │ HTTP/JSON
                    v
┌──────────────────┐         ┌──────────────────┐
│  Raspberry Pi    │         │   Claude API     │
│  - Flask Server  │<------->│  Vision Analysis │
│  - Camera Module │         └──────────────────┘
│  - Data Hub      │
└────────┬─────────┘
         │ HDMI
         v
┌──────────────────┐
│   AR Glasses     │
│  - Pygame Display│
│  - 3 Info Panels │
│  - Frosted Glass │
└──────────────────┘

Technical Stack

Hardware

• Raspberry Pi 4 (4GB+ RAM recommended)
• Pi Camera Module v2/v3 (for scene capture)
• Xreal Air 2 Pro Glasses (AR display)
• Micro-HDMI to USB-C Adapter (with DP Alt Mode support)
• iPhone (iOS 16+)

Software

• iOS App: Swift, SwiftUI, CoreLocation, MapKit
• Backend: Python 3.9+, Flask, Picamera2, Anthropic SDK
• Display: Pygame, NumPy
• APIs: Open-Meteo (weather), Google Cloud (AQI + Maps), Claude Sonnet 4 (AI)

Key Dependencies

Flask 3.0.0
pygame 2.5.2
anthropic 0.39.0
python-dotenv 1.0.0
requests 2.31.0
picamera2 (system package)

Quick Start

1. Raspberry Pi Setup

# Clone repository
git clone https://github.com/Jbae04/OmniScope.git
cd OmniScope

# Install system dependencies
sudo apt update && sudo apt upgrade -y
sudo apt install -y python3-pip python3-venv python3-picamera2
sudo apt install -y libsdl2-dev libsdl2-image-dev

# Create environment
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Configure API keys
cp .env.example .env
nano .env  # Add your Claude API key

# Enable camera
sudo raspi-config
# Navigate to: Interface Options -> Camera -> Enable

2. Start Backend Services

# Terminal 1: Start Flask server
cd OmniScope
source venv/bin/activate
python3 src/server.py

# Terminal 2: Start AR display
cd OmniScope
source venv/bin/activate
export DISPLAY=:0  # Adjust if glasses on different display
python3 src/display/ar_display.py

3. iOS App Setup

# Open Xcode project
open OmniScopeApp/OmniScope.xcodeproj

# Required capabilities:
# - Location (Always)
# - Background Modes: Location updates

# Configure API keys in Config/Secrets.swift:
# - Google Maps API key (for navigation and AQI)
# - Raspberry Pi IP address

# Build and run on iPhone

4. Connect and Use

1. Ensure iPhone and Raspberry Pi are on the same WiFi network
2. Launch iOS app and grant location permissions
3. In Settings tab, enter Raspberry Pi IP address
4. App will automatically start sending weather data
5. Use Capture button to trigger AI scene analysis
6. Use Navigation tab to search and start turn-by-turn directions

Display Layout

The AR glasses show three panels:

┌──────────────────────────────────────────────────┐
│  ┌──────────────┐                 ┌───────────┐ │
│  │ AI ANALYSIS  │   ┌──────────┐  │  WEATHER  │ │
│  │ Landmark Info│   │NAVIGATION│  │  AQI      │ │
│  │ Historical   │   │  Arrows  │  │  GPS      │ │
│  │ Facts (45s)  │   │ Distance │  └───────────┘ │
│  └──────────────┘   └──────────┘                 │
│       [Transparent AR View of Real World]        │
└──────────────────────────────────────────────────┘

• Top-Left: AI analysis (appears on capture, disappears after 45s)
• Top-Center: Navigation arrows and instructions (active during navigation)
• Top-Right: Weather, AQI, and GPS (always visible)

Configuration

Environment Variables (.env)

# Required
CLAUDE_API_KEY=sk-ant-xxxxx

# Optional (defaults provided)
GPS_SERVER_HOST=0.0.0.0
GPS_SERVER_PORT=5000
DISPLAY_WIDTH=1920
DISPLAY_HEIGHT=1080
FONT_SIZE=24
CAMERA_WIDTH=1920
CAMERA_HEIGHT=1080

iOS App Settings

• Pi IP Address: Set in Settings tab
• Location Permissions: Required for weather and navigation
• Background Location: Enables continuous tracking during navigation
• Google Maps API Key: Required for directions (configured in code)

API Keys Required

1. Claude API Key (Anthropic)

   • Sign up at https://console.anthropic.com
   • Used for AI scene analysis
   • Pay-as-you-go pricing

2. Google Maps API Key (Google Cloud)

   • Create project at https://console.cloud.google.com
   • Enable: Maps SDK, Directions API, Places API, Air Quality API
   • Used for navigation, location search, and AQI data
   • Free tier: $200 credit per month

3. Open-Meteo API (No key required)

   • Completely free weather API
   • No registration needed
   • Unlimited requests

Documentation

Detailed guides available in the docs/ directory:

• QUICKSTART.md - Fastest path to running system
• RASPBERRY_PI_SETUP.md - Complete Pi configuration
• NAVIGATION_SETUP.md - Google Maps and AQI setup
• AR_DISPLAY_README.md - Display customization
• DEBUGGING_GUIDE.md - Troubleshooting common issues

Project Structure

OmniScope/
├── OmniScopeApp/              # iOS companion app
│   └── OmniScope/
│       ├── Views/             # SwiftUI views
│       ├── Services/          # Location, Weather, Network
│       ├── Models/            # Data structures
│       └── Config/            # API keys
├── src/
│   ├── server.py              # Flask backend server
│   ├── camera/
│   │   └── vision_service.py  # Claude AI integration
│   └── display/
│       └── ar_display.py      # Pygame AR renderer
├── scripts/                   # Deployment scripts
├── docs/                      # Documentation
└── requirements.txt           # Python dependencies

Customization

Adjusting Analysis Display Time

Edit src/display/ar_display.py:

ANALYSIS_DISPLAY_DURATION = 45  # seconds

Changing UI Colors

Edit color scheme in ar_display.py:

COLOR_PANEL_BG = (40, 40, 45, 150)      # Dark translucent
COLOR_PANEL_BORDER = (80, 80, 85, 100)  # Subtle gray
COLOR_PRIMARY_TEXT = (255, 255, 255)    # White text

Navigation Arrow Images

Custom arrow images stored in src/display/navigation_arrows/:

• turn_left.png, turn_right.png
• slight_left.png, slight_right.png
• straight.png, merge.png, etc.

Performance

• CPU Usage: ~10-15% on Raspberry Pi 4
• Memory: ~150-200 MB total
• Display FPS: 30 FPS
• Network Usage: ~2-3 KB per update
• Battery Life: ~6-8 hours with 20,000mAh power bank

Known Limitations

• Camera-based analysis requires sufficient lighting
• AQI data availability varies by region (WeatherKit limitation)
• Navigation requires active internet connection
• Display resolution locked to AR glasses native resolution (1920x1080)
• Real-time distance calculation depends on GPS accuracy

Troubleshooting

Display Issues

# Check HDMI output
tvservice -s

# Verify display number
DISPLAY=:0 xrandr

Server Connection Issues

# Test server status
curl http://RASPBERRY_PI_IP:5000/status

# Check firewall
sudo ufw status

Camera Problems

# Verify camera detection
vcgencmd get_camera
# Should show: supported=1 detected=1

# Test camera
libcamera-hello

iOS App Issues

• Check Pi IP address in Settings tab
• Verify both devices on same WiFi network
• Check Xcode console for API errors (Open-Meteo, Google Cloud)
• Ensure location permissions granted (Always)
• Verify Google Maps API key is valid and has required APIs enabled

Future Enhancements

• Voice command integration
• Multi-language support
• Offline map caching
• Custom landmark database
• Gesture controls
• Battery optimization mode
• Social features (share locations/routes)
• Integration with other smart home devices

Contributing

This is an academic project developed as part of a hackathon. While not actively maintained, feel free to fork and adapt for your own use cases.

License

This project is provided as-is for educational purposes.

Acknowledgments

• Open-Meteo for free, unlimited weather data
• Google Cloud for AQI data and navigation services
• Anthropic Claude for AI analysis capabilities
• Raspberry Pi Foundation for affordable computing
• Xreal for consumer AR glasses