# Project export: Shepherd This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: A smart cane that guides indoor and outdoor navigation for blind people, costing ~1/20 of existing devices and providing better guidance. - Devpost: https://devpost.com/software/raising-cane - GitHub: https://github.com/tonywangs/shepherd - Video: https://www.youtube.com/embed/nJ5YjK1-0c0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Treehacks Grand Prize (1st); [OpenEvidence] Healthcare Track Grand Prize; [OpenEvidence] Best Use of Clinical Information (4x Apple Watches)) - Team: 6 GitHub contributor(s) — Claude Opus 4.6 (1M context) (13 commits), shanemion (13 commits), Gymnast544 (6 commits), Cursor (4 commits), Tony Wang (3 commits), AnthonyC12321 (2 commits) ## Devpost submission (written by the team) ### Inspiration 1.7 million Americans are legally blind. While canes tell you something is there, they don't tell you where to go. Smart canes actively steer you around obstacles, but they’re prohibitively expensive and inaccessible. We wanted to create a cheaper, intelligent alternative. ### What it does Shepherd is a motorized smart cane that uses iPhone LiDAR and computer vision to detect obstacles and physically guide the user away from them in real time, and is also able to navigate users using GPS with an interactive voice interface. A small motor on the cane applies lateral force, nudging the cane left or right, so the user feels which direction is clear without needing audio cues or screen interaction. The system runs a 60fps depth pipeline: LiDAR frames are split into left/center/right zones, a gap-seeking steering algorithm computes the clearest path, and a 12-byte BLE packet is sent to an ESP32 motor controller every 100ms. The motor uses a leaky integrator for smooth output. End-to-end latency from obstacle detection to motor response is under 50ms. Shepherd also does GPS pedestrian navigation with turn-by-turn voice guidance and waypoint following (wheelchair-accessible routing that avoids stairs), person detection via Apple's Vision framework, terrain detection (grass/mulch avoidance using HSV color analysis of the camera feed), and haptic proximity feedback that pulses faster as obstacles get closer. ### How we built it Hardware: Seeed Studio XIAO ESP32-S3 driving a DC motor, mounted on a standard white cane. The iPhone sits in a phone mount on the cane handle. The battery is inside the pipe, the motor and wheel sit at the bottom, and an enclosure in the handle has the rest of the electronics and wiring. iOS app (Swift 6 / SwiftUI): ARKit captures LiDAR depth maps at 60fps. An ObstacleDetector samples the depth buffer on an 8-pixel grid across three horizontal zones, filtering floor pixels using depth gradient heuristics. A SteeringEngine computes a continuous steering command (-1 to +1) using inverse-depth-weighted lateral bias, with EMA temporal smoothing (alpha=0.08) to prevent hallway oscillation. Communication: 12-byte BLE packets (Float32 angle, Float32 haptic distance, UInt32 mode) sent at 10Hz. The ESP32 normalizes the steering value by dividing by 255 and feeds it through a leaky integrator (tau=0.55s) before driving PWM output. Navigation: Google Geocoding API for address resolution, OpenRouteService wheelchair routing profile for step-free pedestrian directions, Overpass API for crosswalk/traffic signal detection. Micro-waypoints interpolated every 6m along the route polyline with ARKit+compass heading fusion for steering bias. Terrain detection: We use HSV color analysis of the camera buffer. We scan the lower 60% of the YCbCr frame for green-dominant pixels (hue 60°-160°), map them to left/center/right zones, and inject virtual obstacles at 1.5m in zones with >15% grass coverage. Works at night because HSV separates hue from brightness. ### Challenges we ran into Hallway oscillation was the first real problem; the cane would jitter left-right-left in narrow corridors because both walls were equidistant. We solved this by replacing repulsive steering with a gap-seeking algorithm that analyzes 16 depth columns, finds the direction of maximum clearance, and outputs proximity-scaled commands toward the clearest path -- solving overcorrection by steering toward safety rather than away from danger. A similar problem was deciding direction when encountering dead ends. We solved it with EMA temporal smoothing: instead of reacting to each frame independently, the steering engine maintains a running average of zone distances, so it commits to a direction rather than flip-flopping endlessly. Coordinate transforms between ARKit's raw depth buffer (landscape orientation) and the phone's portrait display took several iterations. The depth map's X axis is inverted relative to the display, so "left zone in raw coordinates" is actually the right side of what the user sees. The ESP32 motor would keep spinning if the iPhone app crashed or BLE dropped. We added a frame watchdog (500ms timeout) on the iOS side and a packet timeout (250ms) on the ESP32 side that both independently zero the motor. Fine-tuning PID control was another element requiring constant adjusting; we navigated through this by including dynamic debug sliders within our application during development to manually control and test our many parameters dictating movement. ### Accomplishments we're proud of The steering actually feels good. Not jerky, not laggy -- it applies a smooth lateral force that you can feel through the cane without thinking about it. Getting from "technically working" to "physically intuitive" required a lot of parameter tuning (base scale, EMA alpha, deadband thresholds, leaky integrator tau) that doesn't show up in the code but makes all the difference. The entire obstacle detection to motor response pipeline runs in under 50ms. That's fast enough that the cane reacts before you walk into something. The grass detection works without any ML model. Just pixel color analysis. It's dumb and it works. ### What we learned LiDAR is remarkably good for this use case -- sub-centimeter depth at 60fps with zero calibration. The hard part isn't sensing obstacles, it's deciding what to do about them. The steering algorithm went through five major rewrites. BLE latency matters more than bandwidth. Switching from "write with response" to "write without response" cut our effective latency in half. Spot-welding was another timeless classic that we can proudly say is now part of our repertoire. ### What's next Proper terrain segmentation using a fine-tuned model instead of color heuristics -- mulch and brown grass would be nice to detect. Integration with transit APIs for multi-modal navigation (walk to bus stop, take the 22, walk to destination). And making the hardware smaller -- the current ESP32 + motor setup works but isn't something you'd want to carry every day. ## README (from the GitHub repository) # Shepherd **An open-source, self-navigating smart white cane for the visually impaired.**

Shepherd CAD render

> **Demo Video:** *(Coming soon — showing obstacle avoidance, person detection, and GPS navigation in action)* **Quick Stats:** - 🚀 **<100ms latency** — 30-50× faster than cloud-based alternatives - 💰 **~$50 to build** — 1/20th the cost of commercial smart canes - 🔋 **4-6 hours battery** — charges your phone while you walk - 🌐 **Fully open-source** — CAD, code, and assembly instructions --- ## Table of Contents - [The Problem](#the-problem) - [Our Solution](#our-solution) - [How It Works](#how-it-works) - [Architecture](#architecture) - [Sensing & Steering Pipeline](#sensing--steering-pipeline) - [Technical Highlights: The Gap-Seeking Algorithm](#technical-highlights-the-gap-seeking-algorithm) - [AI Models & Frameworks](#ai-models--frameworks) - [Hardware](#hardware) - [Getting Started](#getting-started) - [Project Structure](#project-structure) - [Performance Metrics](#performance-metrics) - [Roadmap](#roadmap) - [Troubleshooting](#troubleshooting) - [Contributing](#contributing) - [Acknowledgments](#acknowledgments) - [License](#license) --- ## The Problem Over **253 million people** worldwide live with visual impairments. Many rely on guide dogs, AI glasses, or smart canes to navigate safely — but these tools are prohibitively expensive: | Tool | Typical Cost | |------|-------------| | Smart canes (e.g. [WeWalk](https://wewalk.io/en/)) | $800 -- $1,150 | | AI wearables (e.g. OrCam MyEye) | $2,000 -- $5,000 | | Guide dogs | ~$50,000 (with multi-year waitlists) | 85--90% of people with visual impairments live in developing countries, where any of these costs can eclipse an annual salary. Global access to assistive navigation tools is **under 1%**. Existing smart canes on the market rely on cloud-based AI (like GPT) for their intelligence — meaning they're subject to cellular connectivity, server latency (4-5 seconds per query), and subscription fees. That latency isn't just inconvenient; when you're approaching a crosswalk or a moving obstacle, it can be the difference between safety and harm. ## Our Solution Shepherd is a smart cane that **physically guides you** around obstacles using a motorized omni wheel, with all processing done **on-device** on an iPhone. No cloud. No subscriptions. Response time is **under 100ms** — roughly 50x faster than cloud-based alternatives. It costs a fraction of anything on the market, and we've open-sourced the CAD files, bill of materials, and assembly instructions so **anyone with a 3D printer and a soldering iron can build one**. ### Key Features - **Physical steering guidance** — a motorized 3.25" omni wheel at the base pushes the cane laterally to steer you around obstacles. You walk forward; Shepherd handles the rest. - **On-device AI** — LiDAR, camera, and IMU data are processed locally on the iPhone at 30-60 Hz using Apple's Vision framework and ARKit. No internet required for obstacle avoidance. - **Gap-seeking steering algorithm** — instead of pushing away from obstacles (which causes overcorrection), Shepherd finds the direction of maximum clearance and steers you toward the safest path. - **Object recognition** — identifies people, surfaces, signs, and obstacles using Apple's Vision framework (`VNDetectHumanRectanglesRequest`, `VNClassifyImageRequest`). - **GPS navigation** — integrates Google Routes API and OpenRouteService API for turn-by-turn pedestrian routing with infrastructure warnings (crosswalks, traffic signals). - **Voice assistant** — powered by Vapi, providing conversational guidance with real-time situational awareness of your surroundings. - **Haptic feedback** — custom-built from recycled e-waste; pulses faster as you approach obstacles, giving you constant spatial awareness. - **ARKit pose tracking** — 60 Hz heading updates from visual-inertial odometry, fused with compass for drift-resistant orientation. - **Charges your phone** — a built-in 12V-to-5V step-down powers your iPhone while you walk. ### Prior Work Shepherd builds on research from Stanford's [Augmented Cane project](https://hai.stanford.edu/news/stanford-researchers-build-400-self-navigating-smart-cane) ([GitHub](https://github.com/pslade2/AugmentedCane)), which demonstrated the viability of omni-wheel steering for assistive navigation. We extend this concept with on-device AI, GPS navigation, object recognition, and a fully open-source hardware design. --- ## How It Works ### Architecture ``` iPhone 14 Pro Max (LiDAR + Camera + IMU + GPS + Compass) │ ├─ ARKit (30-60 Hz) │ ├─ LiDAR depth maps (sceneDepth) │ ├─ Camera RGB frames │ └─ Pose tracking (heading, position) │ ├─ Vision Framework (~2 Hz) │ ├─ VNDetectHumanRectanglesRequest (person detection) │ └─ VNClassifyImageRequest (scene classification) │ ├─ CoreLocation (1 Hz) │ ├─ GPS position │ └─ Magnetometer (compass heading) │ ├─ Obstacle Detection & Steering │ ├─ Gap profiling (16-column depth analysis) │ ├─ Navigation bias (GPS bearing to next waypoint) │ └─ Merge: gapCommand + navBias × (1 - proximityFactor) │ └─ BLE (10-20 Hz, custom 12-byte protocol) ──► ESP32-S3 │ ┌───────┴────────┐ │ │ Motor Control Haptic Engine (omni wheel PWM) (taptic pulses) │ │ Leaky Integrator Distance-based (smooth accel) (pulse freq) ``` ### Sensing & Steering Pipeline 1. **Depth capture** — ARKit captures LiDAR depth maps at 30-60 Hz, along with camera RGB frames for object recognition 2. **Obstacle detection** — depth map is analyzed in left/center/right zones for obstacles, with vertical filtering to ignore ceiling/floor 3. **Person detection** — Vision framework (`VNDetectHumanRectanglesRequest`) runs at ~2 Hz, mapping bounding boxes to LiDAR depth for distance estimation 4. **Gap profiling** — the depth map is split into 16 vertical columns; average depth per column is computed and smoothed with a [0.25, 0.5, 0.25] kernel to find the direction of maximum clearance 5. **Steering computation** — `command = sqrt(|gapDirection|) × proximityFactor`, where `proximityFactor` ramps from 0 (clear) to 1 (obstacle <0.2m). This produces smooth, non-oscillating steering toward the safest path. 6. **Navigation merge (if active)** — GPS navigation bias is blended additively with obstacle avoidance, scaled by `(1 - proximityFactor)` so obstacles always take priority 7. **BLE transmission** — a custom 12-byte protocol sends `{speed, angle, distance, mode}` at 10-20 Hz over Bluetooth Low Energy (write-without-response for minimal latency) 8. **Motor response** — the ESP32 applies a leaky integrator (boat-like momentum model) for smooth acceleration/deceleration, preventing jarring movements 9. **Safety** — if Bluetooth disconnects, the ESP32 auto-decays motor power to zero over ~500ms (no sudden jolts) ### Why On-Device? As Saqib Shaikh (creator of Microsoft's Seeing AI) has noted, accessibility tech for the visually impaired benefits enormously from edge processing — users can't afford to wait for a cloud round-trip while navigating a crosswalk. Shepherd's core obstacle detection and steering runs entirely on the iPhone with **no network dependency**. ### Technical Highlights: The Gap-Seeking Algorithm Early prototypes used obstacle repulsion steering (push away from detected obstacles). This failed catastrophically at close range — approaching a trash can dead-center would ca [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 39 recognized source files, 357 KB. - Swift (language) — detected in the code - C++ (language) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (60 of 60) ``` .claude/settings.local.json .gitignore BLE_PROTOCOL.md ESP32/SmartCane_ESP32/SmartCane_ESP32.ino Hardware/Assembly Instructions.md Hardware/handle - electronics cover.stl Hardware/handle - handle (3).stl Hardware/handle - handle cover (1).stl Hardware/motor mounting - motor clamp (4).stl Hardware/motor mounting - motor cover (2).stl Hardware/motor mounting - motor mount (2).stl Hardware/vex wheel to gobilda hub - wheel adapter.stl PERSON_DETECTION_EXPLAINED.md QUICKSTART.md README.md SETUP_TROUBLESHOOTING.md SmartCane_Source/SmartCane.xcodeproj/project.pbxproj SmartCane_Source/SmartCane.xcodeproj/project.xcworkspace/contents.xcworkspacedata SmartCane_Source/SmartCane/Communication/BLEManager.swift SmartCane_Source/SmartCane/ContentView.swift SmartCane_Source/SmartCane/Core/SmartCaneController.swift SmartCane_Source/SmartCane/Feedback/HapticManager.swift SmartCane_Source/SmartCane/Feedback/VoiceManager.swift SmartCane_Source/SmartCane/Info.plist SmartCane_Source/SmartCane/Navigation/ObstacleDetector.swift SmartCane_Source/SmartCane/Navigation/SteeringEngine.swift SmartCane_Source/SmartCane/Sensors/DepthSensor.swift SmartCane_Source/SmartCane/SmartCaneApp.swift SmartCane_Source/SmartCane/Vision/ObjectRecognizer.swift SmartCane/SmartCane.xcodeproj/project.pbxproj SmartCane/SmartCane.xcodeproj/project.pbxproj.backup SmartCane/SmartCane.xcodeproj/project.xcworkspace/contents.xcworkspacedata SmartCane/SmartCane/Assets.xcassets/AccentColor.colorset/Contents.json SmartCane/SmartCane/Assets.xcassets/AppIcon.appiconset/Contents.json SmartCane/SmartCane/Assets.xcassets/Contents.json SmartCane/SmartCane/BluetoothPairingView.swift SmartCane/SmartCane/Communication/ESPBluetoothManager.swift SmartCane/SmartCane/ContentView_Backup.swift SmartCane/SmartCane/ContentView.swift SmartCane/SmartCane/Core/SmartCaneController.swift SmartCane/SmartCane/DeepLabV3Int8LUT.mlmodel SmartCane/SmartCane/Feedback/HapticManager.swift SmartCane/SmartCane/Feedback/VoiceManager.swift SmartCane/SmartCane/Info.plist SmartCane/SmartCane/Input/GameControllerManager.swift SmartCane/SmartCane/Navigation/NavigationManager.swift SmartCane/SmartCane/Navigation/NavigationSteering.swift SmartCane/SmartCane/Navigation/ObstacleDetector.swift SmartCane/SmartCane/Navigation/RouteService.swift SmartCane/SmartCane/Navigation/SteeringEngine.swift SmartCane/SmartCane/Navigation/SurfaceClassifier.swift SmartCane/SmartCane/Navigation/WaypointModels.swift SmartCane/SmartCane/NavigationView.swift SmartCane/SmartCane/RouteMapView.swift SmartCane/SmartCane/Sensors/DepthSensor.swift SmartCane/SmartCane/SmartCaneApp.swift SmartCane/SmartCane/Vision/DepthVisualizer.swift SmartCane/SmartCane/Vision/ObjectRecognizer.swift SmartCane/SmartCane/Voice/VapiManager.swift TESTING.md ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - i lied, don't forget the readme - final push?!?! potensh? - Remove HARDWARE_SETUP.md, now covered by Hardware/Assembly Instructions.md - Rewrite README as a project landing page - Add hardware assembly instructions, 3D print files, and CAD images - Add Joy-Con kill button and fix controls - Add navigation compass and micro-waypoint map overlay - Switch pedestrian routing to OpenRouteService wheelchair profile - Add GPS pedestrian navigation with route map and accessibility routing - Add Nintendo Switch Joy-Con steering override - Fix motor output scaling and add live tuning sliders - Tuned ESP code adding constant multiplier - Remove dead BLEManager (1-byte protocol) — ESP32 only accepts 12-byte packets - Real ESP32 Code - Add CoreML import for MLMultiArray support - Fix ContentView parameter order (caneController must precede espBluetooth) - Implement weighted temporal tiebreaker with EMA smoothing - Add frame watchdog to zero steering when depth pipeline stalls - Add EMA temporal smoothing to fix hallway steering oscillation - Add continuous zone weighting, faster person detection, and wall/ahead evasion ## Key source files (fetched from GitHub, selected and truncated for size) ### BLE_PROTOCOL.md ```markdown # BLE Protocol Specification **Ultra-Low-Latency Communication Protocol for Smart Cane** ## Design Philosophy This protocol is optimized for minimal latency in a real-time control system. Every design decision prioritizes speed over features. ### Key Optimizations - **Single-byte packets** - Minimal transmission time - **Write without response** - No ACK overhead - **No JSON/serialization** - Direct binary values - **Persistent connection** - No reconnection overhead - **High connection priority** - iPhone optimization flags ## Service Definition ### Service UUID ``` 4fafc201-1fb5-459e-8fcc-c5c9c331914b ``` This UUID must be identical in both iOS and ESP32 code. ## Characteristics ### 1. Steering Command Characteristic **UUID:** `beb5483e-36e1-4688-b7f5-ea07361b26a8` **Direction:** iPhone → ESP32 **Format:** 1 signed byte (int8_t / Int8) **Values:** | Value | Meaning | Motor Action | |-------|---------|--------------| | -1 | LEFT | Omni wheel rolls left (lateral force) | | 0 | NEUTRAL | Motor stopped (no lateral force) | | +1 | RIGHT | Omni wheel rolls right (lateral force) | **Write Mode:** Write Without Response (fastest) **Update Rate:** ~30 Hz (every 33ms) **Timeout:** ESP32 stops motor if no command received for 500ms (safety) #### iOS Implementation ```swift var command: Int8 = -1 // or 0 or +1 let data = Data(bytes: &command, count: 1) peripheral.writeValue(data, for: characteristic, type: .withoutResponse) ``` #### ESP32 Implementation ```cpp void onWrite(BLECharacteristic *pCharacteristic) { std::string value = pCharacteristic->getValue(); if (value.length() == 1) { int8_t command = (int8_t)value[0]; handleSteeringCommand(command); } } ``` ### 2. Haptic Trigger Characteristic **UUID:** `beb5483e-36e1-4688-b7f5-ea07361b26a9` **Direction:** iPhone → ESP32 **Format:** 1 unsigned byte (uint8_t / UInt8) **Values:** 0-255 (haptic intensity) - 0 = No vibration - 128 = Medium intensity - 255 = Maximum intensity **Write Mode:** Write Without Response **Update Rate:** Variable (triggered by distance changes) **Purpose:** Trigger vibration motor pulses on ESP32 #### iOS Implementation ```swift var intensity: UInt8 = 180 let data = Data(bytes: &intensity, count: 1) peripheral.writeValue(data, for: characteristic, type: .withoutResponse) ``` #### ESP32 Implementation ```cpp void onWrite(BLECharacteristic *pCharacteristic) { std::string value = pCharacteristic->getValue(); if (value.length() == 1) { uint8_t intensity = (uint8_t)value[0]; analogWrite(HAPTIC_PIN, intensity); } } ``` ## Connection Parameters ### iOS Central Configuration ```swift // Scanning centralManager.scanForPeripherals(withServices: [serviceUUID], options: nil) // Connection centralManager.connect(peripheral, options: nil) // No connection interval configuration needed - iOS manages automatically ``` ### ESP32 Peripheral Configuration ```cpp // Advertising parameters optimized f [truncated — 3787 more characters] ``` ### PERSON_DETECTION_EXPLAINED.md ```markdown # Person Detection & Distance Calculation - Technical Explanation ## Overview The Smart Cane uses a combination of **Vision Framework** (for person detection) and **LiDAR depth sensing** (for distance calculation) to identify and measure the distance to people in the user's path. --- ## How It Works: Step-by-Step ### 1. **Camera Frame Capture** (60fps) - ARKit captures RGB camera frames at 60fps from iPhone's camera - Simultaneously captures LiDAR depth map at 60fps - Both are packaged into `DepthFrame` structure ```swift struct DepthFrame { let depthMap: CVPixelBuffer // LiDAR depth data let capturedImage: CVPixelBuffer? // RGB camera frame let timestamp: TimeInterval let cameraTransform: simd_float4x4 } ``` --- ### 2. **Person Detection** (Every 5 seconds) Uses Apple's **Vision Framework** - specifically `VNDetectHumanRectanglesRequest`: ```swift VNDetectHumanRectanglesRequest { request, error in // Returns array of VNHumanObservation with bounding boxes } ``` **What it detects:** - Full human bodies in the frame - Returns normalized bounding box coordinates (0-1 range) - Works in various poses and lighting conditions - Hardware-accelerated on A14+ chips (Neural Engine) **Output:** ```swift DetectionResult { objectName: "person" boundingBox: CGRect(x: 0.3, y: 0.4, width: 0.2, height: 0.4) // Example: person centered at (0.4, 0.6) in frame } ``` --- ### 3. **Coordinate Conversion** Convert Vision's normalized coordinates to pixel coordinates in depth map: **Challenge:** Vision uses **bottom-left** origin, depth map uses **top-left** origin ```swift // Vision bounding box: (0.5, 0.5) = center of frame // Must flip Y coordinate for depth map let centerX = Int(boundingBox.midX * CGFloat(depthMapWidth)) let centerY = Int((1.0 - boundingBox.midY) * CGFloat(depthMapHeight)) // Example: // boundingBox.midX = 0.5, boundingBox.midY = 0.5 // depthMapWidth = 1920, depthMapHeight = 1440 // → centerX = 960, centerY = 720 (center of depth map) ``` --- ### 4. **Depth Sampling Strategy** Sample multiple depth values around the person's center for robustness: ```swift // Sample 11×11 grid (121 points) around detected person for dy in -5...5 { for dx in -5...5 { let x = centerX + dx let y = centerY + dy let depth = depthMap[y][x] // in meters } } ``` **Why multiple samples?** - LiDAR has noise and occasional invalid readings - Person might not be perfectly centered in bounding box - Edges of person might have different depth than center - Taking median is more robust than single point **Filtering:** - Remove invalid values (depth < 0 or depth > 10 meters) - Remove outliers (background objects) - Use **median** instead of mean (less affected by outliers) --- ### 5. **Distance Calculation** ```swift // Example data: depthValues = [2.31, 2.34, 2.33, 2.35, 2.32, 2.34, ...] sorted = [2.31, 2.32, 2.33, 2.34, 2.34, 2.35, ...] median = sorted[60] = 2.34 meters // Result: Pers [truncated — 5335 more characters] ``` ### SmartCane/SmartCane/ContentView_Backup.swift ```swift // // ContentView_Backup.swift // Backup of original ContentView // ``` ### SmartCane_Source/SmartCane/SmartCaneApp.swift ```swift // // SmartCaneApp.swift // SmartCane // // Hackathon MVP - Lateral Steering Smart Cane // import SwiftUI @main struct SmartCaneApp: App { var body: some Scene { WindowGroup { ContentView() } } } ``` ### SmartCane/SmartCane/SmartCaneApp.swift ```swift // // SmartCaneApp.swift // SmartCane // // Hackathon MVP - Lateral Steering Smart Cane // import SwiftUI @main struct SmartCaneApp: App { @StateObject private var espBluetooth = ESPBluetoothManager() @StateObject private var caneController = SmartCaneController() var body: some Scene { WindowGroup { TabView { ContentView(caneController: caneController, espBluetooth: espBluetooth) .tabItem { Label("Navigation", systemImage: "location.fill") } if let navManager = caneController.navigationManager { RouteMapView(navigationManager: navManager) .tabItem { Label("Route", systemImage: "map.fill") } } BluetoothPairingView(ble: espBluetooth, controller: caneController) .tabItem { Label("Bluetooth", systemImage: "antenna.radiowaves.left.and.right") } } .onAppear { // Initialize after views are ready caneController.initialize(espBluetooth: espBluetooth) } } } } ``` ### SmartCane_Source/SmartCane/ContentView.swift ```swift // // ContentView.swift // SmartCane // // Main UI - Simple status display for hackathon demo // import SwiftUI struct ContentView: View { @StateObject private var caneController = SmartCaneController() var body: some View { VStack(spacing: 20) { // Status Section Text("Smart Cane") .font(.largeTitle) .bold() // BLE Connection Status HStack { Circle() .fill(caneController.isConnected ? Color.green : Color.red) .frame(width: 20, height: 20) Text(caneController.isConnected ? "Connected" : "Disconnected") } // ARKit Status HStack { Circle() .fill(caneController.isARRunning ? Color.green : Color.orange) .frame(width: 20, height: 20) Text(caneController.isARRunning ? "LiDAR Active" : "LiDAR Inactive") } Divider() // Live Data Display VStack(alignment: .leading, spacing: 10) { Text("Obstacle Detection") .font(.headline) HStack { ZoneIndicator(label: "Left", distance: caneController.leftDistance) ZoneIndicator(label: "Center", distance: caneController.centerDistance) ZoneIndicator(label: "Right", distance: caneController.rightDistance) } Text("Steering: \(caneController.steeringCommandText)") .font(.title2) .bold() .foregroundColor(caneController.steeringColor) if let object = caneController.detectedObject { Text("Detected: \(object)") .font(.subheadline) .foregroundColor(.blue) } } .padding() .background(Color.gray.opacity(0.1)) .cornerRadius(10) Divider() // Control Buttons VStack(spacing: 15) { Button(action: { caneController.toggleSystem() }) { Text(caneController.isSystemActive ? "Stop System" : "Start System") .frame(maxWidth: .infinity) .padding() .background(caneController.isSystemActive ? Color.red : Color.green) .foregroundColor(.white) .cornerRadius(10) } Button(action: { caneController.testVoice() }) { Text("Test Voice") .frame(maxWidth: .infinity) .padding() .background(Color.blue) .foregroundColor(.white) .cornerRadius(10) } } Spacer() // Debug Info Text("Latency: \(String(format: "%.1f", caneController.latencyMs))ms") .font(.caption) .foregroundColor(.gray) } .padding() .onAppear { caneController.initialize() } } } struct ZoneIndicator: View { let label: String let distance: Float? var color: Color { guard let dist = distance else { return .gray } if dist < 0.5 { return .red } if dist < 1.0 { return .orange } if dist < 1.5 { return .yellow } return .green } var body: some View { VStack { Text(label) .font(.caption) Rectangle() .fill(color) .frame(width: 60, height: 100) if let dist = distance { Text(String(format: "%.2fm", dist)) .font(.caption2) } else { Text("--") .font(.caption2) } } } } #Preview { ContentView() } ``` ### SmartCane/SmartCane/BluetoothPairingView.swift ```swift // // BluetoothPairingView.swift // SmartCane // // Manual ESP32 pairing and motor control interface. // Ported from the Bluetooth branch's standalone app. // import SwiftUI struct BluetoothPairingView: View { @ObservedObject var ble: ESPBluetoothManager @ObservedObject var controller: SmartCaneController var body: some View { NavigationStack { Form { // MARK: - Bluetooth Pairing Section("Bluetooth") { HStack { Label( ble.isBluetoothReady ? "Powered On" : "Unavailable", systemImage: ble.isBluetoothReady ? "bolt.horizontal.circle.fill" : "bolt.horizontal.circle" ) .foregroundStyle(ble.isBluetoothReady ? .green : .red) Spacer() Button(ble.isScanning ? "Stop Scan" : "Scan") { ble.toggleScan() } } if let connectedName = ble.connectedName { HStack { Text("Connected: \(connectedName)") Spacer() Button("Disconnect", role: .destructive) { ble.disconnect() } } } if let statusMessage = ble.statusMessage { Text(statusMessage) .font(.caption) .foregroundStyle(.orange) } if ble.discoveredPeripherals.isEmpty { Text("No ESP32 devices found yet.") .foregroundStyle(.secondary) } else { ForEach(ble.discoveredPeripherals) { device in HStack { VStack(alignment: .leading) { Text(device.name) Text("RSSI: \(device.rssi)") .font(.caption) .foregroundStyle(.secondary) } Spacer() Button("Pair") { ble.connect(device.peripheral) } } } } } // MARK: - Motor Control Section("Motor Control (10 Hz)") { VStack(alignment: .leading, spacing: 6) { Text("Angle: \(ble.angle, specifier: "%.2f")") Slider(value: $ble.angle, in: -180...180) } VStack(alignment: .leading, spacing: 6) { Text("Distance: \(ble.distance, specifier: "%.2f")") Slider(value: $ble.distance, in: 0...100) } VStack(alignment: .leading, spacing: 6) { Text("Mode: \(Int(ble.mode.rounded()))") Slider(value: $ble.mode, in: 0...10, step: 1) } } // MARK: - Steering Tuning Section("Steering Tuning") { VStack(alignment: .leading, spacing: 6) { Text("Sensitivity: \(ble.steeringSensitivity, specifier: "%.1f")m") Slider(value: $ble.steeringSensitivity, in: 0.5...4.0) Text("Start steering when obstacle closer than this") .font(.caption2) .foregroundStyle(.secondary) } VStack(alignment: .leading, spacing: 6) { Text("Motor Base Scale: \(ble.motorBaseScale, specifier: "%.0f")") Slider(value: $ble.motorBaseScale, in: 10...255) Text("Raw speed sent to ESP32 (÷255 on device). Higher = stronger motor.") .font(.caption2) .foregroundStyle(.secondary) } VStack(alignment: .leading, spacing: 6) { Text("Magnitude: \(ble.steeringMagnitude, specifier: "%.1f")×") Slider(value: $ble.steeringMagnitude, in: 0.1...3.0) Text("Extra multiplier on base scale") .font(.caption2) .foregroundStyle(.secondary) } VStack(alignment: .leading, spacing: 6) { Text("Proximity Exponent: \(ble.proximityExponent, specifier: "%.2f")") Slider(value: $ble.proximityExponent, in: 0.2...1.5) Text("Lower = ramps up faster with distance. 1.0 = linear.") .font(.caption2) .foregroundStyle(.secondary) } VStack(alignment: .leading, spacing: 6) { Text("Close Floor: \(ble.closeFloor, specifier: "%.2f")") Slider(value: $ble.closeFloor, in: 0.0...1.0) Text("Min |command| when obstacle < 1m. 0 = disabled.") .font(.caption2) .foregroundStyle(.secondary) } } // MARK: - Steering Debug Section("Steering Debug (Live)") { VStack(alignment: .leading, spacing: 6) { HStack { Text("Gap Direction:") .for [truncated — 2580 more characters] ``` ### SmartCane/SmartCane/NavigationView.swift ```swift // // NavigationView.swift // SmartCane // // Navigation input sheet and HUD overlay for GPS turn-by-turn guidance // import SwiftUI // MARK: - Navigation Input Sheet struct NavigationInputSheet: View { @ObservedObject var navigationManager: NavigationManager @Binding var isPresented: Bool @State private var destination: String = "" @FocusState private var isTextFieldFocused: Bool var body: some View { NavigationStack { VStack(spacing: 24) { VStack(spacing: 8) { Image(systemName: "map.fill") .font(.system(size: 50)) .foregroundColor(.cyan) Text("Where to?") .font(.title) .bold() .foregroundColor(.white) Text("Enter a destination for walking directions") .font(.subheadline) .foregroundColor(.gray) } .padding(.top, 20) TextField("e.g. Tresidder Union, Stanford", text: $destination) .textFieldStyle(.plain) .padding() .background(Color.gray.opacity(0.2)) .cornerRadius(12) .foregroundColor(.white) .focused($isTextFieldFocused) .submitLabel(.go) .onSubmit { startNavigation() } Button(action: startNavigation) { HStack(spacing: 10) { Image(systemName: "location.fill") Text("Navigate") .bold() } .frame(maxWidth: .infinity) .padding(.vertical, 16) .background(destination.isEmpty ? Color.gray : Color.cyan) .foregroundColor(.white) .cornerRadius(15) } .disabled(destination.isEmpty) Spacer() } .padding() .background(Color.black.ignoresSafeArea()) .navigationBarTitleDisplayMode(.inline) .toolbar { ToolbarItem(placement: .cancellationAction) { Button("Cancel") { isPresented = false } .foregroundColor(.gray) } } } .onAppear { isTextFieldFocused = true } } private func startNavigation() { guard !destination.isEmpty else { return } print("[NavigationSheet] Navigate tapped with destination: '\(destination)'") navigationManager.startNavigation(to: destination) isPresented = false } } // MARK: - Navigation HUD struct NavigationHUD: View { @ObservedObject var navigationManager: NavigationManager var body: some View { VStack(spacing: 0) { VStack(spacing: 10) { // State-dependent content switch navigationManager.state { case .planning: planningView case .navigating: navigatingView case .arriving: arrivingView case .arrived: arrivedView case .error(let message): errorView(message) default: EmptyView() } } .padding(.horizontal, 16) .padding(.vertical, 12) .background( RoundedRectangle(cornerRadius: 16) .fill(Color.black.opacity(0.9)) .overlay( RoundedRectangle(cornerRadius: 16) .stroke(hudBorderColor, lineWidth: 2) ) ) .shadow(color: hudBorderColor.opacity(0.3), radius: 10) } .padding(.horizontal, 12) .padding(.top, 8) } // MARK: - Sub-views private var planningView: some View { HStack(spacing: 12) { ProgressView() .progressViewStyle(CircularProgressViewStyle(tint: .cyan)) Text("Planning route...") .font(.subheadline) .foregroundColor(.white) Spacer() } } private var navigatingView: some View { VStack(alignment: .leading, spacing: 8) { // Current instruction HStack(alignment: .top, spacing: 10) { Image(systemName: maneuverIcon) .font(.title2) .foregroundColor(.cyan) .frame(width: 30) VStack(alignment: .leading, spacing: 4) { Text(navigationManager.currentGuidance?.currentInstruction ?? "Continue") .font(.subheadline) .bold() .foregroundColor(.white) .lineLimit(2) HStack(spacing: 16) { Label(formatDistance(navigationManager.distanceToNextManeuver), systemImage: "arrow.turn.up.right") .font(.caption) .foregroundColor(.cyan) Label(formatDistance(navigationManager.distanceToDestination), systemImage: "flag.fill") .font(.caption) .foregroundColor(.gray) } } Spacer() stopButton } // Infrastructure warnings if let guidance = navigationManager.currentGuidance, !guidance.nearbyInfrastructure.isEmpty { HStack(spacing: 8) { ForEach(Array(Set(guidance.nearbyInfrastructure.map(\.type.rawValue))), id: \.self) { type in Label( [truncated — 4277 more characters] ``` ### SmartCane/SmartCane/RouteMapView.swift ```swift // // RouteMapView.swift // SmartCane // // Map tab showing the active route, step waypoints, and infrastructure features // import SwiftUI import MapKit struct RouteMapView: View { @ObservedObject var navigationManager: NavigationManager @State private var showMicroWaypoints = false var body: some View { ZStack { Color.black.ignoresSafeArea() if let route = navigationManager.currentRoute { activeRouteMap(route: route) } else { noRouteView } } } // MARK: - No Route private var noRouteView: some View { VStack(spacing: 16) { Image(systemName: "map") .font(.system(size: 60)) .foregroundColor(.gray) Text("No Active Route") .font(.title2) .bold() .foregroundColor(.white) Text("Start navigation from the main tab to see the route here") .font(.subheadline) .foregroundColor(.gray) .multilineTextAlignment(.center) .padding(.horizontal, 40) } } // MARK: - Active Route Map private func activeRouteMap(route: PedestrianRoute) -> some View { let polylineCoords = route.overviewPolyline let region = Self.regionForCoordinates(polylineCoords, userLocation: navigationManager.userLocation) return VStack(spacing: 0) { // Route info header routeHeader(route: route) // Map Map(initialPosition: .region(region)) { // Route polyline if polylineCoords.count >= 2 { MapPolyline(coordinates: polylineCoords) .stroke(.cyan, lineWidth: 5) } // Origin marker Annotation("Start", coordinate: route.origin) { ZStack { Circle() .fill(.green) .frame(width: 24, height: 24) Image(systemName: "figure.walk") .font(.system(size: 12)) .foregroundColor(.white) } } // Destination marker Annotation(route.destinationName.components(separatedBy: ",").first ?? "End", coordinate: route.destination) { ZStack { Circle() .fill(.red) .frame(width: 24, height: 24) Image(systemName: "flag.fill") .font(.system(size: 12)) .foregroundColor(.white) } } // Step waypoints (turn points) ForEach(Array(route.steps.enumerated()), id: \.offset) { index, step in if step.maneuver != .depart && step.maneuver != .unknown { Annotation("", coordinate: step.startLocation) { stepMarker(index: index, step: step, isCurrentStep: index == navigationManager.currentStepIndex) } } } // Infrastructure features ForEach(allInfrastructure(from: route)) { feature in Annotation("", coordinate: feature.coordinate) { infrastructureMarker(feature: feature) } } // Micro-waypoints (when toggled on) if showMicroWaypoints { let waypoints = navigationManager.waypointTracker.waypoints let currentIdx = navigationManager.waypointTracker.currentIndex ForEach(Array(waypoints.enumerated()), id: \.offset) { index, wp in Annotation("", coordinate: wp.coordinate) { Circle() .fill(index == currentIdx ? Color.yellow : Color.mint.opacity(0.7)) .frame(width: index == currentIdx ? 12 : 8, height: index == currentIdx ? 12 : 8) .overlay( index == currentIdx ? Circle() .stroke(Color.yellow, lineWidth: 2) .frame(width: 18, height: 18) : nil ) } } } // User location if let userLoc = navigationManager.userLocation { Annotation("You", coordinate: userLoc) { ZStack { Circle() .fill(.blue.opacity(0.3)) .frame(width: 32, height: 32) Circle() .fill(.blue) .frame(width: 16, height: 16) .overlay( Circle().stroke(.white, lineWidth: 2) ) } } } } .mapStyle(.standard) // Step list stepList(route: route) } } // MARK: - Route Header private func routeHeader(route: PedestrianRoute) -> some View { HStack(spacing: 12) { VStack(alignment: .leading, spacing: 4) { Text(route.destinationName.components(separatedBy: ",").first ?? route.destinationName) .font(.headline) .foregroundColor(.white) .lineLimit(1) HSt [truncated — 8074 more characters] ``` ### SmartCane_Source/SmartCane/Sensors/DepthSensor.swift ```swift // // DepthSensor.swift // SmartCane // // ARKit + LiDAR depth capture at 30-60fps // import Foundation import ARKit import Combine // Depth data structure struct DepthFrame { let depthMap: CVPixelBuffer let timestamp: TimeInterval let cameraTransform: simd_float4x4 } class DepthSensor: NSObject, ObservableObject { @Published var latestDepthFrame: DepthFrame? private var arSession: ARSession? private let configuration = ARWorldTrackingConfiguration() override init() { super.init() setupARSession() } private func setupARSession() { guard ARWorldTrackingConfiguration.supportsFrameSemantics(.sceneDepth) else { print("[DepthSensor] ERROR: LiDAR not supported on this device!") return } arSession = ARSession() arSession?.delegate = self // Configure for LiDAR depth configuration.frameSemantics = .sceneDepth configuration.planeDetection = [.horizontal, .vertical] // Optimize for real-time performance configuration.videoFormat = ARWorldTrackingConfiguration .supportedVideoFormats .first { $0.framesPerSecond == 60 } ?? ARWorldTrackingConfiguration.supportedVideoFormats[0] print("[DepthSensor] ARKit configured for LiDAR at \(configuration.videoFormat.framesPerSecond)fps") } func start() { print("[DepthSensor] Starting ARKit session...") arSession?.run(configuration, options: [.resetTracking, .removeExistingAnchors]) } func stop() { print("[DepthSensor] Stopping ARKit session...") arSession?.pause() } } // MARK: - ARSessionDelegate extension DepthSensor: ARSessionDelegate { func session(_ session: ARSession, didUpdate frame: ARFrame) { // Extract scene depth (LiDAR) guard let sceneDepth = frame.sceneDepth else { return } let depthFrame = DepthFrame( depthMap: sceneDepth.depthMap, timestamp: frame.timestamp, cameraTransform: frame.camera.transform ) // Publish on main thread (SwiftUI requirement) DispatchQueue.main.async { [weak self] in self?.latestDepthFrame = depthFrame } } func session(_ session: ARSession, didFailWithError error: Error) { print("[DepthSensor] ERROR: \(error.localizedDescription)") } } ``` [27 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]