# Project export: Wingman

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 12.0
- Tagline: Wingman uses Snap Spectacles to control a robot that moves, grabs, and sees where humans can’t by pointing to navigate, gesturing to grasp, and capturing 3D scenes through depth-sensing AR.
- Devpost: https://devpost.com/software/wingman-9xok8b
- GitHub: not linked
- Video: https://www.youtube.com/embed/aqoKeFthpqQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Snap: Best Use of Snap Spectacles; BitRobot Network: Best Robotics Hack - 3rd Place)
- Team: contributor stats unavailable

## Devpost submission (written by the team)

### Inspiration

During disasters and times of crisis, first responders often face an impossible choice: risk their lives to search for survivors or stay back and lose precious time. We wanted to eliminate that choice. Wingman was born from the idea that technology can extend human presence into unreachable worlds - using wearable AR to control a robot as naturally as moving your own body. Instead of doing things that humans can already do, we wanted Wingman to fill the gaps of our abilities.

### What it does

Wingman is a Snap Spectacles–controlled robot that brings human intuition into robotics. It combines three seamless functions: Move - Point to a location through the Spectacles’ depth-sensing AR, and the robot autonomously navigates there. Grab - Use hand tracking to control the robot’s arm and gripper. Snapshot - Capture and explore 3D depth images of the environment, rotating views to reconstruct the scene virtually. Together, these features allow people to move, manipulate, and visualize in places humans can’t safely go.

### How we built it

We integrated Snap Spectacles, AR world data, and robotic motion control through a low-latency pipeline: Hardware: A mobile LeKiwi robot with a 6-DOF arm and gripper, paired with Snap Spectacles for spatial mapping, hand tracking, and 3D capture. Software: FastAPI + WebSocket for real-time communication. Snap Lens Studio API to extract hand-tracking data and AR anchors. OpenCV for vision processing and coordinate mapping. Three.js for 3D depth visualization. Inverse kinematics and path planning for smooth, intuitive arm control. FastAPI + WebSocket for real-time communication. Snap Lens Studio API to extract hand-tracking data and AR anchors. OpenCV for vision processing and coordinate mapping. Three.js for 3D depth visualization. Inverse kinematics and path planning for smooth, intuitive arm control.

### Challenges we ran into

Achieving low-latency control between the Spectacles and the robot. Translating AR spatial anchors from Snap’s coordinate system into real-world robot coordinates. Maintaining precise gesture-to-motion mapping without drift or jitter. Synchronizing depth capture and 3D reconstruction for accurate spatial visualization.

### Accomplishments we're proud of

Building a fully functional prototype that unites AR, robotics, and real-time control. Using Snap Spectacles not just for vision, but as a wearable interface for spatial movement, manipulation, and perception. Creating a 3D visualization system that reconstructs physical environments from depth snapshots. Designing an experience that feels more like collaboration with a companion than controlling a machine.

### What we learned

We discovered how powerful AR becomes when combined with robotics - not as a separate layer of information, but as a natural bridge between human intent and machine action. We also learned the importance of designing for intuition over complexity: when the interface feels human, the technology disappears, and the experience becomes seamless.

### What's next

We plan to expand Wingman’s capabilities with: Voice commands (“Wingman, move to the doorway” or “pick up the red object”). Autonomous exploration using SLAM and AI-driven scene understanding. Edge processing on Spectacles for faster local gesture interpretation. Multi-step planning for performing complex tasks from natural language. Our ultimate goal: make Wingman a universal telepresence assistant - a true wearable-controlled companion for exploration, safety, and discovery.

## README (from the GitHub repository)

No README available.

## Detected evidence (automated analysis)

No repository was indexed for this project. Claimed technologies below could not be checked against code.
- FastAPI (technology) — claimed on Devpost, not found in the code

## Codebase structure

No repository index available.

## Key source files

No repository index available; no source files included.