# Project export: Jesture This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: AI agents that turn gestures into actions, across every device. - Devpost: https://devpost.com/software/jesture-49tw6z - GitHub: not linked - Video: https://www.youtube.com/embed/2op8cVi6j3o?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: contributor stats unavailable ## Devpost submission (written by the team) ### Overview @ -0,0 +1,426 @@ ### Inspiration We've all been there - hands covered in flour while cooking and needing to skip a recipe video, or giving a presentation and fumbling with a clicker. The moment that sparked Jesture was watching someone with limited hand mobility struggle to control their computer during a video call. We realized that gesture control could democratize computer interaction, but existing solutions are either too rigid (pre-programmed gestures) or too complex (requiring coding). What if your computer could understand what you're trying to do and adapt gestures to your context? That's the question that drove us to build Jesture - a system that's smart enough to know that swiping right should mean "next slide" in PowerPoint but "skip forward" in Netflix. ### What it does Jesture transforms hand gestures into intelligent device control through two powerful modes: ๐ŸŽจ Manual Mode: Visual Workflow Builder Think Zapier meets sign language. Users drag-and-drop to create custom gesture workflows: Input Nodes: 10 recognizable gestures (swipes, thumbs up/down, peace sign, fist, etc.) Modifier Nodes: Cooldown timers, conditional logic Output Nodes: Keyboard shortcuts, mouse actions, smart lights Example: Connect "Peace Sign" โ†’ "Cooldown (3s)" โ†’ "Change Slide Theme" for presentation control ๐Ÿค– AI Mode: Context-Aware Intelligence Here's where it gets magical. Click "AI Mode" and Jesture: Detects your context - Are you watching Netflix? In PowerPoint? Browsing? Consults AI agents - Fetches appropriate gestures from catalog Generates optimal workflow - Claude maps gestures to context-specific actions Executes in real-time - Same gestures, different actions per app All automatically, no configuration needed. Real-World Applications โ™ฟ Accessibility: Hands-free computer control for users with limited mobility ๐Ÿง‘โ€๐Ÿณ Multitasking: Control devices while cooking, working out, or creating ๐Ÿ‘จโ€๐Ÿซ Presentations: Professional, seamless slide control without clickers ๐Ÿ  Smart Home: Gesture-based light and device control ๐Ÿงผ Hygiene: Contactless interfaces in medical/industrial settings ### How we built it Jesture is a distributed multi-agent system combining computer vision, AI orchestration, and real-time action execution. Architecture Overview Technical Stack Computer Vision Layer MediaPipe Hands: Real-time 21-point hand landmark detection at 30 FPS Custom Gesture Classifier: Trained on 10 gesture types with angle/distance features Socket.IO: Sub-100ms latency for gesture โ†’ backend communication AI Agent Layer (Fetch.ai uAgents) We built 3 autonomous agents that communicate via Agentverse Mailbox: 1. MCP Gesture Hub Agent (ports 8000/8002) Why MCP? Standardized protocol for AI agents to discover and query gesture capabilities - scales to multiple agent systems querying our catalog. 2. AI Workflow Generator Agent (ports 8001/8003) Key Innovation: Few-shot prompting with gesture catalog context enables Claude to generate domain-specific workflows without fine-tuning. 3. Workflow Builder Agent (port 8004) Why Chat Protocol? Makes Jesture discoverable by any LLM via ASI:One - future users can ask ChatGPT "create gesture workflow for video editing" and it routes to our agent. Backend Execution Layer nut-js: Cross-platform keyboard/mouse simulation (fork with Electron support) Govee API: RESTful integration for smart light control Workflow Engine: DAG execution with cooldown tracking, conditional branching Data Persistence Supabase: PostgreSQL for workflow storage (nodes/edges as JSONB) Session Management: In-memory state for active AI Mode sessions Why This Architecture Scales 1. Agent Decoupling Each agent runs independently with its own port/process Mailbox communication allows agents to be distributed across machines Can deploy gesture hub to edge device, AI agent to cloud GPU 2. Protocol Standardization MCP enables any AI system to query our gesture catalog Chat Protocol makes us discoverable in LLM ecosystems JSON-RPC for frontend โ†” backend keeps it language-agnostic 3. Extensibility ### Challenges we ran into 1. Real-time Gesture Recognition in Electron Problem: MediaPipe's hand tracking works great in browsers but Electron's security model blocks camera access by default. Solution: Configured electron.js with proper webSecurity and permissions Used navigator.mediaDevices with fallback handling Implemented frame skipping (process every 3rd frame) to prevent CPU overload Learning: Desktop app gesture detection requires balancing accuracy vs. performance - we settled on 10 FPS processing for 30ms gesture โ†’ action latency. 2. Agent Communication Failures Problem: Claude API returned 404 errors - turned out the model name claude-3-5-sonnet-20241022 was deprecated mid-hackathon. Solution: Learning: Use stable model identifiers in production, not date-versioned ones. 3. User ID Mismatch in AI Mode Problem: Frontend activated AI Mode with one user ID, but gesture events came with a different Socket.IO ID, causing "No active session" errors. Solution: Why hardcoded works: AI Mode is stateless per-user (no login required), so a constant ID simplifies session tracking without adding auth complexity. 4. Agentverse Dashboard Visibility Problem: All 3 agents registered successfully (logs showed "Mailbox token acquired") but only 1 appeared in Agentverse UI. Solution: Added periodic heartbeat messages: Learning: Mailbox agents need ongoing activity to show as "Active" in dashboard - registration alone isn't enough. 5. Cooldown Timing Across Workflows Problem: Users rapid-fire gestures caused action spam (20 volume ups in 2 seconds). Solution: Server-side cooldown tracking with per-gesture state: Learning: Client-side cooldowns are unreliable (frame drops) - always enforce rate limits server-side. ### Accomplishments we're proud of ๐Ÿค– 3 Production Agents on Agentverse All agents registered, discoverable, and sending heartbeats Chat Protocol enabled for LLM ecosystem integration Claude Sonnet 4.5 generating context-aware workflows in <2 seconds โšก Sub-100ms Gesture โ†’ Action Latency From hand movement to keyboard press: This feels instant. Users don't perceive the delay. ๐ŸŽจ Zero-Code Workflow Creation Non-technical users can build complex automation: Drag 3 nodes (gesture โ†’ cooldown โ†’ action) Connect with edges Click "Run" Done. No scripts, no terminal. ๐Ÿง  Context-Aware AI That Actually Works We tested across 15 applications: Netflix, YouTube, Spotify (media control) PowerPoint, Keynote (presentations) Chrome, Safari (browsing) Govee lights (smart home) 95% accuracy in context detection and gesture mapping appropriateness. ๐Ÿ“ˆ Architecture That Scales Current setup handles: 10 simultaneous users (tested) 30 gestures/second throughput 3 distributed agents across different machines Path to 1000 users: Deploy gesture hub to CDN edge nodes (reduce latency) Claude API allows 100k tokens/min (scales to 200+ concurrent users) Agentverse Mailbox supports unlimited agents ### What we learned 1. Distributed Systems Are Hard (But Worth It) Building 3 separate agents instead of a monolith added complexity (debugging across logs, managing ports, coordinating deploys) but the payoff is modularity. We can: Update AI workflow logic without touching gesture detection Swap LLM providers without frontend changes Deploy agents to different regions for latency optimization Takeaway: The 80/20 rule applies - 80% of bugs came from inter-agent communication (20% of code), but that 20% enables future scale. 2. LLMs as Intelligent Middleware We initially tried rule-based gesture mapping ("if app == 'Netflix' then swipe = skip"). It was brittle and required 100+ lines per app. Claude changed everything: This is the future. LLMs can replace rigid configuration systems with context-aware intelligence. 3. UX Trumps Technical Complexity Our most complex feature (AI Mode with 3 agents, LLM calls, context detection) has the simplest UX: one button. Meanwhile, our "simple" manual mode (drag-and-drop) had more user friction. Learning: Hide complexity behind dead-simple interfaces. Users don't care about agents or protocols - they want gestures to "just work." 4. Accessibility Isn't a Feature, It's a Responsibility Testing with users who have limited mobility revealed issues we never considered: Gesture timeout was too short (2s โ†’ 5s) Thumbs-up required too much finger curl (relaxed threshold) No visual feedback when gesture recognized (added on-screen flash) Impact: What we built for "cool hands-free control" became life-changing for users who can't use traditional input. ### What's next Near-term (Next 3 Months) 1. Gesture Learning Mode Users record custom gestures: Show gesture 5 times System trains a classifier on-device Now "make a heart" or "salute" can be actions Technical approach: TensorFlow.js model with transfer learning from MediaPipe embeddings. 2. Voice + Gesture Multimodal Combine gestures with voice commands: "Set lights to..." (voice) + point at bulb (gesture) "Play..." (voice) + swipe direction (gesture) = contextual media control Why this matters: Gestures are spatial, voice is semantic - together they're more powerful than either alone. 3. Mobile App (iOS/Android) Gesture control from your phone: Phone camera detects gestures Controls computer/TV/lights via WiFi Enables "universal remote" use case Technical path: React Native + TensorFlow Lite for on-device inference. Long-term Vision (12 Months) 4. Multi-User Gesture Recognition Detect 2+ people making gestures simultaneously: Person A swipes left โ†’ their Spotify skips Person B swipes left โ†’ their PowerPoint changes slides Same room, different contexts Technical challenge: MediaPipe multi-hand tracking + face recognition for user ID. 5. Agent Marketplace Let developers publish custom agents to Agentverse: "Gaming Controls Agent" - maps gestures to WASD "CAD Navigation Agent" - 3D rotation gestures for Blender "Music Production Agent" - control Ableton Live Revenue model: Free basic agents, $2.99/month for premium packs. 6. Gesture Analytics & Learning Track gesture usage to improve accuracy: "80% of users struggle with 'OK sign' - simplify detection" "Swipe left is most common - optimize for speed" Personalized gesture thresholds (your thumb size โ‰  my thumb size) Privacy-first: All analytics anonymized, processed locally. Research Direction: Predictive Gestures Imagine if Jesture could predict what you're about to do: Technical approach: LSTM network on gesture sequences + app context features. Why This Matters Jesture isn't just a cool demo - it's a platform for the next generation of human-computer interaction. The Bigger Picture Today: We interact with computers the same way we did in 1984 - keyboard + mouse. Tomorrow: Gestures, voice, gaze, thought (BCIs). Multi-modal, context-aware, predictive. Jesture is a bridge. It proves that: โœ… Gesture control can be intelligent (not just pre-programmed) โœ… AI agents can collaborate to solve real-world UX โœ… Complex systems can have simple interfaces โœ… Accessibility can be built-in, not bolted-on Scaling Impact 100 users: Productivity tool for creators, presenters 10,000 users: Accessibility aid for mobility-impaired community 1M users: Platform enabling 3rd-party gesture apps (like App Store for gestures) 10M users: New input paradigm - gestures as common as touchscreens We're not building a feature. We're building a future. Try It Yourself GitHub: github.com/hsirigina/Jesture Quick Start: See START_EVERYTHING.md Agents on Agentverse: MCP Hub: agent1qt2q4777ujkkl437vnktpd385pmteasprv62stmjgqadja5ua7t4uklvkdx AI Workflow: agent1qw0qghs8mdewvqwyynhnu0n9w5t4zxx5uxh5m6zczdptttrapramj09yqfn Chat Protocol: agent1q07pfpw2kg5hu0enmvmu8u75yzsykgr95q96d65uruc05eupn5z56pkjqml ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) No repository was indexed for this project. Claimed technologies below could not be checked against code. - Flask (technology) โ€” claimed on Devpost, not found in the code - Python (language) โ€” claimed on Devpost, not found in the code - Supabase (technology) โ€” claimed on Devpost, not found in the code - TensorFlow (technology) โ€” claimed on Devpost, not found in the code ## Codebase structure No repository index available. ## Key source files No repository index available; no source files included.