# Project export: Kinetic AI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Kinetic turns any camera into physical intelligence — it coaches movement with voice, watches spaces with goals you set, and understands your physical movements. - Devpost: https://devpost.com/software/kinetic-ai-ilvr9o - GitHub: https://github.com/rajashekarcs2023/treehacks2026-kinetic - Demo: https://frontend-qh6de6ixx-rajashekarvs-projects.vercel.app/ - Video: https://www.youtube.com/embed/Z331RsxLI8Q?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — rajashekarcs2023 (73 commits) ## Devpost submission (written by the team) ### Inspiration AI helps us write, code, and reason — but human physical capability remains unaugmented. Quality coaching costs $80–200/hour, PT is $100+/hour, and 55 million elderly Americans need fall monitoring that costs hospitals billions in patient sitter programs annually. These aren't separate problems. They're all the same gap: no cognitive layer for physical capability. We asked: what if one camera and one AI could coach a squat, detect a grandmother falling, guide PT rehab, and monitor a hospital room overnight? What It Does Kinetic is a real-time Physical Movement Intelligence platform with four modes from one stack: AI Skill Coach — Describe any movement and Kinetic generates the ideal form via a 4-tier pipeline (templates → Claude mapping → Claude angle generation → HY-Motion diffusion on A100 GPU). It tracks your body at 30 FPS, scores form with a triple-metric engine, and coaches via real-time voice. Physical Therapy — PT-specific categories (knee, shoulder, hip, ankle rehab). Tracks range of motion, enforces safety limits, auto-counts reps via phase detection, and adapts voice coaching to gentle rehab context. Goal-Based Intelligence — Give Kinetic any goal and step away. Presets include fall detection, desk security, posture watch, driver alertness, and study focus. It runs a continuous perception→reasoning→action loop, sending Telegram alerts with photos autonomously. Clinical Patient Safety — Hospital-grade monitoring: fall detection, bed exit alerts, immobility/pressure ulcer prevention, IV line safety, post-op distress, and wandering/elopement detection. Alerts nurses via Telegram with snapshot photos instantly. 46 MCP tools are exposed via HTTP so external AI agents (including Poke) can drive the entire stack programmatically. How We Built It DGX Spark — YOLOv8n-pose on the GB10 Superchip for sub-50ms edge pose estimation at 15+ FPS. (17 key points) Modal A100 — Tencent's HY-Motion 1.0-Lite (0.46B param text-to-3D motion diffusion, Dec 2025) generates 30-frame skeleton sequences from text prompts in ~26s. Three model weights (~20GB) cached in Modal Volumes. https://rajashekarvennavelli--aegis-motion-generate-endpoint.modal.run` Claude Agent SDK — 46 MCP tools across 12 categories, 3 sub-agents (perception, coach, communicator), safety hooks, and audit logging. Claude reasons about movement, coaching, and spatial safety across all modes. OpenAI Realtime Voice — Bidirectional audio with interruption handling and context injection every 5 seconds. Voice adapts per mode: gym coach, PT encourager, or urgent fall alerter. Triple-Metric Scoring — Gaussian joint angles (0.4) + cosine spatial similarity (0.3) + COCO OKS (0.3) for scoring that catches errors single metrics miss. Autonomous Monitoring — Background loop every 3s detects falls, bed exits, immobility, and wandering. Sends Telegram photo alerts, voice alerts, and Claude analysis simultaneously. Controllable via Telegram commands. Frontend — Next.js + Tailwind + shadcn/ui. Four mode cards, coaching UI with 10 skill categories, monitoring dashboard with live camera + alert timeline + agent tool call visualization. Challenges We Ran Into HY-Motion on Modal — 20GB of model weights, relative path issues, dozens of build iterations, and mapping 52-joint output to MediaPipe 33-point format. The hardest single technical challenge. https://rajashekarvennavelli--aegis-motion-generate-endpoint.modal.run Real-time scoring at 30 FPS — DTW temporal alignment, phase detection, and triple-metric computation all under 5ms per frame via pre-normalization and caching. Voice + vision sync — Three-way latency (audio 200ms, video 50ms, Claude 1-2s) solved with context injection architecture — voice always references current form, not stale data. Clinical reliability — Fall detection can't have false negatives. Temporal smoothing, priority boosting for safety-critical activities, and hysteresis prevent state flickering. Accomplishments That We're Proud Of Four modes, one stack — coaching, PT, autonomous monitoring, and clinical safety all share the same CV + agent + voice pipeline Working text-to-3D motion — HY-Motion generating real physics-aware skeleton sequences, animated live on the frontend Autonomous fall → Telegram pipeline — zero human intervention from detection to nurse alert with photo 46 MCP tools as HTTP — the entire physical intelligence stack is callable by any external AI agent What We Learned Physical intelligence is AI's next frontier — coaching, rehab, monitoring, and safety are all the same capability: perceiving movement and reasoning about what should change. Edge AI changes everything — sub-50ms pose estimation makes coaching feel human and fall detection truly real-time. Voice is the only viable physical interface — you can't touch a screen mid-pushup. Realtime bidirectional voice makes AI coaching natural. Modularity enables pivoting — separate subsystems let us add PT, monitoring, and clinical modes without breaking existing functionality. What's Next for Kinetic AR glasses — overlay the expert skeleton directly in the user's field of view for truly hands-free coaching Multi-camera fusion — stereo depth for accurate 3D joint angles, eliminating single-camera occlusion blind spots Clinical integration — FHIR/EHR export of ROM data, rep counts, and progress for clinician dashboards Hospital deployment — ceiling-mounted cameras augmenting clinical staff, enabling one nurse to safely monitor multiple rooms simultaneously Personalized models — fine-tuned to individual body proportions and injury history for adaptive coaching thresholds ## README (from the GitHub repository) # Kinetic — Real-Time Physical Movement Intelligence > **"One camera. One AI. Coaches your squat, detects a fall, guides PT rehab, monitors a hospital room."** Kinetic is a unified platform for **physical movement intelligence** — AI skill coaching, physical therapy rehab, autonomous space monitoring, and clinical patient safety — all from the same CV + agent + voice stack. 4 modes. 46 MCP tools. 10 ML models. 17,000+ lines. Built solo in 20 hours at TreeHacks 2026. --- ## Demo 🎥 [Loom Demo Video](https://youtu.be/xTkgjQ2uvHc) 🌐 [Live Frontend](https://frontend-qh6de6ixx-rajashekarvs-projects.vercel.app) **Infrastructure:** - 🖥️ **DGX Spark (Edge)**: Real-time pose, fall detection, monitoring — all on-device, zero cloud dependency > ⚡ **Why both edge and cloud?** DGX Spark's Blackwell GPU doesn't have PyTorch/CUDA wheels for ARM yet — so safety-critical inference (pose, falls) runs on-device at sub-50ms, while one-time heavy generation (0.46B params) offloads to A100. Result is cached locally; after that, coaching is 100% edge. - 🚀 **Modal A100 (Cloud)**: HY-Motion 1.0-Lite text-to-3D motion generation — **Endpoint**: `POST https://rajashekarvennavelli--aegis-motion-generate-endpoint.modal.run` - 🤖 **46 MCP Tools**: Full stack exposed via HTTP for external AI agents (Poke) --- ## 4 Modes, One Stack ``` ┌─────────────────────────────────────────────────────────┐ │ KINETIC PLATFORM │ ├──────────────┬──────────────┬──────────┬────────────────┤ │ 🏋️ Coaching │ 🩺 PT Rehab │ 🎯 Goals │ 🏥 Clinical │ │ │ │ │ │ │ Any skill │ Knee/shoulder│ Fall │ Fall detection │ │ Voice coach │ Safe ROM │ Posture │ Bed exit alert │ │ AI expert │ Rep counting │ Security │ Immobility │ │ Scoring │ Voice rehab │ Focus │ Wandering │ ├──────────────┴──────────────┴──────────┴────────────────┤ │ CV Pipeline: YOLO + MediaPipe + ByteTrack + Depth │ │ Agent: Claude SDK (3 sub-agents, 46 MCP tools, hooks) │ │ Voice: OpenAI Realtime (bidirectional, interruption) │ │ Edge AI: DGX Spark + Modal A100 (HY-Motion 1.0) │ │ Alerts: Telegram + Voice + Frontend dashboard │ └─────────────────────────────────────────────────────────┘ ``` --- ## 6 Infrastructure Pillars ### 1. 🟢 NVIDIA DGX Spark — Edge AI Inference - **GB10 Superchip** (Grace ARM CPU + Blackwell GPU) - **YOLOv8n-pose**: 17-keypoint real-time pose estimation - Runs on-premise with low latency — no cloud roundtrip for pose - Endpoints: `POST /predict`, `GET /health` ### 2. 🔵 Modal + NVIDIA A100 — Cloud GPU for Motion Generation - **HY-Motion 1.0-Lite** (Tencent, SOTA Dec 2025) - 0.46B parameters, DiT + Flow Matching architecture - Trained on 3,000+ hours of 3D motion data (pretrain → finetune → RLHF) - Text prompt → SMPL 22-joint 3D skeleton → MediaPipe 33-point conversion - **Endpoint**: `POST https://rajashekarvennavelli--aegis-motion-generate-endpoint.modal.run` - Serverless A100, scales to zero, $530 credits available ### 3. 🟠 Anthropic Claude Agent SDK — AI Orchestration - **Claude Sonnet 4** as main orchestrator - **3 Sub-Agents**: Perception (11 tools), Coach (14 tools), Progress (10 tools) - **44 MCP Tools** via Model Context Protocol - **3 Agent Hooks**: Safety guard, audit log, session summary - Handles: coaching decisions, expert generation, form analysis, goal tracking ### 4. 🎙️ OpenAI Realtime API — Voice Coaching - **GPT-4o Realtime Preview** with `alloy` voice - **3-Layer Interruption System**: - Layer 1: Server-side VAD (50ms speech detection) - Layer 2: Response state machine (prevents audio overlap) - Layer 3: Single voice source (proactive coaching + reactive Q&A) - Punchy prompts: max 15 words, no filler - Browser `speechSynthesis` TTS fallback for offline use ### 5. 👁️ Computer Vision Pipeline | Model | Purpose | Size | Speed | |---|---|---|---| | YOLO11n | Person detection | 5.4 MB | 15 FPS | | MediaPipe Pose | 33 body landmarks | 5.6 MB | 30 FPS | | MediaPipe Hands | 21 hand landmarks/hand | ~5 MB | 30 FPS | | Depth Anything V2 | Monocular depth | ~50 MB | 22 FPS | | ByteTrack | Multi-person tracking | — | 15 FPS | ### 6. 📊 Triple-Metric Pose Scoring | Metric | Weight | What it measures | |---|---|---| | **Gaussian Angle Scoring** (16 joints) | 50% | Per-joint angular accuracy with tuned σ | | **Cosine Spatial Similarity** | 30% | Overall pose shape matching | | **COCO OKS** | 20% | Industry-standard keypoint similarity | Plus: DTW temporal alignment, phase detection, rep counting, compensation detection. --- ## AI Expert Generation — No Video Required Kinetic can coach **any skill** without a reference video: | Tier | Method | Latency | How | |---|---|---|---| | 1 | Semantic Alias Lookup | 0ms | 53 aliases → 10 canonical exercises | | 2 | Claude Semantic Mapping | ~0.5s | Claude maps novel names to known exercises | | 3 | Claude Angle Generation | ~1-2s | Claude generates biomechanically correct angles | | 4 | HY-Motion 3D Generation | ~5-15s | Full 3D motion on Modal A100 GPU | **10 built-in exercises**: squat, pushup, lunge, deadlift, shoulder press, bicep curl, plank, jumping jack, warrior II, tree pose --- ## Complete Model Zoo | Model | Company | Purpose | Location | |---|---|---|---| | **YOLOv8n-pose** | Ultralytics | 17-keypoint pose | DGX Spark | | **YOLO11n** | Ultralytics | Person detection | Local | | **MediaPipe Pose Lite** | Google | 33 body landmarks | Local | | **MediaPipe Hands** | Google | 21 hand landmarks | Local | | **Depth Anything V2** | ByteDance | Monocular depth | Local (MPS) | | **ByteTrack** | ByteDance | Multi-person tracking | Local | | **HY-Motion 1.0-Lite** | Tencent | Text → 3D motion (SOTA) | Modal A100 | | **Claude Sonnet 4** | Anthropic | Agent orchestration | Cloud API | | **GPT-4o Realtime** | OpenAI | Voice coaching | Cloud API | | **Custom 1D CNN** | Built in-house | Pose scoring (14K params) | Local (NumPy) | --- ## Tech Stack | Layer | Technology | |---|---| | **Frontend** | Next.js 14 + React + TailwindCSS + shadcn/ui | | **Backend** | FastAPI + Python 3.12 (44 routes, 3 WebSockets) | | **AI Orchestration** | Claude Agent SDK (3 sub-agents, 44 MCP tools, 3 hooks) | | **Voice AI** | OpenAI Realtime API (GPT-4o) + browser TTS fallback | | **Edge AI** | NVIDIA DGX Spark (GB10 Superchip, YOLOv8n-pose) | | **Cloud GPU** | Modal + NVIDIA A100 (HY-Motion 1.0-Lite, 0.46B params) | | **Computer Vision** | YOLO11n + MediaPipe Pose/Hands + ByteTrack + Depth Anything V2 | | **Pose Scoring** | Gaussian angles + Cosine spatial + COCO OKS | | **Protocol** | MCP (Model Context Protocol) — 44 tools | --- ## Quick Start ### Prerequisites - Python 3.12+ - Node.js 18+ - API Keys: `ANTHROPIC_API_KEY`, `OPENAI_API_KEY` ### Backend ```bash cd aegis pip install -r requirements.txt python run_server.py # Server starts at http://localhost:8000 ``` ### Frontend ```bash cd frontend npm install npm run dev # UI at http://localhost:3000 ``` ### Test Motion Generation (Modal A100) ```bash curl -X POST https://rajashekarvennavelli--aegis-motion-generate-endpoint.modal.run \ -H "Content-Type: application/json" \ -d '{"prompt": "a person doing a squat", "num_frames": 60}' ``` ### DGX Spark Edge Inference ```bash # On DGX Spark (gx10-eb94) cd dgx pip install -r requirements.txt python inference_server.py # Pose endpoint at http://:8080/predict ``` --- ## Project Structure ``` kinetic/ ├── aegis/ # Backend (internal codename) │ ├── server.py # FastAPI server (44 routes, 3 WebSockets) │ ├── spatial_engine.py # CV pipeline (YOLO + MediaPipe + ByteTrack + Depth) │ ├── pose_comparison.py # Triple-metric scoring (Gaussian + Cosine + OKS) │ ├── ai_expert.py # AI expert generation (aliases + Claude + HY-Motion) │ ├── sdk_agent.py # Claude Agent SDK (3 sub-agents, hooks) │ ├── mcp_serv [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 81 recognized source files, 1055 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (96 of 96) ``` .claude/skills/coaching.md .claude/skills/perception.md .claude/skills/progress.md .gitignore aegis-architecture.jsx aegis/__init__.py aegis/activity_model.py aegis/activity.py aegis/agent.py aegis/ai_expert.py aegis/config.py aegis/data_collector.py aegis/dgx_client.py aegis/gemini_bridge.py aegis/goals.py aegis/hybrid_scorer.py aegis/main.py aegis/mcp_server.py aegis/memory.py aegis/monitor.py aegis/multi_agent.py aegis/openai_voice.py aegis/pose_comparison.py aegis/rooms.py aegis/run_mcp_server.py aegis/run_server.py aegis/sdk_agent.py aegis/sdk_tools.py aegis/server.py aegis/skill_graph.py aegis/skill_scorer_torch.py aegis/skill_scorer.py aegis/spatial_engine.py aegis/static/app.js aegis/static/dashboard.html aegis/static/index.html aegis/telegram_bot.py aegis/video_ingest.py aegis/voice.py data/skill_graphs/fitness.json data/skill_graphs/pt_rehab_knee.json data/skill_graphs/pt_rehab_shoulder.json data/skill_graphs/yoga.json DEVPOST.md dgx/download_models.sh dgx/inference_server.py dgx/modal_motion.py dgx/setup_mld.sh dgx/setup_motion.sh frontend/.gitignore frontend/components.json frontend/eslint.config.mjs frontend/netlify.toml frontend/next.config.ts frontend/package.json frontend/postcss.config.mjs frontend/README.md frontend/src/app/architecture/page.tsx frontend/src/app/coach/page.tsx frontend/src/app/globals.css frontend/src/app/history/page.tsx frontend/src/app/layout.tsx frontend/src/app/monitor/page.tsx frontend/src/app/page.tsx frontend/src/app/paper/page.tsx frontend/src/app/product/page.tsx frontend/src/app/settings/page.tsx frontend/src/app/skills/page.tsx frontend/src/components/score-ring.tsx frontend/src/components/sidebar.tsx frontend/src/components/skeleton-player.tsx frontend/src/components/ui/avatar.tsx frontend/src/components/ui/badge.tsx frontend/src/components/ui/button.tsx frontend/src/components/ui/card.tsx frontend/src/components/ui/progress.tsx frontend/src/components/ui/scroll-area.tsx frontend/src/components/ui/separator.tsx frontend/src/components/ui/tabs.tsx frontend/src/components/ui/tooltip.tsx frontend/src/lib/api.ts frontend/src/lib/utils.ts frontend/tsconfig.json models/pose_landmarker_full.task models/pose_landmarker_lite.task README.md requirements.txt run.py src/__init__.py src/dashboard.py src/decision.py src/intervention.py src/models.py src/motion.py src/perception.py src/risk.py ``` ### Dependencies - frontend/package.json: @radix-ui/react-avatar@^1.1.11, @radix-ui/react-dialog@^1.1.15, @radix-ui/react-dropdown-menu@^2.1.16, @radix-ui/react-progress@^1.1.8, @radix-ui/react-scroll-area@^1.2.10, @radix-ui/react-separator@^1.1.8, @radix-ui/react-slot@^1.2.4, @radix-ui/react-tabs@^1.1.13, @radix-ui/react-tooltip@^1.2.8, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9, eslint-config-next@16.1.6, lucide-react@^0.564.0, next@16.1.6, radix-ui@^1.4.3, react@19.2.3, react-dom@19.2.3, recharts@^3.7.0, shadcn@^3.8.4, tailwind-merge@^3.4.0, tailwindcss@^4, tw-animate-css@^1.4.0, typescript@^5 - requirements.txt: anthropic@>=0.40.0, claude-agent-sdk@>=0.1.36, fastapi@>=0.109.0, fastmcp@>=2.14.5, flask@>=3.0.0, google-genai@>=1.0.0, httpx@>=0.27.0, lap@>=0.4.0, matplotlib@>=3.7.0, mediapipe@>=0.10.9, numpy@>=1.24.0, openai@>=1.0.0, opencv-python@>=4.8.0, Pillow@>=10.0.0, python-dotenv@>=1.0.0, python-multipart@>=0.0.6, requests@>=2.31.0, torch@>=2.1.0, torchvision@>=0.16.0, transformers@>=4.36.0, ultralytics@>=8.1.0, uvicorn@>=0.27.0, websockets@>=12.0 ### Recent commits (newest first) - update README - added custom goals - Update README - Re-add DEVPOST.md - Remove DEVPOST.md from repo - Update README.md - Update README.md - Update README.md - Update README.md - Update README: reflect full product (4 modes, 46 MCP tools, clinical monitoring, not just skill coach) - Fix pose skeleton overlay: correct PoseLandmarks.points access, robust pose-to-person matching, add tracked persons panel to monitor sidebar - Redesign monitor: full-screen layout with big camera feed + sidebar alerts/tools - Fix monitor page: backend /api/frame endpoint, use backend camera feed, fix tool call field mapping, separate Goal-Based vs Clinical modes - Final submission: 4 modes (Skill Coach, PT, Goal-Based Intelligence, Clinical Patient Safety), 17 backend goals, enhanced monitoring loop, trimmed DEVPOST.md, cleaned up internal docs - UI: colored buttons LEFT of greeting, architecture page, text labels in sidebar - UI: text labels in sidebar, Product + Tech Paper buttons on dashboard, /paper route with academic formatting - Product landing page with tech paper, sidebar nav update - Direct Modal A100 integration: backend calls HY-Motion endpoint directly, no DGX proxy needed - Modal HY-Motion WORKING: keypoints3d extraction, batch dim fix, 26s generation on A100 - Modal HY-Motion: add CLIP + Qwen3 text encoders, fastapi, LFS, README Kinetic branding, architecture diagram ## Key source files (fetched from GitHub, selected and truncated for size) ### DEVPOST.md ```markdown # Kinetic — Real-Time Physical Movement Intelligence ## Inspiration AI helps us write, code, and reason — but human physical capability remains unaugmented. Quality coaching costs $80–200/hour, PT is $100+/hour, and 55 million elderly Americans need fall monitoring that costs hospitals billions in patient sitter programs annually. These aren't separate problems. They're all the same gap: no cognitive layer for physical capability. We asked: what if one camera and one AI could coach a squat, detect a grandmother falling, guide PT rehab, and monitor a hospital room overnight? ## What It Does Kinetic is a real-time Physical Movement Intelligence platform with four modes from one stack: **AI Skill Coach** — Describe any movement and Kinetic generates the ideal form via a 4-tier pipeline (templates → Claude mapping → Claude angle generation → HY-Motion diffusion on A100 GPU). It tracks your body at 30 FPS, scores form with a triple-metric engine, and coaches via real-time voice. **Physical Therapy** — PT-specific categories (knee, shoulder, hip, ankle rehab). Tracks range of motion, enforces safety limits, auto-counts reps via phase detection, and adapts voice coaching to gentle rehab context. **Goal-Based Intelligence** — Give Kinetic any goal and step away. Presets include fall detection, desk security, posture watch, driver alertness, and study focus. It runs a continuous perception→reasoning→action loop, sending Telegram alerts with photos autonomously. **Clinical Patient Safety** — Hospital-grade monitoring: fall detection, bed exit alerts, immobility/pressure ulcer prevention, IV line safety, post-op distress, and wandering/elopement detection. Alerts nurses via Telegram with snapshot photos instantly. 46 MCP tools are exposed via HTTP so external AI agents (including Poke) can drive the entire stack programmatically. ## How We Built It **DGX Spark** — YOLOv8n-pose on the GB10 Superchip for sub-50ms edge pose estimation at 15+ FPS. **Modal A100** — Tencent's HY-Motion 1.0-Lite (0.46B param text-to-3D motion diffusion, Dec 2025) generates 30-frame skeleton sequences from text prompts in ~26s. Three model weights (~20GB) cached in Modal Volumes. **Claude Agent SDK** — 46 MCP tools across 12 categories, 3 sub-agents (perception, coach, communicator), safety hooks, and audit logging. Claude reasons about movement, coaching, and spatial safety across all modes. **OpenAI Realtime Voice** — Bidirectional audio with interruption handling and context injection every 5 seconds. Voice adapts per mode: gym coach, PT encourager, or urgent fall alerter. **Triple-Metric Scoring** — Gaussian joint angles (0.4) + cosine spatial similarity (0.3) + COCO OKS (0.3) for scoring that catches errors single metrics miss. **Autonomous Monitoring** — Background loop every 3s detects falls, bed exits, immobility, and wandering. Sends Telegram photo alerts, voice alerts, and Claude analysis simultaneously. Controllable via Telegram commands. **Frontend** — [truncated — 2814 more characters] ``` ### .claude/skills/progress.md ```markdown --- name: skill-progress description: Skill progression tracking, expert references, goals, and memory management. Use when the user asks about their history, wants to set goals, manage references, or review progress. --- # Skill Progress ## Expert References - `list_references` — show all stored expert movements - `load_reference_from_current` — capture live camera as reference - `record_reference_start` / `record_reference_stop` — record a reference sequence ## Goal Management 12 preset goals available: - **Spatial**: desk_watch, posture_coach, driver_monitor, study_focus, elderly_care, general - **Coaching**: skill_coach, pt_rehab, fitness_trainer, dance_teacher, sports_coach, zero_shot_coach - `get_current_goal` — see active goal - `update_goal` — change goal via natural language - `get_goal_presets` — list all presets ## Memory - `save_observation` — store important observations for future sessions - `get_observations` — recall past observations by tag ## Skill Document Parsing - `parse_skill_document` — feed PT protocols, yoga guides, exercise docs - Extracts target angles, phases, rep counts, safety boundaries ``` ### requirements.txt ``` # Core CV & ML opencv-python>=4.8.0 ultralytics>=8.1.0 mediapipe>=0.10.9 numpy>=1.24.0 # Depth estimation torch>=2.1.0 torchvision>=0.16.0 transformers>=4.36.0 # Tracking lap>=0.4.0 # Utilities Pillow>=10.0.0 matplotlib>=3.7.0 # Dashboard flask>=3.0.0 # AEGIS Agent anthropic>=0.40.0 claude-agent-sdk>=0.1.36 requests>=2.31.0 python-dotenv>=1.0.0 httpx>=0.27.0 # Server fastapi>=0.109.0 uvicorn>=0.27.0 python-multipart>=0.0.6 # Voice google-genai>=1.0.0 # MCP fastmcp>=2.14.5 # OpenAI Realtime Voice websockets>=12.0 openai>=1.0.0 ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@radix-ui/react-avatar": "^1.1.11", "@radix-ui/react-dialog": "^1.1.15", "@radix-ui/react-dropdown-menu": "^2.1.16", "@radix-ui/react-progress": "^1.1.8", "@radix-ui/react-scroll-area": "^1.2.10", "@radix-ui/react-separator": "^1.1.8", "@radix-ui/react-slot": "^1.2.4", "@radix-ui/react-tabs": "^1.1.13", "@radix-ui/react-tooltip": "^1.2.8", "class-variance-authority": "^0.7.1", "clsx": "^2.1.1", "lucide-react": "^0.564.0", "next": "16.1.6", "radix-ui": "^1.4.3", "react": "19.2.3", "react-dom": "19.2.3", "recharts": "^3.7.0", "shadcn": "^3.8.4", "tailwind-merge": "^3.4.0", "tw-animate-css": "^1.4.0" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.1.6", "tailwindcss": "^4", "typescript": "^5" } } ``` ### aegis/main.py ```python """ AEGIS Main Entry Point ====================== Starts all components: 1. SpatialEngine (CV pipeline in background thread) 2. TelegramBot (user interface, polling in background) 3. Monitor (proactive heartbeat loop in background) 4. AegisAgent (Claude brain, called by bot and monitor) Usage: # Full system (needs ANTHROPIC_API_KEY, TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID): python -m aegis.main # Without Telegram (agent prints to console): python -m aegis.main --no-telegram # Test spatial engine only (no agent): python -m aegis.main --engine-only Environment variables: ANTHROPIC_API_KEY — Required for agent TELEGRAM_BOT_TOKEN — Required for Telegram bot TELEGRAM_CHAT_ID — Auto-detected on first message if not set """ import argparse import asyncio import signal import sys import time from aegis import config from aegis.spatial_engine import SpatialEngine from aegis.sdk_agent import AegisSDKAgent from aegis.telegram_bot import TelegramBot from aegis.monitor import Monitor from aegis.voice import VoiceNarrator def parse_args(): parser = argparse.ArgumentParser(description="AEGIS — Spatial Intelligence Agent") parser.add_argument("--no-telegram", action="store_true", help="Run without Telegram bot (console only)") parser.add_argument("--engine-only", action="store_true", help="Run spatial engine only (no agent, no bot)") parser.add_argument("--no-monitor", action="store_true", help="Disable proactive monitoring loop") parser.add_argument("--camera", type=int, default=None, help="Camera index (overrides AEGIS_CAMERA env)") parser.add_argument("--heartbeat", type=float, default=None, help="Heartbeat interval in seconds") parser.add_argument("--show-camera", action="store_true", help="Show live camera feed with overlays") parser.add_argument("--voice", action="store_true", help="Enable voice narration (TTS)") parser.add_argument("--voice-interval", type=float, default=4.0, help="Seconds between voice narrations (default 4)") return parser.parse_args() def main(): args = parse_args() # Apply overrides if args.camera is not None: config.CAMERA_INDEX = args.camera if args.heartbeat is not None: config.HEARTBEAT_INTERVAL = args.heartbeat print("=" * 60) print(" AEGIS — Autonomous Spatial Intelligence Agent") print(" OpenClaw for Physical Space") print("=" * 60) print() # ── 1. Start Spatial Engine ────────────────────────────────────────── print("[1/4] Starting Spatial Engine...") engine = SpatialEngine(show_camera=args.show_camera) engine.start() # Wait for first frame print(" Waiting for camera...") for _ in range(50): # 5 second timeout if engine.get_state(): break time.sleep(0.1) if not engine.get_state(): print("ERROR: Spatial engine failed to start. Check camera.") engine.stop() sys.exit(1) print(" ✓ Camera active, CV pipeline running") if args.engine_only: print("\n[Engine-only mode] Printing spatial state every 2 seconds. Ctrl+C to stop.\n") try: while True: print(engine.get_summary()) print() time.sleep(2) except KeyboardInterrupt: engine.stop() return # ── 2. Initialize Agent ────────────────────────────────────────────── print("[2/4] Initializing Agent...") if not config.ANTHROPIC_API_KEY: print(" ⚠ ANTHROPIC_API_KEY not set. Agent will not work.") print(" Set it: export ANTHROPIC_API_KEY=your-key") engine.stop() sys.exit(1) # Telegram bot (for sending) telegram = TelegramBot() # Create send function for the agent def telegram_sender(message, photo_path=None): telegram.send_message(message, photo_path) agent = AegisSDKAgent() telegram.agent = agent print(" ✓ Agent ready (claude-agent-sdk + 3 sub-agents)") # ── 3. Start Telegram Bot ──────────────────────────────────────────── if not args.no_telegram: print("[3/4] Starting Telegram Bot...") if telegram.is_configured: telegram.start_polling() telegram.send_message("🤖 *AEGIS Online*\nSpatial intelligence agent is active. Send /help for commands.") print(" ✓ Telegram bot active") else: print(" ⚠ Telegram not configured (set TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID)") print(" Running in console mode.") else: print("[3/4] Telegram bot disabled (--no-telegram)") # ── 4. Start Monitor ───────────────────────────────────────────────── monitor = None if not args.no_monitor: print("[4/4] Starting Monitor...") monitor = Monitor(engine, agent) monitor.start() print(" ✓ Proactive monitoring active") else: print("[4/4] Monitor disabled (--no-monitor)") # Connect monitor to telegram bot for remote control telegram.monitor = monitor # ── 5. Voice Narrator (optional) ───────────────────────────────────── narrator = VoiceNarrator(enabled=args.voice, min_interval=args.voice_interval) if args.voice: narrator.start() # ── Ready ──────────────────────────────────────────────────────────── print() print("=" * 60) print(" AEGIS is running. Components:") print(f" Spatial Engine: ✓ (camera {config.CAMERA_INDEX})") print(f" Agent: ✓ ({config.AGENT_MODEL})") print(f" Telegram: {'✓' if telegram.is_configured and not args.no_telegram else '✗'}") print(f" Monitor: {'✓' if monitor else '✗'} (every {config.HEARTBEAT_INTERVAL}s)") print(f" Voice: [truncated — 2851 more characters] ``` ### frontend/src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import { TooltipProvider } from "@/components/ui/tooltip"; import { Sidebar } from "@/components/sidebar"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "Kinetic AI — AI Skill Coach", description: "Real-time AI skill coaching with expert motion transfer", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return (
{children}
 ); } ``` ### aegis/static/app.js ```javascript /** * AEGIS Phone Web App * * Architecture: * Camera → captures frames → WebSocket → AEGIS backend (CV pipeline) * Backend → spatial state → WebSocket → overlay on phone screen * Mic audio → Gemini Live (direct client-to-server) → speaker * Spatial state injected into Gemini Live as text context * * Clean separation: * - VideoStream: handles camera + frame sending * - SpatialOverlay: draws bounding boxes + labels on canvas * - GeminiVoice: handles Gemini Live audio I/O * - App: orchestrates everything */ // ═══════════════════════════════════════════════════════════════════════ // VideoStream — Camera capture + WebSocket frame streaming // ═══════════════════════════════════════════════════════════════════════ class VideoStream { constructor() { this.video = document.getElementById('camera-video'); this.ws = null; this.stream = null; this.sendCanvas = document.createElement('canvas'); this.sendCtx = this.sendCanvas.getContext('2d'); this.facingMode = 'environment'; // rear camera this.sending = false; this.frameInterval = null; this.targetFPS = 8; // frames sent to server per second this.onStateUpdate = null; // callback for spatial state } async startCamera() { try { this.stream = await navigator.mediaDevices.getUserMedia({ video: { facingMode: this.facingMode, width: { ideal: 640 }, height: { ideal: 480 }, }, audio: false, // audio handled separately by Gemini }); this.video.srcObject = this.stream; await this.video.play(); return true; } catch (err) { console.error('Camera error:', err); return false; } } async flipCamera() { this.facingMode = this.facingMode === 'environment' ? 'user' : 'environment'; if (this.stream) { this.stream.getTracks().forEach(t => t.stop()); } return this.startCamera(); } connectWebSocket() { const protocol = location.protocol === 'https:' ? 'wss:' : 'ws:'; const wsUrl = `${protocol}//${location.host}/ws/video`; this.ws = new WebSocket(wsUrl); this.ws.onopen = () => { console.log('[Video] WebSocket connected'); this.sending = true; this.startSending(); }; this.ws.onmessage = (event) => { const msg = JSON.parse(event.data); if (msg.type === 'state' && this.onStateUpdate) { this.onStateUpdate(msg.data); } }; this.ws.onclose = () => { console.log('[Video] WebSocket disconnected'); this.sending = false; this.stopSending(); }; this.ws.onerror = (err) => { console.error('[Video] WebSocket error:', err); }; } startSending() { if (this.frameInterval) return; this.frameInterval = setInterval(() => this.sendFrame(), 1000 / this.targetFPS); } stopSending() { if (this.frameInterval) { clearInterval(this.frameInterval); this.frameInterval = null; } } sendFrame() { if (!this.sending || !this.ws || this.ws.readyState !== WebSocket.OPEN) return; if (!this.video.videoWidth) return; // Scale down for bandwidth const scale = 0.5; this.sendCanvas.width = this.video.videoWidth * scale; this.sendCanvas.height = this.video.videoHeight * scale; this.sendCtx.drawImage(this.video, 0, 0, this.sendCanvas.width, this.sendCanvas.height); // Convert to JPEG base64 const dataUrl = this.sendCanvas.toDataURL('image/jpeg', 0.6); const base64 = dataUrl.split(',')[1]; this.ws.send(JSON.stringify({ type: 'frame', data: base64 })); } stop() { this.sending = false; this.stopSending(); if (this.ws) this.ws.close(); if (this.stream) this.stream.getTracks().forEach(t => t.stop()); } } // ═══════════════════════════════════════════════════════════════════════ // SpatialOverlay — Draws bounding boxes, labels, activities on canvas // ═══════════════════════════════════════════════════════════════════════ class SpatialOverlay { constructor() { this.canvas = document.getElementById('overlay-canvas'); this.ctx = this.canvas.getContext('2d'); this.lastState = null; } update(state) { this.lastState = state; this.draw(); } draw() { const state = this.lastState; if (!state) return; const video = document.getElementById('camera-video'); const rect = video.getBoundingClientRect(); // Match canvas to video display size this.canvas.width = rect.width; this.canvas.height = rect.height; const ctx = this.ctx; ctx.clearRect(0, 0, this.canvas.width, this.canvas.height); const frameW = state.frame_size?.width || 640; const frameH = state.frame_size?.height || 480; const scaleX = rect.width / frameW; const scaleY = rect.height / frameH; // ── Draw persons ──────────────────────────────── for (const p of (state.persons || [])) { const b = p.bbox; const x1 = b.x1 * scaleX, y1 = b.y1 * scaleY; const w = (b.x2 - b.x1) * scaleX, h = (b.y2 - b.y1) * scaleY; const activity = p.activity || ''; // Color based on activity let color = '#00ff88'; if (activity === 'fallen' || activity === 'lying_down') color = '#ff4444'; else if (activity === 'running') color = '#ffaa00'; else if (activity === 'waving') color = '#00aaff'; // Bounding box ctx.strokeStyle = colo [truncated — 18006 more characters] ``` ### frontend/src/app/page.tsx ```typescript "use client"; import { useEffect, useState } from "react"; import Link from "next/link"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; import { Button } from "@/components/ui/button"; import { Badge } from "@/components/ui/badge"; import { ScoreRing } from "@/components/score-ring"; import { Video, Flame, TrendingUp, Clock, ChevronRight, Brain, Sparkles, Dumbbell, Music, Swords, HeartPulse, Hand, Monitor, Globe, FileText, Layers, Eye, Shield, Stethoscope, Activity, } from "lucide-react"; interface SessionSummary { id: string; skill: string; category: string; score: number; reps: number; duration: string; time: string; trend: "up" | "down" | "same"; } const MOCK_SESSIONS: SessionSummary[] = [ { id: "1", skill: "Knee Extension", category: "PT Rehab", score: 85, reps: 10, duration: "4:30", time: "1h ago", trend: "up" }, { id: "2", skill: "Warrior Pose", category: "Yoga", score: 91, reps: 6, duration: "6:15", time: "2h ago", trend: "up" }, { id: "3", skill: "Sit-to-Stand", category: "Elderly", score: 78, reps: 8, duration: "3:00", time: "3h ago", trend: "up" }, { id: "4", skill: "ASL Alphabet", category: "Sign Lang", score: 88, reps: 12, duration: "5:00", time: "Yesterday", trend: "up" }, { id: "5", skill: "Salsa Basic Step", category: "Dance", score: 82, reps: 15, duration: "7:00", time: "Yesterday", trend: "same" }, ]; const SKILL_CATEGORIES = [ { name: "PT & Rehab", icon: HeartPulse, color: "text-emerald-400", bg: "bg-emerald-500/10", examples: "Knee, Shoulder, Hip" }, { name: "Yoga", icon: Sparkles, color: "text-purple-400", bg: "bg-purple-500/10", examples: "Warrior, Tree, Sun" }, { name: "Sign Language", icon: Hand, color: "text-yellow-400", bg: "bg-yellow-500/10", examples: "ASL, Greetings" }, { name: "Elderly Care", icon: HeartPulse, color: "text-rose-400", bg: "bg-rose-500/10", examples: "Balance, Sit-Stand" }, { name: "Ergonomics", icon: Monitor, color: "text-slate-400", bg: "bg-slate-500/10", examples: "Posture, Lifting" }, { name: "Dance", icon: Music, color: "text-pink-400", bg: "bg-pink-500/10", examples: "Salsa, Ballet" }, { name: "Fitness", icon: Dumbbell, color: "text-blue-400", bg: "bg-blue-500/10", examples: "Squats, Push-ups" }, { name: "Sports", icon: Swords, color: "text-orange-400", bg: "bg-orange-500/10", examples: "Tennis, Golf" }, ]; const CATEGORY_BADGE_COLOR: Record = { "PT Rehab": "border-emerald-500/40 text-emerald-400", Yoga: "border-purple-500/40 text-purple-400", "Sign Lang": "border-yellow-500/40 text-yellow-400", Elderly: "border-rose-500/40 text-rose-400", Ergo: "border-slate-500/40 text-slate-400", Dance: "border-pink-500/40 text-pink-400", Fitness: "border-blue-500/40 text-blue-400", Sports: "border-orange-500/40 text-orange-400", }; export default function Dashboard() { const [greeting, setGreeting] = useState(""); useEffect(() => { const hour = new Date().getHours(); if (hour < 12) setGreeting("Good morning"); else if (hour < 17) setGreeting("Good afternoon"); else setGreeting("Good evening"); }, []); const todayScore = 79; const streak = 7; const skillsLearned = 12; const totalTime = "48 min"; return (
{/* Header */}
{/* Showcase buttons — LEFT of greeting */}

{greeting}

Physical Movement Intelligence — coaching, monitoring, rehab, and safety in real-time.

{/* Stats Row */}

{todayScore}

Avg Score Today

{streak}

Day Streak

[truncated — 12900 more characters] ``` ### frontend/src/app/settings/page.tsx ```typescript "use client"; import { useState } from "react"; import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"; import { Button } from "@/components/ui/button"; import { Badge } from "@/components/ui/badge"; import { Separator } from "@/components/ui/separator"; import { Camera, Mic, Volume2, Wifi, Server, Brain, Database, RefreshCw, CheckCircle, XCircle, Zap, Shield, Download, } from "lucide-react"; interface SystemStatus { camera: boolean; microphone: boolean; backend: boolean; mlModel: boolean; voiceEngine: boolean; } export default function SettingsPage() { const [status, setStatus] = useState({ camera: false, microphone: false, backend: false, mlModel: false, voiceEngine: false, }); const [isTraining, setIsTraining] = useState(false); const [apiUrl, setApiUrl] = useState("http://localhost:8000"); const checkBackend = async () => { try { const res = await fetch(`${apiUrl}/api/state`); setStatus((s) => ({ ...s, backend: res.ok })); } catch { setStatus((s) => ({ ...s, backend: false })); } }; const trainModel = async () => { setIsTraining(true); try { await fetch(`${apiUrl}/api/model/bootstrap?n_per_skill=30&epochs=50`, { method: "POST", }); setStatus((s) => ({ ...s, mlModel: true })); } catch { // training failed } setIsTraining(false); }; const StatusDot = ({ active }: { active: boolean }) => (
); return (

Settings

Configure your Kinetic setup and check system status.

{/* Connection */} Backend Connection
setApiUrl(e.target.value)} className="w-full mt-1 px-3 py-2 bg-secondary/50 border border-border rounded-lg text-sm focus:outline-none focus:ring-2 focus:ring-primary/50" />
{status.backend ? "Connected" : "Not connected"}
{/* System Status */} System Status {[ { label: "Camera Feed", icon: Camera, active: status.camera, desc: "WebSocket /ws/video" }, { label: "Microphone", icon: Mic, active: status.microphone, desc: "WebSocket /ws/audio" }, { label: "Backend Server", icon: Server, active: status.backend, desc: `${apiUrl}` }, { label: "ML Models", icon: Brain, active: status.mlModel, desc: "Skill scorer + Activity classifier" }, { label: "Voice Engine", icon: Volume2, active: status.voiceEngine, desc: "Gemini Live bidirectional" }, ].map((item) => (

{item.label}

{item.desc}

{item.active ? ( ) : ( )}
))} {/* ML Model Training */} ML Model

Bootstrap the skill scoring model with synthetic training data. This trains a PyTorch 1D CNN that scores movement quality in real-time.