Project Info

Kinetic AI

Inspiration

AI helps us write, code, and reason — but human physical capability remains unaugmented. Quality coaching costs $80–200/hour, PT is $100+/hour, and 55 million elderly Americans need fall monitoring that costs hospitals billions in patient sitter programs annually. These aren't separate problems. They're all the same gap: no cognitive layer for physical capability. We asked: what if one camera and one AI could coach a squat, detect a grandmother falling, guide PT rehab, and monitor a hospital room overnight? What It Does Kinetic is a real-time Physical Movement Intelligence platform with four modes from one stack: AI Skill Coach — Describe any movement and Kinetic generates the ideal form via a 4-tier pipeline (templates → Claude mapping → Claude angle generation → HY-Motion diffusion on A100 GPU). It tracks your body at 30 FPS, scores form with a triple-metric engine, and coaches via real-time voice. Physical Therapy — PT-specific categories (knee, shoulder, hip, ankle rehab). Tracks range of motion, enforces safety limits, auto-counts reps via phase detection, and adapts voice coaching to gentle rehab context. Goal-Based Intelligence — Give Kinetic any goal and step away. Presets include fall detection, desk security, posture watch, driver alertness, and study focus. It runs a continuous perception→reasoning→action loop, sending Telegram alerts with photos autonomously. Clinical Patient Safety — Hospital-grade monitoring: fall detection, bed exit alerts, immobility/pressure ulcer prevention, IV line safety, post-op distress, and wandering/elopement detection. Alerts nurses via Telegram with snapshot photos instantly. 46 MCP tools are exposed via HTTP so external AI agents (including Poke) can drive the entire stack programmatically. How We Built It DGX Spark — YOLOv8n-pose on the GB10 Superchip for sub-50ms edge pose estimation at 15+ FPS. (17 key points) Modal A100 — Tencent's HY-Motion 1.0-Lite (0.46B param text-to-3D motion diffusion, Dec 2025) generates 30-frame skeleton sequences from text prompts in ~26s. Three model weights (~20GB) cached in Modal Volumes. https://rajashekarvennavelli--aegis-motion-generate-endpoint.modal.run` Claude Agent SDK — 46 MCP tools across 12 categories, 3 sub-agents (perception, coach, communicator), safety hooks, and audit logging. Claude reasons about movement, coaching, and spatial safety across all modes. OpenAI Realtime Voice — Bidirectional audio with interruption handling and context injection every 5 seconds. Voice adapts per mode: gym coach, PT encourager, or urgent fall alerter. Triple-Metric Scoring — Gaussian joint angles (0.4) + cosine spatial similarity (0.3) + COCO OKS (0.3) for scoring that catches errors single metrics miss. Autonomous Monitoring — Background loop every 3s detects falls, bed exits, immobility, and wandering. Sends Telegram photo alerts, voice alerts, and Claude analysis simultaneously. Controllable via Telegram commands. Frontend — Next.js + Tailwind + shadcn/ui. Four mode cards, coaching UI with 10 skill categories, monitoring dashboard with live camera + alert timeline + agent tool call visualization. Challenges We Ran Into HY-Motion on Modal — 20GB of model weights, relative path issues, dozens of build iterations, and mapping 52-joint output to MediaPipe 33-point format. The hardest single technical challenge. https://rajashekarvennavelli--aegis-motion-generate-endpoint.modal.run Real-time scoring at 30 FPS — DTW temporal alignment, phase detection, and triple-metric computation all under 5ms per frame via pre-normalization and caching. Voice + vision sync — Three-way latency (audio 200ms, video 50ms, Claude 1-2s) solved with context injection architecture — voice always references current form, not stale data. Clinical reliability — Fall detection can't have false negatives. Temporal smoothing, priority boosting for safety-critical activities, and hysteresis prevent state flickering. Accomplishments That We're Proud Of Four modes, one stack — coaching, PT, autonomous monitoring, and clinical safety all share the same CV + agent + voice pipeline Working text-to-3D motion — HY-Motion generating real physics-aware skeleton sequences, animated live on the frontend Autonomous fall → Telegram pipeline — zero human intervention from detection to nurse alert with photo 46 MCP tools as HTTP — the entire physical intelligence stack is callable by any external AI agent What We Learned Physical intelligence is AI's next frontier — coaching, rehab, monitoring, and safety are all the same capability: perceiving movement and reasoning about what should change. Edge AI changes everything — sub-50ms pose estimation makes coaching feel human and fall detection truly real-time. Voice is the only viable physical interface — you can't touch a screen mid-pushup. Realtime bidirectional voice makes AI coaching natural. Modularity enables pivoting — separate subsystems let us add PT, monitoring, and clinical modes without breaking existing functionality. What's Next for Kinetic AR glasses — overlay the expert skeleton directly in the user's field of view for truly hands-free coaching Multi-camera fusion — stereo depth for accurate 3D joint angles, eliminating single-camera occlusion blind spots Clinical integration — FHIR/EHR export of ROM data, rep counts, and progress for clinician dashboards Hospital deployment — ceiling-mounted cameras augmenting clinical staff, enabling one nurse to safely monitor multiple rooms simultaneously Personalized models — fine-tuned to individual body proportions and injury history for adaptive coaching thresholds

Kinetic — Real-Time Physical Movement Intelligence

"One camera. One AI. Coaches your squat, detects a fall, guides PT rehab, monitors a hospital room."

Kinetic is a unified platform for physical movement intelligence — AI skill coaching, physical therapy rehab, autonomous space monitoring, and clinical patient safety — all from the same CV + agent + voice stack.

4 modes. 46 MCP tools. 10 ML models. 17,000+ lines. Built solo in 20 hours at TreeHacks 2026.

Demo

🎥 Loom Demo Video 🌐 Live Frontend

Infrastructure:

🖥️ DGX Spark (Edge): Real-time pose, fall detection, monitoring — all on-device, zero cloud dependency

⚡ Why both edge and cloud? DGX Spark's Blackwell GPU doesn't have PyTorch/CUDA wheels for ARM yet — so safety-critical inference (pose, falls) runs on-device at sub-50ms, while one-time heavy generation (0.46B params) offloads to A100. Result is cached locally; after that, coaching is 100% edge.

🚀 Modal A100 (Cloud): HY-Motion 1.0-Lite text-to-3D motion generation — Endpoint: POST https://rajashekarvennavelli--aegis-motion-generate-endpoint.modal.run
🤖 46 MCP Tools: Full stack exposed via HTTP for external AI agents (Poke)

4 Modes, One Stack

┌─────────────────────────────────────────────────────────┐
│                    KINETIC PLATFORM                      │
├──────────────┬──────────────┬──────────┬────────────────┤
│  🏋️ Coaching  │  🩺 PT Rehab  │ 🎯 Goals │  🏥 Clinical   │
│              │              │          │                │
│ Any skill    │ Knee/shoulder│ Fall     │ Fall detection │
│ Voice coach  │ Safe ROM     │ Posture  │ Bed exit alert │
│ AI expert    │ Rep counting │ Security │ Immobility     │
│ Scoring      │ Voice rehab  │ Focus    │ Wandering      │
├──────────────┴──────────────┴──────────┴────────────────┤
│  CV Pipeline: YOLO + MediaPipe + ByteTrack + Depth      │
│  Agent: Claude SDK (3 sub-agents, 46 MCP tools, hooks)  │
│  Voice: OpenAI Realtime (bidirectional, interruption)   │
│  Edge AI: DGX Spark + Modal A100 (HY-Motion 1.0)       │
│  Alerts: Telegram + Voice + Frontend dashboard          │
└─────────────────────────────────────────────────────────┘

6 Infrastructure Pillars

1. 🟢 NVIDIA DGX Spark — Edge AI Inference

GB10 Superchip (Grace ARM CPU + Blackwell GPU)
YOLOv8n-pose: 17-keypoint real-time pose estimation
Runs on-premise with low latency — no cloud roundtrip for pose
Endpoints: POST /predict, GET /health

2. 🔵 Modal + NVIDIA A100 — Cloud GPU for Motion Generation

HY-Motion 1.0-Lite (Tencent, SOTA Dec 2025)
0.46B parameters, DiT + Flow Matching architecture
Trained on 3,000+ hours of 3D motion data (pretrain → finetune → RLHF)
Text prompt → SMPL 22-joint 3D skeleton → MediaPipe 33-point conversion
Endpoint: POST https://rajashekarvennavelli--aegis-motion-generate-endpoint.modal.run
Serverless A100, scales to zero, $530 credits available

3. 🟠 Anthropic Claude Agent SDK — AI Orchestration

Claude Sonnet 4 as main orchestrator
3 Sub-Agents: Perception (11 tools), Coach (14 tools), Progress (10 tools)
44 MCP Tools via Model Context Protocol
3 Agent Hooks: Safety guard, audit log, session summary
Handles: coaching decisions, expert generation, form analysis, goal tracking

4. 🎙️ OpenAI Realtime API — Voice Coaching

GPT-4o Realtime Preview with alloy voice
3-Layer Interruption System:
- Layer 1: Server-side VAD (50ms speech detection)
- Layer 2: Response state machine (prevents audio overlap)
- Layer 3: Single voice source (proactive coaching + reactive Q&A)
Punchy prompts: max 15 words, no filler
Browser speechSynthesis TTS fallback for offline use

5. 👁️ Computer Vision Pipeline

Model	Purpose	Size	Speed
YOLO11n	Person detection	5.4 MB	15 FPS
MediaPipe Pose	33 body landmarks	5.6 MB	30 FPS
MediaPipe Hands	21 hand landmarks/hand	~5 MB	30 FPS
Depth Anything V2	Monocular depth	~50 MB	22 FPS
ByteTrack	Multi-person tracking	—	15 FPS

6. 📊 Triple-Metric Pose Scoring

Metric	Weight	What it measures
Gaussian Angle Scoring (16 joints)	50%	Per-joint angular accuracy with tuned σ
Cosine Spatial Similarity	30%	Overall pose shape matching
COCO OKS	20%	Industry-standard keypoint similarity

Plus: DTW temporal alignment, phase detection, rep counting, compensation detection.

AI Expert Generation — No Video Required

Kinetic can coach any skill without a reference video:

Tier	Method	Latency	How
1	Semantic Alias Lookup	0ms	53 aliases → 10 canonical exercises
2	Claude Semantic Mapping	~0.5s	Claude maps novel names to known exercises
3	Claude Angle Generation	~1-2s	Claude generates biomechanically correct angles
4	HY-Motion 3D Generation	~5-15s	Full 3D motion on Modal A100 GPU

10 built-in exercises: squat, pushup, lunge, deadlift, shoulder press, bicep curl, plank, jumping jack, warrior II, tree pose

Complete Model Zoo

Model	Company	Purpose	Location
YOLOv8n-pose	Ultralytics	17-keypoint pose	DGX Spark
YOLO11n	Ultralytics	Person detection	Local
MediaPipe Pose Lite	Google	33 body landmarks	Local
MediaPipe Hands	Google	21 hand landmarks	Local
Depth Anything V2	ByteDance	Monocular depth	Local (MPS)
ByteTrack	ByteDance	Multi-person tracking	Local
HY-Motion 1.0-Lite	Tencent	Text → 3D motion (SOTA)	Modal A100
Claude Sonnet 4	Anthropic	Agent orchestration	Cloud API
GPT-4o Realtime	OpenAI	Voice coaching	Cloud API
Custom 1D CNN	Built in-house	Pose scoring (14K params)	Local (NumPy)

Tech Stack

Layer	Technology
Frontend	Next.js 14 + React + TailwindCSS + shadcn/ui
Backend	FastAPI + Python 3.12 (44 routes, 3 WebSockets)
AI Orchestration	Claude Agent SDK (3 sub-agents, 44 MCP tools, 3 hooks)
Voice AI	OpenAI Realtime API (GPT-4o) + browser TTS fallback
Edge AI	NVIDIA DGX Spark (GB10 Superchip, YOLOv8n-pose)
Cloud GPU	Modal + NVIDIA A100 (HY-Motion 1.0-Lite, 0.46B params)
Computer Vision	YOLO11n + MediaPipe Pose/Hands + ByteTrack + Depth Anything V2
Pose Scoring	Gaussian angles + Cosine spatial + COCO OKS
Protocol	MCP (Model Context Protocol) — 44 tools

Quick Start

Prerequisites

Python 3.12+
Node.js 18+
API Keys: ANTHROPIC_API_KEY, OPENAI_API_KEY

Backend

cd aegis
pip install -r requirements.txt
python run_server.py
# Server starts at http://localhost:8000

Frontend

cd frontend
npm install
npm run dev
# UI at http://localhost:3000

Test Motion Generation (Modal A100)

curl -X POST https://rajashekarvennavelli--aegis-motion-generate-endpoint.modal.run \
  -H "Content-Type: application/json" \
  -d '{"prompt": "a person doing a squat", "num_frames": 60}'

DGX Spark Edge Inference

# On DGX Spark (gx10-eb94)
cd dgx
pip install -r requirements.txt
python inference_server.py
# Pose endpoint at http://<dgx-ip>:8080/predict

Project Structure

kinetic/
├── aegis/                    # Backend (internal codename)
│   ├── server.py             # FastAPI server (44 routes, 3 WebSockets)
│   ├── spatial_engine.py     # CV pipeline (YOLO + MediaPipe + ByteTrack + Depth)
│   ├── pose_comparison.py    # Triple-metric scoring (Gaussian + Cosine + OKS)
│   ├── ai_expert.py          # AI expert generation (aliases + Claude + HY-Motion)
│   ├── sdk_agent.py          # Claude Agent SDK (3 sub-agents, hooks)
│   ├── mcp_server.py         # 44 MCP tools (fastmcp)
│   ├── openai_voice.py       # OpenAI Realtime voice (3-layer interruption)
│   ├── gemini_bridge.py      # Gemini Live fallback voice
│   ├── skill_graph.py        # Skill DAG + PageRank recommendations
│   ├── data_collector.py     # JSONL session storage
│   └── run_server.py         # Orchestrator
├── dgx/                      # NVIDIA DGX Spark deployment
│   ├── inference_server.py   # Edge AI: YOLOv8n-pose + health endpoints
│   └── modal_motion.py       # Modal A100: HY-Motion 1.0-Lite deployment
├── frontend/                 # Next.js 14 UI
│   └── app/coach/page.tsx    # Main coaching interface
└── README.md

Codebase: ~17,000+ lines across 20+ modules

Built at TreeHacks 2026 🌲

Powered by:

Anthropic Claude Sonnet 4 + Agent SDK (3 sub-agents, hooks, MCP)
OpenAI GPT-4o Realtime API (bidirectional voice coaching)
NVIDIA DGX Spark GB10 Superchip (edge pose estimation)
Modal A100 GPU Cloud (HY-Motion 1.0-Lite, text-to-3D motion)
Google MediaPipe Pose + Hands (33 body + 21 hand landmarks)
Poke MCP integration (46 tools exposed via HTTP/SSE)

Analysis

Compare with all teams

View

Metric

rajashekarcs2023
73

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

AnthropicIn code
CSSIn code
FastAPIIn code
FlaskIn code
HTMLIn code
Hugging FaceIn code
JavaScriptIn code
Next.jsIn code
OpenAIIn code
PythonIn code
PyTorchIn code
ReactIn code
Tailwind CSSIn code
TypeScriptIn code

14 of 14 appear in the indexed code.

AI coding agents

Claude CodeConfig

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

1.0 MB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.