Project Info
Inspiration
Robotics companies like Tesla, Figure AI, and Boston Dynamics spend hundreds of thousands of dollars hiring employees to manually record repetitive tasks for robot training data. Inspired by platforms like Poke.com and Scale AI that crowdsource text labeling, we asked: what if we could do the same thing for motion data? We wanted to make training data collection accessible, affordable, and fair — putting money directly in the hands of everyday workers.
What it does
OpenMotion is a crowdsourced marketplace connecting robotics companies with everyday contributors. Our platform has 4 main features: Task posting - Businesses create tasks (e.g., "warehouse picking," "folding laundry"), set prices, and define requirements Real-time pose capture - Contributors record themselves using just their webcam while TensorFlow BlazePose extracts full 3D skeletal data (33 body keypoints) — no special hardware needed AI verification - Gemini Vision API automatically verifies videos match task requirements before submission Data access - Businesses download complete datasets (video + JSON pose data), access a developer API, and analyze patterns using built-in Jupyter notebooks User Incentive - Users are paid when they submit videos of doing tasks Additional features SMS integration via poke.com for accessibility to workers without computers Privacy-first design with client-side face blurring before upload Real-time 3D avatar visualization that mirrors user movements Track earnings and browse available tasks
How we built it
Our tech stack combines modern tools for a robust, scalable solution: Frontend: Next.js 16 with TypeScript, Tailwind CSS, and Framer Motion for smooth animations Backend: Firebase for authentication, Firestore for data storage, and Firebase Storage for video files AI Processing: TensorFlow.js with BlazePose for client-side pose detection and Google Gemini Vision API for video verification 3D Visualization: Three.js with React Three Fiber for the live skeleton avatar that mirrors user movements Developer API: Custom REST endpoints with API key authentication Data Analytics: Built-in Jupyter-style notebook environment for pose data analysis SMS Service: MCP server (FastMCP/Python) powering poke.com text-based platform interaction
Challenges we ran into
Browser Performance: Loading MediaPipe + TensorFlow.js dynamically while maintaining real-time skeleton overlay Canvas Recording: Recording from canvas (not raw video) so face blur and skeleton are baked into the file required canvas.captureStream() synchronization Data Quality: Sanitizing pose frames for NaN/Infinity values before JSON serialization Rate Limits: Redesigning Gemini verification flow to avoid automatic rate limit hits
Accomplishments we're proud of
Full 3D pose capture running entirely in the browser — no server processing, downloads, or special hardware Built a complete end-to-end pipeline from task creation to API access in 36 hours Real-time 3D avatar provides immediate visual feedback while recording Privacy-first with client-side face blurring Made the platform accessible via SMS for workers without computers
What we learned
Browser-based ML is incredibly powerful — TensorFlow.js tracks 33 3D keypoints in real time without lag Designing for two different user types (contributors and businesses) requires careful role-based UI decisions AI verification adds huge quality control value but needs rate limit consideration Accessibility matters — SMS integration opened the platform to excluded users Edge case handling is critical for production-ready applications
What's next
Future enhancements we're planning: 1. Mobile & Advanced Detection Mobile-native recording app with optimized camera controls Multi-person pose detection for collaborative tasks Marketplace matching based on skills and location 2. Payments & Quality Stripe/PayPal integration for real money transfers Automatic quality scoring based on keypoint confidence Advanced analytics with pre-built ML model templates 3. Enterprise Features Dedicated data pipelines and SLAs Priority contributor pools Custom training workflows Our vision is to make OpenMotion the go-to platform for crowdsourced motion data, making robot training more efficient and accessible for everyone. Links Poke.ai recipe link: https://poke.com/r/vq_YGEA8MuC Vercel link: https://treehacks26-flame.vercel.app/
This is a Next.js project bootstrapped with create-next-app.
Getting Started
First, run the development server:
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
Open http://localhost:3000 with your browser to see the result.
You can start editing the page by modifying app/page.tsx. The page auto-updates as you edit the file.
This project uses next/font to automatically optimize and load Geist, a new font family for Vercel.
Learn More
To learn more about Next.js, take a look at the following resources:
- Next.js Documentation - learn about Next.js features and API.
- Learn Next.js - an interactive Next.js tutorial.
You can check out the Next.js GitHub repository - your feedback and contributions are welcome!
Deploy on Vercel
The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js.
Check out our Next.js deployment documentation for more details.
Analysis
View
Metric
- 15
- 13
- 12
- 3
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- FirebaseIn code
- Google GeminiIn code
- Next.jsIn code
- PythonIn code
- ReactIn code
- Tailwind CSSIn code
- TypeScriptIn code
- JavaScriptClaimed
- OpenAIClaimed
- TensorFlowClaimed
- VercelClaimed
8 of 12 appear in the indexed code. 4 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
406 KB
Source files
49
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
CsCats/treehacks26
66 files · 11.5 MB · @ fb7b823
Structure
Interface
28 files · 42%Screens, components and styles rendered to the user.
+10 moreAPI & routing
10 files · 15%Request entry points: routes, handlers and controllers.
Application logic
6 files · 9%Domain rules, services and shared utilities.
Background jobs
3 files · 5%Work run outside a request: tasks, workers and schedules.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- TypeScript92%
- Python4%
- Markdown2%
- CSS2%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
package.json
npm · 23- @google/generative-ai
- @react-three/drei
- @react-three/fiber
- @types/three
- @types/uuid
- firebase
- framer-motion
- next
- react
- react-dom
- react-syntax-highlighter
- replicate
- three
- uuid
- +9 more
poke-mcp/requirements.txt
pypi · 1- fastmcp
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
This project’s features have not been analysed yet.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.