# Project export: OpenMotion
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2026
- Tagline: Crowdsourced Motion Data for Robotics
- Devpost: https://devpost.com/software/openmotion
- GitHub: https://github.com/CsCats/treehacks26
- Video: https://www.youtube.com/embed/rr9VkDqwMuE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 4 GitHub contributor(s) — Grace-Shao (15 commits), Alex (13 commits), Aishwarya Chakravarthy (12 commits), Darryl Tanzil (3 commits)
## Devpost submission (written by the team)
### Inspiration
Robotics companies like Tesla, Figure AI, and Boston Dynamics spend hundreds of thousands of dollars hiring employees to manually record repetitive tasks for robot training data. Inspired by platforms like Poke.com and Scale AI that crowdsource text labeling, we asked: what if we could do the same thing for motion data? We wanted to make training data collection accessible, affordable, and fair — putting money directly in the hands of everyday workers.
### What it does
OpenMotion is a crowdsourced marketplace connecting robotics companies with everyday contributors. Our platform has 4 main features: Task posting - Businesses create tasks (e.g., "warehouse picking," "folding laundry"), set prices, and define requirements Real-time pose capture - Contributors record themselves using just their webcam while TensorFlow BlazePose extracts full 3D skeletal data (33 body keypoints) — no special hardware needed AI verification - Gemini Vision API automatically verifies videos match task requirements before submission Data access - Businesses download complete datasets (video + JSON pose data), access a developer API, and analyze patterns using built-in Jupyter notebooks User Incentive - Users are paid when they submit videos of doing tasks Additional features SMS integration via poke.com for accessibility to workers without computers Privacy-first design with client-side face blurring before upload Real-time 3D avatar visualization that mirrors user movements Track earnings and browse available tasks
### How we built it
Our tech stack combines modern tools for a robust, scalable solution: Frontend: Next.js 16 with TypeScript, Tailwind CSS, and Framer Motion for smooth animations Backend: Firebase for authentication, Firestore for data storage, and Firebase Storage for video files AI Processing: TensorFlow.js with BlazePose for client-side pose detection and Google Gemini Vision API for video verification 3D Visualization: Three.js with React Three Fiber for the live skeleton avatar that mirrors user movements Developer API: Custom REST endpoints with API key authentication Data Analytics: Built-in Jupyter-style notebook environment for pose data analysis SMS Service: MCP server (FastMCP/Python) powering poke.com text-based platform interaction
### Challenges we ran into
Browser Performance: Loading MediaPipe + TensorFlow.js dynamically while maintaining real-time skeleton overlay Canvas Recording: Recording from canvas (not raw video) so face blur and skeleton are baked into the file required canvas.captureStream() synchronization Data Quality: Sanitizing pose frames for NaN/Infinity values before JSON serialization Rate Limits: Redesigning Gemini verification flow to avoid automatic rate limit hits
### Accomplishments we're proud of
Full 3D pose capture running entirely in the browser — no server processing, downloads, or special hardware Built a complete end-to-end pipeline from task creation to API access in 36 hours Real-time 3D avatar provides immediate visual feedback while recording Privacy-first with client-side face blurring Made the platform accessible via SMS for workers without computers
### What we learned
Browser-based ML is incredibly powerful — TensorFlow.js tracks 33 3D keypoints in real time without lag Designing for two different user types (contributors and businesses) requires careful role-based UI decisions AI verification adds huge quality control value but needs rate limit consideration Accessibility matters — SMS integration opened the platform to excluded users Edge case handling is critical for production-ready applications
### What's next
Future enhancements we're planning: 1. Mobile & Advanced Detection Mobile-native recording app with optimized camera controls Multi-person pose detection for collaborative tasks Marketplace matching based on skills and location 2. Payments & Quality Stripe/PayPal integration for real money transfers Automatic quality scoring based on keypoint confidence Advanced analytics with pre-built ML model templates 3. Enterprise Features Dedicated data pipelines and SLAs Priority contributor pools Custom training workflows Our vision is to make OpenMotion the go-to platform for crowdsourced motion data, making robot training more efficient and accessible for everyone. Links Poke.ai recipe link: https://poke.com/r/vq_YGEA8MuC Vercel link: https://treehacks26-flame.vercel.app/
## README (from the GitHub repository)
This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app).
## Getting Started
First, run the development server:
```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.
This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel.
## Learn More
To learn more about Next.js, take a look at the following resources:
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome!
## Deploy on Vercel
The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details.
## Detected evidence (automated analysis)
Indexed codebase: 49 recognized source files, 406 KB.
- CSS (language) — detected in the code
- Firebase (technology) — detected in the code
- Google Gemini (technology) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- JavaScript (language) — claimed on Devpost, not found in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
- TensorFlow (technology) — claimed on Devpost, not found in the code
- Vercel (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (59 of 59)
```
.gitignore
.vercelignore
app/analytics/page.tsx
app/api/billing/route.ts
app/api/developer/[apiKey]/route.ts
app/api/developer/route.ts
app/api/follows/route.ts
app/api/generate-angles/route.ts
app/api/profile/route.ts
app/api/stats/route.ts
app/api/submissions/route.ts
app/api/tasks/fix-description/route.ts
app/api/tasks/review-description/route.ts
app/api/tasks/route.ts
app/api/user-by-email/route.ts
app/api/verify-task/route.ts
app/billing/page.tsx
app/business/page.tsx
app/businessView/page.tsx
app/contributions/page.tsx
app/developer/page.tsx
app/earnings/page.tsx
app/gallery/page.tsx
app/globals.css
app/layout.tsx
app/login/page.tsx
app/notebook/page.tsx
app/page.tsx
app/profile/page.tsx
app/signup/page.tsx
app/stats/page.tsx
app/userUpload/page.tsx
components/ConstellationBackground.tsx
components/ContributionGraph.tsx
components/GlobalThemeToggle.tsx
components/MainContent.tsx
components/NotebookViewer.tsx
components/PoseAvatarViewer.tsx
components/PoseSkeletonViewer.tsx
components/Sidebar.tsx
components/ThemeScript.tsx
components/ThemeToggle.tsx
components/UserUploadClient.tsx
eslint.config.mjs
lib/AuthContext.tsx
lib/firebase.ts
lib/ThemeContext.tsx
next.config.ts
notebooks/.ipynb_checkpoints/training_pipeline_demo-checkpoint.ipynb
notebooks/training_pipeline_demo.ipynb
package.json
poke-mcp/README.md
poke-mcp/requirements.txt
poke-mcp/server.py
postcss.config.mjs
README.md
SETUP.md
tsconfig.json
vercel.json
```
### Dependencies
- package.json: @google/generative-ai@^0.24.1, @react-three/drei@^10.7.7, @react-three/fiber@^9.5.0, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, @types/react-syntax-highlighter@^15.5.13, @types/three@^0.182.0, @types/uuid@^11.0.0, eslint@^9, eslint-config-next@16.1.6, firebase@^12.9.0, framer-motion@^12.34.0, next@16.1.6, react@19.2.3, react-dom@19.2.3, react-syntax-highlighter@^15.6.1, replicate@^1.0.3, tailwindcss@^4, three@^0.182.0, typescript@^5, uuid@^13.0.0
- poke-mcp/requirements.txt: fastmcp@>=2.0.0
### Recent commits (newest first)
- Merge branch 'main' of github.com:Grace-Shao/treehacks26
- Added busniess fix feature
- change to openMotion
- Merge branch 'main' of github.com:Grace-Shao/treehacks26
- update ui
- Fixing deploy issue
- Merge pull request #6 from CsCats/grace3
- fix gemini rate limiting
- fix UI
- Update training pipeline notebook with new random seed
- Merge branch 'main' of github.com:Grace-Shao/treehacks26
- adding Poke AI
- Merge pull request #5 from CsCats/grace2
- dark mode DONE
- Merge remote-tracking branch 'refs/remotes/origin/main'
- adding sample training pileine
- Merge pull request #4 from CsCats/grace2
- Merge branch 'main' into grace2
- no approve button for ai and fix darkmode in profile
- Add some jupyter stuff
## Key source files (fetched from GitHub, selected and truncated for size)
### SETUP.md
```markdown
# Setup Instructions
## Firebase Configuration
1. Create a Firebase project at [https://console.firebase.google.com](https://console.firebase.google.com)
2. Enable **Firestore Database** (start in test mode)
3. Enable **Firebase Storage** (start in test mode)
4. Go to Project Settings > General > Your apps > Add a Web app
5. Copy the config values and create a `.env.local` file in the project root:
```
NEXT_PUBLIC_FIREBASE_API_KEY=your_api_key
NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN=your_project.firebaseapp.com
NEXT_PUBLIC_FIREBASE_PROJECT_ID=your_project_id
NEXT_PUBLIC_FIREBASE_STORAGE_BUCKET=your_project.appspot.com
NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID=your_sender_id
NEXT_PUBLIC_FIREBASE_APP_ID=your_app_id
```
6. Run `npm run dev` to start the development server.
## Firestore Collections (auto-created)
- **tasks** — Business prompts/requirements
- **submissions** — User video uploads with pose data
```
### package.json
```
{
"name": "treehacks26",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"lint": "eslint",
"mcp": "cd poke-mcp && .venv/bin/python server.py"
},
"dependencies": {
"@google/generative-ai": "^0.24.1",
"@react-three/drei": "^10.7.7",
"@react-three/fiber": "^9.5.0",
"@types/three": "^0.182.0",
"@types/uuid": "^11.0.0",
"firebase": "^12.9.0",
"framer-motion": "^12.34.0",
"next": "16.1.6",
"react": "19.2.3",
"react-dom": "19.2.3",
"react-syntax-highlighter": "^15.6.1",
"replicate": "^1.0.3",
"three": "^0.182.0",
"uuid": "^13.0.0"
},
"devDependencies": {
"@tailwindcss/postcss": "^4",
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"@types/react-syntax-highlighter": "^15.5.13",
"eslint": "^9",
"eslint-config-next": "16.1.6",
"tailwindcss": "^4",
"typescript": "^5"
}
}
```
### poke-mcp/requirements.txt
```
fastmcp>=2.0.0
```
### app/layout.tsx
```typescript
import type { Metadata } from "next";
import { Geist, Geist_Mono } from "next/font/google";
import "./globals.css";
import Sidebar from "@/components/Sidebar";
import MainContent from "@/components/MainContent";
import ThemeScript from "@/components/ThemeScript";
import { AuthProvider } from "@/lib/AuthContext";
import { ThemeProvider } from "@/lib/ThemeContext";
const geistSans = Geist({
variable: "--font-geist-sans",
subsets: ["latin"],
});
const geistMono = Geist_Mono({
variable: "--font-geist-mono",
subsets: ["latin"],
});
export const metadata: Metadata = {
title: "OpenMotion - Crowdsourced Robotics Training",
description: "Crowdsourced platform for collecting human motion training data for robotics companies",
};
export default function RootLayout({
children,
}: Readonly<{
children: React.ReactNode;
}>) {
return (
{children}
);
}
```
### poke-mcp/server.py
```python
import subprocess
import json
import os
from pathlib import Path
from urllib.request import urlopen, Request
from urllib.error import URLError, HTTPError
from urllib.parse import quote
from fastmcp import FastMCP
# Project root directory
PROJECT_ROOT = "/Users/achakrav6/Desktop/treehacks26"
mcp = FastMCP("Robotics Motion Training Platform")
@mcp.tool()
def run_dev_server() -> str:
"""Start the Next.js development server on localhost:3000."""
try:
result = subprocess.run(
["npm", "run", "dev"],
capture_output=True,
text=True,
cwd=PROJECT_ROOT,
timeout=5
)
return result.stdout + result.stderr
except subprocess.TimeoutExpired:
return "Dev server is starting in the background. Visit http://localhost:3000"
except Exception as e:
return f"Error starting dev server: {str(e)}"
@mcp.tool()
def build_project() -> str:
"""Build the Next.js project for production."""
try:
result = subprocess.run(
["npm", "run", "build"],
capture_output=True,
text=True,
cwd=PROJECT_ROOT,
timeout=300
)
if result.returncode == 0:
return f"Build successful!\n\n{result.stdout}"
else:
return f"Build failed with errors:\n\n{result.stderr}"
except Exception as e:
return f"Error building project: {str(e)}"
@mcp.tool()
def run_linter() -> str:
"""Run ESLint to check code quality and identify issues."""
try:
result = subprocess.run(
["npm", "run", "lint"],
capture_output=True,
text=True,
cwd=PROJECT_ROOT,
timeout=60
)
return result.stdout + result.stderr
except Exception as e:
return f"Error running linter: {str(e)}"
@mcp.tool()
def list_api_endpoints() -> str:
"""List all API endpoints available in the application."""
try:
api_dir = Path(PROJECT_ROOT) / "app" / "api"
endpoints = []
for route_file in api_dir.rglob("route.ts"):
# Convert path to API endpoint format
rel_path = route_file.relative_to(api_dir)
endpoint = "/" + str(rel_path.parent).replace("\\", "/")
# Read file to detect HTTP methods
with open(route_file, 'r') as f:
content = f.read()
methods = []
if "export async function GET" in content:
methods.append("GET")
if "export async function POST" in content:
methods.append("POST")
if "export async function PATCH" in content:
methods.append("PATCH")
if "export async function DELETE" in content:
methods.append("DELETE")
if "export async function PUT" in content:
methods.append("PUT")
endpoints.append({
"endpoint": f"/api{endpoint}",
"methods": methods,
"file": str(route_file.relative_to(PROJECT_ROOT))
})
return json.dumps(endpoints, indent=2)
except Exception as e:
return f"Error listing API endpoints: {str(e)}"
@mcp.tool()
def get_project_structure() -> str:
"""Get an overview of the project directory structure."""
try:
result = subprocess.run(
["find", ".", "-type", "f", "-name", "*.tsx", "-o", "-name", "*.ts", "-o", "-name", "*.json"],
capture_output=True,
text=True,
cwd=PROJECT_ROOT,
timeout=30
)
lines = result.stdout.strip().split('\n')
# Filter out node_modules and .git
filtered = [l for l in lines if 'node_modules' not in l and '.git' not in l and l.strip()]
return '\n'.join(sorted(filtered)[:100]) # Limit to first 100 files
except Exception as e:
return f"Error getting project structure: {str(e)}"
@mcp.tool()
def get_package_info() -> str:
"""Get package.json information including dependencies and scripts."""
try:
package_path = Path(PROJECT_ROOT) / "package.json"
with open(package_path, 'r') as f:
data = json.load(f)
info = {
"name": data.get("name"),
"version": data.get("version"),
"scripts": data.get("scripts", {}),
"dependencies": list(data.get("dependencies", {}).keys()),
"devDependencies": list(data.get("devDependencies", {}).keys())
}
return json.dumps(info, indent=2)
except Exception as e:
return f"Error reading package.json: {str(e)}"
@mcp.tool()
def check_firebase_config() -> str:
"""Check if Firebase configuration files exist and are properly set up."""
try:
config_files = [
"lib/firebase.ts",
"lib/firebase.js",
".env.local",
".env"
]
results = {}
for config_file in config_files:
file_path = Path(PROJECT_ROOT) / config_file
results[config_file] = {
"exists": file_path.exists(),
"path": str(file_path) if file_path.exists() else "Not found"
}
return json.dumps(results, indent=2)
except Exception as e:
return f"Error checking Firebase config: {str(e)}"
@mcp.tool()
def list_pages() -> str:
"""List all pages/routes in the Next.js application."""
try:
app_dir = Path(PROJECT_ROOT) / "app"
pages = []
for page_file in app_dir.rglob("page.tsx"):
rel_path = page_file.relative_to(app_dir)
route = "/" + str(rel_path.parent).replace("\\", "/")
if route == "/.":
route = "/"
# Try to extract page description from comments or title
with open(page_file, 'r') as f:
content = f.read()
# Look
[truncated — 12080 more characters]
```
### app/userUpload/page.tsx
```typescript
'use client';
import dynamic from 'next/dynamic';
const UserUploadClient = dynamic(() => import('@/components/UserUploadClient'), {
ssr: false,
loading: () => (