# Project export: Robo Jab This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Phone video in, fighting humanoid robot out. We made a markerless mocap, sim-trained balance, deployed straight to a real 29-DoF Unitree G1. - Devpost: https://devpost.com/software/robo-jab - GitHub: https://github.com/abtonmoy/ai_hackathon_calhacks - Video: https://www.youtube.com/embed/XCrEp2mOd_s?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Best Physical AI Hack by UFB) - Team: 3 GitHub contributor(s) — Abdul Basit Tonmoy (25 commits), clemencecoco (1 commits), robearista-cpu (1 commits) ## Devpost submission (written by the team) ### Inspiration We wanted to teach a Unitree G1 humanoid to throw a boxing jab without writing a single keyframe by hand. Motion capture suits are expensive and most reference motions for humanoids come from clean lab data, which does not look like how a real person actually moves. Our bet was simple: a phone camera and one sweaty person in leopard tights should be enough to drive a 29 degree of freedom robot. If we could go from a normal video to a trained policy, anyone could turn human movement into robot behavior. ### What it does The project is a full pipeline that takes ordinary monocular phone videos of a person jabbing and produces robot reference motions plus a trained control policy for the Unitree G1. The flow is: Record short clips of a jab on a phone. Run markerless motion capture to recover the 3D human body per frame. Retarget that human motion onto the G1 skeleton (29 joints). Export each clip as a reference CSV (root position, root rotation, joint angles). Validate every CSV, then render the robot and an overlay so we can eyeball quality. Train a multilayer perceptron (MLP) tracking policy on the reference motions and export it as an ONNX policy. The output is a batch of clean, validated jab references and a policy that makes the robot reproduce the motion. ### How we built it The core chain is GVHMR for markerless capture, GMR for retargeting, and an Isaac style tracking trainer for the policy. GVHMR turns each video into world grounded SMPL-X parameters. It runs YOLO for person detection, ViTPose for 2D keypoints, HMR2 for body shape, and the GVHMR network for world grounded recovery. GMR takes the SMPL-X motion and solves inverse kinematics to fit the G1, giving us joint angles per frame. A small exporter converts the retargeted pickle into the headerless CSV format the trainer expects, and a validator checks units, NaNs, foot contact, and frame count. We wrote a headless MuJoCo renderer that draws the robot only, with a camera that re-centers every frame, so the output video shows the motion clearly instead of the robot drifting off screen. Everything is wrapped in an idempotent batch script that processes around 122 clips, skips work that is already done, and stages every artifact (CSV, pickle, overlay video, robot video, side by side) into one output folder. Training ran on rented GPUs, producing checkpoints, a policy ONNX, and verification videos. All of this ran on a Windows laptop through WSL2 with a 8 GB RTX 4060, which forced us to be careful about memory the whole way through. ### Challenges we ran into 8 GB of VRAM broke the obvious plan.** The naive approach was to load GVHMR, YOLO, ViTPose, and HMR2 and run a clip straight through. Four models plus SMPL-X parameters does not fit on a 4060. We rewrote the flow to load one model at a time, run it to completion, dump its output to disk, free the GPU, and only then load the next stage. That turned a single function call into a staged pipeline with intermediate files between every step. It doubled the disk traffic, but it was the only way the clips ran at all. Numbers passed validation and the motion was still wrong. Our validator checks units, NaNs, foot contact, and frame count, and a clip can clear all four while the robot does something that looks nothing like a jab. A retarget that puts the wrist in the right place with the elbow folded backward is valid by every metric we wrote and useless on the robot. We only caught these by watching the render, so we stopped trusting the CSV and started trusting the video. The MuJoCo render itself fought us. The robot's root translates across the floor during a jab, so a fixed camera lets it walk out of frame within a second and the clip is unwatchable. We wrote a camera that recomputes its target from the robot's root every frame so the body stays centered while the motion still reads. Getting that re-centering to track the body without also cancelling the motion we wanted to see took more iterations than the capture code did. The overlay never lined up on the first try. GVHMR recovers a generic SMPL-X body, and that body is not the specific person in leopard tights, so projecting the recovered mesh back onto the original footage drifts at the shoulders and hips. The capture is correct in 3D and still looks wrong composited on 2D video, which sent us chasing a bug that was not a bug before we added an explicit alignment step for the overlay. Running the batch dozens of times surfaced its own problem. With around 122 clips and a laptop that needed sleep, any crash on clip 80 used to mean restarting from clip 1. We made the batch idempotent so it skips any clip whose artifacts already exist and stages every output, the CSV, pickle, overlay, robot render, and side by side, into one folder per clip. That script was not clever, and it saved us more time than anything clever did. Training had to leave the laptop, and finding a GPU to rent took some hunting. Nebius set us up with free credits, which was a real help going in. The catch was timing: the whole hackathon was reaching for GPUs at once, so the instances with enough VRAM to hold the policy and the reference batch kept coming back as unavailable whenever we tried to grab one. Rather than wait it out against the deadline, we moved the training setup over to RunPod and got a machine there. By the time checkpoints, the ONNX policy, and the verification videos came back, lining up the compute had been as much work as writing the code that ran on it. ### What we learned Validation catches format errors, not semantic ones. We added visual checks (overlay, global view, robot render side by side) because numbers alone lied to us. Off the shelf monocular capture is good but not perfect. The 3D body shape never fits one specific person exactly, so the overlay needs an alignment step if you want it to look right on top of the original footage. Small hardware changes the engineering. Most of our design decisions, from static camera mode to one model at a time scheduling, came from living inside 8 GB of VRAM. A boring, idempotent batch pipeline that puts everything in one folder is worth more than any single clever script, especially when you are re running it dozens of times under a deadline. What is next More motion types beyond the jab, better automatic quality scoring so bad clips get filtered without a human looking, and pushing the trained policy onto real hardware. The longer-term goal is medical: adapting the pipeline for surgery by focusing on precise wrist sensing and signaling, so a surgeon's fine hand motion can be captured from video and faithfully mimicked by a robot. ## README (from the GitHub repository) # ROBO JAB ## Human jab → Unitree G1 jab policy AI Hackathon 2026 UC Berkeley / Ultimate Bots Physical-AI hack. The pipeline takes a phone video of a person throwing a jab and produces a trained, deployable motion-tracking policy for a real 29-DoF Unitree G1. Capture is markerless, training runs in sim, and the output is an ONNX policy the robot can run. ![Pipeline: phone video → markerless mocap → G1 reference → trained policy](assets/pipeline_filmstrip.gif) _Reading the strip left → right: the raw phone video of a human jab; the markerless mocap (GVHMR) that extracts the 3D motion; that motion retargeted onto the G1, which becomes the CSV the model trains on; and the trained G1 policy executing the jab in sim. One of about 120 clips in the dataset._ ## Architecture Three independent stages, capture → train → deploy, connected by one portable artifact: a CSV of G1 joint angles. Each stage runs on different hardware and is swappable on its own. ``` CAPTURE (laptop GPU / WSL) ──────────────────────────────────────────────────────────────────────── phone video │ YOLOv8 → ViTPose-H → HMR2.0 → GVHMR (world-grounded SMPL, run -s) │ detect 17 2D kpts 3D mesh markerless mocap ▼ GMR : inverse-kinematics retarget (human SMPL → robot joints) 29-DoF G1 joint angles │ ▼ CSV root_pos[3] + root_rot_xyzw[4] + 29 joints @ 30 fps (headerless) ~120 clips · format-verified against the trainer and the G1 URDF │ TRAIN (RunPod H100) ──────────────────────────────────────────────────────────────────────── csv_to_npz → npz (+ body vel/accel via forward kinematics) │ ~99 clips concatenated → one reference, sampled per episode ▼ unitree_rl_mjlab · Unitree-G1-Tracking-No-State-Estimation (PPO) MuJoCo-Warp, 4096 envs, domain randomization, ~50 Hz obs: joint encoders + IMU → MLP → 29 joint-position targets │ auto-export ▼ policy.onnx (obs → actions) │ DEPLOY (Jetson Orin NX) ──────────────────────────────────────────────────────────────────────── policy.onnx → unitree_sdk2 LowCmd PD (q_des + Kp/Kd) joint-order map → G1JointIndex · sim-to-sim (MuJoCo) gate first ▼ real 29-DoF Unitree G1 throws the jab ``` ### Capture (laptop GPU) A phone video runs through four vision models and then a retargeter. YOLOv8 detects the person, ViTPose-H finds 17 2D keypoints, HMR2.0 (4D-Humans) lifts that to a 3D body mesh, and GVHMR grounds it in world space as SMPL motion (run with `-s`, no SLAM, for static-camera clips). GMR then retargets the human SMPL motion onto the G1 by inverse kinematics, solving which 29 G1 joint angles reproduce the motion within the robot's joint limits. A human body and a robot body differ, so you cannot copy angles directly. The output is one CSV per clip: headerless, `root_pos[3] + root_rot(xyzw)[4] + 29 joints`, at 30 fps. That CSV is the contract between capture and training, format- verified against the trainer and the robot URDF. ### Train (H100) The mocap gives a kinematic reference that is not dynamically feasible. A real G1 holding those exact angles would topple. The fix is a PPO policy that learns to track the reference while staying balanced, under domain randomization, in `unitree_rl_mjlab` (MuJoCo-Warp, GPU-parallel, 4096 envs). `csv_to_npz` adds body velocities and accelerations via forward kinematics. The `Unitree-G1-Tracking-No-State-Estimation` task is deployable by design: the policy observes only what the real robot can measure (joint encoders and IMU), with no privileged sim state. The task tracks one motion at a time, so to cover every jab we concatenate all ~99 clips into one long reference and let the env sample random start points across it. That gives one policy for all jab variations. The policy itself is a small MLP that maps an observation to 29 joint-position targets at about 50 Hz. ### Deploy (Jetson Orin) Training auto-exports `policy.onnx` with an `obs` input and an `actions` output. On the real 29-DoF G1 it runs on the onboard Jetson Orin NX and commands joints via `unitree_sdk2` LowCmd (PD control: target position plus Kp/Kd, about 50 Hz). A deploy config maps the policy's joint order to the SDK's `G1JointIndex` and sets the gains. It is validated in MuJoCo sim-to-sim before hardware. ### Why this design One portable artifact, the CSV, decouples capture from training, so the H100 side never needs the mocap stack and either trainer (`unitree_rl_mjlab` or Isaac/BeyondMimic) reads the same data. Deployability is built in from the start instead of retrofitted: the observation space, joint order, control rate, and ONNX export all match the real robot. Capture stays markerless, so a phone is the only capture hardware, with no mocap suit or marker rig. ## Status | Stage | State | | ---------------------------------------- | ---------------------------------------------------------------- | | Capture (video → GVHMR → GMR → CSV) | done and verified; ~120 clean CSVs in `data/` | | Data ↔ trainer format match | verified against `csv_to_npz` (xyzw, 29-DoF, joint order) | | Data ↔ hardware (29-DoF G1) | confirmed with Ultimate Bots | | Training (RunPod H100, unitree_rl_mjlab) | done; multi-motion ran 10k iters, converged ~0.68 rad | | Deployable artifact (`policy.onnx`) | exported and validated (obs → actions); in `runpod_out/final/` | | Deploy config (29-DoF G1) | generated and self-verified; drop-in package in `deploy_config/` | | On-robot deploy | deployed on G1 (build deploy stack, sim-to-sim, hardware) | ## Repo layout | Path | What it is | | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `TRAINING_RUNPOD.md` | The real, reproducible training run (RunPod H100, `unitree_rl_mjlab`): version pins, exact commands, results. Start here for training. | | `DEPLOY.md` | Pre-flight package for the real G1: the deploy contract (obs 154-dim, action 29, 50 Hz, gains, joint map) extracted from the saved config, plus the checklist. | | `deploy_config/` | Drop-in deploy package for `unitree_rl_mjlab`'s `deploy/robots/g1` (29-DoF): generated `deploy.yaml`, `policy.onnx`, `jab.npz`, FSM snippet, plus the variant research. | | `CAPTURE_GUIDE.md` | How to film the jab (camera angle, framing). | | `data/README.md` | The CSV → npz → train data spec with format guarantees. | | `G1_PLAN.md` | Approach and key decisions. | | `NEBIUS_TRAINING.md`, `AGENT_TRAIN_RUNBOOK.md` | The Isaac-Lab/BeyondMimic alternative we planned but did not run. Banner-flagged. | | `data/` | The ~120 validated G1-motion CSVs. [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 47 recognized source files, 212 KB. - Python (language) — detected in the code - Hugging Face (technology) — claimed on Devpost, not found in the code - PyTorch (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 271) ``` AGENT_TRAIN_RUNBOOK.md batch_process.py CAPTURE_GUIDE.md config.env data/batch_summary.txt data/batch_total.txt data/csv_to_npz.py data/IMG_3324.csv data/IMG_3325.csv data/IMG_3326.csv data/IMG_3327.csv data/IMG_3328.csv data/IMG_3329.csv data/IMG_3330.csv data/IMG_3331.csv data/IMG_3332.csv data/IMG_3333.csv data/IMG_3334.csv data/IMG_3335.csv data/IMG_3336.csv data/IMG_3337.csv data/IMG_3338.csv data/IMG_3340.csv data/IMG_3341.csv data/IMG_3343.csv data/IMG_3344.csv data/IMG_3345.csv data/IMG_3346.csv data/IMG_3347.csv data/IMG_3348.csv data/IMG_3349.csv data/IMG_3350.csv data/IMG_3351.csv data/IMG_3352.csv data/IMG_3353.csv data/IMG_3354.csv data/IMG_3355.csv data/IMG_3357.csv data/IMG_3358.csv data/IMG_3359.csv data/IMG_3360.csv data/IMG_3361.csv data/IMG_3362.csv data/IMG_3363.csv data/IMG_3364.csv data/IMG_3365.csv data/IMG_3366.csv data/IMG_3367.csv data/IMG_3368.csv data/IMG_3369.csv data/IMG_3370.csv data/IMG_3371.csv data/IMG_3372.csv data/IMG_3373.csv data/IMG_3374.csv data/IMG_3375.csv data/IMG_3376.csv data/IMG_3425.csv data/IMG_3429.csv data/IMG_3430.csv data/IMG_3431.csv data/IMG_3432.csv data/IMG_3433.csv data/IMG_3434.csv data/IMG_3435.csv data/IMG_3436.csv data/IMG_3437.csv data/IMG_3438.csv data/IMG_3439.csv data/IMG_3440.csv data/IMG_3441.csv data/IMG_3442.csv data/IMG_3443.csv data/IMG_3444.csv data/IMG_3445.csv data/IMG_3446.csv data/IMG_3447.csv data/IMG_3448.csv data/IMG_3449.csv data/IMG_3450.csv data/IMG_3451.csv data/IMG_3452.csv data/IMG_3453.csv data/IMG_3454.csv data/IMG_3455.csv data/IMG_3456.csv data/IMG_3457.csv data/IMG_3458.csv data/IMG_3459.csv data/IMG_3460.csv data/IMG_3461.csv data/IMG_3462.csv data/IMG_3463.csv data/IMG_3464.csv data/IMG_3465.csv data/IMG_3466.csv data/IMG_3467.csv data/IMG_3468.csv data/IMG_3469.csv data/IMG_3470.csv data/IMG_3471.csv data/IMG_3472.csv data/IMG_3473.csv data/IMG_3474.csv data/IMG_3475.csv data/IMG_3476.csv data/IMG_3477.csv data/IMG_3478.csv data/IMG_3479.csv data/IMG_3480.csv data/IMG_3481.csv data/IMG_3482.csv data/IMG_3483.csv data/IMG_3484.csv data/IMG_3485.csv data/IMG_3486.csv data/IMG_3487.csv data/IMG_3488.csv data/IMG_3489.csv data/IMG_3490.csv [151 more files omitted for size] ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - update readme - added deployment setup - update readme - update readme - update readme - rename - updated readme - updated readme - updated readme - updated readme - Merge branch 'main' of https://github.com/abtonmoy/ai_hackathon_calhacks - updated readme - add punch motion analysis and robot action export - added training documentation - added first 50 jab data - Merge branch 'main' of https://github.com/abtonmoy/ai_hackathon_calhacks - added more data - Added data training optimizers file - added csv to npz - added a sample ## Key source files (fetched from GitHub, selected and truncated for size) ### G1_PLAN.md ```markdown # G1 Jab — Approach & Key Decisions Why the pipeline is built the way it is. For the *how*, see `g1/README.md` (overview), `g1/CAPTURE_GUIDE.md` (filming), and `g1/TRAINING_RUNPOD.md` (training). ## Goal Turn **human jab videos** into a **deployable motion-tracking policy for a real 29-DoF Unitree G1** — no marker suits, no manual animation. ## The pipeline & why each stage ``` video → GVHMR (markerless mocap) → GMR (retarget to G1) → CSV → unitree_rl_mjlab RL → policy.onnx → G1 ``` - **Markerless mocap (GVHMR), not markers.** A phone video is the lowest-friction capture. GVHMR recovers world-grounded 3D body motion (it's the input GMR's `gvhmr_to_robot` consumes directly). Run with `-s` (SLAM off) — right for a static-camera, feet-planted jab, and skips the painful DPVO build. - **GMR retarget, because human body ≠ robot body.** Different limb lengths, joints, and limits → IK solves "what G1 joint angles reproduce this motion," outputting clean 29-DoF G1 joints. - **RL motion-tracking, because the reference isn't dynamically feasible as-is.** GVHMR/GMR give a *kinematic* reference; an RL policy learns to *execute* it on the real robot's dynamics while staying balanced. This is what makes it deploy. ## Key decisions - **Trainer: `unitree_rl_mjlab` (MuJoCo), not Isaac Lab.** Lighter (no Isaac Sim), ships the **deployable `Unitree-G1-Tracking-No-State-Estimation`** task (obs are hardware-realizable), has its own `csv_to_npz`, and a sim-to-real path into `unitree_sdk2`. Ran on a RunPod H100. (Isaac/BeyondMimic was the original plan — `g1/NEBIUS_TRAINING.md`, kept as the alternative.) - **One policy over all clips via concatenation.** The tracking task loads one motion file, so we concatenate all clips into one long motion; the env samples random start points across it → the policy sees every clip. Single-clip training converges *tighter* (crisper jab); multi-motion is more *general* but plateaus looser (the multi-motion tax). - **Hardware locked: 29-DoF G1.** Confirmed with Ultimate Bots; matches our data exactly (legs 12 + waist 3 + arms 14 = SDK `G1JointIndex` grouping), so no re-retargeting. ## Deployability (designed in, not bolted on) - "No-State-Estimation" task → policy observes only what the real G1 can measure (joint encoders + IMU), no privileged sim state. - Auto-exports `policy.onnx` (`obs → actions`), the contract the Jetson Orin runs. - Deploy via `unitree_rl_mjlab` `deploy/robots/g1` → `unitree_sdk2` LowCmd PD ~50Hz (joint-order map + Kp/Kd in the deploy config). ## What's done vs. open - ✅ Capture → ~120 validated jab CSVs; training running on H100; deployable ONNX. - ⏳ On-robot deploy (needs robot time). Single-clip "hero" jab is the crisp-demo option if the multi-motion policy reads too loose. ## Risk / fallback Every rung is still a valid result: deployable ONNX (have it) → trained policy in sim (have it) → validated motion dataset (have it). We never end with nothing. ``` ### TRAINING_RUNPOD.md ```markdown # G1 Jab — Training (the path we ACTUALLY ran, reproducible) This is the **real, executed** training pipeline — MuJoCo-based `unitree_rl_mjlab` on a RunPod H100, **no Isaac Lab**. It takes the validated CSVs from the capture side (`g1/data/*.csv`) all the way to a trained, deployable G1 jab policy. > The Isaac-Lab / BeyondMimic route in `NEBIUS_TRAINING.md` and > `AGENT_TRAIN_RUNBOOK.md` was the original plan; we switched to `unitree_rl_mjlab` > because it's lighter (no Isaac Sim), runs the **deployable > `Unitree-G1-Tracking-No-State-Estimation`** task, has its own `csv_to_npz`, and > ships a sim-to-real path into `unitree_sdk2`. Our CSVs are format-identical for > both, so only the trainer changed. --- ## 0. Box - **RunPod pod**, **NVIDIA H100 80GB**, image `runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04`. - 100GB **persistent** volume at `/workspace` (put repo + data + checkpoints here so a pod stop doesn't wipe them). - SSH in: `ssh -i ~/.ssh/id_ed25519 -p root@`. ## 1. Install (with the version pins that matter) ```bash cd /workspace git clone https://github.com/unitreerobotics/unitree_rl_mjlab.git cd unitree_rl_mjlab apt-get install -y libyaml-cpp-dev libboost-all-dev libeigen3-dev libspdlog-dev libfmt-dev pip install -e . ``` **Then fix the bleeding-edge deps** — `unitree_rl_mjlab` leaves these UNPINNED, so pip grabs too-new versions that break at runtime. Pin them: ```bash pip install "mujoco==3.5.0" "warp-lang==1.12.0" scipy ``` Why (we hit all three): - `mujoco` 3.9.0 → missing `mjENBL_MULTICCD` enum that `mujoco-warp==3.5.0` needs → **pin 3.5.0**. - `warp-lang` 1.14.0/1.13.0 → removed `wp.context` that `mjlab` calls → **pin 1.12.0** (has it). - `scipy` not pulled but `mjlab.terrains` imports it → **add scipy**. Saved on the box as `/workspace/WORKING_VERSIONS.txt`. Headless rendering also needs (Section 5): `apt-get install -y libegl1 libglvnd0 libgles2 libgl1`. ## 2. Get the CSVs onto the box ```bash mkdir -p /workspace/csvs # from local: scp -P -i ~/.ssh/id_ed25519 g1/data/IMG_*.csv root@:/workspace/csvs/ ``` ## 3. CSV → NPZ `unitree_rl_mjlab`'s own converter (MuJoCo FK, no Isaac, no WandB). Required args: `--robot g1` (29-DoF; `g1_23dof` for the 23-DoF variant) and `--output-name`. CSV format it expects = exactly ours: headerless, `root_pos(3) root_rot_xyzw(4) dof(29)`, 30fps (it interpolates to 50fps internally). ```bash cd /workspace/unitree_rl_mjlab python scripts/csv_to_npz.py --input-file /workspace/csvs/IMG_3429.csv \ --robot g1 --output-name IMG_3429 # -> src/assets/motions/g1/IMG_3429.npz ``` ## 4. Train The tracking task loads ONE `--motion-file`. Two modes: ### 4a. Single clip (crisp, converges tight) ```bash python scripts/train.py Unitree-G1-Tracking-No-State-Estimation \ --motion-file=src/assets/motions/g1/IMG_3429.npz \ --env.scene.num-envs 4096 --agent.max-iterations 10000 \ --agent.save-interval 500 --agent.logger tensorboard --video False \ --agent.run-name [truncated — 3656 more characters] ``` ### make_replay.py ```python import sys, os import numpy as np import pandas as pd import matplotlib.pyplot as plt from matplotlib.animation import FuncAnimation, PillowWriter def load_xyz(path): df = pd.read_csv(path, header=None) df = df.apply(pd.to_numeric, errors="coerce").dropna(axis=1, how="all").dropna() arr = df.to_numpy(dtype=float) if arr.shape[1] < 3: raise ValueError("CSV needs at least 3 numeric columns") return arr[:, :3] if len(sys.argv) < 2: print("Usage: python make_replay.py data/IMG_3343.csv") sys.exit(1) path = sys.argv[1] xyz = load_xyz(path) # Downsample so the GIF is not huge step = max(1, len(xyz) // 180) xyz = xyz[::step] name = os.path.splitext(os.path.basename(path))[0] os.makedirs("output/replays", exist_ok=True) fig = plt.figure() ax = fig.add_subplot(111, projection="3d") center = xyz.mean(axis=0) scale = max(np.ptp(xyz[:, 0]), np.ptp(xyz[:, 1]), np.ptp(xyz[:, 2])) if scale < 1e-6: scale = 1.0 ax.set_xlim(center[0] - scale / 2, center[0] + scale / 2) ax.set_ylim(center[1] - scale / 2, center[1] + scale / 2) ax.set_zlim(center[2] - scale / 2, center[2] + scale / 2) ax.set_title(f"{name} motion replay") ax.set_xlabel("X") ax.set_ylabel("Y") ax.set_zlabel("Z") line, = ax.plot([], [], []) point, = ax.plot([], [], [], marker="o") def update(i): line.set_data(xyz[:i+1, 0], xyz[:i+1, 1]) line.set_3d_properties(xyz[:i+1, 2]) point.set_data([xyz[i, 0]], [xyz[i, 1]]) point.set_3d_properties([xyz[i, 2]]) return line, point ani = FuncAnimation(fig, update, frames=len(xyz), interval=50, blit=False) out = f"output/replays/{name}_replay.gif" ani.save(out, writer=PillowWriter(fps=20)) plt.close() print("Saved:", out) ``` ### export_robot_action.py ```python import sys, os, json import numpy as np import pandas as pd FPS = 30 def load_xyz(path): df = pd.read_csv(path, header=None) df = df.apply(pd.to_numeric, errors="coerce").dropna(axis=1, how="all").dropna() arr = df.to_numpy(dtype=float) if arr.shape[1] < 3: raise ValueError("CSV needs at least 3 numeric columns") return arr[:, :3] def normalize_trajectory(xyz): # center at first frame traj = xyz - xyz[0] # scale to robot-arm-friendly range max_range = np.max(np.ptp(traj, axis=0)) if max_range > 1e-6: traj = traj / max_range * 0.35 # add a default robot arm offset in front of torso offset = np.array([0.35, -0.20, 1.10]) return traj + offset def main(path): xyz = load_xyz(path) robot_xyz = normalize_trajectory(xyz) name = os.path.splitext(os.path.basename(path))[0] os.makedirs("output/robot_actions", exist_ok=True) action = { "name": f"{name}_right_arm_jab", "source_file": path, "description": "Robot-ready end-effector trajectory generated from human punch motion.", "control_mode": "right_wrist_end_effector_tracking", "target_robot": "humanoid_right_arm", "assumed_fps": FPS, "note": "This is not direct motor control. It is an IK/retargeting-ready trajectory.", "trajectory": [ { "t": round(i / FPS, 4), "right_wrist_target": { "x": float(p[0]), "y": float(p[1]), "z": float(p[2]) } } for i, p in enumerate(robot_xyz) ] } out = f"output/robot_actions/{name}_robot_action_plan.json" with open(out, "w") as f: json.dump(action, f, indent=2) print("Saved:", out) print("Frames:", len(robot_xyz)) print("This is IK/retargeting-ready, not direct robot motor commands.") if __name__ == "__main__": if len(sys.argv) < 2: print("Usage: python export_robot_action.py data/IMG_3327.csv") sys.exit(1) main(sys.argv[1]) ``` ### demo_csv.py ```python import sys, json, os, glob import numpy as np import pandas as pd import matplotlib.pyplot as plt def load_csv(path): df = pd.read_csv(path, header=None) df = df.apply(pd.to_numeric, errors="coerce").dropna(axis=1, how="all").dropna() arr = df.to_numpy(dtype=float) # Use first 3 columns as x,y,z trajectory if arr.shape[1] >= 3: xyz = arr[:, :3] else: raise ValueError(f"{path} has fewer than 3 numeric columns") return xyz def analyze(path, fps=30): xyz = load_csv(path) dt = 1.0 / fps vel = np.zeros(len(xyz)) vel[1:] = np.linalg.norm(np.diff(xyz, axis=0), axis=1) / dt peak_idx = int(np.argmax(vel)) peak_velocity = float(vel[peak_idx]) duration = float(len(xyz) * dt) displacement = xyz[-1] - xyz[0] dx, dy, dz = displacement.tolist() # simple rule-based punch type guess abs_disp = np.abs(displacement) if abs_disp[0] >= abs_disp[1] and abs_disp[0] >= abs_disp[2]: punch_type = "jab" elif abs_disp[1] >= abs_disp[0] and abs_disp[1] >= abs_disp[2]: punch_type = "hook" else: punch_type = "uppercut" name = os.path.splitext(os.path.basename(path))[0] os.makedirs("output", exist_ok=True) profile = { "source_file": path, "num_frames": int(len(xyz)), "fps_assumed": fps, "duration_sec": duration, "predicted_punch_type": punch_type, "peak_velocity": peak_velocity, "peak_frame": peak_idx, "displacement": { "dx": float(dx), "dy": float(dy), "dz": float(dz) }, "trajectory": [ {"frame": i, "x": float(p[0]), "y": float(p[1]), "z": float(p[2]), "velocity": float(vel[i])} for i, p in enumerate(xyz) ] } json_path = f"output/{name}_motion_profile.json" with open(json_path, "w") as f: json.dump(profile, f, indent=2) plt.figure() plt.plot(vel) plt.axvline(peak_idx, linestyle="--") plt.title(f"{name} wrist/trajectory velocity") plt.xlabel("Frame") plt.ylabel("Velocity") plot_path = f"output/{name}_velocity.png" plt.savefig(plot_path, dpi=160) plt.close() print("Source:", path) print("Frames:", len(xyz)) print("Predicted punch:", punch_type) print("Duration:", round(duration, 3), "sec") print("Peak velocity:", round(peak_velocity, 3)) print("Saved:", json_path) print("Saved:", plot_path) if __name__ == "__main__": if len(sys.argv) > 1: analyze(sys.argv[1]) else: files = sorted(glob.glob("data/*.csv")) analyze(files[0]) ``` ### batch_process.py ```python import os, glob, csv, json import numpy as np import pandas as pd import matplotlib.pyplot as plt FPS = 30 def load_csv(path): df = pd.read_csv(path, header=None) df = df.apply(pd.to_numeric, errors="coerce").dropna(axis=1, how="all").dropna() arr = df.to_numpy(dtype=float) if arr.shape[1] < 3: raise ValueError(f"{path} has fewer than 3 numeric columns") return arr[:, :3] def analyze_file(path): xyz = load_csv(path) dt = 1.0 / FPS vel = np.zeros(len(xyz)) vel[1:] = np.linalg.norm(np.diff(xyz, axis=0), axis=1) / dt peak_idx = int(np.argmax(vel)) peak_velocity = float(vel[peak_idx]) duration = float(len(xyz) * dt) displacement = xyz[-1] - xyz[0] abs_disp = np.abs(displacement) if abs_disp[0] >= abs_disp[1] and abs_disp[0] >= abs_disp[2]: punch_type = "jab" elif abs_disp[1] >= abs_disp[0] and abs_disp[1] >= abs_disp[2]: punch_type = "hook" else: punch_type = "uppercut" name = os.path.splitext(os.path.basename(path))[0] os.makedirs("output/profiles", exist_ok=True) os.makedirs("output/plots", exist_ok=True) profile = { "source_file": path, "num_frames": int(len(xyz)), "fps_assumed": FPS, "duration_sec": duration, "predicted_punch_type": punch_type, "peak_velocity": peak_velocity, "peak_frame": peak_idx, "displacement": { "dx": float(displacement[0]), "dy": float(displacement[1]), "dz": float(displacement[2]) } } json_path = f"output/profiles/{name}_motion_profile.json" with open(json_path, "w") as f: json.dump(profile, f, indent=2) plt.figure() plt.plot(vel) plt.axvline(peak_idx, linestyle="--") plt.title(f"{name} velocity") plt.xlabel("Frame") plt.ylabel("Velocity") plot_path = f"output/plots/{name}_velocity.png" plt.savefig(plot_path, dpi=160) plt.close() return { "file": os.path.basename(path), "frames": len(xyz), "duration_sec": round(duration, 3), "predicted_punch_type": punch_type, "peak_velocity": round(peak_velocity, 4), "peak_frame": peak_idx, "profile_path": json_path, "plot_path": plot_path } def main(): files = sorted(glob.glob("data/*.csv")) print(f"Found {len(files)} CSV files") results = [] for path in files: try: result = analyze_file(path) results.append(result) print(f"{result['file']}: {result['predicted_punch_type']} | peak={result['peak_velocity']}") except Exception as e: print(f"FAILED {path}: {e}") os.makedirs("output", exist_ok=True) summary_path = "output/batch_summary.csv" with open(summary_path, "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=results[0].keys()) writer.writeheader() writer.writerows(results) print("\nSaved summary:", summary_path) print("Saved profiles: output/profiles/") print("Saved plots: output/plots/") if __name__ == "__main__": main() ``` ### scripts/render_ref.sh ```shell #!/usr/bin/env bash # Render a GMR robot-motion .pkl (the retargeted G1 reference) to video. WSL. set -uo pipefail export MUJOCO_GL=egl PKL="${1:?pkl path}"; OUT="${2:?out mp4}" cd /home/abtonmoy/repos/GMR timeout 240 /home/abtonmoy/repos/GMR/.venv/bin/python scripts/vis_robot_motion.py \ --robot unitree_g1 --robot_motion_path "$PKL" --record_video --video_path "$OUT" pkill -f vis_robot_motion 2>/dev/null || true echo "REF_RENDER_EXIT $? -> $OUT" ls -la "$OUT" 2>/dev/null ``` ### scripts/wsl_local_sim.sh ```shell #!/usr/bin/env bash # Run the trained policy in MuJoCo locally (sim-to-sim on the 4060). WSL. export MUJOCO_GL=egl cd /home/abtonmoy/repos/unitree_rl_mjlab /home/abtonmoy/repos/unitree_rl_mjlab/.venv/bin/python scripts/play.py \ Unitree-G1-Tracking-No-State-Estimation \ --motion-file=src/assets/motions/g1/IMG_3429.npz \ --checkpoint-file=/mnt/c/research_projects/calaihacks/g1/runpod_out/final/checkpoints/model_9999.pt \ --num-envs 1 --video True --video-length 120 echo "LOCALSIM_EXIT $?" find logs -name "rl-video-step-0.mp4" -printf "%p (%s bytes)\n" 2>/dev/null | tail -1 ``` ### scripts/30_train.sh ```shell #!/usr/bin/env bash # Step 5 — Train the G1 jab tracking policy in Isaac Lab (on the Nebius GPU). set -euo pipefail HERE="$(cd "$(dirname "$0")/.." && pwd)" source "${HERE}/config.env" cd "${RL_LAB_DIR}" MOTION="${1:-src/assets/motions/g1/${MOTION_NAME}.npz}" ITERS="${MAX_ITERATIONS:-20000}" echo ">> Training ${TRACK_TASK}" echo ">> motion: ${MOTION} iters: ${ITERS}" python scripts/train.py "${TRACK_TASK}" \ --motion_file="${MOTION}" \ --headless \ --max_iterations "${ITERS}" echo ">> Done. Newest run under ${RL_LAB_DIR}/logs/" echo ">> Eval in sim: bash scripts/31_play.sh" ``` ### scripts/wsl_gvhmr_ckpts.sh ```shell #!/usr/bin/env bash # Run INSIDE WSL natively (avoids Git-Bash->wsl.exe quoting issues): # wsl.exe -d Ubuntu bash /mnt/c/research_projects/calaihacks/g1/scripts/wsl_gvhmr_ckpts.sh # Downloads GVHMR checkpoints (public Google Drive) into inputs/checkpoints. set -euo pipefail export PATH="$HOME/.local/bin:$PATH" CK="$HOME/repos/GVHMR/inputs/checkpoints" mkdir -p "$CK" echo ">> downloading checkpoints into: $CK" uvx gdown --folder "https://drive.google.com/drive/folders/1eebJ13FUEXrKBawHpJroW0sNSxLjh9xD" -O "$CK" echo ">> checkpoint files (excluding body_models):" find "$CK" -type f ! -path "*body_models*" -printf " %p (%s bytes)\n" echo ">> CKPTS-DONE" ``` [34 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]