# Project export: LLM-Generated Robot URDF This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Generate robot URDF from natural language or images. Validate with urdfpy, simulate in PyBullet, and export to MJCF/SDF. One prompt to a working robot. - Devpost: https://devpost.com/software/llm-generated-robot-urdf - GitHub: https://github.com/Kunal2341/tree-hack-robo - Video: https://www.youtube.com/embed/MB1bjE6FRtM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Kunal Aneja (18 commits), Cursor (15 commits) ## Devpost submission (written by the team) ### Inspiration Robotics simulation is powerful, but creating robot models is tedious. URDF (Unified Robot Description Format) requires precise XML: links, joints, inertials, collision geometries. A single typo or misplaced origin can break everything. We wondered: what if you could describe a robot in plain English and get a working, simulated URDF? We were inspired by the gap between natural language (how humans think about robots) and formal representations (how simulators need them). LLMs excel at structured output—why not bridge that gap for robotics? ### What it does The project turns natural language into simulated robot URDFs. You type "A 4-legged dog robot" or "A box with 4 wheels"—and get a valid, physics-tested URDF in seconds. Core features: Natural language generation — Describe any robot; the LLM produces URDF XML Image-to-URDF — Upload a sketch, diagram, or photo; GPT-4o vision analyzes it and generates a matching URDF RAG-augmented generation — Retrieves relevant URDF snippets from a library (quadrupeds, hexapods, wheeled bases) to improve output quality Validation pipeline — Parse checks (urdfpy), link position checks (no overlapping geometry), effort limits (no floppy robots) Physics simulation — PyBullet runs a 5-second sim; detects explosions, fall-over, self-collisions Error feedback loop — When validation or simulation fails, the error is fed back to the LLM for automatic retry (up to 5 attempts) Iterative refinement — "Make it heavier," "add another wheel," "shorter legs"—modify existing robots with follow-up prompts Multi-terrain stress testing — Flat, uneven, stairs, slope; score robots across all terrains Export — URDF → MJCF (MuJoCo) and SDF (Gazebo) conversion Web UI — 3D preview (Three.js + urdf-loader), history, leaderboard, feedback suggestions ### How we built it Architecture: A three-stage pipeline—Generate → Validate → Simulate—with an orchestrator agent that retries on failure. Generation — OpenAI GPT-4o-mini with a system prompt that enforces URDF rules. For multi-legged robots, we inject chain-of-thought: the LLM first computes angles ( \theta_i = \frac{360°}{n} \cdot i ) and mount positions ( (x, y) = (r \cos\theta, r \sin\theta) ) before writing XML, avoiding legs at ((0,0,0)). Generation — OpenAI GPT-4o-mini with a system prompt that enforces URDF rules. For multi-legged robots, we inject chain-of-thought: the LLM first computes angles ( \theta_i = \frac{360°}{n} \cdot i ) and mount positions ( (x, y) = (r \cos\theta, r \sin\theta) ) before writing XML, avoiding legs at ((0,0,0)). RAG — TF-IDF over a corpus of URDF snippets (quadruped, hexapod, wheeled base, etc.). Query tokens are matched; top-k snippets are injected into the prompt as examples. RAG — TF-IDF over a corpus of URDF snippets (quadruped, hexapod, wheeled base, etc.). Query tokens are matched; top-k snippets are injected into the prompt as examples. Validation — urdfpy for parse correctness; custom checks for link offsets (( | \text{origin} | > 0.01 ) m) and joint effort (( \geq 100 ) N·m). Validation — urdfpy for parse correctness; custom checks for link offsets (( | \text{origin} | > 0.01 ) m) and joint effort (( \geq 100 ) N·m). Simulation — PyBullet headless mode. Terrain loaders for flat, uneven (heightfield), stairs, slope. Physics sanity check (0.5 s) catches explosions and self-collisions before full 5 s run. Simulation — PyBullet headless mode. Terrain loaders for flat, uneven (heightfield), stairs, slope. Physics sanity check (0.5 s) catches explosions and self-collisions before full 5 s run. Scoring — Composite score from stability (displacement), uprightness (tilt cosine), and grounding (height). Terrain multipliers: flat 1.0×, slope 1.15×, stairs 1.25×, uneven 1.30×. Final score: [ S = \min\left(100,\; \left(0.4 S_{\text{stab}} + 0.35 S_{\text{upright}} + 0.25 S_{\text{ground}}\right) \cdot m_{\text{terrain}}\right) ] Scoring — Composite score from stability (displacement), uprightness (tilt cosine), and grounding (height). Terrain multipliers: flat 1.0×, slope 1.15×, stairs 1.25×, uneven 1.30×. Final score: [ S = \min\left(100,\; \left(0.4 S_{\text{stab}} + 0.35 S_{\text{upright}} + 0.25 S_{\text{ground}}\right) \cdot m_{\text{terrain}}\right) ] Web stack — Flask backend, vanilla JS frontend, Three.js + urdf-loader for 3D preview. History and leaderboard persisted to JSON. Web stack — Flask backend, vanilla JS frontend, Three.js + urdf-loader for 3D preview. History and leaderboard persisted to JSON. ### Challenges we ran into Leg overlap — Early multi-legged robots had all legs at ((0,0,0)); PyBullet exploded. We added chain-of-thought prompting so the LLM computes angles and positions first. Leg overlap — Early multi-legged robots had all legs at ((0,0,0)); PyBullet exploded. We added chain-of-thought prompting so the LLM computes angles and positions first. Floppy robots — Weak joint effort caused limbs to collapse. We added effort validation (min 100) and mass validation (0.01–500 kg). Floppy robots — Weak joint effort caused limbs to collapse. We added effort validation (min 100) and mass validation (0.01–500 kg). Self-collisions — Links touching at spawn caused instability. We added a sanity check that detects non-adjacent link contacts and feeds that back to the LLM. Self-collisions — Links touching at spawn caused instability. We added a sanity check that detects non-adjacent link contacts and feeds that back to the LLM. URDF extraction — LLMs sometimes wrap XML in markdown or add commentary. We use regex to extract and strip the rest. URDF extraction — LLMs sometimes wrap XML in markdown or add commentary. We use regex to extract and strip the rest. PyBullet on ARM Mac — Some users needed brew install cmake for PyBullet to build. We made simulation optional so generation and validation still work without it. PyBullet on ARM Mac — Some users needed brew install cmake for PyBullet to build. We made simulation optional so generation and validation still work without it. ### Accomplishments we're proud of End-to-end pipeline — From "A 4-legged dog" to a simulated, scored robot in one flow Self-healing — Error feedback loop means the system often fixes its own mistakes without human intervention Image-to-URDF — Two-stage pipeline (analyze → generate) with RAG for sketch/diagram input Multi-format export — URDF, MJCF, SDF from a single description Stress testing — Robots tested on four terrains; leaderboard with terrain-filtered rankings Feedback suggestions — UI suggests refinements ("Robot is unstable — widen the base") that users can one-click apply ### What we learned Structured prompting matters — Chain-of-thought for geometry (angles, positions) dramatically improved multi-legged robot quality Validation layers compound — Parse → physics sanity → full sim catches different failure modes RAG helps — Even a small snippet library (10–15 URDFs) improved generation for similar robot types Vision + text — GPT-4o vision can interpret sketches and diagrams; combining that with the text pipeline opened image-to-URDF ### What's next for TreeHackNow Mesh support — Generate or reference STL/OBJ meshes for more realistic geometry Trajectory optimization — Use simulation feedback to tune joint parameters (PD gains, limits) automatically Multi-robot scenarios — Generate and simulate multiple robots interacting ROS 2 integration — Export to ROS 2 packages with launch files and config Community snippet library — Allow users to contribute URDF snippets to the RAG index Fine-tuned model — Train a small model on URDF examples for faster, cheaper generation ## README (from the GitHub repository) # RoboWhisper — LLM-Generated Robot URDF Generate robot URDF files from natural language using an LLM, validate with urdfpy, and simulate in PyBullet. --- ## Architecture & Design ### High-Level Overview ![RoboWhisper Pipeline](docs/images/pipeline.png) ``` ┌─────────────────────────────────────────────────────────────────────────────────┐ │ RoboWhisper Pipeline │ └─────────────────────────────────────────────────────────────────────────────────┘ User Prompt ┌──────────────┐ "A 4-legged dog" ──────────►│ generate │──────────► Raw URDF XML │ (LLM) │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ validate │──────────► Parse + custom checks │ (urdfpy) │ └──────┬───────┘ │ ▼ ┌──────────────┐ │ simulate │──────────► PyBullet physics │ (PyBullet) │ └──────┬───────┘ │ ▼ output/robot.urdf ``` ### Data Flow ``` Natural Language ──► LLM (GPT-4o-mini) ──► URDF XML ──► Validation ──► Simulation ──► Saved URDF │ │ │ │ │ └── On failure: error feedback → retry │ └── On failure: error feedback → retry └── System prompt + optional chain-of-thought (multi-legged) ``` ### Component Architecture | Module | Responsibility | Key Functions | |--------|----------------|---------------| | **`src/generate.py`** | LLM-based URDF generation | `generate_robot()`, `extract_urdf_from_response()` | | **`src/validate.py`** | URDF correctness & physics sanity | `validate_all()`, `validate_urdf_parse()`, `check_link_positions()`, `check_effort_limits()` | | **`src/simulate.py`** | Physics simulation in PyBullet | `simulate_urdf()` — loads URDF, terrain modes (flat/uneven/stairs/slope), 5s sim | | **`src/agent.py`** | Orchestrator with retry loop | `run_agent()` — Generate → Validate → Simulate, up to 5 retries with error feedback | ### Design Decisions 1. **Separation of concerns** — Generation, validation, and simulation are independent modules. Each can be run standalone (`python -m src.generate`, `python -m src.simulate`) or composed by the agent. 2. **Error feedback loop** — When validation or simulation fails, the error message is fed back into the LLM prompt so it can fix the URDF. Max 5 retries prevents infinite loops. 3. **Multi-legged chain-of-thought** — For prompts like "4-legged dog" or "hexapod", the agent injects a structured prompt that instructs the LLM to: (a) compute angles for each leg, (b) compute (x,y) mount positions from body radius, (c) then generate XML. This avoids legs overlapping at (0,0,0). 4. **Validation layers**: - **Parse** — `urdfpy` ensures valid URDF syntax and structure. - **Link positions** — Child links (wheels, legs) must be offset from parent to avoid self-collision. - **Effort limits** — Joint effort ≥ 100 to prevent "floppy noodle" robots. 5. **Simulation stability check** — If the robot base moves >50m from origin during the 5s sim, it's considered "exploded" (unstable). ### File Structure ``` RoboWhisper/ ├── src/ │ ├── agent.py # Orchestrator: retry loop + error feedback │ ├── generate.py # LLM → URDF (OpenAI API) │ ├── simulate.py # PyBullet physics (headless or GUI) │ └── validate.py # urdfpy + custom checks ├── web/ │ ├── app.py # Flask server — /api/generate, /api/refine, /api/simulate │ ├── templates/ # index.html │ └── static/ # app.js, style.css — 3D preview (Three.js + urdf-loader) ├── prompts/ │ └── system_prompt.txt # LLM system instructions ├── output/ # Generated URDFs (agent_test.urdf, robot.urdf) ├── package.json # npm run web — start localhost server ├── requirements.txt └── environment.yml ``` ### External Dependencies | Dependency | Purpose | |------------|---------| | **OpenAI** | LLM API for natural language → URDF generation | | **urdfpy** | Parse and validate URDF XML | | **PyBullet** | Physics simulation (gravity, collision, stability) | --- ## Setup **Conda (recommended):** ```bash conda env create -f environment.yml conda activate robowhisper ``` **Or pip only:** ```bash pip install -r requirements.txt ``` > **Note:** PyBullet may require building from source on some systems. If simulation fails, generation and validation still work. On macOS ARM, you may need `brew install cmake` first. Set your OpenAI API key: ```bash export OPENAI_API_KEY="your-key-here" ``` ## Usage ### Web UI (recommended) ```bash npm run web ``` Then open **http://localhost:5000** in your browser. You get: - **Generate** — describe a robot (e.g. "A 4-legged dog"), get URDF + 3D preview - **Refine** — select a robot, type a change (e.g. "make it heavier"), get updated URDF - **Simulate** — run PyBullet physics on flat/uneven/stairs/slope terrain ### CLI ```bash # Generate a robot (simple) python -m src.generate "A box with 4 wheels" # Simulate a URDF (with optional terrain mode) python -m src.simulate output/robot.urdf python -m src.simulate output/robot.urdf --terrain uneven # uneven, stairs, slope # Terrain modes: flat (default), uneven, stairs, slope — test robustness # Full agent loop (validate + simulate + retry on failure) python -m src.agent "A 4-legged dog robot" python -m src.agent "A 4-legged dog" --terrain slope # optional terrain for sim # Web UI (same as above) npm run web ``` ## Plan See [PLAN.md](PLAN.md) for the full implementation roadmap. **Phase 5** covers web UI enhancements: Download URDF, View source, Delete from history, Prompt examples, History persistence, Simulation metrics. ## Detected evidence (automated analysis) Indexed codebase: 23 recognized source files, 211 KB. - CSS (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (29 of 29) ``` .gitignore CONTRIBUTING.md environment.yml GITHUB_SETUP.md output/.gitkeep package.json PLAN.md prompts/refine_prompt.txt prompts/system_prompt.txt README.md requirements.txt src/__init__.py src/agent.py src/convert.py src/generate.py src/rag.py src/score.py src/simulate.py src/validate.py src/vision.py SUBMISSION.md tests/__init__.py tests/test_leaderboard_api.py tests/test_score.py UPDATES.md web/app.py web/static/app.js web/static/style.css web/templates/index.html ``` ### Dependencies - requirements.txt: flask@>=3.0.0, openai@>=1.0.0, pybullet@>=3.2.5, pytest@>=8.0.0, python-dotenv@>=1.0.0, urdfpy@>=0.0.20 ### Recent commits (newest first) - Add files via upload - feat: wire up images — hero-bg, example sketch, pipeline, submission docs - Add files via upload - add the images - feat: add MJCF/SDF export UI, image-to-URDF upload, and output format selector - feat: integrate RAG context and vision/conversion endpoints - docs: add Phase 6 (motor control, replay, stress test) to PLAN.md - feat: add RAG and vision modules, update generate.py - feat: add simulation replay, stress test UI, and motor toggle - feat: add physics sanity check UI and interactive feedback tweaking - feat: add motor/trajectory params to simulate API + stress-test endpoint - feat: add robot scoring, leaderboard UI, and comprehensive tests - feat: add joint motor control and trajectory recording to simulation - feat: add scoring system, leaderboard, and score tests - style: add keyboard shortcut hints and improve input UX - feat: add /api/health endpoint and robot count badge - feat: add mass validation to catch unrealistic robot weights - feat: add structured logging across core modules - feat: add simulation metrics (distance, position, upright check) - feat: persist robot history to output/history.json ## Key source files (fetched from GitHub, selected and truncated for size) ### CONTRIBUTING.md ```markdown # Contributing ## Commit often Prefer many small, focused commits over large monolithic ones. Good commit messages: - `feat: add X` - `fix: handle Y` - `docs: update Z` ``` ### GITHUB_SETUP.md ```markdown # GitHub Setup ## Create the repository 1. Go to [github.com/new](https://github.com/new) 2. Name it `TreeHackNow` (or your preferred name) 3. **Do not** initialize with README, .gitignore, or license (we already have these) 4. Click **Create repository** ## Push your code ```bash cd /Users/kunalaneja/TreeHackNow # Add the remote (replace YOUR_USERNAME with your GitHub username) git remote add origin https://github.com/YOUR_USERNAME/TreeHackNow.git # Push git push -u origin main ``` ## Optional: Use GitHub CLI If you install [GitHub CLI](https://cli.github.com/) (`brew install gh`): ```bash gh auth login gh repo create TreeHackNow --source=. --push ``` ``` ### requirements.txt ``` openai>=1.0.0 urdfpy>=0.0.20 pybullet>=3.2.5 flask>=3.0.0 python-dotenv>=1.0.0 pytest>=8.0.0 ``` ### package.json ``` { "name": "treehacknow", "version": "1.0.0", "description": "LLM-generated robot URDF — generate, validate, simulate", "scripts": { "web": "python web/app.py", "start": "python web/app.py" } } ``` ### web/app.py ```python """ Web UI for TreeHackNow — prompt input, history, 3D preview, iterative refinement. Supports: URDF generation, MJCF/SDF conversion, RAG-enhanced generation, Image-to-URDF. """ import json import os import time import uuid import base64 import tempfile from pathlib import Path from flask import Flask, jsonify, render_template, request # Add project root to path import sys sys.path.insert(0, str(Path(__file__).parent.parent)) from src.agent import run_agent, run_agent_refine from src.simulate import simulate_urdf, stress_test_urdf, physics_sanity_check, generate_feedback_suggestions, TERRAIN_MODES from src.score import compute_score, score_label from src.convert import urdf_to_mjcf, urdf_to_sdf, generate_mjcf, generate_sdf from src.vision import image_to_urdf, image_to_urdf_two_stage, analyze_robot_image app = Flask(__name__, static_folder="static", template_folder="templates") # Max upload size: 10MB app.config["MAX_CONTENT_LENGTH"] = 10 * 1024 * 1024 HISTORY_PATH = Path(__file__).parent.parent / "output" / "history.json" LEADERBOARD_PATH = Path(__file__).parent.parent / "output" / "leaderboard.json" # In-memory history (id -> {prompt, urdf, refined_from, timestamp}) _history: dict[str, dict] = {} # In-memory leaderboard (list of score entries) _leaderboard: list[dict] = [] def _load_history(): """Load history from disk on startup.""" global _history if HISTORY_PATH.exists(): try: data = json.loads(HISTORY_PATH.read_text()) _history = {e["id"]: e for e in data} except (json.JSONDecodeError, KeyError): _history = {} def _save_history(): """Persist history to disk.""" HISTORY_PATH.parent.mkdir(exist_ok=True) entries = sorted(_history.values(), key=lambda e: e.get("timestamp", 0)) HISTORY_PATH.write_text(json.dumps(entries, indent=2)) def _load_leaderboard(): """Load leaderboard from disk on startup.""" global _leaderboard if LEADERBOARD_PATH.exists(): try: _leaderboard = json.loads(LEADERBOARD_PATH.read_text()) except (json.JSONDecodeError, KeyError): _leaderboard = [] def _save_leaderboard(): """Persist leaderboard to disk.""" LEADERBOARD_PATH.parent.mkdir(exist_ok=True) LEADERBOARD_PATH.write_text(json.dumps(_leaderboard, indent=2)) def _ensure_api_key(): if not os.environ.get("OPENAI_API_KEY"): raise ValueError("OPENAI_API_KEY not set. Set it before running the web app.") @app.route("/") def index(): return render_template("index.html") @app.route("/api/generate", methods=["POST"]) def api_generate(): _ensure_api_key() data = request.get_json() or {} prompt = (data.get("prompt") or "").strip() if not prompt: return jsonify({"error": "prompt is required"}), 400 success, urdf, msg = run_agent(prompt) if not success: return jsonify({"success": False, "error": msg}), 200 entry_id = str(uuid.uuid4()) _history[entry_id] = { "id": entry_id, "prompt": prompt, "urdf": urdf, "refined_from": None, "timestamp": time.time(), } _save_history() return jsonify({ "success": True, "id": entry_id, "prompt": prompt, "urdf": urdf, }) @app.route("/api/refine", methods=["POST"]) def api_refine(): _ensure_api_key() data = request.get_json() or {} refinement_prompt = (data.get("prompt") or "").strip() base_id = data.get("base_id") base_urdf = data.get("base_urdf") if not refinement_prompt: return jsonify({"error": "prompt is required"}), 400 if base_urdf: urdf_to_use = base_urdf elif base_id and base_id in _history: urdf_to_use = _history[base_id]["urdf"] else: return jsonify({"error": "base_id or base_urdf is required"}), 400 success, urdf, msg = run_agent_refine(refinement_prompt, urdf_to_use) if not success: return jsonify({"success": False, "error": msg}), 200 entry_id = str(uuid.uuid4()) _history[entry_id] = { "id": entry_id, "prompt": refinement_prompt, "urdf": urdf, "refined_from": base_id, "timestamp": time.time(), } _save_history() return jsonify({ "success": True, "id": entry_id, "prompt": refinement_prompt, "urdf": urdf, "refined_from": base_id, }) @app.route("/api/history", methods=["GET"]) def api_history(): entries = sorted( _history.values(), key=lambda e: e["timestamp"], reverse=True, ) return jsonify({ "history": [ { "id": e["id"], "prompt": e["prompt"], "refined_from": e.get("refined_from"), "timestamp": e["timestamp"], } for e in entries ], }) @app.route("/api/robot/", methods=["GET"]) def api_robot(robot_id): if robot_id not in _history: return jsonify({"error": "not found"}), 404 entry = _history[robot_id] return jsonify({ "id": entry["id"], "prompt": entry["prompt"], "urdf": entry["urdf"], "refined_from": entry.get("refined_from"), }) @app.route("/api/robot/", methods=["DELETE"]) def api_robot_delete(robot_id): if robot_id not in _history: return jsonify({"error": "not found"}), 404 del _history[robot_id] _save_history() return jsonify({"success": True}) @app.route("/api/health", methods=["GET"]) def api_health(): """Health check endpoint — useful for monitoring and uptime checks.""" return jsonify({ "status": "ok", "robot_count": len(_history), "pybullet_available": bool( __import__("importlib").util.find_spec("pybullet") ), "version": "1.0.0", }) @app.route("/api/simulate", methods=["POST"]) def api_simulate(): """Run PyBullet simulation with selected te [truncated — 16519 more characters] ``` ### web/static/app.js ```javascript /** * RoboWhisper Web UI — generate, refine, preview robots */ const API = { generate: (prompt) => fetch("/api/generate", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prompt }), }).then((r) => r.json()), generateMjcf: (prompt) => fetch("/api/generate/mjcf", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prompt }), }).then((r) => r.json()), generateSdf: (prompt) => fetch("/api/generate/sdf", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prompt }), }).then((r) => r.json()), convertMjcf: (robotId, urdf) => fetch("/api/convert/mjcf", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ robot_id: robotId, urdf }), }).then((r) => r.json()), convertSdf: (robotId, urdf) => fetch("/api/convert/sdf", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ robot_id: robotId, urdf }), }).then((r) => r.json()), imageToUrdf: (formData) => fetch("/api/image-to-urdf", { method: "POST", body: formData, }).then((r) => r.json()), simulate: (robotId, terrainMode, opts = {}) => fetch("/api/simulate", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ robot_id: robotId, terrain_mode: terrainMode, enable_motors: opts.enableMotors || false, record_trajectory: opts.recordTrajectory || false, }), }).then((r) => r.json()), stressTest: (robotId, enableMotors = false) => fetch("/api/stress-test", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ robot_id: robotId, enable_motors: enableMotors }), }).then((r) => r.json()), refine: (prompt, baseId, baseUrdf) => fetch("/api/refine", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prompt, base_id: baseId || undefined, base_urdf: baseUrdf || undefined, }), }).then((r) => r.json()), history: () => fetch("/api/history").then((r) => r.json()), robot: (id) => fetch(`/api/robot/${id}`).then((r) => r.json()), deleteRobot: (id) => fetch(`/api/robot/${id}`, { method: "DELETE" }).then((r) => r.json()), leaderboard: (terrainMode) => { const params = terrainMode ? `?terrain_mode=${terrainMode}` : ""; return fetch(`/api/leaderboard${params}`).then((r) => r.json()); }, submitScore: (robotId, terrainMode) => fetch("/api/leaderboard/submit", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ robot_id: robotId, terrain_mode: terrainMode }), }).then((r) => r.json()), }; // Track uploaded image file let uploadedImageFile = null; let selectedId = null; let selectedUrdf = null; let scene = null; let renderer = null; let camera = null; let animationId = null; // Replay state let replayData = null; // trajectory object from API let replayPlaying = false; let replayFrame = 0; let replayStartTime = 0; let replayAnimId = null; let currentRobot = null; // Three.js robot object for replay function toast(msg, type = "success") { const el = document.getElementById("toast"); el.textContent = msg; el.className = `toast ${type} show`; setTimeout(() => el.classList.remove("show"), 3000); } function setLoading(loading) { const btn = document.getElementById("btn-generate"); const btnRefine = document.getElementById("btn-refine"); btn.disabled = loading; btn.textContent = loading ? "Generating..." : "Generate URDF"; if (loading) { btnRefine.disabled = true; document.getElementById("btn-simulate").disabled = true; document.getElementById("btn-download").disabled = true; document.getElementById("btn-view-source").disabled = true; document.getElementById("btn-submit-score").disabled = true; document.getElementById("btn-replay").disabled = true; document.getElementById("btn-stress-test").disabled = true; document.getElementById("btn-export-mjcf").disabled = true; document.getElementById("btn-export-sdf").disabled = true; } else { updateRefineButton(); updateSimulateButton(); updateDownloadButton(); updateViewSourceButton(); updateSubmitButton(); updateReplayButton(); updateStressTestButton(); updateExportButtons(); } } function updateRefineButton() { const btn = document.getElementById("btn-refine"); btn.disabled = !selectedId; } function updateSimulateButton() { const btn = document.getElementById("btn-simulate"); btn.disabled = !selectedId; } function updateDownloadButton() { document.getElementById("btn-download").disabled = !selectedId; } function updateViewSourceButton() { document.getElementById("btn-view-source").disabled = !selectedId; } function updateSubmitButton() { document.getElementById("btn-submit-score").disabled = !selectedId; } function updateReplayButton() { document.getElementById("btn-replay").disabled = !selectedId; } function updateStressTestButton() { document.getElementById("btn-stress-test").disabled = !selectedId; } function updateExportButtons() { document.getElementById("btn-export-mjcf").disabled = !selectedId; document.getElementById("btn-export-sdf").disabled = !selectedId; } function updateRobotCount(count) { const badge = document.getElementById("robot-count-badge"); if (badge) { badge.textContent = `${count} robot${count !== 1 ? "s" : ""}`; } } function renderHistory(history) { updateRobotCount(history.length); const ul = document.getElementById("history-list"); ul.innerHTML = history .map( (e) => `
  • str: with open(SYSTEM_PROMPT_PATH) as f: return f.read().strip() def extract_urdf_from_response(text: str) -> str: """ Day 2 Fix: Strip conversational text and markdown. Extract only the XML between . """ # Find match = re.search(r"<\?xml[\s\S]*?", text, re.IGNORECASE | re.DOTALL) if match: logger.debug("Extracted URDF XML (%d chars) from LLM response", len(match.group(0))) return match.group(0).strip() logger.warning("No block found in LLM response; returning raw text") return text.strip() def generate_robot(prompt: str, output_path: Path | None = None, use_rag: bool = True) -> str: """ Generate URDF from natural language using OpenAI. When use_rag=True, retrieves relevant URDF snippets to augment the prompt. Returns the raw URDF string. """ logger.info("Generating URDF for prompt: %s", prompt[:80]) client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY")) system = load_system_prompt() # RAG: retrieve relevant snippets and augment the prompt augmented_prompt = prompt if use_rag: try: from src.rag import build_rag_context rag_context = build_rag_context(prompt, top_k=2) if rag_context: augmented_prompt = f"{rag_context}\n\nUser request: {prompt}" logger.info("RAG: Augmented prompt with retrieved snippets") except Exception as e: logger.warning("RAG retrieval failed (continuing without): %s", e) response = client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": system}, {"role": "user", "content": augmented_prompt}, ], ) raw = response.choices[0].message.content urdf = extract_urdf_from_response(raw) logger.info("URDF generated successfully (%d chars)", len(urdf)) return urdf def validate_urdf(urdf_str: str) -> tuple[bool, str]: """Validate URDF. Returns (valid, error_msg).""" return validate_urdf_parse(urdf_str) def main(): import sys prompt = sys.argv[1] if len(sys.argv) > 1 else "A box with 4 wheels" urdf = generate_robot(prompt) valid, err = validate_urdf(urdf) if valid: OUTPUT_DIR.mkdir(exist_ok=True) out = OUTPUT_DIR / "robot.urdf" out.write_text(urdf) print(f"Saved to {out}") else: print(f"Validation failed: {err}") print("Raw URDF:\n", urdf[:500]) if __name__ == "__main__": main() ``` ### src/validate.py ```python """ URDF validation: urdfpy parse + custom checks (bounding box, effort limits). """ import re from pathlib import Path from urdfpy import URDF def validate_urdf_parse(urdf_str: str) -> tuple[bool, str]: """Parse URDF with urdfpy. Returns (valid, error_msg).""" try: URDF.from_xml_string(urdf_str) return True, "" except Exception as e: return False, str(e) def get_chassis_size(urdf_str: str) -> float: """ Estimate chassis bounding box size from base link geometry. Returns approximate half-extent (radius) in meters. """ robot = URDF.from_xml_string(urdf_str) for link in robot.links: colls = link.collisions if hasattr(link, "collisions") else ([link.collision] if link.collision else []) for coll in colls: if coll is not None and coll.geometry is not None: geom = coll.geometry if hasattr(geom, "size"): s = geom.size return max(s) / 2.0 if hasattr(s, "__len__") else s / 2.0 if hasattr(geom, "radius"): return geom.radius return 0.5 # default def check_link_positions(urdf_str: str, min_offset: float = 0.5) -> tuple[bool, str]: """ Day 6 Fix: Ensure child links (wheels, legs) are not at same position as parent. Returns (valid, error_msg). """ robot = URDF.from_xml_string(urdf_str) chassis_size = get_chassis_size(urdf_str) threshold = max(min_offset, chassis_size) for joint in robot.joints: if joint.parent == joint.child: continue origin = joint.origin if origin is None: continue x, y, z = origin[0, 3], origin[1, 3], origin[2, 3] dist = (x**2 + y**2 + z**2) ** 0.5 if dist < 0.01: # effectively (0,0,0) return False, ( f"Link '{joint.child}' is at same position as parent. " f"Offset must be > {threshold:.2f}m to avoid self-collision." ) return True, "" def check_effort_limits(urdf_str: str, min_effort: float = 100.0) -> tuple[bool, str]: """ Day 8 Fix: Reject URDF if joint effort is too weak (floppy noodle). """ effort_pattern = re.compile(r']*effort\s*=\s*["\']([^"\']+)["\']', re.I) for m in effort_pattern.finditer(urdf_str): try: effort = float(m.group(1)) if effort < min_effort: return False, ( f"Joint effort {effort} is too weak. " f"Use at least {min_effort} for heavy robots." ) except ValueError: pass return True, "" def check_mass_values(urdf_str: str, min_mass: float = 0.01, max_mass: float = 500.0) -> tuple[bool, str]: """ Validate that all link masses are within physically reasonable bounds. Reject zero/negative masses (PyBullet treats mass=0 as static/immovable) and absurdly heavy links that cause simulation instability. """ mass_pattern = re.compile(r' 0 for physics simulation." ) if mass < min_mass: return False, ( f"Link mass {mass}kg is unrealistically light (min {min_mass}kg). " "Increase mass for simulation stability." ) if mass > max_mass: return False, ( f"Link mass {mass}kg exceeds maximum ({max_mass}kg). " "Use realistic masses for a robot." ) except ValueError: return False, f"Non-numeric mass value: {m.group(1)}" return True, "" def validate_all(urdf_str: str) -> tuple[bool, str]: """Run all validations. Returns (valid, error_msg).""" valid, err = validate_urdf_parse(urdf_str) if not valid: return False, f"Parse error: {err}" valid, err = check_link_positions(urdf_str) if not valid: return False, err valid, err = check_effort_limits(urdf_str) if not valid: return False, err valid, err = check_mass_values(urdf_str) if not valid: return False, err return True, "" ``` [13 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]