Project Info

LLM-Generated Robot URDF

Inspiration

Robotics simulation is powerful, but creating robot models is tedious. URDF (Unified Robot Description Format) requires precise XML: links, joints, inertials, collision geometries. A single typo or misplaced origin can break everything. We wondered: what if you could describe a robot in plain English and get a working, simulated URDF? We were inspired by the gap between natural language (how humans think about robots) and formal representations (how simulators need them). LLMs excel at structured output—why not bridge that gap for robotics?

What it does

The project turns natural language into simulated robot URDFs. You type "A 4-legged dog robot" or "A box with 4 wheels"—and get a valid, physics-tested URDF in seconds. Core features: Natural language generation — Describe any robot; the LLM produces URDF XML Image-to-URDF — Upload a sketch, diagram, or photo; GPT-4o vision analyzes it and generates a matching URDF RAG-augmented generation — Retrieves relevant URDF snippets from a library (quadrupeds, hexapods, wheeled bases) to improve output quality Validation pipeline — Parse checks (urdfpy), link position checks (no overlapping geometry), effort limits (no floppy robots) Physics simulation — PyBullet runs a 5-second sim; detects explosions, fall-over, self-collisions Error feedback loop — When validation or simulation fails, the error is fed back to the LLM for automatic retry (up to 5 attempts) Iterative refinement — "Make it heavier," "add another wheel," "shorter legs"—modify existing robots with follow-up prompts Multi-terrain stress testing — Flat, uneven, stairs, slope; score robots across all terrains Export — URDF → MJCF (MuJoCo) and SDF (Gazebo) conversion Web UI — 3D preview (Three.js + urdf-loader), history, leaderboard, feedback suggestions

How we built it

Architecture: A three-stage pipeline—Generate → Validate → Simulate—with an orchestrator agent that retries on failure. Generation — OpenAI GPT-4o-mini with a system prompt that enforces URDF rules. For multi-legged robots, we inject chain-of-thought: the LLM first computes angles ( \theta_i = \frac{360°}{n} \cdot i ) and mount positions ( (x, y) = (r \cos\theta, r \sin\theta) ) before writing XML, avoiding legs at ((0,0,0)). Generation — OpenAI GPT-4o-mini with a system prompt that enforces URDF rules. For multi-legged robots, we inject chain-of-thought: the LLM first computes angles ( \theta_i = \frac{360°}{n} \cdot i ) and mount positions ( (x, y) = (r \cos\theta, r \sin\theta) ) before writing XML, avoiding legs at ((0,0,0)). RAG — TF-IDF over a corpus of URDF snippets (quadruped, hexapod, wheeled base, etc.). Query tokens are matched; top-k snippets are injected into the prompt as examples. RAG — TF-IDF over a corpus of URDF snippets (quadruped, hexapod, wheeled base, etc.). Query tokens are matched; top-k snippets are injected into the prompt as examples. Validation — urdfpy for parse correctness; custom checks for link offsets (( | \text{origin} | > 0.01 ) m) and joint effort (( \geq 100 ) N·m). Validation — urdfpy for parse correctness; custom checks for link offsets (( | \text{origin} | > 0.01 ) m) and joint effort (( \geq 100 ) N·m). Simulation — PyBullet headless mode. Terrain loaders for flat, uneven (heightfield), stairs, slope. Physics sanity check (0.5 s) catches explosions and self-collisions before full 5 s run. Simulation — PyBullet headless mode. Terrain loaders for flat, uneven (heightfield), stairs, slope. Physics sanity check (0.5 s) catches explosions and self-collisions before full 5 s run. Scoring — Composite score from stability (displacement), uprightness (tilt cosine), and grounding (height). Terrain multipliers: flat 1.0×, slope 1.15×, stairs 1.25×, uneven 1.30×. Final score: [ S = \min\left(100,\; \left(0.4 S_{\text{stab}} + 0.35 S_{\text{upright}} + 0.25 S_{\text{ground}}\right) \cdot m_{\text{terrain}}\right) ] Scoring — Composite score from stability (displacement), uprightness (tilt cosine), and grounding (height). Terrain multipliers: flat 1.0×, slope 1.15×, stairs 1.25×, uneven 1.30×. Final score: [ S = \min\left(100,\; \left(0.4 S_{\text{stab}} + 0.35 S_{\text{upright}} + 0.25 S_{\text{ground}}\right) \cdot m_{\text{terrain}}\right) ] Web stack — Flask backend, vanilla JS frontend, Three.js + urdf-loader for 3D preview. History and leaderboard persisted to JSON. Web stack — Flask backend, vanilla JS frontend, Three.js + urdf-loader for 3D preview. History and leaderboard persisted to JSON.

Challenges we ran into

Leg overlap — Early multi-legged robots had all legs at ((0,0,0)); PyBullet exploded. We added chain-of-thought prompting so the LLM computes angles and positions first. Leg overlap — Early multi-legged robots had all legs at ((0,0,0)); PyBullet exploded. We added chain-of-thought prompting so the LLM computes angles and positions first. Floppy robots — Weak joint effort caused limbs to collapse. We added effort validation (min 100) and mass validation (0.01–500 kg). Floppy robots — Weak joint effort caused limbs to collapse. We added effort validation (min 100) and mass validation (0.01–500 kg). Self-collisions — Links touching at spawn caused instability. We added a sanity check that detects non-adjacent link contacts and feeds that back to the LLM. Self-collisions — Links touching at spawn caused instability. We added a sanity check that detects non-adjacent link contacts and feeds that back to the LLM. URDF extraction — LLMs sometimes wrap XML in markdown or add commentary. We use regex to extract <?xml ... </robot> and strip the rest. URDF extraction — LLMs sometimes wrap XML in markdown or add commentary. We use regex to extract <?xml ... </robot> and strip the rest. PyBullet on ARM Mac — Some users needed brew install cmake for PyBullet to build. We made simulation optional so generation and validation still work without it. PyBullet on ARM Mac — Some users needed brew install cmake for PyBullet to build. We made simulation optional so generation and validation still work without it.

Accomplishments we're proud of

End-to-end pipeline — From "A 4-legged dog" to a simulated, scored robot in one flow Self-healing — Error feedback loop means the system often fixes its own mistakes without human intervention Image-to-URDF — Two-stage pipeline (analyze → generate) with RAG for sketch/diagram input Multi-format export — URDF, MJCF, SDF from a single description Stress testing — Robots tested on four terrains; leaderboard with terrain-filtered rankings Feedback suggestions — UI suggests refinements ("Robot is unstable — widen the base") that users can one-click apply

What we learned

Structured prompting matters — Chain-of-thought for geometry (angles, positions) dramatically improved multi-legged robot quality Validation layers compound — Parse → physics sanity → full sim catches different failure modes RAG helps — Even a small snippet library (10–15 URDFs) improved generation for similar robot types Vision + text — GPT-4o vision can interpret sketches and diagrams; combining that with the text pipeline opened image-to-URDF

What's next

for TreeHackNow Mesh support — Generate or reference STL/OBJ meshes for more realistic geometry Trajectory optimization — Use simulation feedback to tune joint parameters (PD gains, limits) automatically Multi-robot scenarios — Generate and simulate multiple robots interacting ROS 2 integration — Export to ROS 2 packages with launch files and config Community snippet library — Allow users to contribute URDF snippets to the RAG index Fine-tuned model — Train a small model on URDF examples for faster, cheaper generation

RoboWhisper — LLM-Generated Robot URDF

Generate robot URDF files from natural language using an LLM, validate with urdfpy, and simulate in PyBullet.

Architecture & Design

High-Level Overview

RoboWhisper Pipeline

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              RoboWhisper Pipeline                                │
└─────────────────────────────────────────────────────────────────────────────────┘

  User Prompt                    ┌──────────────┐
  "A 4-legged dog"    ──────────►│   generate   │──────────► Raw URDF XML
                                 │   (LLM)      │
                                 └──────┬───────┘
                                        │
                                        ▼
                                 ┌──────────────┐
                                 │   validate   │──────────► Parse + custom checks
                                 │  (urdfpy)    │
                                 └──────┬───────┘
                                        │
                                        ▼
                                 ┌──────────────┐
                                 │   simulate   │──────────► PyBullet physics
                                 │  (PyBullet)  │
                                 └──────┬───────┘
                                        │
                                        ▼
                                 output/robot.urdf

Data Flow

Natural Language ──► LLM (GPT-4o-mini) ──► URDF XML ──► Validation ──► Simulation ──► Saved URDF
                         │                      │              │
                         │                      │              └── On failure: error feedback → retry
                         │                      └── On failure: error feedback → retry
                         └── System prompt + optional chain-of-thought (multi-legged)

Component Architecture

Module	Responsibility	Key Functions
`src/generate.py`	LLM-based URDF generation	`generate_robot()`, `extract_urdf_from_response()`
`src/validate.py`	URDF correctness & physics sanity	`validate_all()`, `validate_urdf_parse()`, `check_link_positions()`, `check_effort_limits()`
`src/simulate.py`	Physics simulation in PyBullet	`simulate_urdf()` — loads URDF, terrain modes (flat/uneven/stairs/slope), 5s sim
`src/agent.py`	Orchestrator with retry loop	`run_agent()` — Generate → Validate → Simulate, up to 5 retries with error feedback

Design Decisions

Separation of concerns — Generation, validation, and simulation are independent modules. Each can be run standalone (python -m src.generate, python -m src.simulate) or composed by the agent.
Error feedback loop — When validation or simulation fails, the error message is fed back into the LLM prompt so it can fix the URDF. Max 5 retries prevents infinite loops.
Multi-legged chain-of-thought — For prompts like "4-legged dog" or "hexapod", the agent injects a structured prompt that instructs the LLM to: (a) compute angles for each leg, (b) compute (x,y) mount positions from body radius, (c) then generate XML. This avoids legs overlapping at (0,0,0).
Validation layers:
- Parse — urdfpy ensures valid URDF syntax and structure.
- Link positions — Child links (wheels, legs) must be offset from parent to avoid self-collision.
- Effort limits — Joint effort ≥ 100 to prevent "floppy noodle" robots.
Simulation stability check — If the robot base moves >50m from origin during the 5s sim, it's considered "exploded" (unstable).

File Structure

RoboWhisper/
├── src/
│   ├── agent.py         # Orchestrator: retry loop + error feedback
│   ├── generate.py      # LLM → URDF (OpenAI API)
│   ├── simulate.py      # PyBullet physics (headless or GUI)
│   └── validate.py      # urdfpy + custom checks
├── web/
│   ├── app.py           # Flask server — /api/generate, /api/refine, /api/simulate
│   ├── templates/       # index.html
│   └── static/          # app.js, style.css — 3D preview (Three.js + urdf-loader)
├── prompts/
│   └── system_prompt.txt   # LLM system instructions
├── output/                  # Generated URDFs (agent_test.urdf, robot.urdf)
├── package.json         # npm run web — start localhost server
├── requirements.txt
└── environment.yml

External Dependencies

Dependency	Purpose
OpenAI	LLM API for natural language → URDF generation
urdfpy	Parse and validate URDF XML
PyBullet	Physics simulation (gravity, collision, stability)

Setup

Conda (recommended):

conda env create -f environment.yml
conda activate robowhisper

Or pip only:

pip install -r requirements.txt

Note: PyBullet may require building from source on some systems. If simulation fails, generation and validation still work. On macOS ARM, you may need brew install cmake first.

Set your OpenAI API key:

export OPENAI_API_KEY="your-key-here"

Usage

Web UI (recommended)

npm run web

Then open http://localhost:5000 in your browser. You get:

Generate — describe a robot (e.g. "A 4-legged dog"), get URDF + 3D preview
Refine — select a robot, type a change (e.g. "make it heavier"), get updated URDF
Simulate — run PyBullet physics on flat/uneven/stairs/slope terrain

CLI

# Generate a robot (simple)
python -m src.generate "A box with 4 wheels"

# Simulate a URDF (with optional terrain mode)
python -m src.simulate output/robot.urdf
python -m src.simulate output/robot.urdf --terrain uneven   # uneven, stairs, slope
# Terrain modes: flat (default), uneven, stairs, slope — test robustness

# Full agent loop (validate + simulate + retry on failure)
python -m src.agent "A 4-legged dog robot"
python -m src.agent "A 4-legged dog" --terrain slope   # optional terrain for sim

# Web UI (same as above)
npm run web

Plan

See PLAN.md for the full implementation roadmap. Phase 5 covers web UI enhancements: Download URDF, View source, Delete from history, Prompt examples, History persistence, Simulation metrics.

Analysis

Compare with all teams

View

Metric

Kunal Aneja
18
Cursor
15

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CSSIn code
FlaskIn code
HTMLIn code
JavaScriptIn code
OpenAIIn code
PythonIn code

6 of 6 appear in the indexed code.

AI coding agents

CursorCommits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

211 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.