Project Info

FailureCloud

Inspiration

Robots and self-driving systems do not usually fail in clean demos. They fail in rare edge cases: slippery floors, dropped objects, occluded humans, reflective surfaces, bad lighting, sensor noise, and unexpected physical interactions. Collecting these failures in the real world is expensive, slow, dangerous, and hard to reproduce. That inspired us to build FailureCloud: unit tests for robots. Software engineers have unit tests. We built FailureCloud to give robotics and physical AI teams unit tests for robots. The problem The future of AI-like robots and self-driving cars requires lots and lots of data. Current datasets are extensive, but they cover common senarios, like driving in a sunny or cold day. They overrepresent positive cases and underrepresent edge cases and failure cases, where things could actually go wrong. Conviction is furthered by Waymo beginning similar research, seen here Robotics teams need to know: Will the robot still work on a slippery floor? Can it avoid an unseen obstacle? Can it carry something fragile without spilling? Can its sensors handle noisy or incomplete data? Can the same failure be reproduced again and again? Today, many of these cases are manually built in simulation or discovered through risky real-world * testing. Even after data is collected, labels, rewards, and success conditions are still painful to define.

What it does

FailureCloud turns a natural-language robot task into an executable edge-case test. A user describes a task, selects one of five generated failure scenarios, edits its parameters, previews it, runs it in PyBullet, inspects synchronized sensor evidence, and exports the result. Each test includes: Environment, robot, route, and hazards RGB, depth, segmentation, and LiDAR data Object labels and telemetry Reward and success criteria A measurable pass/fail explanation Simulator and dataset exports Users can also upload custom URDF robot packages. Architecture FailureCloud is built around a versioned, simulator-independent scenario contract: Every layer consumes this contract. PyBullet is the current execution adapter, but FailureCloud is designed to export the same scenario to simulators such as Isaac Sim, Gazebo, and CARLA. Data flow and exports PyBullet records RGB, metric depth, segmentation, ray-cast LiDAR, object poses, collisions, cup stability, water retention, and reward telemetry. These results are stored in a canonical bundle containing the scenario, frame manifest, sensor data, labels, calibration, evaluation, and robot assets. Export adapters convert this bundle into: PyBullet replay packages OpenPCDet datasets ROS-style sensor folders Isaac Sim configuration previews Nebius job manifests Because canonical data is separated from export formats, new integrations can be added without rerunning the simulation.

How we built it

The frontend uses Next.js, React, TypeScript, Three.js, React Three Fiber, SWR, and Playwright. The backend uses FastAPI, Pydantic, PyBullet, NumPy, Pillow, and Claude. Claude generates edge-case ideas, while deterministic templates keep the workflow reliable when external services are unavailable. Best Physical AI Hack by UFB TL;DR: FailureCloud creates unit tests for robots. It fits Physical AI because it directly generates, simulates, evaluates, and exports the data and worlds that train or test robotic systems. Given a natural-language task like “a warehouse robot carries a cup of water,” FailureCloud generates rare edge-case scenarios such as slippery floors, dropped obstacles, noisy sensors, and sudden turns. For each scenario, we generate a runnable simulation, RGB/depth/LiDAR-style sensor data, object labels, reward metrics, and a pass/fail report. The goal is simple: help robotics teams test failures before they happen in the real world. A real robotics team would use this to quickly create repeatable edge-case tests, debug robot failures, evaluate policies, and export reusable test bundles for tools like PyBullet, ROS, Isaac Sim, OpenPCDet, or future physical AI workbenches. Best Use of Claude TL;DR: Claude acts as the scenario compiler for robot unit tests. We use Claude to turn vague human intent into structured physical AI test cases. A user can describe a normal robot task, and Claude expands it into realistic edge cases, simulation parameters, sensor configurations, reward logic, and definitions of success or failure. For example, from “a robot carries a cup,” Claude can infer failure modes like slippery floors, sharp turns, spills, collisions, and sensor noise. It then converts those into an executable scenario with measurable success criteria, such as reaching the goal with at least 70% of the water remaining and no collisions. Claude is essential because it bridges natural language and robotics simulation. It lets non-experts describe what they want to test, while FailureCloud turns that into a runnable physical AI evaluation. Challenges and accomplishments The main challenge was avoiding simulator lock-in. We solved this by separating scenario intent, simulation execution, recorded evidence, and export delivery. We built: A complete seven-step testing workflow A versioned scenario contract Editable reward and success logic Synchronized multimodal sensor recording Automatic labels and failure explanations Validated custom URDF uploads Multiple simulator and dataset export formats

FailureCloud

FailureCloud turns natural-language robot failure cases into executable simulation tests with sensor data, labels, rewards, and exportable artifacts.

The hackathon MVP currently supports:

Prompt-to-scenario compilation using Claude or a deterministic fallback.
Editable, versioned scenario.json.
Interactive Three.js warehouse preview.
Deterministic PyBullet execution.
RGB, metric depth, segmentation, LiDAR, object labels, and telemetry.
Water-retention, collision, reward, and pass/fail evaluation.
OpenPCDet, ROS-style, PyBullet, Isaac-config, and Nebius-manifest exports.
Reactor Helios live cinematic preview with a local SVG fallback.
Nebius-compatible parameter sweeps with a local execution fallback.

Local setup

Requirements:

Node.js 22+
Python 3.12+
Docker, optionally

Install:

cp .env.example .env
make install

Start the backend:

make api

Start the frontend in another terminal:

make web

Open http://localhost:3000.

Environment

Claude is optional. Without a key, scenario compilation uses a deterministic template.

ANTHROPIC_API_KEY=
ANTHROPIC_MODEL=claude-sonnet-4-5

Reactor is optional. When configured, the backend exchanges the API key for a short-lived browser token and the frontend connects to Helios over WebRTC.

REACTOR_API_URL=https://api.reactor.inc
REACTOR_API_KEY=

Nebius currently requires an authorized service account, the official CLI, and a published worker image:

NEBIUS_PROJECT_ID=
NEBIUS_REGION=eu-north1
NEBIUS_EXECUTION_ENDPOINT=api.nebius.cloud:443
NEBIUS_SERVICE_ACCOUNT_KEY_FILE=
NEBIUS_CLI_PATH=nebius
NEBIUS_PROFILE=failurecloud
NEBIUS_JOB_IMAGE=

Never commit .env or service-account credential files.

Verification

.venv/bin/pytest -q services/api/tests
npm run build
npm --workspace apps/web run test:e2e

The browser test executes:

Prompt compilation
Scenario validation
Three.js preview
PyBullet simulation
Failure report
Local parameter sweep and heatmap

API

Primary endpoints:

POST /v1/scenarios/compile
POST /v1/scenarios/validate
POST /v1/runs
GET /v1/runs/{run_id}
GET /v1/runs/{run_id}/events
POST /v1/runs/{run_id}/exports
GET /v1/runs/{run_id}/bundle
POST /v1/previews/reactor
POST /v1/integrations/reactor/token
POST /v1/runs/{run_id}/sweeps/nebius
GET /health/integrations

Generated data is written to artifacts/runs/<run_id>/.

Sponsor integrations

Reactor is used only for an illustrative cinematic preview. PyBullet remains authoritative for physics, labels, and evaluation.

Nebius is intended to execute the parameter-sweep worker at scale. The worker entry point is:

python -m app.sweep_worker \
  --scenario /work/input/scenario.json \
  --specification /work/input/sweep.json \
  --output /work/output/results.json

Until project IAM and a worker image are configured, the same normalized sweep runs locally.

Demo

Use the default warehouse prompt. Compile the scenario, inspect the world, run the test, show the water-retention failure, open the cloud-sweep heatmap, select a difficult variant, and download the complete test bundle.

Analysis

Compare with all teams

View

Metric

elijahzhao24
15

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

AnthropicIn code
CSSIn code
FastAPIIn code
HTMLIn code
Next.jsIn code
PythonIn code
ReactIn code
TypeScriptIn code
RedisClaimed

8 of 9 appear in the indexed code. 1 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

452 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Project Info

FailureCloud

Inspiration

What it does

How we built it

FailureCloud

Local setup

Environment

Verification

API

Sponsor integrations

Demo

Analysis

Technology

AI coding agents

Codebase size

Structure

Interface

API & routing

Application logic

Data & schema

Languages

Dependencies