# Project export: Ember This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: hw-accelerated autonomous firefighting humanoids that you can prompt - Devpost: https://devpost.com/software/ember-3jhgxo - GitHub: https://github.com/conjeevaram/ember - Video: https://www.youtube.com/embed/biA6SK4SClM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — conjeevaram (7 commits), Dhyey Bhatt (5 commits), Claude Sonnet 4.6 (4 commits), Cursor (2 commits) ## Devpost submission (written by the team) ### Inspiration Fire-fighting is exactly the kind of dangerous, time-critical job we'd want a robot to take on — but a humanoid that can walk into a hazard is only useful if it can perceive that hazard cheaply and continuously. We kept hitting the same tension: every always-on sensor and computation on a robot competes for power, money, and cooling. Running a full neural network around the clock just to watch for fire is expensive overkill. So Ember became two halves solving one problem: a fire-fighting humanoid trained in simulation, and a custom FPGA perception accelerator that does the constant, low-power watching — a cheap always-on "first line of detection" that only wakes heavier compute when there's a real reason to. ### What it does Ember is a fire-fighting humanoid with a hardware-accelerated fire-detection front end. The perception layer runs on an FPGA. As camera pixels stream in, it flags fire-colored pixels, filters out false positives by checking whether each flagged pixel is surrounded by other fire pixels (a real fire is a solid region, not scattered specks), locates the fire, and targets the base of the flame — where it would actually be fought. It outputs just a coordinate and a fire/no-fire flag per frame, not a whole image, over UART. The humanoid layer is a Unitree humanoid trained in MuJoCo. Locomotion and approach behaviors are driven by reinforcement-learned policies (PPO), with A* for higher-level path planning toward the detected fire. The detected fire location from the FPGA is the target the humanoid stack acts on. A home-base server ties it together, reading the FPGA's output over WiFi and visualizing detections live — and keeping the architecture open for multi-robot coordination and on-demand heavier AI. ### How we built it Perception (FPGA, Verilog): a streaming pipeline on a Xilinx Zynq — raster scanner → YCbCr color threshold → morphological erosion using two line buffers and a 3×3 sliding window (so all nine neighbors are available in a single clock cycle) → an accumulator that computes the centroid and flame-base aim point → a UART transmitter sending a compact binary packet per frame. We kept division off the fabric by deferring it downstream, and verified the hardware against a pixel-for-pixel Python golden model before programming the board. Humanoid (MuJoCo + PPO + A*): we set up the Unitree humanoid in MuJoCo and trained locomotion/approach policies with PPO, using A* for planning toward a target. Getting stable, useful behavior took real fine-tuning of the training policies — reward shaping, tuning to keep the humanoid balanced while moving toward a goal rather than collapsing or learning degenerate gaits, and adjusting so the learned policy responded sensibly to an externally supplied target coordinate. Software glue (Python, Flask): an image-to-memory converter, a golden-reference verifier, a live serial reader, a Flask web dashboard (phone-viewable over WiFi), and a video annotator that runs the exact FPGA algorithm frame-by-frame on real footage. We used Cursor and Claude through the build. ### Challenges we ran into The hardest problems were at the integration seams between two very different systems: Bridging perception to the policy. The FPGA emits a raw coordinate over a serial link; the humanoid policy expects a target in its own frame. Getting that hand-off — serial packet → server → a target the trained policy could actually act on — was a real integration effort, and we ran out of time to fully close the loop into the live sim, so we route through the home-base server as the connecting layer. Tuning the humanoid policies. PPO didn't just work out of the box — balancing locomotion stability against goal-seeking took repeated reward and hyperparameter tuning, and the policy had to stay robust when handed a target it hadn't seen during training. FPGA pipeline alignment. BRAM read latency and the line-buffer window each add delay, so coordinates and frame-boundary signals needed careful re-alignment — we chased a one-pixel coordinate bias down to the exact register stage. Morphology tradeoffs. Erosion removed noise but over-shrank thin fires; we attempted morphological opening, hit a dilation bug, and made the call to ship reliable erosion-only rather than risk a working demo. Bring-up gremlins. A camera/ESP8266 path we ultimately scoped out, stale bitstreams, sim runtimes too short for UART, cached dashboards — the classic "sim is right but the board isn't" debugging across both hardware and the sim toolchain. ### Accomplishments we're proud of A complete fire-detection pipeline running on real silicon, verified end-to-end: image → color detection → real-time neighborhood filtering → localization → UART → live dashboard. Real-time morphological filtering with line buffers — the piece that genuinely justifies "why an FPGA," doing a neighborhood operation at one result per clock that a CPU can't sustain at frame rate. A trained humanoid that learned to locomote and move toward a goal in MuJoCo, with policies tuned to stay stable. Bringing two hard, separate systems — custom hardware perception and a learned humanoid controller — into one coherent fire-fighting story. Knowing when to scope down to protect a working demo. ### What we learned Why FPGAs win for streaming, per-pixel work: dedicated hardware per stage, neighbors on wires instead of fetched from memory, deterministic latency with no cache jitter — and that much of FPGA design is timing alignment, not the logic itself. How brittle RL policies can be, and how much reward shaping and tuning it takes to get stable, goal-directed humanoid behavior in MuJoCo. That the real work in a multi-part robot is the integration between subsystems, not just each subsystem alone. The value of a software golden model for trusting hardware output. The strongest framing isn't "FPGA beats GPU" — it's a tiered system where a cheap, always-on FPGA gate guards expensive compute that runs only on demand. ### What's next Close the loop fully: FPGA detection → policy target → humanoid response, live and end-to-end. Live camera input via a parallel camera module straight into the FPGA fabric. Richer perception on the same pipeline: morphological opening to preserve thin fires, Sobel texture analysis to reject flat orange surfaces like sunsets, and temporal flicker detection (fire pulses at a few hertz; steady light doesn't) — each drops into the existing line-buffer foundation, as does thermal imaging. More robust policies: further PPO tuning, domain randomization for sim-to-real, and training the humanoid on actual suppression behaviors rather than just approach. The home-base server as an opening for multi-robot coordination and an on-demand heavier confirmer (e.g. a YOLO-style model) that the FPGA gate wakes only when it flags something. ## README (from the GitHub repository) # ember Unitree G1 firefighting-humanoid demo in MuJoCo, rendered headlessly (EGL) and streamed to a browser. - **`ember.sim`** — Unitree's pretrained **12-DOF** RL walker (robust `(vx, vy, yaw)` velocity control + obstacle traversal) with a kinematic full-body arm overlay that carries a fire-hose nozzle, plus the water-jet ballistics and A*/approach navigation hooks. - **`ember.viewer`** — discovers the named + procedural scenes, hot-swaps them, and streams the sim to a browser (driving, nav map, autonomous approach). Port **8088**. ## Install ```bash pip install -e . ``` ## External assets (not vendored) Cloned separately; paths are env-overridable (defaults in parentheses): | Var | Default | Holds | | --- | --- | --- | | `UNITREE_RL_GYM` | `~/unitree_rl_gym` | G1 model, 12-DOF policy, deploy config | Stream/camera knobs: `EMBER_W`/`EMBER_H` (640x360), `EMBER_FPS` (18), `EMBER_QUALITY` (55), `EMBER_HOST`, `EMBER_CAM_W`/`EMBER_CAM_H`/`EMBER_CAM_FOVY`. ## Run ```bash python scripts/run_walker.py --scene fire # 12-DOF walker -> :8088 python scripts/build_scenes.py --force # (re)generate scene XMLs ``` Browser controls: **WASD** drive, **Q/E** strafe, **Space** stop, **H** auto-steer; plus a robot POV inset. The arms are locked in a hose-carry pose. ## Programmatic control `viewer.start` returns the live `G1Sim`, which exposes the control API directly: ```python from ember import viewer sim = viewer.start(block=False, scene="fire") # flat ground + flame at 4 m sim.set_command(vx=0.5, yaw=0.2) # clamped (vx, vy, yaw) sim.get_state() # pose, velocity, fell flag, fires ``` For a procedural scene, pass a `SceneSpec` (`viewer.start(spec=...)`); the robot spawns at `spec.start` and `sim.set_approach(True)` autonomously navigates to the nearest burning fire and faces it. ## Detected evidence (automated analysis) Indexed codebase: 60 recognized source files, 406 KB. - C (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 251) ``` .gitignore FPGA/Fire_Detection_Accelerator.cache/sim/ssm.db FPGA/Fire_Detection_Accelerator.cache/wt/project.wpc FPGA/Fire_Detection_Accelerator.cache/wt/synthesis_details.wdf FPGA/Fire_Detection_Accelerator.cache/wt/synthesis.wdf FPGA/Fire_Detection_Accelerator.cache/wt/webtalk_pa.xml FPGA/Fire_Detection_Accelerator.cache/wt/xsim.wdf FPGA/Fire_Detection_Accelerator.hw/Fire_Detection_Accelerator.lpr FPGA/Fire_Detection_Accelerator.hw/hw_1/hw.xml FPGA/Fire_Detection_Accelerator.ip_user_files/mem_init_files/image.mem FPGA/Fire_Detection_Accelerator.ip_user_files/mem_init_files/pjeevy1.mem FPGA/Fire_Detection_Accelerator.ip_user_files/README.txt FPGA/Fire_Detection_Accelerator.runs/.jobs/vrs_config_1.xml FPGA/Fire_Detection_Accelerator.runs/.jobs/vrs_config_2.xml FPGA/Fire_Detection_Accelerator.runs/.jobs/vrs_config_3.xml FPGA/Fire_Detection_Accelerator.runs/impl_1/.init_design.begin.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.init_design.end.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.opt_design.begin.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.opt_design.end.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.phys_opt_design.begin.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.phys_opt_design.end.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.place_design.begin.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.place_design.end.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.route_design.begin.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.route_design.end.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.Vivado_Implementation.queue.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.vivado.begin.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.vivado.end.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.write_bitstream.begin.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/.write_bitstream.end.rst FPGA/Fire_Detection_Accelerator.runs/impl_1/clockInfo.txt FPGA/Fire_Detection_Accelerator.runs/impl_1/gen_run.xml FPGA/Fire_Detection_Accelerator.runs/impl_1/htr.txt FPGA/Fire_Detection_Accelerator.runs/impl_1/init_design.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/ISEWrap.js FPGA/Fire_Detection_Accelerator.runs/impl_1/ISEWrap.sh FPGA/Fire_Detection_Accelerator.runs/impl_1/opt_design.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/phys_opt_design.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/place_design.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/project.wdf FPGA/Fire_Detection_Accelerator.runs/impl_1/route_design.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/rundef.js FPGA/Fire_Detection_Accelerator.runs/impl_1/runme.bat FPGA/Fire_Detection_Accelerator.runs/impl_1/runme.sh FPGA/Fire_Detection_Accelerator.runs/impl_1/top_29468.backup.vdi FPGA/Fire_Detection_Accelerator.runs/impl_1/top_bus_skew_routed.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/top_bus_skew_routed.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top_bus_skew_routed.rpx FPGA/Fire_Detection_Accelerator.runs/impl_1/top_clock_utilization_routed.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top_control_sets_placed.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top_drc_opted.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/top_drc_opted.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top_drc_opted.rpx FPGA/Fire_Detection_Accelerator.runs/impl_1/top_drc_routed.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/top_drc_routed.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top_drc_routed.rpx FPGA/Fire_Detection_Accelerator.runs/impl_1/top_io_placed.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top_methodology_drc_routed.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/top_methodology_drc_routed.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top_methodology_drc_routed.rpx FPGA/Fire_Detection_Accelerator.runs/impl_1/top_opt.dcp FPGA/Fire_Detection_Accelerator.runs/impl_1/top_physopt.dcp FPGA/Fire_Detection_Accelerator.runs/impl_1/top_placed.dcp FPGA/Fire_Detection_Accelerator.runs/impl_1/top_power_routed.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top_power_routed.rpx FPGA/Fire_Detection_Accelerator.runs/impl_1/top_power_summary_routed.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/top_route_status.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/top_route_status.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top_routed.dcp FPGA/Fire_Detection_Accelerator.runs/impl_1/top_timing_summary_routed.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/top_timing_summary_routed.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top_timing_summary_routed.rpx FPGA/Fire_Detection_Accelerator.runs/impl_1/top_utilization_placed.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/top_utilization_placed.rpt FPGA/Fire_Detection_Accelerator.runs/impl_1/top.bit FPGA/Fire_Detection_Accelerator.runs/impl_1/top.tcl FPGA/Fire_Detection_Accelerator.runs/impl_1/top.vdi FPGA/Fire_Detection_Accelerator.runs/impl_1/vivado_29468.backup.jou FPGA/Fire_Detection_Accelerator.runs/impl_1/vivado.jou FPGA/Fire_Detection_Accelerator.runs/impl_1/vivado.pb FPGA/Fire_Detection_Accelerator.runs/impl_1/write_bitstream.pb FPGA/Fire_Detection_Accelerator.runs/synth_1/__synthesis_is_complete__ FPGA/Fire_Detection_Accelerator.runs/synth_1/.Vivado_Synthesis.queue.rst FPGA/Fire_Detection_Accelerator.runs/synth_1/.vivado.begin.rst FPGA/Fire_Detection_Accelerator.runs/synth_1/.vivado.end.rst FPGA/Fire_Detection_Accelerator.runs/synth_1/.Xil/top_propImpl.xdc FPGA/Fire_Detection_Accelerator.runs/synth_1/gen_run.xml FPGA/Fire_Detection_Accelerator.runs/synth_1/htr.txt FPGA/Fire_Detection_Accelerator.runs/synth_1/ISEWrap.js FPGA/Fire_Detection_Accelerator.runs/synth_1/ISEWrap.sh FPGA/Fire_Detection_Accelerator.runs/synth_1/project.wdf FPGA/Fire_Detection_Accelerator.runs/synth_1/rundef.js FPGA/Fire_Detection_Accelerator.runs/synth_1/runme.bat FPGA/Fire_Detection_Accelerator.runs/synth_1/runme.sh FPGA/Fire_Detection_Accelerator.runs/synth_1/top_utilization_synth.pb FPGA/Fire_Detection_Accelerator.runs/synth_1/top_utilization_synth.rpt FPGA/Fire_Detection_Accelerator.runs/synth_1/top.dcp FPGA/Fire_Detection_Accelerator.runs/synth_1/top.tcl FPGA/Fire_Detection_Accelerator.runs/synth_1/top.vds FPGA/Fire_Detection_Accelerator.runs/synth_1/vivado.jou FPGA/Fire_Detection_Accelerator.runs/synth_1/vivado.pb FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/compile.bat FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/elaborate.bat FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/glbl.v FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/image.mem FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/pjeevy1.mem FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/simulate.bat FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/tb_addr_gen_behav.wdb FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/tb_addr_gen.tcl FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/tb_image_rom_behav.wdb FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/tb_image_rom_vlog.prj FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/tb_image_rom.tcl FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/tb_top_behav.wdb FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/tb_top_vlog.prj FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/tb_top.tcl FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/tb_uart_tx_behav.wdb FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/tb_uart_tx.tcl FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/xelab.pb FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/xsim.dir/tb_addr_gen_behav/Compile_Options.txt FPGA/Fire_Detection_Accelerator.sim/sim_1/behav/xsim/xsim.dir/tb_addr_gen_behav/obj/xsim_0.win64.obj [131 more files omitted for size] ``` ### Dependencies - pyproject.toml: flask@>=3.0, google-genai@>=1.0, gymnasium@>=0.29, mujoco@>=3.2, numpy@>=1.24, pillow@>=10.0, pyyaml@>=6.0, stable-baselines3@>=2.3, torch@>=2.2 ### Recent commits (newest first) - demo_scene + autonomy hardening + demo recorders - NL firefighting pipeline: Gemini mission parser + FSM executor + RL spray-aim, fix executor/sim control conflicts, externalize web UI, dead-code cleanup - Add UART TX module, constraints, synthesis/impl runs, and Wifi Data Server - Merge remote-tracking branch 'origin/main' - Update Vivado simulation output files - Add new Verilog modules, testbenches, and image processing scripts - split locomotion into sim+viewer, add A*/approach nav layer, drop dead fire_controller - Add FPGA Fire Detection Accelerator Vivado project - procedural multi-fire scenes: SceneSpec + scenegen, traversable debris, reachability-validated, live scene switcher - fire scene fx: emissive flame, ballistic water jet, extinguish; clean pass - hose nozzle head added, robot body cam added, nub for controller - g1 walking teleoperated with 12 dof and full body with GMT ## Key source files (fetched from GitHub, selected and truncated for size) ### pyproject.toml ``` [build-system] requires = ["setuptools>=68"] build-backend = "setuptools.build_meta" [project] name = "ember" version = "0.1.0" description = "Firefighting humanoid (Unitree G1) locomotion + whole-body demos in MuJoCo, streamed headlessly to a browser." readme = "README.md" requires-python = ">=3.10" dependencies = [ "mujoco>=3.2", "numpy>=1.24", "torch>=2.2", # CPU build is fine; the policies are small MLPs "pyyaml>=6.0", "pillow>=10.0", "flask>=3.0", ] [project.optional-dependencies] train = [ "gymnasium>=0.29", "stable-baselines3>=2.3", ] llm = [ "google-genai>=1.0", ] [project.scripts] ember-walk = "ember.viewer:main" ember-build-scenes = "ember.scenes:main" [tool.setuptools.packages.find] where = ["src"] [tool.setuptools.package-data] ember = ["web/*.html"] ``` ### tests/conftest.py ```python """Put ``src/`` on sys.path so tests run from a checkout without installing.""" import pathlib import sys SRC = pathlib.Path(__file__).resolve().parent.parent / "src" if str(SRC) not in sys.path: sys.path.insert(0, str(SRC)) ``` ### scripts/run_walker.py ```python #!/usr/bin/env python3 """Entry point: 12-DOF G1 walker + kinematic arm overlay (default port 8088). python scripts/run_walker.py --scene obstacles """ import _bootstrap # noqa: F401 (puts src/ on the path) from ember.viewer import main if __name__ == "__main__": main() ``` ### scripts/_bootstrap.py ```python """Put ``src/`` on sys.path so the entry scripts run without installing the package (the demo box runs them straight from a checkout).""" import pathlib import sys SRC = pathlib.Path(__file__).resolve().parent.parent / "src" if str(SRC) not in sys.path: sys.path.insert(0, str(SRC)) ``` ### scripts/rt_probe.py ```python """Measure sim-time vs wall-time ratio (real-time factor) under render load.""" from __future__ import annotations import os, sys, time, threading, pathlib os.environ.setdefault("OMP_NUM_THREADS", "1") os.environ.setdefault("MKL_NUM_THREADS", "1") REPO = pathlib.Path(__file__).resolve().parent.parent sys.path.insert(0, str(REPO / "src")) import torch torch.set_num_threads(1) from ember import scenes from ember.config import G1_MODEL_DIR, RENDER_W, RENDER_H, CAM_W, CAM_H, RENDER_FPS from ember.sim import G1Sim from ember.spec import from_json spec = from_json(str(REPO / "scenes" / "specs" / "demo_scene.json")) scenes.build(spec) n12, n29 = scenes.spec_scene_names(spec.name) sim = G1Sim(scene_path=str(G1_MODEL_DIR / n12), overlay_scene=str(G1_MODEL_DIR / n29), spec=spec) threading.Thread(target=sim.run, daemon=True).start() time.sleep(2.0) sim.enable_wind(True) sim.set_command(0.4, 0.0, 0.0) t0 = time.time(); s0 = sim.get_state()["sim_time"] DUR = 12.0 time.sleep(DUR) wall = time.time() - t0; simt = sim.get_state()["sim_time"] - s0 print(f"main={RENDER_W}x{RENDER_H} cam={CAM_W}x{CAM_H} fps={RENDER_FPS} " f"-> sim {simt:.1f}s / wall {wall:.1f}s = {simt/wall:.2f}x realtime") sim.stop() ``` ### scripts/build_scenes.py ```python #!/usr/bin/env python3 """Entry point: (re)generate demo or procedural scene XMLs. python scripts/build_scenes.py --force python scripts/build_scenes.py --random 8 --seed 0 """ import argparse from pathlib import Path import _bootstrap # noqa: F401 (puts src/ on the path) import numpy as np from ember import scenegen, scenes from ember.config import G1_MODEL_DIR from ember.spec import to_json SPECS_DIR = Path(__file__).resolve().parent.parent / "scenes" / "specs" def _build_random(n: int, seed: int, n_fires: int | None, n_walls: int, terrain: bool, n_debris: int, n_tiers: int) -> None: if not G1_MODEL_DIR.exists(): raise SystemExit(f"model dir not found: {G1_MODEL_DIR}\n" "Set $UNITREE_RL_GYM to your unitree_rl_gym checkout.") SPECS_DIR.mkdir(parents=True, exist_ok=True) written_xml: list[str] = [] written_json: list[str] = [] fire_rng = np.random.default_rng(seed) if n_fires is None else None for i in range(n): s = seed + i scene_fires = n_fires if n_fires is not None else int(fire_rng.integers(1, 6)) spec = scenegen.random_spec(s, n_fires=scene_fires, n_walls=n_walls, terrain=terrain, n_debris=n_debris, n_tiers=n_tiers) p12, p29 = scenes.build(spec) written_xml.extend([p12, p29]) json_path = SPECS_DIR / f"{spec.name}.json" to_json(spec, json_path) written_json.append(str(json_path)) print(f"generated {n} scene spec(s) (seeds {seed}..{seed + n - 1})") print("XML:\n " + "\n ".join(written_xml)) print("JSON:\n " + "\n ".join(written_json)) def main() -> None: p = argparse.ArgumentParser(description="(Re)generate G1 demo or procedural scenes.") p.add_argument("--force", action="store_true", help="overwrite existing named demo scene files") p.add_argument("--random", type=int, metavar="N", help="generate N procedural scenes from SceneSpec") p.add_argument("--seed", type=int, default=0, help="RNG seed for --random (default: 0)") p.add_argument("--n-fires", type=int, default=None, help="fixed fires per random scene (default: 1–5 per scene, seeded)") p.add_argument("--n-walls", type=int, default=4, help="walls per random scene (default: 4)") p.add_argument("--terrain", action="store_true", help="include gentle heightfield terrain") p.add_argument("--n-debris", type=int, default=4, help="traversable debris per random scene (default: 4)") p.add_argument("--tiers", type=int, default=1, metavar="N", help="low ramp+platform tiers among debris (default: 1)") args = p.parse_args() if args.random is not None: _build_random(args.random, args.seed, args.n_fires, args.n_walls, args.terrain, args.n_debris, args.tiers) return if not G1_MODEL_DIR.exists(): raise SystemExit(f"model dir not found: {G1_MODEL_DIR}\n" "Set $UNITREE_RL_GYM to your unitree_rl_gym checkout.") written = scenes.ensure_scenes(force=args.force) if written: print("wrote:\n " + "\n ".join(written)) else: print("all scenes already present (use --force to overwrite)") if __name__ == "__main__": main() ``` ### scripts/record_demo.py ```python """Record an MP4 of the whole Ember web console for the demo mission. Headless-Chrome screen recording of the **entire console page exactly as it is** (third-person stream, robot-cam PiP, telemetry, nav map, mission panel, wind gauge) while it runs "put out 2 fires, then return home" with gusty wind on ``demo_scene``. Point ``--base`` at a viewer dedicated to recording (so it does not load or disturb the live server). Robot-cam frames are captured separately and offline by ``scripts/render_robot_cam.py``. """ from __future__ import annotations import argparse import pathlib import time import httpx REPO = pathlib.Path(__file__).resolve().parent.parent VIDEO_DIR = REPO / "recordings" / "video_raw" def _try(client: httpx.Client, url: str, tries: int = 5) -> None: for _ in range(tries): try: client.get(url) return except Exception: time.sleep(0.5) def record(base: str, prompt: str, max_s: float) -> pathlib.Path: from playwright.sync_api import sync_playwright VIDEO_DIR.mkdir(parents=True, exist_ok=True) size = {"width": 1920, "height": 1080} with sync_playwright() as p: browser = p.chromium.launch(channel="chrome", headless=True, args=["--force-device-scale-factor=1"]) ctx = browser.new_context(viewport=size, record_video_dir=str(VIDEO_DIR), record_video_size=size, device_scale_factor=1) page = ctx.new_page() page.goto(base + "/", wait_until="domcontentloaded") page.wait_for_timeout(3000) # let the MJPEG streams warm up page.click("#windBtn") # enable gusty wind (visible toggle) page.fill("#missionIn", prompt) page.wait_for_timeout(800) page.click("#missionBtn") # run the mission start = time.time() done = False while time.time() - start < max_s: try: st = page.evaluate( "async () => { const r = await fetch('/mission_status'); return await r.json(); }") except Exception: st = None if st and st.get("done"): done = True break page.wait_for_timeout(500) print(f"mission {'completed' if done else 'TIMED OUT'} after {time.time()-start:.1f}s") page.wait_for_timeout(3000) # hold on the final frame path = page.video.path() ctx.close() # flush the video to disk browser.close() return pathlib.Path(path) def main() -> None: ap = argparse.ArgumentParser(description=__doc__) ap.add_argument("--base", default="http://127.0.0.1:8099") ap.add_argument("--prompt", default="put out 2 fires, then return home") ap.add_argument("--max-s", type=float, default=220.0) args = ap.parse_args() # Clean start: stop any mission, reset pose, wind off (so the toggle shows). with httpx.Client(timeout=10.0) as c: _try(c, args.base + "/mission_stop") _try(c, args.base + "/reset") try: if c.get(args.base + "/state").json().get("wind_enabled"): c.get(args.base + "/wind?on=0") except Exception: pass time.sleep(1.0) webm = record(args.base, args.prompt, args.max_s) print(f"raw video: {webm}") if __name__ == "__main__": main() ``` ### scripts/render_robot_cam.py ```python """Offline full-resolution robot ego-camera capture for the demo mission. Runs ``demo_scene`` headless (its own G1Sim physics + render threads, no web server), enables gusty wind, executes the mission "put out 2 fires, then return home" via the real MissionExecutor, and saves every robot-cam frame straight from the ego renderer to ``recordings/robot_cam_frames/frame_*.jpg`` at full resolution. This is deliberately decoupled from the web viewer so it neither loads nor can crash the live server, and the frames carry no MJPEG re-streaming overhead. """ from __future__ import annotations import argparse import os import pathlib import sys import threading import time # Must be set before mujoco / ember.config import. os.environ.setdefault("MUJOCO_GL", "egl") os.environ.setdefault("MUJOCO_EGL_DEVICE_ID", os.environ.get("EMBER_EGL_DEVICE", "0")) os.environ.setdefault("EMBER_CAM_W", "1280") os.environ.setdefault("EMBER_CAM_H", "960") os.environ.setdefault("EMBER_QUALITY", "95") os.environ.setdefault("EMBER_FPS", "30") REPO = pathlib.Path(__file__).resolve().parent.parent sys.path.insert(0, str(REPO / "src")) from ember import scenes # noqa: E402 from ember.config import G1_MODEL_DIR # noqa: E402 from ember.executor import MissionExecutor # noqa: E402 from ember.mission import parse_mission # noqa: E402 from ember.sim import G1Sim # noqa: E402 from ember.spec import from_json # noqa: E402 OUT = REPO / "recordings" / "robot_cam_frames" def main() -> None: ap = argparse.ArgumentParser(description=__doc__) ap.add_argument("--spec", default=str(REPO / "scenes" / "specs" / "demo_scene.json")) ap.add_argument("--prompt", default="put out 2 fires, then return home") ap.add_argument("--fps", type=float, default=15.0, help="frame save rate") ap.add_argument("--max-s", type=float, default=220.0) ap.add_argument("--use-llm", action="store_true") args = ap.parse_args() OUT.mkdir(parents=True, exist_ok=True) for f in OUT.glob("*.jpg"): f.unlink() spec = from_json(args.spec) scenes.build(spec) name_12, name_29 = scenes.spec_scene_names(spec.name) sim = G1Sim(scene_path=str(G1_MODEL_DIR / name_12), overlay_scene=str(G1_MODEL_DIR / name_29), spec=spec) sim.heading_hold = False thread = threading.Thread(target=sim.run, daemon=True, name="cam-sim") thread.start() time.sleep(2.0) # let physics settle + first render if not sim.has_camera: print("ERROR: scene has no ego camera", file=sys.stderr) sys.exit(1) sim.enable_wind(True) # gusty wind mission = parse_mission(args.prompt, spec, use_llm=args.use_llm) print("parsed plan:", [(t.type.value, t.target) for t in mission]) ex = MissionExecutor(sim, mission, tick_hz=20.0) ex.start() idx = 0 min_dt = 1.0 / args.fps last = 0.0 last_jpeg = None start = time.time() done = False while time.time() - start < args.max_s: now = time.time() if now - last >= min_dt: jpeg = sim.cam_frames.get() if jpeg is not None and jpeg is not last_jpeg: (OUT / f"frame_{idx:05d}.jpg").write_bytes(jpeg) idx += 1 last_jpeg = jpeg last = now st = ex.status() if st and st.get("done"): done = True break time.sleep(0.01) # Capture a couple of extra seconds on the final (home) pose. tail_end = time.time() + 2.0 while time.time() < tail_end: jpeg = sim.cam_frames.get() if jpeg is not None and jpeg is not last_jpeg: (OUT / f"frame_{idx:05d}.jpg").write_bytes(jpeg) idx += 1 last_jpeg = jpeg time.sleep(min_dt) sim.stop() print(f"mission {'completed' if done else 'TIMED OUT'} after {time.time()-start:.1f}s") print(f"saved {idx} robot-cam frames -> {OUT}") if __name__ == "__main__": main() ``` ### scripts/ab_spray.py ```python #!/usr/bin/env python3 """A/B: reactive PID vs learned RL spray-correction, calm air vs gusty wind. The water jet has a real time-of-flight, so where it lands is only *observed* after a short delay, while the wind is sensed instantly at the nozzle. A purely reactive controller (proportional on the delayed landing error) must therefore chase stale information and trails the gusts; the RL policy also gets the instant wind reading and learns to feed it forward, anticipating the deflection. python scripts/ab_spray.py python scripts/ab_spray.py --episodes 80 --plot Runs are paired: every controller faces the same per-episode wind/noise realizations, so the comparison is apples-to-apples. """ from __future__ import annotations import argparse import _bootstrap # noqa: F401 (puts src/ on the path) import numpy as np from ember.spray_rl import kinematics as kin from ember.spray_rl.env import SprayEnv from ember.spray_rl.policy import DEFAULT_CHECKPOINT, PIDSprayController, SprayPolicy def rollout(controller, *, wind: bool, episodes: int, seed: int, sigma: float, tau: float, trace: bool = False): """Paired rollout: returns (on_target_rate, mean_err, p90_err, trace). ``trace`` (steady-state landing error over one episode) is captured for the last episode when requested, for plotting.""" env = SprayEnv(seed=seed, wind=wind) on, errs, last_trace = 0, [], [] n_steady = 0 for ep in range(episodes): obs, _ = env.reset(seed=seed + ep) if wind: # pin the demo gust regime for the A/B env._wind.sigma = sigma env._wind.tau = tau ep_trace = [] for t in range(env.max_steps): obs, _, term, trunc, info = env.step(controller.act(obs)) ep_trace.append(info["landing_dist"]) if t >= env.max_steps // 2: # steady state (after convergence) on += int(info["on_target"]) errs.append(info["landing_dist"]) n_steady += 1 if term or trunc: break if trace and ep == episodes - 1: last_trace = ep_trace return (on / max(n_steady, 1), float(np.mean(errs)), float(np.percentile(errs, 90)), last_trace) def _fmt(rate, err, p90): return f"on-target {rate * 100:5.1f}% mean {err:.3f} m p90 {p90:.3f} m" def main(): ap = argparse.ArgumentParser(description=__doc__) ap.add_argument("--episodes", type=int, default=60) ap.add_argument("--seed", type=int, default=2025) ap.add_argument("--sigma", type=float, default=kin.WIND_SIGMA_DEMO) ap.add_argument("--tau", type=float, default=kin.WIND_TAU) ap.add_argument("--plot", action="store_true", help="save a landing-error-vs-time plot (needs matplotlib)") args = ap.parse_args() controllers: list[tuple[str, object]] = [("Reactive PID", PIDSprayController())] if DEFAULT_CHECKPOINT.is_file(): try: controllers.append(("RL (wind-aware)", SprayPolicy.load(DEFAULT_CHECKPOINT))) except Exception as exc: # noqa: BLE001 print(f"[warn] RL checkpoint failed to load ({exc}); showing PID only.\n") else: print(f"[warn] no RL checkpoint at {DEFAULT_CHECKPOINT}; showing PID only.\n") kw = dict(episodes=args.episodes, seed=args.seed, sigma=args.sigma, tau=args.tau) print(f"Spray-correction A/B | {args.episodes} paired episodes, " f"gust sigma={args.sigma} m/s^2 tau={args.tau}s, " f"flight-lag={kin.FLIGHT_LAG_STEPS} steps\n") header = f"{'controller':<18}{'calm air':<48}{'gusty wind':<48}" print(header) print("-" * len(header)) traces = {} for name, ctrl in controllers: calm = rollout(ctrl, wind=False, **kw) gust = rollout(ctrl, wind=True, trace=args.plot, **kw) traces[name] = gust[3] print(f"{name:<18}{_fmt(*calm[:3]):<48}{_fmt(*gust[:3]):<48}") if args.plot and any(traces.values()): _plot(traces, args) def _plot(traces, args): try: import matplotlib matplotlib.use("Agg") import matplotlib.pyplot as plt except Exception as exc: # noqa: BLE001 print(f"\n[warn] plot skipped ({exc})") return out = "train_5070ti/ab_spray.png" fig, ax = plt.subplots(figsize=(9, 4)) for name, tr in traces.items(): if tr: ax.plot(np.arange(len(tr)) * 0.02, tr, label=name, lw=1.6) ax.axhline(kin.HIT_RADIUS, ls="--", c="k", lw=1, label="hit radius") ax.set_xlabel("time (s)"); ax.set_ylabel("landing error (m)") ax.set_title(f"Gusty wind (sigma={args.sigma} m/s^2): PID trails, RL anticipates") ax.legend(); ax.grid(alpha=0.3) fig.tight_layout(); fig.savefig(out, dpi=120) print(f"\nsaved {out}") if __name__ == "__main__": main() ``` ### tests/test_spray_rl.py ```python """Phase 6 spray-aim RL surrogate.""" from __future__ import annotations import math import numpy as np from ember.nav import SPRAY_STANDOFF from ember.spray_rl.env import SprayEnv from ember.spray_rl import kinematics as kin from ember.spray_rl.kinematics import (HIT_RADIUS, OBS_DIM, aim_point, build_obs, sample_disc_offset) from ember.spray_rl.policy import PIDSprayController def test_spray_env_spaces_and_step(): env = SprayEnv(max_steps=20, seed=42) obs, _ = env.reset() assert obs.shape == (OBS_DIM,) assert env.observation_space.contains(obs) action = env.action_space.sample() assert env.action_space.contains(action) obs2, reward, term, trunc, info = env.step(action) assert obs2.shape == (OBS_DIM,) assert isinstance(reward, float) assert "landing_dist" in info assert "on_target" in info def test_spray_env_reward_sanity(): env = SprayEnv(max_steps=5, seed=0) env.reset() for _ in range(5): _, r, term, trunc, info = env.step(np.zeros(3, dtype=np.float32)) assert math.isfinite(r) assert r <= 5.0 + 1e-6 # bonus(5) - 2*dist - penalties if info["on_target"]: assert r > 0.0 if term or trunc: break def test_aim_point_hits_at_standoff(): """The fixed jet's closest approach to the fire centre is within HIT_RADIUS when the robot faces the fire at the navigation standoff.""" _, dist = aim_point(0.0, 0.0, 0.0, (SPRAY_STANDOFF, 0.0)) assert dist < HIT_RADIUS, f"standoff not hittable: min3d={dist:.3f} m" def test_disc_noise_radius(): rng = np.random.default_rng(0) rs = [np.hypot(*sample_disc_offset(rng, 0.25)) for _ in range(500)] assert max(rs) <= 0.25 + 1e-9 assert np.mean(rs) > 0.05 def test_build_obs_shape(): obs = build_obs(fire_xy=(2.0, 0.0), robot_xy=(0.0, 0.0), aim_xy=np.array([2.1, 0.05]), yaw=0.0, vx=0.0, vy=0.0, yaw_rate=0.0) assert obs.shape == (OBS_DIM,) assert obs.dtype == np.float32 def test_build_obs_is_body_frame(): """Spatial terms must rotate into the body frame so they match the actions. Robot facing +y (yaw=90 deg): a fire that is due east in world (+x) is to the robot's RIGHT, i.e. negative body-lateral; due north (+y) is straight ahead. A world-frame obs would mislabel these and the policy could not correct aim. """ obs = build_obs(fire_xy=(1.0, 0.0), robot_xy=(0.0, 0.0), aim_xy=np.array([1.0, 0.0]), yaw=math.pi / 2, vx=0.0, vy=0.0, yaw_rate=0.0) fire_fwd, fire_lat = float(obs[0]), float(obs[1]) assert abs(fire_fwd) < 1e-5 # east fire is not ahead assert fire_lat < -0.9 # it is to the right (negative lateral) def _pid_steady_on_target(*, wind: bool, episodes: int = 8, seed: int = 7) -> float: """Steady-state on-target fraction for the reactive PID over several episodes. When ``wind`` is on we pin the demo gust regime (strong, fast) rather than averaging over the calm-included training randomization, so the test asserts the actual A/B condition.""" pid = PIDSprayController() env = SprayEnv(seed=seed, wind=wind) steady, on = [], 0 for ep in range(episodes): obs, _ = env.reset(seed=seed + ep) if wind: env._wind.sigma = kin.WIND_SIGMA_DEMO env._wind.tau = kin.WIND_TAU for t in range(env.max_steps): obs, _, term, trunc, info = env.step(pid.act(obs)) if t >= env.max_steps // 2: steady.append(info["landing_dist"]) on += int(info["on_target"]) if term or trunc: break return on / max(len(steady), 1) def test_pid_controller_converges_on_env(): """Body-frame PID must actually reduce landing error in calm air (guards the frame bug: a world-frame controller diverges to >1 m, on-target ~0.05).""" pid = PIDSprayController() env = SprayEnv(seed=7, wind=False) steady, on = [], 0 for ep in range(6): obs, _ = env.reset(seed=7 + ep) for t in range(env.max_steps): obs, _, term, trunc, info = env.step(pid.act(obs)) if t >= env.max_steps // 2: steady.append(info["landing_dist"]) on += int(info["on_target"]) if term or trunc: break assert np.mean(steady) < 0.30, f"steady error too high: {np.mean(steady):.3f} m" assert on / len(steady) > 0.5 def test_wind_degrades_reactive_pid(): """Premise of the A/B: gusty wind the PID cannot anticipate collapses its on-target rate well below its calm-air performance (the RL policy, which sees a noisy wind reading, recovers this in deployment).""" calm = _pid_steady_on_target(wind=False) gusty = _pid_steady_on_target(wind=True) assert calm > 0.5 assert gusty < calm - 0.15, f"wind should hurt reactive PID: calm={calm:.2f} gusty={gusty:.2f}" ``` [50 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]