# Project export: Edge Rescue This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: The First Autonomous Benchmarking Platform for Off-the-Shelf VLA Policies on Real Hardware - Devpost: https://devpost.com/software/edge-rescue - GitHub: https://github.com/eiyer28/treehacks2026 - Demo: https://huggingface.co/datasets/trevorkw7/disaster_multitask - Video: https://www.youtube.com/embed/Uioy06rqzyM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Claude Opus 4.6 (5 commits), eiyer28 (5 commits) ## Devpost submission (written by the team) ### Inspiration It started with a simple question: "How good is this model, actually?" We'd been handed a DGX Spark, a robot arm, and 36 hours. The obvious move was to grab an off-the-shelf Vision-Language-Action (VLA) model, load it up, and watch the robot do something cool. So we did. We pulled Pi 0.5 from Physical Intelligence — a 4-billion parameter flow-matching policy that generates continuous robot trajectories from language and vision. We also grabbed SmolVLA from HuggingFace — a compact discretized model that tokenizes actions. Both had impressive demo videos. Both claimed strong performance on manipulation tasks. But when we actually tried to run them on the same robot, doing the same task, we realized something: there is no standardized way to know if a VLA model works. Every model ships with cherry-picked demo footage. Nobody publishes failure rates. Nobody compares models head-to-head on the same hardware. If you want to know whether Pi 0.5 or SmolVLA is better at stacking cubes, you have to load each one, physically watch every attempt, and manually decide whether it succeeded. That's not engineering. That's vibes. Then we thought about where this actually matters. In disaster response — collapsed buildings after earthquakes, infrastructure failures, hazmat scenarios — robots need to manipulate objects in unstructured environments with zero margin for error. Before you deploy a VLA policy to move rubble in a collapsed building, you need to know it works. Not from a demo video. From rigorous, autonomous, reproducible evaluation. So we built the evaluation platform itself. What It Does Edge Rescue is an autonomous benchmarking platform that can evaluate any VLA policy against real-world manipulation tasks — with an independent VLM judge, on edge hardware, with no cloud dependency. The core loop: A user sends a natural language goal — "stack the orange cubes" A Vision-Language Model (Cosmos-Reason1-7B) analyzes the scene via camera and decomposes the goal into an ordered plan of subtasks The VLA under test executes each subtask — driving the real robot arm The VLM independently verifies each step by comparing before/after camera images: "Did this action actually happen?" If verification fails, the system retries (up to $N$ times), then replans from the current state — considering what succeeded and what failed After all steps complete, a final VLM check verifies the overall goal is achieved. If not, it loops back and replans from scratch The key insight: the VLM that evaluates is not the VLA being tested. It's an independent judge. This is what makes it a benchmarking platform rather than just a demo rig. We tested two fundamentally different VLA architectures through the same pipeline: Same robot. Same cameras. Same tasks. Same judge. Different everything else. How We Built It Architecture The system runs across two machines with five concurrent services, orchestrated over ROS2 Jazzy: DGX Spark (128GB unified memory, aarch64): Cosmos-Reason1-7B — VLM planner and judge, served via llama.cpp (GGUF Q8_0) on port 8080 OpenPI Pi 0.5 Server — VLA inference via websocket on port 8001, serving the pi05_so101 checkpoint VLM Planner Node — ROS2 node implementing the full plan-execute-verify-replan state machine Pi 0.5 Executor Node — ROS2 node bridging LeRobot robot control with the OpenPI client Web Frontend — Next.js/React dashboard with live camera feed, plan visualization, and planner output log Bowman (RTX 3070, 8GB VRAM): SmolVLA Executor Node — same ROS2 interface, different model, different inference stack HTTP/SSE Bridge — REST + MJPEG streaming for alternative access Isaac Sim — physics simulation for structural validation The ROS2 Contract The secret to making model-swapping trivial is the topic contract: To swap VLA models, you only change the executor node. The planner, the judge, the UI, and the camera pipeline are completely unchanged. The Pi 0.5 executor is 300 lines of Python. The SmolVLA executor is 270 lines. Same interface, radically different internals. The VLM Judge Cosmos-Reason1-7B performs four distinct evaluation roles: Scene Understanding — analyzes the workspace image to generate a feasible plan Step Verification — compares before/after images for each subtask: "Did the gripper actually make contact with the cube?" Failure Diagnosis — when verification fails, explains why: "The arm overshot the target position" Goal Completion — after all steps, checks if the overall goal is truly achieved Every before/after image pair is automatically logged to ~/planner_logs/{timestamp}/ with descriptive filenames. One evaluation run of 13 steps produced 53 timestamped images — a full visual audit trail. The Closed Loop The planner operates as a state machine: $$\text{IDLE} \rightarrow \text{PLANNING} \rightarrow \text{EXECUTING} \rightarrow \text{VERIFYING} \rightarrow \begin{cases} \text{next step} \ \text{RETRY} \ \text{REPLAN} \end{cases} \rightarrow \text{GOAL_CHECK} \rightarrow \text{IDLE}$$ Crucially, replanning is context-aware: the VLM receives the list of completed steps, the failed subtask, the failure reason, and the current scene image. It doesn't start from scratch — it builds on what already worked. Challenges We Faced The OpenPI Format Problem (3 AM Discovery) The Pi 0.5 checkpoint (felixmayor/pi05_so101_orange_cube) is in OpenPI format — JAX/flax metadata, no config.json. LeRobot 0.4.3 has a PI05Policy class but it expects LeRobot-format checkpoints. It cannot load OpenPI-format weights. We had to: Write custom SO-101 input/output transforms (so101_policy.py) to map our camera names and joint states to the model's expected format Patch the transformers library for OpenPI's PyTorch mode: cp -r src/openpi/models_pytorch/transformers_replace/* .venv/.../transformers/ Discover at 3 AM that openpi-client requires numpy<2.0.0, which conflicted with half our stack Manage two separate Python environments: OpenPI uses uv with Python 3.11, while LeRobot runs on Python 3.12 The Serial Port Contention Problem LeRobot owns the SO-ARM101's serial port for motor control. The cameras are also initialized through LeRobot. But the VLM planner needs camera frames for verification, and the web frontend needs them for live display. You can't have multiple processes fighting over a USB serial bus. Our solution: a camera republisher thread inside the executor node that grabs frames from LeRobot's observation dictionary and publishes them to ROS2 at 10 Hz, protected by a threading.Lock mutex to prevent serial port contention between camera reads and motor commands. The Action Space Bridge Pi 0.5 returns action chunks — approximately 50 continuous joint positions per inference call. SmolVLA returns single discrete actions — one tokenized step per forward pass. The executor abstraction handles both: Pi 0.5's executor loops over the returned chunk at 50 Hz (matching training frequency), while SmolVLA's executor calls inference on every step. Both publish the same success/fail string to /subtask/status. The planner doesn't know or care which model is running. VLM JSON Reliability Cosmos-Reason1 sometimes wraps its JSON output in markdown code fences, or adds conversational preamble before the JSON object. We built a robust fallback parser: This handles every prompt type (plan, verify, replan, goal_check) with zero parse failures across all our evaluation runs. What We Learned Off-the-shelf VLAs are much more fragile than their demos suggest. Pi 0.5 generates beautifully smooth trajectories but overshoots on precise placement. SmolVLA reaches for objects reliably but its discretized actions lose fine motor control at the quantization boundary. Neither model's demo video would tell you this. Off-the-shelf VLAs are much more fragile than their demos suggest. Pi 0.5 generates beautifully smooth trajectories but overshoots on precise placement. SmolVLA reaches for objects reliably but its discretized actions lose fine motor control at the quantization boundary. Neither model's demo video would tell you this. An independent VLM judge changes everything. When the model evaluating success is separate from the model generating actions, you get honest signal. The VLM caught failures that a human observer might miss in real-time — subtle cases where the gripper closed 2mm too early, or where an object shifted but didn't actually reach the target. An independent VLM judge changes everything. When the model evaluating success is separate from the model generating actions, you get honest signal. The VLM caught failures that a human observer might miss in real-time — subtle cases where the gripper closed 2mm too early, or where an object shifted but didn't actually reach the target. The ROS2 abstraction layer was the highest-leverage decision we made. By defining a clean topic contract up front, swapping between Pi 0.5 and SmolVLA became a one-line change in the launch script. Every minute we spent on that interface saved hours of integration work later. The ROS2 abstraction layer was the highest-leverage decision we made. By defining a clean topic contract up front, swapping between Pi 0.5 and SmolVLA became a one-line change in the launch script. Every minute we spent on that interface saved hours of integration work later. 128GB of unified memory is a superpower. Running a 7B VLM and a 4B VLA simultaneously on the same device — with camera streams and a web server — would be impossible on consumer hardware. The DGX Spark's unified memory architecture meant we never had to choose between model quality and system complexity. 128GB of unified memory is a superpower. Running a 7B VLM and a 4B VLA simultaneously on the same device — with camera streams and a web server — would be impossible on consumer hardware. The DGX Spark's unified memory architecture meant we never had to choose between model quality and system complexity. Evaluation infrastructure is a prerequisite to trust. Before you deploy a VLA to move rubble in a collapsed building, you benchmark it moving cubes on a table. Same platform. Same evaluation loop. Same independent judge. The only thing that changes is the stakes. Evaluation infrastructure is a prerequisite to trust. Before you deploy a VLA to move rubble in a collapsed building, you benchmark it moving cubes on a table. Same platform. Same evaluation loop. Same independent judge. The only thing that changes is the stakes. What's Next The platform is model-agnostic by design. We tested two VLAs, but the architecture supports any policy that can receive a text prompt and camera images and return joint-space actions. The verification loop, the replanning logic, and the visual audit trail work regardless of what's generating the actions. The immediate extensions: Quantitative comparison dashboards — success rates, retry counts, and replanning frequency across models and tasks Isaac Sim integration — physics-validated stress testing before real execution (structural stability, collision prediction) Multi-robot evaluation — run the same task on multiple arms simultaneously to measure consistency The long-term vision: a standardized benchmark suite for embodied AI, running entirely on edge hardware, that gives robotics teams the same confidence in their VLA policies that software teams get from CI/CD test suites. Built With NVIDIA DGX Spark (128GB unified memory) — VLM + VLA inference NVIDIA Cosmos-Reason1-7B — vision-language model for planning and evaluation Pi 0.5 (Physical Intelligence, via OpenPI) — continuous VLA policy SmolVLA (HuggingFace) — discrete VLA policy ROS2 Jazzy — middleware and topic-based orchestration LeRobot — robot control and camera integration llama.cpp — efficient VLM serving (GGUF Q8_0) SO-ARM101 — 6-DOF robot arm React / Next.js / Tailwind — real-time monitoring frontend NVIDIA Isaac Sim — physics simulation Python, PyTorch, JAX, OpenCV, roslib.js ## README (from the GitHub repository) # TreeHacks 2026 Eashan Iyer, Samuel Lihn, Gordon Jin, Trevor Kwan Project: Hardware: Nvidia Jetson Orin Nano Super (2x) Hugging face SO-101 Robot arm (1 leader, 1 follower) 2 Mono Cameras Logitech C920 Webcam Compute: RTX 3070 Mobile Laptop (can run NVIDIA Isaac Sim) RTX 5090 can be accessed remotely for fine tuning models ## Detected evidence (automated analysis) Indexed codebase: 8 recognized source files, 71 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (10 of 10) ``` .claude/settings.local.json 2026 TreeHacks Prize & Challenge List.txt CLAUDE.md create_pptx.py frontend/app.js frontend/index.html frontend/style.css presentation.md README.md server/bridge.py ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Add camera feed from /cam0/compressed ROS2 topic - Replace WebSocket bridge with HTTP + SSE architecture - Fix websockets v16 compatibility and update Spark IP - Add frontend UI and WebSocket bridge server - Add presentation deck and refine CLAUDE.md pitch - added claude file - added deepmind pdf - Update README with team info, hardware, and compute details - added hardware - Add README ## Key source files (fetched from GitHub, selected and truncated for size) ### presentation.md ```markdown # MirrorVerse — Presentation Deck # TreeHacks 2026 --- ## SLIDE 1: Title **MirrorVerse** *Sim-Validated Edge AI Robotics for Disaster Response* Eashan Iyer, Samuel Lihn, Gordon Jin, Trevor Kwan TreeHacks 2026 --- ## SLIDE 2: The Problem **When disasters strike, robots can't think for themselves — and humans can't reach them.** - 2023 Turkey-Syria earthquake: 59,000+ dead, 10,000+ buildings collapsed - Secondary aftershock collapses killed additional victims and endangered 53,000+ deployed rescue workers - Source: UNDP, Center for Disaster Philanthropy - March 2025 Myanmar earthquake: 3,600+ dead, 10,000+ buildings collapsed or severely damaged - A 6.4 aftershock hit while rescue operations were active - Source: WHO, AHA Centre - In these environments, cell towers are down. Cloud APIs are unreachable. But the need for autonomous robots is at its peak. **The question: How do you make a robot that can build, reinforce, and clear — safely — with zero internet?** --- ## SLIDE 3: The Insight **The most dangerous thing an autonomous robot can do is act without checking its work.** An LLM can plan a construction task. But LLMs hallucinate — they can propose structures that are physically impossible or unstable. Current approaches: - Cloud-based VLAs (like Gemini Robotics 1.5) require massive models and internet connectivity - Standard digital twins mirror reality but don't plan ahead - Text-to-robot systems are one-shot — no validation before execution **What's missing: a physics sanity-check that runs locally, before the robot commits.** --- ## SLIDE 4: Our Solution **MirrorVerse: Plan. Simulate. Validate. Build.** An AI system that runs entirely on edge hardware: 1. **You prompt it** — "Build a reinforced shelter" (natural language) 2. **It plans** — Local LLM on DGX Spark decomposes the task into subtasks 3. **It simulates** — Isaac Sim tests each action: will this block fall? Will this structure survive lateral forces? 4. **It validates** — Only actions that pass physics checks are sent to the real robot 5. **It builds** — SO-101 arm executes the validated plan 6. **It adapts** — If the real world doesn't match the sim (block slipped, misaligned), the system re-plans No cloud. No internet. Fully autonomous. --- ## SLIDE 5: The Demo — Earthquake Test [This slide should show a diagram or live demo of the following sequence] **Step 1: Prompt** — "Build a tower" - DGX Spark LLM breaks this into pick-and-place subtasks **Step 2: Build** — Physical arm stacks blocks - Isaac Sim mirrors every move in real time (split screen) **Step 3: Stress Test** — Simulate an earthquake - Isaac Sim applies lateral forces to the digital twin - The tower collapses in simulation (the real tower is untouched) **Step 4: Redesign** — AI sees the failure - LLM analyzes the collapse, redesigns with a wider base and interlocking pattern **Step 5: Rebuild** — Arm deconstructs and rebuilds the reinforced version - Isaac Sim confirms the new de [truncated — 6110 more characters] ``` ### CLAUDE.md ```markdown # TreeHacks 2026 Eashan Iyer, Samuel Lihn, Gordon Jin, Trevor Kwan Project: We are building a generalized edge AI construction system for the Treehacks 2026 hackathon at Stanford. It demonstrates two robotic agents—one physical (in the real world) and one virtual (in NVIDIA Isaac Sim)—working together on a single shared task. The premise is that you can prompt an LLM that runs locally on a DGX spark, which is used for planning and is the brains behind the operation. This can do a wide range of tasks. From there the LLM passes subtasks into the Jetson, which then uses VLA/VLM and a smaller LLM to turn this subtask into code that can run on the robotic arm. To get a sense of capabitilies, we could build something like a tower of blocks. The prompt would be turned into subtasks and then run on the robotic arm. We would simulate actions on Isaac Sim and use that to prevent hallucination. We would also use it to simulate things like lateral forces as a simplified model of earthquakes. From there we could show that the current tower would break, then the AI would redesign the tower and then it would be rebuilt using this feedback loop to reinforce the structure. This system is built entirely on edge AI — the LLM runs locally on the DGX Spark, perception runs on the Jetson, and physics validation runs on a local GPU. No cloud. No internet required. That architectural choice unlocks the highest-stakes use case: disaster response. When an earthquake levels a building, cell towers go down and cloud APIs become unreachable, but the need for autonomous robots is at its peak. A system like this could direct robots to shore up unstable structures, clear debris from access routes, or assemble temporary shelters — all while simulating each action first to avoid making a collapse worse. The demo we show (building a block tower, earthquake-testing it in simulation, watching it fail, and having the AI redesign and rebuild a stronger version) is not a metaphor for disaster response — it is the core capability in miniature. The same plan-simulate-validate-execute loop that reinforces a block tower is what would prevent a rescue robot from pulling the wrong beam out of a rubble pile. Because the system is fully generalizable (prompt in natural language, execute with any manipulator), extending from blocks on a table to rubble in a disaster zone is a matter of scale, not a change in architecture. Here is the criteria that is being used to judge this project: Creativity We want hackers to create a project that makes you say “wow” and tell all your friends about it. We're looking for projects that think so far outside the box that you begin to wonder why there was a box at all in the first place. Technical Complexity In only 36 hours, hackers manage to build projects with remarkably complex infrastructures built on excitingly advanced frameworks. We hope to see projects that are really running some beautiful code or hardware under the hood. Social Impact We're lo [truncated — 9190 more characters] ``` ### frontend/app.js ```javascript // ---- CONFIG ---- const SPARK_IP = "100.123.79.38"; const BRIDGE_URL = `http://${SPARK_IP}:9090`; // ---- DOM ---- const messagesEl = document.getElementById("messages"); const promptEl = document.getElementById("prompt"); const sendBtn = document.getElementById("send"); const statusEl = document.getElementById("status"); const planView = document.getElementById("plan-view"); const subtaskLabel = document.getElementById("subtask-label"); const logPanel = document.getElementById("log-panel"); const logToggle = document.getElementById("log-toggle"); // ---- LOG PANEL ---- logToggle.addEventListener("click", () => { logPanel.classList.toggle("open"); logToggle.textContent = logPanel.classList.contains("open") ? "Hide Logs" : "Logs"; }); function log(message, level = "info") { const ts = new Date().toLocaleTimeString("en-US", { hour12: false, hour: "2-digit", minute: "2-digit", second: "2-digit", fractionalSecondDigits: 3 }); const entry = document.createElement("div"); entry.className = `log-entry log-${level}`; entry.innerHTML = `${ts}${message}`; logPanel.appendChild(entry); logPanel.scrollTop = logPanel.scrollHeight; } // ---- STATE ---- let planSteps = []; let currentSubtask = ""; let evtSource = null; // ---- HELPERS ---- function addMessage(text, role) { const div = document.createElement("div"); div.className = `msg ${role}`; div.textContent = text; messagesEl.appendChild(div); messagesEl.scrollTop = messagesEl.scrollHeight; } function setStatus(connected) { statusEl.textContent = connected ? "connected" : "disconnected"; statusEl.className = connected ? "connected" : ""; sendBtn.disabled = !connected; } function renderPlan() { if (planSteps.length === 0) { planView.innerHTML = '
No plan yet. Send a prompt to begin.
'; return; } planView.innerHTML = ""; planSteps.forEach((step, i) => { const div = document.createElement("div"); div.className = "plan-step"; if (step.status === "done") { div.classList.add("done"); } else if (step.status === "active") { div.classList.add("active"); } div.textContent = `${i + 1}. ${step.label}`; planView.appendChild(div); }); } // ---- SSE CONNECTION ---- let connectAttempt = 0; function connectSSE() { connectAttempt++; log(`SSE connection attempt #${connectAttempt} to ${BRIDGE_URL}/events`, "info"); evtSource = new EventSource(`${BRIDGE_URL}/events`); evtSource.onopen = () => { connectAttempt = 0; setStatus(true); addMessage("Connected to DGX Spark.", "system"); log(`SSE connected to ${BRIDGE_URL}/events`, "ok"); }; evtSource.addEventListener("plan", (e) => { log(`plan received: ${e.data.substring(0, 200)}`, "info"); try { const plan = JSON.parse(e.data); planSteps = (Array.isArray(plan) ? plan : [plan]).map((s) => { if (typeof s === "string") return { label: s, status: "pending" }; return { label: s.label || s.task || JSON.stringify(s), status: s.status || "pending" }; }); renderPlan(); addMessage("Plan received:\n" + planSteps.map((s, i) => ` ${i + 1}. ${s.label}`).join("\n"), "assistant"); } catch (err) { log(`plan parse error: ${err.message}`, "warn"); addMessage(e.data, "assistant"); } }); evtSource.addEventListener("subtask", (e) => { log(`subtask: ${e.data}`, "info"); currentSubtask = e.data; subtaskLabel.textContent = currentSubtask; let foundCurrent = false; planSteps.forEach((step) => { if (foundCurrent) { step.status = "pending"; } else if (step.label === currentSubtask || currentSubtask.includes(step.label)) { step.status = "active"; foundCurrent = true; } else { step.status = "done"; } }); renderPlan(); }); evtSource.onerror = () => { setStatus(false); log(`SSE connection lost. Retrying in 3s (attempt #${connectAttempt})`, "warn"); evtSource.close(); setTimeout(connectSSE, 3000); }; } // ---- SEND ---- function send() { const text = promptEl.value.trim(); if (!text) return; addMessage(text, "user"); log(`Sending goal: "${text}"`, "ok"); planSteps = []; renderPlan(); subtaskLabel.textContent = "Planning..."; fetch(`${BRIDGE_URL}/goal`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ prompt: text }), }) .then((res) => { if (!res.ok) throw new Error(`HTTP ${res.status}`); log(`Goal accepted by server`, "ok"); }) .catch((err) => { log(`Failed to send goal: ${err.message}`, "error"); addMessage(`Error sending goal: ${err.message}`, "system"); }); promptEl.value = ""; } sendBtn.addEventListener("click", send); promptEl.addEventListener("keydown", (e) => { if (e.key === "Enter") send(); }); document.getElementById("hostname").textContent = `${location.hostname || "file"} \u2192 ${BRIDGE_URL}`; // ---- CAMERA FEED ---- const camFeed = document.getElementById("cam-feed"); const camStatus = document.getElementById("cam-status"); function startCamFeed() { camFeed.src = `${BRIDGE_URL}/cam0/stream`; camFeed.onload = () => { camStatus.style.display = "none"; }; camFeed.onerror = () => { camStatus.textContent = "Camera offline"; camStatus.style.display = "block"; // Retry after 5s setTimeout(() => { camFeed.src = `${BRIDGE_URL}/cam0/stream?t=${Date.now()}`; }, 5000); [truncated — 135 more characters] ``` ### frontend/index.html ```html Edge Rescue

Edge Rescue

disconnected
Connecting to DGX Spark...
``` ### frontend/style.css ```css * { margin: 0; padding: 0; box-sizing: border-box; } body { background: #0f0f0f; color: #fff; font-family: 'Segoe UI', Calibri, sans-serif; height: 100vh; display: flex; flex-direction: column; } header { padding: 1rem 1.5rem; border-bottom: 1px solid #222; display: flex; align-items: center; gap: 1rem; } header h1 { font-size: 1.4rem; font-weight: 700; } header h1 span { color: #76b900; } #status { font-size: 0.8rem; padding: 0.25rem 0.6rem; border-radius: 999px; background: #2a1a1a; color: #ff4c4c; } #status.connected { background: #1a2a1a; color: #76b900; } #main { flex: 1; display: flex; overflow: hidden; } #messages { flex: 1; overflow-y: auto; padding: 1.5rem; display: flex; flex-direction: column; gap: 0.75rem; } #sidebar { width: 340px; border-left: 1px solid #222; display: flex; flex-direction: column; overflow: hidden; } #sidebar h2 { font-size: 0.85rem; color: #76b900; padding: 0.75rem 1rem; border-bottom: 1px solid #222; text-transform: uppercase; letter-spacing: 0.05em; } #cam-view { padding: 0.5rem; border-bottom: 1px solid #222; position: relative; background: #0a0a0a; } #cam-feed { width: 100%; display: block; border-radius: 4px; background: #111; min-height: 180px; object-fit: contain; } #cam-status { position: absolute; top: 50%; left: 50%; transform: translate(-50%, -50%); color: #555; font-size: 0.8rem; pointer-events: none; } #plan-view, #subtask-view { flex: 1; overflow-y: auto; padding: 0.75rem 1rem; font-size: 0.85rem; border-bottom: 1px solid #222; } .plan-step { padding: 0.4rem 0; border-bottom: 1px solid #1a1a2e; color: #bbb; } .plan-step.active { color: #76b900; font-weight: 700; } .plan-step.done { color: #555; text-decoration: line-through; } #subtask-label { color: #009eff; font-weight: 600; } .msg { max-width: 80%; padding: 0.75rem 1rem; border-radius: 0.75rem; font-size: 0.95rem; line-height: 1.5; white-space: pre-wrap; } .msg.user { align-self: flex-end; background: #1a3a5c; border: 1px solid #009eff; } .msg.assistant { align-self: flex-start; background: #1a1a2e; border: 1px solid #333; } .msg.system { align-self: center; background: none; color: #666; font-size: 0.8rem; text-align: center; } #input-bar { padding: 1rem 1.5rem; border-top: 1px solid #222; display: flex; gap: 0.75rem; } #prompt { flex: 1; padding: 0.75rem 1rem; font-size: 1rem; background: #1a1a1a; border: 1px solid #333; border-radius: 0.5rem; color: #fff; outline: none; font-family: inherit; } #prompt:focus { border-color: #76b900; } #prompt::placeholder { color: #555; } #send { padding: 0.75rem 1.5rem; background: #76b900; color: #0f0f0f; border: none; border-radius: 0.5rem; font-size: 1rem; font-weight: 700; cursor: pointer; } #send:hover { background: #8ad400; } #send:disabled { background: #333; color: #666; cursor: not-allowed; } /* ---- Log Panel ---- */ #log-toggle { font-size: 0.75rem; padding: 0.2rem 0.5rem; background: #1a1a2e; border: 1px solid #333; border-radius: 4px; color: #888; cursor: pointer; margin-left: 0.5rem; } #log-toggle:hover { border-color: #76b900; color: #bbb; } #log-panel { display: none; border-top: 1px solid #222; background: #0a0a0a; max-height: 220px; overflow-y: auto; font-family: 'Consolas', 'Courier New', monospace; font-size: 0.75rem; padding: 0.5rem 1rem; } #log-panel.open { display: block; } .log-entry { padding: 2px 0; border-bottom: 1px solid #111; display: flex; gap: 0.5rem; } .log-ts { color: #444; flex-shrink: 0; } .log-entry.log-info .log-msg { color: #888; } .log-entry.log-ok .log-msg { color: #76b900; } .log-entry.log-warn .log-msg { color: #ffaa00; } .log-entry.log-error .log-msg { color: #ff4c4c; } ``` ### create_pptx.py ```python from pptx import Presentation from pptx.util import Inches, Pt, Emu from pptx.dml.color import RGBColor from pptx.enum.text import PP_ALIGN, MSO_ANCHOR from pptx.enum.shapes import MSO_SHAPE prs = Presentation() prs.slide_width = Inches(13.333) prs.slide_height = Inches(7.5) # --- Color Palette --- BG_DARK = RGBColor(0x0F, 0x0F, 0x0F) BG_SECTION = RGBColor(0x1A, 0x1A, 0x2E) NVIDIA_GREEN = RGBColor(0x76, 0xB9, 0x00) ACCENT_BLUE = RGBColor(0x00, 0x9E, 0xFF) ACCENT_RED = RGBColor(0xFF, 0x4C, 0x4C) WHITE = RGBColor(0xFF, 0xFF, 0xFF) LIGHT_GRAY = RGBColor(0xBB, 0xBB, 0xBB) DARK_GRAY = RGBColor(0x2A, 0x2A, 0x2A) TABLE_HEADER_BG = RGBColor(0x1E, 0x1E, 0x3A) TABLE_ROW_BG = RGBColor(0x14, 0x14, 0x24) TABLE_ALT_BG = RGBColor(0x1A, 0x1A, 0x30) def set_slide_bg(slide, color): bg = slide.background fill = bg.fill fill.solid() fill.fore_color.rgb = color def add_textbox(slide, left, top, width, height, text, font_size=18, color=WHITE, bold=False, alignment=PP_ALIGN.LEFT, font_name="Calibri"): txBox = slide.shapes.add_textbox(Inches(left), Inches(top), Inches(width), Inches(height)) tf = txBox.text_frame tf.word_wrap = True p = tf.paragraphs[0] p.text = text p.font.size = Pt(font_size) p.font.color.rgb = color p.font.bold = bold p.font.name = font_name p.alignment = alignment return tf def add_paragraph(tf, text, font_size=18, color=WHITE, bold=False, alignment=PP_ALIGN.LEFT, space_before=0, font_name="Calibri"): p = tf.add_paragraph() p.text = text p.font.size = Pt(font_size) p.font.color.rgb = color p.font.bold = bold p.font.name = font_name p.alignment = alignment if space_before: p.space_before = Pt(space_before) return p def add_table(slide, rows, cols, left, top, width, height): table_shape = slide.shapes.add_table(rows, cols, Inches(left), Inches(top), Inches(width), Inches(height)) return table_shape.table def style_cell(cell, text, font_size=14, color=WHITE, bold=False, bg_color=None): cell.text = "" p = cell.text_frame.paragraphs[0] p.text = text p.font.size = Pt(font_size) p.font.color.rgb = color p.font.bold = bold p.font.name = "Calibri" cell.text_frame.word_wrap = True if bg_color: cell.fill.solid() cell.fill.fore_color.rgb = bg_color cell.vertical_anchor = MSO_ANCHOR.MIDDLE def add_accent_line(slide, left, top, width, color=NVIDIA_GREEN): shape = slide.shapes.add_shape( MSO_SHAPE.RECTANGLE, Inches(left), Inches(top), Inches(width), Pt(4) ) shape.fill.solid() shape.fill.fore_color.rgb = color shape.line.fill.background() def add_device_box(slide, left, top, width, height, title, subtitle, details, border_color): shape = slide.shapes.add_shape( MSO_SHAPE.ROUNDED_RECTANGLE, Inches(left), Inches(top), Inches(width), Inches(height) ) shape.fill.solid() shape.fill.fore_color.rgb = RGBColor(0x1A, 0x1A, 0x2E) shape.line.color.rgb = border_color shape.line.width = Pt(2) add_textbox(slide, left + 0.15, top + 0.1, width - 0.3, 0.4, title, font_size=16, color=border_color, bold=True, alignment=PP_ALIGN.CENTER) add_textbox(slide, left + 0.15, top + 0.5, width - 0.3, 0.35, subtitle, font_size=13, color=WHITE, bold=True, alignment=PP_ALIGN.CENTER) add_textbox(slide, left + 0.15, top + 0.85, width - 0.3, height - 1.0, details, font_size=11, color=LIGHT_GRAY, alignment=PP_ALIGN.CENTER) # ============================================================ # SLIDE 1: Title # ============================================================ slide = prs.slides.add_slide(prs.slide_layouts[6]) set_slide_bg(slide, BG_DARK) add_accent_line(slide, 3, 1.8, 7.333) add_textbox(slide, 1, 2.0, 11.333, 1.5, "Edge Rescue", font_size=54, color=WHITE, bold=True, alignment=PP_ALIGN.CENTER) add_textbox(slide, 1, 3.2, 11.333, 0.8, "Humans, Robots, and AI \u2014 Working Together", font_size=28, color=NVIDIA_GREEN, alignment=PP_ALIGN.CENTER) add_textbox(slide, 1, 4.1, 11.333, 0.7, "Sim-validated autonomous robotics for disaster response.", font_size=18, color=LIGHT_GRAY, alignment=PP_ALIGN.CENTER) add_textbox(slide, 1, 4.6, 11.333, 0.5, "Plan. Simulate. Validate. Build. \u2014 No cloud, no internet, fully on edge hardware.", font_size=16, color=LIGHT_GRAY, alignment=PP_ALIGN.CENTER) add_textbox(slide, 1, 5.5, 11.333, 0.5, "Eashan Iyer | Samuel Lihn | Gordon Jin | Trevor Kwan", font_size=18, color=LIGHT_GRAY, alignment=PP_ALIGN.CENTER) add_textbox(slide, 1, 6.1, 11.333, 0.5, "TreeHacks 2026", font_size=16, color=LIGHT_GRAY, alignment=PP_ALIGN.CENTER) # ============================================================ # SLIDE 2: The Story (condensed) # ============================================================ slide = prs.slides.add_slide(prs.slide_layouts[6]) set_slide_bg(slide, BG_DARK) add_textbox(slide, 0.8, 0.4, 11.7, 0.8, "HATAY, TURKEY \u2014 FEBRUARY 6, 2023", font_size=14, color=ACCENT_RED, bold=True) add_accent_line(slide, 0.8, 0.8, 2, color=ACCENT_RED) # Left: the disaster facts tf = add_textbox(slide, 0.8, 1.2, 6.0, 5.5, "", font_size=18) add_paragraph(tf, "4:17 AM. A 7.8-magnitude earthquake hits.", font_size=24, color=WHITE, bold=True) add_paragraph(tf, "", font_size=6) add_paragraph(tf, "59,000 dead across Turkey and Syria.", font_size=20, color=ACCENT_RED, bold=True, space_before=8) add_paragraph(tf, "230,000+ buildings damaged or destroyed.", font_size=18, color=WHITE, space_before=8) add_paragraph(tf, "", font_size=6) add_paragraph(tf, "The flaw wasn't bravery. It was information.", font_size=20, color=WHITE, bold=True, space_before=16) add_paragraph(tf, "", font_size=6) add_paragraph(tf, "Rescue teams couldn't reach remote villages for 3\u20134 days.", font_size=16, color=LIGHT_GRAY, space_before=6) add_paragraph(tf, "Syria's White Helmets had no heavy equipment \u2014 it never came.", font_size=16, color=LIGHT_G [truncated — 21324 more characters] ``` ### server/bridge.py ```python #!/usr/bin/env python3 """ Edge Rescue — HTTP + SSE bridge server. Endpoints: POST /goal — accepts {"prompt": "..."}, publishes to ROS2 topic GET /events — SSE stream of plan and subtask updates GET /cam0/stream — MJPEG stream from /cam0/compressed ROS2 topic GET /cam0/snap — single JPEG snapshot Run: source /opt/ros/humble/setup.bash && python3 bridge.py """ import json import subprocess import threading import queue import time from http.server import HTTPServer, BaseHTTPRequestHandler HOST = "0.0.0.0" PORT = 9090 # ---- SSE state ---- sse_clients = [] # list of queue.Queue sse_lock = threading.Lock() # ---- Camera state ---- latest_frame = None # raw JPEG bytes latest_frame_lock = threading.Lock() cam_clients = [] # list of threading.Event to notify MJPEG clients cam_clients_lock = threading.Lock() cam_running = False def broadcast_sse(event_type, data): """Push an SSE event to all connected clients.""" with sse_lock: dead = [] for q in sse_clients: try: q.put((event_type, data), block=False) except queue.Full: dead.append(q) for q in dead: sse_clients.remove(q) def ros2_pub(topic, message): """Publish a string message to a ROS2 topic via subprocess.""" cmd = [ "ros2", "topic", "pub", "--once", topic, "std_msgs/String", f'{{"data": "{message}"}}' ] print(f"[ros2] {' '.join(cmd)}") try: result = subprocess.run(cmd, capture_output=True, text=True, timeout=10) if result.returncode != 0: print(f"[ros2] stderr: {result.stderr.strip()}") except FileNotFoundError: print("[ros2] ros2 CLI not found — running in demo mode") except subprocess.TimeoutExpired: print("[ros2] publish timed out") def handle_goal(prompt): """Process a mission goal: publish to ROS2, then run placeholder plan logic.""" print(f"\n{'='*50}") print(f" MISSION GOAL: {prompt}") print(f"{'='*50}\n") ros2_pub("/mission/goal", prompt) # --- PLACEHOLDER: replace with actual LLM call --- plan = [ f"Locate objects for: {prompt}", "Pick up first block", "Place block at target position", "Verify placement in simulation", "Report result", ] broadcast_sse("plan", json.dumps(plan)) for i, step in enumerate(plan): broadcast_sse("subtask", step) print(f" [{i+1}/{len(plan)}] {step}") time.sleep(1) broadcast_sse("subtask", "Done.") print(" Mission complete.\n") # ---- Camera subscriber (rclpy) ---- def start_cam_subscriber(): """Background thread: subscribe to /cam0/compressed via rclpy and store frames.""" global latest_frame, cam_running try: import rclpy from rclpy.node import Node from sensor_msgs.msg import CompressedImage except ImportError: print("[cam] rclpy not available — camera feed disabled") print("[cam] Run: source /opt/ros/humble/setup.bash") return rclpy.init() class CamSub(Node): def __init__(self): super().__init__("bridge_cam_sub") self.sub = self.create_subscription( CompressedImage, "/cam0/compressed", self.on_frame, 1 ) self.frame_count = 0 def on_frame(self, msg): global latest_frame frame_bytes = bytes(msg.data) with latest_frame_lock: latest_frame = frame_bytes # Notify all MJPEG clients with cam_clients_lock: for evt in cam_clients: evt.set() self.frame_count += 1 if self.frame_count == 1: print(f"[cam] First frame received ({len(frame_bytes)} bytes, format: {msg.format})") elif self.frame_count % 300 == 0: print(f"[cam] {self.frame_count} frames received") node = CamSub() cam_running = True print("[cam] Subscribed to /cam0/compressed") try: rclpy.spin(node) except Exception as e: print(f"[cam] Subscriber error: {e}") finally: cam_running = False node.destroy_node() rclpy.shutdown() class BridgeHandler(BaseHTTPRequestHandler): """Handles POST /goal, GET /events, GET /cam0/stream, GET /cam0/snap.""" def _cors_headers(self): self.send_header("Access-Control-Allow-Origin", "*") self.send_header("Access-Control-Allow-Methods", "GET, POST, OPTIONS") self.send_header("Access-Control-Allow-Headers", "Content-Type") def do_OPTIONS(self): self.send_response(204) self._cors_headers() self.end_headers() def do_POST(self): if self.path == "/goal": length = int(self.headers.get("Content-Length", 0)) body = self.rfile.read(length) try: data = json.loads(body) prompt = data.get("prompt", "") except (json.JSONDecodeError, AttributeError): prompt = body.decode("utf-8", errors="replace") if not prompt: self.send_response(400) self._cors_headers() self.end_headers() self.wfile.write(b'{"error":"empty prompt"}') return self.send_response(200) self._cors_headers() self.send_header("Content-Type", "application/json") self.end_headers() self.wfile.write(json.dumps({"status": "ok"}).encode()) threading.Thread(target=handle_goal, args=(prompt,), daemon=True).start() else: self.send_response(404) self._cors_headers() self.end_headers() def do_GET(self): if self.path == "/events": self._handle_sse() elif self.path == "/cam0/stream": self._handle_mjpeg( [truncated — 5530 more characters] ```