# Project export: EdgeCase This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: EdgeCase turns hardware debugging into a a closed loop experiment using NVIDIA NIM and Nemotron foundational models. Agents flash firmware, capture output, and triage debug to firmware improvements. - Devpost: https://devpost.com/software/edgecase-lsc926 - GitHub: https://github.com/jtl06/hard-itl - Video: https://www.youtube.com/embed/uO099eBgX7M?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — jtl06 (47 commits) ## Devpost submission (written by the team) ### Inspiration Inspired by the difficulty and repetitive nature of hardware debug, I wanted a way to turn “try random things until it works” into a structured loop with evidence. Hardware issues are often timing-sensitive, hard to reproduce, and expensive to diagnose when your only feedback is vague logs. EdgeCase is our attempt to make debugging feel more like running experiments: capture ground truth, propose the next best test, and converge on a fix. To make that loop fast enough to be practical, we run multiple specialized agents locally on an NVIDIA DGX Spark, so planning, critique, and next-step generation can happen in parallel without waiting on cloud services. ### What it does EdgeCase turns hardware debugging into a closed-loop experiment using NVIDIA NIM and Nemotron foundational models running locally on a DGX Spark. Multiple agents: build and flash RP2350 firmware, capture UART output and logic-analyzer traces (the “truth layer”), extract metrics (drops, timing gaps, framing errors, missing markers), propose the next experiments (parameter sweeps, decode settings, instrumentation tweaks), and iterate until the fix is validated by evidence. For the demo, EdgeCase can solve a real-world bring-up problem interactively: the user picks a nonstandard UART baud rate (e.g., 76200), and the agents automatically test decode settings until the UART stream decodes correctly and passes validation checks. ### How we built it Compute + models (all local): NVIDIA DGX Spark running Ubuntu 24.04, hosting a single local NIM endpoint for Nemotron Nano 9B v2 (OpenAI-compatible API). All agent calls go to the same local endpoint. Hardware-in-the-loop runner: a Runner service that is the only component allowed to touch hardware: runs make to build firmware, flashes the RP2350 (UF2 / picotool / OpenOCD auto-detect), captures USB CDC logs, triggers Saleae Logic 2 automation to capture and export traces + UART decode. runs make to build firmware, flashes the RP2350 (UF2 / picotool / OpenOCD auto-detect), captures USB CDC logs, triggers Saleae Logic 2 automation to capture and export traces + UART decode. Multi-agent orchestration (local fan-out/converge): four agents (planner, coder, critic, summarizer) run concurrently on the DGX Spark for fast iteration, then converge into a single actionable run plan. Evidence bundles: every run produces a reproducible artifact bundle (logs, traces, metrics, and a triage note) so debugging decisions are always backed by data. ### Challenges we ran into Flashing reliability: making firmware flashing deterministic across different boot states and toolchains (mass-storage UF2 vs picotool vs OpenOCD). Truth vs telemetry: USB CDC logs can hide physical-layer issues; the logic analyzer is the ground truth, but automating capture and exports robustly takes care. Reproducibility: turning a flaky symptom into a repeatable experiment required better instrumentation (markers like RUN_START/ERROR/RUN_END and GPIO triggers). Agent safety: ensuring agents propose changes without directly touching hardware or running arbitrary shell commands outside the Runner boundary. ### Accomplishments we're proud of A working closed-loop workflow where “debug” becomes a sequence of experiments with saved evidence, not guesswork. A multi-agent system running locally that proposes targeted next runs and converges quickly, rather than brute-forcing everything. A demo-friendly interactive scenario (UART baud discovery) that feels like real hardware bring-up and has an obvious visual win when decoding locks in. A clean boundary design: hardware access is isolated to a single Runner, making the system safer and more reliable. ### What we learned Hardware debugging gets dramatically easier when you treat the logic analyzer as the source of truth and structure everything around measurable signals. The biggest productivity boost isn’t one perfect AI answer, it’s a loop: hypothesis → experiment → evidence → next experiment. Good instrumentation beats clever reasoning. Simple markers, triggers, and checksums make automation possible. Concurrency and specialization help: planner/coder/critic roles catch different failure modes and reduce blind spots. ### What's next Expand beyond UART into SPI/I2C protocol debugging (including automated decoder selection and timing-violation detection). Add a lightweight UI dashboard to browse runs, compare traces, and visualize clusters of failures over time. Add bisect mode across firmware changes to automatically localize regressions. Support more hardware targets and capture backends (sigrok, other logic analyzers) while keeping the same evidence-bundle format. Turn the demo problems into a library of reproducible debug challenges (baud/framing/inversion/drops/timing races). ## README (from the GitHub repository) # EdgeCase (Multi-Agent LLM HIL on DGX Spark + NVIDIA NIM) Local-first multi-agent hardware-in-the-loop debugger. Only `runner/` touches hardware (`build`, `flash`, `/dev/tty*`). ## Mission and purpose EdgeCase demonstrates an LLM-driven hardware debugging loop designed for live demos and iterative bring-up: - use real UART evidence from the DUT as the single source of truth - run planner/coder/debugger/coordinator/validator roles against that evidence - converge quickly on a stable, passing configuration The goal is to make hardware debugging observable, repeatable, and explainable, not just ad-hoc trial-and-error. ## Architecture block diagram ```text ┌───────────────────────────────────────────┐ │ NVIDIA NIM (local) │ │ http://localhost:8000/v1/chat/completions │ └───────────────▲───────────────────────────┘ │ LLM calls │ ┌──────────────────────────────┐ ┌─────┴─────────────────────────────┐ │ Dashboard (make gui) │ │ orchestrator.py (CLI) │ │ http://127.0.0.1:8765 │ │ - run loop per case │ │ - Planner/Coder/Debugger/... │◄──┤ - emits SSE updates │ │ - Verifier panel + charts │SSE│ - reads/writes run artifacts │ └───────────────▲──────────────┘ └─────┬─────────────────────────────┘ │ user selects case/target│ │ │ invokes │ ▼ │ ┌─────────────────────────┐ │ │ Agents (fan-out/converge)│ │ │ Planner / Coder / Debugger│ │ │ Coordinator / Verifier │ │ └───────────┬──────────────┘ │ │ propose params │ ▼ │ ┌─────────────────────────┐ │ │ runner/ (ONLY hardware) │ │ │ - Build (ELF/UF2) │ │ │ - Flash (picotool/OpenOCD │ │ │ /UF2) │ │ │ - UART capture (/dev/tty*)│ │ │ - Mock mode (synthetic) │ │ └───────────┬──────────────┘ │ │ produces ▼ ▼ ┌────────────────────────────────────────────────────┐ │ Run Evidence Bundle (unchanged contract) │ │ runs/run_x/ │ │ manifest.json firmware.elf firmware.uf2 │ │ uart.log analysis.json triage.md │ └────────────────────────────────────────────────────┘ ``` Real mode path: Runner -> Flash -> RP2350 -> UART -> uart.log Mock/demo path: Runner -> synthetic uart.log + outcomes ## Primary use cases - Interactive demos: show live agent reasoning summaries and UART logs in one dashboard. - Rapid bring-up: validate that firmware boots, emits expected markers, and ends runs cleanly. - Regression checks: replay the same case across multiple runs and compare artifacts. - Safe automation: keep all hardware-touching actions isolated in `runner/`. ## Platform focus: DGX Spark + NIM Primary deployment target is DGX Spark on `Ubuntu 24.04 (ARM64)`: - NIM inference runs locally via Docker (`make nim-start`). - Orchestrator, agents, dashboard, build, flash, and UART capture run on the same host. - Hardware target (RP2350 in this repo) is connected over USB CDC (`/dev/serial/by-id/*` preferred). For real runs, set: - Docker/NGC access for NIM (`NGC_API_KEY`) for local Nemotron inference - target-specific build toolchain (RP2350 example uses `PICO_SDK_PATH` + `arm-none-eabi-gcc`) ## Quick start ```bash make venv make mock ``` Default demo command: ```bash python3 orchestrator.py --case uart_demo --runs 8 ``` ## Make targets - `make mock` / `make demo`: mock run (`uart_demo`, 8 runs) - `make real`: real hardware run (`demo-real`) - `make demo-live`: mock run with live run diagnostics - `make demo-real`: real run with live run diagnostics - `make gui`: start dashboard on `http://127.0.0.1:8765` - `make nim-start`: start local Nemotron Nano 9B NIM container - `make nim-stop`: stop/remove local NIM container - `make nim-smoke`: basic concurrent curl smoke test against NIM ## Truth layer and run artifacts Logic analyzer support is removed. USB CDC UART is the only truth layer. Per-run bundle (`runs/run_*`): - `manifest.json` - `firmware/firmware.elf` - `firmware/firmware.uf2` - `uart.log` - `analysis.json` - `triage.md` ## Runner contract Runner responsibilities: - flash backend auto-detect (`UF2 -> picotool -> OpenOCD`) - serial auto-detect (prefer `/dev/serial/by-id/*`, fallback `/dev/ttyACM*`, `/dev/ttyUSB*`, `/dev/cu.usbmodem*`) - serial re-enumeration handling after flash - timestamp each UART line - capture until `RUN_END ` (or any `RUN_END ...`) or timeout ## Cases - `uart_demo`: baud guess hunt (`guess_baud` vs `target_baud`) - `framing_hunt`: frame guess hunt (`guess_frame` vs `target_frame`) - `parity_hunt`: parity guess hunt (`guess_parity` vs `target_parity`) - `signature_check`: signature semantic check (`guess_magic` vs `target_magic`) You can override targets from CLI: ```bash python3 orchestrator.py --case uart_demo --runs 8 --target-baud 76200 python3 orchestrator.py --case framing_hunt --runs 8 --target-frame 8E1 python3 orchestrator.py --case parity_hunt --runs 8 --target-parity odd python3 orchestrator.py --case signature_check --runs 8 --target-magic 0xC0FFEE42 ``` For `uart_demo`, allowed model-selectable baud candidates are configurable via: - `cases.uart_demo.baud_options_csv` ## Analysis metrics `analysis.json` includes: - `error_count` - `missing_start` - `missing_end` - `lines_per_sec` - `max_gap_ms` - `last_error_code` - `uart_line_count` - `signature_valid` ## Live CLI flags - `--live`: per-run diagnostics + UART tail - `--live-uart`: stream UART lines as captured (`[uart] ...`) - `--trace`: stream short agent reasoning summaries (`[planner]`, `[coder]`, `[critic]`, `[summarizer]`, `[verifier]`) - `--verbose`: enables all live CLI output - `--show-agent-fragments`: print short role output fragments in live mode - `--state-file `: write live state JSON for dashboard Example: ```bash python3 orchestrator.py --case uart_demo --runs 8 --mode mock --live-uart --trace --verbose ``` ## Dashboard Run: ```bash make gui ``` Open `http://127.0.0.1:8765`. UI sections: - Planner, Coder, Debugger, Coordinator panels - Overall Output - Latest UART - Run Tracker - Agent Load / Time chart + Verifier panel (right side) Top controls: - `Case`, `Runs`, `Mode` (`mock`/`real`) - `Agent Mode` (`sequential` tag-team or `parallel`) - `NIM Model` (`Nemotron Nano 9B` or `Nemotron 30B`) - one case-specific target input shown at a time: - baud, frame, parity, or magic API: - `GET /api/stream` (SSE live state/process stream) - `GET /api/state` - `GET /api/process` - `POST /api/run` (also accepts `/api/start`) ## Real hardware mode Real mode requires valid firmware build outputs configured in `config.yaml`: - `runner.build_cmd` - `runner.real_uf2_path` - optional `runner.real_elf_path` - flash backend settings (`runner.flash_method`, optional `runner.openocd_cfg` when using OpenOCD) Default config uses: - `build_cmd: make -C firmware REQUIRE_PICO_SDK=1 rp2350_{case_id}` - `real_uf2_path: firmware/build/firmware.uf2` - `flash_method: picotool` - `auto_bootsel: true` (runner sends `BOOTSEL` command over USB CDC before flashing) If `runner.real_uf2_path` is missing, orchestrator exits with configuration erro [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 28 recognized source files, 176 KB. - C (language) — detected in the code - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (36 of 36) ``` .devcontainer/devcontainer.json .devcontainer/Dockerfile .gitignore agents/__init__.py agents/analyst.py agents/llm_client.py agents/orchestrator_nim.py agents/planner.py agents/triage.py brief.md config.real.example.yaml config.yaml dashboard/server.py firmware/bootsel_helper.h firmware/CMakeLists.txt firmware/Makefile firmware/README.md firmware/rp2350_framing_hunt.c firmware/rp2350_parity_hunt.c firmware/rp2350_signature_check.c firmware/rp2350_uart_demo.c Makefile orchestrator.py pyproject.toml README.md requirements.txt runner/__init__.py runner/cli.py runner/flash.py runner/runner.py runner/serial_capture.py schemas/types.py scripts/demo.py scripts/start_nim_nemotron_9b.sh scripts/stop_nim_nemotron_9b.sh tools/smoke_concurrency.sh ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Add architecture block diagram to README - Flip validator confidence colors and cap NIM tokens at 512 - Tune dashboard agent-load refresh interval to 0.45s - Polish dashboard agent headers and live validator confidence flow - Stream live validator confidence with sparkline and progress bar - Add validator confidence sparkline and coder-audit validation flow - Use unified system memory in GPU panel and remove per-GPU text - Stabilize GPU panel by persisting last good single-GPU sample - Tighten GPU fallback messaging for unified-memory systems - Fill output panes and ignore transient GPU N/A samples - Refine dashboard sizing and stabilize GPU metrics display - Tune dashboard layout and throttle GPU polling to 2s - Add GPU utilization panel and per-run NIM model selector - Make NIM first-class for baud selection and refine multi-agent flow - Use dependency-driven agent flow and common baud dropdown - Reposition validator panel below agent load chart - Add verifier agent and replace agent-calls panel in dashboard - Add blind baud demo hints and richer dashboard call telemetry - Enhance dashboard UX and agent coordination flow - Add agent execution modes, peer messaging rounds, and EdgeCase renaming ## Key source files (fetched from GitHub, selected and truncated for size) ### brief.md ```markdown # EdgeCase Brief (DGX Spark + NVIDIA NIM + Multi-Agent HIL) ## Goal Build a hackathon-ready, LLM-driven hardware-in-the-loop debugger where: - DGX Spark runs the full stack locally. - NVIDIA NIM (Nemotron) provides one shared OpenAI-compatible endpoint. - Multi-agent orchestration drives iterative debug decisions from UART evidence. - `runner/` is the only module allowed to touch hardware. This is a **multi-agent systems demo first**, with RP2350 as the current reference target. ## Core Story Closed-loop debug cycle: 1) Build firmware and flash target 2) Capture UART truth stream 3) Analyze run artifacts 4) Planner/Coder/Debugger/Validator reason over evidence 5) Coordinator emits next experiments and operator guidance 6) Repeat until pass, then stop with a success message ## Environment / Platform - Host: DGX Spark (`Ubuntu 24.04`, `ARM64`) - Local endpoint: - `NIM_CHAT_URL=http://localhost:8000/v1/chat/completions` - `NIM_MODEL=nvidia/nemotron-nano-9b-v2` - No required cloud inference path. - Dashboard + orchestrator + agents + runner all execute on same host. ## Architecture Constraints - `runner/` is the only hardware-touching boundary: - build commands - flash commands - `/dev/tty*` access - USB CDC UART is the only truth layer. - No logic analyzer dependency in current flow. ## Runner Contract Runner must: - auto-detect serial port (`/dev/serial/by-id/*` preferred) - handle re-enumeration after flash - timestamp captured UART lines - stop capture on `RUN_END` or timeout - emit actionable diagnostics on failure Flash strategy (auto-detect, fast failure): 1) UF2 mass-storage copy 2) `picotool` 3) optional OpenOCD ## Artifact Contract (per run) Each run directory under `runs/run__/` must include: - `manifest.json` - `firmware/firmware.elf` - `firmware/firmware.uf2` - `uart.log` - `analysis.json` - `triage.md` No extra bundle files. ## Agent Model Five roles over one NIM endpoint: 1) Planner 2) Coder 3) Critic (UI: Debugger) 4) Verifier (UI: Validator) 5) Summarizer (UI: Coordinator) Behavior requirements: - No hidden chain-of-thought exposure. - Stream short reasoning summaries only: - `Evidence -> Hypothesis -> Next action` - Stream live UART lines to CLI/dashboard. - Prefer dependency-driven scheduling over cosmetic parallelism. ## Dashboard Requirements - Live SSE stream (`GET /api/stream`) - Start runs via `POST /api/run` - Show: - Planner, Coder, Debugger, Coordinator, Validator panes - overall output - latest UART - run tracker - agent load/time chart ## Demo/Real Modes - `demo` (mock): full pipeline without hardware - `real`: build/flash/capture against connected hardware Commands that must work: - `make demo` - `python3 orchestrator.py --case uart_demo --runs 8` ## Current Demo Cases - `uart_demo` (baud hunt, blind-first strategy) - `framing_hunt` - `parity_hunt` - `signature_check` ## Success Criteria - Pipeline runs end-to-end in demo mode with clear agent activity. - Real mode giv [truncated — 196 more characters] ``` ### requirements.txt ``` # No third-party dependencies required. ``` ### pyproject.toml ``` [build-system] requires = ["setuptools>=68", "wheel"] build-backend = "setuptools.build_meta" [project] name = "edgecase" version = "0.1.0" description = "EdgeCase: multi-agent HIL debugger for RP2350 UART with local NIM endpoint" requires-python = ">=3.10" dependencies = [] [tool.setuptools] packages = ["runner", "agents"] ``` ### .devcontainer/Dockerfile ``` FROM mcr.microsoft.com/devcontainers/python:3.11 WORKDIR /workspace ``` ### runner/cli.py ```python from __future__ import annotations import argparse import json from runner import Runner, RunnerConfig def main() -> None: parser = argparse.ArgumentParser(description="Execute one HIL runner iteration") parser.add_argument("--case", default="uart_demo") parser.add_argument("--run-index", type=int, default=1) parser.add_argument("--mode", choices=["mock", "real"], default="mock") parser.add_argument("--params", default='{"uart_rate": 1000000, "buffer_size": 16}') parser.add_argument("--serial-port", default="") parser.add_argument("--serial-baud", type=int, default=115200) parser.add_argument("--build-cmd", default="") parser.add_argument("--build-cwd", default=".") parser.add_argument("--real-elf-path", default="") parser.add_argument("--real-uf2-path", default="") args = parser.parse_args() params = json.loads(args.params) runner = Runner( RunnerConfig( serial_port=args.serial_port, serial_baud=args.serial_baud, build_cmd=args.build_cmd, build_cwd=args.build_cwd, real_elf_path=args.real_elf_path, real_uf2_path=args.real_uf2_path, ) ) result = runner.execute(case_id=args.case, run_index=args.run_index, params=params, mode=args.mode) print(json.dumps(result, indent=2)) if __name__ == "__main__": main() ``` ### dashboard/server.py ```python from __future__ import annotations import json import os import signal import subprocess import sys import threading import time from http import HTTPStatus from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path from urllib.parse import urlparse ROOT = Path(__file__).resolve().parents[1] STATE_PATH = ROOT / "dashboard" / "state.json" LOG_PATH = ROOT / "dashboard" / "orchestrator.log" PROCESS: subprocess.Popen[str] | None = None PROCESS_PAUSED = False LOCK = threading.Lock() GPU_CACHE: dict[str, object] = { "ts": 0.0, "has_good": False, "data": {"available": False, "message": "Waiting for GPU metrics..."}, "last_good": {}, } GPU_UNIFIED_MEM_NOTE = ( "Unified memory expected on DGX Spark. nvidia-smi may not report dedicated VRAM usage.\n" "Use: top, htop, free" ) DGX_SPARK_MAX_POWER_W = 140.0 HTML = """ EdgeCase Dashboard