# Project export: Improving Autonomous Vehicles This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: What if the car had chosen differently? - Devpost: https://devpost.com/software/improving-autonomous-vehicles - GitHub: https://github.com/Steve-Dusty/overflow - Video: https://www.youtube.com/embed/tgMRUUJN_UE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Steve-Dusty (2 commits), Claude Opus 4.8 (1M context) (1 commits) ## Devpost submission (written by the team) ### Overview What if the car had chosen differently? ### Inspiration You can't drive your way to safety. RAND showed self-driving cars would need hundreds of millions — sometimes hundreds of billions — of miles to statistically prove they're safe, because the moments that actually matter (a jaywalker, a near-miss, the half-second a kid steps off the curb) almost never happen per mile. No fleet can drive far enough to catch them all. So the industry simulates instead. But the world models that do it live inside Waymo and Nvidia, behind proprietary fleets and TPU clusters. We wanted the version a small team can actually run and inspect: take one logged scene, branch it into the futures that didn't happen, and let a human teach the policy which future was better. RLHF for driving, not chatbots — the same GRPO that taught reasoning models to think, pointed at a steering wheel. ### What it does Improving Autonomous Cars turns a single driving log into a training signal. Pick a scenario — "Near Miss," "Jaywalker" — and watch the ego car drive it in 3D: live LiDAR, bounding boxes, and an Autonomy Stack panel streaming the model's actions and rewards, with an "Explain last decision" button for the reasoning behind each move. Then the counterfactuals kick in — every 10 seconds the dashboard spawns three new sims where the ego chose differently, each drawn as a top-down map of where that decision led. A human ranks those rollouts. A reward model learns the ranking. GRPO trains the policy against it — and the trick is that each scenario's counterfactual set is exactly the "group" GRPO needs for its baseline. A 3D knowledge graph wires every scenario, run, action, and reward together so you can trace why a decision happened, and an analytics page tracks rewards, runs, and incidents over time. Above all of it runs a governed multi-agent safety system: perception, planner, and safety-auditor agents that coordinate in a shared room, run on a durable runtime, and operate under cryptographic intent enforcement — so the agents auditing a driving decision can't go rogue while doing it. ### How we built it Frontend. React 19, TypeScript, Vite. Three.js + React Three Fiber for the 3D sim and LiDAR, Zustand for state, react-force-graph-3d for the knowledge graph, and Hyparquet to read Waymo parquet files straight in the browser. Backend. An Express server that proxies the OpenAI reasoning calls server-side, fully traced. Training. A Python pipeline: a Bradley-Terry reward model over human-ranked rollouts, a GRPO trainer that uses each scene's counterfactual set as its group, and an orchestrator tying them together. Durable agent runtime — Orkes AgentSpan. The long-running work — the counterfactual rollout fleet and the GRPO/reward-model training jobs — runs as durable workflows on AgentSpan. Execution state lives on the server, tool calls retry on failure, and if a process dies the run resumes from the exact step instead of restarting. Crucially, the human reranker is an AgentSpan human-in-the-loop approval gate: the workflow pauses with no timeout, holds state on the server, and waits for a human to rank the rollouts before training continues — exactly the primitive AgentSpan is built for. Agent collaboration — Band. The perception, planner, and safety-auditor agents don't call each other directly. They live in a shared Band room and coordinate by @mention with deterministic (non-LLM) routing, so context stays synchronized and every exchange lands in one audit trail. If one agent misinforms another, Band's control plane catches the cascade instead of letting a bad safety verdict propagate. Intent governance — ArmorIQ. Before the autonomous safety agent acts, ArmorIQ captures its plan, compiles it into a signed Canonical Structured Reasoning Graph, and issues a short-lived intent token with per-step cryptographic proofs. Every action is checked against that signed plan; anything that drifts outside it is blocked at the gate, fail-closed. It governs the agent by intent, not just credentials — which is what you want before an autonomous agent gets anywhere near a driving policy. Observability — Sentry. Full-stack distributed tracing front to back, error boundaries, and performance profiling, with the backend doubling as the Sentry tunnel — so the whole system is traceable instead of a black box. ### Challenges we ran into Counterfactual realism. The instant the ego leaves the logged path, you have no ground-truth sensor data for where it went — the hardest unsolved problem in the field. We used synthetic generation to keep divergent rollouts coherent, and learned firsthand why the frontier pours money into generative world models here. Learning the reward instead of writing it. "Good driving" is too fuzzy to hand-code without going brittle, so the whole point became learning it from human preference — building the ranking → reward-model → GRPO chain end to end and getting the signal to actually move the policy. An intent-governance SDK that fought back. Integrating ArmorIQ surfaced real bugs in its intent-verification layer — a verifyToken() that returned true even when the signed planHash had been tampered with (an integrity hole in the exact thing the SDK exists to protect), and a delegate() that was dead against the live backend. We shipped workarounds and wrote up all eight findings with root-cause traces. Backend from scratch, mid-hackathon. The app started frontend-only; the LLM proxy and the Sentry tunnel meant standing up a server tier under the clock. ### Accomplishments we're proud of The loop actually closes: log → counterfactual → human rank → GRPO → knowledge graph. Not a diagram of it — a running version. Four sponsor integrations that each own a real subsystem, not logos bolted on: AgentSpan runs the durable pipeline and holds the human-rank approval gate, Band is the agents' shared room and audit trail, ArmorIQ is the cryptographic intent gate on the safety agent, and Sentry is full-stack tracing across the whole thing. We found and documented 8 real bugs (2 high-severity) in a production intent-assurance SDK while wiring it in — with honest triage separating the SDK's faults from our own pre-existing dependencies. We can say exactly where we sit next to Waymo, and built the honest, transparent version of a technique the frontier keeps locked up. ### What we learned The RLHF/GRPO playbook ports cleanly from language to driving: a scene's counterfactual rollouts are the GRPO group, and a human ranking them is the preference signal. AV safety is a long-tail and reward-specification problem far more than an average-driving one — both squarely in human-feedback RL's wheelhouse. Counterfactual realism breaks the moment trajectories diverge; reconstructive methods can't follow, which is why generative world models are the real frontier. Multi-agent safety is three different problems — coordination (Band), durable execution with human checkpoints (AgentSpan), and intent enforcement (ArmorIQ) — and they don't collapse into one tool. Humility: Waymo's world model, the open-source Waymax simulator, and Wayve's GAIA-1 are years ahead. Our edge is transparency and access, not scale. ### What's next for Improving Autonomous Cars Real closed-loop data — swap synthetic scenes for the Waymo Open Motion Dataset via Waymax, so the counterfactuals are grounded in real driving. A generative world model so divergent rollouts stay realistic once the ego leaves the logged path. Active preference collection — surface the most informative rollouts to rank, so every human label trains the reward model harder. A continuous flywheel — re-rank as the policy shifts so the reward model never goes stale. From advisory to audit-and-veto — combine ArmorIQ's fail-closed intent gate with AgentSpan's approval checkpoints so the safety system can actually override an unsafe policy, not just flag it. ## README (from the GitHub repository) # Improving Autonomous Vehicles Multi-sim dashboard for autonomous vehicle perception. Replays Waymo scenes in 3D, proposes ego actions via the OpenEnv model, and continuously spawns counterfactual rollouts to compare "what if the ego chose differently?" ## Quick Start ```bash npm install npm run dev ``` Open `http://localhost:5173`. The app defaults to mock data mode (no Waymo files needed). ## Pages | Route | Description | |-------|-------------| | `/sim` | Main 3D simulator. LiDAR point cloud, bounding boxes, ego vehicle. Autonomy Stack panel shows OpenEnv actions/rewards in real time. | | `/dashboard` | Camera-grid multi-sim view. Ground truth + auto-spawning counterfactual rollouts. Every 10s, 3 new sims appear with different ego decisions. | | `/graph` | 3D knowledge graph (ForceGraph3D). Connects scenarios, incidents, runs, actions, metrics, and rewards. Click any node to inspect. | | `/analytics` | Overview cards, sortable run table, reward timeline chart, incident ticket feed. | ## OpenEnv Configuration The OpenEnv model provides actions and rewards for the ego vehicle. **Mock mode** (default): Deterministic pseudo-random actions based on scene context. No external API needed. **Real mode**: Set an environment variable to point at your OpenEnv endpoint: ```bash VITE_OPENENV_ENDPOINT=http://localhost:8080/predict VITE_OPENENV_MODE=real ``` Then in your code, call `configureOpenEnv({ mode: "real", endpoint: import.meta.env.VITE_OPENENV_ENDPOINT })`. The client module lives at `src/lib/openenvClient.ts` and exposes: - `getActionAndReward(input)` — single action/reward query - `getCounterfactualVariants(input, count)` — N variant actions for branching ## Waymo Data Place Waymo Open Dataset parquet files in `public/waymo_data/`: - `vehicle_pose.parquet` - `lidar.parquet` - `lidar_box.parquet` (optional) - `lidar_calibration.parquet` Or drag and drop files directly onto the simulator. ## Demo Script 1. Open `http://localhost:5173/sim` 2. The sim loads with mock data. Use the scenario selector (top-left) to pick "Near Miss" or "Jaywalker" 3. Press Play. Watch the Autonomy Stack panel (right) update every 3s with OpenEnv actions/rewards 4. Click "Explain last decision" to see the model's reasoning 5. Navigate to `/dashboard` — the camera grid auto-populates with counterfactual sims 6. Watch new tiles appear every 10s. Each shows a 2D top-down mini-map of the ego's divergent trajectory 7. Click any tile to see full metrics and action stream 8. Go to `/graph` — the knowledge graph connects runs, actions, metrics, and rewards. Click nodes to inspect 9. Go to `/analytics` — overview cards, sortable table, timeline chart, and incident feed ## Tech Stack - React 19 + TypeScript + Vite - Three.js + React Three Fiber (3D rendering) - Zustand (state management) - react-force-graph-3d (knowledge graph) - Sonner (toast notifications) - Lucide React (icons) - Hyparquet (browser-native Parquet reader) ## Project Structure ``` src/ pages/ SimPage, DashboardPage, GraphPage, AnalyticsPage components/ Scene3D, Timeline, and existing 3D components (kept intact) components/ui/ AppShell, Card, Badge (design system) lib/ openenvClient, simManager, simTypes utils/ parquet, waymoLoader, rangeImage, trajectoryData, scenarioAI store.ts Zustand global state theme.ts Design tokens (colors, typography, spacing) mockData.ts Synthetic scenario generation + LiDAR raytracing ``` ## Agent stack — ArmorIQ · AgentSpan · Band Overflow runs an AV-safety agent fleet on three layers of agent infrastructure: - **ArmorIQ** (`scripts/armoriq_*.mjs`) — governance: every tool call gated by a signed intent token. `npm run agent`, `npm run fleet`. - **AgentSpan** (`scripts/agentspan_*.py`) — durable execution: crash-resume, retries, structured output, guardrails, human approval. `npm run agentspan`, `npm run agentspan:fleet`. - **Band** (`scripts/band_agents.py`) — cross-agent discovery + `@mention` coordination. `npm run band -- auditor`. Python setup (once): `uv venv .venv && uv pip install -r requirements-agents.txt` — the agents reuse the OpenAI key in `server/.env`. See **[AGENT_STACK.md](AGENT_STACK.md)** for the full story and run guide. ## Detected evidence (automated analysis) Indexed codebase: 85 recognized source files, 884 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - Next.js (technology) — claimed on Devpost, not found in the code - PyTorch (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (116 of 116) ``` .env.example .gitignore AGENT_STACK.md ARMORIQ_SDK_BUGS.md band_agent_config.example.yaml checkpoints/agentspan_safety_report.md checkpoints/safety_agent_report.md data_tools/download_e2e_subset.sh data_tools/extract_e2e_to_json.py data_tools/generate_sample.py data_tools/requirements_extractor.txt data_tools/sample_e2e.json eslint.config.js index.html package.json public/demo_data/armoriq_analytics.json public/demo_data/overflow_demo_dataset.json public/demo_data/waymo_scene_final_model.json public/demo_data/waymo_scene_jaywalker.json public/demo_data/waymo_scene_near_miss.json public/demo_data/waymo_scene_normal.json public/demo_data/waymo_scene_rear_end.json public/demo_data/waymo_scene_red_light.json public/demo_data/waymo_scene_swerving.json public/models/car.glb public/models/CREDITS.md public/models/cyclist.glb public/models/person.glb public/models/sign.glb README.md requirements-agents.txt scripts/agentspan_agent.py scripts/agentspan_fleet.py scripts/agentspan_hitl.py scripts/agentspan_resume.py scripts/agentspan_test.py scripts/agentspan_tools.py scripts/armoriq_agent.mjs scripts/armoriq_fleet.mjs scripts/band_agents.py scripts/find_incidents.mjs scripts/generate-scenarios.ts scripts/inspect_box_schema.mjs scripts/scan_incidents.mjs scripts/sentry_obs.py scripts/train_full_pipeline.py scripts/train_grpo.py scripts/train_openenv.py scripts/train_reward_model.py server/.gitignore server/index.mjs server/instrument.mjs server/package.json src/App.tsx src/components/AnalyticsPanel.tsx src/components/ArmorIQStatus.tsx src/components/BoundingBoxes.tsx src/components/CameraViews.tsx src/components/ControlPanel.tsx src/components/DataBrowser.tsx src/components/EgoVehicle.tsx src/components/FrameOverrideContext.ts src/components/IncidentAlert.tsx src/components/InfoBar.tsx src/components/KnowledgeGraph.tsx src/components/ObjectModels.tsx src/components/PointCloud.tsx src/components/ScenarioChat.tsx src/components/Scene3D.tsx src/components/SceneBoundary.tsx src/components/TicketConsole.tsx src/components/Timeline.tsx src/components/ToastNotifications.tsx src/components/TrajectoryControls.tsx src/components/TrajectoryViewer.tsx src/components/ui/AppShell.tsx src/components/ui/Badge.tsx src/components/ui/Card.tsx src/index.css src/instrument.ts src/lib/armoriq.ts src/lib/incidentDetector.ts src/lib/llmClient.ts src/lib/mockTraining.ts src/lib/openenvClient.ts src/lib/sentry.ts src/lib/simManager.ts src/lib/simTypes.ts src/lib/types.ts src/main.tsx src/mockData.ts src/pages/AnalysisPage.tsx src/pages/AnalyticsPage.tsx src/pages/ArmorIQPage.tsx src/pages/ComparePage.tsx src/pages/DashboardPage.tsx src/pages/ExportPage.tsx src/pages/GraphPage.tsx src/pages/RankPage.tsx src/pages/SimPage.tsx src/pages/TrainPage.tsx src/pages/UploadPage.tsx src/store.ts src/theme.ts src/utils/parquet.ts src/utils/rangeImage.ts src/utils/scenarioAI.ts src/utils/scenarioLoader.ts src/utils/sceneCache.ts src/utils/trajectoryData.ts src/utils/waymoLoader.ts src/vite-env.d.ts tsconfig.app.json tsconfig.json tsconfig.node.json vite.config.ts ``` ### Dependencies - package.json: @armoriq/sdk@^0.3.8, @eslint/js@^9.39.1, @react-three/drei@^10.7.7, @react-three/fiber@^9.5.0, @sentry/react@^10.59.0, @sentry/vite-plugin@^5.3.0, @types/node@^24.10.1, @types/react@^19.2.7, @types/react-dom@^19.2.3, @types/three@^0.183.1, @vitejs/plugin-react@^5.1.1, eslint@^9.39.1, eslint-plugin-react-hooks@^7.0.1, eslint-plugin-react-refresh@^0.4.24, globals@^16.5.0, hyparquet@^1.25.1, hyparquet-compressors@^1.1.1, lucide-react@^0.577.0, react@^19.2.0, react-dom@^19.2.0, react-force-graph-3d@^1.29.1, react-router-dom@^7.13.1, sonner@^2.0.7, three@^0.183.2, tsx@^4.21.0, typescript@~5.9.3, typescript-eslint@^8.48.0, vite@^7.3.1, zustand@^5.0.11 - server/package.json: @sentry/node@^10.59.0, @sentry/profiling-node@^10.59.0, cors@^2.8.5, dotenv@^17.2.0, express@^4.21.2, openai@^4.77.0 ### Recent commits (newest first) - Rename project display name → Improving Autonomous Vehicles - Sentry observability + instrumented OpenAI backend; scenario data (combined session snapshot) - Overflow — multi-sim AV perception dashboard + RLHF/GRPO training pipeline ## Key source files (fetched from GitHub, selected and truncated for size) ### AGENT_STACK.md ```markdown # The Overflow Agent Stack Overflow isn't just an AV-perception dashboard — it runs a **fleet of AI agents** that review autonomous-vehicle safety. Those agents sit on three complementary pieces of agent infrastructure, each answering a different hard question: | Layer | Question it answers | SDK | Where | |-------|--------------------|-----|-------| | **ArmorIQ** | *Is this agent allowed to do this?* — governance | `@armoriq/sdk` | `scripts/armoriq_*.mjs` | | **AgentSpan** | *Does the work survive a crash?* — durability | `agentspan` | `scripts/agentspan_*.py` | | **Band** | *Can agents find each other and collaborate?* — coordination | `band-sdk` | `scripts/band_agents.py` | One domain — AV safety over the Waymo demo scenes (`public/demo_data/`) — made **governed**, **durable**, and **collaborative**. All three stacks share the same domain logic (`scripts/agentspan_tools.py`), so they operate on one set of tools. ## The story in five acts 1. **Perception.** The perception agent classifies every road agent in each scene (vehicles, pedestrians, cyclists). 2. **Risk audit.** The safety auditor scores each scenario's worst incident on a 0–1 collision-risk scale — on the real data: **Jaywalker 0.86, Red-light 0.84, Rear-end 0.83**. It returns a *typed, validated* `FleetRiskReport` (AgentSpan **structured output**). 3. **Planning.** The planner proposes a safe ego maneuver per high-risk scenario (jaywalker → `emergency_brake`, rear_end → `brake_and_widen_gap`, …). 4. **Policy, gated.** The policy agent drafts updates and, for the worst scenario, calls the **sensitive** `override_safety_limit` and `deploy_policy` tools. These are `approval_required` — AgentSpan **pauses for a human** before they run — and a **guardrail** (`no_unsafe_policy`) makes it impossible for the agent to ever recommend disabling safety. 5. **Durability.** Every step is a Conductor workflow task with state on the server. If the process dies mid-review, `agentspan_resume.py` reconnects by execution-id and finishes from the exact step. *The process dies; the agent doesn't.* Meanwhile **Band** makes the same agents discoverable: the auditor doesn't call the planner through hard-coded wiring — it `@mentions` `overflow-planner-agent`, and Band routes by description. A new agent can join the review just by registering a description. ## Run it ```bash # one-time setup (agents reuse the OpenAI key in server/.env) uv venv .venv && uv pip install -r requirements-agents.txt npm run agentspan # durable single safety agent (verified: COMPLETED) npm run agentspan:fleet # full durable pipeline (verified: COMPLETED) npm run agentspan:resume # crash & resume-by-id demo npm run agentspan:hitl # sensitive action pauses for human approval (verified) npm run agentspan:test # deterministic test, no LLM/server npm run agentspan:tui # inspect the runtime (or http://localhost:6767) npm run band -- auditor --selfte [truncated — 1520 more characters] ``` ### ARMORIQ_SDK_BUGS.md ```markdown # `@armoriq/sdk` — Bug & Issue Report **Package:** `@armoriq/sdk@0.3.8` (the version that `npm install @armoriq/sdk` resolved to) **Reported:** 2026-06-20 **Backend:** production (`iap.armoriq.ai` / `proxy.armoriq.ai` / `api.armoriq.ai`), `ak_live_*` key **How tested:** the Overflow agent + fleet (`npm run agent`, `npm run fleet`) plus two throwaway probe scripts that exercised every public client method against the live backend. All file:line references are into the installed `node_modules/@armoriq/sdk/dist/`. > Scope note on accuracy: `npm audit` reports 11 vulnerabilities for this project, but **only one is attributable to this SDK** (see §Security). The other ten are the app's pre-existing dev dependencies (vite, react-router, esbuild, …) and are **not** ArmorIQ's. Likewise, one earlier "bug" we saw turned out to be our own probe passing wrong params — documented under §Not-a-bug so it isn't double-counted. --- ## Summary | # | Severity | Area | Issue | Workaround | |---|----------|------|-------|------------| | 1 | **High** | Plan Assurance | `verifyToken()` returns `true` for a token whose `planHash` was mutated | Don't treat `verifyToken` as tamper-detection; re-check `planHash` yourself | | 2 | **High** | Delegation | `delegate()` always throws against the live backend (response-shape mismatch) | Use `delegateSubtree()` | | 3 | **Medium** | Delegation | `createDelegationRequest()` TS type marks `arguments`/`amount` optional, but the backend requires them | Always pass `arguments: {}` and a valid `amount` | | 4 | **Medium** | DX / logging | 43 unconditional `console.*` calls; no `silent`/`logger` option | Monkey-patch `console` or filter stdout | | 5 | **Low** | Packaging | Node-only (axios + Node `crypto`, secret key) but nothing stops you importing it client-side | Keep it server-side only | | 6 | **Low** | Security (deps) | `js-yaml@4.1.1` pulls a moderate DoS advisory | Only hit if you call `fromConfig()`; avoid or override | | 7 | **Low** | Consistency | unregistered `defaultMcpName` accepted silently; `getMcpToolSchemas` 404s but `resolveRole` succeeds for the same name | Register MCPs before referencing them | | 8 | **Low** | Logging hygiene | logs the last 8 chars of the secret API key to stdout | n/a (cosmetic, but a log-leak) | --- ## Detailed findings ### 1. `verifyToken()` does not detect a tampered token object — **High** **What:** `verifyToken()` is the SDK's "Plan Assurance" check, but it returns `true` even when the `IntentToken`'s `planHash` has been replaced with garbage. **Repro:** ```js const token = await session.startPlan(plan, goal) await client.verifyToken(token) // → true (expected) await client.verifyToken({ ...token, planHash: 'deadbeef'.repeat(8) }) // → true ❌ ``` **Observed:** `true` for both. **Expected:** the tampered token should fail (or the method's contract should be documented as "validates the signed JWT only, not the client-side object fields"). **Likely [truncated — 6373 more characters] ``` ### package.json ``` { "name": "improving-autonomous-vehicles", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "lint": "eslint .", "preview": "vite preview", "generate-scenarios": "tsx scripts/generate-scenarios.ts", "agent": "node scripts/armoriq_agent.mjs", "fleet": "node scripts/armoriq_fleet.mjs", "agentspan": "./.venv/bin/python scripts/agentspan_agent.py", "agentspan:fleet": "./.venv/bin/python scripts/agentspan_fleet.py", "agentspan:hitl": "./.venv/bin/python scripts/agentspan_hitl.py", "agentspan:resume": "./.venv/bin/python scripts/agentspan_resume.py", "agentspan:test": "./.venv/bin/python scripts/agentspan_test.py", "agentspan:server": "./.venv/bin/agentspan server start", "agentspan:tui": "./.venv/bin/agentspan tui", "band": "./.venv/bin/python scripts/band_agents.py" }, "dependencies": { "@armoriq/sdk": "^0.3.8", "@react-three/drei": "^10.7.7", "@react-three/fiber": "^9.5.0", "@sentry/react": "^10.59.0", "@sentry/vite-plugin": "^5.3.0", "hyparquet": "^1.25.1", "hyparquet-compressors": "^1.1.1", "lucide-react": "^0.577.0", "react": "^19.2.0", "react-dom": "^19.2.0", "react-force-graph-3d": "^1.29.1", "react-router-dom": "^7.13.1", "sonner": "^2.0.7", "three": "^0.183.2", "zustand": "^5.0.11" }, "devDependencies": { "@eslint/js": "^9.39.1", "@types/node": "^24.10.1", "@types/react": "^19.2.7", "@types/react-dom": "^19.2.3", "@types/three": "^0.183.1", "@vitejs/plugin-react": "^5.1.1", "eslint": "^9.39.1", "eslint-plugin-react-hooks": "^7.0.1", "eslint-plugin-react-refresh": "^0.4.24", "globals": "^16.5.0", "tsx": "^4.21.0", "typescript": "~5.9.3", "typescript-eslint": "^8.48.0", "vite": "^7.3.1" } } ``` ### server/package.json ``` { "name": "overflow-server", "private": true, "type": "module", "version": "0.0.0", "description": "Overflow backend — instrumented OpenAI proxy + Sentry distributed tracing/tunnel.", "scripts": { "start": "node --import ./instrument.mjs index.mjs", "dev": "node --watch --import ./instrument.mjs index.mjs" }, "dependencies": { "@sentry/node": "^10.59.0", "@sentry/profiling-node": "^10.59.0", "cors": "^2.8.5", "dotenv": "^17.2.0", "express": "^4.21.2", "openai": "^4.77.0" } } ``` ### src/main.tsx ```typescript // MUST be first: initializes Sentry before any other module runs (Sentry React skill). import "./instrument"; import { StrictMode } from "react"; import { createRoot } from "react-dom/client"; import { BrowserRouter } from "react-router-dom"; import * as Sentry from "@sentry/react"; import "./index.css"; import App from "./App"; import { colors, fonts } from "./theme"; function RootFallback({ error }: { error: unknown }) { return (
Something went wrong
{error instanceof Error ? error.message : String(error)}
); } createRoot(document.getElementById("root")!, { // React 19 error hooks (Sentry React skill). We report uncaught + recoverable // errors here; errors *caught* by our ErrorBoundaries are already reported by // those boundaries, so onCaughtError is intentionally omitted to avoid double-counting. onUncaughtError: Sentry.reactErrorHandler(), onRecoverableError: Sentry.reactErrorHandler(), }).render( }> , ); ``` ### server/index.mjs ``` /** * Overflow backend — a small, instrumented Express server that: * 1. Proxies OpenAI chat completions (key stays server-side, off the browser), * wrapping each call in a Sentry `gen_ai` span with model + token usage → * shows up in Sentry's AI/LLM monitoring + as a child of the browser's * distributed trace. * 2. Mocks the OpenEnv policy (/api/predict) so the sim loop produces * server-side inference spans too. * 3. Tunnels browser Sentry envelopes (/api/tunnel) to bypass ad-blockers. * * Sentry init happens in instrument.mjs (loaded via `node --import`). */ import express from "express"; import cors from "cors"; import * as Sentry from "@sentry/node"; import OpenAI from "openai"; const PORT = process.env.PORT || 8787; // SSRF guard: the tunnel only forwards to Sentry ingest hosts. const SENTRY_INGEST_HOST = /(^|\.)ingest\.(\w+\.)?sentry\.io$/; const apiKey = process.env.OPENAI_API_KEY || ""; const hasOpenAI = apiKey.length > 20; const openai = new OpenAI({ apiKey }); const app = express(); app.use(cors()); app.use(express.json({ limit: "1mb" })); // --------------------------------------------------------------------------- // Health // --------------------------------------------------------------------------- app.get("/api/health", (_req, res) => { res.json({ ok: true, openai: hasOpenAI, sentry: Boolean(process.env.SENTRY_DSN_SERVER || process.env.SENTRY_DSN), }); }); // --------------------------------------------------------------------------- // Sentry tunnel — forward browser envelopes so ad-blockers can't drop events. // --------------------------------------------------------------------------- app.post("/api/tunnel", express.text({ type: () => true, limit: "1mb" }), async (req, res) => { try { const envelope = req.body; const header = JSON.parse(envelope.split("\n")[0]); const dsn = new URL(header.dsn); if (!SENTRY_INGEST_HOST.test(dsn.host)) { return res.status(400).json({ error: "untrusted dsn host" }); } const projectId = dsn.pathname.replace(/^\//, ""); const upstream = `https://${dsn.host}/api/${projectId}/envelope/`; const r = await fetch(upstream, { method: "POST", body: envelope, headers: { "Content-Type": "application/x-sentry-envelope" }, }); res.status(r.status).send(await r.text()); } catch { res.status(400).json({ error: "bad envelope" }); } }); // --------------------------------------------------------------------------- // AI chat proxy — instrumented gen_ai span with token usage. // --------------------------------------------------------------------------- async function chat({ system, user, model = "gpt-4o-mini", temperature = 0.7, maxTokens = 1500 }) { return Sentry.startSpan( { op: "gen_ai.chat", name: `chat ${model}`, attributes: { "gen_ai.system": "openai", "gen_ai.operation.name": "chat", "gen_ai.request.model": model, "gen_ai.request.temperature": temperature, "gen_ai.request.max_tokens": maxTokens, }, }, async (span) => { const completion = await openai.chat.completions.create({ model, temperature, max_tokens: maxTokens, messages: [ { role: "system", content: system }, { role: "user", content: user }, ], }); const u = completion.usage; if (u) { span.setAttribute("gen_ai.usage.input_tokens", u.prompt_tokens); span.setAttribute("gen_ai.usage.output_tokens", u.completion_tokens); span.setAttribute("gen_ai.usage.total_tokens", u.total_tokens); } span.setAttribute("gen_ai.response.model", completion.model); return { text: completion.choices?.[0]?.message?.content ?? "", usage: u ?? null, model: completion.model, }; }, ); } app.post("/api/chat", async (req, res, next) => { try { if (!hasOpenAI) return res.status(503).json({ error: "no_api_key" }); const { system, user, model, temperature, maxTokens } = req.body ?? {}; if (!system || !user) return res.status(400).json({ error: "system and user are required" }); Sentry.logger?.info?.("ai.chat.request", { model: model ?? "gpt-4o-mini" }); const out = await chat({ system, user, model, temperature, maxTokens }); res.json(out); } catch (err) { next(err); // hand to Sentry's express error handler } }); // --------------------------------------------------------------------------- // Mock OpenEnv policy — server-side inference span (distributed tracing demo). // --------------------------------------------------------------------------- const ACTIONS = [ "keep_lane", "brake_mild", "brake_hard", "accelerate", "merge_left", "merge_right", "yield", "nudge_left", "nudge_right", ]; app.post("/api/predict", (req, res) => { const { nearestObjectDist = 20, frameIndex = 0, scenarioId = "unknown" } = req.body ?? {}; Sentry.startSpan( { op: "openenv.predict", name: "policy.getActionAndReward", attributes: { scenario: scenarioId, frame: frameIndex } }, (span) => { const close = nearestObjectDist < 5; const action = close ? (Math.random() < 0.7 ? "brake_hard" : "yield") : ACTIONS[Math.floor(Math.random() * 4)]; const reward = close ? 0.5 + Math.random() * 0.4 : 0.6 + Math.random() * 0.3; span.setAttribute("action", action); span.setAttribute("reward", reward); res.json({ action, reward: Math.round(reward * 1000) / 1000, branchId: `srv-${frameIndex}`, explanation: "Server-side OpenEnv policy (mock).", timestamp: Date.now(), latencyMs: 0, }); }, ); }); // Sentry's express error handler (only if initialized), then a JSON fallback. if (Sentry.getClient()) { Sentry.setupExpressErrorHandler(app); } app.use((err, _req, res, _next) => { res.status(500).json({ error: err?.message || "internal error" }); [truncated — 213 more characters] ``` ### src/App.tsx ```typescript /** * App — Router + data loading + app shell. */ import { useEffect, useState, useCallback, useRef } from "react"; import { Routes, Route, Navigate } from "react-router-dom"; import { Toaster, toast } from "sonner"; import AppShell from "./components/ui/AppShell"; import SimPage from "./pages/SimPage"; import DashboardPage from "./pages/DashboardPage"; import GraphPage from "./pages/GraphPage"; import AnalyticsPage from "./pages/AnalyticsPage"; import { lazy, Suspense } from "react"; const UploadPage = lazy(() => import("./pages/UploadPage")); const RankPage = lazy(() => import("./pages/RankPage")); const ArmorIQPage = lazy(() => import("./pages/ArmorIQPage")); import { useStore } from "./store"; import type { DataSource } from "./store"; import { ALL_SCENARIOS } from "./mockData"; import type { ScenarioId } from "./mockData"; import { loadScenario, preloadAllScenarios } from "./utils/scenarioLoader"; import { generateTrajectoryMoments } from "./utils/trajectoryData"; import { loadWaymoFromUrls, loadWaymoFromFiles, scanDroppedFiles, type WaymoLoadResult, } from "./utils/waymoLoader"; import { getCachedScene, setCachedScene, cacheKey } from "./utils/sceneCache"; import { colors, fonts, typeScale } from "./theme"; import { captureError, setSentryScenario, Sentry } from "./lib/sentry"; // Route-aware tracing wrapper for react-router v7 (parameterized transactions // like "/sim" instead of raw URLs). No-op when Sentry isn't initialized. const SentryRoutes = Sentry.withSentryReactRouterV7Routing(Routes); // --------------------------------------------------------------------------- // Auto-detect waymo data layout // --------------------------------------------------------------------------- async function detectWaymoLayout( basePath: string, overrideSegment?: string | null, ): Promise<{ basePath: string; segmentName?: string }> { if (overrideSegment) { return { basePath, segmentName: overrideSegment }; } try { const resp = await fetch(`${basePath}/manifest.json`); if (resp.ok) { const manifest = await resp.json(); const segId = manifest.segment; if (segId) return { basePath, segmentName: segId }; } } catch { /* no manifest */ } try { const resp = await fetch(`${basePath}/vehicle_pose.parquet`, { method: "HEAD" }); if (resp.ok) return { basePath }; } catch { /* not flat */ } throw new Error("No Waymo data found. Place parquet files in public/waymo_data/ or drag & drop."); } // --------------------------------------------------------------------------- // Scenario pre-generation hook — eagerly generates all scenarios on mount // --------------------------------------------------------------------------- function useScenarioPreloader() { const [ready, setReady] = useState(false); const actions = useStore((s) => s.actions); useEffect(() => { const defaultScenario: ScenarioId = "normal"; actions.setLoadStatus("loading"); actions.setLoadMessage("Loading scenario…"); loadScenario(defaultScenario, "ground_truth", (msg, progress) => { actions.setLoadMessage(msg); actions.setLoadProgress(progress); }) .then((sceneData) => { actions.setScenarioId(defaultScenario); actions.setDataSource("scenario"); actions.setSceneData(sceneData); const moments = generateTrajectoryMoments(sceneData); actions.setTrajectoryMoments(moments); setReady(true); // Pre-fetch remaining scenarios in background const remaining = ALL_SCENARIOS.filter((s) => s !== defaultScenario); preloadAllScenarios(remaining); }) .catch((e) => { console.error("[preloader] Failed to load scenario:", e); captureError(e, { tags: { phase: "preload" }, contexts: { scenario: { id: defaultScenario } }, }); actions.setLoadError(e instanceof Error ? e.message : String(e)); actions.setLoadStatus("error"); }); }, []); // eslint-disable-line react-hooks/exhaustive-deps return ready; } // --------------------------------------------------------------------------- // Data loading hook (handles waymo loading; scenarios are pre-generated) // --------------------------------------------------------------------------- function useDataLoader() { const dataSource = useStore((s) => s.dataSource); const loadStatus = useStore((s) => s.loadStatus); const scenarioId = useStore((s) => s.scenarioId); const waymoSegment = useStore((s) => s.waymoSegment); const actions = useStore((s) => s.actions); useEffect(() => { if (loadStatus !== "idle") return; if (dataSource === "waymo-drop") return; if (dataSource === "scenario") { actions.setLoadStatus("loading"); actions.setLoadMessage(`Loading "${scenarioId}" scenario`); actions.setLoadProgress(0.1); setSentryScenario(scenarioId); loadScenario(scenarioId, "ground_truth", (msg, progress) => { actions.setLoadMessage(msg); actions.setLoadProgress(progress); }) .then((sceneData) => { actions.setSceneData(sceneData); const moments = generateTrajectoryMoments(sceneData); actions.setTrajectoryMoments(moments); }) .catch((e) => { captureError(e, { tags: { phase: "load-scenario" }, contexts: { scenario: { id: scenarioId } }, }); actions.setLoadError(e instanceof Error ? e.message : String(e)); actions.setLoadStatus("error"); }); } else if (dataSource === "waymo") { actions.setLoadStatus("loading"); actions.setLoadMessage("Checking cache…"); actions.setLoadProgress(0); const key = cacheKey("waymo", waymoSegment); getCachedScene(key) .then((cached) => { if (cached) { actions.setLoadMessage("Restoring from cache…"); actions.setLoadProgress(0.9); actions.setSceneData(cached); [truncated — 9541 more characters] ``` ### index.html ```html Improving Autonomous Vehicles
``` ### eslint.config.js ```javascript import js from '@eslint/js' import globals from 'globals' import reactHooks from 'eslint-plugin-react-hooks' import reactRefresh from 'eslint-plugin-react-refresh' import tseslint from 'typescript-eslint' import { defineConfig, globalIgnores } from 'eslint/config' export default defineConfig([ globalIgnores(['dist']), { files: ['**/*.{ts,tsx}'], extends: [ js.configs.recommended, tseslint.configs.recommended, reactHooks.configs.flat.recommended, reactRefresh.configs.vite, ], languageOptions: { ecmaVersion: 2020, globals: globals.browser, }, }, ]) ``` ### band_agent_config.example.yaml ```yaml # Band remote-agent registry for the Overflow AV-safety fleet. # # 1. In the Band dashboard (app.band.ai): Agents → New Agent → Remote Agent, # one per role below. # 2. Paste each agent's UUID into its agent_id. # 3. Copy this file to band_agent_config.yaml (gitignored), and put your key in .env: # BAND_API_KEY=band_u_... # # The agent *description* (in scripts/band_agents.py) is what other agents discover # and @mention — that's Band's no-orchestrator coordination model. overflow-perception-agent: agent_id: "REPLACE_WITH_BAND_UUID" overflow-safety-auditor: agent_id: "REPLACE_WITH_BAND_UUID" overflow-planner-agent: agent_id: "REPLACE_WITH_BAND_UUID" overflow-policy-agent: agent_id: "REPLACE_WITH_BAND_UUID" overflow-fleet-coordinator: agent_id: "REPLACE_WITH_BAND_UUID" ``` [74 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]