# Project export: benchwarmer.ai
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2026
- Tagline: benchwarmer.ai automates algorithm bench marking from papers to live results, so you can warm your seat while we run the benchmarks.
- Devpost: https://devpost.com/software/benchwarmer-7t2fr6
- GitHub: https://github.com/pdd23001/Benchwarmer.AI
- Video: https://www.youtube.com/embed/EmfTlF5OH04?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 3 GitHub contributor(s) — Parth Danve (8 commits), Gurmeher Singh (4 commits), Potatoboy9999 (3 commits)
## Devpost submission (written by the team)
### Inspiration
Everyone on our team has worked in research at some point. If there’s one thing you learn quickly in research, it’s this: The idea is exhilarating. The benchmarking is exhausting. You devise a new algorithm. You’re confident it’s better. Now you have to prove it. That means: Searching for multiple existing algorithms in the space Reading dense research papers Re-implementing those baselines yourself SSH’ing into different machines to run experiments Waiting for results Writing additional scripts to compile metrics Generating comparison plots Finally, analyze the results Benchmarking is the most important part of research and also the most tedious. We kept asking: Why does getting an idea of how an algorithm performs, take longer than coding it? What if benchmarking could be reduced to a few inputs? Drop them into an engine. Sit back. Relax. Warm your bench. We couldn't believe such a technology did not exist considering today's advancements. That’s where benchwarmer.ai was born. What benchwarmer does benchwarmer automates the painful workflow of algorithm benchmarking. You upload: Your .py algorithm The research papers you want to compete against Our multi-agent framework automatically: Extract algorithm logic from the papers Generate runnable challenger implementations Execute all algorithms in serverless sandboxed environments Aggregate results and generate comparison charts Instead of spending days re-implementing baselines and running for results, it becomes a matter of minutes. How We Built It benchwarmer is a multi-agent AI + systems pipeline designed for reliability, isolation, and scalability. Parallel Execution (Modal Sandboxes) Each algorithm runs in its own isolated Modal sandbox. One sandbox per algorithm Parallel execution No shared state Fail-soft isolation We are executing AI-generated code, so isolation is mandatory. If one implementation fails, the rest of the benchmark continues. Modal’s infrastructure is designed for exactly this workload: dynamic code execution at scale. Their platform supports scaling to 50,000+ concurrent sessions, making this architecture viable for large benchmark suites. Paper Ingestion + Orchestration (NVIDIA DGX Spark + Nemotron) Scientific PDFs are messy. Multi-column layouts, pseudocode blocks, dense notation. We integrated with NVIDIA DGX Spark and used Nemotron-3-Nano-30B as both an ingestion and orchestration agent. First, Nemotron extracts a structured representation of the algorithm including problem class, key steps, and assumptions. Then, based on this structured output, Nemotron routes control flow through the pipeline: Determines which implementation template to use Passes structured inputs to the Claude implementation agent Validates required components before proceeding Signals whether to retry, reject, or advance to execution Instead of relying on an external orchestration framework (e.g., LangGraph), we embed routing and control logic directly into Nemotron’s structured outputs. This reduces orchestration overhead and keeps the pipeline tightly integrated. Live Observability We built a Next.js frontend with WebSocket streaming so users can: See per-algorithm status transitions Watch live terminal logs Track benchmark progress View automatically generated comparison charts Benchmarking becomes observable instead of opaque. AI-Assisted Implementation (Claude Opus 4.6) The structured summary from Nemotron is passed to implementation agent powered by Claude Opus 4.6 We specifically chose Opus 4.6 for its reliability in structured code generation and long-context reasoning across technical documents. However, generation is only step one. Before any implementation is registered for benchmarking, we run automated smoke tests to verify: The algorithm compiles successfully Required interfaces are implemented correctly The function signatures match the selected problem class The algorithm produces valid outputs on small test instances No unsafe operations or obvious runtime failures occur If the implementation fails any of these checks, it is rejected and does not proceed to benchmarking. AI-generated code is treated as untrusted by default. Only validated implementations move forward to execution. Challenges We Faced PDF Extraction Reliability Scientific formatting breaks naïve parsing. Structured ingestion via Nemotron significantly improved robustness. Executing AI-Generated Code Safely Generated implementations must be sandboxed and validated to prevent failures from cascading. Infrastructure Pivot (brev.dev -> NVIDIA DGX Spark) Mid-hackathon, we ran out of brev.dev compute credits and had to pivot to NVIDIA DGX Spark, a more complex, less abstracted infrastructure layer What We Learned Parallel sandbox isolation is essential for safe AI execution Observability builds trust in automated systems Structured ingestion dramatically improves downstream reliability Most research tooling hasn’t evolved alongside AI capabilities Most importantly: Industry does not need another chat bot. They need tools that remove friction from validation. Our Vision Benchmarking should not become a bottleneck of innovation. If you build a new algorithm, you shouldn’t spend days rebuilding baselines. You should upload your code, upload the papers, and let them compete. benchwarmer turns paper ideas into live competitors. Automatically.
## README (from the GitHub repository)
# benchwarmer.ai
**Benchmarking should not be a bottleneck of innovation.**
benchwarmer.ai automates the painful workflow of algorithm benchmarking. Upload your algorithm and the research papers you want to compete against — a multi-agent orchestration framework extracts algorithms from the papers, generates runnable implementations, executes everything in parallel cloud sandboxes, and produces publication-ready comparison charts. What used to take days now takes minutes.
---
## Table of Contents
- [How It Works](#how-it-works)
- [Architecture](#architecture)
- [Execution Modes](#execution-modes)
- [Tech Stack](#tech-stack)
- [Project Structure](#project-structure)
- [Prerequisites](#prerequisites)
- [Setup](#setup)
- [Running the App](#running-the-app)
- [Deploying to Vercel](#deploying-to-vercel)
- [Environment Variables](#environment-variables)
- [License](#license)
---
## How It Works
1. **Upload** — Drop in your `.py` algorithm and the research papers you want to benchmark against.
2. **Orchestration** — The orchestrator agent understands your intent, routes through the pipeline, and drives the entire session conversationally — no manual steps required.
3. **Intake** — An AI agent parses your description and PDFs, classifies the problem class (Max-Cut, TSP, etc.), and builds a structured benchmark configuration.
4. **Implementation** — Claude generates runnable Python implementations of each challenger algorithm extracted from the papers, smoke-tests them in an isolated sandbox, and registers them for execution.
5. **Execution** — All algorithms run in parallel inside isolated [Modal](https://modal.com) cloud sandboxes — one sandbox per algorithm, full isolation, automatic scaling. One crash never takes down the benchmark.
6. **Analysis** — Results are aggregated into a DataFrame and an AI-powered plot agent generates comparison charts on demand.
7. **Conversation** — The entire flow is driven through a multi-turn chat interface. Ask follow-up questions, tweak parameters, re-run with different instances, request new visualizations — all in natural language.
---
## Architecture
```
┌────────────────────────────────────────────────────────────────────────┐
│ Frontend (Vite + React + TS) │
│ │
│ ChatPage ─── SSE stream ◄──── /api/chat ────► OrchestratorAgent │
│ Sidebar ─── REST ◄──── /api/sessions, /api/algorithms │
│ SandboxPanel ◄──────────────── benchmark_progress events │
└────────────────────────────────────────────────────────────────────────┘
│
FastAPI (uvicorn :8000)
│
┌───────────────┴───────────────────┐
│ │
▼ ▼
OrchestratorAgent SQLite (chat.db)
(conversational router) session persistence
│
┌───────────┼────────────────┐
│ │ │
▼ ▼ ▼
IntakeAgent ImplementationAgent PlotAgent
(config) (code generation) (visualizations)
│ │ │
▼ ▼ ▼
LLM Backends AlgorithmWrapper matplotlib
├─ Claude smoke-test →
└─ Nemotron register
(DGX Spark) │
▼
BenchmarkRunner
├─ Sequential (local subprocesses)
└─ Modal CPU Sandbox (parallel cloud)
├─ 1 sandbox per algorithm
├─ all instances × runs per sandbox
└─ real-time progress via SSE
```
### Multi-Agent Orchestration
The system is driven by a central **Orchestrator Agent** that acts as the conversational router. It receives every user message, maintains the full pipeline state, and dispatches tools to specialized sub-agents as needed. The orchestrator supports context forwarding — if the user's initial message specifies algorithms, instances, and parameters, it drives straight through the pipeline without re-asking.
| Agent | Role | Model |
|---|---|---|
| **Orchestrator** | Central router — understands intent, dispatches tools, manages multi-turn state | Claude Sonnet 4 |
| **Intake** | Parses NL problem descriptions + research PDFs into structured benchmark configs | Claude Sonnet 4 / Nemotron |
| **Implementation** | Generates `AlgorithmWrapper` subclasses from algorithm specs, smoke-tests in sandbox | Claude Opus 4.6 |
| **Plot** | Generates matplotlib visualizations from NL requests over benchmark results | Claude Sonnet 4 |
### LLM Backend: Claude vs Nemotron
The intake and orchestration stages are the most token-intensive parts of the pipeline — they process full research papers, lengthy problem descriptions, and maintain multi-turn conversation context. We designed the system to support two LLM backends:
- **Claude Opus 4.6** (Anthropic) — Used for the **Implementation Agent** where code generation accuracy is critical. Opus 4.6 is Anthropic's most capable model for coding tasks, ensuring the generated algorithm implementations are correct, efficient, and faithful to the source papers.
- **Claude Sonnet 4** (Anthropic) — Used for orchestration, intake, and plot generation where speed and tool-use capability matter more than raw coding power.
- **Nemotron-3-Nano-30B** (NVIDIA, open-source) — Deployed locally on **NVIDIA DGX Spark** hardware. As an open-source model, Nemotron eliminates per-token API costs entirely, making it a strong choice for the high-context intake stage where papers and descriptions can consume tens of thousands of tokens per request. Running on DGX Spark also means inference stays on-premises with zero network latency and full data privacy — important when processing unpublished research.
Users can select either backend from the chat UI. This dual-backend design lets teams balance cost, speed, and capability based on their workload.
---
## Execution Modes
### Sequential (Local)
Each algorithm runs in an isolated subprocess on your machine with hard timeout enforcement via `multiprocessing`. Algorithms execute one at a time. Best for quick tests and debugging.
### Modal CPU Sandbox (Parallel)
Each algorithm gets its own [Modal](https://modal.com) cloud sandbox — a fully isolated container running in Modal's infrastructure. All algorithm sandboxes run **in parallel**, with instances and runs executing sequentially within each sandbox. This means a benchmark with 5 algorithms runs ~5x faster than sequential mode.
The frontend provides a real-time **sandbox visualization panel** during Modal execution: each algorithm gets a visual progress indicator showing completion percentage, with a sand-fill animation that rises as runs complete. When all sandboxes finish, the panel shows "Complete" and transitions back to the full chat view.
```
Modal Execution Architecture:
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Sandbox #1 │ │ Sandbox #2 │ │ Sandbox #3 │
│ random_cut │ │ goemans │ │ your_algo │
│ │ │ │ │ │
│ inst1×run1 │ │ inst1×run1 │ │ inst1×run1 │
│ inst1×run2 │ │ inst1×run2 │ │ inst1×run2 │
│ inst2×run1 │ │ inst2×run1 │ │ inst2×run1 │
│ ... │ │ ... │ │ ... │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
└─────── progress events ───────────┘
│
SSE →
[README truncated for size]
## Detected evidence (automated analysis)
Indexed codebase: 94 recognized source files, 586 KB.
- Anthropic (technology) — detected in the code
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- AI coding agent: Claude Code — evidence: commit authorship or trailers
## Codebase structure (from repository index)
### Files (119 of 119)
```
.DS_Store
.gitignore
agent-backend/.DS_Store
agent-backend/.env.example
agent-backend/.gitignore
agent-backend/benchmark_results.csv
agent-backend/benchwarmer/__init__.py
agent-backend/benchwarmer/agents/__init__.py
agent-backend/benchwarmer/agents/backends.py
agent-backend/benchwarmer/agents/implementation.py
agent-backend/benchwarmer/agents/intake.py
agent-backend/benchwarmer/agents/orchestrator.py
agent-backend/benchwarmer/agents/plot.py
agent-backend/benchwarmer/agents/tools.py
agent-backend/benchwarmer/algorithms/__init__.py
agent-backend/benchwarmer/algorithms/base.py
agent-backend/benchwarmer/config.py
agent-backend/benchwarmer/database.py
agent-backend/benchwarmer/engine/__init__.py
agent-backend/benchwarmer/engine/modal_runner.py
agent-backend/benchwarmer/engine/runner.py
agent-backend/benchwarmer/engine/sandbox_pool.py
agent-backend/benchwarmer/generators/__init__.py
agent-backend/benchwarmer/generators/barabasi_albert.py
agent-backend/benchwarmer/generators/base.py
agent-backend/benchwarmer/generators/erdos_renyi.py
agent-backend/benchwarmer/generators/grid_2d.py
agent-backend/benchwarmer/generators/planar_random.py
agent-backend/benchwarmer/generators/planted_partition.py
agent-backend/benchwarmer/problem_classes/__init__.py
agent-backend/benchwarmer/problem_classes/maximum_cut.py
agent-backend/benchwarmer/problem_classes/minimum_vertex_cover.py
agent-backend/benchwarmer/problem_classes/registry.py
agent-backend/benchwarmer/utils/__init__.py
agent-backend/benchwarmer/utils/algorithm_sandbox.py
agent-backend/benchwarmer/utils/benchmark_suites.py
agent-backend/benchwarmer/utils/instance_loader.py
agent-backend/benchwarmer/utils/loader.py
agent-backend/benchwarmer/utils/modal_sandbox.py
agent-backend/benchwarmer/utils/sandbox.py
agent-backend/chat.db
agent-backend/custom_algo_template.py
agent-backend/debug_intake.py
agent-backend/debug_output.txt
agent-backend/explanation.md
agent-backend/nixpacks.toml
agent-backend/Procfile
agent-backend/pyproject.toml
agent-backend/railway.toml
agent-backend/reproduce_error.py
agent-backend/requirements.txt
agent-backend/runtime.txt
agent-backend/scripts/demo_phase1.py
agent-backend/scripts/run_benchmark.py
agent-backend/server.py
agent-backend/tests/test_agents.py
agent-backend/tests/test_algorithm_sandbox.py
agent-backend/tests/test_backends.py
agent-backend/tests/test_benchmark_suites.py
agent-backend/tests/test_engine.py
agent-backend/tests/test_generators.py
agent-backend/tests/test_instance_loader.py
agent-backend/tests/test_modal_runner.py
agent-backend/tests/test_pdf_native.py
agent-backend/tests/test_problem_classes.py
agent-backend/tests/test_sandbox.py
chat.db
frontend-vite/.env.example
frontend-vite/.gitignore
frontend-vite/.nvmrc
frontend-vite/eslint.config.js
frontend-vite/index.html
frontend-vite/package.json
frontend-vite/postcss.config.js
frontend-vite/README.md
frontend-vite/src/App.css
frontend-vite/src/App.tsx
frontend-vite/src/assets/placeholder.ts
frontend-vite/src/components/BenchmarkChart.tsx
frontend-vite/src/components/BenchwarmerLogo.tsx
frontend-vite/src/components/chat/AlgorithmSelector.tsx
frontend-vite/src/components/chat/ChatInput.tsx
frontend-vite/src/components/chat/ChoiceSelector.tsx
frontend-vite/src/components/chat/CodeViewer.tsx
frontend-vite/src/components/chat/MessageList.tsx
frontend-vite/src/components/chat/SandboxPanel.tsx
frontend-vite/src/components/chat/UploadZone.tsx
frontend-vite/src/components/CodeViewer.tsx
frontend-vite/src/components/FileViewer.tsx
frontend-vite/src/components/Header.tsx
frontend-vite/src/components/Layout.tsx
frontend-vite/src/components/Sidebar.tsx
frontend-vite/src/components/ui/avatar.tsx
frontend-vite/src/components/ui/button.tsx
frontend-vite/src/components/ui/card.tsx
frontend-vite/src/components/ui/collapsible.tsx
frontend-vite/src/components/ui/dialog.tsx
frontend-vite/src/components/ui/input.tsx
frontend-vite/src/components/ui/label.tsx
frontend-vite/src/components/ui/scroll-area.tsx
frontend-vite/src/components/ui/select.tsx
frontend-vite/src/hooks/use-chat.ts
frontend-vite/src/index.css
frontend-vite/src/lib/utils.ts
frontend-vite/src/main.tsx
frontend-vite/src/pages/BenchmarksPage.tsx
frontend-vite/src/pages/ChatPage.tsx
frontend-vite/src/vite-env.d.ts
frontend-vite/tailwind.config.js
frontend-vite/tsconfig.app.json
frontend-vite/tsconfig.json
frontend-vite/tsconfig.node.json
frontend-vite/vercel.json
frontend-vite/vite.config.ts
package.json
README.md
revised-architecture.md
SPEC.md
vercel.json
```
### Dependencies
- agent-backend/pyproject.toml: anthropic@>=0.30, matplotlib@>=3.7, networkx@>=3.0, numpy@>=1.24, pandas@>=2.0, pydantic@>=2.0, pytest@>=7.0, scipy@>=1.10
- agent-backend/requirements.txt: anthropic@>=0.3.0, fastapi@>=0.109.0, matplotlib@>=3.7, modal@>=0.72.0, networkx@>=3.0, numpy@>=1.24, openai@>=1.0.0, pandas@>=2.0, pydantic@>=2.0, pypdf@>=5.0.0, pytest@>=7.0, python-dotenv@>=1.0.0, python-multipart@>=0.0.6, scipy@>=1.10, uvicorn@>=0.27.0
- frontend-vite/package.json: @eslint/js@^9.39.1, @radix-ui/react-avatar@^1.1.11, @radix-ui/react-collapsible@^1.1.12, @radix-ui/react-dialog@^1.1.15, @radix-ui/react-scroll-area@^1.2.10, @radix-ui/react-select@^2.2.6, @radix-ui/react-slot@^1.2.4, @radix-ui/react-tooltip@^1.2.8, @types/node@^24.10.13, @types/react@^19.2.7, @types/react-dom@^19.2.3, @types/uuid@^10.0.0, @vitejs/plugin-react@^5.1.1, autoprefixer@^10.4.24, axios@^1.13.5, class-variance-authority@^0.7.1, clsx@^2.1.1, eslint@^9.39.1, eslint-plugin-react-hooks@^7.0.1, eslint-plugin-react-refresh@^0.4.24, globals@^16.5.0, lucide-react@^0.564.0, postcss@^8.5.6, react@^19.2.0, react-dom@^19.2.0, react-markdown@^10.1.0, react-router-dom@^7.13.0, recharts@^3.7.0, remark-gfm@^4.0.1, tailwind-merge@^3.4.1, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7, typescript@~5.9.3, typescript-eslint@^8.48.0, uuid@^13.0.0, vite@^7.3.1
### Recent commits (newest first)
- fix: harden nemotron orchestrator fallback and backend flows
- Persist rich UI metadata (buttons, tools, choices) across refresh
- Persist chat session across page refresh
- bruh
- railway
- vercel
- vercel
- Extend token limt
- Fix nemotron
- final
- Readme
- holycrap'
- vercel
- one left
- ui
- opus holy
- context forwarding
- bruh
- pipeline fixed
- only plotting error left
## Key source files (fetched from GitHub, selected and truncated for size)
### SPEC.md
```markdown
📜 Benchwarmer.ai — Technical Specification & Hackathon Plan
1. The Vision
Benchwarmer.ai is a conversational research assistant that scientifically validates algorithmic claims. Instead of just "generating code," it behaves like a scientist: it researches algorithms, implements them, runs head-to-head benchmarks in a secure sandbox, and discusses the results with you dynamically.
The Loop:
User: "Benchmark Dijkstra vs. A* Search on a grid." (Natural Language)
Agent: Writes the code + Test Harness.
Infrastructure: Executed in Modal Sandbox (User code is never run locally).
Result: Interactive graphs + Analysis.
User: "Now show me memory usage instead of time." (Conversational Refinement)
Agent: Re-renders the analysis instantly.
2. Prize Targets (The "Why")
Modal (Sandbox Challenge): Core execution engine.
Greylock (Best Multi-turn Agent): The conversational analysis loop.
Anthropic (Best Use of Claude): The "Coder" and "Analyst" agents.
Perplexity (Sonar API): Finding SOTA baselines when the user asks vague questions (e.g., "Compare the fastest sorting algos").
Decagon (Best Conversational Assistant): The natural language interface.
Vercel (Best Use of Vercel/Best Deployed on Vercel)
3. System Architecture
A. The "Modal-First" Backend
Why: We avoid Vercel's 10s timeout. Modal Web Endpoints support up to 15-minute executions, essential for running benchmarks. We will run all our agent code inside the Modal container.
Entrypoint: backend/modal_app.py.
Endpoints:
POST /generate: Streaming endpoint. Receives chat history, yields text chunks (logs) and final JSON (data).
B. The Frontend (Next.js 15)
Host: Vercel.
UI Library: Tailwind. No ShadCN/component library. Let's build our own unique UI.
Visualization: Recharts (Line/Bar charts).
State: Uses ai/react (Vercel AI SDK) to handle the streaming chat.
The "Lab" View: A dynamic component that renders:
Loading State: Live terminal logs from the backend.
Success State: Interactive Graph + Statistical Summary table.
C. The Agent Logic (LangGraph)
Orchestrator: A state machine running inside the Modal container.
Tools:
search_perplexity(query): Finds algorithm logic.
generate_code_claude(prompt): Writes Python.
run_sandbox(code): Executes in isolation.
4. Detailed Data Flow
Step 1: User Request
"Benchmark QuickSort vs MergeSort on random integers."
Step 2: Research & Planning (Agent)
Intent Classification: Agent identifies [QuickSort, MergeSort] as targets and List[int] as input.
Baseline Search: (Optional) If user asks "Compare QuickSort to SOTA", call Perplexity to find "Timsort".
Step 3: Code Generation (Claude)
The Prompt: "Implement QuickSort and MergeSort. You must adhere to the BenchmarkHarness class interface."
The Interface:
Python
class Algorithm(ABC):
def load_data(self, size: int):
[truncated — 3045 more characters]
```
### revised-architecture.md
```markdown
# Revised Architecture: Natural Language Intake + User-Driven Plots
---
## Change 1: Natural Language Problem Description
### Before (rigid)
```yaml
problem_class: maximum_cut
instances:
generators:
- type: erdos_renyi
params: {p: 0.3}
sizes: [50, 100, 200, 500]
```
### After (natural language)
```
User: "I'm working on partitioning social networks to maximize
cross-group connections. My graphs are sparse, undirected,
with weighted edges representing interaction frequency.
Typical size is 500-5000 nodes. I care most about solution
quality but need results within 60 seconds."
```
The system figures out: this is Max-Cut on sparse weighted undirected graphs,
needs instance generators that mimic social network topology (Barabási-Albert,
planted partition), benchmark sizes should go up to 5000, and the analysis
should weight quality heavily but flag anything over 60s.
---
### Intake Agent Design
**Model:** Claude Sonnet 4 via tool_use
**Job:** Turn a freeform problem description into a structured `BenchmarkConfig`
that the rest of the pipeline can consume.
**System prompt (core logic):**
```
You are the intake agent for an algorithm benchmarking platform.
The user will describe their optimization problem in natural language.
Your job is to:
1. UNDERSTAND the problem — ask clarifying questions if needed
2. CLASSIFY it into a known problem class (or flag it as custom)
3. INFER the right benchmarking setup:
- What graph types match their real-world scenario?
- What sizes to test at?
- What matters more: speed, quality, memory, consistency?
- Any hard constraints (e.g., "must run under 60 seconds")?
4. OUTPUT a structured BenchmarkConfig JSON
You have access to the following tools:
- classify_problem(description) → returns candidate problem classes with confidence
- get_generators(problem_class) → returns available instance generators
- validate_config(config) → checks if a config is valid and complete
IMPORTANT BEHAVIORS:
- If the problem clearly maps to a known class, don't over-ask. Confirm and move on.
- If it's ambiguous (could be Max-Cut OR graph partitioning), ask ONE clarifying question.
- Always infer instance generators from the user's domain description:
- "social networks" → Barabási-Albert, planted partition
- "road networks" → grid-like graphs, planar graphs
- "molecular structures" → sparse, bounded-degree graphs
- "internet topology" → power-law graphs
- "random benchmarks" → Erdős-Rényi
- Extract any implicit constraints the user mentioned.
- Don't ask about things you can set sensible defaults for.
```
**Conversation flow:**
```
User: "I'm trying to find the minimum set of sensors that covers all
corridors in a building. Each corridor connects two junctions.
I want to compare my greedy approach against whatever the
literature recommends."
Agent (internally):
→ classify_problem("minimum set of nodes covering all
[truncated — 15533 more characters]
```
### package.json
```
{
"name": "benchwarmer-ai",
"private": true,
"scripts": {
"build": "cd frontend-vite && npm run build"
}
}
```
### agent-backend/requirements.txt
```
# Core
networkx>=3.0
pandas>=2.0
pydantic>=2.0
numpy>=1.24
matplotlib>=3.7
# Optional — used by planar_random generator
scipy>=1.10
# LLM agents (Phase 2+)
# anthropic>=0.30
# Testing
pytest>=7.0
# Web API
fastapi>=0.109.0
uvicorn>=0.27.0
python-multipart>=0.0.6 # Required for file uploads in FastAPI
# LLM & Utils
anthropic>=0.3.0
python-dotenv>=1.0.0
modal>=0.72.0
openai>=1.0.0
pypdf>=5.0.0
```
### agent-backend/pyproject.toml
```
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "benchwarmer"
version = "0.1.0"
description = "Algorithm benchmarking platform with NL-driven intake and interactive analysis"
readme = "README.md"
requires-python = ">=3.10"
dependencies = [
"networkx>=3.0",
"pandas>=2.0",
"pydantic>=2.0",
"numpy>=1.24",
"matplotlib>=3.7",
"scipy>=1.10",
]
[project.optional-dependencies]
dev = ["pytest>=7.0"]
agents = ["anthropic>=0.30"]
[tool.setuptools.packages.find]
include = ["benchwarmer*"]
```
### frontend-vite/package.json
```
{
"name": "frontend-vite",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc -b && vite build",
"lint": "eslint .",
"preview": "vite preview"
},
"dependencies": {
"@radix-ui/react-avatar": "^1.1.11",
"@radix-ui/react-collapsible": "^1.1.12",
"@radix-ui/react-dialog": "^1.1.15",
"@radix-ui/react-scroll-area": "^1.2.10",
"@radix-ui/react-select": "^2.2.6",
"@radix-ui/react-slot": "^1.2.4",
"@radix-ui/react-tooltip": "^1.2.8",
"axios": "^1.13.5",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"lucide-react": "^0.564.0",
"react": "^19.2.0",
"react-dom": "^19.2.0",
"react-markdown": "^10.1.0",
"react-router-dom": "^7.13.0",
"recharts": "^3.7.0",
"remark-gfm": "^4.0.1",
"tailwind-merge": "^3.4.1",
"tailwindcss-animate": "^1.0.7",
"uuid": "^13.0.0"
},
"devDependencies": {
"@eslint/js": "^9.39.1",
"@types/node": "^24.10.13",
"@types/react": "^19.2.7",
"@types/react-dom": "^19.2.3",
"@types/uuid": "^10.0.0",
"@vitejs/plugin-react": "^5.1.1",
"autoprefixer": "^10.4.24",
"eslint": "^9.39.1",
"eslint-plugin-react-hooks": "^7.0.1",
"eslint-plugin-react-refresh": "^0.4.24",
"globals": "^16.5.0",
"postcss": "^8.5.6",
"tailwindcss": "^3.4.1",
"typescript": "~5.9.3",
"typescript-eslint": "^8.48.0",
"vite": "^7.3.1"
}
}
```
### frontend-vite/src/main.tsx
```typescript
import React from 'react'
import ReactDOM from 'react-dom/client'
import App from './App.tsx'
import './index.css'
ReactDOM.createRoot(document.getElementById('root')!).render(
,
)
```
### frontend-vite/src/App.tsx
```typescript
import { BrowserRouter as Router, Routes, Route } from "react-router-dom"
import ChatPage from "@/pages/ChatPage"
function App() {
return (
} />
)
}
export default App
```
### agent-backend/server.py
```python
import logging
import sys
import pandas as pd
import tempfile
import os
import re
from pathlib import Path
from fastapi import FastAPI, HTTPException, UploadFile, File, Form
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import List, Dict, Any, Optional
# Ensure we can import from benchwarmer package
import asyncio
import uuid
import json
import time
import subprocess
from collections import defaultdict
from fastapi import FastAPI, HTTPException, UploadFile, File, Form, WebSocket, WebSocketDisconnect
from fastapi.responses import JSONResponse
import httpx
from benchwarmer.algorithms.base import AlgorithmWrapper
from benchwarmer.config import BenchmarkConfig, GeneratorConfig, InstanceConfig
from benchwarmer.engine.runner import BenchmarkRunner
from benchwarmer.agents.intake import IntakeAgent
from benchwarmer.agents.implementation import ImplementationAgent
from benchwarmer.agents.backends import OpenAIBackend
from benchwarmer.engine.modal_runner import ModalRunner
from dotenv import load_dotenv
# Load environment variables
load_dotenv()
# Configure logging
logging.basicConfig(level=logging.INFO, format="%(levelname)s | %(message)s")
app = FastAPI()
# Edge Nemotron defaults (DGX Spark / Ollama-compatible API).
NEMOTRON_BASE_URL = os.environ.get("NEMOTRON_BASE_URL", "http://10.19.177.52:11434/api")
NEMOTRON_MODEL = os.environ.get(
"NEMOTRON_MODEL",
"hf.co/unsloth/Nemotron-3-Nano-30B-A3B-GGUF:Q4_K_M",
)
NEMOTRON_API_KEY = (
os.environ.get("NEMOTRON_API_KEY")
or os.environ.get("NVIDIA_API_KEY")
or os.environ.get("OPENAI_API_KEY")
)
# Add CORS middleware to allow frontend requests
app.add_middleware(
CORSMiddleware,
allow_origins=["http://localhost:3000"], # Next.js default port
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
MAX_PY_UPLOAD_BYTES = 512 * 1024 # 512 KB
MAX_PDF_UPLOAD_BYTES = 20 * 1024 * 1024 # 20 MB
# ─── Session & WebSocket Management ──────────────────────────────────────────
class SessionManager:
def __init__(self):
# session_id -> { "files": [], "algorithms": [], "config": None, "logs": [] }
self.sessions: Dict[str, Dict[str, Any]] = {}
def create_session(self) -> str:
session_id = str(uuid.uuid4())
self.sessions[session_id] = {
"created_at": time.time(),
"user_algo_path": None,
"user_algo_name": None,
"problem_class": None,
"challengers": [], # List of {id, type, status, path, name}
"config": None,
"base_dir": tempfile.mkdtemp(prefix=f"benchwarmer_{session_id}_")
}
return session_id
def get_session(self, session_id: str) -> Dict[str, Any]:
if session_id not in self.sessions:
raise HTTPException(status_code=404, detail="Session not found")
return self.sessions[session_id]
def add_challenger(self, session_id: str, type: str, path: str, name: str) -> str:
session = self.get_session(session_id)
challenger_id = str(uuid.uuid4())
session["challengers"].append({
"id": challenger_id,
"type": type, # "pdf", "text", "baseline"
"status": "pending", # "analyzing", "ready", "error"
"path": path,
"name": name,
"implementation": None # Will hold the AlgorithmWrapper instance later
})
return challenger_id
def get_challenger(self, session_id: str, challenger_id: str):
session = self.get_session(session_id)
for c in session["challengers"]:
if c["id"] == challenger_id:
return c
return None
session_manager = SessionManager()
class ConnectionManager:
def __init__(self):
# session_id -> list of WebSockets
self.active_connections: Dict[str, List[WebSocket]] = defaultdict(list)
async def connect(self, websocket: WebSocket, session_id: str):
await websocket.accept()
self.active_connections[session_id].append(websocket)
def disconnect(self, websocket: WebSocket, session_id: str):
if session_id in self.active_connections:
if websocket in self.active_connections[session_id]:
self.active_connections[session_id].remove(websocket)
async def broadcast(self, session_id: str, message: dict):
if session_id in self.active_connections:
stale_connections: list[WebSocket] = []
for connection in self.active_connections[session_id]:
try:
await connection.send_json(message)
except Exception as e:
logging.warning(f"Failed to send to websocket: {e}")
stale_connections.append(connection)
for connection in stale_connections:
self.disconnect(connection, session_id)
ws_manager = ConnectionManager()
def _validate_upload(
file_name: str,
content: bytes,
allowed_suffixes: set[str],
max_bytes: int,
label: str,
) -> None:
if not file_name:
raise HTTPException(status_code=400, detail=f"{label} filename is required")
suffix = Path(file_name).suffix.lower()
if suffix not in allowed_suffixes:
raise HTTPException(
status_code=400,
detail=(
f"Invalid {label} file extension '{suffix or '(none)'}'. "
f"Allowed: {', '.join(sorted(allowed_suffixes))}"
),
)
if not content:
raise HTTPException(status_code=400, detail=f"{label} file is empty")
if len(content) > max_bytes:
raise HTTPException(
status_code=413,
detail=(
f"{label} file too large ({len(content)} bytes). "
f"Max allowed: {max_bytes} bytes"
),
)
def _extract_json_object(text: str) -> dict[str, Any]:
"""Extract the first JSON object from p
[truncated — 35782 more characters]
```
### frontend-vite/postcss.config.js
```javascript
export default {
plugins: {
tailwindcss: {},
autoprefixer: {},
},
}
```
[86 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]