# Project export: EcholoKernel This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Fast, multigpu AI kernels using LLM agents - Devpost: https://devpost.com/software/echolokernel - GitHub: https://github.com/vincent65/FERB - Demo: https://docs.google.com/presentation/d/12uf2YyxNKq3BruRxDVUQghrfFtUMrXOBLcEHfJjQNrI/edit?usp=sharing - Video: https://player.vimeo.com/video/1165110679?byline=0&portrait=0&title=0#t= - Team: 1 GitHub contributor(s) — Vincent Yip (7 commits) ## Devpost submission (written by the team) ### Overview So long, and thanks for all the fish. - Douglas Adams TLDR - Writing multi-GPU kernels is hard. We built a slick UI + multi-turn LLM agent (RAG, RLM, profiler feedback, etc.) to make it easier. This equals a better assisted, iterative, human-in-the-loop way to write real, useful multiGPU code. EcholoKernel can generate NVSHMEM4py + Triton–compatible kernels (despite the near-absence of documentation) to get up to 2.4× speedup over AG + GEMM baselines on small tensors and ~1.3× speedup on the Ulysses attention combine step. ### Inspiration Writing GPU kernels is painful. Writing multi-GPU kernels is worse. Companies love to advertise TFLOPs, but the real bottleneck in modern AI systems is networking. Especially in sparse and MoE-style models, runtime is dominated by just getting data where it needs to be, not doing the math inside a kernel. There’s been real progress on LLM agents that can write GPU kernels using RL and RAG, but most of that work stops at a single GPU. To use multiple GPUs, researchers default to NCCL via torch.distributed, which works well for standard patterns but leaves performance on the table for workloads that need fine-grained communication. So we asked: can an AI agent translate high-level torch.distributed code and turn it into fine-grained, fused compute + communication kernels? And can we get better performance? ### What it does We built a UI and an RLM-powered agent that translates torch.distributed code into NVSHMEM-style multigpu kernels. The core idea is to keep a human in the loop, but let the agent handle the monotony and grind of kernel iteration/ideation. Our backend uses Triton + NVSHMEM4Py, chosen deliberately to stay close to a DSL kernel developers already understand. The agent generates candidate kernels, compiles and runs them, and then reasons over profiler and compiler feedback to iteratively improve performance. Through the UI, a human can see prior agent attempts and step in at any point to nudge the algorithm/syntax if needed. We found this makes multiGPU kernel prototyping a lot less painful! Results We were able to get a 2.4x speedup on AG + GEMM for 256 x 256 matrices, and consistently lower latencies on smaller matrix sizes for AG + GEMM (see slides). We also tried it with something harder, namely ulysses attention, and got ~1.3x speedup across 2 H100s for sequence lengths ranging from 256 - 16384. These are real-world workloads and kernels and algorithms used in production today: dropping in our replacement led GPT-2 to consistently have a higher tokens/second generated! We're excited to see what would happen if we took the time to drop in a model end-to-end and have a giant reasoning model go to town on optimizing it. ### How we built it The UI is built using Next and FastAPI. Benchmarking and orchestration are written primarily in Python, with heavy use of Modal and Together to compile, run, and evaluate kernels across GPUs. At the core is a multi-turn RLM agent that generates candidate kernels, runs them, and incorporates compiler errors, runtime behavior, and profiler feedback into subsequent iterations. Rather than treating kernel generation as a one-shot problem, the system is explicitly designed around having a human in the loop, able to intervene between LLM loops using the UI. See the slides) for more details on the agentic system. ### Challenges we ran into Writing correct multi-GPU kernels is still hard, for humans and for LLMs. Correctness was the dominant challenge: many generated kernels deadlocked, produced subtly wrong outputs, or violated NVSHMEM and Triton semantics in ways that only showed up at runtime. The agent often struggled with synchronization, memory ordering, and the implicit assumptions baked into these DSLs. ### Accomplishments we're proud of That said, it was still striking how far we could get given our limited prior experience with NVSHMEM + Triton and the scarcity of documentation. We're producing coherent NVSHMEM4Py + Triton kernels with minimal public examples, and made a working end-to-end system that generates, benchmarks, and iterates on real multi-GPU kernels. We also demonstrated real-world applications: we have legit performance gains on production-relevant GEMM and attention workloads, demonstrating that agent-assisted kernel development is both within reach and actually useful. ### What's next Improving the agent, fleshing out benchmarks, and getting some sleep. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 229 recognized source files, 1179 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 1189) ``` .modalignore .playwright-mcp/console-2026-02-15T06-01-16-137Z.log .playwright-mcp/console-2026-02-15T06-03-20-106Z.log .playwright-mcp/console-2026-02-15T06-04-58-643Z.log AGENT_ARCHITECTURE_SLIDE.md AGENT_ARCHITECTURE.md AGENT_CODEBASE_GUIDE.md agent/__init__.py agent/config.py agent/core/__init__.py agent/core/bootstrap.py agent/core/optimizer.py agent/core/rollback.py agent/core/run_log.py agent/eval/__init__.py agent/eval/modal_evaluator.py agent/llm_client.py agent/openai_client.py agent/strategies/__init__.py agent/strategies/memory/__init__.py agent/strategies/memory/base.py agent/strategies/memory/best_so_far.py agent/strategies/proposers/__init__.py agent/strategies/proposers/base.py agent/strategies/proposers/single_shot.py agent/strategies/registry.py agent/strategies/retrieval/__init__.py agent/strategies/retrieval/corpus.py agent/strategies/retrieval/docs_loader.py agent/strategies/retrieval/retriever.py agent/strategies/retrieval/rlm_retriever.py agent/strategies/scorers/__init__.py agent/strategies/scorers/base.py agent/strategies/scorers/speedup_mean.py DETAILED_AGENT_FLOW_DIAGRAMS.html DETAILED_AGENT_FLOW.md docs/OPTIMIZATION_LOOP_EXAMPLE.md experiments/default.yaml experiments/problem11.yaml experiments/problem3.yaml experiments/problem4.yaml experiments/problem4base.yaml experiments/problem5.yaml experiments/problem50.yaml experiments/problem6.yaml experiments/problem7.yaml experiments/problem8.yaml frontend/api/__init__.py frontend/api/main.py frontend/api/requirements.txt frontend/api/routers/__init__.py frontend/api/routers/experiments.py frontend/api/routers/launch.py frontend/api/routers/runs.py frontend/api/services/__init__.py frontend/api/services/demo_runner.py frontend/api/services/file_watcher.py frontend/api/services/run_launcher.py frontend/api/services/run_reader.py frontend/api/ws/__init__.py frontend/api/ws/handler.py frontend/app/.gitignore frontend/app/eslint.config.mjs frontend/app/next.config.ts frontend/app/package.json frontend/app/postcss.config.mjs frontend/app/README.md frontend/app/src/app/globals.css frontend/app/src/app/layout.tsx frontend/app/src/app/page.tsx frontend/app/src/app/providers.tsx frontend/app/src/app/runs/[runId]/page.tsx frontend/app/src/components/bottom-panel.tsx frontend/app/src/components/code-viewer.tsx frontend/app/src/components/correctness-panel.tsx frontend/app/src/components/experiment-builder.tsx frontend/app/src/components/iteration-history.tsx frontend/app/src/components/llm-reasoning.tsx frontend/app/src/components/modal-logs.tsx frontend/app/src/components/profiler-panel.tsx frontend/app/src/components/run-card.tsx frontend/app/src/components/run-launcher.tsx frontend/app/src/components/timing-charts.tsx frontend/app/src/hooks/use-run-data.ts frontend/app/src/hooks/use-websocket.ts frontend/app/src/lib/api-client.ts frontend/app/src/lib/types.ts frontend/app/src/stores/run-store.ts frontend/app/tsconfig.json frontend/README.md frontend/start.sh logs/problem_1/agent/rank_0_perf.json logs/problem_1/agent/rank_1_perf.json logs/problem_1/agent/rank_2_perf.json logs/problem_1/agent/rank_3_perf.json logs/problem_1/agent/rank_4_perf.json logs/problem_1/agent/rank_5_perf.json logs/problem_1/agent/rank_6_perf.json logs/problem_1/agent/rank_7_perf.json logs/problem_1/agent/summary_rank0.json logs/problem_1/agent/trace_candidate_rank0.json logs/problem_1/agent/trace_candidate_rank1.json logs/problem_1/agent/trace_candidate_rank2.json logs/problem_1/agent/trace_candidate_rank3.json logs/problem_1/agent/trace_candidate_rank4.json logs/problem_1/agent/trace_candidate_rank5.json logs/problem_1/agent/trace_candidate_rank6.json logs/problem_1/agent/trace_candidate_rank7.json logs/problem_1/agent/trace_reference_rank0.json logs/problem_1/agent/trace_reference_rank1.json logs/problem_1/agent/trace_reference_rank2.json logs/problem_1/agent/trace_reference_rank3.json logs/problem_1/agent/trace_reference_rank4.json logs/problem_1/agent/trace_reference_rank5.json logs/problem_1/agent/trace_reference_rank6.json logs/problem_1/agent/trace_reference_rank7.json logs/problem_1/reference/candidate_outputs/rank_0.pt logs/problem_1/reference/candidate_outputs/rank_1.pt logs/problem_1/reference/candidate_outputs/rank_2.pt logs/problem_1/reference/candidate_outputs/rank_3.pt [1069 more files omitted for size] ``` ### Dependencies - frontend/api/requirements.txt: fastapi@>=0.115.0, pyyaml@>=6.0, uvicorn[standard]@>=0.32.0, watchfiles@>=1.0.0 - frontend/app/package.json: @monaco-editor/react@^4.7.0, @tailwindcss/postcss@^4, @tanstack/react-query@^5.90.21, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@16.1.6, framer-motion@^12.34.0, monaco-editor@^0.55.1, next@16.1.6, react@19.2.3, react-dom@19.2.3, react-resizable-panels@^4.6.4, recharts@^3.7.0, tailwindcss@^4, typescript@^5, zustand@^5.0.11 - requirements.txt: modal, openai, pydantic, PyYAML, rlms ### Recent commits (newest first) - ui fixes and loading issues - more experiments added claude sdk - frontend created - RAG added - fixed modal stalling on broken code, agentic loop works now with more visibility over changes form iteration to iteration - agent architecture and modal containers running - reference data and solutions data ## Key source files (fetched from GitHub, selected and truncated for size) ### AGENT_ARCHITECTURE_SLIDE.md ```markdown # EcholoKernel Agent Architecture — Slide Summary Copy the Mermaid block below to [mermaid.live](https://mermaid.live) → Export as PNG/SVG for Google Slides. ## Recommended (fits 16:9 slide) ```mermaid %%{init: {'theme':'dark'}}%% flowchart LR C[Config] --> B[Bootstrap] B --> E[Evaluate] E --> P[Propose] P --> E subgraph Retrieval R[RAG + RLM] end R -.-> P subgraph State M[Memory] end M -.-> P style C fill:#475569 style B fill:#475569 style E fill:#475569 style P fill:#475569 style M fill:#475569 style R fill:#475569 style State fill:#334155 style Retrieval fill:#334155 ``` ## Compact (minimal labels) ```mermaid flowchart LR C[Config] --> B[Bootstrap] B --> E[Evaluate] E --> P[Propose] P --> E R[RAG+RLM] -.-> P ``` ## Vertical (narrow layout) ```mermaid flowchart TD C[Config] --> B[Bootstrap] B --> E[Evaluate] E --> P[Propose] P --> E R[RAG + RLM] -.-> P ``` ``` ### AGENT_CODEBASE_GUIDE.md ```markdown # AGENT Codebase Guide This file is a handoff/reference doc for future agents and conversations. It summarizes what has been implemented, how the system is wired, and how to run it. ## Project Goal Build an iterative OpenAI-powered kernel optimizer that improves Triton solutions against reference PyTorch distributed kernels using Modal-based H100 evaluation, correctness checks, timing, and profiling feedback. ## What Was Added In This Implementation Pass ### 1) Distributed worker harness - Added `scripts/worker.py` - This is the runtime worker launched by `torchrun` from `run_modal.py`. - Responsibilities: - backend init/finalize (reference/triton/agent/numba_cuda) - dynamic module loading (reference and candidate solution) - standardized input generation - correctness check - timing benchmark (CUDA events + barriers) - optional `torch.profiler` traces - metrics artifact writing (`rank_*.json`, `summary_rank0.json`) ### 2) Input/output standardization + extension hook - Reused `utils/input_output_tensors.py` as the default input path. - Updated `create_input_tensor(...)` to support optional override functions. - Added `utils/problem_specs.py` as an override registry for edge-case problems. ### 3) Agent backend directory - Added `solutions_agent/` for editable candidate kernels. - Added seed wrappers: - `solutions_agent/1_agent.py` - `solutions_agent/2_agent.py` - `solutions_agent/10_agent.py` - These currently import corresponding Triton seeds from `solutions_triton/`. ### 4) Modular optimizer framework - Added `agent/` package with pluggable components. - Core orchestrator: - `agent/core/optimizer.py` - Bootstrap stage for unseen problems: - `agent/core/bootstrap.py` - Modal evaluation: - `agent/eval/modal_evaluator.py` - OpenAI integration: - `agent/openai_client.py` - Run logging and rollback: - `agent/core/run_log.py` - `agent/core/rollback.py` ### 5) Strategy architecture - Added strategy folders for easy variant testing: - `agent/strategies/proposers/` - `agent/strategies/memory/` - `agent/strategies/scorers/` - Added a registry factory: - `agent/strategies/registry.py` ### 6) Experiment runner + config - Added `run_experiment.py` - Added default config: - `experiments/default.yaml` - Added dependencies: - `requirements.txt` ## High-Level Architecture ### Single evaluation run 1. `modal run run_modal.py ...` 2. Modal launches 8x H100 and runs `torchrun` 3. `scripts/worker.py` evaluates reference + candidate 4. Artifacts are written to logs and downloaded locally ### Iterative optimization run 1. `python3 run_experiment.py --config experiments/default.yaml` 2. Bootstrap stage ensures a candidate exists in `solutions_agent/`: - copy seed from solved backend if available - otherwise generate initial candidate from reference + solved examples 3. Evaluator precomputes reference once and caches outputs/timing metadata 4. Evaluator runs candidate jobs and returns structured fee [truncated — 5139 more characters] ``` ### requirements.txt ``` modal openai pydantic PyYAML rlms ``` ### frontend/api/requirements.txt ``` fastapi>=0.115.0 uvicorn[standard]>=0.32.0 pyyaml>=6.0 watchfiles>=1.0.0 ``` ### frontend/app/package.json ``` { "name": "echolokernel-app", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@monaco-editor/react": "^4.7.0", "@tanstack/react-query": "^5.90.21", "framer-motion": "^12.34.0", "monaco-editor": "^0.55.1", "next": "16.1.6", "react": "19.2.3", "react-dom": "19.2.3", "react-resizable-panels": "^4.6.4", "recharts": "^3.7.0", "zustand": "^5.0.11" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.1.6", "tailwindcss": "^4", "typescript": "^5" } } ``` ### frontend/api/main.py ```python """FastAPI backend for the EcholoKernel Frontend Dashboard.""" from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware from .routers import runs, experiments, launch from .ws.handler import router as ws_router app = FastAPI( title="EcholoKernel API", description="Backend API for the EcholoKernel dashboard", version="1.0.0", ) # CORS - allow Next.js dev server app.add_middleware( CORSMiddleware, allow_origins=["http://localhost:3000", "http://127.0.0.1:3000"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Mount routers app.include_router(runs.router) app.include_router(experiments.router) app.include_router(launch.router) app.include_router(ws_router) @app.get("/api/health") async def health(): return {"status": "ok"} ``` ### frontend/app/src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { JetBrains_Mono } from "next/font/google"; import "./globals.css"; import { Providers } from "./providers"; const jetbrainsMono = JetBrains_Mono({ variable: "--font-mono", subsets: ["latin"], weight: ["300", "400", "500", "600", "700", "800"], }); export const metadata: Metadata = { title: "EcholoKernel — Triton Kernel Optimizer", description: "Iterative optimization with OpenAI + Modal H100 evaluation", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### frontend/app/src/app/page.tsx ```typescript "use client"; import { useState, useCallback } from "react"; import { useRouter } from "next/navigation"; import { motion, AnimatePresence } from "framer-motion"; import { useQueryClient } from "@tanstack/react-query"; import { useRuns } from "@/hooks/use-run-data"; import { RunCard } from "@/components/run-card"; import { RunLauncher } from "@/components/run-launcher"; import { ExperimentBuilder } from "@/components/experiment-builder"; import { api } from "@/lib/api-client"; export default function DashboardPage() { const { data: runs, isLoading } = useRuns(); const [showBuilder, setShowBuilder] = useState(false); const [demoLaunching, setDemoLaunching] = useState(false); const router = useRouter(); const queryClient = useQueryClient(); const handleDemoLaunch = useCallback( async (name: string, delayMs = 30000) => { setDemoLaunching(true); try { const result = await api.launchDemo(name); setShowBuilder(false); queryClient.invalidateQueries({ queryKey: ["runs"] }); router.push(`/runs/${result.run_id}?demo=true&delay=${delayMs}`); } catch (e) { console.error("Demo launch failed:", e); } finally { setDemoLaunching(false); } }, [queryClient, router] ); return (
{/* Header */}

EcholoKernel

Iterative optimization with LLMs (OpenAI, Anthropic) + Modal H100 evaluation

{/* Action Buttons */}
{/* Experiment Builder */} {showBuilder && (
setShowBuilder(false)} onLaunch={handleDemoLaunch} />
)} {/* Run List */}

Optimization Runs

{isLoading ? (
Loading runs...
) : runs && runs.length > 0 ? (
{runs.map((run, i) => ( ))}
) : (

No optimization runs yet

Launch a new run to get started

)}
); } ``` ### frontend/app/src/app/runs/[runId]/page.tsx ```typescript "use client"; import { useEffect, useCallback, useRef, useState } from "react"; import { useParams, useSearchParams } from "next/navigation"; import Link from "next/link"; import { motion } from "framer-motion"; import { useQueryClient } from "@tanstack/react-query"; import { useRunDetail, useIterations } from "@/hooks/use-run-data"; import { useWebSocket } from "@/hooks/use-websocket"; import { useRunStore } from "@/stores/run-store"; import { IterationHistory } from "@/components/iteration-history"; import { CodeViewer } from "@/components/code-viewer"; import { BottomPanel } from "@/components/bottom-panel"; import { api } from "@/lib/api-client"; import type { WSEvent } from "@/lib/types"; function ConnectionIndicator({ connected, isDemo, }: { connected: boolean; isDemo?: boolean; }) { if (isDemo) { return (
OPTIMIZING
); } return (
{connected ? "LIVE" : "DISCONNECTED"}
); } function DemoProgressBar({ current, total, }: { current: number; total: number; }) { const pct = total > 0 ? (current / total) * 100 : 0; return (
PROGRESS
{current}/{total}
); } export default function RunDetailPage() { const params = useParams(); const searchParams = useSearchParams(); const runId = params.runId as string; const isDemoParam = searchParams.get("demo") === "true"; const delayMs = Number(searchParams.get("delay")) || 30000; const queryClient = useQueryClient(); const { data: run, isLoading: runLoading } = useRunDetail(runId); const { data: iterations } = useIterations(runId); const { selectedIteration, setSelectedIteration, activeProblemId, setActiveProblemId, } = useRunStore(); // Demo simulation state const [isDemo, setIsDemo] = useState(isDemoParam); const [demoRunning, setDemoRunning] = useState(false); const [demoIteration, setDemoIteration] = useState(0); const [demoTotal, setDemoTotal] = useState(10); const demoTimerRef = useRef(null); const demoRunningRef = useRef(false); // WebSocket for live updates (non-demo) const handleWSEvent = useCallback( (event: WSEvent) => { if (event.type === "iteration_event" || event.type === "llm_proposal") { queryClient.invalidateQueries({ queryKey: ["iterations", runId] }); queryClient.invalidateQueries({ queryKey: ["run", runId] }); } if (event.type === "snapshot") { queryClient.invalidateQueries({ queryKey: ["snapshots"] }); queryClient.invalidateQueries({ queryKey: ["snapshot-code"] }); } }, [queryClient, runId] ); const { connected } = useWebSocket({ runId, enabled: run?.status === "running" && !isDemo, onEvent: handleWSEvent, }); // Demo: auto-advance iterations const advanceDemoIteration = useCallback(async () => { if (!demoRunningRef.current) return; try { const result = await api.advanceDemo(runId); if (result.status === "complete" || result.is_complete) { setDemoRunning(false); demoRunningRef.current = false; setDemoIteration(result.iteration || demoTotal); // Refetch everything queryClient.invalidateQueries({ queryKey: ["iterations", runId] }); queryClient.invalidateQueries({ queryKey: ["run", runId] }); queryClient.invalidateQueries({ queryKey: ["snapshots"] }); queryClient.invalidateQueries({ queryKey: ["snapshot-code"] }); return; } const newIter = result.iteration; setDemoIteration(newIter); setDemoTotal(result.total); // Refetch data so UI updates queryClient.invalidateQueries({ queryKey: ["iterations", runId] }); queryClient.invalidateQueries({ queryKey: ["run", runId] }); queryClient.invalidateQueries({ queryKey: ["snapshots"] }); queryClient.invalidateQueries({ queryKey: ["snapshot-code"] }); // Auto-select the new iteration setSelectedIteration(newIter); // Schedule next iteration if (!result.is_complete && demoRunningRef.current) { demoTimerRef.current = setTimeout(advanceDemoIteration, delayMs); } } catch (e) { console.error("Demo advance failed:", e); setDemoRunning(false); demoRunningRef.current = false; } }, [runId, queryClient, setSelectedIteration, demoTotal, delayMs]); // Start demo auto-play when page loads with demo=true useEffect(() => { if (isDemoParam && !demoRunning && !demoRunningRef.current) { // Small delay to let the page render first const startTimer = setTimeout(() => { setDemoRunning(true); demoRunningRef.current = true; setIsDemo(true); advanceDemoIteration(); }, 1500); return () => clearTimeout(startTimer); } }, [isDemoParam]); // eslint-disable-line react-hooks/exhaustive-deps // Cleanup on unmount useEffect(() => { return () => { [truncated — 6564 more characters] ``` ### run_experiment.py ```python #!/usr/bin/env python3 from __future__ import annotations import argparse from pathlib import Path def main() -> None: parser = argparse.ArgumentParser(description="Run modular Triton optimization experiment") parser.add_argument( "--config", type=str, default="experiments/default.yaml", help="Path to experiment YAML config", ) args = parser.parse_args() from agent.config import ExperimentConfig from agent.core.optimizer import Optimizer repo_root = Path(__file__).resolve().parent cfg = ExperimentConfig.from_yaml(repo_root / args.config) optimizer = Optimizer(repo_root=repo_root, cfg=cfg) run_dir = optimizer.run() print(f"Experiment complete: {run_dir}") if __name__ == "__main__": main() ``` [213 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]