# Project export: Codex Benchmark Guardian This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: A developer tool that detects performance regressions, creates bounded Codex repair goals, and verifies fixes through protected checks before merge. - Devpost: https://devpost.com/software/codex-benchmark-guardian - GitHub: https://github.com/OmprakashSahani/codex-benchmark-guardian - Demo: https://codex-benchmark-guardian.vercel.app/ - Video: https://www.youtube.com/embed/nw5s-3TwPQY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Omprakash Sahani (80 commits) ## Devpost submission (written by the team) ### Overview A developer tool that detects performance regressions, creates bounded Codex repair goals, and verifies fixes through protected checks before merge. Live dashboard: https://codex-benchmark-guardian.vercel.app Source code: https://github.com/OmprakashSahani/codex-benchmark-guardian Real regression-to-fix proof: https://github.com/OmprakashSahani/codex-benchmark-guardian/pull/20 ### Inspiration Functional tests can pass while performance quietly gets worse. A pull request may preserve correct behavior but increase latency, consume more memory, slow down an important workflow, or reduce throughput. These regressions are often discovered only after deployment, when they are more expensive and risky to investigate. Many benchmark tools can compare numbers or fail a CI job, but the workflow often stops at the alert. Developers must still determine why the change matters, investigate the likely cause, prepare a focused repair task, verify that the repair genuinely restored performance, and ensure that the benchmark policy was not weakened merely to obtain a passing result. I built Codex Benchmark Guardian to close that gap. It turns benchmark evidence into a clear release decision, actionable triage, a bounded Codex repair goal, and protected verification before merge. The project follows one central principle: Codex may help investigate and repair a regression, but it must not control the benchmark policy, manipulate the evidence, approve its own work, or merge the pull request. ### What it does Codex Benchmark Guardian compares baseline and current benchmark evidence and determines whether a proposed change introduces a material performance regression. It supports metrics where higher values are worse, including: latency; runtime; memory usage; error rate. It also supports metrics where lower values are worse, including: throughput; accuracy; recall; success rate. For every comparison, Guardian can produce: percentage changes; regression decisions; severity classifications; deterministic release-readiness scoring; metric-specific triage guidance; Markdown and HTML reports; a GitHub issue handoff; a bounded Codex Goal; an immutable Repair Contract; CI and pull-request guardrail artifacts. The complete workflow is: benchmark evidence → regression decision → triage → bounded Codex repair → protected verification → human approval The project can be used through: a production Next.js dashboard; a FastAPI analysis endpoint; a Python command-line interface; an additional Streamlit interface; a protected GitHub pull-request benchmark gate. Production experience The primary product experience is a production dashboard built with Next.js 15, TypeScript, Tailwind CSS, FastAPI, and a deterministic Python analysis engine. The results are organized into five sections: Overview Metrics Triage Codex Goal Handoff Pack Developers can: choose from permanent example scenarios; upload or edit baseline and current benchmark JSON; configure per-metric directions; choose a regression threshold; inspect readiness, severity, and metric changes; review investigation guidance; copy or download the generated Codex Goal; inspect the complete Handoff Pack; replay a real pull request from regression to verified repair. The frontend does not duplicate the benchmark or repair policy. It sends the evidence to FastAPI, while the Python engine remains the only source of truth. The verified Codex Repair Loop The central feature of the project is the verified Codex Repair Loop. When a material regression is detected, Guardian builds an immutable Repair Contract from the benchmark evidence. The contract defines: whether a repair is required; the exact evidence requiring attention; the repair objective; actions Codex is allowed to perform; actions Codex is forbidden from performing; required validation commands; protected completion criteria; conditions under which Codex must stop and report. For a genuine regression, Codex may: inspect the relevant implementation and history; form evidence-backed root-cause hypotheses; implement the smallest maintainable correction; add or update regression tests; run approved project checks; rerun the relevant protected benchmark; review the final diff. Codex may not: lower or bypass regression thresholds; change metric directions to hide a failure; modify or replace the original benchmark evidence; selectively choose a passing run; weaken the protected harness or evaluator; delete or relax tests merely to pass; hard-code expected benchmark results; declare the pull request Ready; merge automatically. A repair is complete only when: all required project checks pass; fresh evidence is produced by the protected workflow or trusted evaluator; the fresh evidence reports zero material regressions; the release status is Ready; the final diff has been reviewed; a human approves the merge. This prevents the repair process from changing the rules used to judge its own success. Verification-only behavior A trustworthy agent workflow must also know when not to change code. When valid benchmark evidence contains no material regressions, Guardian creates a verification-only contract. In this state: no implementation change is authorized; no test modification is authorized; speculative cleanup and refactoring are prohibited; Codex inspects and verifies the supplied evidence; protected verification is still required; human approval is still required; the system reports that no code repair is needed. This distinction prevents an AI agent from inventing work simply because it was given access to a repository. Proven on a real pull request I validated the full workflow using a real pull request in the project repository. The pull request introduced an intentionally inefficient repeated lookup in PR-comment generation. Functional tests continued to pass, but the protected benchmark gate detected a serious performance regression: Metric: pr_gate_generation_latency_ms Regression: +135.21% Severity: Critical Release readiness: Needs Review Score: 70/100 Guardian generated benchmark evidence, investigation guidance, a Codex handoff, validation commands, and protected finish conditions. Codex traced the regression to a repeated linear lookup performed for every regressed metric, producing approximately quadratic behavior. The focused repair: removed the redundant lookup; restored linear behavior; preserved output content and ordering; added deterministic regression coverage. The protected workflow then produced fresh evidence: Material regressions: 0 Release readiness: Ready Score: 100/100 The benchmark threshold, metric directions, protected evaluator, provenance, and human approval requirement remained unchanged throughout the repair. This demonstrates the project’s main value: Guardian can detect a performance problem that ordinary functional testing misses and guide a safe repair without allowing the repair process to weaken the guardrail. Protected pull-request benchmark gate The project includes a protected GitHub pull-request benchmark workflow. It compares the exact protected base revision and pull-request head using: the same GitHub-hosted runner; separate restricted containers; identical workload and measurement settings; a benchmark harness copied from the protected base branch; a trusted evaluator from the protected base branch; deterministic benchmark evidence and provenance. The untrusted pull-request code cannot modify the evaluator, threshold, metric-direction policy, protected baseline, or workflow definition used to judge it. A separate trusted publisher validates the workflow and pull-request identity before posting or updating the benchmark decision on GitHub. The publisher never executes pull-request code. This separation is important because an automated performance gate is only trustworthy when the code being evaluated cannot redefine the rules used to evaluate itself. Codex Handoff Pack Guardian generates a complete set of artifacts that developers can inspect, assign, download, or use in another workflow. Depending on the interface and whether a repair is required, the Handoff Pack includes equivalent forms of: the Codex repair goal; the Repair Contract in Markdown and JSON; benchmark reports in Markdown and HTML; release-readiness guidance; regression triage; a GitHub issue handoff; a generated CI workflow; pull-request decision content; trusted gate summaries. When a regression exists, Guardian exposes the bounded repair workflow. When no repair is required, speculative implementation instructions are hidden and only verification-safe guidance is presented. Every representation is generated deterministically from the same Python evidence and policy. How I built it The project uses: Python 3.12 for benchmark analysis and repair policy; Typer for the command-line interface; Rich for readable terminal output; FastAPI for the production analysis API; Next.js 15 and TypeScript for the production frontend; Tailwind CSS for responsive presentation; Streamlit for an additional local Python-first interface; Pytest for automated testing; Ruff for linting and formatting; GitHub Actions and Docker for protected pull-request evaluation; deterministic Markdown, HTML, JSON, and YAML generation. The project currently includes 162 automated Python tests, along with frontend linting, TypeScript checks, production builds, workflow validation, protected benchmark checks, and manual production verification. Judges and developers can start with the live dashboard or run the project locally: The project supports modern desktop and mobile browsers for the hosted dashboard. The local Python tools require Python 3.12 or newer, while the protected repository gate runs through GitHub Actions and Docker on Ubuntu runners. How I used Codex and GPT-5.6 I used Codex with GPT-5.6 throughout the project as an engineering collaborator. Codex helped with: implementation planning; focused code changes; test generation; debugging; CLI and API development; frontend integration; workflow hardening; documentation; pull-request review; security and edge-case analysis. Codex contributed to features including: multi-metric benchmark comparison; higher-is-worse and lower-is-worse metric handling; deterministic readiness scoring; regression triage; Markdown and HTML reporting; CI failure behavior; the Handoff Pack; GitHub issue and workflow generation; the protected PR benchmark gate; FastAPI and Next.js integration; the Repair Contract; repair-required and verification-only Codex Goals. Codex reviews also found meaningful issues that were corrected before merge. For example: verification-only contracts initially inherited actions that could authorize implementation and test changes; verification-only Codex Goals initially ended with an unconditional repair-loop instruction; fresh protected verification evidence needed to be distinguished from immutable original failing evidence; a legacy speculative fix prompt remained accessible after switching to a verification-only result; stale artifact selection could preserve content from a previous dashboard scenario; protected workflow paths and read-only container behavior required focused corrections. These findings improved correctness, security, compatibility, and the safety of the agent workflow. I retained responsibility for: defining the product problem; selecting the architecture; designing the readiness policy; establishing the protected evidence model; deciding the Codex safety boundaries; reviewing and accepting changes; validating production behavior; approving and merging pull requests. Codex assisted with implementation and review. It did not define the benchmark policy, approve its own work, declare a repair successful, deploy the application, or merge changes. Challenges Separating real regressions from benchmark noise Performance measurements can fluctuate because of runner noise and environmental variation. I addressed this through paired base-versus-head measurements on the same runner, consistent benchmark settings, explicit thresholds, deterministic evaluation, and preserved provenance. Protecting the evaluator from untrusted code A pull request should not be able to change the workflow, harness, or evaluator that decides whether it passes. Designing a secure boundary between protected base code, untrusted pull-request code, and a write-capable GitHub publisher required careful workflow architecture, isolated containers, restricted permissions, and several rounds of testing and review. Supporting different metric meanings An increase is harmful for latency but beneficial for throughput. The system therefore supports per-metric directions instead of applying one assumption to every benchmark value. Designing safe Codex behavior A prompt that simply says “fix the benchmark” could encourage threshold changes, evidence manipulation, test weakening, overfitting, or unnecessary edits. The Repair Contract required a more explicit model of: allowed actions; prohibited changes; trusted evidence; completion criteria; stop conditions; human authority. Keeping every interface consistent The CLI, Streamlit app, FastAPI service, Next.js dashboard, GitHub workflows, reports, and generated artifacts all needed to use the same underlying policy. I kept the Python engine as the canonical source of truth and avoided duplicating decision logic in the frontend. What I learned This project taught me that adding an AI agent to a developer workflow is not only a prompting problem. It is also a systems, security, evidence, and authority-design problem. I learned how to: design deterministic benchmark comparisons; model metric direction and severity correctly; create CI-friendly release decisions; build trusted base-versus-head performance evaluation; preserve benchmark provenance; separate untrusted execution from trusted publication; turn benchmark evidence into bounded agent instructions; distinguish repair from verification; design for human approval rather than uncontrolled automation; use Codex iteratively for implementation, debugging, testing, and review. Most importantly, I learned that trustworthy agentic development requires more than asking an AI to produce a patch. The workflow must define what the agent is allowed to change, preserve the evidence used to judge it, verify the result independently, and keep final authority with a human. ### What's next The next stage would focus on making Guardian easier to adopt across more engineering stacks. Potential improvements include: adapters for pytest-benchmark, ASV, JMH, k6, Locust, Lighthouse, and ML evaluation outputs; historical benchmark storage and trend visualization; repository-specific readiness policies; organization-level dashboards; GitHub App installation and configuration; richer measurement-noise analysis; ownership routing and notifications for affected components. The core principle will remain unchanged: Codex can help repair performance regressions, but trusted evidence and human approval must determine whether a change is safe to merge. ## README (from the GitHub repository)
# Codex Benchmark Guardian ### Evidence-backed performance repair for pull requests [Production dashboard](https://codex-benchmark-guardian.vercel.app) · [Repository](https://github.com/OmprakashSahani/codex-benchmark-guardian) · [Real PR #20 proof](https://github.com/OmprakashSahani/codex-benchmark-guardian/pull/20)
--- **Codex Benchmark Guardian turns a real performance regression into a bounded Codex repair task, verifies the proposed fix against protected benchmark evidence, and safely returns the pull request to Ready.** It connects deterministic benchmark analysis to a conditional repair workflow without allowing an agent to decide its own success. Python is the only source of benchmark, readiness, and repair-policy truth; the production Next.js dashboard and FastAPI API expose that engine without reimplementing its business rules. The final merge remains a human decision. ## Why It Matters A benchmark failure is only the start of a repair. A useful system must preserve the evidence, explain what regressed, constrain what may change, distinguish repair from verification, and prove the outcome with a trusted evaluator. Codex Benchmark Guardian provides that chain: - deterministic comparison across higher-is-worse and lower-is-worse metrics - severity classification, readiness scoring, and evidence-backed triage - an immutable Repair Contract and a conditional Codex Goal - bounded repair instructions only when material regressions exist - a verification-only path that avoids speculative changes when evidence is clean - a protected GitHub PR gate that makes the final benchmark-readiness decision - a complete, portable Handoff Pack for review and automation ## Judge Quickstart Start with the [live production dashboard](https://codex-benchmark-guardian.vercel.app), then verify the repository locally: ```bash git clone https://github.com/OmprakashSahani/codex-benchmark-guardian.git cd codex-benchmark-guardian pip install -e ".[dev,dashboard]" make lint make format-check make test npm install npm run lint npm run typecheck npm run build ``` Expected Python test result: ```text 162 passed ``` Generate a CLI handoff and exercise both protected-gate outcomes: ```bash make demo-handoff make demo-pr-gate-block make demo-pr-gate-ready ``` The first command writes the CLI Handoff Pack to `reports/handoff/`. The gate demonstrations intentionally show a blocked regression and a Ready result. For the additional local Python-first interface: ```bash streamlit run app.py ``` ## Live Production Experience The primary product experience is the [production Next.js/FastAPI dashboard](https://codex-benchmark-guardian.vercel.app). Its result workspace is organized into five tabs: 1. **Overview** 2. **Metrics** 3. **Triage** 4. **Codex Goal** 5. **Handoff Pack** Users can: - load permanent example scenarios - upload or edit baseline, current, and direction JSON - configure the threshold and per-metric directions - run the deterministic Python analysis engine through FastAPI - inspect readiness, metrics, severity, and triage - copy or download the conditional Codex Goal - inspect and download the complete Handoff Pack - replay the real PR #20 regression and verified fix The dashboard defaults to a **10%** regression threshold. This repository's protected PR gate uses **25%** to evaluate pull requests. Until a new submission video is recorded, the production dashboard is the primary demo. The existing [YouTube video](https://youtu.be/nw5s-3TwPQY) is an earlier prototype demonstration and does not represent the current production experience. ## Verified Repair Loop ```mermaid flowchart LR A[Benchmark evidence] --> B[Deterministic regression analysis] B --> C[Readiness decision] C --> D[Immutable Repair Contract] D --> E[Conditional Codex Goal] E --> F{Repair required?} F -->|Yes| G[Bounded repair workflow] F -->|No| H[Verification-only workflow] G --> I[Protected benchmark verification] H --> I I --> J{Zero material regressions
and Ready?} J -->|No| B J -->|Yes| K[Human approval] K --> L[Merge] ``` ### Repair-required path When material regressions exist, the generated goal directs Codex to: 1. inspect the relevant implementation and history 2. identify evidence-backed root-cause hypotheses 3. implement the smallest maintainable correction 4. add or update regression tests 5. run only approved validation commands 6. rerun the relevant benchmark 7. review the final diff 8. obtain fresh protected evidence 9. confirm zero material regressions and **Ready** status 10. leave merge approval to a human Failure or incomplete evidence repeats the bounded analysis-and-repair loop; it does not authorize policy changes. ### Verification-only path When the result has zero material regressions, the system does not invent repair work. The generated goal requires the user or agent to: 1. make no speculative code repair or speculative test change 2. inspect the supplied evidence 3. confirm that the protected workflow or trusted evaluator completed 4. confirm zero material regressions and **Ready** status 5. review the final diff only when changes already exist 6. preserve human approval before merge 7. continue monitoring benchmark stability The dashboard hides the legacy speculative fix prompt from verification-only users. The API retains its `codex_fix_prompt` field solely for backward compatibility; `codex_repair_goal.md` is the canonical handoff for both paths. ## Real Regression-to-Fix Proof: PR #20 [Pull request #20](https://github.com/OmprakashSahani/codex-benchmark-guardian/pull/20) demonstrates the complete loop with real protected evidence. | Stage | Result | | --- | --- | | Regression metric | `pr_gate_generation_latency_ms` | | Harmful change | **+135.21%** | | Protected threshold | **25%** | | Severity | **critical** | | Initial readiness | **Needs Review** | | Initial score | **70/100** | | Verified fix | **zero material regressions** | | Final readiness | **Ready** | | Final score | **100/100** | - [Regression workflow](https://github.com/OmprakashSahani/codex-benchmark-guardian/actions/runs/29634627579) - [Verified-fix workflow](https://github.com/OmprakashSahani/codex-benchmark-guardian/actions/runs/29635217444) Workflow artifacts eventually expire, so the evidence used to replay both states is committed permanently under `examples/scenarios/pr-20/`. The protected verifier established benchmark readiness; Codex did not autonomously merge the pull request. ## Repair Contract and Safety Model Every analysis produces an immutable Repair Contract from the Python engine. It records the evidence, whether repair is required, allowed and forbidden actions, validation commands, and completion criteria. The Codex Goal is derived from this contract rather than from an open-ended request to improve performance. Codex may inspect, edit, test, benchmark, and review a bounded repair. It may not: - lower or bypass thresholds - change metric directions to suppress failures - manipulate original or fresh benchmark evidence - substitute a selectively chosen passing run - weaken the protected harness - weaken the protected evaluator - relax or delete tests merely to pass - hard-code expected benchmark results - self-declare the pull request Ready - automatically merge the pull request Protected verification declares benchmark readiness. A human still reviews and approves the merge. ## Handoff Pack `codex_repair_goal.md` is always the primary and default handoff. Dashboard downloads use review-friendly names. ### Dashboard artifacts Repair-required results expose: 1. `codex_repair_goal.md` 2. `repair_contract.md` 3. `repair_contract.json` 4. `benchmark-report.md` 5. `benchmark-report.html` 6. `codex-fix-prompt.txt` 7. `github-issue.md` 8. `benchmark-workflow.yml` 9. `release-readiness.md` Verification-only results expose the same set except `codex-fix-prom [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 62 recognized source files, 333 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Streamlit (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Docker (technology) — claimed on Devpost, not found in the code - Node.js (technology) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Codex — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (93 of 93) ``` .github/workflows/benchmark-pr-gate-publish.yml .github/workflows/benchmark-pr-gate.yml .github/workflows/ci.yml .gitignore AGENTS.md api/__init__.py api/index.py app.py app/globals.css app/layout.tsx app/page.tsx benchmarks/directions.json benchmarks/run_project_benchmarks.py components/analysis-workspace.tsx components/artifact-viewer.tsx components/codex-fix-panel.tsx components/decision-summary.tsx components/handoff-pack.tsx components/header.tsx components/json-upload.tsx components/metric-change-chart.tsx components/metrics-table.tsx components/pr-replay.tsx components/pr-story.tsx components/results.tsx components/theme-provider.tsx components/theme-toggle.tsx components/triage-panel.tsx components/ui.tsx eslint.config.mjs examples/baseline.json examples/current_no_regression.json examples/current.json examples/directions.json examples/pr_gate_current_block.json examples/pr_gate_current_ready.json examples/pr_gate_current.json examples/scenarios/clean/baseline.json examples/scenarios/clean/current.json examples/scenarios/clean/directions.json examples/scenarios/pr-20/regression/baseline.json examples/scenarios/pr-20/regression/current.json examples/scenarios/pr-20/regression/directions.json examples/scenarios/pr-20/verified-fix/baseline.json examples/scenarios/pr-20/verified-fix/current.json examples/scenarios/pr-20/verified-fix/directions.json examples/scenarios/scenarios.json examples/scenarios/standard/baseline.json examples/scenarios/standard/current.json examples/scenarios/standard/directions.json lib/example-scenarios.ts lib/number-format.ts lib/pr-demo.ts lib/sample-data.ts lib/types.ts LICENSE Makefile next-env.d.ts next.config.ts package.json postcss.config.mjs pyproject.toml README.md reports/benchmark_guardian_ci.yml reports/codex_fix_prompt.md reports/report.html reports/report.md requirements.txt src/codex_benchmark_guardian/__init__.py src/codex_benchmark_guardian/benchmarks.py src/codex_benchmark_guardian/ci.py src/codex_benchmark_guardian/cli.py src/codex_benchmark_guardian/handoff.py src/codex_benchmark_guardian/pr_gate.py src/codex_benchmark_guardian/regression.py src/codex_benchmark_guardian/release_readiness.py src/codex_benchmark_guardian/repair.py src/codex_benchmark_guardian/report.py src/codex_benchmark_guardian/triage.py tests/test_api.py tests/test_benchmark_runner.py tests/test_benchmarks.py tests/test_ci.py tests/test_cli.py tests/test_example_scenarios.py tests/test_pr_gate.py tests/test_regression.py tests/test_release_readiness.py tests/test_repair.py tests/test_report.py tests/test_triage.py tsconfig.json uv.lock ``` ### Dependencies - package.json: @eslint/eslintrc@^3.2.0, @tailwindcss/postcss@^4.0.0, @types/node@^22.10.0, @types/react@^19.0.0, @types/react-dom@^19.0.0, eslint@^9.16.0, eslint-config-next@^15.1.0, lucide-react@^0.468.0, next@^15.1.0, next-themes@^0.4.4, postcss@^8.4.49, react@^19.0.0, react-dom@^19.0.0, tailwindcss@^4.0.0, typescript@^5.7.0 - pyproject.toml: fastapi@>=0.115.0, httpx@>=0.27.0, pytest@>=8.0.0, PyYAML@>=6.0, rich@>=13.0.0, ruff@>=0.6.0, streamlit@>=1.36.0, typer@>=0.12.0, uvicorn@>=0.30.0 ### Recent commits (newest first) - Fix YouTube video link in README - Update README with architecture details and Codex usage - Refresh README for final Build Week submission (#30) - Expose verified Codex Repair Loop in API and dashboard (#29) - Add Codex repair contract foundation (#27) - Add example benchmark scenarios and JSON guidance (#26) - Polish dashboard and add live PR replay (#25) - Add polished Vercel dashboard foundation (#24) - Demonstrate Benchmark Gate Needs Review to Ready flow (#20) - Fix benchmark publisher artifact paths (#23) - Fix PR gate install from read-only source (#22) - Fix benchmark gate container output permissions (#21) - Merge pull request #19 from OmprakashSahani/codex/add-github-pr-benchmark-gate-feature - Finalize paired gate provenance and immutable image use - Complete paired gate evidence and runtime isolation - Harden paired runtime image workflow YAML - Pair benchmark measurements on one isolated runner - Restrict trusted publisher to PR head repository - Align CI tests with protected full gate workflow - Assert protected three-job gate architecture ## Key source files (fetched from GitHub, selected and truncated for size) ### AGENTS.md ```markdown # Codex Benchmark Guardian Repository Guide ## Architecture - `src/codex_benchmark_guardian/` contains the Python benchmark, regression, readiness, reporting, and gate logic. This Python engine is the source of truth. - `api/` exposes the Python engine through FastAPI. - `app/`, `components/`, and `lib/` contain the Next.js dashboard. It renders engine results and must not duplicate benchmark business logic. - `app.py` is the supported Streamlit interface. - `benchmarks/` contains the protected benchmark harness; `tests/` covers the Python engine and interfaces; `examples/` contains sample evidence and configuration. - The protected GitHub evaluator makes the final PR readiness decision. ## Repair Boundaries Prefer minimal root-cause fixes. Codex may investigate, edit, test, benchmark, review the diff, and prepare a patch. It may not declare its patch Ready or merge automatically. Only protected benchmark evidence may return a PR to Ready, and human review is required before merge. Never lower thresholds to make a repair pass, alter metric directions to hide a regression, weaken, skip, replace, or bypass the protected harness or evaluator, or delete or skip tests to obtain Ready. ## Required Checks Run `make lint`, `make format-check`, `make test`, `npm run lint`, `npm run typecheck`, and `npm run build`. Performance repairs must also run the relevant benchmark or PR gate. ``` ### reports/report.md ```markdown # Benchmark Comparison Report Compared metrics: 4 Regressions detected: 2 | Metric | Direction | Baseline | Current | Change | Threshold | Status | Severity | | --- | --- | ---: | ---: | ---: | ---: | --- | --- | | latency_ms | higher_is_worse | 100 | 125 | 25.00% | 10.00% | Regression | high | | memory_mb | higher_is_worse | 256 | 260 | 1.56% | 10.00% | OK | none | | runtime_s | higher_is_worse | 2.5 | 2.7 | 8.00% | 10.00% | OK | none | | throughput_rps | lower_is_worse | 1000 | 850 | -15.00% | 10.00% | Regression | medium | ## Regression Triage ### latency_ms - **Likely area:** Request path latency or dependency wait time - **Why it matters:** Higher latency slows developer and user workflows and can hide downstream bottlenecks. - **Suggested checks:** - Inspect recent changes on the hot path for added I/O, sleeps, retries, or serialization work. - Compare dependency timing, network calls, database queries, and cache hit rates against the baseline. - Check benchmark host load and input size to rule out environmental noise. ### throughput_rps - **Likely area:** Capacity, concurrency, or request processing rate - **Why it matters:** Lower throughput means the system handles less work with the same resources. - **Suggested checks:** - Review concurrency limits, worker counts, queue behavior, and backpressure changes. - Inspect CPU, lock contention, database pool usage, and external service rate limits. - Verify the benchmark duration and request mix match the baseline run. ``` ### requirements.txt ``` -e .[dashboard] ``` ### package.json ``` { "name": "codex-benchmark-guardian-dashboard", "version": "1.0.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint . --max-warnings=0", "typecheck": "tsc --noEmit" }, "dependencies": { "lucide-react": "^0.468.0", "next": "^15.1.0", "next-themes": "^0.4.4", "react": "^19.0.0", "react-dom": "^19.0.0" }, "devDependencies": { "@eslint/eslintrc": "^3.2.0", "@types/node": "^22.10.0", "@types/react": "^19.0.0", "@types/react-dom": "^19.0.0", "eslint": "^9.16.0", "eslint-config-next": "^15.1.0", "postcss": "^8.4.49", "tailwindcss": "^4.0.0", "@tailwindcss/postcss": "^4.0.0", "typescript": "^5.7.0" } } ``` ### pyproject.toml ``` [project] name = "codex-benchmark-guardian" version = "0.1.0" description = "A Codex-powered developer tool for detecting performance regressions and improving software reliability." readme = "README.md" requires-python = ">=3.12" dependencies = [ "typer>=0.12.0", "rich>=13.0.0", "fastapi>=0.115.0", "uvicorn>=0.30.0", ] [project.optional-dependencies] dashboard = [ "streamlit>=1.36.0", ] dev = [ "httpx>=0.27.0", "pytest>=8.0.0", "ruff>=0.6.0", "PyYAML>=6.0", ] [project.scripts] cbg = "codex_benchmark_guardian.cli:app" [tool.ruff] line-length = 100 [tool.ruff.lint] select = ["E", "F", "I", "B"] [build-system] requires = ["setuptools>=68.0"] build-backend = "setuptools.build_meta" [tool.pytest.ini_options] pythonpath = ["src", "."] ``` ### app.py ```python from __future__ import annotations import json from typing import Any import streamlit as st from codex_benchmark_guardian.benchmarks import ( compare_benchmark_metrics, parse_benchmark_metrics, parse_directions_config, ) from codex_benchmark_guardian.ci import ( generate_github_actions_workflow, select_dashboard_workflow_context, ) from codex_benchmark_guardian.regression import MetricDirection from codex_benchmark_guardian.release_readiness import ( calculate_release_readiness, generate_release_readiness_markdown, ) from codex_benchmark_guardian.report import ( generate_codex_fix_prompt, generate_github_issue, generate_html_report, generate_markdown_report, ) from codex_benchmark_guardian.triage import generate_triage_notes SAMPLE_BASELINE = { "latency_ms": 100.0, "memory_mb": 256.0, "runtime_s": 2.5, "throughput_rps": 1000.0, } SAMPLE_CURRENT = { "latency_ms": 125.0, "memory_mb": 260.0, "runtime_s": 2.7, "throughput_rps": 850.0, } SAMPLE_DIRECTIONS = { "latency_ms": MetricDirection.HIGHER_IS_WORSE, "memory_mb": MetricDirection.HIGHER_IS_WORSE, "runtime_s": MetricDirection.HIGHER_IS_WORSE, "throughput_rps": MetricDirection.LOWER_IS_WORSE, } def _load_uploaded_json(uploaded_file: Any) -> Any: if uploaded_file is None: return None return json.loads(uploaded_file.getvalue().decode("utf-8")) def _results_table(results: list[Any]) -> list[dict[str, str | float]]: return [ { "metric name": result.metric_name, "direction": result.direction.value, "baseline": result.baseline_value, "current": result.current_value, "change percent": f"{result.change_percent:.2f}%", "threshold": f"{result.threshold_percent:.2f}%", "status": "Regression" if result.is_regression else "OK", "severity": result.severity, } for result in results ] st.set_page_config(page_title="Codex Benchmark Guardian", page_icon="🛡️", layout="wide") st.title("Codex Benchmark Guardian") st.write( "Upload baseline and current benchmark JSON files, configure a regression threshold, " "and run the same comparison engine used by the CLI to produce triage guidance, " "Codex fix prompts, reports, and CI guardrails." ) with st.sidebar: st.header("Analysis controls") threshold = st.number_input( "Regression threshold (%)", min_value=0.0, value=10.0, step=1.0, help=( "A metric is a regression when it moves in the worse direction " "by at least this percent." ), ) fallback_direction = st.selectbox( "Fallback metric direction", options=[direction.value for direction in MetricDirection], index=0, help="Used for metrics not listed in the directions config.", ) use_sample_data = st.checkbox("Use built-in sample data", value=True) st.subheader("Benchmark inputs") col1, col2, col3 = st.columns(3) with col1: baseline_upload = st.file_uploader("Baseline JSON", type="json") with col2: current_upload = st.file_uploader("Current JSON", type="json") with col3: directions_upload = st.file_uploader("Directions config JSON", type="json") run_analysis = st.button("Run Analysis", type="primary") if run_analysis: try: if use_sample_data: baseline_metrics = SAMPLE_BASELINE current_metrics = SAMPLE_CURRENT directions = SAMPLE_DIRECTIONS else: if baseline_upload is None or current_upload is None: st.error( "Upload both baseline and current benchmark JSON files, or use sample data." ) st.stop() baseline_metrics = parse_benchmark_metrics(_load_uploaded_json(baseline_upload)) current_metrics = parse_benchmark_metrics(_load_uploaded_json(current_upload)) directions_data = _load_uploaded_json(directions_upload) directions = ( parse_directions_config(directions_data) if directions_data is not None else None ) results = compare_benchmark_metrics( baseline_metrics=baseline_metrics, current_metrics=current_metrics, threshold_percent=threshold, direction=MetricDirection(fallback_direction), directions=directions, ) if not results: st.warning( "No matching numeric metrics were found between the baseline and current inputs." ) st.stop() regression_count = sum(result.is_regression for result in results) release_readiness = calculate_release_readiness(results) release_readiness_markdown = generate_release_readiness_markdown(results) markdown_report = generate_markdown_report(results) html_report = generate_html_report(results) codex_prompt = generate_codex_fix_prompt(results) workflow_context = select_dashboard_workflow_context( use_sample_data=use_sample_data, has_directions_upload=directions_upload is not None, ) github_issue = generate_github_issue(results) ci_workflow = generate_github_actions_workflow( baseline_path=workflow_context.baseline_path, current_path=workflow_context.current_path, directions_config_path=workflow_context.directions_config_path, threshold=threshold, direction=MetricDirection(fallback_direction), ) metric_col, regression_col = st.columns(2) metric_col.metric("Total compared metrics", len(results)) regression_col.metric("Regressions detected", regression_count) st.subheader("Benchmark Release Readiness") readiness_col, score_col = st.columns(2) readiness_col.metric("Readiness", release_readine [truncated — 3125 more characters] ``` ### app/layout.tsx ```typescript import type { Metadata } from "next"; import { ThemeProvider } from "@/components/theme-provider"; import "./globals.css"; export const metadata: Metadata = { title: "Codex Benchmark Guardian", description: "Catch performance regressions before they merge with deterministic benchmark analysis.", openGraph: { title: "Codex Benchmark Guardian", description: "Deterministic performance regression analysis and Codex-ready handoff.", type: "website", }, }; export default function RootLayout({ children }: Readonly<{ children: React.ReactNode }>) { return ( {children} ); } ``` ### app/page.tsx ```typescript import { ArrowDown, Binary, Bot, Fingerprint } from "lucide-react"; import { AnalysisWorkspace } from "@/components/analysis-workspace"; import { Header } from "@/components/header"; import { PrStory } from "@/components/pr-story"; export default function Home() { return
Performance confidence for every merge

Catch performance regressions before they merge.

Functional tests can pass while latency climbs, memory grows, and throughput falls. Benchmark Guardian turns those hidden shifts into a deterministic release decision and an actionable Codex handoff.

Analyze benchmarks
Deterministic analysisRepeatable, explainable results
Protected evaluatorPython remains the source of truth
Codex-ready handoffFrom signal to focused fix
; } ``` ### api/index.py ```python from __future__ import annotations import json from typing import Annotated, Literal from fastapi import FastAPI, Request from fastapi.exceptions import RequestValidationError from fastapi.responses import JSONResponse from pydantic import BaseModel, ConfigDict, Field, field_validator from codex_benchmark_guardian.benchmarks import compare_benchmark_metrics from codex_benchmark_guardian.ci import ( generate_github_actions_workflow, select_dashboard_workflow_context, ) from codex_benchmark_guardian.pr_gate import build_pr_gate_summary from codex_benchmark_guardian.regression import MetricDirection from codex_benchmark_guardian.release_readiness import generate_release_readiness_markdown from codex_benchmark_guardian.repair import ( build_repair_contract, generate_codex_repair_goal, generate_repair_contract_json, generate_repair_contract_markdown, ) from codex_benchmark_guardian.report import ( generate_codex_fix_prompt, generate_github_issue, generate_html_report, generate_markdown_report, ) from codex_benchmark_guardian.triage import generate_triage_notes MAX_METRICS = 250 DirectionValue = Literal["higher_is_worse", "lower_is_worse"] FiniteNumber = Annotated[float, Field(strict=True, allow_inf_nan=False)] class AnalyzeRequest(BaseModel): model_config = ConfigDict(extra="forbid") baseline: dict[str, FiniteNumber] current: dict[str, FiniteNumber] threshold_percent: FiniteNumber = Field(default=10.0, ge=0, le=10000) fallback_direction: DirectionValue = "higher_is_worse" directions: dict[str, DirectionValue] | None = None use_sample_data: bool = False @field_validator("baseline", "current") @classmethod def validate_metrics(cls, metrics: dict[str, float]) -> dict[str, float]: if not metrics: raise ValueError("must contain at least one metric") if len(metrics) > MAX_METRICS: raise ValueError(f"must contain at most {MAX_METRICS} metrics") if any(not name.strip() for name in metrics): raise ValueError("metric names must not be empty") return metrics @field_validator("directions") @classmethod def validate_direction_count( cls, directions: dict[str, DirectionValue] | None ) -> dict[str, DirectionValue] | None: if directions is not None and len(directions) > MAX_METRICS: raise ValueError(f"must contain at most {MAX_METRICS} metrics") return directions app = FastAPI(title="Codex Benchmark Guardian API", version="1.0.0") @app.exception_handler(RequestValidationError) async def validation_error_handler(_request: Request, exc: RequestValidationError) -> JSONResponse: errors = [] for error in exc.errors(): location = ".".join(str(part) for part in error["loc"] if part != "body") or "body" errors.append({"field": location, "message": error["msg"]}) return JSONResponse( status_code=422, content={"error": "Invalid analysis request", "details": errors}, ) @app.get("/api/health") async def health() -> dict[str, str]: return {"status": "ok"} @app.post("/api/analyze") async def analyze(payload: AnalyzeRequest) -> dict[str, object]: matching_metrics = payload.baseline.keys() & payload.current.keys() if not matching_metrics: return JSONResponse( status_code=422, content={ "error": "Invalid analysis request", "details": [{"field": "baseline,current", "message": "no matching metrics"}], }, ) if any(payload.baseline[name] == 0 for name in matching_metrics): return JSONResponse( status_code=422, content={ "error": "Invalid analysis request", "details": [ { "field": "baseline", "message": "matching baseline metric values must not be zero", } ], }, ) direction = MetricDirection(payload.fallback_direction) directions = ( {name: MetricDirection(value) for name, value in payload.directions.items()} if payload.directions else None ) results = compare_benchmark_metrics( baseline_metrics=payload.baseline, current_metrics=payload.current, threshold_percent=payload.threshold_percent, direction=direction, directions=directions, ) summary = build_pr_gate_summary(results) triage = generate_triage_notes(results) repair_contract = build_repair_contract(results) repair_contract_json = generate_repair_contract_json(repair_contract) workflow_context = select_dashboard_workflow_context( use_sample_data=payload.use_sample_data, has_directions_upload=payload.directions is not None, ) workflow = generate_github_actions_workflow( baseline_path=workflow_context.baseline_path, current_path=workflow_context.current_path, directions_config_path=workflow_context.directions_config_path, threshold=payload.threshold_percent, direction=direction, ) return { "compared_metric_count": summary.compared_metrics, "regression_count": summary.regression_count, "release_readiness_score": summary.readiness_score, "release_readiness_label": summary.readiness_label, "recommendation": summary.recommendation, "should_block": summary.should_block, "metrics": [ { "metric_name": result.metric_name, "baseline": result.baseline_value, "current": result.current_value, "percentage_change": result.change_percent, "threshold": result.threshold_percent, "direction": result.direction.value, "is_regression": result.is_regression, "severity": result.severity, [truncated — 1006 more characters] ``` ### src/codex_benchmark_guardian/cli.py ```python from __future__ import annotations from pathlib import Path from typing import Annotated import typer from rich.console import Console from codex_benchmark_guardian.benchmarks import ( compare_benchmark_metrics, load_benchmark_file, load_directions_config, ) from codex_benchmark_guardian.ci import ( DEFAULT_BASELINE_PATH, DEFAULT_CURRENT_PATH, DEFAULT_DIRECTIONS_CONFIG_PATH, DEFAULT_OUTPUT_PATH, DEFAULT_PR_GATE_OUTPUT_PATH, DEFAULT_PR_GATE_PUBLISHER_OUTPUT_PATH, DEFAULT_PYTHON_VERSION, DEFAULT_THRESHOLD, write_github_actions_workflow, write_pr_gate_publisher_workflow, write_pr_gate_workflow, ) from codex_benchmark_guardian.handoff import generate_handoff_pack from codex_benchmark_guardian.pr_gate import PRGateResult from codex_benchmark_guardian.regression import MetricDirection, detect_regression from codex_benchmark_guardian.report import ( generate_codex_fix_prompt, generate_html_report, generate_markdown_report, ) app = typer.Typer( name="cbg", help=( "Codex Benchmark Guardian: detect performance regressions and improve software reliability." ), no_args_is_help=True, ) console = Console() @app.callback() def main() -> None: """Codex Benchmark Guardian CLI.""" return None @app.command() def about() -> None: """Show project information.""" console.print("[bold]Codex Benchmark Guardian[/bold]") console.print( "A developer tool for testing, benchmarking, regression detection, " "and software reliability reporting." ) @app.command() def version() -> None: """Show project version.""" console.print("0.1.0") @app.command() def compare( metric_name: str, baseline_value: float, current_value: float, threshold: Annotated[ float, typer.Option( "--threshold", "-t", help="Regression threshold percentage.", ), ] = 10.0, direction: Annotated[ MetricDirection, typer.Option( "--direction", help="Metric direction that determines which movement is worse.", ), ] = MetricDirection.HIGHER_IS_WORSE, ) -> None: """Compare a baseline benchmark value against a current value.""" result = detect_regression( metric_name=metric_name, baseline_value=baseline_value, current_value=current_value, threshold_percent=threshold, direction=direction, ) console.print(f"[bold]Metric:[/bold] {result.metric_name}") console.print(f"[bold]Baseline:[/bold] {result.baseline_value}") console.print(f"[bold]Current:[/bold] {result.current_value}") console.print(f"[bold]Change:[/bold] {result.change_percent:.2f}%") console.print(f"[bold]Threshold:[/bold] {result.threshold_percent:.2f}%") console.print(f"[bold]Direction:[/bold] {result.direction.value}") if result.is_regression: console.print(f"[red]Regression detected[/red] | Severity: {result.severity}") else: console.print("[green]No regression detected[/green]") @app.command("init-ci") def init_ci( baseline_path: Annotated[ Path, typer.Option( "--baseline", help="Path to the baseline benchmark JSON file used by the workflow.", ), ] = DEFAULT_BASELINE_PATH, current_path: Annotated[ Path, typer.Option( "--current", help="Path to the current benchmark JSON file used by the workflow.", ), ] = DEFAULT_CURRENT_PATH, directions_config_path: Annotated[ Path, typer.Option( "--directions-config", help="Path to the per-metric directions JSON file used by the workflow.", ), ] = DEFAULT_DIRECTIONS_CONFIG_PATH, threshold: Annotated[ float, typer.Option( "--threshold", "-t", help="Regression threshold percentage used by the workflow.", ), ] = DEFAULT_THRESHOLD, python_version: Annotated[ str, typer.Option( "--python-version", help="Python version configured for actions/setup-python.", ), ] = DEFAULT_PYTHON_VERSION, output_path: Annotated[ Path, typer.Option( "--output", "-o", help="Path where the GitHub Actions workflow YAML should be written.", ), ] = DEFAULT_OUTPUT_PATH, ) -> None: """Generate a GitHub Actions benchmark regression guardrail workflow.""" write_github_actions_workflow( output_path=output_path, baseline_path=baseline_path, current_path=current_path, directions_config_path=directions_config_path, threshold=threshold, python_version=python_version, ) console.print(f"CI guardrail workflow written to: {output_path}") @app.command("init-pr-gate") def init_pr_gate( output_path: Annotated[Path, typer.Option("--output", "-o")] = DEFAULT_PR_GATE_OUTPUT_PATH, ) -> None: """Generate a GitHub pull-request benchmark gate workflow.""" write_pr_gate_workflow(output_path) publisher_path = output_path.parent / DEFAULT_PR_GATE_PUBLISHER_OUTPUT_PATH.name write_pr_gate_publisher_workflow(publisher_path) console.print(f"PR gate workflow written to: {output_path}") console.print(f"PR gate publisher workflow written to: {publisher_path}") @app.command("enforce-gate") def enforce_gate(summary_path: Annotated[Path, typer.Argument(exists=True)]) -> None: """Enforce the stored deterministic PR gate summary.""" import json try: data = json.loads(summary_path.read_text(encoding="utf-8")) summary = PRGateResult(**data) except (OSError, TypeError, ValueError, json.JSONDecodeError) as exc: raise typer.BadParameter(f"invalid gate summary JSON: {exc}") from exc required = {"Ready", "Needs Review", "Block"} if ( summary.readin [truncated — 7377 more characters] ``` [53 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]