Codex Benchmark Guardian

Evidence-backed performance repair for pull requests

Production dashboard · Repository · Real PR #20 proof

---

Codex Benchmark Guardian turns a real performance regression into a bounded Codex repair task, verifies the proposed fix against protected benchmark evidence, and safely returns the pull request to Ready.

It connects deterministic benchmark analysis to a conditional repair workflow without allowing an agent to decide its own success. Python is the only source of benchmark, readiness, and repair-policy truth; the production Next.js dashboard and FastAPI API expose that engine without reimplementing its business rules.

The final merge remains a human decision.

Why It Matters

A benchmark failure is only the start of a repair. A useful system must preserve the evidence, explain what regressed, constrain what may change, distinguish repair from verification, and prove the outcome with a trusted evaluator.

Codex Benchmark Guardian provides that chain:

• deterministic comparison across higher-is-worse and lower-is-worse metrics
• severity classification, readiness scoring, and evidence-backed triage
• an immutable Repair Contract and a conditional Codex Goal
• bounded repair instructions only when material regressions exist
• a verification-only path that avoids speculative changes when evidence is clean
• a protected GitHub PR gate that makes the final benchmark-readiness decision
• a complete, portable Handoff Pack for review and automation

Judge Quickstart

Start with the live production dashboard, then verify the repository locally:

git clone https://github.com/OmprakashSahani/codex-benchmark-guardian.git
cd codex-benchmark-guardian
pip install -e ".[dev,dashboard]"
make lint
make format-check
make test
npm install
npm run lint
npm run typecheck
npm run build

Expected Python test result:

162 passed

Generate a CLI handoff and exercise both protected-gate outcomes:

make demo-handoff
make demo-pr-gate-block
make demo-pr-gate-ready

The first command writes the CLI Handoff Pack to reports/handoff/. The gate demonstrations intentionally show a blocked regression and a Ready result. For the additional local Python-first interface:

streamlit run app.py

Live Production Experience

The primary product experience is the production Next.js/FastAPI dashboard. Its result workspace is organized into five tabs:

1. Overview
2. Metrics
3. Triage
4. Codex Goal
5. Handoff Pack

Users can:

• load permanent example scenarios
• upload or edit baseline, current, and direction JSON
• configure the threshold and per-metric directions
• run the deterministic Python analysis engine through FastAPI
• inspect readiness, metrics, severity, and triage
• copy or download the conditional Codex Goal
• inspect and download the complete Handoff Pack
• replay the real PR #20 regression and verified fix

The dashboard defaults to a 10% regression threshold. This repository's protected PR gate uses 25% to evaluate pull requests.

Until a new submission video is recorded, the production dashboard is the primary demo. The existing YouTube video is an earlier prototype demonstration and does not represent the current production experience.

Verified Repair Loop

flowchart LR
    A[Benchmark evidence] --> B[Deterministic regression analysis]
    B --> C[Readiness decision]
    C --> D[Immutable Repair Contract]
    D --> E[Conditional Codex Goal]
    E --> F{Repair required?}
    F -->|Yes| G[Bounded repair workflow]
    F -->|No| H[Verification-only workflow]
    G --> I[Protected benchmark verification]
    H --> I
    I --> J{Zero material regressions<br/>and Ready?}
    J -->|No| B
    J -->|Yes| K[Human approval]
    K --> L[Merge]

Repair-required path

When material regressions exist, the generated goal directs Codex to:

1. inspect the relevant implementation and history
2. identify evidence-backed root-cause hypotheses
3. implement the smallest maintainable correction
4. add or update regression tests
5. run only approved validation commands
6. rerun the relevant benchmark
7. review the final diff
8. obtain fresh protected evidence
9. confirm zero material regressions and Ready status
10. leave merge approval to a human

Failure or incomplete evidence repeats the bounded analysis-and-repair loop; it does not authorize policy changes.

Verification-only path

When the result has zero material regressions, the system does not invent repair work. The generated goal requires the user or agent to:

1. make no speculative code repair or speculative test change
2. inspect the supplied evidence
3. confirm that the protected workflow or trusted evaluator completed
4. confirm zero material regressions and Ready status
5. review the final diff only when changes already exist
6. preserve human approval before merge
7. continue monitoring benchmark stability

The dashboard hides the legacy speculative fix prompt from verification-only users. The API retains its codex_fix_prompt field solely for backward compatibility; codex_repair_goal.md is the canonical handoff for both paths.

Real Regression-to-Fix Proof: PR #20

Pull request #20 demonstrates the complete loop with real protected evidence.

Stage	Result
Regression metric	pr_gate_generation_latency_ms
Harmful change	+135.21%
Protected threshold	25%
Severity	critical
Initial readiness	Needs Review
Initial score	70/100
Verified fix	zero material regressions
Final readiness	Ready
Final score	100/100

• Regression workflow
• Verified-fix workflow

Workflow artifacts eventually expire, so the evidence used to replay both states is committed permanently under examples/scenarios/pr-20/. The protected verifier established benchmark readiness; Codex did not autonomously merge the pull request.

Repair Contract and Safety Model

Every analysis produces an immutable Repair Contract from the Python engine. It records the evidence, whether repair is required, allowed and forbidden actions, validation commands, and completion criteria. The Codex Goal is derived from this contract rather than from an open-ended request to improve performance.

Codex may inspect, edit, test, benchmark, and review a bounded repair. It may not:

• lower or bypass thresholds
• change metric directions to suppress failures
• manipulate original or fresh benchmark evidence
• substitute a selectively chosen passing run
• weaken the protected harness
• weaken the protected evaluator
• relax or delete tests merely to pass
• hard-code expected benchmark results
• self-declare the pull request Ready
• automatically merge the pull request

Protected verification declares benchmark readiness. A human still reviews and approves the merge.

Handoff Pack

codex_repair_goal.md is always the primary and default handoff. Dashboard downloads use review-friendly names.

Dashboard artifacts

Repair-required results expose:

1. codex_repair_goal.md
2. repair_contract.md
3. repair_contract.json
4. benchmark-report.md
5. benchmark-report.html
6. codex-fix-prompt.txt
7. github-issue.md
8. benchmark-workflow.yml
9. release-readiness.md

Verification-only results expose the same set except codex-fix-prompt.txt, because no speculative repair prompt is appropriate.

CLI artifacts

Run:

cbg handoff-pack \
  --baseline examples/baseline.json \
  --current examples/current.json \
  --directions-config examples/directions.json \
  --threshold 10 \
  --output-dir reports/handoff

The CLI preserves established filenames. The equivalents of the dashboard report, prompt, issue, workflow, and readiness downloads are report.md, report.html, codex_fix_prompt.md, github_issue.md, benchmark_guardian_ci.yml, and release_readiness.md. It also writes codex_repair_goal.md, repair_contract.md, repair_contract.json, pr_comment.md, and gate_summary.json.

The CLI currently retains codex_fix_prompt.md in a generated verification-only pack for compatibility. Consumers should use the conditional codex_repair_goal.md as the authoritative handoff.

Architecture

The production system is organized around explicit trust boundaries. The Next.js 15 App Router, TypeScript, and Tailwind CSS provide the primary product experience; FastAPI exposes the deterministic Python engine; the CLI and Streamlit reuse the same core directly. Benchmark comparison, readiness scoring, triage, and repair policy remain centralized in Python.

flowchart TB
    EVIDENCE["Benchmark evidence<br/>baseline · current · directions · threshold"]

    subgraph INTERFACES["Product interfaces"]
        direction LR
        WEB["Next.js 15 dashboard<br/>TypeScript · Tailwind CSS"]
        API["FastAPI<br/>POST /api/analyze"]
        CLI["Python CLI"]
        ST["Streamlit<br/>local Python interface"]
        WEB --> API
    end

    subgraph CORE["Trusted Python core — only policy source of truth"]
        direction LR
        CMP["Benchmark<br/>comparison"] --> REG["Regression<br/>classification"]
        REG --> READY["Release-readiness<br/>scoring"]
        REG --> TRIAGE["Deterministic<br/>triage"]
        READY --> CONTRACT["Immutable<br/>Repair Contract"]
        TRIAGE --> CONTRACT
        CONTRACT --> GOAL["Conditional<br/>Codex Goal"]
        CONTRACT --> PACK["Reports and<br/>Handoff Pack"]
    end

    subgraph EXECUTION["Bounded execution"]
        direction LR
        ROUTE{"Repair<br/>required?"}
        REPAIR["Minimal evidence-backed repair<br/>plus regression tests"]
        VERIFY["Verification only<br/>no speculative changes"]
        CHECKS["Approved<br/>project checks"]
        ROUTE -->|Yes| REPAIR
        ROUTE -->|No| VERIFY
        REPAIR --> CHECKS
        VERIFY --> CHECKS
    end

    subgraph PROTECTED["Protected verification boundary"]
        direction LR
        HARNESS["Protected-base harness<br/>controls what is measured"]
        EVALUATOR["Protected-base evaluator<br/>controls readiness"]
        DECISION{"Zero material regressions<br/>and Ready?"}
        HARNESS --> EVALUATOR --> DECISION
    end

    HUMAN["Human review<br/>and approval"]
    MERGE["Merge"]

    EVIDENCE --> WEB
    EVIDENCE --> CLI
    EVIDENCE --> ST
    API --> CMP
    CLI --> CMP
    ST --> CMP
    GOAL --> ROUTE
    PACK -.-> ROUTE
    CHECKS --> HARNESS
    DECISION -->|No| CONTRACT
    DECISION -->|Yes| HUMAN --> MERGE

    classDef evidence fill:#f8fafc,stroke:#64748b,color:#0f172a,stroke-width:1.5px;
    classDef interface fill:#eff6ff,stroke:#2563eb,color:#172554,stroke-width:1.5px;
    classDef api fill:#ecfeff,stroke:#0891b2,color:#164e63,stroke-width:1.5px;
    classDef core fill:#f0fdf4,stroke:#16a34a,color:#14532d,stroke-width:1.5px;
    classDef contract fill:#fff7ed,stroke:#ea580c,color:#7c2d12,stroke-width:2px;
    classDef execution fill:#faf5ff,stroke:#9333ea,color:#581c87,stroke-width:1.5px;
    classDef protected fill:#fef2f2,stroke:#dc2626,color:#7f1d1d,stroke-width:2px;
    classDef human fill:#fdf4ff,stroke:#c026d3,color:#701a75,stroke-width:2px;

    class EVIDENCE evidence;
    class WEB,CLI,ST interface;
    class API api;
    class CMP,REG,READY,TRIAGE core;
    class CONTRACT,GOAL,PACK contract;
    class ROUTE,REPAIR,VERIFY,CHECKS execution;
    class HARNESS,EVALUATOR,DECISION protected;
    class HUMAN,MERGE human;

    style INTERFACES fill:#f8fbff,stroke:#60a5fa,stroke-width:2px
    style CORE fill:#f7fff9,stroke:#4ade80,stroke-width:2px
    style EXECUTION fill:#fdfaff,stroke:#c084fc,stroke-width:2px
    style PROTECTED fill:#fff8f8,stroke:#f87171,stroke-width:2px

The diagram makes the authority boundaries explicit:

• Frontend and API adapters present results; they do not recalculate readiness or repair policy.
• The Python core generates the immutable Repair Contract and conditional Codex Goal from benchmark evidence.
• The protected-base harness controls measurement, and the protected-base evaluator controls benchmark readiness.
• Only a human approves the final merge.

Benchmark rules, readiness scoring, and repair policy are not duplicated in TypeScript. The frontend renders API results from the Python source of truth.

Permanent Example Scenarios

The production dashboard includes five scenarios:

• Standard regression — the normal regression walkthrough
• Material regression — the same evidence evaluated at the higher, more permissive 25% threshold rather than the standard scenario's 10% threshold; 10% flags smaller harmful changes, while 25% tolerates more variation and therefore flags fewer regressions, and the repository's protected PR gate uses 25% to account conservatively for shared-runner timing noise
• Clean benchmark — the verification-only path
• PR #20 regression — the real protected regression evidence
• PR #20 verified fix — the real Ready evidence after repair

Scenario inputs live in examples/scenarios/ as baseline, current, and metric-direction JSON. They can be loaded, edited, and rerun through the current Python engine.

CLI and Local Interfaces

Compare benchmark files

cbg compare-files examples/baseline.json examples/current.json \
  --threshold 10 \
  --directions-config examples/directions.json \
  --report reports/report.md \
  --html-report reports/report.html \
  --fail-on-regression

--fail-on-regression still generates reports and exits nonzero when a material regression is detected. Metric directions may be higher_is_worse (for latency, runtime, memory, or error rate) or lower_is_worse (for throughput, accuracy, recall, precision, or success rate). --direction sets a fallback and --directions-config supplies per-metric overrides.

Useful discovery commands:

cbg version
cbg about
cbg --help

Streamlit

Streamlit is an additional local, Python-first interface—not the primary production dashboard:

streamlit run app.py
# or
make dashboard

It uses the same Python comparison and reporting engine as the CLI and API.

GitHub PR Benchmark Gate

The protected gate benchmarks the exact protected base SHA and PR head in separate environments on the same runner. Its fixed workload uses warm-ups and repeated high-resolution measurements, then compares median results. The 25% repository threshold is intentionally separate from the dashboard's 10% exploratory default.

The protected-base harness controls what is measured. The protected-base evaluator controls comparison, readiness, PR comments, and final enforcement. Provenance records the harness and evaluator sources, while fork pull requests benchmark and upload evidence without receiving write access for comments.

Local demonstrations:

cbg init-pr-gate
make demo-pr-gate-block
make demo-pr-gate-ready
make demo-init-pr-gate

Repositories adopting the generated gate should make benchmark-pr-gate a required status check. They may replace the project benchmark workload, but a proposed repair must not weaken the protected harness or evaluator.

Built with Codex and GPT-5.6

I used Codex with GPT-5.6 throughout the project for implementation planning, focused code changes, testing, debugging, and review.

Codex contributions

Codex helped implement and refine:

• JSON benchmark comparison and multi-metric regression detection
• mixed metric-direction handling and severity classification
• deterministic readiness scoring and regression triage guidance
• Markdown and self-contained HTML benchmark reports
• CLI commands and Codex Handoff Pack generation
• GitHub issue, workflow, and pull-request artifact generation
• Streamlit dashboard workflows
• protected GitHub PR Benchmark Gate behavior
• FastAPI analysis and Repair Contract integration
• Next.js and TypeScript dashboard integration
• permanent benchmark scenarios and real PR replay workflows
• conditional repair-required and verification-only Codex Goals
• unit, CLI, API, repair-contract, scenario, and PR-gate tests

Codex reviews also helped identify and resolve edge cases involving:

• passing and intentionally failing CI demonstrations
• generated workflow paths and safe command quoting
• mixed metric directions and relative, ./, and absolute benchmark input paths
• dashboard-generated CI YAML matching the selected evidence and policy inputs
• deterministic structured and serialized Repair Contract outputs
• inert rendering of untrusted metric names
• protected benchmark evidence, harness, and evaluator boundaries
• verification-only results exposing a speculative legacy fix prompt
• stale artifact selection while switching from a repair result to a verification-only result

Human design and approval

The human developer defined and approved:

• the product problem and overall system direction
• the benchmark readiness policy and scoring behavior
• the protected harness, evaluator, and evidence model
• the safety boundaries for Codex-assisted repair
• repair-required versus verification-only behavior
• the product scope and user experience
• review acceptance and production deployment
• final pull-request merge decisions

Every Codex-assisted change was reviewed and validated using Ruff, Pytest, TypeScript checks, production builds, protected benchmark workflows, focused local Codex reviews, GitHub Codex reviews, and manual production verification.

Codex assisted with implementation and review; it did not independently define the policy, approve its own work, declare a pull request Ready, deploy the application, or merge changes.

Installation and Verification

Requirements include Python 3.12 or newer, Node.js with npm, pip, and Git.

git clone https://github.com/OmprakashSahani/codex-benchmark-guardian.git
cd codex-benchmark-guardian
pip install -e ".[dev,dashboard]"
npm install

Run the complete repository checks:

make lint
make format-check
make test
npm run lint
npm run typecheck
npm run build

The current Python suite contains 162 tests.

Project Structure

.
├── api/
│   └── index.py                       # FastAPI adapter for the Python engine
├── app/                               # Next.js 15 App Router application
├── components/                        # Production dashboard components
├── lib/                               # Frontend types, fixtures, and adapters
├── src/codex_benchmark_guardian/      # Deterministic Python source of truth
│   ├── regression.py                  # Comparison and classification
│   ├── release_readiness.py           # Readiness scoring
│   ├── repair.py                      # Repair Contract and conditional goal
│   ├── report.py                      # Markdown and HTML generation
│   ├── handoff.py                     # Canonical CLI Handoff Pack
│   └── pr_gate.py                     # Protected-gate result generation
├── benchmarks/                        # Protected benchmark harness
├── examples/scenarios/                # Permanent dashboard and PR #20 evidence
├── tests/
│   ├── test_api.py                    # API contract coverage
│   └── test_repair.py                 # Repair-policy coverage
├── app.py                             # Additional Streamlit interface
├── AGENTS.md                          # Repository repair boundaries
└── README.md

License

MIT. See LICENSE.

---

Omprakash Sahani — ML Systems Engineer · Software Engineer · Distributed Systems