# Project export: Relevance Incident Lab

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: OpenAI Build Week
- Tagline: Investigate how search results change across retrieval, deterministic ranking, and LLM reranking before deploying to prod
- Devpost: https://devpost.com/software/relevance-incident-lab
- GitHub: https://github.com/tpremrud/relevance-incident-lab
- Demo: https://relevance-incident-lab.nobthiti.chatgpt.site/
- Video: https://www.youtube.com/embed/pGur076pIfE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Thiti Premrudeepreechacharn (72 commits)

## Devpost submission (written by the team)

### Overview

TL;DR We see that LLM can rerank search results, and we should have a way of inspecting the whole chain from retrieval to results. With DuckDB, Python with FastAPI, and GPT-5.6 (Luna), we performed search on synthetic pet food product data with ground truth labeling and item of interest highlighted as protected and inspect each layer in the Relevance Incident Lab (RIL) We found that LLM reranking can be beneficial in some cases, and not others (i.e., hurting the ranking of protected items, or unchanged so that the LLM calls are additional costs we do not need) Codex and GPT-5.6 were used extensively in planning and implementation of the code and synthesizing data, as well as the video demo via plugins and debugging the UI with skills Future direction includes extending RIL's capabilities to be able to run as a containerized companion to a developer’s search stack, support hybrid, semantic, and AI-search pipelines, expose its evidence through an MCP server, and evaluate real public datasets beyond the current synthetic examples

### Inspiration

Research such as RankGPT and RankZephyr showed that large language models can rerank search results. That made me wonder: when a ranking looks wrong, is it the LLM's fault or was the issue in retrieval or the deterministic ranking baseline, or some other parts such as output validation? With that, we built Relevance Incident Lab (RIL) to make that whole chain inspectable end-to-end, to see between the layers what each is doing, an essential tool when we might be accusing LLM being the blackbox.

### What it does

With DuckDB full-text retrieval, a deterministic baseline, GPT reranking, response validation, and other features like p95 latency, RIL lets a developer be able to trace the search result through each layer from end-to-end. Each test case includes frozen relevance judgments (i.e., our version of ground truth) so RIL can show exactly which items moved, why a protected relevant item matters, and whether the change helped or hurt. The caveat regarding the suggestion and summary or final outcome is that they are pieces of evidence for a human reviewer, not an automatic release decision.

### How we built it

We used DuckDB Full-Text Search and BM25 for lexical retrieval, Python with FastAPI and Pydantic for the evidence and validation layer, GPT-5.6 Luna through the OpenAI API for reranking, and React with Vite for the review interface. In addition to the rerunnable nature of the data, the package runs in Docker without an API key or network dependency, while the public site is a read-only demonstration of the same evidence. Codex was part of the development workflow too. Sol at ultra reasoning helped refine the PRD and technical design, Goal mode helped with longer implementation and verification work, installed UI skills (i.e., Impeccable) helped polish the interface and accessibility, and HyperFrames helped produce the demo video. Moreover, we used Sites to deploy a static demo with ease.

### Challenges we ran into

The hardest part was producing a truthful BLOCK case. Our first organic case did not regress. A later chicken-free case failed before reranking because its protected item landed at retrieval rank 14, outside the top-12 candidate pool. That was useful evidence: a reranker cannot rescue an item it never receives. The puppy case exposed a different problem when early model attempts returned invalid or hallucinated product IDs. We redesigned the protocol around opaque candidate handles, exact validation, one bounded repair attempt, explicit abstention, and deterministic fallback. Only after freezing a separately motivated beef-exclusion fixture (yet another case) before observing the provider result did we obtain a defensible BLOCK demonstration. This took longer than simply generating a convenient output, but it kept the evidence honest.

### Accomplishments we're proud of

We produced both sides of the story: A GPT_PREFERRED case where the protected puppy result moved from baseline rank 11 to GPT rank 1. A BLOCK case where the protected beef-exclusion result began at baseline rank 1 and the GPT reranker demoted it, triggering investigation. A 100-pair response-time campaign for the qualified puppy case, giving us a real p95 measurement instead of relying on one fast call. A provider-free recorded demo that preserves attempts, validation, replay, metrics, policy reasoning, and exact artifacts for review. We are also proud that RIL keeps INCONCLUSIVE as a real outcome. Missing or invalid evidence does not silently become a success.

### What we learned

LLM reranking can help, but it is only one part of the search pipeline. Retrieval recall comes first: if the relevant item is outside the candidate pool, reranking cannot recover it. An aggregate metric can improve while a business-critical protected item gets worse, so both views need to be evaluated. Model output should be treated as untrusted input: IDs must be validated, and the system needs abstention and deterministic fallback behavior. One successful response is not enough to establish latency or reliability; repeated, comparable runs matter. Synthetic fixtures are useful for controlled regression tests, but they are not proof of production performance on real customer data. BLOCK should remain an advisory, human-gated decision. RIL explains the evidence; a person owns the release action.

### What's next

Turn RIL into a containerized companion for a developer's existing search stack, with a documented fixture/import contract for bringing their own queries, candidates, judgments, and rankings. Extend RIL beyond lexical retrieval to inspect hybrid, semantic, and broader AI-search pipelines. Create an MCP server so a developer can connect a self-hosted RIL instance to Codex and ask questions about recorded incidents and evaluation results. Evaluate broader public datasets such as the Amazon Shopping Queries ESCI dataset, then extend beyond product search to news and other changing corpora. Add connectors that snapshot evidence from production search systems and run RIL as a human-reviewed search-quality gate in CI/CD, without giving it automatic release authority.

## README (from the GitHub repository)

# Relevance Incident Lab

Relevance Incident Lab (RIL) is a developer tool for debugging search-ranking
changes before release. It compares a deterministic lexical baseline with an
LLM reranker, validates the response, replays the ranking, computes quality and
latency evidence, and presents an advisory outcome for human review.

**Try the read-only recorded demo:**
[relevance-incident-lab.nobthiti.chatgpt.site](https://relevance-incident-lab.nobthiti.chatgpt.site)

The demo uses frozen synthetic data. It needs no API key, makes no provider
calls, and cannot approve a release.

## Run locally

The fastest source-based path runs the same recorded console used by the public
demo.

Requirements:

- Node.js 22.13 or newer
- npm 10 or newer
- macOS, Linux, or Windows

```sh
git clone https://github.com/tpremrud/relevance-incident-lab.git
cd relevance-incident-lab/site
npm ci
npm run dev -- --host 127.0.0.1 --port 3000
```

Open [http://127.0.0.1:3000](http://127.0.0.1:3000). Run the offline site test
with:

```sh
npm test
```

No Docker image, database, OpenAI account, or environment variable is required
for this judge path. The bundled sample data is synthetic and read-only.

## What to inspect

- **Beef exclusion · BLOCK** — an engineered regression control in which the
  protected relevant result starts at baseline rank 1 and is demoted by the
  recorded reranker. RIL recommends investigation; it does not block a release
  automatically.
- **Puppy v2 · GPT preferred** — a positive control in which the protected
  result moves from baseline rank 11 to reranked rank 1 and passes the recorded
  100-pair response-time qualification.
- **Organic** and **Chicken-free** — honest nonqualifying and inconclusive
  states that demonstrate fail-closed evidence handling.

Each case exposes retrieval, baseline and reranked positions, validation,
replay, metrics, policy reasoning, record details, and exact source artifacts.
The final decision form is session-only and keeps the release decision with a
human reviewer.

## Architecture

```text
Synthetic fixture + judgments
            │
            ▼
      DuckDB FTS retrieval
            │
            ├──────────────► deterministic baseline
            │
            └──────────────► validated GPT-5.6 ranking
                                  │
                                  ▼
                   replay + metrics + policy proof
                                  │
                                  ▼
                       advisory human review UI
```

The recorded demo replays immutable, digest-verified evidence. Missing or
ambiguous facts remain visibly unavailable or `INCONCLUSIVE`; the UI does not
infer them.

## Canonical Docker package

The container is the canonical reproducible developer-tool package. The public
site is a convenient read-only mirror, not the complete FastAPI/DuckDB runtime.
If you have received or loaded the prebuilt `linux/amd64` image named
`relevance-incident-lab:recorded`, use:

```sh
./scripts/incident-lab doctor
./scripts/incident-lab run recorded --port 8000
# open http://127.0.0.1:8000
./scripts/incident-lab status
./scripts/incident-lab stop
```

Recorded mode is non-root, verifies evidence by digest, uses a read-only root
filesystem and private data directory, and denies application egress. The
prebuilt image archive is intentionally not committed to Git; judges can test
the project without it through the public URL or `site/` setup above. See the
[runbook](RUNBOOK.md) for detailed operation and troubleshooting.

## Verification

```sh
cd site && npm test
cd ../web && npm test && npm run typecheck && npm run build
cd .. && ./scripts/incident-lab verify fast
```

The security matrix covers query and candidate injection, unknown output IDs,
route/source bounds, bundle tampering, unsafe SQL, secret exposure, error
channels, and policy bypass. Recorded mode makes zero provider calls.

## How Codex and GPT-5.6 were used

- Codex with Sol at ultra reasoning helped refine the PRD and technical design,
  then implemented, tested, reviewed, and integrated the project under explicit
  owner gates. Goal mode was used for longer environment and verification work.
- Installed UI-quality skills helped polish accessibility and responsive
  behavior. The HyperFrames plugin produced the scripted demonstration video.
- GPT-5.6 Luna produced separately authorized ranking observations and the
  bounded 100-pair puppy response-time campaign.
- The submitted demo is recorded playback: it does not contact Luna or read an
  API key.

## Repository map

- `site/` — installable, provider-free judge demo
- `src/relevance_incident_lab/` — Python contracts, verification, and runtime
- `web/` — React evidence console
- `fixtures/` — synthetic source and compiled fixtures
- `evidence/recorded/` — immutable recorded evidence
- `security/` and `tests/` — security matrix and verification controls
- `packaging/` — container inputs and recorded-package metadata
- `RUNBOOK.md` — operation, local development, troubleshooting, and demo flow

Historical milestone IDs remain inside immutable evidence paths and generated
records because changing them would invalidate their hashes. They are not
required to understand or operate the public project.

## License

Original project code, documentation, and synthetic fixtures are available
under the [MIT License](LICENSE). Recorded model outputs, bundled evidence, and
third-party material retain the separate treatment described in
[NOTICE.md](NOTICE.md) and [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).


## Detected evidence (automated analysis)

Indexed codebase: 307 recognized source files, 2373 KB.
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- HTML (language) — detected in the code
- Next.js (technology) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Docker (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (120 of 937)

```
.dockerignore
.editorconfig
.gitattributes
.gitignore
.node-version
.python-version
evidence/presentation/m4-presentation-view-v1/rank-binding.json
evidence/presentation/m4-presentation-view-v1/source-binding.json
evidence/presentation/rel60-candidate-a-exploration-v2/chicken-free-adult-dog-food-regression--chicken-free-adult-dog-food-regression-v1-preflight-1.json
evidence/presentation/rel60-candidate-a-exploration-v2/grain-free-puppy-food-improvement--rel26-grain-free-puppy-food-improvement-v2-live-1.json
evidence/presentation/rel60-candidate-a-exploration-v2/organic-dog-food-regression--rel26-organic-dog-food-regression-v2-live-1.json
evidence/recorded/.gitkeep
evidence/recorded/m3-truthful-evidence-portfolio-v1/manifest.json
evidence/recorded/m3-truthful-evidence-portfolio-v1/manifest.sha256
evidence/recorded/m3-truthful-evidence-portfolio-v1/portfolio/entries/chicken-free-adult-dog-food-regression-portfolio-entry-v1.json
evidence/recorded/m3-truthful-evidence-portfolio-v1/portfolio/entries/grain-free-puppy-food-improvement-portfolio-entry-v1.json
evidence/recorded/m3-truthful-evidence-portfolio-v1/portfolio/entries/organic-dog-food-regression-portfolio-entry-v1.json
evidence/recorded/m3-truthful-evidence-portfolio-v1/portfolio/m3-truthful-evidence-portfolio-assessment-v1.json
evidence/recorded/puppy-protocol-v2-qualification-additional-1/artifacts/generation/puppy-protocol-v2-qualification-additional-1-observation-generation-1-v1.json
evidence/recorded/puppy-protocol-v2-qualification-additional-1/artifacts/gpt-candidate-handle-map/puppy-protocol-v2-qualification-additional-1-handle-map-v1.json
evidence/recorded/puppy-protocol-v2-qualification-additional-1/artifacts/gpt-semantic-request/puppy-protocol-v2-qualification-additional-1-v1.json
evidence/recorded/puppy-protocol-v2-qualification-additional-1/artifacts/gpt-stage-result/puppy-protocol-v2-qualification-additional-1-stage-v1.json
evidence/recorded/puppy-protocol-v2-qualification-additional-1/artifacts/trace/puppy-protocol-v2-qualification-additional-1-trace-v1.json
evidence/recorded/puppy-protocol-v2-qualification-additional-1/snapshot.json
evidence/recorded/puppy-protocol-v2-qualification-additional-1/snapshot.sha256
evidence/recorded/puppy-protocol-v2-qualification/artifacts/generation/puppy-protocol-v2-qualification-observation-generation-1-v1.json
evidence/recorded/puppy-protocol-v2-qualification/artifacts/generation/puppy-protocol-v2-qualification-observation-generation-2-v1.json
evidence/recorded/puppy-protocol-v2-qualification/artifacts/gpt-candidate-handle-map/puppy-protocol-v2-qualification-handle-map-v1.json
evidence/recorded/puppy-protocol-v2-qualification/artifacts/gpt-semantic-request/puppy-protocol-v2-qualification-request-2-v1.json
evidence/recorded/puppy-protocol-v2-qualification/artifacts/gpt-semantic-request/puppy-protocol-v2-qualification-v1.json
evidence/recorded/puppy-protocol-v2-qualification/artifacts/gpt-stage-result/puppy-protocol-v2-qualification-stage-v1.json
evidence/recorded/puppy-protocol-v2-qualification/artifacts/trace/puppy-protocol-v2-qualification-trace-v1.json
evidence/recorded/puppy-protocol-v2-qualification/snapshot.json
evidence/recorded/puppy-protocol-v2-qualification/snapshot.sha256
evidence/recorded/rel58-block-demonstration-v1/capture/artifacts/generation/beef-exclusion-evidence-depth-block-experiment-observation-generation-1-v1.json
evidence/recorded/rel58-block-demonstration-v1/capture/artifacts/gpt-candidate-handle-map/beef-exclusion-evidence-depth-block-experiment-handle-map-v1.json
evidence/recorded/rel58-block-demonstration-v1/capture/artifacts/gpt-semantic-request/beef-exclusion-evidence-depth-block-experiment-v1.json
evidence/recorded/rel58-block-demonstration-v1/capture/artifacts/gpt-stage-result/beef-exclusion-evidence-depth-block-experiment-stage-v1.json
evidence/recorded/rel58-block-demonstration-v1/capture/artifacts/gpt-validation-result/beef-exclusion-evidence-depth-block-experiment-validation-1-v1.json
evidence/recorded/rel58-block-demonstration-v1/capture/artifacts/safe-provider-response/beef-exclusion-evidence-depth-block-experiment-response-1-v1.json
evidence/recorded/rel58-block-demonstration-v1/capture/artifacts/trace/beef-exclusion-evidence-depth-block-experiment-trace-v1.json
evidence/recorded/rel58-block-demonstration-v1/capture/snapshot.json
evidence/recorded/rel58-block-demonstration-v1/capture/snapshot.sha256
evidence/recorded/rel58-block-demonstration-v1/experiment-report.md
evidence/recorded/rel58-block-demonstration-v1/manifest.json
evidence/recorded/rel58-block-demonstration-v1/manifest.sha256
evidence/recorded/rel58-block-demonstration-v1/verdict.json
evidence/recorded/rel59-puppy-qualification-v1/latency-campaign/campaign-capture-summary.json
evidence/recorded/rel59-puppy-qualification-v1/latency-campaign/campaign-declaration.json
evidence/recorded/rel59-puppy-qualification-v1/latency-campaign/final-analysis/campaign-cost.json
evidence/recorded/rel59-puppy-qualification-v1/latency-campaign/final-analysis/latency.json
evidence/recorded/rel59-puppy-qualification-v1/latency-campaign/final-analysis/policy.json
evidence/recorded/rel59-puppy-qualification-v1/latency-campaign/final-analysis/qualification.json
evidence/recorded/rel59-puppy-qualification-v1/latency-campaign/final-analysis/summary.json
evidence/recorded/rel59-puppy-qualification-v1/manifest.json
evidence/recorded/rel59-puppy-qualification-v1/offline-eligibility/cost.json
evidence/recorded/rel59-puppy-qualification-v1/offline-eligibility/diagnosis.json
evidence/recorded/rel59-puppy-qualification-v1/offline-eligibility/latency-campaign-eligibility.json
evidence/recorded/rel59-puppy-qualification-v1/offline-eligibility/latency-pre-campaign.json
evidence/recorded/rel59-puppy-qualification-v1/offline-eligibility/policy-pre-campaign.json
evidence/recorded/rel59-puppy-qualification-v1/offline-eligibility/qualification-pre-campaign.json
evidence/recorded/rel59-puppy-qualification-v1/offline-eligibility/quality.json
evidence/recorded/rel59-puppy-qualification-v1/offline-eligibility/replay.json
evidence/recorded/rel59-puppy-qualification-v1/offline-eligibility/summary.json
evidence/recorded/rel59-puppy-qualification-v1/qualification-capture/artifacts/generation/puppy-protocol-v2-qualification-additional-2-observation-generation-1-v1.json
evidence/recorded/rel59-puppy-qualification-v1/qualification-capture/artifacts/gpt-candidate-handle-map/puppy-protocol-v2-qualification-additional-2-handle-map-v1.json
evidence/recorded/rel59-puppy-qualification-v1/qualification-capture/artifacts/gpt-semantic-request/puppy-protocol-v2-qualification-additional-2-v1.json
evidence/recorded/rel59-puppy-qualification-v1/qualification-capture/artifacts/gpt-stage-result/puppy-protocol-v2-qualification-additional-2-stage-v1.json
evidence/recorded/rel59-puppy-qualification-v1/qualification-capture/artifacts/gpt-validation-result/puppy-protocol-v2-qualification-additional-2-validation-1-v1.json
evidence/recorded/rel59-puppy-qualification-v1/qualification-capture/artifacts/safe-provider-response/puppy-protocol-v2-qualification-additional-2-response-1-v1.json
evidence/recorded/rel59-puppy-qualification-v1/qualification-capture/artifacts/trace/puppy-protocol-v2-qualification-additional-2-trace-v1.json
evidence/recorded/rel59-puppy-qualification-v1/qualification-capture/snapshot.json
evidence/recorded/rel59-puppy-qualification-v1/qualification-capture/snapshot.sha256
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/case-reviews/beef-exclusion-lexical-trap-regression-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/cases/beef-exclusion-lexical-trap-regression-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/constraint-sets/us-available-catalog-under-90-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/evaluation-policies/relevance-incident-lab-evaluation-policy-v1-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/fixture/relevance-incident-lab-beef-exclusion-lexical-trap-suite-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/judgment-sets/beef-exclusion-lexical-trap-judgments-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/pricing-snapshots/openai-gpt-5p6-luna-default-usd-2026-07-17-v1-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b001-quietmeadow-lamb-adult-recipe-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b002-pureclaim-beef-free-adult-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b003-labelwise-beef-free-adult-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b004-simpletrail-limited-adult-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b005-gentlebowl-lamb-adult-diet-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b006-clearcoat-beef-free-adult-supplement-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b007-brightbite-beef-free-adult-treats-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b008-pantrymark-beef-free-food-bin-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b009-dailymix-beef-free-adult-mixer-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b010-bowlboost-beef-free-topper-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b011-quietwhisker-beef-free-adult-cat-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b012-firststep-beef-free-puppy-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b013-silvermuzzle-beef-free-senior-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b014-pureclaim-beef-free-canned-adult-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b015-fieldpup-lamb-simple-adult-meal-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b016-riverstone-turkey-simple-adult-recipe-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b017-openfield-duck-adult-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b018-coastsalmon-limited-adult-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b019-northfork-no-cattle-adult-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b020-labelguide-beef-free-shopping-card-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b021-background-unscented-cat-litter-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b022-background-tropical-fish-flakes-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b023-background-garden-bird-seed-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b024-background-reflective-dog-leash-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b025-background-orthopedic-dog-bed-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b026-background-ceramic-food-bowl-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b027-background-puppy-training-pads-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b028-background-beef-jerky-dog-treats-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b029-background-chicken-adult-dog-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b030-background-grain-free-beef-dog-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b031-background-rabbit-hay-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b032-background-hamster-bedding-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b033-background-flea-comb-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b034-background-salmon-cat-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b035-background-dog-shampoo-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/products/b036-background-food-scoop-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/artifacts/queries/beef-free-limited-ingredient-adult-dog-food-v1.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/evidence/role-leakage-proof.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/evidence/source-snapshot.json
fixtures/compiled/relevance-incident-lab-beef-exclusion-lexical-trap-suite/1/evidence/validation-result.json
[817 more files omitted for size]
```

### Dependencies

- pyproject.toml: duckdb@==1.5.4, fastapi@==0.139.2, openai@==2.46.0, pydantic@==2.13.4, rfc8785@==0.1.4, uvicorn@==0.51.0
- site/package.json: @cloudflare/vite-plugin@1.45.1, @tailwindcss/postcss@4.2.1, @types/node@22.19.19, @types/react@19.2.14, @types/react-dom@19.2.3, @vitejs/plugin-react@6.0.2, @vitejs/plugin-rsc@0.5.26, eslint@9.39.4, eslint-config-next@16.2.10, next@16.2.10, react@19.2.7, react-dom@19.2.7, react-router-dom@7.18.1, react-server-dom-webpack@19.2.6, tailwindcss@4.2.1, typescript@5.9.3, vinext@0.0.50, vite@8.1.5, wrangler@4.112.0
- web/package.json: @axe-core/playwright@4.12.1, @eslint/js@10.0.1, @playwright/test@1.61.1, @testing-library/react@16.3.2, @testing-library/user-event@14.6.1, @types/node@24.13.3, @types/react@19.2.17, @types/react-dom@19.2.3, @vitejs/plugin-react@6.0.3, eslint@10.7.0, jsdom@29.1.1, react@19.2.7, react-dom@19.2.7, react-router-dom@7.18.1, typescript@6.0.3, typescript-eslint@8.64.0, vite@8.1.5, vitest@4.1.10

### Recent commits (newest first)

- docs: clarify Codex model usage
- docs: prepare judge-facing submission
- docs: bind M5 remediation candidate
- docs: clarify network-denied smoke path
- fix: preserve offline app with loopback proxy
- fix: harden M5 recorded package
- docs: publish recorded demo URL
- feat: ship judge-ready evidence console
- feat(web): complete REL-60 navigation and artifact explorer
- feat: adopt puppy GPT-preferred qualification
- feat: adopt REL-58 BLOCK demonstration
- fix: type REL-58 control handles exactly
- Merge branch 'implement/rel58-chg005' into integrate/rel58-rel59-rel60-main
- Merge branch 'preserve/puppy-protocol-v2-captures-2026-07-20' into integrate/rel58-rel59-rel60-main
- Merge branch 'preserve/m5-closeout-audit-2026-07-20' into integrate/rel58-rel59-rel60-main
- feat: complete REL-58 offline BLOCK proof
- chore: preserve puppy protocol-v2 capture snapshots
- chore: preserve unreviewed REL-58 slice A state
- feat: implement REL-60 Candidate A prototype
- Add bounded M5 closeout security audit evidence

## Key source files (fetched from GitHub, selected and truncated for size)

### NOTICE.md

```markdown
# Relevance Incident Lab notices and data/evidence terms

Copyright (c) 2026 Thiti Premrudeepreechacharn.

The repository's `LICENSE` applies to original project source code,
project-authored documentation, and the synthetic fixture source and compiled
derivatives, except where a file states otherwise.

The MIT license does **not** relicense third-party material. In particular:

- recorded model outputs are included only as immutable evaluation evidence;
  their use remains subject to applicable law and the provider terms under
  which they were generated;
- bundled evidence may combine MIT-licensed project-authored manifests,
  analysis, and schemas with recorded model outputs that are not granted a
  separate license by this project;
- third-party dependencies, base-image components, and their notices remain
  under their respective licenses, summarized in `THIRD_PARTY_NOTICES.md` and
  retained in installed package metadata; and
- owner-controlled planning material, credentials, customer data, and any
  material explicitly marked private or excluded are not covered by the MIT
  grant and are not part of the public package.

All included fixtures are synthetic. The recorded evidence is provided to
inspect the tool's behavior, not as a production-quality, incident-rate,
provider-endorsement, or automated-release claim.

The public Sites mirror is a sanitized, read-only convenience view. It is not
the complete Docker runtime, does not persist proposal decisions, and does not
replace the digest-bound, network-denied recorded container.

```

### SECURITY.md

```markdown
# Security Policy

## Scope

Relevance Incident Lab is a synthetic, recorded evidence package and developer
tool. It is not a production ranking service and cannot authorize a release.

## Secret boundary

Recorded startup, offline verification, the public site, and the judge demo do
not read, probe, or require an OpenAI API key, proxy credential, or provider
state. The source repository excludes `.env` files, customer data, runtime
secrets, provider headers, and source maps. Live provider work is outside the
recorded package.

## Runtime boundary

The canonical container runs non-root with a read-only root filesystem, all
Linux capabilities dropped, `no-new-privileges`, and a private `0700` data
directory. The app is placed on an internal-only network. A separate
credential-free, fixed-target proxy may expose the UI on loopback while the app
retains no external egress.

State-changing proposal routes require an exact Origin/Host match and a
reviewed non-simple CSRF header. Static evidence assets are content-addressed
and allowlisted. Unknown routes, IDs, manifest mutations, SQL expressions, and
unsafe error paths fail closed.

## Evidence boundary

Recorded portfolio bytes are verified against their manifests before playback.
Fixtures, captures, fallbacks, reports, and generated records are never
rewritten by startup or verification. Historical IDs remain inside immutable
paths because renaming them would invalidate provenance.

## Security verification

`security/red-team-matrix-v1.json` covers query and candidate injection,
unknown output IDs, route/source bounds, manifest tampering, unsafe SQL, secret
exposure, error-channel disclosure, and policy/release bypass. Recorded checks
make zero provider calls.

## Reporting

Do not publish suspected credentials or sensitive evidence in a public issue.
Use GitHub's private vulnerability-reporting channel when available, or contact
the repository owner privately.

```

### pyproject.toml

```
[project]
name = "relevance-incident-lab"
version = "0.0.0"
description = "Local synthetic search-ranking incident investigation tool"
requires-python = "==3.14.*"
dependencies = [
  "duckdb==1.5.4",
  "fastapi==0.139.2",
  "uvicorn==0.51.0",
  "pydantic==2.13.4",
  "openai==2.46.0",
  "rfc8785==0.1.4",
]

[dependency-groups]
dev = [
  "pytest==9.1.1",
  "pytest-asyncio==1.4.0",
  "hypothesis==6.156.6",
  "httpx==0.28.1",
  "coverage==7.15.2",
  "ruff==0.15.22",
  "mypy==2.3.0",
]

[tool.uv]
package = false

```

### web/package.json

```
{
  "name": "relevance-incident-lab-web",
  "version": "0.0.0",
  "private": true,
  "type": "module",
  "scripts": {
    "build": "vite build",
    "test": "vitest run",
    "typecheck": "tsc -p tsconfig.app.json",
    "test:contract": "vitest run src/judge-contract.test.ts"
  },
  "engines": {
    "node": "24.18.0",
    "npm": "11.16.0"
  },
  "dependencies": {
    "react": "19.2.7",
    "react-dom": "19.2.7",
    "react-router-dom": "7.18.1"
  },
  "devDependencies": {
    "@axe-core/playwright": "4.12.1",
    "@eslint/js": "10.0.1",
    "@playwright/test": "1.61.1",
    "@testing-library/react": "16.3.2",
    "@testing-library/user-event": "14.6.1",
    "@types/node": "24.13.3",
    "@types/react": "19.2.17",
    "@types/react-dom": "19.2.3",
    "@vitejs/plugin-react": "6.0.3",
    "eslint": "10.7.0",
    "jsdom": "29.1.1",
    "typescript": "6.0.3",
    "typescript-eslint": "8.64.0",
    "vite": "8.1.5",
    "vitest": "4.1.10"
  }
}

```

### site/package.json

```
{
  "name": "relevance-incident-lab-public-site",
  "version": "0.1.0",
  "private": true,
  "engines": {
    "node": ">=22.13.0"
  },
  "scripts": {
    "dev": "WRANGLER_LOG_PATH=.wrangler/wrangler.log vinext dev",
    "build": "WRANGLER_LOG_PATH=.wrangler/wrangler.log vinext build",
    "start": "WRANGLER_LOG_PATH=.wrangler/wrangler.log vinext start",
    "test": "npm run build && node --test tests/rendered-html.test.mjs",
    "lint": "eslint . --ignore-pattern dist --ignore-pattern .next"
  },
  "dependencies": {
    "next": "16.2.10",
    "react": "19.2.7",
    "react-dom": "19.2.7",
    "react-router-dom": "7.18.1"
  },
  "devDependencies": {
    "@cloudflare/vite-plugin": "1.45.1",
    "@tailwindcss/postcss": "4.2.1",
    "@types/node": "22.19.19",
    "@types/react": "19.2.14",
    "@types/react-dom": "19.2.3",
    "@vitejs/plugin-react": "6.0.2",
    "@vitejs/plugin-rsc": "0.5.26",
    "eslint": "9.39.4",
    "eslint-config-next": "16.2.10",
    "react-server-dom-webpack": "19.2.6",
    "tailwindcss": "4.2.1",
    "typescript": "5.9.3",
    "vinext": "0.0.50",
    "vite": "8.1.5",
    "wrangler": "4.112.0"
  },
  "overrides": {
    "@babel/core": "7.29.7",
    "brace-expansion@1.x": "1.1.16",
    "brace-expansion@5.x": "5.0.7",
    "js-yaml": "4.3.0",
    "postcss": "8.5.20"
  },
  "type": "module"
}

```

### web/src/main.tsx

```typescript
import { StrictMode } from "react";
import { createRoot } from "react-dom/client";
import { BrowserRouter } from "react-router-dom";

import { App } from "./App";
import "./styles.css";

const root = document.getElementById("root");
if (root === null) {
  throw new Error("Application root is missing.");
}

createRoot(root).render(
  <StrictMode>
    <BrowserRouter>
      <App />
    </BrowserRouter>
  </StrictMode>,
);

```

### site/app/page.tsx

```typescript
"use client";

import { useSyncExternalStore } from "react";
import { HashRouter } from "react-router-dom";

import { App } from "../../web/src/App";
import "../../web/src/styles.css";

export default function Home() {
  const mounted = useSyncExternalStore(
    () => () => undefined,
    () => true,
    () => false,
  );

  if (!mounted) {
    return <main className="site-recorded-loading">Verifying recorded evidence…</main>;
  }

  return (
    <HashRouter>
      <App />
    </HashRouter>
  );
}

```

### web/src/App.tsx

```typescript
import { Navigate, Route, Routes } from "react-router-dom";

import { Investigation } from "./components/Investigation";
import { EvidenceSelection } from "./components/EvidenceSelection";
import { BlockDemo } from "./components/BlockDemo";
import { PuppyDemo } from "./components/PuppyDemo";
import { CaseNavigator } from "./components/CaseNavigator";

export function App() {
  return (
    <>
      <a className="skip-link" href="#main-content">
        Skip to main content
      </a>
      <CaseNavigator />
      <Routes>
        <Route path="/" element={<EvidenceSelection />} />
        <Route
          path="/cases/:caseId/runs/:runId"
          element={<Investigation />}
        />
        <Route path="/demonstrations/rel58-block" element={<BlockDemo />} />
        <Route path="/demonstrations/rel59-puppy" element={<PuppyDemo />} />
        <Route path="*" element={<Navigate replace to="/" />} />
      </Routes>
    </>
  );
}

```

### site/app/layout.tsx

```typescript
import type { Metadata } from "next";
import { headers } from "next/headers";
import "./globals.css";

export async function generateMetadata(): Promise<Metadata> {
  const requestHeaders = await headers();
  const host =
    requestHeaders.get("x-forwarded-host") ?? requestHeaders.get("host");
  const protocol = requestHeaders.get("x-forwarded-proto") ?? "https";
  const metadataBase = new URL(host ? `${protocol}://${host}` : "http://localhost:3000");

  return {
    metadataBase,
    title: "Relevance Incident Lab",
    description:
      "Inspect recorded search-ranking evidence and keep release decisions human.",
    openGraph: {
      title: "Relevance Incident Lab",
      description:
        "Inspect ranking evidence. Keep release decisions human.",
      images: [{ alt: "Relevance Incident Lab", url: "/og.png" }],
      type: "website",
    },
    twitter: {
      card: "summary_large_image",
      title: "Relevance Incident Lab",
      description:
        "Inspect ranking evidence. Keep release decisions human.",
      images: ["/og.png"],
    },
  };
}

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html data-ril-runtime="static-recorded" lang="en">
      <body>{children}</body>
    </html>
  );
}

```

### site/worker/index.ts

```typescript
/** Cloudflare Worker entry point for the vinext-starter template. */
import { handleImageOptimization, DEFAULT_DEVICE_SIZES, DEFAULT_IMAGE_SIZES } from "vinext/server/image-optimization";
import handler from "vinext/server/app-router-entry";

interface Env {
  ASSETS: Fetcher;
  DB: D1Database;
  IMAGES: {
    input(stream: ReadableStream): {
      transform(options: Record<string, unknown>): {
        output(options: { format: string; quality: number }): Promise<{ response(): Response }>;
      };
    };
  };
}

interface ExecutionContext {
  waitUntil(promise: Promise<unknown>): void;
  passThroughOnException(): void;
}

// Image security config. SVG sources with .svg extension auto-skip the
// optimization endpoint on the client side (served directly, no proxy).
// To route SVGs through the optimizer (with security headers), set
// dangerouslyAllowSVG: true in next.config.js and uncomment below:
// const imageConfig: ImageConfig = { dangerouslyAllowSVG: true };

const worker = {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    const url = new URL(request.url);

    if (url.pathname === "/_vinext/image") {
      const allowedWidths = [...DEFAULT_DEVICE_SIZES, ...DEFAULT_IMAGE_SIZES];
      return handleImageOptimization(request, {
        fetchAsset: (path) => env.ASSETS.fetch(new Request(new URL(path, request.url))),
        transformImage: async (body, { width, format, quality }) => {
          const result = await env.IMAGES.input(body).transform(width > 0 ? { width } : {}).output({ format, quality });
          return result.response();
        },
      }, allowedWidths);
    }

    return handler.fetch(request, env, ctx);
  },
};

export default worker;

```

[291 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]