# Apprentice
**The apprentice watches the expensive model work. Then earns the job.**
[](https://runapprentice.com)
[](LICENSE)
[](https://pypi.org/project/runapprentice/)
[](https://openai.com/codex)
[](https://platform.openai.com/docs/models)
[](#the-numbers)
**OpenAI Build Week · Developer Tools**
[](https://youtu.be/ZjQGyK_V4FU)
**[▶ Watch the demo film](https://youtu.be/ZjQGyK_V4FU)** — 1:47, the whole loop, ending on
the eval gate refusing a model.
---
## About this repository
**The production codebase is a private monorepo. This repository is the Build Week
artifact, and it is public.**
Saying that plainly up front, because the alternative would be to imply this is the whole
system, and it is not. What is here is real and runnable:
- the **feature we shipped during the submission window**, running standalone,
- the **benchmark that reproduces every GPT-5.6 number we claim**, with nothing of ours
in the loop,
- an honest account of **how Codex was used**, and which work is new this week.
The product itself is live. You do not need our source to test it: sign in at
[runapprentice.com](https://runapprentice.com), `pip install runapprentice`, and the
console will show your own traffic. Instructions below.
## What is in here
| Piece | What it does | Needs |
|---|---|---|
| [`drift-demo/`](drift-demo) | **The feature built during the window**, running standalone: the drift endpoints and the console page, over a seeded SQLite database. `seed.py` writes 30 days of traffic whose feedback score decays from 0.95 to 0.55, so you can watch a model quietly get worse. | nothing |
| [`benchmark/`](benchmark) | **Reproduces the GPT-5.6 numbers.** Talks to OpenAI and DSPy directly. No Apprentice API, no account, no database. Bring a key and check our arithmetic. | `OPENAI_API_KEY` |
| [`drift-panel/`](drift-panel) | The **production source** of the two endpoints and their tests, lifted verbatim from the private monorepo, so the code being judged is the code that shipped. | reading only |
---
## What Apprentice is
Everyone can make a model cheaper. Nobody can tell you when the cheap one is *safe*.
That is the whole problem. Swapping a frontier model for a small fine-tuned one is a
two-hour job; convincing an engineering lead that quality will not quietly collapse on the
3% of inputs nobody looked at is the part that never happens. So teams keep paying frontier
prices for extraction and classification work a 4B model could do in its sleep.
Apprentice watches a task your app already runs on a frontier model, then earns the job:
1. **Capture** — two lines of SDK, and real production traffic starts landing.
2. **Verify** — rows become **gold** (a human checked it) or **silver** (deterministic
checks passed). Everything else stays raw and never counts.
3. **Optimize** — DSPy GEPA rewrites the prompt against the verified set.
4. **Train** — a small model is fine-tuned on gold rows only.
5. **Eval gate** — every candidate is scored on held-out gold. It is promoted, or it is
**refused**. There is no third option and no human override.
6. **Watch for drift** — after takeover, the panel shows captured traffic, the feedback
your app reports, and offers a retrain only when enough new gold has arrived to be
worth it.
The refuse path is the product. A model that fails the gate does not ship, and you keep
paying the model that works.
---
## Built with Codex
Codex is the implementing engineer on the backend, not an autocomplete. The process is
written into the monorepo's `AGENTS.md` and followed every time:
1. **A bounded contract, not a vibe.** The brief names the exact unit of work, the files to
mirror, the hard guardrails, and the gate commands (mypy, ruff, pytest).
2. **Isolation.** Feature work runs in a dedicated git worktree on its own branch, so a bad
run cannot touch `main`.
3. **Codex writes and verifies.** It implements, runs the gates itself, and reports what
passed. It does not commit; a human does.
4. **A senior review is mandatory.** Green gates are necessary and never sufficient.
**Codex Session ID:** `019f5eb6-27b4-7e00-af1b-04285e89a907`
*(This is the Codex thread. The Devpost field calls it a "/feedback Session ID"; it has
nothing to do with this project's own `POST /v1/feedback` endpoint.)*
**Scale:** 152 Codex sessions on the monorepo between 2026-06-10 and 2026-07-14.
### The rule that earned its keep
On this very feature, Codex's tests went green and the code was still wrong three ways —
all caught by the human review, none caught by the gates:
1. The endpoint counted **gold + silver** rows toward retraining. **Training reads gold
only**, so silver rows would never reach the model. The panel would have overstated the
retrain payload — on a surface whose entire job is to be trusted.
2. The cutoff used the job's **queue** time instead of the moment the worker **snapshots**
rows, so rows created during a long training run were miscounted.
3. `eligible` ignored the minimum-rows gate, so the panel would have offered a one-click
retrain that the API rejects with a 400. A dead end, one click away from the user.
The tests passed because **they encoded the same wrong contract the brief did.** The brief
was mine, not Codex's. The lesson is that a spec must be written against the function that
enforces the rule, not against a doc that describes it.
Codex is also a good engineer inside the thread. In the session above it root-caused its
own failing test run (the reused venv's editable install pointed at the main checkout, so
pytest was exercising the old API) and caught a UTC bucketing bug in its own diff during
self-review, before handing anything back.
### GPT-5.6 inside the product
The GEPA optimizer's student and reflection models are configurable. The submission run
uses **`gpt-5.6-luna`** for both. Reproduce it yourself: [`benchmark/`](benchmark).
Codex itself runs on **`gpt-5.6-sol`** at medium effort.
---
## New this week, and what predates it
The rules require this distinction, so here it is without spin.
| | |
|---|---|
| **Built during the window (Jul 13–21)** | **The drift panel.** Two read-only endpoints and the console page that reads them. It wires the previously orphaned `POST /v1/feedback` signal into a surface a user can act on, and closes the loop after a model takes over. Written by driving Codex; see the session above. |
| **Also in-window** | The first scored GEPA run on `gpt-5.6-luna`, and a real bug it exposed: our run-cost accounting was billing for DSPy **cache hits** — LLM calls that never reached the provider — overstating cost ~15x. Fixed, and the run below is the verification. |
| **Predates the window** | Everything else: capture, the tier system, the GEPA optimizer, fine-tuning, the eval gate, the console, the SDK, the docs site. Apprentice has been in development since June 2026. |
---
## The numbers
Measured on 2026-07-14. Nothing here is projected.
| | |
|---|---|
| Baseline prompt (held out) | **50.17** |
| GEPA-optimized (held out) | **100.00** |
| Rows improved / regressed | **12 of 12 / 0** |
| Wall time | **67 seconds** |
| Cost | **$0.047** |
| Models | `gpt-5.6-luna`, student and reflection |
| Metric | deterministic JSON field F1 — no LLM judge marks its own homework |
| Split | 24 rows, seed 42, 12-row holdout |
Those numbers come from **Apprentice's own optimizer**, which i
[README truncated for size]
## Detected evidence (automated analysis)
Indexed codebase: 8 recognized source files, 53 KB.
- FastAPI (technology) — detected in the code
- HTML (language) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- Next.js (technology) — claimed on Devpost, not found in the code
- PostgreSQL (technology) — claimed on Devpost, not found in the code
- React (technology) — claimed on Devpost, not found in the code
- Supabase (technology) — claimed on Devpost, not found in the code
- Tailwind CSS (technology) — claimed on Devpost, not found in the code
- TypeScript (language) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (13 of 13)
```
.gitignore
assets/cover.html
benchmark/gepa_bench.py
benchmark/golden.csv
benchmark/pyproject.toml
drift-demo/app.py
drift-demo/pyproject.toml
drift-demo/seed.py
drift-demo/static/index.html
drift-panel/endpoints.py
drift-panel/test_endpoints.py
LICENSE
README.md
```
### Dependencies
- benchmark/pyproject.toml: dspy@>=3.0.0, openai@>=1.0.0
- drift-demo/pyproject.toml: fastapi@>=0.110, sqlalchemy@>=2.0, uvicorn@>=0.27
### Recent commits (newest first)
- docs: make the feedback example use sync capture, async returns no trace_id
- assets: gallery at 4K, rendered rather than screen-grabbed
- assets: re-grab the gallery frames at settled scene midpoints
- assets: logo-forward cover
- assets: Devpost gallery images (3:2)
- docs: link the demo film
- docs: the benchmark reproduces 100.00, and the 72.92 story stays
- fix: the benchmark does not reproduce the production number, and the README now says so
- Apprentice for OpenAI Build Week
## Key source files (fetched from GitHub, selected and truncated for size)
### benchmark/pyproject.toml
```
[project]
name = "apprentice-gepa-bench"
version = "1.0.0"
description = "Reproduce Apprentice's GPT-5.6 prompt-optimization numbers"
requires-python = ">=3.11"
dependencies = ["dspy>=3.0.0", "openai>=1.0.0"]
```
### drift-demo/pyproject.toml
```
[project]
name = "apprentice-drift-demo"
version = "1.0.0"
description = "The Apprentice drift panel, standalone"
requires-python = ">=3.11"
dependencies = ["fastapi>=0.110", "uvicorn>=0.27", "sqlalchemy>=2.0"]
```
### drift-demo/app.py
```python
"""The drift panel, standalone and runnable.
This is the feature Apprentice shipped during OpenAI Build Week (2026-07-13 to 07-21),
written by driving Codex. The production version lives in a private monorepo behind auth,
multi-tenancy and a job queue. This file is the same two endpoints and the same rules,
over a seeded SQLite database, so a judge can run it in one command and see it work.
The rules it enforces are the ones that matter, and they are not cosmetic:
* `drift` counts only rows whose source is a captured trace. Uploads and synthetic rows
are not production traffic and must never appear on a traffic chart.
* `retrain-candidate` counts GOLD rows only. Training reads gold only, so counting
silver would overstate what a retrain would actually learn from. (This is the bug the
human review caught after Codex's tests went green: the tests encoded the same wrong
contract the brief did. See the README.)
* The cutoff for "new" rows is when the last training job STARTED, not when it was
queued: the worker snapshots rows at start, so anything created after that was never
seen by the model.
* `eligible` is false unless a retrain would actually clear the same gate the training
endpoint enforces (MIN_TRAIN_ROWS). Offering a button that the API would reject with a
400 is a dead end, and a dead end is a bug.
Run: uv run uvicorn app:app --reload (then open http://localhost:8000)
"""
from __future__ import annotations
from datetime import date, datetime, timedelta, timezone
from pathlib import Path
from typing import Any
from fastapi import FastAPI, HTTPException
from fastapi.responses import FileResponse
from sqlalchemy import (
Boolean,
DateTime,
Float,
ForeignKey,
String,
Text,
create_engine,
func,
)
from sqlalchemy.orm import DeclarativeBase, Mapped, Session, mapped_column
DB_PATH = Path(__file__).parent / "drift-demo.db"
ENGINE = create_engine(f"sqlite:///{DB_PATH}")
# The production gate. A task cannot train below this many gold rows, so the panel must
# not offer a retrain below it either.
MIN_TRAIN_ROWS = 500
class Base(DeclarativeBase):
pass
class Task(Base):
__tablename__ = "tasks"
id: Mapped[str] = mapped_column(String(32), primary_key=True)
name: Mapped[str] = mapped_column(String(120), unique=True)
class Row(Base):
"""One captured input/output pair.
`tier` is the trust level: gold means a human verified it, silver means it passed
deterministic checks, raw means neither. `source` distinguishes production traffic
(trace) from an upload.
"""
__tablename__ = "rows"
id: Mapped[str] = mapped_column(String(32), primary_key=True)
task_id: Mapped[str] = mapped_column(ForeignKey("tasks.id"), index=True)
input: Mapped[str] = mapped_column(Text)
output: Mapped[str] = mapped_column(Text)
tier: Mapped[str] = mapped_column(String(10), default="raw", index=True)
source: Mapped[str] = mapped_column(String(20), default="upload")
feedback_score: Mapped[float | None] = mapped_column(Float, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True))
class Job(Base):
__tablename__ = "jobs"
id: Mapped[str] = mapped_column(String(32), primary_key=True)
task_id: Mapped[str] = mapped_column(ForeignKey("tasks.id"), index=True)
kind: Mapped[str] = mapped_column(String(20))
status: Mapped[str] = mapped_column(String(20))
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True))
app = FastAPI(title="Apprentice drift panel (Build Week demo)")
def _utc(value: datetime) -> datetime:
"""SQLite hands back naive datetimes. Treat them as UTC rather than local time.
Getting this wrong shifts every row into the wrong day bucket, which is exactly the
kind of quiet error a drift chart would show as real behaviour.
"""
return value.replace(tzinfo=timezone.utc) if value.tzinfo is None else value.astimezone(timezone.utc)
def _iso(value: datetime | None) -> str | None:
return _utc(value).isoformat() if value else None
def _task(session: Session, name: str) -> Task:
task = session.query(Task).filter(Task.name == name).one_or_none()
if task is None:
raise HTTPException(status_code=404, detail=f"No task named {name!r}. Run `python seed.py`.")
return task
@app.get("/v1/tasks/{name}/drift")
def task_drift(name: str, days: int = 30) -> dict[str, Any]:
"""Daily captured traffic and the feedback your app reported, over 1 to 90 days.
Every day in the window is present, zero-filled. A quiet day is a real fact and must
read as a quiet day, not as a gap in the line.
"""
days = max(1, min(days, 90))
today = datetime.now(timezone.utc).date()
start_date = today - timedelta(days=days - 1)
start_at = datetime.combine(start_date, datetime.min.time(), tzinfo=timezone.utc)
end_at = datetime.combine(today + timedelta(days=1), datetime.min.time(), tzinfo=timezone.utc)
series: dict[date, dict[str, Any]] = {
start_date
+ timedelta(days=offset): {
"date": (start_date + timedelta(days=offset)).isoformat(),
"traces": 0,
"feedback_count": 0,
"mean_feedback_score": None,
}
for offset in range(days)
}
with Session(ENGINE) as session:
task = _task(session, name)
traces = (
session.query(Row.created_at, Row.feedback_score)
.filter(
Row.task_id == task.id,
Row.source == "trace", # production traffic only
Row.created_at >= start_at,
Row.created_at < end_at,
)
.all()
)
scores: dict[date, list[float]] = {}
for created_
[truncated — 2351 more characters]
```
### assets/cover.html
```html