ClassLens

Turn a class's wrong answers into tomorrow's lesson — and verify the fix before a student sees it.

ClassLens takes a class's responses to an Algebra-1 diagnostic quiz and maps each wrong answer to a named misconception (grounded in the Eedi misconception taxonomy), with measured accuracy on held-out official Eedi diagnostic item–distractor mappings. It then encodes the class's top misconceptions as executable buggy-math rules, and runs a factory that manufactures a verified recheck quiz provably targeting them. Output: a class report (static HTML), an editable reteach plan (Markdown), and the recheck quiz.

Built with Codex and GPT-5.6 for OpenAI Build Week. Track: Education.

Judge snapshot: GPT-5.6 Luna’s selected three-hypothesis arm achieved 41.1% @1 / 55.4% @3 on n=716 real held-out Eedi pairs, a +33.5-point audited @1 lift over local retrieval. Executable rules achieved 41.1% grammar coverage × 67.0% in-grammar fidelity = 27.5% strict reproduction (71/258), and the blind factory accepted 10/33 (30.3%) items while visibly rejecting 23 attempts.

See it with zero setup — view the live report (no clone needed), or open the committed examples/sample_report.html directly. It is the pre-rendered judge-facing artifact; it does not run the pipeline or call an API.

Status: all core CLI paths are implemented and locally tested. Retrieval has completed its held-out calibration; factory and executable-rule measurements are seeded, bounded real-Eedi calibrations.

Why

Algebra 1 is the gatekeeper course, and teachers run the same daily loop — teach, quick-check, reteach — but cannot turn 150 exit-ticket responses into "what does this class actually misunderstand, and what do I reteach tomorrow" in a 20-minute prep. They see scores, not causes. ClassLens surfaces the causes, writes the reteach, and verifies the recheck questions actually target the diagnosed misconceptions.

How it works

1. Ingest quiz questions + student responses from a CSV export.
2. Diagnose — for each wrong answer, retrieve candidate misconceptions from the Eedi taxonomy (local embeddings) and use GPT-5.6 to rerank/verify the best match against the specific question and answer.
3. Ground — verify computable items symbolically with SymPy, so correctness is checked, not guessed.
4. Profile — aggregate a class-level misconception profile ranked by prevalence, with the students affected by each.
5. Flag broken questions — where "wrong" answers reflect valid reasoning or ambiguity, flag the question, not the students.
6. Reteach — GPT-5.6 writes an editable plan per top misconception: a worked example, the counterexample that breaks it, targeted practice, and a recheck item.
7. Verify the recheck — the recheck items come from a factory: GPT-5.6 writes the stem, SymPy computes the correct answer, an executable buggy-math rule generates the distractor, and the diagnosis engine must map that distractor back to the intended misconception (blind) or the item is rejected and regenerated.

Design principle: ClassLens attaches to the invariant (misconceptions are curriculum-independent), not the variable (lesson plans, textbooks, formats). Bring your own questions in any format; it maps errors to a universal taxonomy. Narrow on subject, general across teachers.

Measured results

On a held-out split of official Eedi question–wrong-answer–misconception pairs:

• Final seeded 716-item held-out Eedi evaluation, seed 47: BAAI BGE-large hybrid local retrieval reached 7.5% @1 / 20.0% @3 (95% CI @1 5.8–9.7%). The selected three-hypothesis + GPT-5.6 Luna rerank arm reached 41.1% @1 / 55.4% @3 (95% CIs 37.5–44.7% / 51.8–59.0%) with 63.8% recall@25: a measured +33.5-point @1 lift with conservative 95% CI +27.8 to +38.9 points. The rerank-only arm is a clearly labeled supplementary n=75 decomposition (30.7% / 41.3% @1/@3), not the headline comparison. The earlier fixed 75-item retrieval calibration selected BGE-large hybrid plus three hypotheses (64.0% recall@25); k=50 did not improve that calibration, so the candidate pool remains 25.
• Executable-misconception distractor reproduction: 27.5% (71/258) across the full eligible seeded official-Eedi development set, after widening only grammar recognition against the same cached 258 GPT extractions. Grammar coverage is 41.1% (106/258) and in-grammar fidelity is 67.0% (71/106), so coverage × fidelity = strict rate. The strict rate remains the headline because parse misses are counted rather than hidden.
• Factory round-trip acceptance: 30.3% (10/33) across three executable misconception targets; 23 attempts were visibly rejected. Blind local retrieval hit the intended target on 10 of 26 viable attempts (38.5%) and missed it on 16, which is surfaced rather than hidden by target injection.

Executable-rule validation (full eligible set, n=258, seed 47)

Rule	Funnel: eligible -> in-grammar -> reproduced	Coverage	Fidelity	Strict rate
same-operation-not-inverse	41 -> 2 -> 1	4.9%	50.0%	2.4%
left-to-right priority	27 -> 6 -> 5	22.2%	83.3%	18.5%
negative-times-negative	24 -> 3 -> 3	12.5%	100.0%	12.5%
first-term-only expansion	17 -> 7 -> 7	41.2%	100.0%	41.2%
multiply-by-index	16 -> 10 -> 10	62.5%	100.0%	62.5%
add-fraction-denominators	15 -> 9 -> 8	60.0%	88.9%	53.3%
positive-negative-is-positive	15 -> 6 -> 4	40.0%	66.7%	26.7%
add-instead-of-expand	12 -> 3 -> 1	25.0%	33.3%	8.3%
subtraction-is-commutative	11 -> 4 -> 4	36.4%	100.0%	36.4%
double-instead-of-square	31 -> 23 -> 13	74.2%	56.5%	41.9%
division-is-commutative	17 -> 10 -> 5	58.8%	50.0%	29.4%
add-instead-of-multiply	11 -> 8 -> 4	72.7%	50.0%	36.4%
reverse-inequality	21 -> 15 -> 6	71.4%	40.0%	28.6%

Class-aggregation robustness

Class prevalence is calculated from distinct affected learners rather than repeated responses, and identical (question, answer, explanation) pairs are deduplicated before runtime mapping. The report keeps per-question evidence and symbolically identified slips separate from named misconceptions, so a repeated answer pattern cannot silently inflate a reteach target. The shipped class is explicitly an illustrative composition of real Eedi answers, not a claim of longitudinal student data.

How GPT-5.6 is used

The retrieval-only baseline is local and never calls an API. GPT-5.6 writes three cached misconception hypotheses for the strongest retrieval arm, then reranks and verifies the fixed local candidate pool against each specific question and wrong answer; the held-out benchmark measures all three arms, and the runtime diagnose command ships that same selected best arm (three cached hypotheses + hybrid retrieval + rerank/verify), so the product runs the configuration the benchmark measured. Later runtime roles are structured stem extraction, factory-item generation, reteach generation, class summaries, and broken-question detection. Calls are cached by model, prompt, and reasoning effort; GPT-5.6 reasoning models do not accept a temperature parameter, so deterministic roles use its fixed default behavior. The default runtime model is gpt-5.6-luna for cost-disciplined batch diagnosis; gpt-5.6-terra is reserved for a small final comparison or difficult verification cases.

How Codex was used

Codex built the core of ClassLens over a three-day session: the benchmark harness, the executable misconception rules, the recheck factory, and the report renderer. The engineering decisions below were made or corrected in that thread.

Measurement discipline. Codex built the held-out benchmark before optimizing against it, then corrected its own first ablation so both arms shared the same local candidate pool, accuracy@3 meant the first three ranked candidates, and evaluation order was seeded. The uncorrected early numbers were discarded rather than reported. It later replayed the retrieval-only baseline with no GPT client and reproduced exactly 54/716 @1 and 143/716 @3, confirming the published 7.5% / 20.0% baseline and the +33.5-point lift were uncontaminated.

Retrieval calibration. Codex compared free local embedding models on a fixed held-out slice and selected BGE-large hybrid on recall@25, added a TF-IDF lexical arm with reciprocal-rank fusion, and tested a larger candidate pool. k=50 did not improve recall, so k=25 was retained. A high-effort selector variant did not improve top-1 and was discarded.

Honest failure surfaces. Codex removed the factory's target-injection shortcut so retrieval misses and invalid rule outputs became visible rejection reasons, and made parse failures count against the executable-rule reproduction rate rather than being excluded from the denominator. Both choices lower the headline numbers and were kept anyway.

Correctness fixes it found. Option letters such as B are resolved to their actual answer text before retrieval and diagnosis, preventing meaningless letter-only diagnoses in standard multiple-choice exports. The rule validator compares symbolic equivalents rather than raw strings, so 1/4 and 3/12 match.

Reproducibility. Every GPT-5.6 response is cached by model, prompt, and reasoning effort and committed, so the published report and metrics can be regenerated with no API key. Setting CLASSLENS_OFFLINE=1 makes any cache miss fail before a live request, which is how the offline audits above were run.

Setup

Python 3.10+ is required (pyproject.toml is tested with Python 3.11). On Windows, keep the checkout path short or enable long-path support before installing the local torch dependency.

# Windows PowerShell
py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1

# macOS/Linux equivalent: python3.11 -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements.txt
python -m pip install -e .
Copy-Item .env.example .env  # add OPENAI_API_KEY only for live GPT commands

ClassLens reads OPENAI_API_KEY from your environment (or a local .env, which is gitignored). No key is bundled with this repo — bring your own.

Get the data

See data/README.md — download the Eedi misconception dataset from Kaggle (accept the competition rules first). Data is gitignored.

Usage

Every CLI command below requires the Kaggle Eedi download described in data/README.md (at minimum, the taxonomy; benchmark, validation, and factory commands also need their corresponding prepared splits). For a zero-setup judge-facing artifact, open examples/sample_report.html directly instead of rerunning the pipeline.

# Diagnose a class -> HTML report + Markdown reteach plan (best-arm GPT-5.6 diagnosis)
python -m classlens diagnose --responses examples/sample_class.csv --questions examples/sample_questions.csv --out report.html

# Regenerate the committed sample report from the local GPT cache only; any cache miss fails
# before a live call (PowerShell: $env:CLASSLENS_OFFLINE="1")
CLASSLENS_OFFLINE=1 python -m classlens diagnose --responses examples/sample_class.csv --questions examples/sample_questions.csv --out examples/sample_report.html --seed 47

# Fully local fallback (no GPT anywhere): retrieval-only diagnosis, offline reteach scaffold
python -m classlens diagnose --responses examples/sample_class.csv --questions examples/sample_questions.csv --out report.html --local-only

# Benchmark the diagnosis engine on held-out real Eedi data (accuracy@1/@3, GPT-5.6 lift)
python -m classlens bench

# Validate executable misconceptions against real Eedi distractors
python -m classlens validate-rules

# Generate + verify recheck items for a misconception
python -m classlens factory --misconception <id> --n 5

All commands that call the API support --limit N and --dry-run, and print a cost estimate first.

Suggested zero-cost planning checks:

python -m classlens validate-rules --dry-run --limit 12 --seed 47
python -m classlens factory --misconception 1507 --n 1 --limit 2 --dry-run --seed 47

After any fresh API run, inspect local token accounting with:

python -m classlens costs

The OpenAI billing dashboard remains the source of truth for charges; ClassLens records no prompt text in this ledger.

Sample data (run it without the full dataset)

examples/ contains a small illustrative class (30 anonymized learner IDs, 7 official Eedi Algebra-1 diagnostic questions plus one blind-verified factory recheck item, and 240 composed responses), the committed GPT-5.6-diagnosed report (sample_report.html / .md), two canonical verified recheck items across two distinct executable-rule targets (duplicates are intentionally omitted), and the cached validation summary behind the report. On this composition, 9 of the 10 distinct official-Eedi wrong answers are diagnosed to their official gold misconception label at rank 1 (10 of 10 within the top 3); the single rank-2 case is exactly the item the report flags for teacher review, and one further response is a SymPy-detected equivalent-answer slip rather than a misconception. The class's top two misconceptions are the two with blind-verified factory recheck items, so the diagnose -> reteach -> verified-recheck loop closes on real targets. The official competition release contains item/distractor labels, not row-level student histories; the composition is therefore clearly labeled as illustrative rather than claimed as raw individual-response data. The committed report is the zero-setup artifact; regenerating it requires the Eedi data and the local GPT cache (every call behind the committed report is cached, and CLASSLENS_OFFLINE=1 makes any cache miss fail before a live request).

Demo video

https://www.youtube.com/watch?v=n6QWy5xHBMY

Disclosures

• Uses the official Eedi "Mining Misconceptions in Mathematics" Kaggle competition release: 1,869 diagnostic questions, 4,370 labeled distractor pairs, and a 2,587-label misconception taxonomy. Dataset license and scope are recorded in data/README.md; used per its terms.

License

MIT. See LICENSE.