# Project export: Re:Compress
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: UC Berkeley AI Hackathon 2026
- Tagline: A query-aware rewriting layer that extends compression into the regime deletion can't reach — distilled into a 1.5B model, then carried into multi-turn conversations.
- Devpost: https://devpost.com/software/re-compress
- GitHub: https://github.com/Kart-ing/ReCompress
- Demo: https://zenodo.org/records/20786357
- Video: https://www.youtube.com/embed/2cx1czVpz8I?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Best Compression Model for The Token Company)
- Team: 3 GitHub contributor(s) — kartikey-ship-it (57 commits), Claude Opus 4.8 (1M context) (53 commits), Parth12358 (10 commits)
## Devpost submission (written by the team)
### Overview
Paste each section into the matching Devpost field. This covers the WHOLE project — both acts and all five distillation experiments. Tagline + "Built With" + links at the bottom. Project name ReCompress: rewrite-don't-delete context compression, distilled to 1.5B and extended to flat-context multi-turn Tagline (Devpost "tagline" field) A query-aware rewriting layer that extends compression into the regime deletion can't reach — distilled into a 1.5B model, then carried into multi-turn conversations to keep a 12-turn chat flat (184 tok) while a naive agent balloons to 1,482.
### Inspiration
The Token Company's bear-2 compresses prompts by deleting low-value tokens — fast and lossless-by-design, but blind in two ways deletion structurally can't fix: it can't read your question, and it can't rewrite. We wanted to measure what those two abilities are worth — and prove you don't need a giant model to get them. Compression isn't only a single-prompt problem, though. In a long conversation, context grows every turn (O(n²) cost). So we built the project in two acts: Act 1 — a query-aware compressor distilled into a small offline model; Act 2 — a multi-turn memory ("Re:Zero") that uses that compressor to keep context flat forever.
### What it does
ReCompress is one system in two acts. Act 1 — single-shot compression. Given a long context + a question, it drops the passages irrelevant to that question and rewrites the rest densely; a downstream LLM then answers. We distilled this behavior (from a DeepSeek teacher) into Qwen2.5-1.5B + LoRA so it runs offline and cheap. Act 2 — Re:Zero multi-turn memory. A fixed ~300-token budget per turn — protected facts + a compressed checkpoint of older turns + the recent raw delta — so context stays flat instead of growing. The checkpoints are compressed by the Act 1 model, so the whole agent runs on our distilled compressor. Headline results. Act 1 — distilled 1.5B vs bear-2, same compression instruction (ours realizes ~8.5× fewer tokens: ~48 vs ~409 on HotpotQA), QA-F1, paired bootstrap 95% CI: Act 2 — multi-turn HotpotQA, 6 turns, n=20: Re:Zero powered by our distilled model wins on both axes — best answer quality at the fewest tokens. Over 12 turns a naive agent grows to 1,482 context tokens while Re:Zero stays flat at ~184 (8.1× less, and diverging).
### How we built it
— and the five experiments behind the winning model The Act 1 distilled model is the result of five distillation experiments, each a deliberate test. We name them so the trajectory is legible (not "v1…v5"): Plus a sixth idea we designed, tested, and dropped — "Bear-Booster": train the small model to make bear's output better (optimize bear(model(text))). Our own data showed it's dominated — the standalone model beats model→bear on every benchmark, and a pre-processor is strictly costlier than bear alone. A clean negative result that sharpened the thesis: rewriting must replace deletion, not augment it. Pipeline: DeepSeek teacher → 5,000 query-aware compression pairs → LoRA fine-tune Qwen2.5-1.5B (4-bit, Unsloth) on a Modal H100 → eval vs bear (TTC SDK) under the same compression instruction with bootstrap CIs across 4 benchmarks → wire the winner (Hearth) into Re:Zero as a pluggable backend → custom multi-turn benchmark. ~$10–15 total compute.
### Challenges we ran into
Distillation failed twice before Hearth worked (Spark wash → Bonfire overfit). Fixing it took the full anti-overfitting playbook: more data, lower rank, dropout, weight-decay, early-stopping on best-eval. A "smarter" idea (answer-grounded, Oracle/Oracle-Lite) lost — twice. Optimizing against downstream answer success overfit the frozen solver and hurt out-of-distribution generalization. Documented as a negative result. The Modal + Unsloth + trl stack is version-brittle — ~7 distinct runtime failures before the first clean train (dependency conflicts, container data paths, the formatting_func contract, eval-time CUDA OOM, a packing incompatibility). All written up as reproducibility notes. Integration: wiring the Act 1 Modal model into Act 2's synchronous checkpoint loop without breaking either codebase. Accomplishments we're proud of A 1.5B model that recovers the query-aware regime bear cedes — beating bear with statistical significance while emitting ~8.5× fewer tokens, and transferring to a near-in-distribution benchmark it never trained on (2Wiki, +46%; directional-but-unproven on the dissimilar OOD sets). It complements deletion rather than replacing it: deletion stays best for fast/verbatim/reusable; rewriting adds the query-specific case. We stress-tested our own headline before a judge could. The teacher and solver are both DeepSeek (a circularity a sharp reviewer attacks first), so we re-scored with an independent solver (Claude Sonnet): the gap is invariant — Δ vs bear = +0.288 (independent) vs +0.285 (in-family), CI excludes zero both ways. The result is not a same-family artifact. A unified system: the same distilled compressor works as a single-shot compressor and as a multi-turn memory engine (the strongest backend of the three we tested, vs DeepSeek and bear). Research-grade rigor + intellectual honesty in 24h: bootstrap CIs on every claim, a cross-solver audit, a mask-the-answer audit (measured against ourselves), 5 named experiments with a clear winner, three documented negative results, a conceptual finding (the "deletion ceiling"), a 13-figure visualization suite, and a custom multi-turn benchmark.
### What we learned
Query-aware rewriting beats blind *deletion* at far fewer tokens — most where there are distractors (multi-hop QA). On purely abstractive QA (MS MARCO) it ties bear — an honest boundary we report. The win survives an independent judge (Claude Sonnet, +0.288) — it's not teacher↔solver affinity. Much of the margin is span-selection, not reasoning — and we proved it on ourselves. Masking the gold answer from the compression drops our F1 by 65% (vs bear's 31%). So our edge is largely "query-aware compression keeps the answer-bearing span at a 3.5% budget where bear's deletion at 30% truncates it" — a real, useful property, stated precisely rather than oversold. We measured our multi-turn overhead honestly — and found our own expensive component was useless. The "8.1× flatter context" is the solver-context axis; counting the per-turn compression LLM calls, the system actually costs more in total tokens at short horizons. Digging in, the LLM "Echidna" checkpoint-trigger turned out to decide checkpoint 98.3% of the time — no real decision. We replaced it with a free rule (2.6× cheaper, same F1) and swept conversation length: with the cheap trigger the system beats an uncached growing-history agent on total tokens from ~6 turns, widening monotonically to ~4× by 20 turns (6,886 vs 28,838). The LLM-trigger version was actually so expensive it got overtaken by the naive agent around turn 11 — it was counterproductive, not just wasteful. Against a cached agent we still don't win on raw tokens, and we say so. You can't fix deletion by stacking it after rewriting (model→bear < model everywhere) — the "deletion ceiling." Downstream-grounded distillation isn't free — selecting by a frozen judge overfits it (Oracle/Oracle-Lite both lost to imitation-based Hearth). The same compressor composes — single-shot quality transfers to keeping conversations flat.
### What's next
Direct comparison to LLMLingua-2 (closest prior work — they delete/classify tokens; we rewrite and distilled it into a generative 1.5B). Scale teacher data + ratios to push MuSiQue/SQuAD into significance. A live demo of Re:Zero holding a long conversation flat in real time. "Built With" python · qwen2.5-1.5b · lora · unsloth · modal · h100 · deepseek-api · the-token-company-sdk · huggingface-datasets · hotpotqa · 2wikimultihop · musique · squad · ms-marco · matplotlib Links 📄 Paper (Zenodo, DOI): https://doi.org/10.5281/zenodo.20786357 📓 Reproduce in Google Colab: https://colab.research.google.com/github/Kart-ing/ReCompress/blob/main/notebooks/ReCompress_reproduce.ipynb 🎛 Interactive demo: https://demo-eight-olive-97.vercel.app 🖥 Slides: https://slides-teal-tau.vercel.app 💻 Code (both acts): https://github.com/Kart-ing/ReCompress Trained adapters + full experiment archive (all 5 experiments, incl. failures) + figures: in the repo (Git LFS) One-line differentiator (keep ready for the pitch + Q&A) "LLMLingua deletes tokens; we rewrite them — distilled into a 1.5B (after five experiments — Hearth won) that extends The Token Company's bear into the query-aware regime it cedes, measured head-to-head with CIs across 4 benchmarks, and powers a multi-turn memory that keeps a 12-turn chat flat at 184 tokens while a naive agent hits 1,482."
## README (from the GitHub repository)
# ReCompress
[](https://doi.org/10.5281/zenodo.20786357)
**A query-aware *rewriting* layer that extends [The Token Company](https://thetokencompany.com)'s compression into the regime deletion can't reach — distilled into a 1.5B model, then carried into multi-turn conversations.**
📄 **Paper:** [Zenodo (DOI: 10.5281/zenodo.20786357)](https://doi.org/10.5281/zenodo.20786357) · 🎛 **Interactive demo:** [demo-eight-olive-97.vercel.app](https://demo-eight-olive-97.vercel.app) · 🖥 **Slides:** [slides-teal-tau.vercel.app](https://slides-teal-tau.vercel.app) · 📓 **Reproduce in Colab:** [`notebooks/ReCompress_reproduce.ipynb`](https://colab.research.google.com/github/Kart-ing/ReCompress/blob/main/notebooks/ReCompress_reproduce.ipynb)
The Token Company's **bear-2** is an excellent foundation: it compresses prompts by deleting low-value tokens — fast, verbatim-faithful, query-agnostic, and reusable across many questions. By design, it doesn't paraphrase or generate ("nothing is paraphrased or generated"). **ReCompress takes up exactly where that design leaves off:** a small, question-conditioned model that *rewrites* — dropping passages irrelevant to *this* question and densifying the rest — then we **distill that behavior into Qwen2.5-1.5B + LoRA** so it runs offline and cheap, in the same product category as bear. It is **not a competitor to bear; it's the abstractive, query-aware regime bear explicitly cedes**, packaged as a small model that complements a deletion-based compressor.
ReCompress is one research project in **two acts**:
| | **Act 1 — Single-shot compression** | **Act 2 — Re:Zero multi-turn memory** |
|---|---|---|
| Question | Can a small model recover the *query-aware* gains deletion leaves on the table? | Can it keep a *long conversation* from growing O(n²)? |
| Method | Query-aware **rewrite** (drop distractors + densify), distilled into **Qwen2.5-1.5B + LoRA** | A flat ~300-token memory (protected facts + compressed checkpoint + recent delta), whose checkpoints are compressed **by the Act 1 model** |
| Result | A 1.5B that compresses HotpotQA context to **~3.5% of tokens** and still answers correctly — recovering the distractor-dropping that blind deletion can't do | Context stays **flat (~184 tok) while a naive agent grows to 1,482 over 12 turns (8.1× less)**, with our distilled model as the engine |
**The two acts are one system:** Act 1 distills a query-aware compressor into a small offline model; Act 2 makes that model the memory engine of a multi-turn agent. The thesis: **query-aware rewriting recovers the gains deletion can't — and a 1.5B can carry it, for single prompts and whole conversations.**
> **Where this complements The Token Company.** bear wins on the axes it was built for — speed (non-autoregressive deletion), verbatim fidelity (no hallucination), and compress-once-serve-many reuse. ReCompress adds the *one* thing deletion structurally can't (reading the question and rewriting), and shows a 1.5B can deliver it offline. Together they cover both regimes: deletion for fast/reusable/verbatim, rewriting for query-specific distractor-heavy context. **The rigorous head-to-head benchmarks, confidence intervals, cross-solver audits, and honest negative results live in the [Appendix: Methodology & Benchmarks](#appendix-methodology--benchmarks) below** — this project's core is depth of research, so the evidence is all there for anyone who wants it.
*Built in 24h for the **Token Company Compression Challenge**, UC Berkeley AI Hackathon 2026. This README doubles as the research writeup — it preserves the full failure→success trajectory, not just the wins.*
---
## What we built
- **A distilled 1.5B query-aware compressor** (`recompress/`) — DeepSeek teacher → Qwen2.5-1.5B + LoRA on a Modal H100, ~$10 total. Runs offline.
- **Re:Zero, a flat-context multi-turn memory** (`rezero/`) — trauma (protected facts) + compressed checkpoint + recent delta, capped at ~300 tokens, with the Act 1 model as the checkpoint compressor. Keeps a 12-turn conversation flat at ~184 tokens vs a naive agent's 1,482.
- **A research-grade evaluation harness** — a 5-bar paired benchmark (same compression instruction, ours realizing ~8.5× fewer tokens) with bootstrap 95% CIs across four QA datasets, a cross-solver audit (independent judge), a mask-the-answer audit, and a documented v1→v3 distillation trajectory including the failures. *(All numbers in the Appendix.)*
## Architecture
### Act 1 — single-shot query-aware compression
```mermaid
flowchart TD
A[HotpotQA context + question] --> B[DeepSeek teacher\nquery-aware compress]
B --> C[5,000 training pairs\nfiltered for answer leakage]
C --> D[LoRA fine-tune\nQwen2.5-1.5B on Modal H100]
D --> E[Distilled student\n1.5B offline model]
A --> F[bear-2\nblind deletion baseline]
E --> G[Frozen DeepSeek solver]
F --> G
G --> H[QA-F1 vs ground truth]
style E fill:#E1F5EE,stroke:#0F6E56,color:#085041
style F fill:#FCEBEB,stroke:#A32D2D,color:#791F1F
style H fill:#E6F1FB,stroke:#185FA5,color:#0C447C
```
### Act 2 — RbD-Compress multi-turn memory
```mermaid
flowchart TD
U[User message] --> TE[Trauma extractor\nscans for critical facts]
TE -->|updates| TM[(Trauma memory\n~50 tok protected)]
TE --> E[Echidna trigger\nreads trauma first]
TM --> E
E -->|checkpoint| CP[Checkpoint builder\ndistilled 1.5B compressor]
E -->|revert| CS
E -->|pass| CB
CP --> CS[(Checkpoint stack\nversioned history)]
CS --> CB[Context builder]
TM --> CB
U --> CB
CB --> S[Solver\nfrozen DeepSeek]
S --> R[Answer]
style TM fill:#FAECE7,stroke:#993C1D,color:#712B13
style CS fill:#E1F5EE,stroke:#0F6E56,color:#085041
style E fill:#EEEDFE,stroke:#534AB7,color:#3C3489
style CB fill:#E6F1FB,stroke:#185FA5,color:#0C447C
```
### Distillation trajectory
```mermaid
flowchart LR
V1[v1\n261 examples\nrank 16] -->|underfitting\nno significant gain| V2
V2[v2\n2500 examples\nrank 64\n6 epochs] -->|overfit\nloss rises epoch 3+| V3
V3[v3\n5000 examples\nrank 32\n3 epochs + regularization]
style V1 fill:#FCEBEB,stroke:#A32D2D,color:#791F1F
style V2 fill:#FAEEDA,stroke:#854F0B,color:#633806
style V3 fill:#E1F5EE,stroke:#0F6E56,color:#085041
```
### The roles
| Role | Model | Notes |
|---|---|---|
| Teacher (compressor) | DeepSeek (API) | query-aware; generates distillation data; also the standalone upper-bound bar |
| Baseline | bear-2 (TheTokenCompany SDK) | blind deletion; the system we measure against |
| **Student (the submission)** | **Qwen2.5-1.5B-Instruct + LoRA, 4-bit** | distilled on Modal H100; runs offline |
| Solver (judge) | DeepSeek (API), **frozen** | identical across all bars — it only ever sees the compressed context, never the original |
Holding the solver fixed is what makes the comparison fair: every bar is judged by the same downstream reader, so any F1 difference is attributable to the **compressor**, not the solver.
---
## 2. The idea
Retrieval-augmented prompts are mostly noise: 10 passages retrieved, 2 relevant, 8 distractors. A QA model then has to find the needle. Two ways to shrink that prompt:
| | **bear-2 (deletion)** | **ReCompress (rewrite)** |
|---|---|---|
| Operation | Deletes low-value tokens, char-for-char | Reads the question, **drops** off-topic passages, **rewrites** the rest densely |
| Sees the question? | **No** (query-agnostic) | **Yes** (query-aware) |
| Can paraphrase? | **No** (extractive) | **Yes** (abstractive) |
| Failure mode | Keeps distractors; truncates mid-passage; preserves the *wrong* facts faithfully | Can hallucinate / drop a fact if it mis-reads the question |
bear's design is honest about its blind spots ("nothing is paraphrased or generated"). ReCompress fills exactly that gap. It is **not** a competitor to bear — it's the abstractive, query-conditioned regime bear explicitly does
[README truncated for size]
## Detected evidence (automated analysis)
Indexed codebase: 122 recognized source files, 544 KB.
- Anthropic (technology) — detected in the code
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- TypeScript (language) — detected in the code
- AI coding agent: Claude Code — evidence: commit authorship or trailers
## Codebase structure (from repository index)
### Files (120 of 247)
```
.env.example
.gitattributes
.gitignore
ACTIII/baselines/__init__.py
ACTIII/baselines/coin_toss.py
ACTIII/baselines/naive.py
ACTIII/baselines/token_company.py
ACTIII/engine/__init__.py
ACTIII/engine/deepseek.py
ACTIII/engine/ratelimit.py
ACTIII/engine/tokens.py
ACTIII/experiments/__init__.py
ACTIII/experiments/runner.py
ACTIII/requirements.txt
ACTIII/tests/__init__.py
ACTIII/tests/test_coin_toss.py
artifacts/adapter
artifacts/adapter_v3
artifacts/adapter_v3_clean/adapter/adapter_config.json
artifacts/adapter_v3_clean/adapter/adapter_model.safetensors
artifacts/adapter_v3_clean/adapter/added_tokens.json
artifacts/adapter_v3_clean/adapter/chat_template.jinja
artifacts/adapter_v3_clean/adapter/merges.txt
artifacts/adapter_v3_clean/adapter/README.md
artifacts/adapter_v3_clean/adapter/special_tokens_map.json
artifacts/adapter_v3_clean/adapter/tokenizer_config.json
artifacts/adapter_v3_clean/adapter/tokenizer.json
artifacts/adapter_v3_clean/adapter/vocab.json
artifacts/adapter_v5/adapter/adapter_config.json
artifacts/adapter_v5/adapter/adapter_model.safetensors
artifacts/adapter_v5/adapter/added_tokens.json
artifacts/adapter_v5/adapter/chat_template.jinja
artifacts/adapter_v5/adapter/merges.txt
artifacts/adapter_v5/adapter/README.md
artifacts/adapter_v5/adapter/special_tokens_map.json
artifacts/adapter_v5/adapter/tokenizer_config.json
artifacts/adapter_v5/adapter/tokenizer.json
artifacts/adapter_v5/adapter/vocab.json
CITATION.cff
data/distill/_gen.log
data/distill/answergrounded.jsonl
data/distill/train_clean.jsonl
data/distill/train_v2.jsonl
data/distill/train_v3.jsonl
data/distill/train.jsonl
data/distill/v5_greedy.jsonl
data/echidna/echidna_train.jsonl
demo/.gitignore
demo/index.html
demo/package.json
demo/README.md
demo/slides/.gitignore
demo/slides/index.html
demo/src/App.tsx
demo/src/components/Benchmarks.tsx
demo/src/components/Crossover.tsx
demo/src/components/Explorer.tsx
demo/src/components/Hero.tsx
demo/src/components/Honesty.tsx
demo/src/data/5bar_2wiki.json
demo/src/data/5bar_hotpotqa.json
demo/src/data/5bar_musique.json
demo/src/data/5bar_squad.json
demo/src/data/cross_solver_audit.json
demo/src/data/echidna_ablation_sweep.json
demo/src/data/index.ts
demo/src/data/mask_symmetric.json
demo/src/main.tsx
demo/src/styles.css
demo/src/vite-env.d.ts
demo/tsconfig.json
demo/vercel.json
demo/vite.config.ts
docs/DEVPOST.md
docs/EXPERIMENT_LOG.md
docs/FINDINGS_deletion_ceiling.md
docs/MULTITURN_OVERHEAD.md
docs/PRD_act1.md
docs/PRD_act2.md
docs/PRD_demo.md
docs/PRD_integration.md
docs/prd.canvas.json
docs/PROGRESS.md
experiments/EXPERIMENT_LOG.md
experiments/v1/data/gen_v1.log
experiments/v1/data/train_v1_clean_290.jsonl
experiments/v1/data/train_v1_raw_300.jsonl
experiments/v1/eval/5bar_api_baseline.json
experiments/v1/eval/5bar_distilled_v1.json
experiments/v1/logs/modal_eval_v1.log
experiments/v1/logs/modal_train_v1.log
experiments/v1/loss_curve_v1.txt
experiments/v3/eval/5bar_distilled_v3.json
experiments/v3/logs/eval_v3.log
experiments/v3/logs/train_v3.log
experiments/v3/train_v3.jsonl
LICENSE
logs/5bar_run.log
logs/combined_bench.log
logs/gen_answergrounded.log
logs/gen_v2_ext.log
logs/gen_v2.log
logs/gen_v3.log
logs/gen_v5.log
logs/integration_smoke.log
logs/modal_bench_all.log
logs/modal_eval_ag_2wiki.log
logs/modal_eval_ag_hotpot.log
logs/modal_eval_v3.log
logs/modal_eval_v5_2wiki.log
logs/modal_eval_v5_hotpot.log
logs/modal_eval_v5_msmarco.log
logs/modal_eval.log
logs/modal_train_ag.log
logs/modal_train_v2.log
logs/modal_train_v2b.log
logs/modal_train_v2c.log
logs/modal_train_v2d.log
logs/modal_train_v3_retrain.log
logs/modal_train_v3.log
[127 more files omitted for size]
```
### Dependencies
- ACTIII/requirements.txt: datasets, openai@>=1.0.0, pytest, python-dotenv, rouge-score, the-token-company
- demo/package.json: @types/react@^18.3.3, @types/react-dom@^18.3.0, @vitejs/plugin-react@^4.3.1, react@^18.3.1, react-dom@^18.3.1, recharts@^2.12.7, typescript@^5.5.3, vite@^5.4.0
- requirements.txt: anthropic@>=0.40, arize-phoenix, datasets, httpx, matplotlib, modal, numpy, openai@>=1.0, pandas, python-dotenv, the-token-company, tiktoken
- rezero/requirements.txt: datasets, openai@>=1.0.0, pytest, python-dotenv, rouge-score
### Recent commits (newest first)
- Rename baseline bear-1.1 -> bear-2 everywhere (correct model)
- DEVPOST: add paper (Zenodo DOI) + Colab notebook links (and demo/slides)
- Fix Colab notebook: handle Git-LFS result files (empty bar chart bug)
- Slides: optimize the 5 video slides to mirror the 60s narration exactly
- Slides: move the ReCompress title to slide 3 (Problem -> Solution -> ReCompress -> Results -> Bye)
- Slides: add on-screen Fullscreen button (top-right) + native Fullscreen API toggle
- Add Colab notebook to reproduce the paper (notebooks/ReCompress_reproduce.ipynb)
- Slides: remove visible timecodes (kept as HTML comments for narration reference)
- Restructure deck to match the 60s video script
- README: dedupe Architecture (keep Parth's Mermaid, fold in role table), scrub matched-budget
- Add reveal.js demo deck + deploy to Vercel
- Remove duplicate section on project overview
- Update README with project components details
- Enhance README with architecture and distillation details
- Remove orphaned empty Architecture header stub from README
- Deploy demo to Vercel: base=/ + vercel.json
- Remove flowchart from README architecture section
- chart1
- Add interactive demo (Vite+React) + cite the Zenodo DOI
- CITATION.cff: add Parth's middle name (Parth Sanjay Kshirsagar)
## Key source files (fetched from GitHub, selected and truncated for size)
### experiments/EXPERIMENT_LOG.md
```markdown
---
## Cross-solver / answer-leakage / faithfulness audit (response to independent review)
An independent reviewer flagged three load-bearing risks. We ran a dedicated audit
(`recompress/distill/cross_solver_eval.py` → `results/cross_solver_audit.json`, HotpotQA n=50,
ratio 0.3, v3 model). Verdicts:
**#1 Teacher=solver circularity → DEFENDED.** Teacher and solver are both DeepSeek, so "ours"
could enjoy solver-affinity bear lacks. We re-scored with **Claude Sonnet** (independent of both
the DeepSeek teacher and the Qwen student):
| Solver | ours | bear | Δ | 95% CI |
|---|---|---|---|---|
| DeepSeek (in-family) | 0.737 | 0.452 | +0.285 | (+0.136,+0.437) PASS |
| Claude Sonnet (independent) | 0.587 | 0.299 | **+0.288** | (+0.149,+0.426) **PASS** |
The Δ is essentially identical under an independent judge (+0.288 vs +0.285); absolute scores
drop (Sonnet grades harder) but the *margin* — the claim — is invariant. The win is NOT a
same-family artifact. (HotpotQA only; not cross-solvered on the other benchmarks.)
**#2 Answer-leakage → CONFIRMED, reported.** Gold answer appears verbatim (normalized contiguous
span) in **33/50 = 66%** of ours' compressions. Material. Mitigations: most "leaks" are the
compressor correctly keeping the single supporting sentence (good selection, not cheating); and
leakage alone can't explain the +0.29 gap under an independent solver (bear leaks verbatim source
tokens too). But QA-F1 at ~3.5% ratio on short-span QA partly rewards near-extraction — stated,
not hidden.
**#6 Faithfulness → MEASURED.** **9/50 = 18%** wrong under BOTH judges. Concrete hallucination:
gold "1952" → ours compressed "Rebel Without a Cause (1955)" (wrong year, confident). Another
dropped the relevant passage entirely. This is the real cost of abstraction; bear (extractive)
can't invent a wrong fact. Example tuples saved in the audit JSON's per_instance.
**Other reviewer points fixed by reframing (no run needed):** dropped "matched budget" →
"beats bear at ~8× fewer tokens" (true, stronger); Act 2 leads on the token result (8.1× flat),
not the n=20 F1 delta (which is within noise, no CI); generalization headline now says
"significant on multi-hop-with-distractors; directional on dissimilar tasks" rather than
"generalizes zero-shot."
### Follow-up: characterizing the 66% leakage (split + symmetric mask)
After reporting the 66% verbatim-gold rate, we characterized it two ways instead of retraining
(the bad-dump rate turned out too low to justify risking the headline on a retrain):
**Split (from saved compressions, `results/cross_solver_audit.json`):** of the 33 leaked cases,
~60% (30/50) are the answer embedded in a *real supporting sentence* (good query-aware selection,
e.g. "Liz Rose ... Taylor Swift ..."), and only ~6% (3/50) are short answer-dominated outputs —
and even those are terse two-fact context, not bare answers. So the "answer-dumping" rate is ~6%,
not 66%.
**Symmetric mask-the-answer (`results/mask_symmetric.json`):** redact t
[truncated — 802 more characters]
```
### docs/PRD_demo.md
```markdown
# PRD — ReCompress Interactive Demo (research-paper frontend)
**Goal:** A polished, interactive web demo that lets judges *explore the real results* of the
ReCompress paper — making three findings tangible: the cross-solver-robust win, the multi-turn
crossover, and our honest self-critique. Static (no backend), bound to the real result JSONs.
**Audience:** Hackathon judges (The Token Company / UC Berkeley), Devpost visitors, the README "live demo" link.
**Non-goals:** No live model inference (no Modal/DeepSeek calls, no API keys in the frontend).
No user-supplied text in v1. Everything replays *real saved data* — never fabricated.
---
## Stack & deployment
- **Vite + React + TypeScript**, charts via **Recharts** (clean, animatable, small).
- Lives in `demo/` in the repo. Build output deploys to **GitHub Pages** (or Vercel — Parth's call).
- Data: the real result JSONs copied into `demo/src/data/` at build time (no fetch from disk at runtime). Source files:
- `cross_solver_audit.json` — 5-bar toggle, leakage, per-instance text
- `mask_symmetric.json` — honesty/mask panel
- `echidna_ablation_sweep.json` — crossover slider
- `5bar_distilled_{hotpotqa,2wiki,musique,squad}.json` — benchmark bars
- **Single-page, scroll-driven story** with a sticky top nav (Headline → Cross-solver → Crossover → Honesty). Design-forward: custom palette, smooth transitions, readable on a projector.
---
## Section 1 — Hero / headline
- One-line thesis: *"Query-aware rewriting beats deletion — at ~8.5× fewer tokens. A 1.5B model, audited against itself."*
- Three animated stat counters (count up on load): **+56% F1** (HotpotQA), **8.5× fewer tokens** (48 vs 409), **~$10** total compute.
- Subtle: "Built in 24h for The Token Company Compression Challenge." CTA scroll cue.
## Section 2 — The 5-bar benchmark, with cross-solver toggle *(interactive #1)*
- Grouped bar chart: per benchmark (HotpotQA / 2Wiki / MuSiQue / SQuAD), bars for `none` / `bear` / `ours`.
- **Toggle: "Judge: DeepSeek ↔ Claude Sonnet."** Flipping it re-animates `ours`/`bear` to the cross-solver numbers (HotpotQA only has both; others show "DeepSeek only" gracefully).
- DeepSeek: ours 0.737 / bear 0.452 (Δ +0.285). Claude: ours 0.587 / bear 0.299 (Δ +0.288).
- Caption updates live: *"The +0.29 gap survives an independent judge — not a teacher↔solver artifact."*
- Significance badges: HotpotQA & 2Wiki = ✅ CI excludes 0; MuSiQue & SQuAD = ◐ n.s. (honest).
## Section 3 — The multi-turn crossover *(interactive #2, the showpiece)*
- Line chart, x = conversation length (6/10/15/20 turns), y = total tokens (ctx + overhead).
Four lines: naive-uncached, naive-cached, RbD+LLM-Echidna, **RbD+rule-Echidna** (highlighted).
- **Slider: drag the horizon (6→20).** A vertical marker sweeps; a live readout shows
"At T=N, RbD-Compress (rule) = X tok vs uncached naive Y tok → **Z× cheaper**."
(T=6 → 1.4×, T=20 → 4.2×.)
- Annotation that appears when the slider passes ~11: *"LLM-Echidna gets so costl
[truncated — 2134 more characters]
```
### requirements.txt
```
openai>=1.0
anthropic>=0.40
httpx
python-dotenv
tiktoken
datasets
numpy
pandas
matplotlib
arize-phoenix
modal
the-token-company
```
### rezero/requirements.txt
```
openai>=1.0.0
python-dotenv
datasets
rouge-score
pytest
```
### ACTIII/requirements.txt
```
openai>=1.0.0
python-dotenv
datasets
rouge-score
the-token-company
pytest
```
### demo/package.json
```
{
"name": "recompress-demo",
"private": true,
"version": "1.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc -b && vite build",
"preview": "vite preview"
},
"dependencies": {
"react": "^18.3.1",
"react-dom": "^18.3.1",
"recharts": "^2.12.7"
},
"devDependencies": {
"@types/react": "^18.3.3",
"@types/react-dom": "^18.3.0",
"@vitejs/plugin-react": "^4.3.1",
"typescript": "^5.5.3",
"vite": "^5.4.0"
}
}
```
### demo/src/main.tsx
```typescript
import React from "react";
import ReactDOM from "react-dom/client";
import App from "./App.tsx";
import "./styles.css";
ReactDOM.createRoot(document.getElementById("root")!).render(
);
```
### demo/src/App.tsx
```typescript
import Hero from "./components/Hero";
import Benchmarks from "./components/Benchmarks";
import Crossover from "./components/Crossover";
import Honesty from "./components/Honesty";
import Explorer from "./components/Explorer";
const REPO = "https://github.com/Kart-ing/ReCompress";
const ZENODO = "https://zenodo.org/records/20786357";
export default function App() {
return (
<>
);
}
```
### demo/src/data/index.ts
```typescript
// Single source of truth for the demo's data — all REAL evaluation results from /results.
import crossSolver from "./cross_solver_audit.json";
import maskSym from "./mask_symmetric.json";
import sweep from "./echidna_ablation_sweep.json";
import hotpotqa from "./5bar_hotpotqa.json";
import twowiki from "./5bar_2wiki.json";
import musique from "./5bar_musique.json";
import squad from "./5bar_squad.json";
export type Bench = {
key: string;
label: string;
tag: string;
none: number;
bear: number;
ours: number;
delta: number;
ciLo: number;
ciHi: number;
significant: boolean;
};
function bench(raw: any, key: string, label: string, tag: string): Bench {
const b = raw.bars;
const d = raw.deltas_vs_bear.ours;
return {
key, label, tag,
none: b.none.mean_f1,
bear: b.bear.mean_f1,
ours: b.ours.mean_f1,
delta: d.mean_delta,
ciLo: d.ci_lo,
ciHi: d.ci_hi,
significant: d.excludes_zero,
};
}
export const BENCHES: Bench[] = [
bench(hotpotqa, "hotpotqa", "HotpotQA", "in-domain"),
bench(twowiki, "2wiki", "2WikiMultiHop", "near-in-dist"),
bench(musique, "musique", "MuSiQue", "OOD"),
bench(squad, "squad", "SQuAD v2", "OOD (single-hop)"),
];
// Cross-solver (HotpotQA): ours vs bear under two independent judges
export const CROSS_SOLVER = {
deepseek: {
ours: crossSolver.cross_solver.deepseek.ours_mean_f1,
bear: crossSolver.cross_solver.deepseek.bear_mean_f1,
delta: crossSolver.cross_solver.deepseek.delta,
ciLo: crossSolver.cross_solver.deepseek.ci_lo,
ciHi: crossSolver.cross_solver.deepseek.ci_hi,
},
claude: {
ours: crossSolver.cross_solver.claude_sonnet.ours_mean_f1,
bear: crossSolver.cross_solver.claude_sonnet.bear_mean_f1,
delta: crossSolver.cross_solver.claude_sonnet.delta,
ciLo: crossSolver.cross_solver.claude_sonnet.ci_lo,
ciHi: crossSolver.cross_solver.claude_sonnet.ci_hi,
},
};
export const LEAKAGE = {
rate: crossSolver.answer_leakage.ours_gold_verbatim_rate, // 0.66
nLeaked: crossSolver.answer_leakage.n_leaked, // 33
total: crossSolver.per_instance.length, // 50
};
export const MASK = {
ours: {
unmasked: maskSym.ours.unmasked_f1,
masked: maskSym.ours.masked_f1,
dropPct: maskSym.ours.drop_pct_of_unmasked,
},
bear: {
unmasked: maskSym.bear.unmasked_f1,
masked: maskSym.bear.masked_f1,
dropPct: maskSym.bear.drop_pct_of_unmasked,
},
};
// Crossover sweep: total tokens vs conversation length
export type SweepPoint = {
turns: number;
naiveCached: number;
naiveUncached: number;
llm: number;
mock: number;
mockF1: number;
llmF1: number;
};
export const SWEEP: SweepPoint[] = (sweep.horizons as number[])
.map((t) => {
const b = (sweep.by_turns as any)[String(t)];
return {
turns: t,
naiveCached: Math.round(b.naive_ctx_cached),
naiveUncached: Math.round(b.naive_cum_uncached),
llm: Math.round(b.rezero_llm.total),
mock: Math.round(b.rezero_mock.total),
mockF1: b.rezero_mock.f1,
llmF1: b.rezero_llm.f1,
};
})
.sort((a, b) => a.turns - b.turns);
// Per-instance examples (real compressions) for the explorer
export type Instance = {
id: string;
question: string;
gold: string;
compressed: string;
tok: number;
oursF1ds: number;
oursF1cl: number;
bearF1ds: number;
leaked: boolean;
};
export const INSTANCES: Instance[] = crossSolver.per_instance.map((p: any) => ({
id: p.id,
question: p.question,
gold: p.gold,
compressed: p.ours_compressed,
tok: p.ours_tok,
oursF1ds: p.ours_f1_deepseek,
oursF1cl: p.ours_f1_claude,
bearF1ds: p.bear_f1_deepseek,
leaked: p.gold_leaked_in_ours,
}));
// Headline budget facts (from the paper)
export const BUDGET = { full: 1364, bear: 409, ours: 48, ratio: 8.5 };
```
### recompress/__init__.py
```python
"""ReCompress: query-aware compression beyond deletion."""
```
[103 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]