# Project export: Winnow
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: UC Berkeley AI Hackathon 2026
- Tagline: Winnow turns your voice into the cheapest, sharpest LLM context you've ever paid for. Speak, and we transcribe, prune, and pipe; losing words, keeping meaning.
- Devpost: https://devpost.com/software/winnow-dzb281
- GitHub: https://github.com/TC960/winnow
- Video: https://www.youtube.com/embed/Sv02GeJrh5I?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 4 GitHub contributor(s) — Swastik Agrawal (16 commits), Claude Opus 4.8 (1M context) (11 commits), lakshgoyal06-eng (4 commits), Mohak Akul Prakash (3 commits)
## Devpost submission (written by the team)
### Inspiration
Tokens are the meter on every LLM bill, and almost all of them are waste. Spoken language, retrieved RAG context, and long documents are massively redundant — but the redundancy lives in different places. Some is low-information tokens ("um, so basically the idea is…"), some is irrelevant passages that a query never touches, and some is the sheer bit-width of the KV cache the model carries while it generates. No single paper kills all three. So instead of picking one compression technique and hoping, we built Winnow: a pipeline that runs several state-of-the-art compressors in parallel, merges their decisions, and then — optionally — hands the result to an LLM whose KV cache is itself compressed. Every stage is independently justified by a recent paper; the contribution is making them compose. The Product: showcasing use cases of compression in "Test" -> users can upload multiple PDFs or talk and witness our hard-token pruning themselves. playground: tweak and play with our compression algorithm and compare and find the best one that works for your use case. There's plenty of variety, idk how we managed to get all these methods in and working in 24 hours lol.
### What it does
(compression-specific) We tried our best to include every strong, parallelizable and patchable compression method we could find — and where no public repo existed, we re-implemented the paper ourselves from scratch (realistically, only LLMLingua shipped usable code). The papers Winnow builds on: LLMLingua-2 — Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — arXiv:2403.12968 · repo LongLLMLingua — Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — arXiv:2310.06839 · repo AttentionRAG — Attention-Guided Context Pruning in Retrieval-Augmented Generation — arXiv:2503.10720 (no public repo — self-implemented) LCLM — End-to-End Context Compression at Scale — arXiv:2606.09659 · repo TurboQuant — Online Vector Quantization with Near-Optimal Distortion Rate (KV-cache compression) — arXiv:2504.19874 · reference impl CompactPrompt — A Unified Pipeline for Prompt & Data Compression in LLM Workflows — arXiv:2510.18043 (explored; no public dataset to reproduce its style metric) Winnow compresses a prompt along two orthogonal axes and lets the user choose how far to push each one. 1. Token-space compression (which words survive). Given a piece of text, Winnow fans it out to two independent compressor families at once: LLMLingua-2 + LongLLMLingua. LLMLingua-2 is an extractive token classifier (a fine-tuned XLM-RoBERTa that labels each token keep/drop). LongLLMLingua adds the query-aware arm: a BGE cross-encoder reranker does coarse, question-aware document selection and reordering (to fight "lost-in-the-middle"), then the causal perplexity path compresses the survivors conditioned on the question. AttentionRAG. A faithful implementation of Attention-Guided Context Pruning in RAG (arXiv:2503.10720). It reformulates the query into an incomplete-answer template with a single "focal" blank, runs next-token prediction over each context chunk, and uses the focal token's summed-over-all-layers attention to rank tokens — keeping the sentences that hold the top-k. Chunks the model answers none on are dropped entirely. This requires a question to be run only when we are querying context (users are invited to try this in the Learn tab). Both arms produce per-token keep decisions. Winnow then merges them token-by-token over the original text, preserving chronology, under a boolean rule the user picks: intersection (keep a word only if both methods kept it — maximally aggressive) or union (keep if either did — safer, recall-oriented). The merge is a true alignment to one canonical token sequence (LLMLingua-2's word labels), not a fragile string-membership test, with a fallback to LLMLingua-only if AttentionRAG gates everything out so we never wipe the text. 2. Model-space compression (how the answer is generated). Once Winnow has the compressed string, the user chooses where it goes: Black box. Send the compressed prompt straight to a hosted API — Anthropic Claude or OpenAI GPT, picked per request. The savings are immediate and provider-agnostic, because the compressed prompt is just text. Self-hosted + TurboQuant. Route to our own Qwen model on Modal, generating with a TurboQuant-compressed KV cache (Google Research, ICLR 2026, arXiv:2504.19874) — a data-free random-rotation quantizer that drops the KV cache to ~4 bits (3–4× smaller) as a DynamicCache subclass, validated on Mistral-7B and Qwen2.5-14B. Self-hosted + LCLM + TurboQuant. Route to LCLM (End-to-End Context Compression at Scale, latent-context's released encoder→adapter→decoder checkpoints, ~2 weeks old at build time). LCLM compresses the long context into a handful of latent soft tokens that the decoder consumes as input embeddings — shrinking the KV cache's sequence length. TurboQuant then compresses the bits per entry of that same decoder cache. The two are orthogonal, so the savings multiply: fewer KV entries × fewer bits each. The net effect: token-level pruning before the model, sequence-length and bit-width compression inside it — three independent papers stacked end to end.
### How we built it
Compression workers on Modal GPUs. LLMLingua-2/LongLLMLingua, AttentionRAG, TurboQuant, and LCLM+TurboQuant each run as a warm Modal A100 worker, tied to a FastAPI server's lifecycle so models load once at startup and real requests pay no cold-start cost. /compress fans LLMLingua and AttentionRAG out concurrently and merges; /generate routes between the Qwen-TurboQuant and LCLM-TurboQuant workers on an lclm flag. The merge engine (token_merge.py) is pure Python: it normalizes LLMLingua's labels, reconstructs each word's char-span in the original by an in-order forward scan, tests AttentionRAG's kept sentence-spans by char overlap, applies the union/intersection rule, and splices survivors back together preserving original spacing. TurboQuant is a drop-in DynamicCache subclass — Lloyd-Max Gaussian quantization with an optional outlier-channel path — so it slots straight in as any HF model's past_key_values, including the LCLM decoder. Frontend is a Next.js app that shows the raw vs. compressed diff live, token counts, % and $ saved, and an A/B Q&A box that asks the same question against both prompts to prove meaning survived.
### Challenges we ran into
Pairing TurboQuant with LCLM. Getting TurboQuant's low-bit (down to ~3.5-bit outlier) quantization to drop cleanly into LCLM's decoder was the hardest integration. LCLM feeds the decoder inputs_embeds (the soft tokens), not token ids, and we had to confirm our TQCache survives that path and the encoder's single prefill without corrupting the latent memory block. Inventing the merge algorithm. Unionizing/intersecting two different compressors is not a set operation on strings — the two methods segment text differently. We had to align both to one canonical token sequence and operate on char-spans so the merge is order-preserving and unambiguous even with repeated words, plus a fallback so an empty AttentionRAG result never erases everything. Tuning LongLLMLingua. Choosing the right lead anchors / causal backbone and reranker for the question-aware arm took real iteration — what to condition on, how to reorder context against position bias, and where the causal path actually beats the cheaper extractive classifier. CompactPrompt had no public data. We tried to implement CompactPrompt's style-metric approach, but the paper shipped no public datasets to fit/evaluate the metric against, so we couldn't reproduce it faithfully and left it out rather than ship something unvalidated. Accomplishments we're proud of A working two-axis compressor: token-space (LLMLingua ∪/∩ AttentionRAG) and model-space (LCLM × TurboQuant) stacked in one pipeline. A genuinely novel merge between two heterogeneous compression algorithms that stays order-preserving and never destroys the text. TurboQuant validated end-to-end (3–4× KV compression, output matching FP16) and composing with LCLM so the KV savings multiply.
### What we learned
The big wins come from compressing along different axes and stacking them — token count, sequence length, and bit-width are independent levers. Heterogeneous methods need a common canonical representation before you can combine them; alignment, not set math, is the real work. Faithfully reproducing a paper is gated by its artifacts: no public dataset (CompactPrompt) effectively means no faithful reimplementation.
### What's next
Learned (rather than user-picked) routing between intersection/union and between black-box vs. self-hosted, per prompt and budget. Custom CUDA kernels for TurboQuant to turn the memory win into a latency win. Revisiting CompactPrompt if/when evaluation data becomes available. Built with Modal · FastAPI · Next.js · LLMLingua-2 · LongLLMLingua · AttentionRAG (arXiv:2503.10720) · LCLM (End-to-End Context Compression at Scale) · TurboQuant (arXiv:2504.19874) · BGE rerankers · Qwen · Anthropic Claude · OpenAI GPT · Deepgram
## README (from the GitHub repository)
# Winnow
> Voice-first, real-time token compression for LLM pipelines.
> Speak, and we transcribe, prune, and pipe — losing words, keeping meaning.
Winnow is a hackathon project that compresses live speech transcripts in real time before they reach an LLM, demonstrably without losing meaning. Spoken language is verbose; LLM input tokens cost money. Winnow shows you the savings, the diff, and a side-by-side fidelity audit, then lets you talk to the compressed transcript through a Claude-Projects-style workspace.
---
## What it does
Two views, one pipeline.
### Compare
Watch compression happen live. Raw transcript on the left as you speak, pruned transcript on the right within milliseconds of each pause. Pruned words are struck through. A stats bar tracks running token counts, % saved, $ saved per the chosen LLM, average pipeline latency, and a sparkline of compression ratio over time. A built-in A/B Q&A box asks the same question against both transcripts in parallel — when the answers match, you have proof compression preserved the buried details.
### Learn
A voice-first Claude-Projects-style workspace built around the compressed transcript. Tap the mic, talk naturally, each pause auto-sends a chat message. Claude streams answers back. One-tap **Insights** generate a summary, decision log, action items, flashcards (flippable carousel), and glossary — all backed by Anthropic prompt caching so multi-turn sessions stay cheap.
### Stage-safety features
- **Swappable input source.** Live mic and a pre-recorded JSON fixture both implement the same `TranscriptSource` interface. One click swaps them; downstream code can't tell which is active. If the live mic ever fails on stage, the fallback is a literal replay of you doing the demo, with original utterance timing.
- **In-app fixture recorder.** Capture a live session, download as JSON, drop into `public/fixtures/demo-transcript.json` — and your fallback becomes you.
- **Director mode.** Keyboard shortcuts (`Space` start/stop, `R` swap source, `1/2/3` fire preset Q&A probes, `X` clear, `?` show overlay) let you drive the demo without touching a trackpad.
---
## Compression pipeline
Winnow compresses along two orthogonal axes: **token-space** (which words survive,
before the model) and **model-space** (sequence length + KV-cache bit-width, inside
the model). The same input is fanned out to several papers in parallel, their
keep-decisions are merged token-by-token, and the user picks how the compressed
prompt is finally answered.
```mermaid
flowchart TD
A[Raw text / transcript] --> B[Compression stage]
subgraph PAR[Parallel compressors over the same input]
direction LR
C[LLMLingua-2 extractive token classifier]
D[LongLLMLingua query-aware perplexity + BGE reranker]
E[AttentionRAG attention-guided sentence pruning]
end
B --> C
B --> D
B --> E
C --> F[LLMLingua keep-mask]
D --> F
E --> G[AttentionRAG keep-spans]
F --> H{Token-by-token merge over original text}
G --> H
H -->|intersection: both kept| I[Compressed prompt]
H -->|union: either kept| I
I --> J{User routing}
J -->|Black box| K[Anthropic Claude / OpenAI GPT API]
J -->|Self-hosted + TurboQuant| L[Qwen + TurboQuant KV-cache bit quantization]
J -->|Self-hosted + LCLM| M[LCLM encoder→decoder soft tokens + TurboQuant on decoder KV]
K --> N[Answer]
L --> N
M --> N
```
**Token-space (which words survive).** The text fans out to two compressor
families at once — LLMLingua-2 (+ LongLLMLingua's query-aware reranker/perplexity
arm) and AttentionRAG (attention-guided sentence pruning). Both produce per-token
keep decisions, which Winnow merges token-by-token over the *original* text under a
user-picked boolean rule: **intersection** (keep only if both kept — aggressive) or
**union** (keep if either kept — recall-safe). The merge aligns both methods to one
canonical token sequence rather than doing string membership, and falls back to
LLMLingua-only if AttentionRAG gates everything out.
**Model-space (how it's answered).** The compressed prompt then goes either to a
**black-box API** (Claude / GPT, provider-agnostic), or to our **self-hosted**
workers: **Qwen + TurboQuant** (KV-cache quantized to ~4 bits, 3–4× smaller via a
data-free random-rotation `DynamicCache`), or **LCLM + TurboQuant** — LCLM
compresses the context into a few latent soft tokens (fewer KV *entries*) while
TurboQuant compresses the *bits per entry* of the same decoder cache, so the savings
multiply.
> **Papers:** LLMLingua-2 · LongLLMLingua · AttentionRAG (arXiv:2503.10720) · LCLM /
> End-to-End Context Compression at Scale · TurboQuant (arXiv:2504.19874). We also
> explored CompactPrompt's style-metric approach but it shipped no public dataset to
> reproduce against. A full project write-up lives in [`DEVPOST.md`](DEVPOST.md).
---
## Architecture
```
┌──────────────────────────┐
│ Browser (Next.js 15) │
│ │
┌──────────────┤ • Compare tab │
│ getUserMedia │ • Learn tab (voice) │
│ │ • Zustand store │
│ └────────┬──────────┬──────┘
│ │ │
│ WebSocket │ /api/ │ /api/project-chat
│ (opus chunks) │ compress │ /api/project-action
▼ ▼ ▼ /api/qa
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Deepgram │ │ FastAPI │ │ Anthropic │
│ Nova-3 │ │ (server.py) │ │ Claude │
│ streaming │ │ │ │ (streaming) │
└──────────────┘ └──────┬───────┘ └──────────────┘
│ modal RPC
▼
┌──────────────┐
│ Modal │
│ (T4 GPU) │
│ + LLMLingua │
│ -2 XLM │
└──────────────┘
```
- The **browser** opens its own WebSocket to Deepgram (auth via `/api/deepgram-token`) and pushes Opus-encoded audio chunks.
- Each `speech_final` utterance fires a request to `/api/compress`, which proxies to the local FastAPI.
- FastAPI calls the **Modal** worker over Modal's RPC. The worker hosts **LLMLingua-2** on a T4, with memory + GPU snapshots so cold starts are ~1 second.
- Q&A and Project chat go through Next.js API routes directly to **Anthropic** (Claude Sonnet 4.6 by default; pickable in the UI).
---
## Tech stack
**Frontend** (`web/`)
- TypeScript, Next.js 15 (App Router), React 18
- Tailwind CSS, Framer Motion, Radix UI primitives (Slider, Switch, Slot)
- Zustand for state, `gpt-tokenizer` for client-side token counting fallback
- `lucide-react`, `canvas-confetti`, `clsx + tailwind-merge`
**Speech**
- Deepgram Nova-3 streaming (interim results, `speech_final`, VAD, diarization, multilingual)
**Compression**
- LLMLingua-2 (`microsoft/llmlingua-2-xlm-roberta-large-meetingbank`)
- Hosted on Modal — T4 GPU, persistent HF cache volume, memory + GPU snapshots for ~1s cold start
- FastAPI / Pydantic / Uvicorn as a thin local proxy
**LLM**
- Anthropic Claude Sonnet 4.6 (default), Opus 4.7, Haiku 4.5; cost math also covers GPT-4o
- Streaming via SSE; prompt caching with `cache_control: ephemeral` on the sources block
---
## Local setup
Prerequisites:
- Python 3.11+
- Node.js 18+
- A Modal account (free tier works)
- A Deepgram API key
- An Anthropic API key
### 1. Set up the Python backend
```bash
# from repo root
python3 -m venv .venv
.venv/bin/pip install --upgrade pip
.venv/bin/pip install modal fastapi "uvicorn[standard]" pydantic
```
### 2. Authenticate Modal and deploy the GPU worker
```bash
.venv/bin/modal setup # one-time, opens a browser
.venv/bin/modal deploy llmlingua2_modal.py # deploys the Compressor class
```
The first deploy ta
[README truncated for size]
## Detected evidence (automated analysis)
Indexed codebase: 82 recognized source files, 525 KB.
- Anthropic (technology) — detected in the code
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- Hugging Face (technology) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- PyTorch (technology) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- FastAPI (technology) — claimed on Devpost, not found in the code
- Vercel (technology) — claimed on Devpost, not found in the code
- AI coding agent: Claude Code — evidence: commit authorship or trailers
## Codebase structure (from repository index)
### Files (101 of 101)
```
.gitignore
attentionrag/__init__.py
attentionrag/core.py
attentionrag/hf_backend.py
attentionrag/modal_app.py
attentionrag/prompts.py
attentionrag/README.md
attentionrag/requirements.txt
attentionrag/test_core.py
DEVPOST.md
downstream.py
experiments/bench/bench_llm_modal.py
experiments/bench/build_data.py
experiments/bench/compress_devpost.py
experiments/bench/compressed.json
experiments/bench/data.json
experiments/bench/lclm_answers.json
experiments/bench/llm_answers.json
experiments/bench/REPORT.md
experiments/bench/run_compress.py
experiments/bench/run_lclm.py
experiments/bench/run_llm.py
experiments/bench/SCHEMA.md
experiments/bench/score.py
experiments/eval_modal.py
experiments/eval_report.md
experiments/eval_sets.py
experiments/lclm_tq_timing_report.md
experiments/lclm_tq_timing_results.json
experiments/lclm_tq_timing.py
experiments/README.md
experiments/run_arms.py
experiments/run_eval.py
experiments/run_lclm_timing.py
experiments/score_eval.py
experiments/scratchpad.ipynb
experiments/test_data.py
lclm_worker_modal.py
llmlingua2_modal.py
README.md
researchpaper_ss
run.sh
server.py
test_token_merge.py
token_merge.py
turboquant_modal.py
turboquant-poc/.gitignore
turboquant-poc/lclm_modal.py
turboquant-poc/logs/run_14b_35bit.log
turboquant-poc/logs/run_14b_4bit.log
turboquant-poc/modal_app.py
turboquant-poc/README.md
turboquant-poc/turboquant_poc.py
two_stage_compressor.py
warmup.py
web/.gitignore
web/app/api/compress/route.ts
web/app/api/deepgram-token/route.ts
web/app/api/downstream/route.ts
web/app/api/extract-pdf/route.ts
web/app/api/playground/route.ts
web/app/api/qa/route.ts
web/app/globals.css
web/app/layout.tsx
web/app/page.tsx
web/components/AnimatedNumber.tsx
web/components/CompareView.tsx
web/components/CompressedColumn.tsx
web/components/DirectorOverlay.tsx
web/components/LanguagePicker.tsx
web/components/ModelPicker.tsx
web/components/PlaygroundPanel.tsx
web/components/QABox.tsx
web/components/RateSlider.tsx
web/components/SourceToggle.tsx
web/components/Sparkline.tsx
web/components/StatsBar.tsx
web/components/Tabs.tsx
web/components/test/DocumentCard.tsx
web/components/test/PipelineStrip.tsx
web/components/test/SpeakCard.tsx
web/components/test/StrikeText.tsx
web/components/TestView.tsx
web/components/TranscriptColumn.tsx
web/lib/cn.ts
web/lib/pipeline.ts
web/lib/playground.ts
web/lib/sources/index.ts
web/lib/sources/live-mic.ts
web/lib/sources/recorded.ts
web/lib/sources/types.ts
web/lib/store.ts
web/lib/tokens.ts
web/next.config.mjs
web/package.json
web/postcss.config.mjs
web/public/fixtures/demo-transcript.json
web/tailwind.config.ts
web/tsconfig.json
web/tsconfig.tsbuildinfo
winnow-deck/winnow.html
```
### Dependencies
- attentionrag/requirements.txt: accelerate, hf_transfer, huggingface_hub, modal, protobuf, sentencepiece, torch, transformers@>=4.44
- web/package.json: @anthropic-ai/sdk@^0.40.0, @radix-ui/react-slider@^1.2.1, @radix-ui/react-slot@^1.1.0, @radix-ui/react-switch@^1.1.1, @types/canvas-confetti@^1.6.4, @types/node@^22, @types/react@^18.3.12, @types/react-dom@^18.3.1, autoprefixer@^10.4.20, canvas-confetti@^1.9.3, clsx@^2.1.1, framer-motion@^11.11.0, gpt-tokenizer@^2.5.1, lucide-react@^0.460.0, next@15.0.3, postcss@^8.4.49, react@^18.3.1, react-dom@^18.3.1, tailwind-merge@^2.5.4, tailwindcss@^3.4.15, typescript@^5.6.3, unpdf@^1.6.2, zustand@^5.0.1
### Recent commits (newest first)
- we want better slides :angy:
- just slides
- fixed union token explosion
- Merge branch 'main' of https://github.com/TC960/winnow
- Create researchpaper_ss
- Merge branch 'main' of https://github.com/TC960/winnow
- final
- Merge branch 'main' of github.com:TC960/winnow
- exp
- Test tab v2: question-aware DocumentCard, speak-and-ask SpeakCard
- Add playground: unified /playground endpoint + two-panel Compare UI
- Rename Learn → Test; rebuild as two compress-and-download flows
- adding option to upload docs
- merged :)
- AttentionRAG: don't drop single-chunk (short) input on a none anchor/hint
- wiring UI to backend
- AttentionRAG: load OPENAI_API_KEY from Modal openai-secret
- Merge worktree-attentionrag-impl into main
- add downstream blackbox llm
- AttentionRAG + token-merge compression, A100 for LLMLingua worker
## Key source files (fetched from GitHub, selected and truncated for size)
### DEVPOST.md
```markdown
# Winnow — multi-path prompt compression for LLM pipelines
## Inspiration
Tokens are the meter on every LLM bill, and almost all of them are waste.
Spoken language, retrieved RAG context, and long documents are massively
redundant — but the redundancy lives in different places. Some is *low-information
tokens* ("um, so basically the idea is…"), some is *irrelevant passages* that a
query never touches, and some is the sheer *bit-width of the KV cache* the model
carries while it generates. No single paper kills all three.
So instead of picking one compression technique and hoping, we built Winnow: a
pipeline that runs several state-of-the-art compressors **in parallel**, merges
their decisions, and then — optionally — hands the result to an LLM whose KV
cache is itself compressed. Every stage is independently justified by a recent
paper; the contribution is making them *compose*.
## What it does
We tried our best to include every strong, **parallelizable and patchable**
compression method we could find — and where no public repo existed, we
re-implemented the paper ourselves from scratch (realistically, only LLMLingua
shipped usable code). The papers Winnow builds on:
- **LLMLingua-2** — Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression — [arXiv:2403.12968](https://arxiv.org/abs/2403.12968) · [repo](https://github.com/microsoft/LLMLingua)
- **LongLLMLingua** — Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression — [arXiv:2310.06839](https://arxiv.org/abs/2310.06839) · [repo](https://github.com/microsoft/LLMLingua)
- **AttentionRAG** — Attention-Guided Context Pruning in Retrieval-Augmented Generation — [arXiv:2503.10720](https://arxiv.org/abs/2503.10720) *(no public repo — self-implemented)*
- **LCLM** — End-to-End Context Compression at Scale — [arXiv:2606.09659](https://arxiv.org/abs/2606.09659) · [repo](https://github.com/LeonLixyz/LCLM)
- **TurboQuant** — Online Vector Quantization with Near-Optimal Distortion Rate (KV-cache compression) — [arXiv:2504.19874](https://arxiv.org/abs/2504.19874) · [reference impl](https://github.com/OmarHory/turboquant)
- **CompactPrompt** — A Unified Pipeline for Prompt & Data Compression in LLM Workflows — [arXiv:2510.18043](https://arxiv.org/abs/2510.18043) *(explored; no public dataset to reproduce its style metric)*
Winnow compresses a prompt along two orthogonal axes and lets the user choose how
far to push each one.
**1. Token-space compression (which words survive).**
Given a piece of text, Winnow fans it out to two independent compressor families
at once:
- **LLMLingua-2 + LongLLMLingua.** LLMLingua-2 is an extractive token classifier
(a fine-tuned XLM-RoBERTa that labels each token keep/drop). LongLLMLingua adds
the query-aware arm: a BGE cross-encoder reranker does coarse, question-aware
document selection and reordering (to fight "lost-in-the-middle"), then the
causal perplexity path compresses the survivors conditioned on the ques
[truncated — 6438 more characters]
```
### experiments/lclm_tq_timing_report.md
```markdown
# LCLM + TurboQuant — timing & memory benchmark
- LCLM checkpoint: `latent-context/0.6b-4b-LCLM-16x` (encoder Qwen3-Embedding-0.6B → adapter → decoder Qwen3-4B-Instruct-2507; 36 layers, 8 KV heads, head_dim=128)
- GPU: **A100-80GB**, bf16 weights | decode: greedy (`do_sample=False`)
- Two **isolated** warm Modal containers: `LCLMVanilla` (fp16 `DynamicCache`) vs `LCLMTurboQuant` (`TQCache`)
- TQ configs: **4-bit**, **3-bit**, **3.5-bit** (`bit_width=3` + 32 outlier channels @ 4-bit)
- TTFT = prefill + first decoded token (separate `max_new_tokens=1` generate). `kv_bytes` = real packed quantized cache size (TQ) or fp16 cache size (vanilla). `KV ×` = fp16-equivalent ÷ TQ packed bytes for the same decoder sequence length.
The LCLM encoder compresses the input doc into a small number of latent soft
tokens (the decoder's effective prompt). With the 16x checkpoint, ~16 input
tokens → 1 latent, so a 16k-token doc becomes ~1000 latent tokens — that's the
decoder sequence length that drives KV-cache size.
> **Honest trade-off.** TurboQuant here is **pure-PyTorch dequant with no custom
> CUDA kernel**: every decode step de-quantizes the whole cache, so wall-clock
> decode is **slower** than fp16. The payoff is **memory** — KV-cache bytes drop
> ~3.8–4.9×. This matches the paper, which needs custom kernels to also win on
> speed. We report both, plainly: **TQ = slower tok/s, smaller KV, lower peak GPU.**
## Headline — per context length (decode = 512 tokens)
KV bytes are the actual decoder KV-cache size at end of generation (MB). Peak GPU
is `torch.cuda.max_memory_allocated`. Decode tok/s excludes the prefill/TTFT.
| ctx (input tok) | latents | arm | TTFT (s) | decode tok/s | peak GPU (MB) | KV (MB) | KV × | eff bits | needle |
|---|---|---|---|---|---|---|---|---|---|
| ~2k (2018) | 127 | vanilla fp16 | 0.118 | 23.75 | 9613 | 104.0 | 1.0× | 16.0 | ✅ |
| | | TQ-4bit | 0.142 | 16.68 | 9657 | 27.6 | 3.77× | 4.0 | ✅ |
| | | TQ-3bit | 0.155 | 16.64 | 9657 | 21.1 | 4.93× | 3.0 | ❌ |
| | | TQ-3.5bit | 0.193 | 9.73 | 9657 | 24.4 | 4.27× | 3.25 | ✅ |
| ~8k (8007) | 501 | vanilla fp16 | 0.217 | 26.55 | 10495 | 159.1 | 1.0× | 16.0 | ✅ |
| | | TQ-4bit | 0.273 | 16.85 | 10633 | 42.3 | 3.77× | 4.0 | ✅ |
| | | TQ-3bit | 0.238 | 16.42 | 10633 | 32.3 | 4.93× | 3.0 | ✅ |
| | | TQ-3.5bit | 0.319 | 9.89 | 10644 | 37.3 | 4.27× | 3.25 | ✅ |
| ~16k (16000) | 1000 | vanilla fp16 | 0.347 | 25.35 | 11670 | 232.7 | 1.0× | 16.0 | ✅ |
| | | TQ-4bit | 0.359 | 16.79 | 11916 | 61.8 | 3.77× | 4.0 | ✅ |
| | | TQ-3bit | 0.377 | 16.69 | 11916 | 42.1 | 4.93× | 3.0 | ✅ |
| | | TQ-3.5bit | 0.438 | 9.97 | 11914 | 54.5 | 4.27× | 3.25 | ✅ |
## Short decode (decode = 128 tokens)
| ctx | arm | TTFT (s) | decode tok/s | peak GPU (MB) | KV (MB) | KV × | needle |
|---|---|---|---|---|---|---|---|
| ~2k | vanilla fp16 | 0.530 | 27.54 | 9613 | 47.3 | 1.0× | ✅ |
| | TQ-4bit | 0.619 | 17.39 | 9657 | 12.6 | 3.78× | ✅ |
| | TQ-3bit | 0.140 | 16.80 | 9657 | 9.6 | 4.94× | ❌ |
| | TQ-3.5bit | 0.219 | 9.
[truncated — 3572 more characters]
```
### attentionrag/requirements.txt
```
# Core (selection logic + tests) needs only the stdlib.
# The HF backend / Modal run needs the following (installed inside the Modal image):
torch
transformers>=4.44
accelerate
huggingface_hub
hf_transfer
sentencepiece
protobuf
modal
# optional: faithful hint-prefix authoring via GPT-4o Mini (set OPENAI_API_KEY)
# openai
```
### web/package.json
```
{
"name": "winnow-web",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev -p 3000",
"build": "next build",
"start": "next start -p 3000",
"lint": "next lint"
},
"dependencies": {
"@anthropic-ai/sdk": "^0.40.0",
"@radix-ui/react-slider": "^1.2.1",
"@radix-ui/react-slot": "^1.1.0",
"@radix-ui/react-switch": "^1.1.1",
"canvas-confetti": "^1.9.3",
"clsx": "^2.1.1",
"framer-motion": "^11.11.0",
"gpt-tokenizer": "^2.5.1",
"lucide-react": "^0.460.0",
"next": "15.0.3",
"react": "^18.3.1",
"react-dom": "^18.3.1",
"tailwind-merge": "^2.5.4",
"unpdf": "^1.6.2",
"zustand": "^5.0.1"
},
"devDependencies": {
"@types/canvas-confetti": "^1.6.4",
"@types/node": "^22",
"@types/react": "^18.3.12",
"@types/react-dom": "^18.3.1",
"autoprefixer": "^10.4.20",
"postcss": "^8.4.49",
"tailwindcss": "^3.4.15",
"typescript": "^5.6.3"
}
}
```
### server.py
```python
"""
FastAPI server in front of the Modal compression + generation workers.
All GPU workers are tied to THIS server's lifecycle (identical mechanism):
* on startup -> each Modal app starts (`app.run()`) and its container
cold-starts and loads its model (we warm each once), so real
requests have no model-startup cost.
* on shutdown -> the Modal apps stop, tearing down their GPU containers
immediately (GPUs released; no idle lingering).
Workers:
* LLMLingua-2 (Compressor) -> /compress, /compress_rag (A100)
* AttentionRAG (AttentionRAGService) -> /compress (when `question` set; A100)
* TurboQuant (TurboQuantModel) -> /generate (default route) (A100-80GB)
* LCLM+TurboQuant (LCLMTurboQuantModel) -> /generate (lclm=true) (A100-80GB)
/compress picks behavior by the request's `question`:
* question empty (default) -> LLMLingua-2 token compression only (back-compat).
* question set -> run LLMLingua-2 AND AttentionRAG in parallel and
MERGE the two keep-decisions token-by-token over the
original text (intersection or union).
/generate picks a worker by the request's `lclm` flag:
* lclm=false (default) -> Qwen TurboQuant route (KV-cache bit quantization).
* lclm=true -> LCLM (encoder-decoder context compression) + TurboQuant
on the decoder. Pass the long context in `context`; it
is compressed into latent soft tokens, while `prompt`
(the question/instruction) stays verbatim.
Prereqs:
pip install fastapi "uvicorn[standard]" pydantic modal torch transformers ...
Run (no --reload: the lifespan owns the Modal apps; reload would double-start them):
uvicorn server:app --port 8000
Try it (Qwen TurboQuant route, default):
curl -X POST http://localhost:8000/generate \
-H "Content-Type: application/json" \
-d '{"prompt": "Explain KV-cache quantization.", "bit_width": 4, "max_new_tokens": 120}'
Try it (LCLM + TurboQuant route):
curl -X POST http://localhost:8000/generate \
-H "Content-Type: application/json" \
-d '{"lclm": true, "prompt": "What is the calibration passphrase?", "context": "your long document with a planted fact ...", "bit_width": 4, "max_new_tokens": 120}'
# plain token-level compression of one blob of text:
curl -X POST http://localhost:8000/compress \
-H "Content-Type: application/json" \
-d '{"text": "your long text here ...", "rate": 0.5}'
# question-aware: LLMLingua + AttentionRAG merged over the original text:
curl -X POST http://localhost:8000/compress \
-H "Content-Type: application/json" \
-d '{"text": "your long text ...", "question": "What is X?", "mode": "intersection", "return_labels": true}'
"""
import asyncio
import time
from contextlib import ExitStack, asynccontextmanager
from typing import List, Optional
import modal
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
# Import the worker app modules so we can run them ephemerally, bound to this
# process. (Module import is light: heavy deps like torch are imported lazily
# inside the Modal methods, not at module top-level.)
import attentionrag.modal_app as attentionrag_modal
import lclm_worker_modal
import llmlingua2_modal
import turboquant_modal
# Token-by-token merge of LLMLingua + AttentionRAG keep-decisions (pure-python).
from token_merge import merge_compress, normalize_labels
# Black-box downstream LLM callers (Claude / ChatGPT) reused for the playground.
from downstream import (
DEFAULT_MODEL as BLACKBOX_DEFAULT_MODEL,
_call_claude,
_call_openai,
_canonical_provider,
_key_for,
)
# Worker handles, populated in the lifespan once the Modal apps are running.
compressor = None
attn_service = None
turboquant = None
lclm = None
@asynccontextmanager
async def lifespan(_app: FastAPI):
"""Start all Modal GPU apps when the server boots; stop them when it exits.
`app.run()` starts an EPHEMERAL Modal app bound to this process: its
containers live only while this server lives. We warm each worker with one
tiny request so the model loads now (the one-time cold start happens here at
server startup, not on a user's first request). Closing the ExitStack on
shutdown stops the apps and releases their GPU containers immediately.
"""
global compressor, attn_service, turboquant, lclm
with ExitStack() as stack:
# NB: no modal.enable_output() — its rich live-display can't be shared
# across concurrent app.run() contexts (LiveError). Apps run quietly.
# Start all GPU apps, tied to this process (identical mechanism).
stack.enter_context(llmlingua2_modal.app.run())
stack.enter_context(attentionrag_modal.app.run())
stack.enter_context(turboquant_modal.app.run())
stack.enter_context(lclm_worker_modal.app.run())
compressor = llmlingua2_modal.Compressor()
attn_service = attentionrag_modal.AttentionRAGService()
turboquant = turboquant_modal.TurboQuantModel()
lclm = lclm_worker_modal.LCLMTurboQuantModel()
# Cold-start + load all models now, concurrently.
print("[startup] warming Modal workers (loading models on GPU)...", flush=True)
await asyncio.gather(
compressor.compress.remote.aio("warmup", rate=0.5),
attn_service.compress_spans.remote.aio("warmup", "warmup"),
turboquant.generate.remote.aio("warmup", max_new_tokens=1),
lclm.generate.remote.aio("warmup", max_new_tokens=1),
)
print("[startup] all workers warm; ready to serve.", flush=True)
yield # ----------------- server handles requests -----------------
# ExitStack closed -> all Modal apps stopped -> GPU containers torn down.
print("[
[truncated — 19987 more characters]
```
### web/app/layout.tsx
```typescript
import type { Metadata } from "next";
import "./globals.css";
export const metadata: Metadata = {
title: "Winnow — real-time speech compression",
description: "Voice → Deepgram → LLMLingua-2 → LLM. Strip tokens, keep meaning.",
};
export default function RootLayout({ children }: { children: React.ReactNode }) {
return (
{children}
);
}
```
### web/app/page.tsx
```typescript
"use client";
import { Wind } from "lucide-react";
import { useStore } from "@/lib/store";
import { Tabs } from "@/components/Tabs";
import { SourceToggle } from "@/components/SourceToggle";
import { CompareView } from "@/components/CompareView";
import { TestView } from "@/components/TestView";
import { AnimatePresence, motion } from "framer-motion";
export default function Page() {
const tab = useStore((s) => s.tab);
return (
{/* Top bar */}
winnow
v0.1
voice → deepgram → llmlingua-2 → llm
{/* Body — single AnimatePresence for tab transitions */}
{tab === "compare" ? (
) : (
)}
);
}
```
### web/lib/sources/index.ts
```typescript
"use client";
import { LiveMicSource } from "./live-mic";
import { RecordedSource } from "./recorded";
import type { SourceConfig, TranscriptSource } from "./types";
export type SourceKind = "live" | "recorded";
export function createSource(kind: SourceKind, cfg: SourceConfig = {}): TranscriptSource {
return kind === "live" ? new LiveMicSource(cfg) : new RecordedSource(cfg);
}
export type { TranscriptSource, SourceConfig } from "./types";
export type { Utterance, Word, SourceEvent } from "./types";
```
### web/app/api/playground/route.ts
```typescript
import { NextRequest, NextResponse } from "next/server";
// Thin proxy to the FastAPI /playground orchestrator (Layer 1 compression ->
// Layer 2 LLM). Keeps the backend URL + keys server-side; same pattern as
// app/api/compress/route.ts.
const BACKEND = process.env.COMPRESS_BACKEND_URL ?? "http://localhost:8000";
export const runtime = "nodejs";
export const maxDuration = 120;
export async function POST(req: NextRequest) {
const body = await req.json();
try {
const r = await fetch(`${BACKEND}/playground`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
});
const text = await r.text();
return new NextResponse(text, {
status: r.status,
headers: { "content-type": r.headers.get("content-type") ?? "application/json" },
});
} catch (e: any) {
return NextResponse.json({ error: `backend unreachable: ${e.message}` }, { status: 502 });
}
}
```
### web/app/api/downstream/route.ts
```typescript
import { NextRequest, NextResponse } from "next/server";
// Proxy to downstream.py (provider-agnostic blackbox LLM API on :8100).
// Keeps provider API keys server-side. Used by the Test-tab SpeakCard to send
// the LLMLingua-compressed prompt to the chosen blackbox model and get a
// non-streaming response back.
const DOWNSTREAM = process.env.DOWNSTREAM_URL ?? "http://localhost:8100";
export async function POST(req: NextRequest) {
const body = await req.json();
try {
const r = await fetch(`${DOWNSTREAM}/generate`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
});
const text = await r.text();
return new NextResponse(text, {
status: r.status,
headers: { "content-type": r.headers.get("content-type") ?? "application/json" },
});
} catch (e: any) {
return NextResponse.json({ error: `downstream unreachable: ${e.message}` }, { status: 502 });
}
}
```
[70 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]