# Project export: Fluently This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Fluently transcribes your reading in real time, aligns it to the target passage, and classifies every deviation against the syntactic structure of the text with pattern recognition. - Devpost: https://devpost.com/software/fluently-rm76lw - GitHub: https://github.com/appleorange/fluently - Video: https://www.youtube.com/embed/64zQlaumeMw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Sabella Han (37 commits), Claude Sonnet 4.6 (7 commits), shannon-xiao (7 commits) ## Devpost submission (written by the team) ### Inspiration Growing up, we had a family friend on the autism spectrum who struggled with reading and speaking aloud. We watched how much early intervention mattered for kids like him, and how much of that intervention depended on having access to the right specialists at the right time. That gap stuck with us. When we started building Fluently, we wanted to create something that could do what a speaking specialist does. We aren't aiming to replace the human connection, but instead, make that level of precision accessible to every family, regardless of income or location. The earlier a reading difficulty is caught and understood, the better the outcome. Fluently is built on that belief. ### What it does Fluently listens to your child read aloud and does what a reading/speaking specialist does, in real time. A child reads a passage out loud. Every word lights up on screen: green if correct, red if an error, yellow if they hesitated. At the end of 60 seconds, Fluently produces a report that doesn't just count mistakes but explains what they mean. Is this a decoding issue or a phrasing fluency issue? These have different causes and different interventions, and most reading tools can't tell the difference. A 2D PassageMap lets you navigate material by complexity and register, with Claude generating fresh passages on demand. After each session, Redis vector search finds the optimal next passage based on where errors were concentrated, moving the reader harder in exactly the right dimension. Over multiple sessions, Fluently builds a longitudinal model of the reader's error patterns, shifting from a snapshot of today to a genuine reading profile that gets sharper every session. ### How we built it Part 1: Deterministic pipeline The scoring engine is fully deterministic and AI-free. Deepgram Nova-3 streams word objects with start, duration, and confidence fields. A custom Levenshtein alignment function matches the transcript against the expected passage word-by-word, classifying each word as correct, substitution, omission, insertion, hesitation (\(> 500\text{ms}\) pause), or acoustically uncertain (confidence \(< 0.75\), excluded from all error metrics to avoid penalizing accent variation). A separate metrics pass computes WCPM benchmarked against DIBELS 8th Edition grade-level thresholds (Grade 2: \(\geq 125\), Grade 4: \(\geq 141\), Grade 6: \(\geq 135\), accuracy threshold \(\geq 96\%\)), pause placement using compromise.js to identify syntactic boundaries in the target text, and self-correction rate as its own positive signal separate from the error taxonomy. Part 2: AI layer Claude receives a structured JSON object (never raw audio or transcript) and returns a plain-language report calibrated to the reader's DIBELS tier (intensive: at risk, strategic: some risk, core: on track), the passage's register (informal passages don't penalize contractions or casual phrasing), and any persisting error patterns across prior sessions. The prompt runs in one of three modes: snapshot (first session), comparison (second session), or pattern-recognition (third session onward), each with distinct framing language. In pattern-recognition mode, Claude receives a full markdown table of historical metrics and is explicitly instructed to re-read every number from the table rather than recall from context. This eliminated hallucinated historical values during testing. Part 3: Redis AI integration Every session stores a 5-dimensional skill vector $$\mathbf{v} = [\text{complexityHandling},\ \text{registerHandling},\ \text{wcpmPercentile},\ \text{pausePlacementScore},\ \text{selfCorrectionRate}]$$ and full error metrics in Redis. After each session, computeNextTarget() identifies the weakest map-axis dimension (complexity or register only, since non-map dimensions like WCPM are addressed through Claude's exercise recommendations rather than passage movement) and computes the optimal next position in the 2D skill space. The target only escalates on an advance recommendation. On retry, the position stays fixed so the reader consolidates at the same level rather than compounding difficulty. A KNN search (FT.SEARCH) finds the nearest existing passage. If no close match exists within distance \(0.1\), Claude auto-generates a fresh passage at the exact target coordinates and stores it in Redis, growing the library organically with every session. The full reader history is fetched from Redis before every Claude call, enabling longitudinal pattern recognition across sessions. Part 4: PassageMap Instead of a grade picker, a draggable 2D SVG canvas lets users place a pin anywhere across the full K–adult complexity and casual–formal register space. Each pin placement calls Claude to generate a fresh ~70-word passage at those exact coordinates. After a session, a dashed blue arrow on the map shows where the recommendation moves the reader next, visually grounding the concept of "harder in the right dimension" in something a parent or child can immediately understand. ### Challenges we ran into Accent fairness A child who pronounces "th" as "d" should not have that counted as an error. We implemented confidence score filtering and switched to accent-agnostic English recognition, then found and fixed a subtle bug where uncertain words were excluded from error counts but still dragging down the accuracy denominator silently, penalizing the reader anyway through a different metric. Longitudinal prompt hallucination In early testing, Claude would misstate historical WCPM values when given prior session data as prose. Switching to a structured markdown table with an explicit instruction to re-read every number from the table rather than recall from context eliminated the issue entirely. Redis vector search projection When running KNN search against our passage index, the query was returning passage identifiers and titles as literal undefined strings. The issue was a subtle mismatch between which fields Redis indexes for search and which fields it actually returns in query results. ### Accomplishments we're proud of A fully deterministic scoring pipeline where Claude interprets but never detects Accent fairness built into the confidence filtering layer so no child is penalized for how they speak Redis powering genuine vector search across a 2D pedagogical skill space, not just caching Longitudinal error tracking that shifts Claude's diagnostic language after three sessions ### What we learned The most powerful thing you can do with an LLM is constrain what it has to guess. Every time we moved a computation out of Claude and into a deterministic function, the output got more reliable and the AI layer got more useful. Real equity has to be designed into the architecture, not added as an afterthought. Accent fairness required deliberate decisions at the data layer, not just the UI. And longitudinal context changes everything: a system that remembers is fundamentally different from one that scores. ### What's next Expanding to ESL adult learners with register-specific passage sets calibrated to workplace and academic English Support for speech therapy use cases including fluency disorders, apraxia, and progressive speech conditions like Huntington's disease, where tracking subtle degradation in prosody and phrasing over time could serve as an early clinical signal Difficulty regression on retry: backing off in the weak dimension before re-attempting, which is what reading specialists actually do ## README (from the GitHub repository) # Fluently Oral reading fluency assessment tool. A child reads a passage aloud, Deepgram transcribes with word-level timestamps, a deterministic Levenshtein alignment pipeline scores the reading against the expected passage, and Claude generates a plain-language diagnostic report. Fluently also tracks a reader's skill profile across sessions in Redis (vector search over an AI-generated passage library) to recommend the next passage's difficulty and register, and to give Claude longitudinal context ("this is the student's 3rd session — has phrasing improved?") instead of grading every session in isolation. ## Prerequisites - Node.js 18+ - **Redis Stack** (not plain Redis — vector search needs the RediSearch module, which plain Redis doesn't include) - A [Deepgram](https://console.deepgram.com) API key - An [Anthropic](https://console.anthropic.com) API key ### Installing Redis Stack (macOS) ```bash brew tap redis-stack/redis-stack brew install --cask redis-stack-server ``` Start it manually before running the app (it's a cask, so `brew services` doesn't manage it): ```bash redis-stack-server ``` Leave that running in its own terminal tab. It listens on `redis://localhost:6379` by default. For other platforms, see [redis.io/docs/install/install-stack](https://redis.io/docs/install/install-stack/) — or point `REDIS_URL` at a hosted Redis Cloud database (free tier supports RediSearch) if you don't want to run it locally. ## Setup ```bash # 1. Install dependencies npm install # 2. Add API keys cp .env.example .env.local # Fill in DEEPGRAM_API_KEY, ANTHROPIC_API_KEY, and REDIS_URL in .env.local # (REDIS_URL=redis://localhost:6379 if you're running Redis Stack locally per above) # 3. Make sure redis-stack-server is running (see Prerequisites) # 4. Run dev server npm run dev ``` Then open [http://localhost:3000](http://localhost:3000). ## Scripts | Command | What it does | |---|---| | `npm run dev` | Start the Next.js dev server | | `npm run build` | Production build | | `npm run start` | Run the production build | | `npm run lint` | ESLint | ## Pages - `/` — landing page - `/practice` — the core flow: pick a passage (drag a point on the complexity/register map, or read an existing one), record yourself reading it, get a diagnostic report and a recommended next passage - `/progress` — longitudinal view of a reader's session history ## Architecture See `docs/architecture.md` for full data flow, and `docs/ROADMAP.md` for current build status vs. what's planned. **The pipeline:** 1. Deepgram streams word-level transcription with timestamps 2. Levenshtein alignment scores every word (correct / substitution / omission / insertion / uncertain) 3. Metrics computation derives WCPM, error counts, pause placement, self-corrections 4. A skill vector (complexity handling, register handling, WCPM, pause placement, self-correction) is computed per session and stored in Redis 5. Redis vector search (`FT.CREATE`/`FT.SEARCH`) matches the reader's next-best passage target against a library of AI-generated passages, or generates a new one if nothing close exists 6. Claude receives a structured JSON object (metrics + DIBELS tier + prior-session history table) and generates the diagnostic report Claude never sees raw audio or transcript — only structured data from the deterministic pipeline. ## Tech Stack - Next.js 14 (App Router), TypeScript, Tailwind CSS - Deepgram SDK (streaming, word timestamps) - Anthropic SDK (diagnostic report generation, passage generation) - Redis Stack (`redis` npm package) — vector search over the passage library, per-reader session history - compromise.js (syntactic boundary detection for pause placement) - gsap (animated passage-map dot grid) ## Hackathon — UC Berkeley AI Hackathon 2026 Track: Ddoski's World Sponsors: Deepgram, Anthropic, Redis ## Detected evidence (automated analysis) Indexed codebase: 31 recognized source files, 145 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - React (technology) — detected in the code - Redis (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (36 of 36) ``` .env.example .gitignore architecture next-env.d.ts next.config.js package.json postcss.config.js README.md src/app/api/deepgram-token/route.ts src/app/api/diagnose/route.ts src/app/api/generate-passage/route.ts src/app/api/session/route.ts src/app/globals.css src/app/layout.tsx src/app/page.tsx src/app/practice/page.tsx src/app/progress/page.tsx src/components/AudioRecorder.tsx src/components/DiagnosticReport.tsx src/components/DotGrid.css src/components/DotGrid.tsx src/components/LoadingDots.tsx src/components/MetricsDashboard.tsx src/components/Nav.tsx src/components/PassageDisplay.tsx src/components/PassageMap.tsx src/lib/alignment.ts src/lib/deepgram.ts src/lib/generatePassage.ts src/lib/metrics.ts src/lib/passageVectors.ts src/lib/redis.ts src/lib/sessionVector.ts src/lib/types.ts tailwind.config.ts tsconfig.json ``` ### Dependencies - package.json: @anthropic-ai/sdk@^0.20.0, @deepgram/sdk@^3.0.0, @types/diff-match-patch@^1.0.36, @types/node@^20, @types/react@^18, @types/react-dom@^18, autoprefixer@^10.0.1, compromise@^14.0.0, diff-match-patch@^1.0.5, eslint@^8, eslint-config-next@14.2.0, gsap@^3.15.0, next@14.2.0, postcss@^8, react@^18, react-dom@^18, redis@^6.0.0, tailwindcss@^3.3.0, typescript@^5 ### Recent commits (newest first) - Add files via upload - Rename Diagram Improvement Suggestions.png to architecture - Add files via upload - chore: remove internal planning docs from the public repo - feat: redesign practice page layout and recording state - feat: replace all loading animations with ellipsis dots loader - feat: remove reports/resources pages, clean up PassageMap and result UI - fix: improve retry-same-difficulty messaging for WCPM and phrasing - feat: UI polish — page transitions, gradient, waveform, step tracker, copy update - feat: animated dot-grid PassageMap with frozen click indicator - merge: resolve divergent recommendation-logic changes - fix: self-correction detection, weakest-dimension bias, map dot visibility - feat: DIBELS/IEP-aligned recommendation logic + home page learn more expand - fix: wire the PassageMap recommendation arrow into the results view - feat: real session data on home/progress pages, passage markup in results - docs: update roadmap for accent fairness work, fix stale confidence threshold - feat: accent fairness via confidence filtering and language param update - feat: passage-vector KNN search, auto-generation, and next-passage UI - feat: Redis longitudinal tracking + homepage nav scaffold - feat: self-correction signal, edge-case fixes, confidence-flagging extension ## Key source files (fetched from GitHub, selected and truncated for size) ### package.json ``` { "name": "fluently", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@anthropic-ai/sdk": "^0.20.0", "@deepgram/sdk": "^3.0.0", "compromise": "^14.0.0", "diff-match-patch": "^1.0.5", "gsap": "^3.15.0", "next": "14.2.0", "react": "^18", "react-dom": "^18", "redis": "^6.0.0" }, "devDependencies": { "@types/diff-match-patch": "^1.0.36", "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10.0.1", "eslint": "^8", "eslint-config-next": "14.2.0", "postcss": "^8", "tailwindcss": "^3.3.0", "typescript": "^5" } } ``` ### src/app/layout.tsx ```typescript import type { Metadata } from 'next' import { Inter } from 'next/font/google' import Nav from '@/components/Nav' import './globals.css' const inter = Inter({ subsets: ['latin'] }) export const metadata: Metadata = { title: 'Fluently', description: 'Oral reading fluency assessment', } export default function RootLayout({ children, }: { children: React.ReactNode }) { return (