# Project export: Cadence This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: A real-time communication companion that helps people with ALS join fast conversations in their own voice, with ready replies, control, and one-tap speech. - Devpost: https://devpost.com/software/cadence-j7lsan - GitHub: https://github.com/usv240/cadence - Demo: https://cadence-orcin-kappa.vercel.app/ - Video: https://www.youtube.com/embed/fonqmTtUGCk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — usv240 (12 commits), ArpitaMKalburgi (1 commits) ## Devpost submission (written by the team) ### Inspiration A conversation can move at 100 to 140 words per minute, while many AAC users communicate far more slowly. The problem is not only typing speed. It is timing, personality, repair after a misunderstanding, and the ability to start a topic instead of always reacting. Cadence is a real-time communication companion for people with ALS and other AAC users. It listens to a conversation, prepares a small set of possible replies in the user's own style, and keeps the user in control of every word that is spoken. We were inspired by a simple question: what if someone could join a fast dinner conversation, make a joke, say “I love you,” hold the floor, or start a topic of their own before the moment moves on? ### What it does Cadence turns a live conversation into ready-to-review communication choices. Live captions: Uses the browser's Speech Recognition API for no-cost, on-device live transcription. Prepared replies: Uses GPT-5.6 with structured JSON output to create diverse reply options grounded in the latest conversation, the user's saved voice style, personal details, and local memory. User control: A person can speak a suggestion, edit it, make it shorter, ask for something more like them, save it, reject it, or mark the context as wrong. Initiation, not only reaction: “Start something” prepares conversation openers so users can bring up a story, concern, question, affection, or news on their own terms. Fast communication boards: Quick reactions, feelings, care needs, saved phrases, and “Hold the floor” are always easy to reach. Offline resilience: Needs, feelings, saved replies, phrase boards, and instant device voice remain useful even when online AI is unavailable. Voice options: Users can choose OpenAI text-to-speech or instant device speech, with tone-aware delivery. Accessible input: Large tap targets, keyboard support, screen-reader labels, scanning mode, and experimental local eye-gaze focus support different access needs. Privacy-first local memory: Voice style, personal details, preferences, remembered people and topics, session data, phrases, and access settings are stored locally on the device. Memory is visible and clearable. Safety controls: Cadence never speaks automatically from a model suggestion or gaze focus. The person confirms with touch, keyboard, switch scanning, or Select. Why Cadence is different Many communication tools help compose text faster. Cadence focuses on participation in the whole conversational turn: Be ready before the moment passes. Choose language that sounds like the person. Repair a wrong suggestion or misunderstanding. Hold the floor when more time is needed. Initiate something meaningful, not only react. The goal is not simply faster typing. It is timing, choice, and connection. ### How we built it Cadence is built with: Next.js 14 App Router and TypeScript Tailwind CSS for a responsive, tablet-first interface OpenAI Responses API with gpt-5.6-luna, low reasoning effort, lean prompts, and structured JSON outputs OpenAI TTS for server-side speech synthesis with configurable voice and tone instructions Browser Web Speech API for live captions MediaPipe Face Landmarker for experimental, local eye-gaze focus localStorage for style cards, personal details, phrase boards, remembered people/topics, sessions, access preferences, and optional local debug recordings Upstash Redis for distributed per-IP rate limiting in real mode Vercel for deployment, analytics, and production hosting The language layer is intentionally separated behind interfaces and API routes: predict() creates grounded candidate replies expand() turns a short idea into full replies toneAdjust() rewrites a message in warm, firm, or funny tone initiate() prepares conversation starters speak() supports OpenAI speech or instant device voice transcribe() supports browser captions This makes the system replaceable and testable. MOCK_MODE=1 runs offline mock behavior without model or speech costs. MOCK_MODE=0 enables the protected real AI path. Privacy and safety by design Cadence is an early prototype, not a medical device or emergency service. We designed for meaningful control: No account is required. Personal voice data, details, memory, phrases, and sessions stay on the device. Camera video and gaze samples are processed locally for the experimental eye-gaze feature and are not uploaded or stored by Cadence. AI receives only the limited conversation context needed to prepare a reply. The server validates input size, requires explicit real-mode consent, rate-limits requests per IP, and keeps API keys server-side. Eye gaze is an experimental focus method, not a general mouse replacement. It never triggers speech automatically. Care phrases such as “I need suction” do not alert caregivers or emergency services. They only speak the selected phrase. ### Challenges we ran into The hardest challenge was balancing speed with control. A system that reacts too slowly misses the conversational moment. A system that reacts too early can misunderstand an unfinished sentence. We addressed this with speculative prediction from interim captions, a short final-caption debounce, cancellation of outdated requests, and clear “preparing replies” states. We also learned that reliability matters more than a flashy demo. When real AI, network access, or TTS is unavailable, Cadence does not leave the user without communication. It keeps quick needs, feelings, saved replies, phrase boards, hold-the-floor phrases, and device voice available. Eye gaze was another important lesson. Browser camera tracking is sensitive to lighting, camera position, head movement, calibration, and device performance. We keep it clearly labeled as experimental, local, confirmation-based, and optional. Research that shaped Cadence Cadence is informed by publicly available AAC, ALS, accessibility, and communication-participation research. AAC users can face high abandonment rates because of training burden, fatigue, partner burden, and effort. Cadence reduces friction with one-tap prepared replies, scanning, and no required partner app. Communication partner instruction can improve AAC communication. Cadence does not require a partner to install anything, while still offering a concise optional partner tip. AAC design must account for cultural and linguistic context. Cadence supports user-controlled language preferences and a “do not translate or normalize my wording” preference. Recent AAC UI co-design research emphasizes that researcher-only design misses real access barriers. Cadence includes a voluntary tester feedback process and treats participatory co-design as an ongoing requirement. Research on communicative participation in ALS supports measuring more than speed. Cadence tracks replies spoken, topics initiated, time-to-response, repairs, rejected suggestions, and whether a reply sounded like the user. Voice banking and voice identity research informed our emphasis on user choice, device voice fallback, tone, and configurable speaking voices. Selected sources: Inclusive AAC UI co-design study Communication partner instruction meta-analysis ASHA multilingual service-delivery guidance ALS communicative participation study Voice banking and identity in motor neurone disease These sources guide our product decisions. They do not prove clinical effectiveness of this prototype. Accomplishments that we are proud of Built a complete live conversation flow from caption to prepared reply to confirmed speech. Made initiation a first-class feature, so users can lead conversations rather than only respond. Kept user agency central: AI suggests, the person chooses. Built offline-safe communication tools instead of depending completely on cloud AI. Added meaningful access options: touch, keyboard, scanning, instant voice, and experimental local eye-gaze focus. Added local-first personalization, memory, needs, feelings, phrase boards, and session recovery. Added production safeguards including structured outputs, consent checks, input caps, CSP, rate limiting, and server-side secret handling. Built a responsive experience for desktop, tablet, and mobile use. Added unit, integration, and end-to-end coverage for core flows including onboarding, local sessions, offline fallback, rate limits, privacy, scanning, and input validation. ### What we learned We learned that a strong AAC experience cannot be reduced to a text box and a faster model. The most important design decision was making Cadence a system for conversation participation, not just message generation. The person needs to be ready, recognized, able to correct the system, able to express care or urgency, and able to decide when not to speak. We also learned that accessibility features must be designed with failure states in mind. A tool is more trustworthy when it explains uncertainty, keeps a clear off switch, and still offers communication when a network or model call fails. What is next for Cadence Our next priority is co-design with people who use AAC, people living with ALS, caregivers, speech-language pathologists, and assistive-technology professionals. We want to learn what feels useful, inaccurate, tiring, or missing in real conversations. We plan to improve reply relevance, voice match, personal profiles for family, care, work, and medical settings, editable captions, custom names and vocabulary, multilingual support, and clearer confidence when listening is uncertain. We will continue strengthening access through Bluetooth switches, configurable scanning, and experimental target-based eye-gaze focus with better calibration and validation. Eye gaze will remain clearly labeled as experimental until it is tested reliably with users. We also plan to expand offline communication boards, saved replies, care phrases, conversation repair, and user-controlled session history, so communication remains available even when AI or network access is unavailable. Most importantly, we will measure participation, not only taps saved: meaningful replies, topics initiated, repairs completed, fatigue, time-to-response, and whether the user feels the words sound like them. Cadence is an early prototype, not a medical or emergency tool. Our goal is to build responsibly with the community, so more people can be present in the conversations that matter, in words that remain their own. We will use feedback to improve: Reply relevance, voice match, and rejection/repair behavior Partner-tip content and communication setup Multilingual and code-switching support Per-person and per-setting communication profiles Target-based eye-gaze reliability, calibration, fatigue, and validation Accessibility for switches, eye-gaze hardware, and assistive browsers Outcome measures centered on participation, autonomy, fatigue, satisfaction, and “this sounded like me” Cadence is early, but the direction is clear: help more people be present in the conversations that matter, in words that remain their own. ## README (from the GitHub repository) # Cadence **Cadence helps people with ALS and other AAC users stay part of a live conversation.** When someone speaks, Cadence listens, prepares a few short replies in the user's voice, and makes them ready to speak with one tap. The user can also correct the context, hold their turn, express a need, or start a topic themselves. ## Impact Thesis **The potential impact is not just faster text entry. It is helping a person be heard in the live moment, stay the author of their words, and initiate what matters to them.** Cadence is designed around four moments that can otherwise be lost in a fast conversation: responding before the topic changes, repairing a misunderstanding, expressing an essential need, and starting a conversation instead of only reacting. > Cadence is an early assistive communication prototype. It is not a medical device, emergency tool, or replacement for an AAC assessment, speech-language pathologist, or care plan. ## The Problem Conversation moves faster than many AAC systems can be operated. Even a thoughtful message can arrive after the subject has changed. That can turn a group conversation into something a person only reacts to from the edge. Cadence is built around a different goal: **make the user's next turn ready before the moment passes.** - It prepares several grounded choices, not one forced answer. - It keeps every spoken word under the user's control. - It supports starting a conversation, not only replying. - It keeps essential communication available when the network or model is unavailable. **What is different:** many tools help compose a message faster. Cadence is a live conversation layer that helps a person respond, repair a misunderstanding, hold the floor, and initiate in their own voice. ## What a Person Can Do | In the moment | How Cadence helps | | --- | --- | | Someone says something | Shows captions and prepares 3 to 4 reply options. | | A reply feels right | Tap it to preview or speak it, then make it shorter, more like the user, saved, or rejected. | | A caption or reply is wrong | Use **Wrong context**, edit the caption, rename the speaker locally, or use editable repair phrases such as “Please repeat that.” | | The user needs time | Tap **Hold the floor** for a natural floor-holding phrase. | | The user has a need | Open **My needs** for editable care, comfort, and urgency phrases. A local help reminder can point to an existing plan, but Cadence never sends an alert. | | The user wants to lead | Choose **Start something** for user-led conversation openers. | | The user wants their wording respected | Use **Conversation setup** to choose a language and preserve wording without automatic translation or normalization. | | Internet is down | Use saved replies, needs, feelings, repair phrases, the backup board, and device voice. | ## Why It Matters Many people with ALS eventually lose functional speech. AAC can be life-changing, but adoption can be affected by training time, fatigue, access method fit, and communication-partner support. Cadence is designed to reduce the pressure of live conversation without taking authorship away from the user. | Common barrier | Cadence response | | --- | --- | | A long learning curve | Ready-to-tap replies, plain-language onboarding, and contextual help. | | High effort or eye-gaze fatigue | Large targets, low-effort quick phrases, and adjustable single-switch scanning. | | Partner-training burden | Conversation partners need no special app or account. A short optional guide helps them pause and leave authorship with the AAC user. | | Loss of agency | User-led openers, personal voice, editing, rejection, and context repair. | | A service or connection failure | Local fallback replies, device speech, a backup board, and saved essentials. | Background reading: [ALS speech loss](https://pubmed.ncbi.nlm.nih.gov/37760880/), [AAC fit and support](https://pmc.ncbi.nlm.nih.gov/articles/PMC6924798/), [AAC abandonment research](https://pubmed.ncbi.nlm.nih.gov/17114167/), and [eye-gaze AAC in ALS](https://pmc.ncbi.nlm.nih.gov/articles/PMC11530652/). Cadence does not claim clinical effectiveness. The next validation step is voluntary testing with AAC users, family members, caregivers, and speech-language pathologists. ## Research-Informed Design Cadence is informed by publicly available research and guidance. These sources shaped the design priorities below. They do not validate Cadence itself, and Cadence does not reproduce their text, figures, branding, datasets, or participant data. See [UX and performance research notes](docs/UX-PERFORMANCE-RESEARCH.md) for the current lightweight runtime budget, eye-gaze safeguards, and the design review checklist. See [competitive landscape and source boundaries](docs/COMPETITIVE-LANDSCAPE.md) or the public [research and sources page](/research) for a plain-language comparison with established AAC, predictive text, personal voice, atypical-speech, and dedicated eye-gaze solutions. | Public source | Design insight used | Cadence response | | --- | --- | --- | | [ASHA AAC Practice Portal](https://www.asha.org/practice-portal/professional-issues/augmentative-and-alternative-communication/) | AAC use depends on the person, access method, environment, and communication partners. | Large targets, switch scanning, personal setup, conversation kits, and a partner-friendly live flow. | | [AAC abandonment study](https://pubmed.ncbi.nlm.nih.gov/17114167/) | Adoption can be limited by effort, training, support, and fit. | One-tap staged choices, short onboarding, local phrase editing, and no required partner app or account. | | [AAC fit and support study](https://pmc.ncbi.nlm.nih.gov/articles/PMC6924798/) | Communication technology must fit real goals and daily contexts. | Personal voice, details, boundaries, saved kits, needs, feelings, and user-led openers. | | [Eye-gaze AAC in ALS study](https://pmc.ncbi.nlm.nih.gov/articles/PMC11530652/) | Access effort and fatigue matter in ALS communication. | Minimal taps, configurable scanning, fast quick phrases, hold-the-floor, and offline backup access. | | [ALS speech analysis study](https://pubmed.ncbi.nlm.nih.gov/37760880/) | Speech changes and speech loss are significant parts of ALS progression. | Captions plus user-controlled text-to-speech, with no requirement that the user speak to use Cadence. | | [Google SpeakFaster paper](https://research.google/pubs/using-large-language-models-to-accelerate-communication-for-eye-gaze-typing-users-with-als/) | Language models can reduce interaction effort for AAC communication. | Diverse structured reply choices, style-guided wording, and an internal tap-savings metric with a non-comparability caveat. | | [W3C WCAG 2.2](https://www.w3.org/TR/WCAG22/) | Target size, visible focus, keyboard operation, and clear status feedback are essential access requirements. | Large controls, focus rings, modal focus trapping, Escape support, keyboard navigation, live announcements, and responsive layouts. | | [Inclusive AAC UI co-design study](https://pubmed.ncbi.nlm.nih.gov/39868412/) | Iterative co-design can reveal access barriers that designer-only work misses. | Voluntary AAC-user, caregiver, and SLP feedback is a required validation step, not a claimed result. | | [AAC partner-instruction meta-analysis](https://www.tandfonline.com/doi/full/10.3109/07434618.2015.1052153) | Conversation partners can affect AAC communication outcomes. | A compact partner guide: no partner app or account, speak to the person, pause, and let the person decide. | | [ASHA multilingual service guidance](https://www.asha.org/practice-portal/professional-issues/multilingual-service-delivery/) | Communication assessment and support should consider linguistic and cultural context. | Device-local language choice and a preserve-wording setting. Cadence does not intentionally translate or normalize a person's dialect or code-switching. | | [Voice banking and identity in MND](https://p [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 64 recognized source files, 383 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Codex — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (72 of 72) ``` .env.example .eslintrc.json .github/workflows/ci.yml .gitignore AGENTS.md app/api/expand/route.ts app/api/health/route.ts app/api/initiate/route.ts app/api/predict/route.ts app/api/speak/route.ts app/api/style/route.ts app/api/tone/route.ts app/app/eye-gaze-controller.tsx app/app/eye-gaze.worker.ts app/app/page.tsx app/globals.css app/landing-theme-toggle.tsx app/layout.tsx app/manifest.ts app/page.tsx app/research/page.tsx docs/Beta-Tester-Outreach.md docs/COMPETITIVE-LANDSCAPE.md docs/EYE-GAZE-NEXT.md docs/Pilot-Protocol.md docs/RESEARCH-IMPLEMENTATION.md docs/Using-Cadence.md docs/UX-PERFORMANCE-RESEARCH.md e2e/cadence.spec.ts evals/fixtures.ts evals/run.ts lib/api-guard.ts lib/browser-transcribe.ts lib/conversation-kits.ts lib/conversation-service.ts lib/conversation-settings.ts lib/conversation.ts lib/debug-log.ts lib/expand.ts lib/eye-gaze.ts lib/feelings.ts lib/help-plan.ts lib/impact.ts lib/initiate.ts lib/local-session.ts lib/memory.ts lib/mock-language.ts lib/needs.ts lib/offline-fallback.ts lib/openai.ts lib/participation.ts lib/personal-vocabulary.ts lib/predict.ts lib/profile.ts lib/repair-phrases.ts lib/reply-preferences.ts lib/speak.ts lib/style-card.ts lib/theme.ts lib/toneAdjust.ts lib/voices.ts LICENSE next-env.d.ts next.config.mjs package.json playwright.config.ts postcss.config.mjs public/sw.js README.md tailwind.config.ts tests/core.test.ts tsconfig.json ``` ### Dependencies - package.json: @mediapipe/tasks-vision@^0.10.35, @playwright/test@^1.61.1, @types/node@20.17.16, @types/react@18.3.18, @types/react-dom@18.3.5, @upstash/ratelimit@^2.0.8, @upstash/redis@^1.35.3, @vercel/analytics@^2.0.1, autoprefixer@10.4.20, eslint@8.57.1, eslint-config-next@14.2.25, next@14.2.25, openai@^6.22.0, postcss@8.5.1, react@18.3.1, react-dom@18.3.1, tailwindcss@3.4.17, tsx@^4.21.0, typescript@5.7.3 ### Recent commits (newest first) - Eye gaze icon change - fix: allow local gaze wasm and add favicon - fix: default to light mode and style onboarding - feat: strengthen accessible conversation experience - feat: add eye gaze controls and supporting updates - feat: strengthen Cadence reliability and research story - feat: harden real-mode API protection - docs: finalize impact story and tester guidance - feat: improve responsive conversation experience - feat: improve real-time feedback and observability - feat: improve real-mode consent and voice steering - feat: harden Cadence for offline use and testing - Initial Cadence conversation copilot ## Key source files (fetched from GitHub, selected and truncated for size) ### AGENTS.md ```markdown # AGENTS.md — Cadence ## What this is Cadence is a real-time conversational communication aid for people with ALS. It listens to a live conversation, predicts what the user may want to say in their own voice and style, and stages replies so speaking can take one tap instead of a minute of typing. **North star:** a person with ALS joins a fast dinner conversation in their own words, in time to matter. ## Architecture - **Ambient captions:** browser Web Speech API feeds a rolling room transcript. - **Context:** recent turns, learned style card, local profile, and local people/topic memory. - **Prediction:** OpenAI Responses API with `gpt-5.6-luna`, low reasoning, and structured JSON returns 3–4 diverse candidates; interim captions pre-warm a request and final captions replace it. - **Speech:** streamed OpenAI Audio Speech in real mode, with immediate browser device speech as the offline and low-latency fallback. - **Resilience:** `MOCK_MODE`, local session recovery, saved replies, editable needs/feelings, and an offline backup board. ## Stack - Next.js App Router, TypeScript strict mode, Tailwind CSS. - OpenAI SDK for structured generation and real-mode TTS. - Browser Web Speech API for captions; Upstash Redis for durable real-mode rate limiting. - Vercel deployment with an account-level WAF rule for `/api/*`. ## Conventions - Keep modules small and single-purpose; server logic belongs in `app/api` route handlers. - Keep model-facing logic behind `predict`, `expand`, `toneAdjust`, `speak`, `transcribe`, and related interfaces so mocks and real services swap cleanly. - Keep prompts lean and return structured data. - Prioritize large targets, high contrast, keyboard/switch access, and screen-reader support. - Never put secrets in code. Real mode needs `OPENAI_API_KEY`, `UPSTASH_REDIS_REST_URL`, and `UPSTASH_REDIS_REST_TOKEN` server-side only. ## Validation Run `npm run typecheck`, `npm run lint`, `npm run build`, `npm run eval`, and `npm run test:e2e` for changes that affect those areas. Keep mock mode runnable without keys or API cost. ## Scope Make focused local changes and run non-destructive validation. Ask before external, destructive, costly, or materially scope-expanding actions. ``` ### docs/RESEARCH-IMPLEMENTATION.md ```markdown # Research Implementation Boundaries Cadence uses public research and professional guidance to choose design priorities. These sources do not validate Cadence, establish clinical benefit, or replace participatory testing with people who use AAC. | Source | What Cadence takes from it | Product boundary | | --- | --- | --- | | [Inclusive AAC UI co-design study](https://pubmed.ncbi.nlm.nih.gov/39868412/) | Real access barriers emerge through iterative co-design. | Cadence collects only voluntary tester feedback. It does not claim co-design outcomes until those testers participate. | | [AAC partner-instruction meta-analysis](https://www.tandfonline.com/doi/full/10.3109/07434618.2015.1052153) | Partner behavior matters for AAC interaction. | Cadence offers an optional brief partner guide. It does not claim partners need no support. | | [ASHA multilingual guidance](https://www.asha.org/practice-portal/professional-issues/multilingual-service-delivery/) | Linguistic and cultural context matters. | Users choose a recognition language and can preserve wording. Cadence is not a language assessment or translation tool. | | [Voice banking and identity in MND](https://pubmed.ncbi.nlm.nih.gov/33350040/) | Speech voice can be identity-sensitive. | Users can select a device or OpenAI speech voice and associate a preferred one with a kit. Cadence does not clone a user's voice. | | [ALS participation study](https://pubmed.ncbi.nlm.nih.gov/38837773/) | Participation includes more than speed. | Cadence records local interaction indicators only. They are not clinical outcomes or evidence of efficacy. | ## Eye-gaze boundary Eye-gaze focus is experimental. It runs locally, selects only from large actionable targets, requires explicit confirmation before speech, and can be turned off immediately. It is not a general pointer, a medical device, or a replacement for dedicated eye-gaze hardware. ``` ### package.json ``` { "name": "cadence", "version": "0.1.0", "private": true, "scripts": { "dev": "node node_modules/next/dist/bin/next dev", "build": "node node_modules/next/dist/bin/next build", "start": "node node_modules/next/dist/bin/next start", "lint": "node node_modules/next/dist/bin/next lint", "typecheck": "node node_modules/typescript/bin/tsc --noEmit", "test:unit": "node --import tsx --test tests/**/*.test.ts", "eval": "node node_modules/tsx/dist/cli.mjs evals/run.ts", "test:e2e": "playwright test --workers=1" }, "dependencies": { "@mediapipe/tasks-vision": "^0.10.35", "@upstash/ratelimit": "^2.0.8", "@upstash/redis": "^1.35.3", "@vercel/analytics": "^2.0.1", "next": "14.2.25", "openai": "^6.22.0", "react": "18.3.1", "react-dom": "18.3.1" }, "devDependencies": { "@playwright/test": "^1.61.1", "@types/node": "20.17.16", "@types/react": "18.3.18", "@types/react-dom": "18.3.5", "autoprefixer": "10.4.20", "eslint": "8.57.1", "eslint-config-next": "14.2.25", "postcss": "8.5.1", "tailwindcss": "3.4.17", "tsx": "^4.21.0", "typescript": "5.7.3" } } ``` ### app/layout.tsx ```typescript import type { Metadata } from "next"; import { Analytics } from "@vercel/analytics/next"; import "./globals.css"; export const metadata: Metadata = { title: "Cadence | Communication companion", description: "A real-time communication aid for faster, more personal conversations.", robots: { index: false, follow: false }, }; export default function RootLayout({ children }: Readonly<{ children: React.ReactNode }>) { return (