# Project export: MatchVision This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Empowering blind and low-vision fans to follow the action, context, and emotion of live sports independently - Devpost: https://devpost.com/software/match-vision - GitHub: https://github.com/bbioren/match-vision - Demo: http://match-vision-nine.vercel.app/ - Result: winner (Best Use of Terac) - Team: 5 GitHub contributor(s) — Ben Bioren (57 commits), Justin Zhou (57 commits), Claude Sonnet 4.6 (36 commits), Christopher Tan (5 commits), Cursor (3 commits) ## Devpost submission (written by the team) ### Inspiration Justin’s father is low vision, and watching sports with him made one problem impossible to ignore: broadcasts are still designed for people who can clearly see the screen. Soccer commentary captures the emotion of a match, but it often skips the visual context that sighted fans take for granted. A commentator might yell “what a chance,” but a low-vision fan may still not know where the ball was, who was open, which direction the attack was moving, or why the stadium reacted. Existing accessibility solutions for sports are still limited. Most tools focus on subtitles, screen readers, or generic text-to-speech, but those do not solve the core problem of live sports: the most important information is spatial and visual. Fans need to understand the field, the movement, the pressure, and the moment, not just hear a transcript. This is not just one family’s problem. Vision impairment affects approximately 2.2 billion people worldwide, and sports remain one of the most shared cultural experiences in the world. MatchVision was built to make that experience more accessible, independent, and emotionally complete. ### What it does MatchVision is a voice-first accessibility companion for blind and low-vision soccer fans. It acts as a missing visual layer for the match, turning soccer video into concise audio-descriptive commentary and natural voice Q&A. Instead of giving generic commentary, MatchVision explains the visual details that matter: where the ball is who has possession which direction the attack is moving where pressure is coming from who has space what key event just happened why the crowd reacted Users can ask questions like: “What just happened?” “Where is the ball?” “Who has space?” “Give me tactical detail.” MatchVision responds with short, spoken answers designed for live play. The key idea behind MatchVision is that not every part of a frame matters equally. We used human gaze heatmaps to identify the regions people naturally focus on during soccer clips, then used those heatmaps to guide the vision-language model toward the most important parts of the play. We also used human annotations from Terac to improve the captions themselves, teaching the model to produce descriptions that are useful for accessibility: spatially clear, concise, grounded, and focused on the moment. Conceptually, each visual region is weighted by human attention: $$ \text{Attention Weight} \propto \text{Heatmap Intensity} $$ Then the captioning objective is guided by both visual attention and human caption quality: $$ \mathcal{L}{\text{total}} = \mathcal{L}{\text{caption}} + \lambda \mathcal{L}{\text{heatmap}} + \beta \mathcal{L}{\text{human}} $$ This lets MatchVision learn not only what is in the frame, but what parts of the frame matter most to explain. ### How we built it We built MatchVision as a full human-in-the-loop accessibility pipeline. First, we collected gaze data from users watching soccer clips. Using webcam-based eye tracking with WebGazer.js and MediaPipe FaceMesh, we took advantage of the human annotation marketplace Terac to capture where people looked during each play. We aggregated those gaze points into heatmaps, which showed the visually important regions of the field over time. These heatmaps became an attention signal for the model, helping it focus more on the hottest regions of the frame instead of treating the entire video equally. Using Terac, we also collected human annotations for audio-descriptive commentary. Annotators reviewed soccer moments and ranked or improved captions based on what would be most useful to a blind or low-vision fan. We focused on features like ball location, direction of attack, key-event coverage, concision, and hallucination avoidance. We then used these two human signals together to fine-tune a vision-language model. The heatmaps guided the visual side of the model by emphasizing the parts of the frame humans cared about most, while the Terac captions guided the language side by teaching the model what high-quality accessibility commentary sounds like. Our pipeline looked like this: Collect soccer clips. Record human gaze while users watch each clip. Convert gaze points into frame-level heatmaps. Use Terac annotators to create and rank accessibility captions. Fine-tune a vision-language model with heatmap-weighted frames and human-improved captions. Generate audio-descriptive commentary. Let users ask voice questions about the match. On the product side, we built the app with JavaScript, HTML, CSS, and Node.js. We created a Chrome Extension using Manifest V3 for gaze-controlled video interaction, a web app for audio-descriptive commentary and Q&A, and API routes for storing labels, sessions, and context. Technologies we used include: JavaScript / HTML / CSS Node.js WebGazer.js MediaPipe FaceMesh Terac MCP Redis / Upstash Deepgram LLM/VLM APIs Python analytics scripts JSON match timelines and event logs ### Challenges we ran into One of the biggest challenges was defining what “good” actually means. A caption can be grammatically correct and still be bad for accessibility. If it says “a dangerous chance develops” but does not say where the ball is, who is attacking, or why the chance is dangerous, it fails the user. We had to design our annotation rubric around accessibility quality instead of generic caption quality. Another challenge was aligning gaze heatmaps with model attention. Human gaze data is noisy, especially when collected through webcams in different lighting conditions and screen setups. We had to smooth the signal, calibrate users, aggregate data across sessions, and think carefully about how to turn heatmap intensity into a useful training signal. Sports video itself was also difficult. Soccer is fast, crowded, and often ambiguous. The ball is small, important actions happen off-ball, and camera angles change constantly. We had to make the model concise while still grounded, and we had to avoid hallucinations because incorrect descriptions can be worse than no description at all. We also had to combine a lot of moving parts into one coherent system: gaze tracking, heatmaps, Terac annotation, caption fine-tuning, voice Q&A, Redis-backed storage, and browser-based playback. Making all of those pieces feel like one product instead of separate demos was one of the hardest parts of the project. ### Accomplishments we're proud of We are proud that MatchVision is grounded in a real accessibility need. The project began with Justin’s father, but the problem extends to millions of blind and low-vision fans who are excluded from the full visual experience of live sports. We are especially proud of building a human-in-the-loop model improvement pipeline. Instead of just prompting a model to “describe this soccer clip,” we used two forms of human feedback: gaze heatmaps to show where people look, and Terac annotations to show what people find useful. That made the system more intentional and more accessibility-focused. We are also proud of making the product voice-first. A low-vision user should not need to navigate a complicated visual interface to understand a visual moment. They should be able to ask a question naturally and hear a useful answer immediately. Finally, we are proud that MatchVision reframes sports accessibility. We are not trying to replace commentators. We are building the missing visual layer that helps fans understand the parts of the match that normal commentary assumes they can already see. ### What we learned We learned that accessibility is not just a feature. It changes the entire objective of the system. A normal video captioning model optimizes for fluent descriptions. MatchVision has to optimize for trust, timing, spatial clarity, and usefulness. We also learned how powerful human data can be. Heatmaps gave us a way to teach the model what parts of the frame mattered visually, while Terac annotations taught it what kinds of descriptions mattered linguistically. Combining those two signals helped us think about model fine-tuning in a more human-centered way. We learned that live sports are uniquely hard because the important context is constantly changing. It is not enough to identify objects. The system has to understand movement, direction, pressure, spacing, and why a moment matters in the match. Most importantly, we learned that personal motivation makes technical decisions sharper. Thinking about whether this would actually help Justin’s father made us focus less on flashy AI output and more on clarity, reliability, and dignity. ### What's next Next, we want to move MatchVision closer to real-time live match support. Our goal is for blind and low-vision fans to use it during a live broadcast, not only on preprocessed clips. We also want to collect more gaze and caption data from blind and low-vision soccer fans directly. Their feedback should define what the model optimizes for. We plan to expand the Terac annotation pipeline, improve the heatmap-guided fine-tuning process, and evaluate the system on more matches, camera angles, and levels of play. On the product side, we want to improve personalization. Some users may want one-sentence updates, while others may want tactical detail or beginner-friendly explanations. MatchVision should adapt to each fan’s preferences. Long term, we want to expand beyond soccer. The same approach could support basketball, tennis, racing, concerts, theater, and any live visual experience where important context is trapped on screen. Our vision is simple: when the crowd erupts, everyone should know what happened, where it happened, and why it mattered. MatchVision is our step toward making live sports accessible to everyone. ## README (from the GitHub repository) # MatchVision **Commentary tells you the game. MatchVision lets you see it.** MatchVision is a voice-first accessibility companion that gives blind and low-vision soccer fans the missing visual layer of a soccer match: ball location, player positioning, direction of attack, and why key moments matter. ## What's in this repo Two parts that share one accessibility mission: 1. **Chrome extension (`extension/`)** — a gaze-controlled zoom/pan tracker that works on *any* web video, plus an always-on Claude voice agent. Say "Match Vision, what just happened?" and Claude answers out loud via Deepgram TTS (browser TTS as fallback), and can drive the tracker itself (zoom in, reset, follow the ball) via tool calls. 2. **Web app (`src/`, `local-server.mjs`)** — the original ADC (audio description) demo: structured match-moment timelines, a voice/text Q&A flow, a Terac annotation lab for collecting human accessibility labels, and an eval dashboard showing measured improvement from those labels. ## Run the demo ### Web app ```bash npm run dev ``` Open: - Main demo: - Annotation Lab: - Eval Dashboard: No install or API keys required for the static MVP — it falls back to a deterministic local description and browser speech APIs. Add `GEMINI_API_KEY`/`ANTHROPIC_API_KEY` (see below) for real model-generated descriptions. ### Chrome extension 1. `cp extension/secrets.example.js extension/secrets.js` and fill in `MV_ANTHROPIC_KEY`/`MV_DEEPGRAM_KEY`. `extension/secrets.js` is gitignored — it's never committed, so real keys never hit GitHub. 2. Open `chrome://extensions`, enable Developer Mode, "Load unpacked", select `extension/`. 3. Open any page with video (YouTube, a broadcast stream, etc.) and click the MatchVision icon. 4. Click the mic button and talk — Claude answers (spoken via Deepgram, falling back to browser TTS), and can zoom/pan/reset the tracker for you. ## Current status - Gaze-controlled video zoom/pan extension (WebGazer-based), with a Claude voice agent layered on top that can both answer questions and control the tracker. - Structured match-moment web demo with voice/text Q&A and spoken fallback. - Terac annotation lab: drag-rank 5 commentary variations per clip, collect human accessibility labels. - Real Terac MCP integration (`scripts/terac-agent.mjs`) that creates a paid labeling opportunity, polls for submissions, and approves them against a budget guardrail — not just a UI mockup. - Gemini/Claude-generated commentary candidates (`scripts/generate-candidates.mjs`) feed the annotation lab instead of hand-written examples. - Prompt-optimization pipeline that learns a "champion prompt" from Terac preference labels, plus a DPO dataset exporter for fine-tuning. - Analytics-replay data source: real StatsBomb matches converted to structured moment timelines via `kloppy` + `socceraction` (xT + VAEP), synced to real broadcast footage — no live VLM required. - `npm run check` / `npm run metrics` / `npm run eval-ranker` pass; see numbers via those scripts rather than stale copy here. ## Validate data ```bash npm run check npm run metrics ``` ## Analytics-replay clips (StatsBomb ground truth, no live VLM needed) Alongside the live-video pipeline, the demo includes a second data source: real StatsBomb open-data matches converted to a structured moment timeline via `kloppy` + `socceraction` (xT + VAEP), with no broadcast video required. See `analytics/build_state_frames.py`. Currently wired into `data/clips.json`: - **Turkey vs Italy, Euro 2020 group stage** (`turkey_vs_italy_euro2020_analytics`) — ticker only, no video. - **Argentina vs France, 2022 World Cup Final** (`argentina_vs_france_wc2022_final_analytics`) — ticker synced to FIFA's official full-match YouTube upload via a `video_offset_seconds` kickoff offset. Regenerate a timeline (requires the `analytics/.venv` — Python 3.12, see `analytics/requirements.txt`): ```bash cd analytics && source .venv/bin/activate python fit_models.py # fits xT + VAEP once, caches to analytics/cache/ python build_state_frames.py # default: Turkey vs Italy python build_state_frames.py --match-id 3869685 --out ../data/analytics/argentina_vs_france_wc2022_final_timeline.json ``` ## AI generation credentials For the annotation candidate generator (`scripts/generate-candidates.mjs`) and `/api/describe`, use either Gemini or Anthropic — both support vision (frame analysis) and text generation: ```bash # Gemini GEMINI_API_KEY=your_google_ai_studio_key GEMINI_MODEL=gemini-2.5-flash # optional, this is the default # Anthropic ANTHROPIC_API_KEY=your_anthropic_key ANTHROPIC_MODEL=claude-haiku-4-5-20251001 # optional, this is the default ``` `GOOGLE_API_KEY` is also accepted as an alias for `GEMINI_API_KEY`. Set `LLM_PROVIDER=gemini` or `LLM_PROVIDER=anthropic` to force one; otherwise both `local-server.mjs` and `generate-candidates.mjs` auto-detect (Gemini first if both keys are set). The Chrome extension's voice agent calls the Anthropic API directly from the browser using `MV_ANTHROPIC_KEY` from `extension/secrets.js` (gitignored — see `extension/secrets.example.js`). TTS uses `MV_DEEPGRAM_KEY` from the same file, falling back to browser TTS if it's unset or the call fails. ## Terac fine-tune pipeline (human labels → better commentary prompt) See [`docs/TERAC_FINETUNE_PLAN.md`](./docs/TERAC_FINETUNE_PLAN.md) for the full design. Short version — zero labels to a champion prompt: 1. **Generate real AI candidates** (replaces hand-written commentary variations): ```bash GEMINI_API_KEY=your_key node scripts/generate-candidates.mjs # or Anthropic instead: LLM_PROVIDER=anthropic ANTHROPIC_API_KEY=your_key node scripts/generate-candidates.mjs # or one clip at a time: GEMINI_API_KEY=your_key node scripts/generate-candidates.mjs --clip yt_eng_cro_12 # sanity-check without burning quota / without a key at all: node scripts/generate-candidates.mjs --dry-run ``` Writes real Gemini or Anthropic outputs (5 prompt strategies per clip) into `data/annotation_tasks.json`, tagged with `generation_provider`/`generation_model`. 2. **Collect Terac rankings** — open `annotate.html`, rank the 5 real candidates per clip. Locally this POSTs to `/api/labels` and is stored in `data/labels.local.json` (gitignored) by `local-server.mjs`; hosted Terac sessions use the same shape. For a real paid Terac run, see `scripts/terac-agent.mjs` (`npm run terac`), which launches and manages the opportunity end-to-end via Terac's MCP API. 3. **Build the preference dataset**: ```bash node scripts/build-preference-dataset.mjs --api=http://localhost:5173 # -> data/training/preference_pairs.jsonl (+ summary.json) ``` 4. **Learn the champion prompt** (Phase 3A — no GPU, no fine-tune budget needed): ```bash node scripts/optimize-prompt.mjs # -> data/prompts/champion_prompt.txt + data/prompts/champion_eval.json ``` Finds the prompt strategy that wins most often, extracts the linguistic patterns of winning vs. losing commentary (ball-location rate, direction mentions, hedging, sentence length), and bakes them into an explicit system prompt. If `data/training/preference_pairs.jsonl` doesn't exist yet, this exits cleanly with instructions instead of crashing. `local-server.mjs` automatically loads `data/prompts/champion_prompt.txt` at startup (if present) and uses it as the system prompt for every `/api/describe` call, overriding the client-sent default — no code changes needed once the file exists. The response body includes `usedChampionPrompt: true/false` for debugging. 5. **Export a DPO fine-tune file** (Option B — stronger, needs a training budget): ```bash node scripts/export-dpo-dataset.mjs # -> data/training/dpo_dataset.jsonl, one {"prompt","chosen","rejected"} object per line node scripts/export-dpo-datas [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 81 recognized source files, 2940 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code - Redis (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (119 of 119) ``` .env.example .gitignore .mcp.json .vercelignore analytics/build_state_frames.py analytics/fit_models.py analytics/requirements.txt analytics/statsbomb_pipeline.py annotate.html api/context.js api/describe.js api/extract.js api/gaze.js api/labels.js api/mediapipe.js api/sessions.js api/tts.js check_redis.mjs data/analytics/argentina_vs_france_wc2022_final_timeline.json data/analytics/argentina_vs_france_wc2022_h2_kickoff_5min_timeline.json data/analytics/turkey_vs_italy_euro2020_timeline.json data/annotation_tasks.json data/annotations/sample_labels.csv data/clips.json data/sample_event_logs.json data/survey_clips.json docs/CHROME_STORE_SUBMISSION.md docs/DEMO_VIDEO_SHOTLIST.md docs/DEVPOST_DRAFT.md docs/EYETRACK_FEATURE.md docs/FINAL_HACKATHON_CHECKLIST.md docs/HACKATHON_WINNING_PITCH.md docs/INTEGRATIONS.md docs/JUDGING_SCRIPT.md docs/LABEL_RECRUITING_MESSAGE.md docs/REAL_CLIPS.md docs/SPONSOR_TALKING_POINTS.md docs/TERAC_ANNOTATION_PLAN.md docs/TERAC_ANNOTATOR_CONTEXT.md docs/TERAC_CENTRAL_STORY.md docs/TERAC_FINETUNE_PLAN.md eval.html extension/background.js extension/content.js extension/manifest.json extension/mediapipe/face_mesh/face_mesh_solution_packed_assets_loader.js extension/mediapipe/face_mesh/face_mesh_solution_packed_assets.data extension/mediapipe/face_mesh/face_mesh_solution_simd_wasm_bin.js extension/mediapipe/face_mesh/face_mesh_solution_simd_wasm_bin.wasm extension/mediapipe/face_mesh/face_mesh_solution_wasm_bin.js extension/mediapipe/face_mesh/face_mesh_solution_wasm_bin.wasm extension/mediapipe/face_mesh/face_mesh.binarypb extension/model-interceptor.js extension/models/face_detection/model.json extension/models/face_landmarks/model.json extension/offscreen.html extension/offscreen.js extension/panel.html extension/panel.js extension/raf-override.js extension/secrets.example.js extension/tracker-window.html extension/tracker-window.js extension/tracker.js extension/webgazer.min.js eyetrack-api.html eyetrack-webgazer.html eyetrack.html gaze-results.html index.html local-server.mjs matchvision_adc_terac_architecture.md package.json privacy.html public/mediapipe/face_mesh/face_mesh_solution_packed_assets_loader.js public/mediapipe/face_mesh/face_mesh_solution_simd_wasm_bin.js public/mediapipe/face_mesh/face_mesh.binarypb public/terac-annotator-context.html README.md run.sh SCOPE.md scripts/build-preference-dataset.mjs scripts/build-static.mjs scripts/check-data.mjs scripts/compute-metrics.mjs scripts/evaluate-ranker.mjs scripts/export-dpo-dataset.mjs scripts/generate-candidates.mjs scripts/optimize-prompt.mjs scripts/package-extension.sh scripts/rehearsal-timer.mjs scripts/segment-videos.mjs scripts/terac-agent.mjs scripts/update-blob-urls.mjs scripts/upload-clips-to-blob.mjs src/annotate.js src/app.js src/eval.js src/gaze-results.js src/ranker.js src/services/description.js src/services/eyetrack-api.js src/services/eyetrack-player-webgazer.js src/services/eyetrack-player.js src/services/eyetrack-real.js src/services/eyetrack-simple.js src/services/eyetrack-webgazer.js src/services/eyetrack.js src/services/match-context.js src/services/match-memory.js src/services/memory.js src/services/vision-extract.js src/services/voice.js src/styles.css src/survey.js survey.html SYSTEMS_OVERVIEW.md TODO.md vercel.json ``` ### Dependencies - analytics/requirements.txt: kloppy@==3.15.0, multimethod@<2.0, pandas, scikit-learn, socceraction, xgboost - package.json: @upstash/redis@^1.38.0, @vercel/blob@^2.4.1, ioredis@^5.11.1 ### Recent commits (newest first) - Remove the Brazil vs Haiti clip option from the eye-tracking demo - Remove Annotation Lab and Eye-Tracking Survey links from homepage - Point eye-tracking demo link at eyetrack-webgazer.html, include it in build - Commit missing timeline JSON for the Argentina vs France h2 kickoff clip - Fix .vercelignore blanket-excluding clips/, breaking all images/videos in prod - Point the eye-tracking survey link at gaze-results.html instead - Point the eye-tracking survey link at annotate.html - Add @vercel/blob dep, include privacy.html in static build, extend blob upload script - vid - Add missing /api/describe, /api/tts, /api/extract serverless functions - Merge branch 'main' of https://github.com/bbioren/match-vision - Added gaze heatmap survey w/ vercel - Prepare extension for Chrome Web Store submission - add .vercelignore to exclude large clip directories from deployment - cache bust vercel: force annotation_tasks.json reload - Update default tracking params to tuned values - update video URLs to use Vercel Blob Storage - trigger vercel rebuild - add 42 first_9_mins clips (010-051) and 18 kaggle segments to annotation tasks - Speak calibration instructions to completion before starting it; speed up TTS ## Key source files (fetched from GitHub, selected and truncated for size) ### TODO.md ```markdown # 17-hour sprint todo ## Critical path - [ ] Lock scope, team roles, sponsor targets, and demo narrative from `SCOPE.md` - [ ] Collect 3-5 short soccer clips and create structured event logs for each clip - [ ] Build minimal web demo shell: clip selector, transcript/response pane, voice button, metrics section - [ ] Implement Claude accessible-description generator from structured event logs and user questions - [ ] Integrate Deepgram STT for voice questions and TTS for spoken responses - [ ] Design Terac annotation task: baseline vs improved descriptions, accessibility rubric, held-out eval set - [ ] Launch Terac labeling and collect enough human labels for before/after evaluation - [ ] Use labels to improve prompt/ranker and compute before/after metrics - [ ] Prepare Devpost submission: pitch, screenshots, track selections, sponsor explanations - [ ] Record 2-3 minute demo video with problem, live voice demo, Terac metrics, sponsor stack - [ ] Run full judging rehearsal under 5 minutes and fix demo-breaking bugs ## Prize-stack polish - [ ] Implement Redis memory for recent match events and user preference modes - [ ] Integrate Arize traces/evals or create equivalent dashboard evidence showing improvement - [ ] Polish UI for accessibility story: large text, voice-first flow, clear baseline vs improved comparison - [ ] Write README with sponsor usage, setup, architecture, and judging talking points ``` ### SCOPE.md ```markdown # MatchVision Hackathon Scope ## One-liner **MatchVision turns live or preselected soccer video into personalized, queryable audio description for blind and low-vision fans, then improves its descriptions using human accessibility labels that train/tune the description selector collected during the hackathon.** ## Tagline > Commentary tells you the game. MatchVision lets you see it. ## Core problem Existing soccer commentary is designed for people who can already see the match. It often says things like “great ball in,” “what a chance,” or “he had options,” but blind and low-vision fans still miss the visual layer: - where the ball is - direction of attack - player positioning - off-ball movement - why the crowd reacted - how close a chance was - what changed in the last few seconds - whether the user wants brief, tactical, beginner, or emotional detail **Framing:** MatchVision is not an AI commentator. It is an accessibility-grade visual description layer. ## Grand prize positioning Submit under **Ddoski’s World**. Narrative: > The World Cup is one of the most watched events on Earth, but sports video remains fundamentally visual. MatchVision gives blind and low-vision fans real-time, personalized access to the spatial and tactical context sighted fans take for granted. We prove improvement with human accessibility labels that train/tune the description selector collected during the hackathon. This hits: - social impact - timely World Cup theme - technical complexity - creativity - functionality - polished user experience ## Sponsor prize strategy ### Primary target: Terac Terac is the safest sponsor-win path because the rubric is explicit: - collect real human-labeled data through Terac during the event - use those labels to improve a model/system - show the improved system beats the base system on unseen examples - build a creative annotation environment - use the human data intelligently within the credit budget For MatchVision, Terac labels should measure accessibility quality of generated descriptions. Annotation dimensions: - Did it describe ball location? - Did it mention direction of attack? - Did it capture the key event? - Was it useful for a blind/low-vision fan? - Was it concise enough? - Did it hallucinate? - Which description is better: baseline or improved? Before/after metrics to show: - helpfulness score - key-event coverage - spatial detail coverage - hallucination/error rate - preference win rate against baseline Demo metric shape: - baseline helpfulness: 60-65% - improved helpfulness: 80-85% - hallucination rate reduced from ~20% to <10% - key-event coverage improved from ~55-60% to ~80% The exact numbers must come from the collected labels/eval, but the demo should clearly show measurable improvement. ### Secondary target: Deepgram Voice must be essential, not bolted on. Use Deepgram for: - speech-to-text user questions - text-to-speech spoken match descriptions - voice-first interaction flow D [truncated — 5714 more characters] ``` ### package.json ``` { "name": "match-vision", "version": "0.1.0", "private": true, "description": "Voice-first accessibility companion for blind and low-vision soccer fans.", "scripts": { "dev": "node local-server.mjs", "start": "node local-server.mjs", "build": "node scripts/build-static.mjs", "check": "node scripts/check-data.mjs", "metrics": "node scripts/compute-metrics.mjs", "rehearse": "node scripts/rehearsal-timer.mjs", "eval-ranker": "node scripts/evaluate-ranker.mjs", "segment-videos": "node scripts/segment-videos.mjs", "terac": "node --env-file-if-exists=.env --env-file-if-exists=.env.local scripts/terac-agent.mjs", "terac:dry": "node --env-file-if-exists=.env --env-file-if-exists=.env.local scripts/terac-agent.mjs --dry-run", "terac:survey": "node --env-file-if-exists=.env --env-file-if-exists=.env.local scripts/terac-agent.mjs --survey", "terac:survey:dry": "node --env-file-if-exists=.env --env-file-if-exists=.env.local scripts/terac-agent.mjs --survey --dry-run", "build-pairs": "node --env-file-if-exists=.env --env-file-if-exists=.env.local scripts/build-preference-dataset.mjs", "generate": "node --env-file-if-exists=.env --env-file-if-exists=.env.local scripts/generate-candidates.mjs", "generate:dry": "node --env-file-if-exists=.env --env-file-if-exists=.env.local scripts/generate-candidates.mjs --dry-run" }, "keywords": [ "accessibility", "soccer", "voice", "hackathon", "deepgram", "terac" ], "license": "MIT", "type": "module", "dependencies": { "@upstash/redis": "^1.38.0", "@vercel/blob": "^2.4.1", "ioredis": "^5.11.1" } } ``` ### analytics/requirements.txt ``` # Python 3.12 required — socceraction pins Python <3.13. # soccer-xg is abandoned (pins pre-2.0 pandas, won't build) — socceraction's # built-in xT and VAEP cover the metrics we need without it. # kloppy pinned to 3.15.0: socceraction 1.5.3's kloppy bridge calls a # coordinate-system constructor whose signature changed in kloppy >=3.16. kloppy==3.15.0 socceraction pandas multimethod<2.0 xgboost scikit-learn # macOS only: xgboost needs the OpenMP runtime. # brew install libomp ``` ### src/app.js ```javascript import { generateAdc, generateQnaAnswer } from './services/description.js'; import { setupWakeWordListening, speakWithDeepgramOrFallback, stopSpeaking } from './services/voice.js'; import { saveMemory } from './services/memory.js'; import { currentMoment, memoryAt } from './services/match-memory.js'; import { resolveContextAt, formatContextSummary } from './services/match-context.js'; import { captureVideoFrameSequence, extractMomentFromFrame, formatVideoTimestamp, resetProbeVideo } from './services/vision-extract.js'; let clips = []; let timeline = []; let liveTimeline = []; let liveMode = true; // True for clips loaded from a precomputed ground-truth timeline_asset // (e.g. StatsBomb/socceraction analytics replay) instead of live video + // VLM extraction. Disables the live-extraction toggle/video element for // that clip without touching the live-vision path used by video clips. let isAnalyticsReplay = false; let currentSeconds = 0; let pendingBuckets = new Set(); let bucketWaiters = new Map(); let queueRunning = false; let priorityBucket = null; let extractTimings = []; let extractInterval = Number(window.MATCHVISION_EXTRACT_INTERVAL) || 2; let liveExtractBlocked = false; let seekTimer = null; // Live caption + auto-speak: track the last moment captioned/spoken so we // don't re-caption an unchanged moment on every timeupdate tick, and don't // re-speak a key moment the user has already heard (e.g. re-seeking nearby). let lastCaptionKey = null; let lastSpokenKey = null; let lastSpokenAtSecond = -Infinity; // Generated commentary lines take a few seconds to speak even when short; // without a minimum gap, moments arriving faster than that would constantly // interrupt each other before finishing. Key moments (goals/danger) always // cut through immediately regardless of this gap. const MIN_CAPTION_GAP_SECONDS = 3; // True while a wake-word question is being answered — the live ticker keeps // updating captions visually but stays silent so it doesn't talk over the // answer, then resumes once the spoken answer finishes. let commentaryMuted = false; // The most recent ADC/Q&A answer text — there's no visible answer panel // (voice is the primary interface), so this is just internal state for // speak()'s default argument and for re-speaking after a clip/mode change. let lastAnswerText = ''; const $ = (id) => document.getElementById(id); function momentKey(m) { return `${m?.atSecond ?? ''}|${m?.event || m?.summary || ''}`; } // "Key" moments (goals, high-danger plays) get spoken aloud automatically as // the video plays; everything else only updates the visible caption text — // speaking every single pass/touch nonstop would be unusable, not helpful. function isKeyMoment(m) { if (!m) return false; const danger = (m.danger_level || m.urgency_level || '').toLowerCase(); return danger === 'high' || /goal/i.test(m.event || ''); } function renderCaption(m) { const el = $('liveCaption'); if (!el) return; const text = m?.commentary || m?.summary || m?.event; if (!text) { el.classList.remove('visible', 'key-moment'); return; } const key = momentKey(m); if (key === lastCaptionKey) return; const isKey = isKeyMoment(m); const gapElapsed = (m.atSecond ?? 0) - lastSpokenAtSecond >= MIN_CAPTION_GAP_SECONDS; if (!isKey && !gapElapsed) return; // too soon after the last caption — let it finish, skip this one lastCaptionKey = key; el.textContent = text; el.classList.add('visible'); el.classList.toggle('key-moment', isKey); // Speak every caption that makes it through the gap above — interrupt- // replace (newest wins) is handled inside speakWithDeepgramOrFallback/ // speechSynthesis, so back-to-back moments cut cleanly into each other. // While a voice question is being answered, keep updating the caption // text but stay silent so the ticker doesn't talk over the answer. if (key !== lastSpokenKey) { lastSpokenKey = key; lastSpokenAtSecond = m.atSecond ?? lastSpokenAtSecond; if (!commentaryMuted) speakWithDeepgramOrFallback(text, { shouldSpeak: () => !commentaryMuted }); } } function currentClip() { return clips[$('clipSelect').selectedIndex]; } function getResolvedContext(seconds = currentSeconds) { const clip = currentClip(); if (!clip?.match_context) return null; return resolveContextAt(clip.match_context, seconds); } function currentMemory() { const ctx = getResolvedContext(); return memoryAt(timeline, currentSeconds, 4, 16, ctx?.match_half ?? null); } function normalize(text) { return text.toLowerCase().replace(/[^a-z0-9 ]/g, '').trim(); } // Local, deterministic fallbacks used when no LLM credentials are configured. function localMoment() { if (!timeline.length) { return { team_in_possession: 'unknown', direction: 'unknown', ball_location: 'unknown', event: liveMode ? 'Waiting for live frame analysis…' : 'unknown', danger_level: 'unknown', summary: liveMode ? 'Press play with live extraction enabled.' : '' }; } return currentMoment(timeline, currentSeconds); } function localAdc(mode) { const m = localMoment(); if (!timeline.length) { return liveMode ? 'Press play to analyze the video. Match memory builds live from each frame.' : 'Enable live vision extraction and press play to build match memory.'; } if (mode === 'brief') { const zone = m.ball_zone || m.ball_location; return `${m.team_in_possession} in possession. Ball in ${zone}. ${m.phase || m.event}.`; } return m.commentary || m.summary || `${m.team_in_possession} — ${m.phase || m.event}. ${m.ball_zone || m.ball_location}.`; } function localAnswer(question) { const q = normalize(question); const recap = currentMemory().map((m) => m.commentary || m.summary).filter(Boolean).join(' '); if (!recap) { return liveMode ? 'No match memory yet. Press play and wait for live frame analysis.' : 'Enable live vision extraction and press play first.'; } if [truncated — 14150 more characters] ``` ### run.sh ```shell #!/usr/bin/env bash # MatchVision dev server — finds Node, validates data, starts server. set -euo pipefail cd "$(dirname "$0")" # Prefer Homebrew Node, then system node, then Cursor helper. if [ -x /opt/homebrew/bin/node ]; then NODE=/opt/homebrew/bin/node elif command -v node >/dev/null 2>&1; then NODE=node elif [ -x /Applications/Cursor.app/Contents/Resources/app/resources/helpers/node ]; then NODE=/Applications/Cursor.app/Contents/Resources/app/resources/helpers/node else echo "Error: Node.js not found. Install with: brew install node" exit 1 fi echo "Using Node: $("$NODE" -v) ($NODE)" echo "Validating event logs..." "$NODE" scripts/check-data.mjs echo "" PORT="${PORT:-5173}" if lsof -nP -iTCP:"$PORT" -sTCP:LISTEN >/dev/null 2>&1; then echo "Error: port $PORT is already in use." echo " Free it: kill \$(lsof -t -iTCP:$PORT -sTCP:LISTEN)" echo " Or use: PORT=5180 ./run.sh" exit 1 fi echo "Starting MatchVision at http://localhost:$PORT" echo " Main demo: http://localhost:$PORT/" echo " Annotation: http://localhost:$PORT/annotate.html" echo " Press Ctrl+C to stop." echo "" exec "$NODE" local-server.mjs ``` ### eval.html ```html MatchVision Eval Dashboard

Arize-style eval evidence

Quality Dashboard

Track whether accessibility descriptions actually improve.

← Back to demo

Baseline

Helpfulness63%
Key-event coverage58%
Spatial detail coverage52%
Hallucination rate21%

After human labels

Helpfulness84%
Key-event coverage81%
Spatial detail coverage88%
Hallucination rate9%

Evaluation traces

ClipQuestionEval resultIssue caught

This page is the local fallback. Replace/augment with Arize Phoenix/AX traces when credentials are available.

``` ### index.html ```html MatchVision

World Cup-level access for blind and low-vision fans

MatchVision

Commentary tells you the game. MatchVision lets you see it.

👁️ Eye-Tracking Demo for Low-Vision Fans →

1. Pick a match moment

Simplified soccer field diagram for selected match moment

Press play — match memory builds live from the video

Match memory

Rolling structured match state up to the current video time. Visual perception (kit colors, screen position) is merged with half-aware team context before narration.

    ``` ### privacy.html ```html MatchVision Eye Tracker — Privacy Policy

    MatchVision Eye Tracker — Privacy Policy

    Last updated: 2026-06-21

    MatchVision Eye Tracker ("the Extension") is an accessibility tool that gives low-vision users gaze-controlled video zoom/pan, plus a voice assistant for hands-free control.

    What the Extension processes

    • Webcam video. Used entirely in your browser to estimate where you're looking (via WebGazer.js). Webcam frames are never uploaded, stored, or sent to any server — gaze estimation happens locally on your device.
    • Microphone audio. When you use the voice assistant, your spoken audio is sent to Deepgram for speech-to-text, the resulting text is sent to Anthropic (Claude) to generate a response, and the response may be converted back to speech via Deepgram. See Deepgram's and Anthropic's own privacy policies for how they handle data sent to their APIs.
    • Page content. The Extension reads the structure of the page you're viewing to locate video elements and can control video playback/fullscreen state. It does not collect or transmit page content elsewhere.

    What the Extension does not do

    • We do not sell or share your data with advertisers.
    • We do not persist voice conversation history beyond your current browsing session — it lives in memory and clears when you stop the voice assistant or close the tab.
    • We do not track your browsing activity beyond what's needed to operate gaze tracking and voice control on the page you're actively using the Extension on.

    Permissions, explained

    • Camera / microphone — required for gaze tracking and voice interaction.
    • Host access (all sites) — the Extension works on any page with video, so it needs to be able to run wherever you choose to use it.
    • debugger — used briefly, only when you ask the voice assistant to make the video fullscreen, to grant the page the same fullscreen permission a real click would. It is not used to inspect, log, or transmit any other browsing activity.

    Contact

    Questions about this policy: ben.bioren@gmail.com

    ``` ### gaze-results.html ```html MatchVision — Gaze Results

    Eye-tracking data collection

    Where people looked

    Aggregated gaze heatmaps from every survey session, per clip.

    ← Take the survey    MatchVision home    Raw JSON →

    —
    sessions
    —
    gaze samples
    —
    clips with data
    —
    avg calibration
    coldmost looked at
    ``` [69 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]