Project Info

CupVoice

🌍

Inspiration

AI is already helping people understand information in new ways. CupVoice brings that idea into live sports. Live sports are exciting, but they can also be hard to follow because everything happens fast: the movement, the sound, the visuals, the crowd, and the commentary. We built CupVoice around one question: What if the same match could be translated into the format that works best for each fan? CupVoice turns live-style match events into audio commentary, text cards, high-contrast visual cues, calmer interface modes, and Braille-ready output. Instead of giving every fan the same broadcast experience, CupVoice helps each fan follow the game in the way that works best for them. Same match. Different ways to follow it. Every fan included. 🧠

What it does

CupVoice transforms live-style football match events into multiple accessibility-focused outputs. Instead of showing the same fast-paced broadcast to everyone, our prototype adapts each event to different user needs. A single match event can become: 🎧 Spatial audio commentary for blind and low-vision fans 📝 Dynamic captions and event cards for deaf and hard-of-hearing fans 🎨 High-contrast, pattern-first visuals for color-blind users ⠿ Braille-ready structured text for users who cannot rely on sight or sound 🌙 A calmer, lower-stimulation mode for sensory-sensitive fans For example, a goal is not just shown as a visual highlight. CupVoice can communicate: who scored where the play happened how the ball moved what changed in the score why the moment matters in the match context For Braille-ready output, the same event is compressed into a short structured line such as: 26:14 | ARG GOAL | Messi | right box | low shot | ARG 1-0 FRA This makes the information easier to send to a refreshable Braille display or a Braille simulator. 🏗️

How we built it

We built CupVoice around a structured live match event pipeline powered by the Claude API and Deepgram API. Instead of trying to process a full sports broadcast video feed during the hackathon, we focused on a more reliable MVP: simulated live match events. Each event contains structured fields such as: time team player event type location score impact match context These structured events are sent to the Claude API, which translates the same play into different accessibility-focused outputs, including beginner-friendly explanations, audio commentary text, caption-ready summaries, and Braille-ready structured lines. For voice output, we use the Deepgram API to convert the generated commentary into natural spoken audio. This allows CupVoice to turn the same match event into a format that can be heard, read, displayed visually, or sent toward Braille-ready hardware. In short, our pipeline is: Simulated match event → structured event data → Claude accessibility translation → Deepgram voice output → accessible UI / captions / Braille-ready output 🔁 MVP Architecture CupVoice Technical Architecture: User Profile Selection → Simulated Match Event Feed → Structured Event Schema → AI Accessibility Translation Layer → Audio Commentary / Caption Cards / Visual Cues / Braille-Ready Text / ESP32 Hardware Output ⚙️ System Flow 📡 Simulated Match Feed We input football-style live events such as goals, passes, substitutions, and key momentum shifts. 📡 Simulated Match Feed We input football-style live events such as goals, passes, substitutions, and key momentum shifts. 🧩 Structured Event Schema Each event is represented in a predictable format with time, team, player, location, event type, and score impact. 🧩 Structured Event Schema Each event is represented in a predictable format with time, team, player, location, event type, and score impact. 🤖 AI Accessibility Translation Layer The system translates the same event into modality-specific outputs such as audio commentary, caption cards, visual cues, and Braille-ready text. 🤖 AI Accessibility Translation Layer The system translates the same event into modality-specific outputs such as audio commentary, caption cards, visual cues, and Braille-ready text. 📱 Accessible Fan Interface Users interact with the output mode that best fits their accessibility profile. 📱 Accessible Fan Interface Users interact with the output mode that best fits their accessibility profile. 🛠️ Prototype Components Our MVP is designed around the following components: simulated match event feed accessibility profile selection accessible event cards pattern-first visual design Braille-ready text output / simulator multi-modal accessibility translation logic 💻 Tech Stack Frontend: React Backend / AI layer: Node.js + Express with Anthropic Claude API Voice layer: Deepgram TTS with browser Web Speech API fallback Storage / memory: In-memory simulated event feed for the MVP Braille layer: Braille-ready formatted text protocol with ESP32 hardware bridge using Web Serial API 🧗

Challenges we ran into

One of our biggest challenges was scope. Accessibility is a broad space, and CupVoice could easily have become just a concept deck if we tried to build everything at once. We had to narrow the project into a realistic hackathon MVP while still showing the core idea: one live match event → multiple accessible outputs. We also had to design for users with very different needs. A blind fan, a deaf fan, a deafblind fan, a color-blind fan, and a sensory-sensitive fan do not need the same interface, the same pacing, or the same information density. Another key challenge was balancing speed and clarity. Live sports updates must be fast, but accessibility outputs cannot be vague or overloaded. This was especially important for the Braille-ready stream, where concise, structured information is much more useful than long-form commentary. Finally, because this was a hackathon, we had to simulate the sports feed and focus on the translation layer itself rather than building a full live broadcast ingestion pipeline. 🏆

Accomplishments we're proud of

We are proud that we turned a broad social impact idea into a concrete, demo-able prototype. Instead of only saying that sports should be more inclusive, we built a system that demonstrates how the same match event can become different accessible experiences for different users. We are especially proud of the Braille-ready pathway. Many accessibility tools stop at captions or audio, but CupVoice also considers users who may not be able to rely on either sight or sound. By creating a structured text protocol, we show how live sports could become more accessible through tactile interfaces as well. We are also proud of designing CupVoice as an example of universal design. It is not a separate simplified version of the game — it is the same match, translated into the format that works best for each fan. 📚

What we learned

We learned that accessibility is not just one feature — it is a full product design challenge. Different users need different sensory channels, timing, structure, and levels of detail. A good accessibility system should not force everyone into the same experience. It should adapt. We also learned that structured data is incredibly powerful for accessible AI systems. Once match events are represented clearly, they can be translated much more reliably across audio, captions, visual cues, and Braille-ready output. Most importantly, we learned that social impact projects need both empathy and execution. A meaningful idea matters, but the demo needs to clearly show how the user experience changes. 🚀

What's next

🔧 Improve the hardware prototype Our current hardware setup is still an early prototype. Next, we would improve the stability, portability, and compatibility of the hardware layer so that CupVoice can better support real assistive devices, including refreshable Braille displays and Bluetooth-based accessibility tools. 🌐 Add multilingual commentary Live sports are global, and accessibility should not be limited to one language. We would expand CupVoice to support multilingual audio commentary, captions, and Braille-ready text so that international fans can follow the match in the language that works best for them. 🧩 Support more sensory profiles and personalization Different fans have different accessibility needs. Next, we would add more sensory profiles and personalization settings, allowing users to adjust commentary detail, caption density, visual contrast, pacing, audio intensity, and Braille-ready update formats. Our long-term vision is simple: every fan should be able to experience the same match through the sensory path that works best for them.

🎙️ CupVoice - Accessible Sports Commentary System

Making live sports accessible for blind, deaf, deafblind, color-blind, and sensory-sensitive fans through AI-powered multi-modal commentary.

🌍 Inspiration

Live sports are meant to be a shared experience, but the way they are broadcast today is not equally accessible to every fan.

Football matches move fast and rely heavily on sight, sound, color, motion, crowd noise, and dense visual graphics. For blind, low-vision, deaf, hard-of-hearing, deafblind, color-blind, and sensory-sensitive fans, the same match can become fragmented, overwhelming, or inaccessible.

We were inspired by a simple question:

What if the same live match could be translated into the sensory format that works best for each fan?

That idea led us to build CupVoice — a multi-modal accessibility layer for live sports broadcasts.

🧠 What it does

CupVoice transforms live-style football match events into multiple accessibility-focused outputs.

Instead of showing the same fast-paced broadcast to everyone, our prototype adapts each event to different user needs. A single match event can become:

🎧 AI-powered audio commentary for blind and low-vision fans (powered by Claude API + Deepgram TTS)
📝 Dynamic captions and event cards for deaf and hard-of-hearing fans
⠿ Braille-ready structured text + hardware output for deafblind users (ESP32 via Web Serial API)
👶 Beginner-friendly explanations for new fans unfamiliar with sports terminology
🎨 High-contrast Apple-inspired UI with accessible color palette and typography

For example, a goal is not just shown as a visual highlight. CupVoice can communicate:

who scored
where the play happened
how the ball moved
what changed in the score
why the moment matters in the match context

For Braille-ready output, the same event is compressed into a short structured line such as:

26:14 | ARG GOAL | Messi | right box | low shot | ARG 1-0 FRA

This is automatically sent to a refreshable Braille display via ESP32 hardware over USB.

🏗️ How we built it

We built CupVoice around a structured live match event pipeline.

Instead of trying to process a full sports broadcast video feed during the hackathon, we focused on a more reliable MVP: simulated live match events. Each event contains structured fields such as:

time
team
player
event type
location
score impact
match context

These structured events are then translated into multiple accessibility outputs.

🔁 System Architecture

┌─────────────────────────────────────────────────────────────┐
│                    CupVoice Architecture                     │
└─────────────────────────────────────────────────────────────┘

┌─────────────────┐
│  User Profile   │  Select: Blind / Deaf / Beginner
│   Selection     │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────┐
│              Simulated Match Event Feed                      │
│  (Soccer, Basketball, American Football live-style events)  │
└────────┬────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────┐
│         AI Commentary Generation (Claude API)                │
│  • Beginner mode: Simple, calm, jargon-free explanations    │
│  • Technical mode: Tactical analysis and formations         │
│  • Emotional mode: Energetic, passionate commentary         │
└────────┬────────────────────────────────────────────────────┘
         │
         ├────────────────────┬──────────────────┬─────────────┐
         ▼                    ▼                  ▼             ▼
┌──────────────┐    ┌──────────────┐    ┌─────────────┐  ┌──────────┐
│ Audio Output │    │ Visual Cards │    │   Braille   │  │ Captions │
│ (Deepgram/   │    │ (Real-time   │    │  Hardware   │  │ (Screen) │
│  Browser TTS)│    │  React UI)   │    │  (ESP32 +   │  │          │
│              │    │              │    │  Web Serial)│  │          │
└──────────────┘    └──────────────┘    └─────────────┘  └──────────┘

⚙️ System Flow

📡 Simulated Match Feed
We input football-style live events such as goals, passes, substitutions, and key momentum shifts every 10-15 seconds.
🧩 Structured Event Schema
Each event is represented in a predictable format with time, team, player, location, event type, and score impact.
🤖 AI Accessibility Translation Layer (Claude API)
The system uses Anthropic Claude Sonnet 4.5 to translate the same event into modality-specific outputs:
- Beginner Fan mode: Short, calm, jargon-free explanations
- Technical mode: Tactical analysis with formations
- Emotional mode: Energetic, passionate commentary
🔊 Text-to-Speech (Deepgram TTS)
Commentary text is converted to natural speech using Deepgram's Aura-Asteria voice model (fallback: browser Web Speech API).
⠿ Braille Hardware Output (ESP32 + Web Serial API)
Structured Braille data is sent to ESP32 microcontroller via USB using Chrome's Web Serial API for physical Braille display control.
📱 Accessible Fan Interface (React + Apple Design)
Users interact with a clean, high-contrast UI inspired by Apple's design system with pill buttons, soft shadows, and SF Pro typography.

💻 Tech Stack

Frontend

React 18 - Component-based UI
Axios - HTTP client for API calls
Web Speech API - Browser-native text-to-speech fallback
Web Serial API - Direct USB communication with ESP32 hardware
Apple-inspired CSS - SF Pro font stack, Action Blue (#0066cc), rounded pill buttons

Backend

Node.js + Express - REST API server
Anthropic Claude API - AI commentary generation (Sonnet 4.5)
Deepgram SDK - Text-to-speech voice synthesis (Aura-Asteria model)
WebSocket (ws) - Real-time hardware event broadcasting
dotenv - Environment variable management

Hardware Integration

ESP32 Microcontroller - Braille display controller
Web Serial API - Browser-to-hardware USB communication at 115200 baud
Arduino C++ - ESP32 firmware for serial data parsing

AI & Voice

Claude Sonnet 4.5 (claude-sonnet-4-20250514) - Natural language commentary generation
Deepgram TTS (aura-asteria-en model) - High-quality voice synthesis
Ollama (optional) - Local AI alternative for offline mode

Accessibility Features

Braille 6-dot system - Standard Braille encoding (dots 1-6)
ARIA labels - Full screen reader support
Keyboard navigation - Tab-accessible UI
High contrast mode - @media (prefers-contrast: high) support
Reduced motion - @media (prefers-reduced-motion) support

Deployment

Development: npm run dev (concurrent client + server)
Production: Static build + Node.js backend
Environment: localhost:3000 (client), localhost:3001 (server)

🚀 Installation & Setup

Prerequisites

# Required
Node.js 18+
npm 9+
Chrome or Edge browser (for Web Serial API)

# Optional
ESP32 microcontroller + USB cable
Arduino IDE (for ESP32 firmware upload)

1. Clone Repository

git clone https://github.com/Abulala1/cupvoice.git
cd cupvoice

2. Install Dependencies

npm run install-all

3. Configure Environment Variables

# Create .env file in root directory
cp .env.example .env

# Add your API keys
CLAUDE_API_KEY=sk-ant-xxxxx
DEEPGRAM_API_KEY=xxxxx
USE_MOCK=false
USE_OLLAMA=false
PORT=3001

4. Start Development Server

npm run dev

This will start:

Backend: http://localhost:3001
Frontend: http://localhost:3000

5. (Optional) Connect ESP32 Hardware

See ESP32_WEB_SERIAL_GUIDE.md for full hardware setup instructions.

# Quick test
1. Upload Arduino sketch from guide
2. Connect ESP32 via USB
3. Open http://localhost:3000 in Chrome
4. Click "Connect ESP32"
5. Select "Blind / Braille User" profile
6. Start auto-commentary
7. Watch Serial Monitor for JSON data

🎮 Usage

For Blind / Low-Vision Users

Select "Blind / Braille User" profile
Choose a sport (Soccer, Basketball, American Football)
Commentary will automatically play via audio every 10-15 seconds
Optionally connect ESP32 for Braille hardware output

For Deaf / Hard of Hearing Users

Select "Deaf / Hard of Hearing" profile
Audio is automatically disabled
Visual commentary cards show event descriptions
Clear text labels and icons

For Beginner Fans

Select "Beginner Fan" profile
Commentary uses simple, jargon-free language
Technical terms are explained immediately
Calm, supportive tone

🧗 Challenges we ran into

One of our biggest challenges was scope.

Accessibility is a broad space, and CupVoice could easily have become just a concept deck if we tried to build everything at once. We had to narrow the project into a realistic hackathon MVP while still showing the core idea:
one live match event → multiple accessible outputs.

We also had to design for users with very different needs. A blind fan, a deaf fan, a deafblind fan, and a sensory-sensitive fan do not need the same interface, the same pacing, or the same information density.

Another key challenge was balancing speed and clarity. Live sports updates must be fast, but accessibility outputs cannot be vague or overloaded. This was especially important for the Braille-ready stream, where concise, structured information is much more useful than long-form commentary.

Technical challenges:

Integrating Web Serial API for direct USB hardware communication
Designing Apple-style UI while maintaining WCAG AAA accessibility
Creating beginner-friendly AI prompts that avoid sports jargon
Handling Deepgram TTS failures gracefully with browser TTS fallback

Finally, because this was a hackathon, we had to simulate the sports feed and focus on the translation layer itself rather than building a full live broadcast ingestion pipeline.

🏆 Accomplishments that we're proud of

We are proud that we turned a broad social impact idea into a concrete, demo-able prototype.

Instead of only saying that sports should be more inclusive, we built a system that demonstrates how the same match event can become different accessible experiences for different users.

Key achievements:

✅ Full hardware integration - ESP32 Braille display via Web Serial API
✅ Multi-modal AI - Claude API generates 3 different commentary styles
✅ Production-quality UI - Apple-inspired design with 95% design guideline coverage
✅ Real accessibility - Screen reader support, keyboard navigation, ARIA labels
✅ Beginner-friendly AI - Custom prompts that explain sports without jargon

We are especially proud of the Braille-ready pathway. Many accessibility tools stop at captions or audio, but CupVoice also considers users who may not be able to rely on either sight or sound. By creating a structured text protocol and Web Serial integration, we show how live sports could become more accessible through tactile interfaces as well.

We are also proud of designing CupVoice as an example of universal design. It is not a separate simplified version of the game — it is the same match, translated into the format that works best for each fan.

📚 What we learned

We learned that accessibility is not just one feature — it is a full product design challenge.

Different users need different sensory channels, timing, structure, and levels of detail. A good accessibility system should not force everyone into the same experience. It should adapt.

We also learned that structured data is incredibly powerful for accessible AI systems. Once match events are represented clearly, they can be translated much more reliably across audio, captions, visual cues, and Braille-ready output.

Technical learnings:

Web Serial API enables powerful browser-to-hardware communication without native apps
Claude API's prompt engineering is critical for accessibility - small wording changes significantly impact user experience
Apple's design system principles (minimal shadows, pill buttons, SF Pro typography) can coexist with accessibility requirements
Browser TTS (Web Speech API) is a reliable fallback when cloud TTS fails

Most importantly, we learned that social impact projects need both empathy and execution. A meaningful idea matters, but the demo needs to clearly show how the user experience changes.

🚀 What's next for CupVoice

🔧 Improve the hardware prototype

Our current hardware setup is still an early prototype. Next, we would improve the stability, portability, and compatibility of the hardware layer so that CupVoice can better support real assistive devices, including refreshable Braille displays and Bluetooth-based accessibility tools.

🌐 Add multilingual commentary

Live sports are global, and accessibility should not be limited to one language. We would expand CupVoice to support multilingual audio commentary, captions, and Braille-ready text so that international fans can follow the match in the language that works best for them.

🧩 Support more sensory profiles and personalization

Different fans have different accessibility needs. Next, we would add more sensory profiles and personalization settings, allowing users to adjust commentary detail, caption density, visual contrast, pacing, audio intensity, and Braille-ready update formats.

📹 Real-time video processing

Integrate computer vision (e.g., MediaPipe, YOLO) to extract live match events directly from broadcast video feeds instead of simulated events.

🎯 Deepgram Voice Agent integration

Add real-time conversational AI using Deepgram's Voice Agent API so users can ask questions like "What just happened?" or "Who's winning?" and get instant beginner-friendly answers.

🌍 Partner with sports organizations

Work with FIFA, UEFA, NBA, NFL to integrate CupVoice into official broadcast platforms and stadium experiences.

Our long-term vision is simple: every fan should be able to experience the same match through the sensory path that works best for them.

📂 Project Structure

cpuvoice/
├── client/                 # React frontend
│   ├── src/
│   │   ├── App.js         # Main React component (profiles, commentary, ESP32)
│   │   ├── App.css        # Apple-inspired accessibility CSS
│   │   └── index.js       # React entry point
│   ├── public/
│   └── package.json
├── server/                # Node.js backend
│   ├── index.js          # Express server + WebSocket
│   ├── services/
│   │   ├── commentary.js           # Claude API commentary generation
│   │   ├── auto-commentary-mock.js # Simulated live events
│   │   └── tts.js                  # Deepgram TTS integration
│   └── package.json
├── ESP32_WEB_SERIAL_GUIDE.md  # Hardware setup instructions
├── FILES_OVERVIEW.md           # Codebase documentation
├── README.md                   # This file
├── .env.example               # Environment template
└── package.json               # Root package (scripts for dev/install)

🤝 Contributing

Contributions are welcome! Please read CONTRIBUTING.md for guidelines.

Areas we'd love help with:

🌐 Multilingual support (Spanish, French, Arabic, Chinese)
♿ Additional accessibility profiles (low vision, cognitive disabilities)
🎮 More sports (Tennis, Cricket, Rugby, Hockey)
🔧 Hardware integrations (other Braille displays, haptic devices)
📱 Mobile app (React Native port)

📜 License

MIT License - see LICENSE file for details.

🙏 Acknowledgments

Anthropic for Claude API access
Deepgram for TTS voice synthesis
Apple for design system inspiration
Web Serial API community for hardware integration resources
Accessibility advocates who inspired this project

📧 Contact

GitHub: @Abulala1
Project: cupvoice
Demo: Live Demo Link (coming soon)

Built with ❤️ for accessible sports experiences

Because every fan deserves to experience the game.

Analysis

Compare with all teams

View

Metric

ella
1

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

AnthropicIn code
ExpressIn code
JavaScriptIn code
Node.jsClaimed
ReactClaimed

3 of 5 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

27 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.