Project Info

BlindTube

Inspiration

Watching movies with friends who are blind made us realize how much visual detail never makes it into the usual audio-description tracks. We wanted to build something that turns any video—whether it’s a blockbuster, a vlog, or a Saturday-morning cartoon—into a richly narrated audiobook so that blind and low-vision audiences can enjoy the full story without waiting for an official audio-described release.

What it does

Takes a YouTube URL link or file Extracts the core story – Gemini’s API with video parsing summarizes plot points, scene changes, character actions, and dialogue. Generates natural-language narration that links scenes together smoothly. Adds emotion and vocal variety – We pass the text through Hume.ai and ElevenLabs to produce a ready-to-listen audiobook track. Outputs a single MP3 (or WAV) file you can play on any device or splice back into the original video as an alternate audio track.

How we built it

Backend: Python + Django handles video and audio files. Video parsing: Gemini Vision extracts frame-level captions and scene metadata. Narration engine and Text-To-Speech (TTS): Hume.ai and ElevenLabs converts the tagged script to high-quality speech.

Challenges we ran into

Keeping the story engaging and detailed – We tweaked prompts to for LLM to make the story as interesting as possible. API rate limits – We Implemented the best performing Text-to-Speech LLM, but it is too expensive to utilize. Emotion markup standards – Hume and ElevenLabs use different tags, so we built a small mapping layer. Implementation of Different APIs for TST – We Implemented google, Hume, and ElevenLabs to test which performs the best. We discovered Hume and ElevenLabs performs the best.

Accomplishments we're proud of

Turned a 5-minutes cartoon into a highly engaging audiobook. End-to-end pipeline (upload → MP3).

What we learned

Good narration is about context, not just describing every frame. Voice synthesis APIs are powerful, but emotion cues make or break the final experience. Accessibility tools benefit people who wanted to listen the audiobooks and don't have the time to watch the long movie.

What's next

Multi-voice casting (different speakers for characters and narrator). Mobile app with AirPods-friendly playback controls. Lower cost cheaper API calls by training and fine-tuning our own model.

BlindTube-mini 🎥 ➡️ 🎧

BlindTube is an innovative platform that transforms visual content into rich audio experiences, making movies, entertainment videos, and cartoons accessible to visually impaired individuals. By combining advanced AI technologies, we create immersive audiobook-style narratives from video content.

This project is conducted by [Jianyu Hou] (https://github.com/houjer23), [Simin Fan] (https://github.com/Olivia-fsm), and [Luoyi Zhang] (https://github.com/louisazz). This project continues from BlindTube.

🌟 Features

Video to narrative conversion using Google's Gemini AI
Emotional context analysis with Hume.ai
High-quality voice synthesis using ElevenLabs
Dynamic background music selection based on scene context
Web interface for easy content management
Support for various video formats

🚀 Getting Started

Prerequisites

Python
API keys for:
- Google Gemini AI
- ElevenLabs
- Hume.ai

Installation

Clone the repository:

git clone [your-repository-url]
cd BlindTube

Create and activate a virtual environment:

python3 -m venv
source venv/bin/activate  # On Windows, use: venv\Scripts\activate

Install dependencies:

pip install -r requirements.txt

Set up your environment variables: Create a .env file in the root directory with:

ELEVENLABS_API_KEY=your_elevenlabs_key
GOOGLE_API_KEY=your_google_key
HUME_API_KEY=your_hume_key

Initialize the database:

python manage.py migrate

Run the development server:

python manage.py runserver

The application will be available at http://localhost:8000

🎯 How It Works

Video Processing:
- Videos are processed and analyzed frame by frame
- Key scenes and moments are identified
- Visual content is converted into descriptive narratives
AI Enhancement:
- Gemini AI transforms visual content into engaging stories
- Hume.ai analyzes emotional context
- ElevenLabs converts text to natural-sounding speech
Audio Production:
- Dynamic background music selection
- Professional-grade audio mixing
- Seamless narrative flow

📁 Project Structure

BlindTube/
├── audio_processor.py         # Audio processing and mixing
├── background_music/         # Background music assets
├── descriptions/            # Django app for managing descriptions
├── text_to_speech_*.py     # Various TTS implementations
└── video_processing.py     # Video analysis and processing

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

🙏 Acknowledgments

Google Gemini AI for video understanding
ElevenLabs and Hume.ai for Text-To-Speech

Analysis

Compare with all teams

View

Metric

Olivia-fsm
8
Jianyu Hou
3

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

DjangoIn code
HTMLIn code
PythonIn code
Google GeminiClaimed
JavaScriptClaimed

3 of 5 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

124 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.