# Project export: BlindTube This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: From screen to sound—turn any video into a vivid audiobook for the blind. - Devpost: https://devpost.com/software/blindtube - GitHub: https://github.com/Olivia-fsm/BlindTube-mini - Demo: https://docs.google.com/presentation/d/1jCdTS7T0tAY3xa4zu-90_wEtgydGVnChXhImAfyI-tk/edit?usp=sharing - Video: https://www.youtube.com/embed/SMK9wzM1mGA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Olivia-fsm (8 commits), Jianyu Hou (3 commits) ## Devpost submission (written by the team) ### Inspiration Watching movies with friends who are blind made us realize how much visual detail never makes it into the usual audio-description tracks. We wanted to build something that turns any video—whether it’s a blockbuster, a vlog, or a Saturday-morning cartoon—into a richly narrated audiobook so that blind and low-vision audiences can enjoy the full story without waiting for an official audio-described release. ### What it does Takes a YouTube URL link or file Extracts the core story – Gemini’s API with video parsing summarizes plot points, scene changes, character actions, and dialogue. Generates natural-language narration that links scenes together smoothly. Adds emotion and vocal variety – We pass the text through Hume.ai and ElevenLabs to produce a ready-to-listen audiobook track. Outputs a single MP3 (or WAV) file you can play on any device or splice back into the original video as an alternate audio track. ### How we built it Backend: Python + Django handles video and audio files. Video parsing: Gemini Vision extracts frame-level captions and scene metadata. Narration engine and Text-To-Speech (TTS): Hume.ai and ElevenLabs converts the tagged script to high-quality speech. ### Challenges we ran into Keeping the story engaging and detailed – We tweaked prompts to for LLM to make the story as interesting as possible. API rate limits – We Implemented the best performing Text-to-Speech LLM, but it is too expensive to utilize. Emotion markup standards – Hume and ElevenLabs use different tags, so we built a small mapping layer. Implementation of Different APIs for TST – We Implemented google, Hume, and ElevenLabs to test which performs the best. We discovered Hume and ElevenLabs performs the best. ### Accomplishments we're proud of Turned a 5-minutes cartoon into a highly engaging audiobook. End-to-end pipeline (upload → MP3). ### What we learned Good narration is about context, not just describing every frame. Voice synthesis APIs are powerful, but emotion cues make or break the final experience. Accessibility tools benefit people who wanted to listen the audiobooks and don't have the time to watch the long movie. ### What's next Multi-voice casting (different speakers for characters and narrator). Mobile app with AirPods-friendly playback controls. Lower cost cheaper API calls by training and fine-tuning our own model. ## README (from the GitHub repository) # BlindTube-mini 🎥 ➡️ 🎧 BlindTube is an innovative platform that transforms visual content into rich audio experiences, making movies, entertainment videos, and cartoons accessible to visually impaired individuals. By combining advanced AI technologies, we create immersive audiobook-style narratives from video content. This project is conducted by [Jianyu Hou] (https://github.com/houjer23), [Simin Fan] (https://github.com/Olivia-fsm), and [Luoyi Zhang] (https://github.com/louisazz). This project continues from [BlindTube](https://github.com/Olivia-fsm/BlindTube). ## 🌟 Features - Video to narrative conversion using Google's Gemini AI - Emotional context analysis with Hume.ai - High-quality voice synthesis using ElevenLabs - Dynamic background music selection based on scene context - Web interface for easy content management - Support for various video formats ## 🚀 Getting Started ### Prerequisites - Python - API keys for: - Google Gemini AI - ElevenLabs - Hume.ai ### Installation 1. Clone the repository: ```bash git clone [your-repository-url] cd BlindTube ``` 2. Create and activate a virtual environment: ```bash python3 -m venv source venv/bin/activate # On Windows, use: venv\Scripts\activate ``` 3. Install dependencies: ```bash pip install -r requirements.txt ``` 4. Set up your environment variables: Create a `.env` file in the root directory with: ``` ELEVENLABS_API_KEY=your_elevenlabs_key GOOGLE_API_KEY=your_google_key HUME_API_KEY=your_hume_key ``` 5. Initialize the database: ```bash python manage.py migrate ``` 6. Run the development server: ```bash python manage.py runserver ``` The application will be available at `http://localhost:8000` ## 🎯 How It Works 1. **Video Processing**: - Videos are processed and analyzed frame by frame - Key scenes and moments are identified - Visual content is converted into descriptive narratives 2. **AI Enhancement**: - Gemini AI transforms visual content into engaging stories - Hume.ai analyzes emotional context - ElevenLabs converts text to natural-sounding speech 3. **Audio Production**: - Dynamic background music selection - Professional-grade audio mixing - Seamless narrative flow ## 📁 Project Structure ``` BlindTube/ ├── audio_processor.py # Audio processing and mixing ├── background_music/ # Background music assets ├── descriptions/ # Django app for managing descriptions ├── text_to_speech_*.py # Various TTS implementations └── video_processing.py # Video analysis and processing ``` ## 🤝 Contributing Contributions are welcome! Please feel free to submit a Pull Request. ## 🙏 Acknowledgments - Google Gemini AI for video understanding - ElevenLabs and Hume.ai for Text-To-Speech ## Detected evidence (automated analysis) Indexed codebase: 33 recognized source files, 124 KB. - Django (technology) — detected in the code - HTML (language) — detected in the code - Python (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - JavaScript (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (35 of 35) ``` app.html audio_outputs/__init__.py audio_processor.py background_music/ATTRIBUTION.md background_music/README.md blindtube/__init__.py blindtube/asgi.py blindtube/settings.py blindtube/urls.py blindtube/wsgi.py db.sqlite3 description_detail.html descriptions/__init__.py descriptions/admin.py descriptions/migrations/__init__.py descriptions/migrations/0001_initial.py descriptions/migrations/0002_audiodescription_audio_url.py descriptions/models_copy.py descriptions/models.py descriptions/serializers.py descriptions/templates/descriptions.html descriptions/tests.py descriptions/urls.py descriptions/views_copy.py descriptions/views.py download_music.py manage.py media/videos/__init__.py README.md requirements.txt text_to_speech_eleven.py text_to_speech_factory.py text_to_speech_google.py text_to_speech_hume.py video_processing.py ``` ### Dependencies - requirements.txt: django, djangorestframework, elevenlabs@>=0.3.0, ffmpeg-python, ffprobe, google-generativeai, gtts, gTTS@>=2.5.0, hume, nltk@>=3.8.1, opencv-python, pydub@>=0.25.1, python-dotenv@==1.0.0, tenacity@>=8.2.3, yt-dlp ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - Add files via upload - Create __init__.py - Add files via upload - Create __init__.py - Add files via upload - Add files via upload - Add files via upload - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### background_music/ATTRIBUTION.md ```markdown # Music Attribution This directory contains the following royalty-free music tracks: ## the_entertainer.mp3 Category: comedy Attribution: The Entertainer by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0 ## monkeys_spinning_monkeys.mp3 Category: comedy Attribution: Monkeys Spinning Monkeys by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0 ## circus_tent.mp3 Category: comedy Attribution: Circus Tent by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0 ## sneaky_snitch.mp3 Category: chase Attribution: Sneaky Snitch by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0 ## scheming_weasel.mp3 Category: chase Attribution: Scheming Weasel by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0 ## merry_go.mp3 Category: chase Attribution: Merry Go by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0 ## sneaky_adventure.mp3 Category: dramatic Attribution: Sneaky Adventure by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0 ## hidden_agenda.mp3 Category: dramatic Attribution: Hidden Agenda by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0 ## investigation.mp3 Category: dramatic Attribution: Investigation by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0 ``` ### requirements.txt ``` elevenlabs==0.2.26 python-dotenv==1.0.0 opencv-python google-generativeai pydub>=0.25.1 yt-dlp django djangorestframework gtts ffprobe ffmpeg-python hume nltk>=3.8.1 tenacity>=8.2.3 gTTS>=2.5.0 elevenlabs>=0.3.0 ``` ### manage.py ```python #!/usr/bin/env python """Django's command-line utility for administrative tasks.""" import os import sys def main(): """Run administrative tasks.""" os.environ.setdefault("DJANGO_SETTINGS_MODULE", "blindtube.settings") try: from django.core.management import execute_from_command_line except ImportError as exc: raise ImportError( "Couldn't import Django. Are you sure it's installed and " "available on your PYTHONPATH environment variable? Did you " "forget to activate a virtual environment?" ) from exc execute_from_command_line(sys.argv) if __name__ == "__main__": main() ``` ### text_to_speech_google.py ```python import os from pathlib import Path from gtts import gTTS from typing import Optional import uuid class GoogleTTS: """Google Text-to-Speech implementation""" def text_to_speech(self, text: str, output_dir: str = "audio_outputs", filename: Optional[str] = None) -> str: """ Convert text to speech using Google Text-to-Speech Args: text (str): The text to convert to speech output_dir (str): Directory to store the audio output filename (str, optional): Optional filename for the output file Returns: str: Path to the generated audio file """ if not text: raise ValueError("Text is empty") # Create output directory if it doesn't exist output_path = Path(output_dir) output_path.mkdir(exist_ok=True) # Create output filename if filename is None: filename = f"{uuid.uuid4()}_audio.mp3" output_file = output_path / filename # Generate audio using gTTS tts = gTTS(text=text, lang='en', slow=False) tts.save(str(output_file)) return str(output_file) if __name__ == "__main__": # Example usage input_file = "/Users/jianyuhou/Downloads/video_description.txt" try: output_file = text_to_speech(input_file) print(f"Audio file generated successfully: {output_file}") except Exception as e: print(f"Error: {str(e)}") ``` ### text_to_speech_eleven.py ```python import os from pathlib import Path from typing import Optional import uuid from elevenlabs import generate, save, set_api_key import requests class ElevenLabsTTS: """Eleven Labs Text-to-Speech implementation""" def __init__(self): # Load API key from environment api_key = os.getenv("ELEVEN_LABS_API_KEY") if not api_key: raise ValueError("Please set ELEVEN_LABS_API_KEY environment variable") set_api_key(api_key) self.api_key = api_key def text_to_speech(self, text: str, output_dir: str = "audio_outputs", filename: Optional[str] = None) -> str: """ Convert text from a file to speech using ElevenLabs Text to Speech API Args: input_file_path (str): Path to the input text file output_dir (str): Directory to store the audio output """ # Create output directory if it doesn't exist output_path = Path(output_dir) output_path.mkdir(exist_ok=True) # ElevenLabs Text to Speech API endpoint url = "https://api.elevenlabs.io/v1/text-to-speech/21m00Tcm4TlvDq8ikWAM" # Default voice ID headers = { "Accept": "audio/mpeg", "Content-Type": "application/json", "xi-api-key": self.api_key } data = { "text": text, "model_id": "eleven_monolingual_v1", "voice_settings": { "stability": 0.5, "similarity_boost": 0.5 } } # Make the API request response = requests.post(url, json=data, headers=headers) if response.status_code != 200: raise Exception(f"API request failed with status code {response.status_code}: {response.text}") # Create output filename if filename is None: filename = f"{uuid.uuid4()}_audio.mp3" output_file = output_path / filename # Save the audio file with open(str(output_file), 'wb') as f: f.write(response.content) return str(output_file) # def text_to_speech(self, text: str, output_dir: str = "audio_outputs", filename: Optional[str] = None) -> str: # """ # Convert text to speech using Eleven Labs Text-to-Speech # Args: # text (str): The text to convert to speech # output_dir (str): Directory to store the audio output # filename (str, optional): Optional filename for the output file # Returns: # str: Path to the generated audio file # """ # if not text: # raise ValueError("Text is empty") # # Create output directory if it doesn't exist # output_path = Path(output_dir) # output_path.mkdir(exist_ok=True) # # Create output filename # if filename is None: # filename = f"{uuid.uuid4()}_audio.mp3" # output_file = output_path / filename # # Generate audio using Eleven Labs # audio = generate( # text=text, # voice="Josh", # Using a default voice, can be made configurable # model="eleven_monolingual_v1" # ) # # Save the audio file # save(audio, str(output_file)) # return str(output_file) # if __name__ == "__main__": # # Example usage # input_file = "/Users/jianyuhou/Downloads/video_description.txt" # try: # output_file = text_to_speech(input_file) # print(f"Audio file generated successfully: {output_file}") # except Exception as e: # print(f"Error: {str(e)}") ``` ### text_to_speech_factory.py ```python from enum import Enum from typing import Optional from text_to_speech_google import GoogleTTS from text_to_speech_eleven import ElevenLabsTTS from text_to_speech_hume import HumeTTS class TTSProvider(Enum): GOOGLE = "google" ELEVEN_LABS = "eleven_labs" HUME = "hume" class TTSFactory: """Factory class to create and manage different TTS providers""" @staticmethod def create_tts(provider: TTSProvider): """ Create a TTS instance based on the specified provider Args: provider (TTSProvider): The TTS provider to use Returns: A TTS instance that implements text_to_speech method Raises: ValueError: If the provider is not supported """ if provider == TTSProvider.GOOGLE: return GoogleTTS() elif provider == TTSProvider.ELEVEN_LABS: return ElevenLabsTTS() elif provider == TTSProvider.HUME: return HumeTTS() else: raise ValueError(f"Unsupported TTS provider: {provider}") @staticmethod def text_to_speech( text: str, provider: TTSProvider, output_dir: str = "audio_outputs", filename: Optional[str] = None ) -> str: """ Convert text to speech using the specified provider Args: text (str): The text to convert to speech provider (TTSProvider): The TTS provider to use output_dir (str): Directory to store the audio output filename (str, optional): Optional filename for the output file Returns: str: Path to the generated audio file Raises: ValueError: If the provider is not supported or if text is empty """ if not text: raise ValueError("Text is empty") tts = TTSFactory.create_tts(provider) return tts.text_to_speech(text, output_dir, filename) def get_recommended_provider(text: str) -> TTSProvider: """ Get recommended TTS provider based on text characteristics Args: text (str): The input text Returns: TTSProvider: Recommended TTS provider """ text_lower = text.lower() # Check text length if len(text) > 5000: return TTSProvider.GOOGLE # Google handles long texts well # Check for technical content technical_keywords = ['feel', 'emotion', 'story', 'experience', 'journey', 'icecream', 'tom', 'jerry', 'code', 'function', 'algorithm', 'technical', 'documentation'] if any(keyword in text_lower for keyword in technical_keywords): return TTSProvider.HUME # Hume handles technical content well # Check for emotional/narrative content emotional_keywords = ['feel', 'emotion', 'story', 'experience', 'journey'] if any(keyword in text_lower for keyword in emotional_keywords): return TTSProvider.ELEVEN_LABS # Eleven Labs is good for emotional content # Default to Hume as it has good general-purpose quality return TTSProvider.HUME # Example usage if __name__ == "__main__": example_text = "This is a test of the text-to-speech system." # Use automatic provider selection recommended_provider = get_recommended_provider(example_text) print(f"Recommended provider: {recommended_provider.value}") try: # Convert text using recommended provider output_file = TTSFactory.text_to_speech( text=example_text, provider=recommended_provider ) print(f"Audio generated successfully: {output_file}") # Or specify a provider manually output_file = TTSFactory.text_to_speech( text=example_text, provider=TTSProvider.HUME, filename="manual_provider_test.mp3" ) print(f"Audio generated successfully: {output_file}") except Exception as e: print(f"Error: {str(e)}") ``` ### download_music.py ```python #!/usr/bin/env python3 import os import requests from pathlib import Path import urllib.parse import logging # Configure logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) def download_file(url: str, output_path: str) -> bool: """ Download a file from URL and save it to output_path. Returns True if successful, False otherwise. """ try: response = requests.get(url, stream=True) response.raise_for_status() with open(output_path, 'wb') as f: for chunk in response.iter_content(chunk_size=8192): f.write(chunk) return True except Exception as e: logger.error(f"Error downloading {url}: {str(e)}") return False def setup_music_directories(): """Create music category directories if they don't exist.""" base_dir = Path("background_music") categories = ['chase', 'comedy', 'dramatic'] for category in categories: (base_dir / category).mkdir(parents=True, exist_ok=True) return base_dir def main(): base_dir = setup_music_directories() # List of royalty-free music tracks to download tracks = [ # Comedy/Light-hearted tracks (Tom and Jerry style) { 'name': 'monkeys_spinning_monkeys.mp3', 'url': 'https://incompetech.com/music/royalty-free/mp3-royaltyfree/Monkeys%20Spinning%20Monkeys.mp3', 'category': 'comedy', 'attribution': 'Monkeys Spinning Monkeys by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0' }, { 'name': 'circus_tent.mp3', 'url': 'https://incompetech.com/music/royalty-free/mp3-royaltyfree/Circus%20Tent.mp3', 'category': 'comedy', 'attribution': 'Circus Tent by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0' }, { 'name': 'sneaky_snitch.mp3', 'url': 'https://incompetech.com/music/royalty-free/mp3-royaltyfree/Sneaky%20Snitch.mp3', 'category': 'comedy', 'attribution': 'Sneaky Snitch by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0' }, # Chase/Action tracks (Fast-paced cartoon style) { 'name': 'chase.mp3', 'url': 'https://incompetech.com/music/royalty-free/mp3-royaltyfree/Chase.mp3', 'category': 'chase', 'attribution': 'Chase by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0' }, { 'name': 'merry_go.mp3', 'url': 'https://incompetech.com/music/royalty-free/mp3-royaltyfree/Merry%20Go.mp3', 'category': 'chase', 'attribution': 'Merry Go by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0' }, { 'name': 'cartoon_battle.mp3', 'url': 'https://incompetech.com/music/royalty-free/mp3-royaltyfree/Cartoon%20Battle.mp3', 'category': 'chase', 'attribution': 'Cartoon Battle by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0' }, # Dramatic/Suspense tracks (For tense moments) { 'name': 'sneaky_adventure.mp3', 'url': 'https://incompetech.com/music/royalty-free/mp3-royaltyfree/Sneaky%20Adventure.mp3', 'category': 'dramatic', 'attribution': 'Sneaky Adventure by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0' }, { 'name': 'hidden_agenda.mp3', 'url': 'https://incompetech.com/music/royalty-free/mp3-royaltyfree/Hidden%20Agenda.mp3', 'category': 'dramatic', 'attribution': 'Hidden Agenda by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0' }, { 'name': 'spy_glass.mp3', 'url': 'https://incompetech.com/music/royalty-free/mp3-royaltyfree/Spy%20Glass.mp3', 'category': 'dramatic', 'attribution': 'Spy Glass by Kevin MacLeod (incompetech.com) - Licensed under Creative Commons: By Attribution 3.0' } ] # Download tracks for track in tracks: output_path = base_dir / track['category'] / track['name'] if output_path.exists(): logger.info(f"Track {track['name']} already exists, skipping...") continue logger.info(f"Downloading {track['name']}...") if download_file(track['url'], str(output_path)): logger.info(f"Successfully downloaded {track['name']}") else: logger.error(f"Failed to download {track['name']}") # Create attribution file attribution_path = base_dir / "ATTRIBUTION.md" with open(attribution_path, "w") as f: f.write("# Music Attribution\n\n") f.write("This directory contains the following royalty-free music tracks:\n\n") for track in tracks: f.write(f"## {track['name']}\n") f.write(f"Category: {track['category']}\n") f.write(f"Attribution: {track['attribution']}\n\n") if __name__ == "__main__": main() ``` ### video_processing.py ```python import cv2 import base64 import os from typing import List, Optional import time import google.generativeai as genai from google.generativeai.types import HarmCategory, HarmBlockThreshold import sys from django.conf import settings class VideoProcessor: def __init__(self, api_key: Optional[str] = None): """Initialize the VideoProcessor with Gemini client.""" self.api_key = api_key or settings.GOOGLE_API_KEY if not self.api_key: raise ValueError("Google API key is required") genai.configure(api_key=self.api_key) self.model = genai.GenerativeModel('models/gemini-2.5-flash') def extract_frames(self, video_path: str, frame_interval: int = 5) -> List[str]: """ Extract frames from a video file and convert them to base64. Args: video_path: Path to the video file frame_interval: Interval between frames to extract (default: 5) Returns: List of base64 encoded frames """ video = cv2.VideoCapture(video_path) base64_frames = [] frame_count = 0 while video.isOpened(): success, frame = video.read() if not success: break # Only keep frames at the specified interval if frame_count % frame_interval == 0: _, buffer = cv2.imencode(".jpg", frame) base64_frames.append(base64.b64encode(buffer).decode("utf-8")) frame_count += 1 video.release() return base64_frames def generate_description(self, frames: List[str]) -> str: """ Generate a description of the video using Gemini Vision. Args: frames: List of base64 encoded frames Returns: Generated description text """ # Convert base64 frames to image parts image_parts = [] for frame in frames: image_parts.append({ "mime_type": "image/jpeg", "data": frame }) # Create the prompt # prompt = """These are frames from a video that I want to upload. # Generate only one compelling description that I can upload along with the # video. Description should describe every detail in the video. This will # be narrated for people who can not see as a story, so it should be very # interesting and engaging. This should be like a book. Start right into the # story. You can descript the scence like a book, but do not anything like The # scene opens on that makes it not like the story.""" prompt = """These are frames from a video that I want to upload. Generate only one lively andcompelling text script that I can upload along with the video. The script should describe every detail in the video. This will be narrated for people who can not see as a lively story and radio drama, so it should be very interesting and engaging. Start right into the story. You can describ the scene like a book, but do not anything like The scene opens on that makes it not like the story. Do not be too long or too short.""" #You can add soound effects #and wrap around with [], like [gunshot], [applause], [clapping], [explosion], #[swallows], [gulps] ... # Generate content using Gemini response = self.model.generate_content( contents=[prompt] + image_parts, generation_config={ "temperature": 0.4, "top_p": 1, "top_k": 32, "max_output_tokens": 10000, }, safety_settings={ HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE, HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE, HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE, HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE, } ) return response.text def main(): """Example usage of the VideoProcessor class.""" if len(sys.argv) != 2: print("Usage: python video_processing.py ") sys.exit(1) video_path = sys.argv[1] if not os.path.exists(video_path): print(f"Error: Video file not found at {video_path}") sys.exit(1) # Initialize the processor processor = VideoProcessor() # Extract frames print("Extracting frames...") frames = processor.extract_frames(video_path, 120) # modify based on the video length print(f"Extracted {len(frames)} frames") # Generate description print("\nGenerating description...") description = processor.generate_description(frames) print("\nDescription:") print(description) # Save the results to a text file output_dir = os.path.dirname(video_path) output_path = os.path.join(output_dir, "video_description.txt") with open(output_path, "w") as f: f.write(description) print(f"\nDescription saved to {output_path}") if __name__ == "__main__": main() ``` ### audio_processor.py ```python from pydub import AudioSegment import os from pathlib import Path import random import re from typing import Optional FFMPEG_PATH = "/opt/homebrew/bin/ffmpeg" FFPROBE_PATH = "/opt/homebrew/bin/ffprobe" AudioSegment.converter = FFMPEG_PATH AudioSegment.ffmpeg = FFMPEG_PATH # AudioSegment.ffprobe = FFPROBE_PATH os.environ['FFMPEG_BINARY'] = FFMPEG_PATH os.environ['FFPROBE_BINARY'] = FFPROBE_PATH class AudioProcessor: def __init__(self, music_dir: str = "background_music"): """Initialize the AudioProcessor with a directory for background music.""" self.music_dir = Path(music_dir) self.music_dir.mkdir(exist_ok=True) print(f"Using ffmpeg at: {AudioSegment.converter}") # print(f"Using ffprobe at: {AudioSegment.ffprobe}") print(f"Environment FFMPEG_BINARY: {os.environ.get('FFMPEG_BINARY')}") print(f"Environment FFPROBE_BINARY: {os.environ.get('FFPROBE_BINARY')}") # Create subdirectories if they don't exist self.categories = ['chase', 'comedy', 'dramatic'] for category in self.categories: (self.music_dir / category).mkdir(exist_ok=True) def _select_music_by_content(self, text: str) -> Optional[str]: """ Select appropriate background music based on text content. Args: text: The narration text to analyze Returns: Path to selected music file or None if no music available """ # Keywords for each category keywords = { 'chase': ['chase', 'run', 'escape', 'catch', 'follow', 'rush', 'speed'], 'comedy': ['funny', 'laugh', 'silly', 'joke', 'prank', 'amusing', 'ridiculous'], 'dramatic': ['dramatic', 'serious', 'intense', 'emotional', 'suspense', 'mystery'] } # Count keyword matches for each category scores = {category: 0 for category in self.categories} text_lower = text.lower() for category, words in keywords.items(): for word in words: scores[category] += len(re.findall(r'\b' + word + r'\b', text_lower)) # Select category with highest score, default to comedy if no matches selected_category = max(scores.items(), key=lambda x: x[1])[0] if any(scores.values()) else 'comedy' # Get available music files for the category music_files = list((self.music_dir / selected_category).glob("*.mp3")) if not music_files: # Fallback to any available music if selected category is empty music_files = [] for category in self.categories: music_files.extend((self.music_dir / category).glob("*.mp3")) if not music_files: return None # No music files available return str(random.choice(music_files)) def mix_audio(self, narration_path: str, narration_text: str = "", output_path: Optional[str] = None, music_volume: float = -20) -> str: """ Mix narration with background music. Args: narration_path: Path to the narration audio file narration_text: Text content of the narration for mood analysis output_path: Path for the output mixed audio file (optional) music_volume: Volume of background music in dB (default: -20) Returns: Path to the mixed audio file """ # Load narration print(f"Loading narration from: {narration_path}") narration = AudioSegment.from_mp3(narration_path) # Select appropriate background music music_path = self._select_music_by_content(narration_text) if narration_text else None if not music_path: # If no text provided or no matching music found, try random selection music_files = [] for category in self.categories: music_files.extend((self.music_dir / category).glob("*.mp3")) if not music_files: return narration_path # Return original narration if no music available music_path = str(random.choice(music_files)) # Load and prepare background music background_music = AudioSegment.from_mp3(music_path) # Loop music if it's shorter than narration while len(background_music) < len(narration): background_music = background_music + background_music # Trim music to match narration length background_music = background_music[:len(narration)] # Add fade in/out effects fade_duration = min(3000, len(background_music) // 2) # 3 seconds or half duration background_music = background_music.fade_in(fade_duration).fade_out(fade_duration) # Adjust music volume and mix background_music = background_music + music_volume mixed_audio = narration.overlay(background_music) # Generate output path if not provided if output_path is None: narration_filename = Path(narration_path).stem output_path = str(Path(narration_path).parent / f"{narration_filename}_with_music.mp3") # Export mixed audio mixed_audio.export(output_path, format="mp3", bitrate="320k", parameters=[ '-codec:a', 'libmp3lame', '-q:a', '0', # Highest quality '-ar', '44100', # Sample rate '-ac', '2', # Stereo '-b:a', '192k' # Bitrate ]) return output_path ``` ### description_detail.html ```html Description Details
Back to List