# Project export: Sonetica This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Where Stories Come to Life - Devpost: https://devpost.com/software/sonetica - GitHub: https://github.com/sgoyal453/SceneCreator - Video: https://www.youtube.com/embed/ueShg9mYx4g?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Vibhor Narang (3 commits), Sarvagya Goyal (1 commits) ## Devpost submission (written by the team) ### Inspiration As creative writers, we’ve always felt that stories, whether they’re fictional pieces, poems, or just everyday journal entries, can showcase an incredible amount of meaning and emotion. But sometimes, just reading them doesn’t capture the full picture we may all visualize. We started wondering: What if a story could not only be read, but actually be seen, heard, and experienced just like a short film? This is why we created Sonetica, a tool that brings writing to life through short, expressive videos. Sonetica helps turn your words into something visual, personal, and alive. Our goal is to make storytelling more immersive, even for everyday moments. ### What it does Sonetica turns your words into short, expressive videos. Whether it’s a story you’ve written, a poem, or even a personal journal entry, you just type it in. Additionally, if you want to, you can add a song that captures the mood. Maybe it’s a track that inspires you, or just something that feels right. From there, Sonetica reads between the lines picking up on the emotion, the mood, and the general theme and uses that to generate an image using Stable Diffusion. That image becomes the heart of a short video that is carefully created by the pipeline to help your story come to life. ### How we built it Here’s how Sonetica’s pipeline works: the user enters their text (short story, journal entry, poem etc) and optionally uploads a song. The audio is analyzed using Librosa to extract features like mood and rhythm. Both the text and audio data are processed and combined to create a detailed prompt using Gemini. This prompt is then fed into Stable Diffusion to generate an image that reflects the emotion, meaning, and themes of the input. Next, the generated image is passed to Veo, which creates an immersive short video. To enhance the audio experience, we leverage Lyria to produce a unique sound that complements the visuals and the user’s chosen song. The frontend is built with React for a smooth user experience, while the backend is powered by Python to handle the processing and coordinate the pipeline. This setup allows us to seamlessly transform written stories and music into vivid, expressive videos. ### Challenges we ran into Implementing Stable Diffusion was one of the toughest parts since none of us had worked with it before. Figuring out the optimal level of complexity for generating images took a lot of experimentation — we wanted to maximize performance without losing the essence and emotion of the user’s input. Integrating audio analysis with text processing was tricky because the audio features we extracted using Librosa sometimes conflicted with the tone suggested by the text. For example, a calm poem paired with an upbeat song created mixed signals that made prompt generation inconsistent. We had to develop heuristics and weighting strategies to balance these inputs so the final prompt truly reflected both the music’s mood and the story’s meaning. On the integration side, a big challenge was figuring out how each API worked and the specific data formats they required. Gemini, Lyria, and Veo all had different inputs and outputs, but we were able to figure out how to make them work seamlessly. Making sure the data flowed smoothly between these tools took careful planning and experimentation. We also had to build fallback mechanisms in case any part of the pipeline failed or caused delays, so the system could still produce a video without crashing or freezing. This made the whole process more reliable and user-friendly despite the complexity behind the scenes. ### Accomplishments we're proud of We’re proud of successfully building a full pipeline that transforms written stories, poems, or journal entries into short, expressive videos combining custom-generated visuals and user-selected music. Despite not having prior experience with Stable Diffusion, we learned how to fine-tune prompts and generate images that truly capture the mood and meaning of the input text. We also managed to integrate multiple complex tools, including Librosa for audio analysis, Gemini and Lyria for prompt creation and sound design, and Veo for video generation, into a smooth, reliable system. We built fallback mechanisms to keep the experience seamless even when parts of the pipeline encountered issues. All of this came together within the tight timeframe of the hackathon, and we’re excited that Sonetica can bring words to life in a new, immersive way. ### What we learned Building Sonetica gave us valuable experience working with multimodal AI by combining text and audio inputs to create meaningful, expressive videos. We learned how to extract and fuse features from written stories and songs, two very different data types, to generate visuals that capture the emotion and mood behind both. Beyond deep learning and AI, we also gained hands-on experience integrating a chatbot interface and developing a responsive frontend using React. This taught us how to design smooth user interactions that connect seamlessly with complex backend AI pipelines. We faced challenges coordinating multiple AI models and APIs while ensuring real-time responsiveness and reliability, which pushed us to develop effective error handling and fallback strategies. ### What's next Moving forward, we want to expand Sonetica’s capabilities to support longer videos and richer storytelling formats, allowing users to create mini-movies from their writings. We’re also excited to explore adding more customization options, like different visual styles or mood filters, so users can better tailor the videos to their unique voices. This can be accomplished through integrated feedback loops for example. In addition, we would like to improve the speed, so generating videos feels instant and seamless. We’d also like to build mobile-friendly versions to make it easier for users to create and share on the go. Finally, we hope to open Sonetica to a wider community, from casual journalers to poets and storytellers, and potentially explore partnerships or integrations with creative platforms to bring storytelling to life in new ways. ## README (from the GitHub repository) # 🤖 Gemini AI Chatbot A beautiful and responsive React application featuring Google's Gemini AI chatbot with a modern light blue theme. ## ✨ Features - 💬 Real-time chat with Google's Gemini AI - 🖼️ **Image Analysis** - Upload and analyze images - 🎵 **Audio Processing** - Upload and analyze audio files - 🎨 Beautiful light blue gradient design - 📱 Fully responsive for all devices - ⚡ Real-time typing indicators - 📎 Drag-and-drop file upload - ✨ Smooth animations and transitions - 🚀 Easy setup and deployment ## 🛠 Prerequisites Before running this application, make sure you have: - **Node.js** (version 14 or higher) - [Download here](https://nodejs.org/) - **npm** (comes with Node.js) - A Google AI API key (already configured in this app) ## 🚀 Quick Start ### Method 1: Automated Setup (Recommended) 1. **Clone or download this project** 2. **Navigate to the project directory:** ```bash cd Scene_Creator ``` 3. **Run the setup script:** ```bash chmod +x setup.sh ./setup.sh ``` 4. **Start the application:** ```bash npm start ``` ### Method 2: Manual Setup 1. **Install dependencies:** ```bash npm install ``` 2. **Start the development server:** ```bash npm start ``` 3. **Open your browser and go to:** ``` http://localhost:3000 ``` ## 🎮 How to Use 1. **Text Chat**: Type a message in the input field and press Enter or click send (📤) 2. **Upload Files**: Click the attach button (📎) to upload images or audio files 3. **Multimodal Analysis**: Upload files with or without text for AI analysis 4. **File Management**: Remove individual files or clear all before sending 5. **Wait for Response**: See typing indicator while Gemini AI processes your request 6. **Continue Conversation**: Chat history is preserved during the session ### 📎 Supported File Types - **Images**: JPG, PNG, GIF, WebP, and other common image formats - **Audio**: MP3, WAV, M4A, and other common audio formats - **File Size**: Up to 20MB per file ## 🎨 Design Features - **Light Blue Theme**: Calming gradient backgrounds and accents - **Modern UI**: Clean, minimalist design with smooth animations - **Responsive**: Works perfectly on desktop, tablet, and mobile devices - **Accessibility**: Keyboard navigation and screen reader friendly - **Visual Feedback**: Loading states, hover effects, and smooth transitions ## 🔧 Technical Stack - **React 18** - Modern React with hooks - **Google Generative AI** - Official Gemini AI SDK - **CSS3** - Custom styling with gradients and animations - **ES6+** - Modern JavaScript features ## 📁 Project Structure ``` gemini-chatbot-app/ ├── public/ │ ├── index.html │ └── favicon.ico ├── src/ │ ├── App.js # Main chatbot component │ ├── App.css # Chatbot styling │ ├── index.js # React entry point │ └── index.css # Global styles ├── package.json # Dependencies and scripts ├── setup.sh # Automated setup script └── README.md # This file ``` ## 🔐 API Configuration The Google Gemini AI API key is already configured in this application. The chatbot uses the `gemini-pro` model for generating responses. ## 🚀 Available Scripts - `npm start` - Runs the app in development mode - `npm build` - Builds the app for production - `npm test` - Launches the test runner - `npm eject` - Ejects from Create React App (not recommended) ## 🐛 Troubleshooting ### Common Issues: 1. **"Module not found" errors** - Make sure you've run `npm install` - Delete `node_modules` folder and `package-lock.json`, then run `npm install` again 2. **API errors** - Check your internet connection - The API key is pre-configured, but ensure it's still valid 3. **Port already in use** - The app runs on port 3000 by default - If port 3000 is busy, React will automatically suggest another port ### Getting Help: If you encounter any issues: 1. Check the browser console for error messages 2. Ensure all dependencies are properly installed 3. Verify that Node.js and npm are up to date ## 🌟 Features in Detail ### Chat Interface - Clean, modern chat bubble design - User messages appear on the right (blue) - AI responses appear on the left (white with blue border) - Automatic scrolling to newest messages ### Loading States - Animated typing indicator while AI is responding - Disabled input during processing - Visual feedback for all user interactions ### Responsive Design - Optimized for mobile devices - Touch-friendly interface - Adaptive layout for different screen sizes ## 🎯 Future Enhancements Potential improvements for this chatbot: - 💾 Chat history persistence - 🌙 Dark mode toggle - 📄 Export chat conversations - 🔄 Regenerate responses - 📎 File upload support - 🎵 Voice input/output - 👥 Multiple conversation threads ## 📝 License This project is open source and available under the MIT License. --- **Enjoy chatting with your Gemini AI assistant! 🤖✨** ## Detected evidence (automated analysis) Indexed codebase: 4467 recognized source files, 66559 KB. - C (language) — detected in the code - CSS (language) — detected in the code - Flask (technology) — detected in the code - Google Gemini (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 5560) ``` .DS_Store .gitignore AUDIO_ANALYSIS_README.md backend/.DS_Store backend/app.py backend/audio_analyzer.py backend/audio_env/.DS_Store backend/audio_env/bin/activate backend/audio_env/bin/activate.csh backend/audio_env/bin/activate.fish backend/audio_env/bin/easy_install backend/audio_env/bin/easy_install-3.7 backend/audio_env/bin/f2py backend/audio_env/bin/f2py3 backend/audio_env/bin/f2py3.7 backend/audio_env/bin/flask backend/audio_env/bin/fonttools backend/audio_env/bin/futurize backend/audio_env/bin/normalizer backend/audio_env/bin/numba backend/audio_env/bin/pasteurize backend/audio_env/bin/pip backend/audio_env/bin/pip3 backend/audio_env/bin/pip3.10 backend/audio_env/bin/pip3.7 backend/audio_env/bin/pycc backend/audio_env/bin/pyftmerge backend/audio_env/bin/pyftsubset backend/audio_env/bin/python backend/audio_env/bin/python3 backend/audio_env/bin/ttx backend/audio_env/lib/.DS_Store backend/audio_env/lib/python3.7/.DS_Store backend/audio_env/lib/python3.7/site-packages/_soundfile_data/COPYING backend/audio_env/lib/python3.7/site-packages/_soundfile_data/libsndfile.dylib backend/audio_env/lib/python3.7/site-packages/_soundfile.py backend/audio_env/lib/python3.7/site-packages/.DS_Store backend/audio_env/lib/python3.7/site-packages/audioread-3.0.1.dist-info/INSTALLER backend/audio_env/lib/python3.7/site-packages/audioread-3.0.1.dist-info/LICENSE backend/audio_env/lib/python3.7/site-packages/audioread-3.0.1.dist-info/METADATA backend/audio_env/lib/python3.7/site-packages/audioread-3.0.1.dist-info/RECORD backend/audio_env/lib/python3.7/site-packages/audioread-3.0.1.dist-info/WHEEL backend/audio_env/lib/python3.7/site-packages/audioread/__init__.py backend/audio_env/lib/python3.7/site-packages/audioread/base.py backend/audio_env/lib/python3.7/site-packages/audioread/exceptions.py backend/audio_env/lib/python3.7/site-packages/audioread/ffdec.py backend/audio_env/lib/python3.7/site-packages/audioread/gstdec.py backend/audio_env/lib/python3.7/site-packages/audioread/macca.py backend/audio_env/lib/python3.7/site-packages/audioread/maddec.py backend/audio_env/lib/python3.7/site-packages/audioread/rawread.py backend/audio_env/lib/python3.7/site-packages/audioread/version.py backend/audio_env/lib/python3.7/site-packages/certifi-2025.6.15.dist-info/INSTALLER backend/audio_env/lib/python3.7/site-packages/certifi-2025.6.15.dist-info/licenses/LICENSE backend/audio_env/lib/python3.7/site-packages/certifi-2025.6.15.dist-info/METADATA backend/audio_env/lib/python3.7/site-packages/certifi-2025.6.15.dist-info/RECORD backend/audio_env/lib/python3.7/site-packages/certifi-2025.6.15.dist-info/top_level.txt backend/audio_env/lib/python3.7/site-packages/certifi-2025.6.15.dist-info/WHEEL backend/audio_env/lib/python3.7/site-packages/certifi/__init__.py backend/audio_env/lib/python3.7/site-packages/certifi/__main__.py backend/audio_env/lib/python3.7/site-packages/certifi/cacert.pem backend/audio_env/lib/python3.7/site-packages/certifi/core.py backend/audio_env/lib/python3.7/site-packages/certifi/py.typed backend/audio_env/lib/python3.7/site-packages/cffi-1.15.1.dist-info/entry_points.txt backend/audio_env/lib/python3.7/site-packages/cffi-1.15.1.dist-info/INSTALLER backend/audio_env/lib/python3.7/site-packages/cffi-1.15.1.dist-info/LICENSE backend/audio_env/lib/python3.7/site-packages/cffi-1.15.1.dist-info/METADATA backend/audio_env/lib/python3.7/site-packages/cffi-1.15.1.dist-info/RECORD backend/audio_env/lib/python3.7/site-packages/cffi-1.15.1.dist-info/top_level.txt backend/audio_env/lib/python3.7/site-packages/cffi-1.15.1.dist-info/WHEEL backend/audio_env/lib/python3.7/site-packages/cffi/__init__.py backend/audio_env/lib/python3.7/site-packages/cffi/_cffi_errors.h backend/audio_env/lib/python3.7/site-packages/cffi/_cffi_include.h backend/audio_env/lib/python3.7/site-packages/cffi/_embedding.h backend/audio_env/lib/python3.7/site-packages/cffi/api.py backend/audio_env/lib/python3.7/site-packages/cffi/backend_ctypes.py backend/audio_env/lib/python3.7/site-packages/cffi/cffi_opcode.py backend/audio_env/lib/python3.7/site-packages/cffi/commontypes.py backend/audio_env/lib/python3.7/site-packages/cffi/cparser.py backend/audio_env/lib/python3.7/site-packages/cffi/error.py backend/audio_env/lib/python3.7/site-packages/cffi/ffiplatform.py backend/audio_env/lib/python3.7/site-packages/cffi/lock.py backend/audio_env/lib/python3.7/site-packages/cffi/model.py backend/audio_env/lib/python3.7/site-packages/cffi/parse_c_type.h backend/audio_env/lib/python3.7/site-packages/cffi/pkgconfig.py backend/audio_env/lib/python3.7/site-packages/cffi/recompiler.py backend/audio_env/lib/python3.7/site-packages/cffi/setuptools_ext.py backend/audio_env/lib/python3.7/site-packages/cffi/vengine_cpy.py backend/audio_env/lib/python3.7/site-packages/cffi/vengine_gen.py backend/audio_env/lib/python3.7/site-packages/cffi/verifier.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer-3.4.2.dist-info/entry_points.txt backend/audio_env/lib/python3.7/site-packages/charset_normalizer-3.4.2.dist-info/INSTALLER backend/audio_env/lib/python3.7/site-packages/charset_normalizer-3.4.2.dist-info/licenses/LICENSE backend/audio_env/lib/python3.7/site-packages/charset_normalizer-3.4.2.dist-info/METADATA backend/audio_env/lib/python3.7/site-packages/charset_normalizer-3.4.2.dist-info/RECORD backend/audio_env/lib/python3.7/site-packages/charset_normalizer-3.4.2.dist-info/top_level.txt backend/audio_env/lib/python3.7/site-packages/charset_normalizer-3.4.2.dist-info/WHEEL backend/audio_env/lib/python3.7/site-packages/charset_normalizer/__init__.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/__main__.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/api.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/cd.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/cli/__init__.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/cli/__main__.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/constant.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/legacy.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/md.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/models.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/py.typed backend/audio_env/lib/python3.7/site-packages/charset_normalizer/utils.py backend/audio_env/lib/python3.7/site-packages/charset_normalizer/version.py backend/audio_env/lib/python3.7/site-packages/click-8.1.8.dist-info/INSTALLER backend/audio_env/lib/python3.7/site-packages/click-8.1.8.dist-info/LICENSE.txt backend/audio_env/lib/python3.7/site-packages/click-8.1.8.dist-info/METADATA backend/audio_env/lib/python3.7/site-packages/click-8.1.8.dist-info/RECORD backend/audio_env/lib/python3.7/site-packages/click-8.1.8.dist-info/WHEEL backend/audio_env/lib/python3.7/site-packages/click/__init__.py backend/audio_env/lib/python3.7/site-packages/click/_compat.py backend/audio_env/lib/python3.7/site-packages/click/_termui_impl.py backend/audio_env/lib/python3.7/site-packages/click/_textwrap.py backend/audio_env/lib/python3.7/site-packages/click/_winconsole.py backend/audio_env/lib/python3.7/site-packages/click/core.py [5440 more files omitted for size] ``` ### Dependencies - backend/audio_env/lib/python3.7/site-packages/matplotlib/backends/web_backend/package.json: @jsxtools/resize-observer@^1.0.4, eslint@^6.8.0, eslint-config-prettier@^6.10.1, prettier@^2.0.2 - backend/requirements.txt: accelerate@==0.15.0, diffusers@==0.10.2, ffmpeg-python@==0.2.0, filelock@==3.9.0, flask@==2.0.3, flask-cors@==3.0.10, huggingface-hub@==0.11.1, importlib_metadata@==6.0.0, librosa@==0.9.2, matplotlib@==3.5.3, numba@==0.56.4, numpy@==1.21.6, packaging@==21.3, pillow@==9.5.0, psutil@==5.9.4, pydub@==0.25.1, python-dotenv@==1.0.0, pyyaml@==6.0, regex@==2022.10.31, requests@==2.28.2, resampy@==0.3.1, safetensors@==0.2.8, scikit-learn@==1.0.2, scipy@==1.7.3, soundfile@==0.10.3.post1, torch@==1.13.1, torchvision@==0.14.1, tqdm@==4.64.1, transformers@==4.25.1 - package.json: @google/generative-ai@^0.15.0, @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, react@^18.2.0, react-dom@^18.2.0, react-scripts@5.0.1, web-vitals@^2.1.4 ### Recent commits (newest first) - Fixed animations - Fixed errors - Finished Project - Commit 1 ## Key source files (fetched from GitHub, selected and truncated for size) ### ENV_SETUP.md ```markdown # 🔐 Environment Variables Setup Guide This guide will help you securely configure your API keys and environment variables for SceneCreator. ## 🚀 Quick Setup Run the automated setup script: ```bash ./setup_env.sh ``` This will create the necessary `.env` files from templates. You'll still need to add your actual API keys. ## 📝 Manual Setup ### 1. Frontend Environment Variables Create a `.env` file in the root directory: ```bash cp env.example .env ``` Edit `.env` and configure: ```env # Google Gemini API Key (Required) REACT_APP_GEMINI_API_KEY=your_gemini_api_key_here # Backend API URL (Optional - defaults to localhost:5001) REACT_APP_BACKEND_URL=http://localhost:5001 # Debug mode (Optional) REACT_APP_DEBUG=false ``` ### 2. Backend Environment Variables Create a `.env` file in the backend directory: ```bash cp backend/env.example backend/.env ``` Edit `backend/.env` and configure: ```env # Google API Key (Required for Veo 2 and Gemini) GOOGLE_API_KEY=your_google_api_key_here # Google Cloud Project ID (Optional - for Lyria) GOOGLE_CLOUD_PROJECT=your_project_id # Flask Configuration FLASK_DEBUG=true FLASK_ENV=development FLASK_PORT=5001 # Audio Settings MAX_FILE_SIZE_MB=50 SUPPORTED_AUDIO_FORMATS=mp3,wav,m4a,flac,ogg,aac # Video Generation Settings VIDEO_TIMEOUT_SECONDS=1800 VIDEO_ASPECT_RATIO=16:9 VIDEO_DURATION_SECONDS=8 ``` ## 🔑 Getting API Keys ### Google Gemini API Key 1. Go to [Google AI Studio](https://aistudio.google.com/app/apikey) 2. Sign in with your Google account 3. Click "Create API Key" 4. Copy the generated key 5. Add it to both `.env` files: - Frontend: `REACT_APP_GEMINI_API_KEY=your_key_here` - Backend: `GOOGLE_API_KEY=your_key_here` ### Google Cloud Project (Optional) For Lyria music generation: 1. Go to [Google Cloud Console](https://console.cloud.google.com/) 2. Create a new project or select existing one 3. Enable the Vertex AI API 4. Note your project ID 5. Add to `backend/.env`: `GOOGLE_CLOUD_PROJECT=your_project_id` ## 🔒 Security Best Practices ### ✅ Do's - Keep your `.env` files private (they're in `.gitignore`) - Use different API keys for development and production - Regularly rotate your API keys - Set API key usage limits in Google Cloud Console ### ❌ Don'ts - Never commit `.env` files to version control - Don't share API keys in screenshots or logs - Don't use production keys in development - Don't store keys in plain text outside of `.env` files ## 🧪 Testing Your Setup ### 1. Check Environment Variables Frontend (React): ```bash # In your browser console after starting the app console.log(process.env.REACT_APP_GEMINI_API_KEY ? '✅ API key loaded' : '❌ API key missing'); ``` Backend (Python): ```bash cd backend python -c "import os; from dotenv import load_dotenv; load_dotenv(); print('✅ API key loaded' if os.getenv('GOOGLE_API_KEY') else '❌ API key missing')" ``` ### 2. Test API Connection Start the application and try generating a video. Check the browser console and ba [truncated — 1199 more characters] ``` ### AUDIO_ANALYSIS_README.md ```markdown # 🎵 Audio Analysis with Librosa Integration Your Gemini AI chatbot now includes **advanced audio analysis** using librosa to extract musical features and map them to visual descriptions! ## 🚀 **New Features** ### 🎼 **Librosa Audio Analysis** - **Tempo Detection** - BPM analysis for pacing - **Spectral Features** - Brightness, contrast, and frequency analysis - **Rhythm Tracking** - Beat detection and onset analysis - **Harmonic Analysis** - Chroma features and key detection - **Energy Profiling** - RMS and dynamic range analysis - **Timbral Analysis** - MFCCs for texture classification - **Major/Minor Detection** - Mood classification ### 🎨 **Audio-to-Visual Mapping** - **Dynamic Visuals** - Fast tempo → sharp angles, slow tempo → soft transitions - **Lighting Control** - Spectral centroid → brightness levels - **Color Palettes** - Chroma features → vibrant vs muted colors - **Visual Intensity** - RMS energy → saturation and boldness - **Pattern Generation** - Spectral contrast → smooth gradients vs harsh textures - **Animation Timing** - Onset detection → visual bursts and transitions ## 🛠 **Setup Instructions** ### 1. **Backend Setup (Required)** ```bash # Navigate to backend directory cd backend # Run setup script ./setup_backend.sh # Start the audio analysis server source audio_env/bin/activate python app.py ``` ### 2. **Frontend Setup** ```bash # Start the React app (in main directory) npm start ``` The backend runs on `http://localhost:5001` and the frontend on `http://localhost:3000`. ## 🎯 **How It Works** ### **Audio Upload Process:** 1. **Upload Audio** - Select MP3, WAV, M4A, or other supported formats 2. **Librosa Analysis** - Python backend extracts 10+ musical features 3. **Visual Mapping** - Features are mapped to visual characteristics 4. **AI Integration** - Gemini AI receives detailed audio analysis 5. **Scene Generation** - AI creates visual descriptions based on audio features ### **Extracted Features:** #### 🎵 **Tempo & Rhythm** - **BPM Detection** → Visual pacing (fast/slow motion) - **Beat Tracking** → Animation timing alignment - **Onset Detection** → Camera flashes and transitions #### 🌈 **Spectral Analysis** - **Spectral Centroid** → Brightness (bright/dark scenes) - **Spectral Contrast** → Pattern style (bold/smooth) - **Chroma Features** → Color palette selection #### 🔊 **Energy & Dynamics** - **RMS Energy** → Visual intensity and saturation - **Zero Crossing Rate** → Texture quality (harsh/smooth) #### 🧠 **Advanced Features** - **MFCCs** → Style classification (acoustic/electronic/natural) - **Key Detection** → Mood mapping (major/minor tonality) ## 📊 **Example Mapping** ``` Input Audio: Classical piano piece in C minor, 72 BPM ↓ Librosa Analysis: - Tempo: 72 BPM (slow) - Spectral Centroid: 1200 Hz (moderate brightness) - RMS: 0.03 (low energy) - Chroma: C minor detected - MFCCs: Acoustic texture ↓ Visual Output: "Create a foggy, watercolor scene with slow pan movements, soft lighting i [truncated — 3154 more characters] ``` ### package.json ``` { "name": "gemini-chatbot-app", "version": "0.1.0", "private": true, "dependencies": { "@google/generative-ai": "^0.15.0", "@testing-library/jest-dom": "^5.17.0", "@testing-library/react": "^13.4.0", "@testing-library/user-event": "^13.5.0", "react": "^18.2.0", "react-dom": "^18.2.0", "react-scripts": "5.0.1", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### backend/requirements.txt ``` flask==2.0.3 flask-cors==3.0.10 python-dotenv==1.0.0 librosa==0.9.2 numpy==1.21.6 scipy==1.7.3 soundfile==0.10.3.post1 resampy==0.3.1 numba==0.56.4 scikit-learn==1.0.2 matplotlib==3.5.3 pydub==0.25.1 ffmpeg-python==0.2.0 torch==1.13.1 torchvision==0.14.1 diffusers==0.10.2 transformers==4.25.1 accelerate==0.15.0 pillow==9.5.0 tqdm==4.64.1 safetensors==0.2.8 huggingface-hub==0.11.1 regex==2022.10.31 packaging==21.3 pyyaml==6.0 filelock==3.9.0 psutil==5.9.4 requests==2.28.2 importlib_metadata==6.0.0 ``` ### backend/audio_env/lib/python3.7/site-packages/matplotlib/backends/web_backend/package.json ``` { "devDependencies": { "eslint": "^6.8.0", "eslint-config-prettier": "^6.10.1", "prettier": "^2.0.2" }, "scripts": { "eslint": "eslint . --fix", "eslint:check": "eslint .", "lint": "npm run prettier && npm run eslint", "lint:check": "npm run prettier:check && npm run eslint:check", "prettier": "prettier --write \"**/*{.ts,.tsx,.js,.jsx,.css,.json}\"", "prettier:check": "prettier --check \"**/*{.ts,.tsx,.js,.jsx,.css,.json}\"" }, "dependencies": { "@jsxtools/resize-observer": "^1.0.4" } } ``` ### src/index.js ```javascript import React from 'react'; import ReactDOM from 'react-dom/client'; import './index.css'; import App from './App'; const root = ReactDOM.createRoot(document.getElementById('root')); root.render( ); ``` ### backend/app.py ```python from flask import Flask, request, jsonify, send_from_directory from flask_cors import CORS import os import tempfile from datetime import datetime from audio_analyzer import AudioAnalyzer import traceback import logging from dotenv import load_dotenv # Load environment variables load_dotenv() # Set up logging logging.basicConfig( level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s' ) logger = logging.getLogger(__name__) app = Flask(__name__) CORS(app) # Enable CORS for React frontend # Initialize audio analyzer analyzer = AudioAnalyzer() # Configure upload settings from environment variables MAX_FILE_SIZE_MB = int(os.getenv('MAX_FILE_SIZE_MB', '50')) app.config['MAX_CONTENT_LENGTH'] = MAX_FILE_SIZE_MB * 1024 * 1024 # Audio formats from environment variable audio_formats_str = os.getenv('SUPPORTED_AUDIO_FORMATS', 'mp3,wav,m4a,flac,ogg,aac') ALLOWED_EXTENSIONS = set(format.strip() for format in audio_formats_str.split(',')) def allowed_file(filename): return '.' in filename and filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS @app.route('/') def home(): logger.info("🏠 Home endpoint accessed") return jsonify({ "status": "success", "message": "Audio Analysis Server is running!", "version": "2.0.0", "supported_formats": list(ALLOWED_EXTENSIONS), "endpoints": { "/": "Server status", "/analyze-audio": "POST - Upload and analyze audio file" } }) @app.route('/analyze-audio', methods=['POST']) def analyze_audio(): logger.info("🎵 Audio analysis request received") try: # Check if file was uploaded if 'audio' not in request.files: logger.warning("❌ No audio file in request") return jsonify({ "status": "error", "message": "No audio file provided" }), 400 file = request.files['audio'] if file.filename == '': logger.warning("❌ Empty filename") return jsonify({ "status": "error", "message": "No file selected" }), 400 # Get file info filename = file.filename file_size = len(file.read()) file.seek(0) # Reset file pointer logger.info(f"📁 File received: {filename} ({file_size} bytes)") # Check file extension if filename: file_ext = filename.rsplit('.', 1)[1].lower() if '.' in filename else '' if file_ext not in ALLOWED_EXTENSIONS: logger.warning(f"❌ Unsupported format: {file_ext}") return jsonify({ "status": "error", "message": f"Unsupported file format: {file_ext}. Supported formats: {', '.join(ALLOWED_EXTENSIONS)}" }), 400 logger.info(f"✅ File format supported: {file_ext}") # Save uploaded file to temporary location temp_file = tempfile.NamedTemporaryFile(delete=False, suffix=f'.{file_ext}') temp_path = temp_file.name logger.debug(f"💾 Saving file to: {temp_path}") file.save(temp_path) temp_file.close() logger.info(f"✅ File saved successfully") logger.info(f"🔍 Analyzing audio file: {filename}") # Analyze the audio try: features = analyzer.extract_features(temp_path) logger.info("✅ Audio analysis completed successfully") # Create response response_data = { "status": "success", "message": "Audio analysis completed", "filename": filename, "file_size": file_size, "features": features } # Log summary logger.info(f"📊 Analysis Summary:") logger.info(f" Duration: {features.get('duration', 0):.2f}s") logger.info(f" Tempo: {features.get('tempo', 0):.1f} BPM") logger.info(f" Key: {features.get('estimated_key', 'Unknown')}") logger.info(f" Energy: {features.get('rms_energy_mean', 0):.6f}") return jsonify(response_data) except Exception as analysis_error: logger.error(f"❌ Audio analysis failed: {str(analysis_error)}") logger.error(f" File: {filename}") logger.error(f" Path: {temp_path}") # Clean up temp file if os.path.exists(temp_path): try: os.unlink(temp_path) logger.debug(f"🗑️ Cleaned up temp file after error") except: pass return jsonify({ "status": "error", "message": f"Audio analysis failed: {str(analysis_error)}", "details": { "filename": filename, "error_type": type(analysis_error).__name__ } }), 500 # Re-raise for outer try-catch raise analysis_error finally: # Clean up temporary file if os.path.exists(temp_path): try: os.unlink(temp_path) logger.debug(f"🗑️ Cleaned up temporary file: {temp_path}") except Exception as cleanup_error: logger.warning(f"⚠️ Could not delete temp file: {cleanup_error}") except Exception as e: logger.error(f"❌ Unexpected error in analyze_audio: {str(e)}") logger.error(f" Error type: {type(e).__name__}") return jsonify({ "status": "error", "message": "Internal server error", "details": str(e) }), 500 @app.route('/health', methods=['GET']) def health_check(): """Health c [truncated — 12439 more characters] ``` ### src/App.js ```javascript import React, { useState, useRef, useEffect } from 'react'; import './App.css'; // Initialize Gemini AI const GEMINI_API_KEY = process.env.REACT_APP_GEMINI_API_KEY; // Backend API configuration const BACKEND_URL = process.env.REACT_APP_BACKEND_URL || 'http://localhost:5001'; // Check if API key is configured if (!GEMINI_API_KEY) { console.error('⚠️ REACT_APP_GEMINI_API_KEY is not configured. Please check your .env file.'); } const App = () => { const [activeTab, setActiveTab] = useState('chat'); const [messages, setMessages] = useState([ { id: 1, text: "Hello! I'm Sonetica, and I help bring your poems and stories to life through custom videos. Share your poem or story with me to get started!", sender: 'ai', timestamp: new Date() } ]); const [input, setInput] = useState(''); const [isLoading, setIsLoading] = useState(false); const [connectionStatus, setConnectionStatus] = useState('connected'); const [audioFile, setAudioFile] = useState(null); const [waitingForAudio, setWaitingForAudio] = useState(false); const [storyText, setStoryText] = useState(''); const [audioAnalysisData, setAudioAnalysisData] = useState(null); const [videos, setVideos] = useState([]); const [loadingVideos, setLoadingVideos] = useState(false); const messagesEndRef = useRef(null); const fileInputRef = useRef(null); const textareaRef = useRef(null); // Auto-scroll to bottom const scrollToBottom = () => { messagesEndRef.current?.scrollIntoView({ behavior: "smooth" }); }; useEffect(() => { scrollToBottom(); }, [messages]); // Fetch videos for EchoVerse const fetchVideos = async () => { try { setLoadingVideos(true); const response = await fetch(`${BACKEND_URL}/api/videos`); if (response.ok) { const videoData = await response.json(); setVideos(videoData.videos || []); } else { console.error('Failed to fetch videos'); } } catch (error) { console.error('Error fetching videos:', error); } finally { setLoadingVideos(false); } }; // Fetch videos when EchoVerse tab is opened useEffect(() => { if (activeTab === 'echoverse') { fetchVideos(); // Set up polling to check for new videos every 10 seconds const interval = setInterval(fetchVideos, 10000); return () => clearInterval(interval); } }, [activeTab]); // Also refresh videos when a new video is successfully generated useEffect(() => { if (activeTab === 'echoverse') { fetchVideos(); // Restart auto-playing videos when switching to EchoVerse setTimeout(() => { const backgroundVideos = document.querySelectorAll('.background-video'); backgroundVideos.forEach(video => { video.play().catch(err => { console.log('Auto-play prevented:', err); }); }); }, 100); } }, [messages, activeTab]); // Refresh when messages change (new video generated) or tab changes // Connection status check useEffect(() => { const checkConnection = async () => { try { setConnectionStatus('connecting'); // Test Gemini API with a simple request const response = await fetch(`https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-latest:generateContent?key=${GEMINI_API_KEY}`, { method: 'POST', headers: { 'Content-Type': 'application/json', }, body: JSON.stringify({ contents: [{ parts: [{ text: 'Hello' }] }] }), signal: AbortSignal.timeout(10000) }); if (response.ok) { setConnectionStatus('connected'); } else { throw new Error(`API responded with status: ${response.status}`); } } catch (error) { console.error('Connection test failed:', error); setConnectionStatus('connected'); // Set as connected anyway to avoid blocking the UI } }; checkConnection(); }, []); const sendMessage = async () => { if (!input.trim() && !audioFile) return; const userMessage = { id: Date.now(), text: input || (audioFile ? `[Audio file: ${audioFile.name}]` : ''), sender: 'user', timestamp: new Date() }; setMessages(prev => [...prev, userMessage]); setIsLoading(true); try { if (!waitingForAudio && !audioFile && input.trim()) { // First interaction - collect the story/poem console.log('Storing story text:', input); setStoryText(input); setWaitingForAudio(true); const aiResponse = { id: Date.now() + 1, text: "Beautiful! Now you can optionally upload some music to enhance your story, or just type 'create video' to generate your custom video.", sender: 'ai', timestamp: new Date() }; setMessages(prev => [...prev, aiResponse]); setInput(''); setIsLoading(false); return; } if (input.toLowerCase().includes('create video') && storyText) { // Generate video with story text (no audio) console.log('Creating video with story only:', storyText); await generateVideo(); return; } // Handle other responses during audio waiting phase const aiResponse = { id: Date.now() + 1, text: "You can upload music to enhance your story, or type 'create video' to generate your video now.", sender: 'ai', timestamp: new Date() }; setMessages(prev => [...prev, aiResponse]); } catch (error) { console.error('Error:', error); const errorResponse = { id: Date.now() + 1, text: "I'm having trouble right now. Please try again in a moment.", sender: 'ai', timestamp: new Date() }; [truncated — 20623 more characters] ``` ### backend/audio_env/lib/python3.7/site-packages/libpasteurize/main.py ```python """ pasteurize: automatic conversion of Python 3 code to clean 2/3 code =================================================================== ``pasteurize`` attempts to convert existing Python 3 code into source-compatible Python 2 and 3 code. Use it like this on Python 3 code: $ pasteurize --verbose mypython3script.py This removes any Py3-only syntax (e.g. new metaclasses) and adds these import lines: from __future__ import absolute_import from __future__ import division from __future__ import print_function from __future__ import unicode_literals from future import standard_library standard_library.install_hooks() from builtins import * To write changes to the files, use the -w flag. It also adds any other wrappers needed for Py2/3 compatibility. Note that separate stages are not available (or needed) when converting from Python 3 with ``pasteurize`` as they are when converting from Python 2 with ``futurize``. The --all-imports option forces adding all ``__future__`` imports, ``builtins`` imports, and standard library aliases, even if they don't seem necessary for the current state of each module. (This can simplify testing, and can reduce the need to think about Py2 compatibility when editing the code further.) """ from __future__ import (absolute_import, print_function, unicode_literals) import sys import logging import optparse from lib2to3.main import main, warn, StdoutRefactoringTool from lib2to3 import refactor from future import __version__ from libpasteurize.fixes import fix_names def main(args=None): """Main program. Returns a suggested exit status (0, 1, 2). """ # Set up option parser parser = optparse.OptionParser(usage="pasteurize [options] file|dir ...") parser.add_option("-V", "--version", action="store_true", help="Report the version number of pasteurize") parser.add_option("-a", "--all-imports", action="store_true", help="Adds all __future__ and future imports to each module") parser.add_option("-f", "--fix", action="append", default=[], help="Each FIX specifies a transformation; default: all") parser.add_option("-j", "--processes", action="store", default=1, type="int", help="Run 2to3 concurrently") parser.add_option("-x", "--nofix", action="append", default=[], help="Prevent a fixer from being run.") parser.add_option("-l", "--list-fixes", action="store_true", help="List available transformations") # parser.add_option("-p", "--print-function", action="store_true", # help="Modify the grammar so that print() is a function") parser.add_option("-v", "--verbose", action="store_true", help="More verbose logging") parser.add_option("--no-diffs", action="store_true", help="Don't show diffs of the refactoring") parser.add_option("-w", "--write", action="store_true", help="Write back modified files") parser.add_option("-n", "--nobackups", action="store_true", default=False, help="Don't write backups for modified files.") # Parse command line arguments refactor_stdin = False flags = {} options, args = parser.parse_args(args) fixer_pkg = 'libpasteurize.fixes' avail_fixes = fix_names flags["print_function"] = True if not options.write and options.no_diffs: warn("not writing files and not printing diffs; that's not very useful") if not options.write and options.nobackups: parser.error("Can't use -n without -w") if options.version: print(__version__) return 0 if options.list_fixes: print("Available transformations for the -f/--fix option:") for fixname in sorted(avail_fixes): print(fixname) if not args: return 0 if not args: print("At least one file or directory argument required.", file=sys.stderr) print("Use --help to show usage.", file=sys.stderr) return 2 if "-" in args: refactor_stdin = True if options.write: print("Can't write to stdin.", file=sys.stderr) return 2 # Set up logging handler level = logging.DEBUG if options.verbose else logging.INFO logging.basicConfig(format='%(name)s: %(message)s', level=level) unwanted_fixes = set() for fix in options.nofix: if ".fix_" in fix: unwanted_fixes.add(fix) else: # Infer the full module name for the fixer. # First ensure that no names clash (e.g. # lib2to3.fixes.fix_blah and libfuturize.fixes.fix_blah): found = [f for f in avail_fixes if f.endswith('fix_{0}'.format(fix))] if len(found) > 1: print("Ambiguous fixer name. Choose a fully qualified " "module name instead from these:\n" + "\n".join(" " + myf for myf in found), file=sys.stderr) return 2 elif len(found) == 0: print("Unknown fixer. Use --list-fixes or -l for a list.", file=sys.stderr) return 2 unwanted_fixes.add(found[0]) extra_fixes = set() if options.all_imports: prefix = 'libpasteurize.fixes.' extra_fixes.add(prefix + 'fix_add_all__future__imports') extra_fixes.add(prefix + 'fix_add_future_standard_library_import') extra_fixes.add(prefix + 'fix_add_all_future_builtins') explicit = set() if options.fix: all_present = False for fix in options.fix: if fix == 'all': all_present = True else: if ".fix_" in fix: explicit.add(fix) else: # Infer the full [truncated — 2186 more characters] ``` ### backend/audio_env/lib/python3.7/site-packages/libfuturize/main.py ```python """ futurize: automatic conversion to clean 2/3 code using ``python-future`` ====================================================================== Like Armin Ronacher's modernize.py, ``futurize`` attempts to produce clean standard Python 3 code that runs on both Py2 and Py3. One pass -------- Use it like this on Python 2 code: $ futurize --verbose mypython2script.py This will attempt to port the code to standard Py3 code that also provides Py2 compatibility with the help of the right imports from ``future``. To write changes to the files, use the -w flag. Two stages ---------- The ``futurize`` script can also be called in two separate stages. First: $ futurize --stage1 mypython2script.py This produces more modern Python 2 code that is not yet compatible with Python 3. The tests should still run and the diff should be uncontroversial to apply to most Python projects that are willing to drop support for Python 2.5 and lower. After this, the recommended approach is to explicitly mark all strings that must be byte-strings with a b'' prefix and all text (unicode) strings with a u'' prefix, and then invoke the second stage of Python 2 to 2/3 conversion with:: $ futurize --stage2 mypython2script.py Stage 2 adds a dependency on ``future``. It converts most remaining Python 2-specific code to Python 3 code and adds appropriate imports from ``future`` to restore Py2 support. The command above leaves all unadorned string literals as native strings (byte-strings on Py2, unicode strings on Py3). If instead you would like all unadorned string literals to be promoted to unicode, you can also pass this flag: $ futurize --stage2 --unicode-literals mypython2script.py This adds the declaration ``from __future__ import unicode_literals`` to the top of each file, which implicitly declares all unadorned string literals to be unicode strings (``unicode`` on Py2). All imports ----------- The --all-imports option forces adding all ``__future__`` imports, ``builtins`` imports, and standard library aliases, even if they don't seem necessary for the current state of each module. (This can simplify testing, and can reduce the need to think about Py2 compatibility when editing the code further.) """ from __future__ import (absolute_import, print_function, unicode_literals) import future.utils from future import __version__ import sys import logging import optparse import os from lib2to3.main import warn, StdoutRefactoringTool from lib2to3 import refactor from libfuturize.fixes import (lib2to3_fix_names_stage1, lib2to3_fix_names_stage2, libfuturize_fix_names_stage1, libfuturize_fix_names_stage2) fixer_pkg = 'libfuturize.fixes' def main(args=None): """Main program. Args: fixer_pkg: the name of a package where the fixers are located. args: optional; a list of command line arguments. If omitted, sys.argv[1:] is used. Returns a suggested exit status (0, 1, 2). """ # Set up option parser parser = optparse.OptionParser(usage="futurize [options] file|dir ...") parser.add_option("-V", "--version", action="store_true", help="Report the version number of futurize") parser.add_option("-a", "--all-imports", action="store_true", help="Add all __future__ and future imports to each module") parser.add_option("-1", "--stage1", action="store_true", help="Modernize Python 2 code only; no compatibility with Python 3 (or dependency on ``future``)") parser.add_option("-2", "--stage2", action="store_true", help="Take modernized (stage1) code and add a dependency on ``future`` to provide Py3 compatibility.") parser.add_option("-0", "--both-stages", action="store_true", help="Apply both stages 1 and 2") parser.add_option("-u", "--unicode-literals", action="store_true", help="Add ``from __future__ import unicode_literals`` to implicitly convert all unadorned string literals '' into unicode strings") parser.add_option("-f", "--fix", action="append", default=[], help="Each FIX specifies a transformation; default: all.\nEither use '-f division -f metaclass' etc. or use the fully-qualified module name: '-f lib2to3.fixes.fix_types -f libfuturize.fixes.fix_unicode_keep_u'") parser.add_option("-j", "--processes", action="store", default=1, type="int", help="Run 2to3 concurrently") parser.add_option("-x", "--nofix", action="append", default=[], help="Prevent a fixer from being run.") parser.add_option("-l", "--list-fixes", action="store_true", help="List available transformations") parser.add_option("-p", "--print-function", action="store_true", help="Modify the grammar so that print() is a function") parser.add_option("-v", "--verbose", action="store_true", help="More verbose logging") parser.add_option("--no-diffs", action="store_true", help="Don't show diffs of the refactoring") parser.add_option("-w", "--write", action="store_true", help="Write back modified files") parser.add_option("-n", "--nobackups", action="store_true", default=False, help="Don't write backups for modified files.") parser.add_option("-o", "--output-dir", action="store", type="str", default="", help="Put output files in this directory " "instead of overwriting the input files. Requires -n. " "For Python >= 2.7 only.") parser.add_option("-W", "--write-unchanged-files", action="store_true", help="Also write files even if no changes were required" " (useful with --output-dir); implies -w.") par [truncated — 7733 more characters] ``` [3969 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]