# Project export: TuneAI
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: Cal Hacks 10.0
- Tagline: Elevate Your Visuals with TuneAI: Seamless Soundscapes for Content Creators, Filmmakers, and Videographers. Simply upload a video, and TuneAI will give you the perfect musical accompaniment.
- Devpost: https://devpost.com/software/tuneai
- GitHub: https://github.com/ryanmckim/CalHacks2023/
- Demo: https://tune-ai.vercel.app/
- Video: https://www.youtube.com/embed/cyrZp7SlkEM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Best Use of Intel Developer Cloud)
- Team: 4 GitHub contributor(s) — Iris Xu (19 commits), Ryan Kim (14 commits), LenaWang03 (11 commits), Owen Yap (3 commits)
## Devpost submission (written by the team)
### Inspiration
TuneAI was born out of a deep appreciation for the art of visual storytelling and the recognition that audio plays in enhancing the impact of videos. Our motivation to create TuneAI lies in the desire to provide content creators, filmmakers, and videographers with a powerful tool that simplifies the process of generating background audio and sound effects, making it more accessible and efficient than ever before.
### What it does
TuneAI is an intelligent visual content to audio generator tool. We provide users with access to generative AI technology to create unique musical samples based on the context of their images or videos. Our model analyzes the video for its mood, tone, objects, scenery, and a variety of other contextual elements, and generates music based on a suitable genre, instrumentation, style, etc. Our music is sure to inspire you!
### How we built it
After the video file is uploaded, we splice the video into multiple frames which are then analyzed by Salesforce BLIP image-to-text model, which gives us a list of image descriptors. These are then passed on to OpenAI GPT, which takes the image descriptors and comes up with the musical prompt, which is fed into Meta's MusicGen. The audio generated by MusicGen can be edited to fit the video. This whole pipeline is squashed into a function which is served on the backend (Flask). We used React to build our frontend and deployed on Vercel. Intel Cloud Max Series GPU allows us to perform inferencing at speed, allowing us to generate the audio in reasonable time.
### Challenges we ran into
Complex tasks: how do we ... do video understanding? ... translate that understanding into a type of music? ... how do we define or generate music? We answered all of these questions after a thorough investigation of current methods, models, and applications. Model selection: Making decisions to select the best models for our choice of architecture and design constraints (compute speed and memory and accuracy tradeoffs). Attempts to fine tune our models: We are venturing into rather new territory in the generative AI space for videos and audio. There are limited datasets that would allow us to tune (Example: In targeting our model for short form content, we wanted to use Tiktok videos to fine-tune the BLIP model. However, we cannot easily analyze sentiment or understand moods with a "describe what you see" type of model, especially since Tiktok present a lot of social trends. Video descriptors and metadata also do not provide much context. This proved to be unreliable upon testing and set us back initially.) A second challenge was to figure out a way to increase the speed of inference, by setting up our environment in Intel Cloud as none of us had deep infrastructure experience . A huge shout-out to Rahul from Intel who helped us through some parts of the cloud setup, allowing us to use Max Series GPUs and 4th gen Xeon Processors for model training and inference.
### Accomplishments we're proud of
Producing the first video-to-audio GenAI model pipeline in under 3 days. Overcoming various technical and non-technical obstacles along the way (overloaded network, laptop memory limit, little sleep) to produce a fully functioning web application which content creators, videographers and filmmakers can now use for audio inspiration.
### What we learned
State of the art generative AI Picked up new tech stack and skills: first time using Flask and websockets, experimented OpenAI prompt generation, learned about Transformer models. Quick problem solving, iterating, and pivoting if blocked.
### What's next
Accept longer form content, which would require access to more powerful compute (i.e. Intel Developer Cloud). Allow users to control the music generation process, providing additional context or specifications to their use cases. Fine tune the captioning model to understand context and moods for more agreeable generations. Speed up inference process by optimizing prompt generation.
## README (from the GitHub repository)
# TuneAI: Video/Image to Audio
Generative AI technology for audio content creation.
## Going from Video to Audio, an example process:
**Visual Input**: [Video background party](https://www.youtube.com/watch?v=4qz6x8y3tNw)
**One Snapshot**: 
**Captions with Blip**: ['purple light shining on a crowd of people at a concert', 'purple light shining on a crowd of people at a concert', 'crowd of people at a concert with their hands in the air']
**Prompt generation with OpenAI**: EDM, Energetic, Pop with pulsating beats, synths, and euphoric crowd samples.
**Audio generation with MusicGen**: [10s music](backend/audio/party.wav)
### Other Video Sources
[Australia vs USA | Women's Beach Volleyball Gold Medal Match | Tokyo Replays](https://www.youtube.com/watch?v=459Oda8XPy0)
[Video background party](https://www.youtube.com/watch?v=4qz6x8y3tNw)
[Fly Me To The Moon - Stringspace Jazz Band](https://www.youtube.com/watch?v=EpjcSfnWqAk)
[Eating at Grand Central Oyster Bar NYC. Tourist Trap? or Classic Restaurant?](https://www.youtube.com/watch?v=BbrHLzx_D_8)
## Detected evidence (automated analysis)
Indexed codebase: 21 recognized source files, 34 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- Hugging Face (technology) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- PyTorch (technology) — detected in the code
- React (technology) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
- Vercel (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (30 of 30)
```
.gitignore
backend/README.md
backend/requirements.txt
backend/server.py
docker-compose.yml
Dockerfile.api
Dockerfile.client
environment.yml
frontend/package.json
frontend/public/index.html
frontend/public/manifest.json
frontend/public/robots.txt
frontend/README.md
frontend/src/App.css
frontend/src/App.js
frontend/src/components/Button.js
frontend/src/components/NavBar.js
frontend/src/index.js
frontend/src/pages/Demo.js
frontend/src/pages/Home.js
frontend/src/reportWebVitals.js
frontend/src/setupTests.js
frontend/src/VideoInput.js
gen_model.py
humeai.py
Makefile
README.md
requirements/clean.py
requirements/requirements.txt
utilities.py
```
### Dependencies
- frontend/package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, axios@^1.6.0, http-proxy-middleware@^2.0.6, react@^18.2.0, react-dom@^18.2.0, react-router-dom@^6.17.0, react-scripts@5.0.1, web-vitals@^2.1.4
- requirements/requirements.txt: accelerate@==0.24.0, anyio@==4.0.0, argon2-cffi@==23.1.0, argon2-cffi-bindings@==21.2.0, arrow@==1.2.3, asttokens, async-lru@==2.0.4, attrs@==23.1.0, av@==10.0.0, Babel@==2.12.1, backcall, backports.functools-lru-cache, beautifulsoup4@==4.12.2, bleach@==6.0.0, Bottleneck, Brotli, certifi@==2023.7.22, cffi, cfgv, charset-normalizer, colorama, comm, contourpy, cycler, debugpy, decorator, defusedxml@==0.7.1, diffusers@==0.21.4, distlib, exceptiongroup, executing, fastjsonschema@==2.18.0, filelock, fonttools@==4.25.0, fqdn@==1.5.1, fsspec@==2023.10.0, huggingface-hub@==0.17.3, identify, idna, importlib-metadata, iniconfig, intel-extension-for-pytorch@==2.1.0, ipykernel, ipython, ipywidgets, isoduration@==20.11.0, jedi, Jinja2, json5@==0.9.14, jsonpointer@==2.4, jsonschema@==4.19.1, jsonschema-specifications@==2023.7.1, jupyter_client, jupyter_core, jupyter_server@==2.7.3, jupyter_server_terminals@==0.4.4, jupyter-events@==0.7.0, jupyter-lsp@==2.2.0, jupyterlab@==4.0.6, jupyterlab_server@==2.25.0, jupyterlab-pygments@==0.2.2, jupyterlab-widgets, kiwisolver, MarkupSafe, matplotlib, matplotlib-inline, mistune@==3.0.1, mkl-fft@==1.3.6, mkl-random, mkl-service@==2.4.0, mpmath, munkres@==1.1.4, nbclient@==0.8.0, nbconvert@==7.8.0, nbformat@==5.9.2, nest-asyncio, networkx, nodeenv, notebook@==7.0.4, notebook_shim@==0.2.3, numexpr, numpy, opencv-python@==4.8.1.78, overrides@==7.4.0, packaging, pandas, pandocfilters@==1.5.0, parso, pexpect, pickleshare, Pillow@==9.4.0, platformdirs, plotly, pluggy, ply@==3.11, pre-commit, prometheus-client@==0.17.1, prompt-toolkit, psutil, ptyprocess, pure-eval, pycparser, Pygments, pyparsing, PyQt5-sip@==12.11.0, pyserial, PySocks, pytest, python-dateutil, python-json-logger@==2.0.7, pytz, PyYAML, pyzmq, referencing@==0.30.2, regex@==2023.10.3, requests, rfc3339-validator@==0.1.4, rfc3986-validator@==0.1.1, rpds-py@==0.10.3, safetensors@==0.4.0, scipy, Send2Trash@==1.8.2, sip, six, sniffio@==1.3.0, soupsieve@==2.5, stack-data, sympy, tenacity, terminado@==0.17.1, tinycss2@==1.2.1, tokenizers@==0.14.1, toml, tomli, torch@==2.0.1, torchaudio@==2.0.2, torchvision@==0.15.2, tornado, tqdm@==4.66.1, traitlets, transformers@==4.34.1, typing_extensions, tzdata, ukkonen, uri-template@==1.3.0, urllib3, virtualenv, wcwidth, webcolors@==1.13, webencodings@==0.5.1, websocket-client@==1.6.3, widgetsnbextension, zipp
### Recent commits (newest first)
- set up docker
- Update dependencies
- Update README.md
- Delete video-processing.ipynb
- Update video-processing.ipynb
- done
- changed name
- deployment
- Merge branch 'main' of https://github.com/ryanmckim/CalHacks2023
- Add image audio output
- allow image upload too
- Fix filename
- Minor fix
- Fix missing import
- Merge branch 'main' of github.com:ryanmckim/CalHacks2023 into main
- Add image functionality
- Edit README.md
- change name
- Organize requirements
- Add video files
## Key source files (fetched from GitHub, selected and truncated for size)
### docker-compose.yml
```yaml
version: '0.0.1'
services:
api:
build:
context: .
dockerfile: Dockerfile.api
image: tuneai-app-api
client:
build:
context: .
dockerfile: Dockerfile.client
image: tuneai-app-client
ports:
- "3000:80"
```
### frontend/package.json
```
{
"name": "frontend",
"version": "0.1.0",
"private": true,
"dependencies": {
"@testing-library/jest-dom": "^5.17.0",
"@testing-library/react": "^13.4.0",
"@testing-library/user-event": "^13.5.0",
"axios": "^1.6.0",
"http-proxy-middleware": "^2.0.6",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-router-dom": "^6.17.0",
"react-scripts": "5.0.1",
"web-vitals": "^2.1.4"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
```
### backend/requirements.txt
```
��a i o h t t p = = 3 . 9 . 3
a i o s i g n a l = = 1 . 3 . 1
a t t r s = = 2 3 . 2 . 0
b l i n k e r = = 1 . 7 . 0
c e r t i f i = = 2 0 2 4 . 2 . 2
c h a r s e t - n o r m a l i z e r = = 3 . 3 . 2
c l i c k = = 8 . 1 . 7
c o l o r a m a = = 0 . 4 . 6
f i l e l o c k = = 3 . 1 3 . 1
F l a s k = = 3 . 0 . 0
F l a s k - C o r s = = 4 . 0 . 0
f r o z e n l i s t = = 1 . 4 . 1
f s s p e c = = 2 0 2 3 . 1 2 . 2
h u g g i n g f a c e - h u b = = 0 . 2 0 . 3
i d n a = = 3 . 6
i t s d a n g e r o u s = = 2 . 1 . 2
J i n j a 2 = = 3 . 1 . 3
M a r k u p S a f e = = 2 . 1 . 4
m p m a t h = = 1 . 3 . 0
m u l t i d i c t = = 6 . 0 . 5
n e t w o r k x = = 3 . 2 . 1
n u m p y = = 1 . 2 6 . 3
o p e n a i = = 0 . 2 8 . 1
o p e n c v - p y t h o n = = 4 . 9 . 0 . 8 0
p a c k a g i n g = = 2 3 . 2
p i l l o w = = 1 0 . 2 . 0
p y t h o n - d o t e n v = = 1 . 0 . 0
P y Y A M L = = 6 . 0 . 1
r e g e x = = 2 0 2 3 . 1 2 . 2 5
r e q u e s t s = = 2 . 3 1 . 0
s a f e t e n s o r s = = 0 . 4 . 2
s c i p y = = 1 . 1 2 . 0
s y m p y = = 1 . 1 2
t o k e n i z e r s = = 0 . 1 5 . 1
t o r c h = = 2 . 2 . 0
t o r c h a u d i o = = 2 . 2 . 0
t o r c h v i s i o n = = 0 . 1 7 . 0
t q d m = = 4 . 6 6 . 1
t r a n s f o r m e r s = = 4 . 3 7 . 2
t y p i n g _ e x t e n s i o n s = = 4 . 9 . 0
u r l l i b 3 = = 2 . 2 . 0
W e r k z e u g = = 3 . 0 . 1
y a r l = = 1 . 9 . 4
```
### requirements/requirements.txt
```
accelerate==0.24.0
anyio==4.0.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
asttokens
async-lru==2.0.4
attrs==23.1.0
av==10.0.0
Babel==2.12.1
backcall
backports.functools-lru-cache
beautifulsoup4==4.12.2
bleach==6.0.0
Bottleneck
Brotli
certifi==2023.7.22
cffi
cfgv
charset-normalizer
colorama
comm
contourpy
cycler
debugpy
decorator
defusedxml==0.7.1
diffusers==0.21.4
distlib
exceptiongroup
executing
fastjsonschema==2.18.0
filelock
fonttools==4.25.0
fqdn==1.5.1
fsspec==2023.10.0
huggingface-hub==0.17.3
identify
idna
importlib-metadata
iniconfig
intel-extension-for-pytorch==2.1.0
ipykernel
ipython
ipywidgets
isoduration==20.11.0
jedi
Jinja2
json5==0.9.14
jsonpointer==2.4
jsonschema==4.19.1
jsonschema-specifications==2023.7.1
jupyter-events==0.7.0
jupyter-lsp==2.2.0
jupyter_client
jupyter_core
jupyter_server==2.7.3
jupyter_server_terminals==0.4.4
jupyterlab==4.0.6
jupyterlab-pygments==0.2.2
jupyterlab-widgets
jupyterlab_server==2.25.0
kiwisolver
MarkupSafe
matplotlib
matplotlib-inline
mistune==3.0.1
mkl-fft==1.3.6
mkl-random
mkl-service==2.4.0
mpmath
munkres==1.1.4
nbclient==0.8.0
nbconvert==7.8.0
nbformat==5.9.2
nest-asyncio
networkx
nodeenv
notebook==7.0.4
notebook_shim==0.2.3
numexpr
numpy
opencv-python==4.8.1.78
overrides==7.4.0
packaging
pandas
pandocfilters==1.5.0
parso
pexpect
pickleshare
Pillow==9.4.0
platformdirs
plotly
pluggy
ply==3.11
pre-commit
prometheus-client==0.17.1
prompt-toolkit
psutil
ptyprocess
pure-eval
pycparser
Pygments
pyparsing
PyQt5-sip==12.11.0
pyserial
PySocks
pytest
python-dateutil
python-json-logger==2.0.7
pytz
PyYAML
pyzmq
referencing==0.30.2
regex==2023.10.3
requests
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rpds-py==0.10.3
safetensors==0.4.0
scipy
Send2Trash==1.8.2
sip
six
sniffio==1.3.0
soupsieve==2.5
stack-data
sympy
tenacity
terminado==0.17.1
tinycss2==1.2.1
tokenizers==0.14.1
toml
tomli
torch==2.0.1
torchaudio==2.0.2
torchvision==0.15.2
tornado
tqdm==4.66.1
traitlets
transformers==4.34.1
typing_extensions
tzdata
ukkonen
uri-template==1.3.0
urllib3
virtualenv
wcwidth
webcolors==1.13
webencodings==0.5.1
websocket-client==1.6.3
widgetsnbextension
zipp
```
### backend/server.py
```python
import os
from flask import Flask, send_from_directory, request
from flask_cors import cross_origin
app = Flask(__name__)
video_audio_mapping = {
"nature_img.jpg": "nature_img.wav",
"party.mp4": "party.wav",
"xmas.mp4": "xmas.wav"
}
# Socket IO connection
# def generate_audio():
# audio_data = "data"
# yield f"data: {audio_data}\n\n"
# @app.route('/get_audio_stream')
# def get_audio_stream():
# return Response(generate_audio(), content_type="text/event-stream")
@app.route("/get_audio/")
@cross_origin()
def get_audio(filename):
return send_from_directory("./audio/", video_audio_mapping.get(filename))
@app.route("/upload", methods=['POST'])
@cross_origin()
def upload_video():
video = request.files['video']
if video:
file_path = "./video/" + video.filename
video.save(file_path)
# send file path to ML model
return "Upload successful!"
return "Upload unsuccessful."
if __name__ == "__main__":
app.run(debug=True)
```
### frontend/src/index.js
```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';
import reportWebVitals from './reportWebVitals';
import { BrowserRouter } from "react-router-dom";
const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
);
reportWebVitals();
```
### frontend/src/App.js
```javascript
import "./App.css";
import { Routes, Route } from "react-router-dom";
import Home from "./pages/Home";
import Demo from "./pages/Demo";
export default function App() {
return (
} />
} />
Page not found
}
/>
);
}
```
### utilities.py
```python
from pathlib import Path
from typing import Any, Dict, List
from urllib.request import urlretrieve
def download_file(url: str) -> Path:
example_dirpath = Path(__file__).parent
data_dirpath = example_dirpath / "data"
data_dirpath.mkdir(exist_ok=True)
filepath = data_dirpath / Path(url).name
urlretrieve(url, filepath)
return filepath
def print_emotions(emotions: List[Dict[str, Any]]) -> None:
emotion_map = {e["name"]: e["score"] for e in emotions}
for emotion in ["Joy", "Sadness", "Anger"]:
print(f"- {emotion}: {emotion_map[emotion]:4f}")
```
### humeai.py
```python
import asyncio
import traceback
from utilities import download_file, print_emotions
from hume import HumeStreamClient, StreamSocket
from hume.models.config import FaceConfig
import cv2
import os
from dotenv import load_dotenv # for loading api keys
filepath = download_file("https://storage.googleapis.com/hume-test-data/image/obama.png")
async def main():
try:
load_dotenv()
client = HumeStreamClient(os.environ.get("HUMEAI_API_KEY"))
config = FaceConfig(identify_faces=True)
async with client.connect([config]) as socket:
cap = cv2.VideoCapture(0)
if not cap.isOpened():
print("Cannot open camera")
return
while True:
ret, frame = cap.read()
if not ret:
print("Can't receive frame (stream end?). Exiting ...")
break
result = await socket.send_frame(frame)
print(result)
if cv2.waitKey(1) == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
#result = await socket.send_file(filepath)
#print(result)
except Exception:
print(traceback.format_exc())
asyncio.run(main())
```
### gen_model.py
```python
import cv2
import os
import sys
import numpy as np
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration, AutoProcessor, MusicgenForConditionalGeneration
import scipy # for saving to wav file
import openai
from dotenv import load_dotenv # for loading api keys
class BlipModelSingleton:
_instance = None
@classmethod
def get_instance(cls):
if cls._instance is None:
cls._instance = cls._create_instance()
return cls._instance
@classmethod
def _create_instance(cls):
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
return processor, model
class MusicModelSingleton:
_instance = None
@classmethod
def get_instance(cls):
if cls._instance is None:
cls._instance = cls._create_instance()
return cls._instance
@classmethod
def _create_instance(cls):
music_processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
music_model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
return music_processor, music_model
def create_snapshots(video_path, output_path='data/frames_out', interval_seconds=2):
cap = cv2.VideoCapture(video_path)
directory = output_path
os.makedirs(directory, exist_ok=True)
fps = cap.get(cv2.CAP_PROP_FPS)
interval_seconds = interval_seconds
interval_frames = int(fps * interval_seconds)
count = 1
if not cap.isOpened():
print("Error opening video file")
else:
while cap.isOpened():
# Capture frame-by-frame
ret, frame = cap.read()
if not ret:
break
if count % interval_frames == 0:
output_path = os.path.join(directory, f"frame_{count}.jpg")
cv2.imwrite(output_path, frame)
print(f"Saved frame {count}")
count += 1
cap.release()
# output directory is a directory of jpg images, returns a list of string captions
def caption_snapshots(directory):
processor, model = BlipModelSingleton.get_instance()
snapshots = []
if os.path.exists(directory):
for filename in os.listdir(directory):
if filename.endswith(".jpg"):
raw_image = cv2.imread(os.path.join(directory, filename))
raw_image = cv2.cvtColor(raw_image, cv2.COLOR_BGR2RGB)
# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt")
out = model.generate(**inputs)
snapshots.append(processor.decode(out[0], skip_special_tokens=True))
else:
print(f"Directory '{directory}' does not exist.")
return snapshots
# prompt is a list of descriptions taken from snapshots in the video
def openai_prompt(prompt):
prompt_str = '\", \"'.join(prompt)
conversation = [
{"role": "user", "content": "You are an expert musician, with the ability to convert scenery to emotions."},
{"role": "user", "content": 'Give me the mood, genre, and feeling of this description: [\"' + prompt_str + '\"]'}
]
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=conversation
)
print("asking gpt to create a music description...")
conversation.append({"role": "user", "content": "Summarize your response describing matching music in the single expression format: style, adjectives, genre with instrumentation, sounds, musical descriptors - like the example: \"80s pop track with bassy drums and synth.\""})
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=conversation
)
print(response)
return response['choices'][0]['message']['content']
# Function for generating audio from text descriptions
def generate_audio(descriptions, path="musicgen_out.wav"):
processor, model = MusicModelSingleton.get_instance()
#descriptions = ["acoustic folk song to play during roadtrips: guitar flute choirs"]
inputs = processor(
text=descriptions,
padding=True,
return_tensors="pt",
)
audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=512)
sampling_rate = model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write(path, rate=sampling_rate, data=audio_values[0, 0].numpy())
def main(input_path):
np.random.seed(45)
load_dotenv()
openai.api_key = os.environ.get("OPENAI_API_KEY")
print("loaded keys")
if not os.path.exists(input_path):
print("Error: The specified input path does not exist.")
return
if input_path.endswith('.mp4'):
filename = input_path.split('/')[-1].split('.')[0]
output_path = f"data/{filename}_out"
create_snapshots(input_path, output_path)
print("Done snapshots.")
captions = caption_snapshots(output_path)
print(captions)
music_description = openai_prompt(captions)
print(music_description)
generate_audio(music_description, path=f"backend/audio/{filename}.wav")
elif input_path.endswith('.jpg'):
filename = input_path.split('/')[-1].split('.')[0]
# describe image
raw_image = cv2.imread(input_path)
raw_image = cv2.cvtColor(raw_image, cv2.COLOR_BGR2RGB)
processor, model = BlipModelSingleton.get_instance()
inputs = processor(raw_image, return_tensors="pt")
out = model.generate(**inputs)
description = (processor.decode(out[0], skip_special_tokens=True))
print(description)
music_description = openai_prompt([description])
print(music_description)
generate_audio(music_description, path=f"backend/audio/{filename}.wav")
else:
print("Er
[truncated — 661 more characters]
```
[11 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]