# Project export: Spielberg AI
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2025
- Tagline: AI-powered video editing with CUDA optimization—our advanced AI agent executes complex edits with natural language, slashing 10-minute processing tasks to seconds using the GPU!
- Devpost: https://devpost.com/software/spielburg-ai
- GitHub: https://github.com/adeng27/treehacks-25
- Video: https://www.youtube.com/embed/EWgFPhaG48E?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Rox: Best Agents Hack ($7k Cash))
- Team: 2 GitHub contributor(s) — Alastair Deng (25 commits), mrTSB (19 commits)
## Devpost submission (written by the team)
### Inspiration
& Social Impact Our inspiration came from a close friend—a YouTuber who constantly struggled with the repetitive and time-consuming nature of short-form content editing. Despite having great content ideas, they found the process of cutting clips, applying effects, and formatting videos for multiple platforms incredibly tedious. We wanted to eliminate the frustration and automate as much of the workflow as possible. Our goal was to build an AI-powered video editor that understands natural language commands and applies professional-grade edits in seconds, letting creators focus on their content, not the editing grind. We achieved this goal through Spielberg AI, a novel video editor that works solely with natural language (and one small button that does all the work for you—if you want). Currently, there are so many stories that people want to share and virality will spread them across the entire world. This is all limited by editing which is extremely difficult and unintuitive. People often do not know the formula for virality and Spielberg AI will eliminate the barrier of people telling their stories by making editing trivial.
### What it does
Spielberg AI is an AI-powered video editor that allows creators to edit videos using only natural language. Instead of manually cutting, trimming, and applying effects, users can describe their desired edits in plain English, and Spielberg AI—powered by the agent, our AI agent—handles the rest. There is a strong emphasis on developing and editing viral videos for the users.
### How we built it
FFMPEG - Video Encoding The core of our project is built around FFMPEG, the industry-standard video encoding and decoding backend. We started with a simple real-time preview system using FFMPEG to encode frames and allow for precise, frame-focused cuts. From there, we integrated AI-driven enhancements to streamline and automate complex video editing tasks. To push performance to the extreme, we custom-compiled FFMPEG with NVENC encoding and CUDA hardware acceleration. This optimization massively reduced processing times, enabling near-instantaneous rendering of complex edits. What once took minutes now executes in mere seconds (on our gaming computer). AI Agent Spielberg is driven by the agent, a highly advanced AI agent designed to interpret natural language commands and execute sophisticated editing tasks seamlessly. The agent's intelligence is built upon a multi-layered NLP and processing pipeline, leveraging: Google Gemini for high-level natural language parsing and all multimodal support. All image and video understanding is completed through Gemini to enable the agent to understand what is occurring in the video that is provided by the user. We tested the video and frame processing capabilities of many models and Gemini's multimodal capabilities far surpassed all other models. The agent would fail to understand what is happening in the video, and thereby be unable to process and edit the videos properly without Gemini. We would say using Gemini to make a video editor possible is a pretty cool use case! Google Gemini for high-level natural language parsing and all multimodal support. All image and video understanding is completed through Gemini to enable the agent to understand what is occurring in the video that is provided by the user. We tested the video and frame processing capabilities of many models and Gemini's multimodal capabilities far surpassed all other models. The agent would fail to understand what is happening in the video, and thereby be unable to process and edit the videos properly without Gemini. We would say using Gemini to make a video editor possible is a pretty cool use case! OpenAI models & Whisper for speech-to-text, intelligent audio processing, and chat integrations. The agent utilizes OpenAI in a fairly creative way leveraging structured output and repeated sampling to understand what the user wants accomplished and how we can go about accomplishing it. The Whispr API is also used for video understanding. We found that audio is a great way to understand the video and this is all accomplished through Whispr, as other APIs were incredibly faulty. Through the agent's understanding developed with Whispr combined with GPT calls, we could identify where to put sound effects and trim the video based on dead audio. OpenAI models & Whisper for speech-to-text, intelligent audio processing, and chat integrations. The agent utilizes OpenAI in a fairly creative way leveraging structured output and repeated sampling to understand what the user wants accomplished and how we can go about accomplishing it. The Whispr API is also used for video understanding. We found that audio is a great way to understand the video and this is all accomplished through Whispr, as other APIs were incredibly faulty. Through the agent's understanding developed with Whispr combined with GPT calls, we could identify where to put sound effects and trim the video based on dead audio. Multimodal Retrieval-Augmented Generation (RAG) Pipeline for dynamically sourcing editing patterns and context-aware adjustments. We consulted various YouTubers with millions of subscribers and views on how to make viral short-form content. They pointed us to resources and gave us some of their own personal guides. We also used OpenAI DeepResearch & Perplexity to create informational documents on how to make a viral video. The agent then uses these documents in a Multimodal RAG pipeline to understand how to make a viral video so with the press of a button, the agent knows what's best and can execute it by interacting with our software infrastructure. Multimodal Retrieval-Augmented Generation (RAG) Pipeline for dynamically sourcing editing patterns and context-aware adjustments. We consulted various YouTubers with millions of subscribers and views on how to make viral short-form content. They pointed us to resources and gave us some of their own personal guides. We also used OpenAI DeepResearch & Perplexity to create informational documents on how to make a viral video. The agent then uses these documents in a Multimodal RAG pipeline to understand how to make a viral video so with the press of a button, the agent knows what's best and can execute it by interacting with our software infrastructure. Verification Layers for the agent to ensure correct outputs at each step of the process and limit hallucinations. Given our research experience in inference time compute at Hazy Research in Stanford AI Laboratory (SAIL), we have a strong understanding of SOTA Verification. We ran quick small-scale experiments regarding sampling, statistical methods, Unit Test Generation/Evaluation, LM Judges, and Reward Models, and used the best judge for each layer. More often than not, it was LM Judges with some small combination of statistical methods. Verification Layers for the agent to ensure correct outputs at each step of the process and limit hallucinations. Given our research experience in inference time compute at Hazy Research in Stanford AI Laboratory (SAIL), we have a strong understanding of SOTA Verification. We ran quick small-scale experiments regarding sampling, statistical methods, Unit Test Generation/Evaluation, LM Judges, and Reward Models, and used the best judge for each layer. More often than not, it was LM Judges with some small combination of statistical methods. The backend is structured into multiple Python layers, each optimized for efficiency, with distinct modules handling inference, video processing, and audio synthesis. The agent seamlessly works across these layers. Advanced Hardware Acceleration with CUDA Our project takes full advantage of NVIDIA GPUs and CUDA acceleration to handle the most computationally intensive workloads. Encoding & Decoding: By using a custom-compiled FFMPEG with CUDA-powered NVENC, we parallelized the encoding of 4K video, dramatically improving performance for long-form content like podcasts. Encoding & Decoding: By using a custom-compiled FFMPEG with CUDA-powered NVENC, we parallelized the encoding of 4K video, dramatically improving performance for long-form content like podcasts. Real-time Color Grading: The most computationally demanding process was applying AI-driven color filters to videos. Instead of relying on FFMPEG’s built-in filters (which took 40+ seconds for a 10-minute 720p video), we wrote a fully custom CUDA kernel from scratch. Real-time Color Grading: The most computationally demanding process was applying AI-driven color filters to videos. Instead of relying on FFMPEG’s built-in filters (which took 40+ seconds for a 10-minute 720p video), we wrote a fully custom CUDA kernel from scratch. Using OpenAI APIs, we generated LUT-based color filters from natural language prompts. Our CUDA kernel applied these filters nearly instantly, outperforming traditional GPU-accelerated FFMPEG operations. Thread-Level Parallelism: CUDA threads were optimized not only for video processing but also for speeding up AI operations, such as frame analysis via Gemini, transcription, and audio synthesis tasks. API Integrations Beyond video processing, we integrated several key APIs to enhance the intelligence and automation of the editor: Luma API for advanced visual effects. Luma AI enables us to generate videos using their SOTA VLM. We aim to keep image characteristics the same and generate videos. This is incredibly important as these days some of the leading viral short-form content is AI-generated videos, so we needed a way for creators to develop AI videos using natural language, our integration allows the user to do this. Luma API for advanced visual effects. Luma AI enables us to generate videos using their SOTA VLM. We aim to keep image characteristics the same and generate videos. This is incredibly important as these days some of the leading viral short-form content is AI-generated videos, so we needed a way for creators to develop AI videos using natural language, our integration allows the user to do this. ElevenLabs API is used to generate voiceovers. Most viral short-form content uses AI voiceovers and does not use a real person's voice. Our integration allows the user to generate an AI voiceover which is trendy and goes viral quicker, using their natural language chats. ElevenLabs API is used to generate voiceovers. Most viral short-form content uses AI voiceovers and does not use a real person's voice. Our integration allows the user to generate an AI voiceover which is trendy and goes viral quicker, using their natural language chats.
### Challenges we ran into
The biggest challenge we faced was the sheer computational demand of our backend operations. Web GPUs are significantly weak for video processing, which is incredibly demanding, and we quickly realized we needed a proper GPU to achieve this. To solve this Reilly drove down to UC Santa Cruz, convincing a friend to lend their high-end gaming PC for the project. This borrowed GPU became the foundation of our CUDA optimizations, allowing us to crank out the custom CUDA kernel that enabled real-time AI-driven edits. We wanted to use a GPU that someone's actual computer has so real editors could leverage features in Spielberg and not need to use brev.dev to rent an H100. Additionally, this only optimizes the best features of Spielberg, but the average person can still use Spielberg, but they will just wait for longer. With the newfound power, we tuned our CUDA kernels, implemented multi-threaded AI inference, and reduced processing times by orders of magnitude. What started as an impossible challenge turned into our biggest breakthrough—a GPU-powered AI editor that processes in seconds what used to take minutes. Also, we found version control when we had a sloppy codebase was very difficult. In the end, "adeng" is our working branch as the main branch has been slightly corrupted and is no longer functional.
### Accomplishments we're proud of
Building a video editor in 36 hours? That's pretty cool and something we are proud of doing. Working with videos is extremely difficult and processing them with LLMs and having agents work with them and interact with them is very difficult. Successfully built an AI-powered video editor that edits using only natural language. Achieved real-time AI-driven video editing using custom CUDA-accelerated processing. Developed a Multimodal RAG system that learns from YouTube creators to optimize viral content. Integrated Google Gemini, OpenAI Whisper, Luma AI, and ElevenLabs to create a seamless editing experience.
### What we learned
As corny as it sounds, we learned that we can literally do anything if we put our minds to it. We learned so many new APIs, tools, and tricks and accomplished so much stuff we thought would be impossible without an army of engineers like Netflix or Adobe.
### What's next
We want to expand the agent's capabilities and finally take the time to deploy this project for the world to use. We know this is a serious problem for creators and we want to democratize content creation so everyone can tell stories about their lives. We would love to talk to Neo about turning this into a company.
## README (from the GitHub repository)
# 🚨 DISCLAIMER: USE THE `ADENG` BRANCH, MAIN SUFFERED AN UNFORTUNATE DEMISE 🚨
**This project requires you to use the `adeng` branch.**
Make sure you switch to `adeng` before making any changes.
This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app).
## Getting Started
First, run the development server:
```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.
This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel.
## Learn More
To learn more about Next.js, take a look at the following resources:
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome!
## Deploy on Vercel
The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details.
## Detected evidence (automated analysis)
Indexed codebase: 39 recognized source files, 216 KB.
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- Google Gemini (technology) — detected in the code
- Hugging Face (technology) — detected in the code
- LangChain (technology) — detected in the code
- Next.js (technology) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- PyTorch (technology) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
## Codebase structure (from repository index)
### Files (54 of 54)
```
.gitignore
eslint.config.mjs
i_love_nvidia/create_lut.py
i_love_nvidia/ffmpeg_lut
i_love_nvidia/ffmpeg_lut.cu
next.config.ts
package.json
postcss.config.mjs
python-layer/app.py
python-layer/captions.py
python-layer/create_video.py
python-layer/frame_refiner.py
python-layer/key_frame_detector.py
python-layer/readme.txt
python-layer/refiner.py
python-layer/requirements.txt
python-layer/voiceover.py
README.md
second-python-layer/main.py
second-python-layer/requirements.txt
second-python-layer/subtitled_3bfc763e.srt
src/app/api/analyze-frame/route.ts
src/app/api/combine-video-audio/route.ts
src/app/api/cut-video/route.ts
src/app/api/get-task-list/route.ts
src/app/api/handle-virality/route.ts
src/app/api/read-video/route.ts
src/app/api/save-video/route.ts
src/app/globals.css
src/app/layout.tsx
src/app/page.tsx
src/app/test/page.tsx
src/assets/ArrowRightIcon.tsx
src/assets/IdeateIcon.tsx
src/assets/PlayheadIcon.tsx
src/assets/TicksIcon.tsx
src/components/EmptyState.tsx
src/components/MessageInput.tsx
src/components/MessageItem.tsx
src/components/NavBar.tsx
src/components/VideoControls.tsx
src/utils/formatTime.ts
src/utils/getRandomInt.ts
src/utils/getRandomString.ts
src/utils/parseTimeToSeconds.ts
tailwind.config.ts
third-python-layer/app.py
third-python-layer/main.py
third-python-layer/pdf_hashes.pkl
third-python-layer/pdf_vectors.pkl
third-python-layer/readme.txt
third-python-layer/requirements.txt
third-python-layer/utils.py
tsconfig.json
```
### Dependencies
- package.json: @eslint/eslintrc@^3, @ffmpeg/ffmpeg@^0.12.15, @google/generative-ai@^0.21.0, @radix-ui/react-slot@^1.1.2, @radix-ui/react-toast@^1.2.6, @types/fluent-ffmpeg@^2.1.27, @types/node@^20, @types/react@^19, @types/react-dom@^19, clsx@^2.1.1, eslint@^9, eslint-config-next@15.1.7, fluent-ffmpeg@^2.1.3, lucide-react@^0.475.0, lumaai@^1.4.0, next@15.1.7, node-fetch@^3.3.2, openai@^4.85.1, postcss@^8, react@^19.0.0, react-dom@^19.0.0, react-icons@^5.4.0, tailwind-merge@^3.0.1, tailwindcss@^3.4.1, typescript@^5, zod@^3.24.2
- python-layer/requirements.txt: aiohappyeyeballs@==2.4.6, aiohttp@==3.11.12, aiosignal@==1.3.2, annotated-types@==0.7.0, anyio@==4.8.0, attrs@==25.1.0, cachetools@==5.5.1, certifi@==2025.1.31, charset-normalizer@==3.4.1, click@==8.1.8, contourpy@==1.3.1, cycler@==0.12.1, dataclasses-json@==0.6.7, decorator@==4.4.2, distro@==1.9.0, fastapi@==0.115.8, fonttools@==4.56.0, frozenlist@==1.5.0, google-auth@==2.38.0, google-genai@==1.2.0, h11@==0.14.0, httpcore@==1.0.7, httpx@==0.28.1, httpx-sse@==0.4.0, idna@==3.10, imageio@==2.37.0, imageio-ffmpeg@==0.6.0, jiter@==0.8.2, jsonpatch@==1.33, jsonpointer@==3.0.0, kiwisolver@==1.4.8, langchain@==0.3.18, langchain-community@==0.3.17, langchain-core@==0.3.35, langchain-text-splitters@==0.3.6, langsmith@==0.3.8, lumaai@==1.4.0, marshmallow@==3.26.1, matplotlib@==3.10.0, moviepy@==1.0.3, multidict@==6.1.0, mypy-extensions@==1.0.0, numpy@>=1.26.4,<2.0.0, openai@==1.63.0, opencv-python@==4.11.0.86, orjson@==3.10.15, packaging@==24.2, PeakUtils@==1.3.5, pillow@==10.4.0, proglog@==0.1.10, propcache@==0.2.1, pyasn1@==0.6.1, pyasn1_modules@==0.4.1, pydantic@==2.10.6, pydantic_core@==2.27.2, pydantic-settings@==2.7.1, pydub@==0.25.1, pyparsing@==3.2.1, python-dateutil@==2.9.0.post0, python-dotenv@==1.0.1, python-multipart@==0.0.20, PyYAML@==6.0.2, requests@==2.32.3, requests-toolbelt@==1.0.0, rsa@==4.9, scipy@==1.15.1, setuptools@==75.8.0, six@==1.17.0, sniffio@==1.3.1, SQLAlchemy@==2.0.38, starlette@==0.45.3, tenacity@==9.0.0, tqdm@==4.67.1, typing_extensions@==4.12.2, typing-inspect@==0.9.0, urllib3@==2.3.0, uvicorn@>=0.27.0, webrtcvad@==2.0.10, websockets@==14.2, wheel@==0.45.1, yarl@==1.18.3, zstandard@==0.23.0
- second-python-layer/requirements.txt: av@==14.1.0, clipsai@==0.2.1, dlib@==19.24.2, elevenlabs@==1.51.0, face-recognition@==1.3.0, fastapi@==0.115.8, git@+https://github.com/m-bain/whisperx.git, httpx@==0.27.2, matplotlib@==3.10.0, moviepy@==1.0.3, numpy@<2.0, openai@==1.63.0, opencv-python@==4.9.0.80, pydub@==0.25.1, python-dotenv@==1.0.1, uvicorn@==0.34.0
- third-python-layer/requirements.txt: elasticsearch@==8.11.1, fastapi@==0.109.2, google-generativeai@==0.3.2, huggingface-hub@==0.19.4, numpy@==1.26.3, pillow@==10.2.0, pydantic@==2.6.1, pypdf2@==3.0.1, python-dotenv@==1.0.1, python-multipart@==0.0.6, sentence-transformers@==2.2.2, torch@==2.2.1, transformers@==4.36.2, uvicorn@==0.27.1
### Recent commits (newest first)
- Update README.md
- Update README.md
- fix third layer requirements
- Merge pull request #3 from adeng27/tsb_dev
- Merge branch 'main' of https://github.com/adeng27/treehacks-25 into tsb_dev
- I LOVE NVIDIA
- add viralize back
- Merge branch 'main' of https://github.com/adeng27/treehacks-25 into tsb_dev
- buttons
- trim based on visual
- viralize button done
- Merge branch 'main' into adeng
- FRONTEND
- Merge pull request #2 from adeng27/tsb_dev
- Merge branch 'main' of https://github.com/adeng27/treehacks-25 into tsb_dev
- rag pipeline and tsx functional
- fix
- mount targon
- Merge branch 'main' of https://github.com/adeng27/treehacks-25 into tsb_dev
- SONA
## Key source files (fetched from GitHub, selected and truncated for size)
### package.json
```
{
"name": "treehacks-25",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev --turbopack",
"build": "next build",
"start": "next start",
"lint": "next lint"
},
"dependencies": {
"@ffmpeg/ffmpeg": "^0.12.15",
"@google/generative-ai": "^0.21.0",
"@radix-ui/react-slot": "^1.1.2",
"@radix-ui/react-toast": "^1.2.6",
"@types/fluent-ffmpeg": "^2.1.27",
"clsx": "^2.1.1",
"fluent-ffmpeg": "^2.1.3",
"lucide-react": "^0.475.0",
"lumaai": "^1.4.0",
"next": "15.1.7",
"node-fetch": "^3.3.2",
"openai": "^4.85.1",
"react": "^19.0.0",
"react-dom": "^19.0.0",
"react-icons": "^5.4.0",
"tailwind-merge": "^3.0.1",
"zod": "^3.24.2"
},
"devDependencies": {
"@eslint/eslintrc": "^3",
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"eslint": "^9",
"eslint-config-next": "15.1.7",
"postcss": "^8",
"tailwindcss": "^3.4.1",
"typescript": "^5"
}
}
```
### third-python-layer/requirements.txt
```
fastapi==0.109.2
uvicorn==0.27.1
google-generativeai==0.3.2
elasticsearch==8.11.1
python-multipart==0.0.6
pydantic==2.6.1
python-dotenv==1.0.1
pypdf2==3.0.1
numpy==1.26.3
sentence-transformers==2.2.2
pillow==10.2.0
torch==2.2.1
transformers==4.36.2
huggingface-hub==0.19.4
```
### second-python-layer/requirements.txt
```
fastapi==0.115.8
uvicorn==0.34.0
git+https://github.com/openai/whisper.git
python-dotenv==1.0.1
elevenlabs==1.51.0
httpx==0.27.2
openai==1.63.0
pydub==0.25.1
moviepy==1.0.3
opencv-python==4.9.0.80
dlib==19.24.2
face-recognition==1.3.0
av==14.1.0
matplotlib==3.10.0
numpy<2.0
clipsai==0.2.1
git+https://github.com/m-bain/whisperx.git
```
### python-layer/requirements.txt
```
aiohappyeyeballs==2.4.6
aiohttp==3.11.12
aiosignal==1.3.2
annotated-types==0.7.0
anyio==4.8.0
attrs==25.1.0
cachetools==5.5.1
certifi==2025.1.31
charset-normalizer==3.4.1
click==8.1.8
contourpy==1.3.1
cycler==0.12.1
dataclasses-json==0.6.7
decorator==4.4.2
distro==1.9.0
fastapi==0.115.8
fonttools==4.56.0
frozenlist==1.5.0
google-auth==2.38.0
google-genai==1.2.0
h11==0.14.0
httpcore==1.0.7
httpx==0.28.1
httpx-sse==0.4.0
idna==3.10
imageio==2.37.0
imageio-ffmpeg==0.6.0
jiter==0.8.2
jsonpatch==1.33
jsonpointer==3.0.0
kiwisolver==1.4.8
langchain==0.3.18
langchain-community==0.3.17
langchain-core==0.3.35
langchain-text-splitters==0.3.6
langsmith==0.3.8
lumaai==1.4.0
marshmallow==3.26.1
matplotlib==3.10.0
moviepy==1.0.3
multidict==6.1.0
mypy-extensions==1.0.0
numpy>=1.26.4,<2.0.0
openai==1.63.0
opencv-python==4.11.0.86
orjson==3.10.15
packaging==24.2
PeakUtils==1.3.5
pillow==10.4.0
proglog==0.1.10
propcache==0.2.1
pyasn1==0.6.1
pyasn1_modules==0.4.1
pydantic==2.10.6
pydantic-settings==2.7.1
pydantic_core==2.27.2
pydub==0.25.1
pyparsing==3.2.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-multipart==0.0.20
PyYAML==6.0.2
requests==2.32.3
requests-toolbelt==1.0.0
rsa==4.9
scipy==1.15.1
setuptools==75.8.0
six==1.17.0
sniffio==1.3.1
SQLAlchemy==2.0.38
starlette==0.45.3
tenacity==9.0.0
tqdm==4.67.1
typing-inspect==0.9.0
typing_extensions==4.12.2
urllib3==2.3.0
uvicorn>=0.27.0
webrtcvad==2.0.10
websockets==14.2
wheel==0.45.1
yarl==1.18.3
zstandard==0.23.0
```
### third-python-layer/app.py
```python
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import Dict
from utils import (
setup_storage,
index_documents,
format_timestamp_info,
hybrid_search,
vectorize_pdfs
)
# Initialize FastAPI app
app = FastAPI(title="Video Virality RAG API")
# Setup storage and index documents on startup
@app.on_event("startup")
async def startup_event():
setup_storage()
index_documents()
@app.post("/vectorize")
async def force_vectorize():
"""Force re-vectorization of all PDFs."""
try:
result = vectorize_pdfs(force=True)
return {"status": "success", "message": result}
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
class VideoQuery(BaseModel):
video_info: Dict[str, str] # timestamp -> information mapping
@app.post("/get-viral-recommendations")
async def get_viral_recommendations(video: VideoQuery):
try:
# Format the timestamp information into a structured query
query = format_timestamp_info(video.video_info)
# Get relevant contexts using hybrid search
relevant_contexts = hybrid_search(query)
# Format the response
formatted_contexts = []
for ctx in relevant_contexts:
formatted_contexts.append({
"content": ctx["content"],
"source": ctx["source"],
"page_number": ctx["page_num"],
"chunk_number": ctx["chunk_num"],
"total_chunks": ctx["total_chunks"]
})
return {
"status": "success",
"recommendations": formatted_contexts,
"query_used": query # Include the formatted query for reference
}
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8003)
```
### python-layer/app.py
```python
from fastapi import FastAPI, UploadFile, File, HTTPException
from fastapi.middleware.cors import CORSMiddleware
import os
from refiner import process_video
from create_video import generate_video
from captions import generate_subtitled_video
from voiceover import generate_voiceover
app = FastAPI()
# Update your allowed origins to include your development domain only
app.add_middleware(
CORSMiddleware,
allow_origins=[
"http://3.85.183.100:3000",
"http://localhost:3000" # production frontend domain
],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
expose_headers=["*"]
)
# Create directories if they don't exist
os.makedirs("raw", exist_ok=True)
os.makedirs("edited", exist_ok=True)
@app.post("/upload-video")
async def upload_video(filename: str):
if not filename.lower().endswith('.mp4'):
return {"error": "Only MP4 files are allowed"}
file_path = os.path.join("raw", filename)
# Check if file exists in raw directory
if not os.path.exists(file_path):
return {"error": "File not found in raw directory"}
# Process the video
try:
process_video(file_path)
output_path = os.path.join("edited", filename)
return {
"message": "Video processed successfully",
"original_file": filename,
"processed_file": output_path
}
except Exception as e:
return {"error": f"Error processing video: {str(e)}"}
@app.post("/create-video")
async def create_video(filename: str):
if not filename.lower().endswith('.mp4'):
return {"error": "Only MP4 files are allowed"}
file_path = os.path.join("raw", filename)
# Check if file exists in raw directory
if not os.path.exists(file_path):
return {"error": "File not found in raw directory"}
try:
output_path = generate_video(file_path)
return {
"message": "Video created successfully",
"original_file": filename,
"processed_file": output_path
}
except Exception as e:
return {"error": f"Error creating video: {str(e)}"}
@app.post("/generate-captions")
async def generate_captions(filename: str):
if not filename.lower().endswith('.mp4'):
return {"error": "Only MP4 files are allowed"}
file_path = os.path.join("raw", filename)
try:
output_path = generate_subtitled_video(file_path)
return {
"message": "Subtitled video created successfully",
"original_file": filename,
"processed_file": output_path
}
except Exception as e:
return {"error": f"Error generating captions: {str(e)}"}
@app.post("/generate-voiceover")
async def generate_voiceover_endpoint(filename: str):
"""
Endpoint to generate a voiceover for a video file.
Parameters:
filename (str): Name of the video file in the raw directory
Returns:
dict: Contains the path to the processed video file
"""
try:
input_path = os.path.join("raw", filename)
if not os.path.exists(input_path):
raise HTTPException(status_code=404, detail="Video file not found")
processed_file = await generate_voiceover(input_path)
return {
"status": "success",
"processed_file": processed_file
}
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
```
### third-python-layer/main.py
```python
# import os
# from fastapi import FastAPI, HTTPException
# from pydantic import BaseModel
# import google.generativeai as genai
# from elasticsearch import Elasticsearch
# import PyPDF2
# from pathlib import Path
# import numpy as np
# from dotenv import load_dotenv
# import json
# from typing import List, Dict
# import re
# # Load environment variables
# load_dotenv()
# # Initialize Gemini
# genai.configure(api_key=os.getenv("GOOGLE_API_KEY"))
# model = genai.GenerativeModel('gemini-pro')
# embedding_model = genai.GenerativeModel('embedding-001')
# # Initialize Elasticsearch
# es = Elasticsearch(os.getenv("ELASTICSEARCH_URL", "http://localhost:9200"))
# # Constants for chunking
# CHUNK_SIZE = 1000 # characters
# CHUNK_OVERLAP = 200 # characters
# MIN_CHUNK_SIZE = 100 # Minimum characters for a chunk to be considered valid
# # Create index if it doesn't exist
# INDEX_NAME = "viral_videos_knowledge"
# if not es.indices.exists(index=INDEX_NAME):
# es.indices.create(
# index=INDEX_NAME,
# body={
# "mappings": {
# "properties": {
# "content": {"type": "text"},
# "embedding": {"type": "dense_vector", "dims": 768},
# "page_num": {"type": "integer"},
# "chunk_num": {"type": "integer"}
# }
# }
# }
# )
# def clean_text(text: str) -> str:
# """Clean and normalize text."""
# # Remove excessive whitespace
# text = re.sub(r'\s+', ' ', text)
# # Remove special characters but keep punctuation
# text = re.sub(r'[^\w\s.,!?;:-]', '', text)
# return text.strip()
# def create_chunks(text: str, chunk_size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> List[str]:
# """Create overlapping chunks from text."""
# cleaned_text = clean_text(text)
# chunks = []
# # Split text into sentences (rough approximation)
# sentences = re.split(r'(?<=[.!?])\s+', cleaned_text)
# current_chunk = []
# current_length = 0
# for sentence in sentences:
# sentence_length = len(sentence)
# # If adding this sentence would exceed chunk size
# if current_length + sentence_length > chunk_size and current_length >= MIN_CHUNK_SIZE:
# # Save current chunk
# chunks.append(' '.join(current_chunk))
# # Start new chunk with overlap
# overlap_point = max(0, len(current_chunk) - int(len(current_chunk) * (overlap / chunk_size)))
# current_chunk = current_chunk[overlap_point:]
# current_length = sum(len(s) for s in current_chunk)
# current_chunk.append(sentence)
# current_length += sentence_length
# # Add the last chunk if it's long enough
# if current_length >= MIN_CHUNK_SIZE:
# chunks.append(' '.join(current_chunk))
# return chunks
# def process_pdf(pdf_path: str) -> List[Dict]:
# """Process PDF and extract text with page numbers and chunks."""
# documents = []
# with open(pdf_path, 'rb') as file:
# pdf_reader = PyPDF2.PdfReader(file)
# for page_num in range(len(pdf_reader.pages)):
# page = pdf_reader.pages[page_num]
# text = page.extract_text()
# if text.strip(): # Only process non-empty pages
# # Create chunks for this page
# chunks = create_chunks(text)
# # Create a document for each chunk
# for chunk_num, chunk in enumerate(chunks):
# if len(chunk.strip()) >= MIN_CHUNK_SIZE:
# documents.append({
# "content": chunk,
# "page_num": page_num + 1,
# "chunk_num": chunk_num + 1,
# "source": pdf_path,
# "total_chunks": len(chunks)
# })
# return documents
# def get_embedding(text: str) -> List[float]:
# """Get embeddings using Gemini."""
# result = embedding_model.embed_content(text)
# return result.embedding
# def index_documents():
# """Index all PDF documents in the pdfs directory."""
# pdf_dir = Path("pdfs")
# for pdf_path in pdf_dir.glob("*.pdf"):
# documents = process_pdf(str(pdf_path))
# for doc in documents:
# embedding = get_embedding(doc["content"])
# es.index(
# index=INDEX_NAME,
# document={
# "content": doc["content"],
# "embedding": embedding,
# "page_num": doc["page_num"],
# "chunk_num": doc["chunk_num"],
# "source": doc["source"],
# "total_chunks": doc["total_chunks"]
# }
# )
# es.indices.refresh(index=INDEX_NAME)
# # Index documents on startup
# index_documents()
# class VideoQuery(BaseModel):
# video_info: Dict[str, str] # timestamp -> information mapping
# def format_timestamp_info(video_info: Dict[str, str]) -> str:
# """Format timestamp information into a structured query."""
# formatted_query = "Video Timeline Information:\n"
# # Sort timestamps to maintain chronological order
# sorted_timestamps = sorted(video_info.keys())
# for timestamp in sorted_timestamps:
# formatted_query += f"At {timestamp}: {video_info[timestamp]}\n"
# return formatted_query
# def hybrid_search(query: str, k: int = 5):
# """Perform hybrid search using BM25 and vector similarity."""
# # Get query embedding
# query_embedding = get_embedding(query)
# # Hybrid search using both BM25 and vector similarity
# response = es.search(
# index=INDEX_NAME,
# body={
# "query": {
# "combined_fields": {
# "query": query,
#
[truncated — 1557 more characters]
```
### src/app/layout.tsx
```typescript
import type { Metadata } from "next";
import { Inter } from "next/font/google";
import "./globals.css";
import NavBar from "@/components/NavBar";
const inter = Inter({
variable: "--font-inter",
subsets: ["latin"],
});
export const metadata: Metadata = {
title: "Create Next App",
description: "Generated by create next app",
};
export default function RootLayout({
children,
}: Readonly<{
children: React.ReactNode;
}>) {
return (
Spielberg
{children}
);
}
```
### second-python-layer/main.py
```python
from dotenv import load_dotenv
from elevenlabs.client import ElevenLabs
from openai import OpenAI
import whisper
from fastapi import FastAPI, Body, HTTPException
from fastapi.responses import FileResponse
from moviepy.editor import VideoFileClip, AudioFileClip
from pydub import AudioSegment
from clipsai import ClipFinder, Transcriber
from moviepy.video.io.ffmpeg_tools import ffmpeg_extract_subclip
from pydantic import BaseModel
import threading
import face_recognition
import cv2
from fastapi.middleware.cors import CORSMiddleware
import boto3
import requests
from pathlib import Path
import time
import os
import subprocess
import uuid
from lumaai import LumaAI
app = FastAPI()
# Add CORS middleware configuration
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # Allows all origins
allow_credentials=True,
allow_methods=["*"], # Allows all methods
allow_headers=["*"], # Allows all headers
)
# Load environment variables
load_dotenv()
elevenlabs_client = ElevenLabs(api_key=os.getenv("ELEVENLABS_API_KEY"))
openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
# Load the Whisper model once to avoid reloading every request
model = whisper.load_model("base")
# Add after existing imports and before app initialization
class TaskResponse(BaseModel):
task_id: str
status: str
result: dict = None
# Add after app initialization
task_store = {}
task_lock = threading.Lock()
def get_video_duration(video_path: str) -> float:
"""Extracts and returns the duration of the video in seconds using MoviePy."""
video = VideoFileClip(video_path)
return video.duration
def estimate_speech_duration(text: str, wpm: int = 178) -> float:
"""Estimates the duration of a given text when spoken."""
word_count = len(text.split())
return (word_count / wpm) * 60 # duration in seconds
def change_audio_speed(audio_path: str, target_duration: float) -> str:
"""Changes the speed of the audio file to match the target duration using pydub."""
audio = AudioSegment.from_file(audio_path)
current_duration = len(audio) / 1000 # Convert ms to seconds
speed_change = current_duration / target_duration
# Adjust speed
new_audio = audio._spawn(audio.raw_data, overrides={
"frame_rate": int(audio.frame_rate * speed_change)
}).set_frame_rate(audio.frame_rate)
output_path = audio_path.replace(".mp3", "_adjusted.mp3")
new_audio.export(output_path, format="mp3")
return output_path
def merge_audio_with_video(video_path: str, audio_path: str, output_path: str):
"""Merges generated voiceover with the original video."""
video = VideoFileClip(video_path)
audio = AudioFileClip(audio_path)
video = video.set_audio(audio)
video.write_videofile(output_path, codec='libx264', audio_codec='aac')
def elevenlabs_voiceover(text: str, video_duration: float, voice_id: str = "JBFqnCBsd6RMkjVDRZzb") -> str:
"""Generates voiceover using ElevenLabs API and adjusts speed to match video duration."""
audio_stream = elevenlabs_client.text_to_speech.convert_as_stream(
text=text,
voice_id=voice_id,
model_id="eleven_multilingual_v2"
)
audio_chunks = [chunk for chunk in audio_stream if isinstance(chunk, bytes)]
complete_audio = b''.join(audio_chunks)
audio_path = f"voiceover_{uuid.uuid4().hex[:8]}.mp3"
with open(audio_path, "wb") as f:
f.write(complete_audio)
# Adjust the audio duration to match the video duration
adjusted_audio_path = change_audio_speed(audio_path, video_duration)
return adjusted_audio_path
def generate_transcript(prompt: str) -> str:
"""Generates transcript using OpenAI API."""
response = openai_client.chat.completions.create(
model="gpt-4-turbo-preview",
messages=[{"role": "user", "content": f"Generate a transcript for the following prompt: {prompt}"}]
)
return response.choices[0].message.content
def srt_timestamp_to_seconds(timestamp: str) -> float:
"""
Converts an SRT timestamp (e.g., "00:00:12,500") to seconds as a float.
"""
try:
hours, minutes, sec_milli = timestamp.split(":")
seconds, milliseconds = sec_milli.split(",")
total_seconds = int(hours) * 3600 + int(minutes) * 60 + int(seconds) + int(milliseconds) / 1000.0
return total_seconds
except Exception:
return 0.0
@app.post("/generate_sound_effects_video")
def generate_sound_effects_video(
video_path: str = Body(..., embed=True),
srt_path: str = Body(..., embed=True)
) -> dict:
"""
Adds sound effects to the provided video based on cues found in the SRT file.
The function:
- Reads the SRT file and looks for keywords in each subtitle.
- When a keyword is found (e.g., "meme", "explosion", "serious", "emotional"),
the corresponding local sound effect file is selected.
- It then uses ffmpeg to overlay these sound effect audio tracks (delayed to the proper timestamp)
on top of the original audio track.
Expected request body example:
{
"video_path": "/path/to/video.mp4",
"srt_path": "/path/to/subtitles.srt"
}
"""
# Check if the provided video and SRT files exist
if not os.path.isfile(video_path):
return {"error": f"Video file {video_path} does not exist."}
if not os.path.isfile(srt_path):
return {"error": f"SRT file {srt_path} does not exist."}
# Define a mapping from keywords to local sound effect files.
# You can adjust the keywords and file paths as needed.
effects_mapping = {
("meme", "funny"): "sounds/baby-laughing-meme.mp3",
("explosion", "boom", "kaboom"): "sounds/shocked-sound-effect.mp3",
("serious",): "sounds/y2mate_5gbydy1",
("emotional", "sad", "cry", "tears"): "sounds/emotional-damage-meme.mp3"
}
# Parse the SRT file to extract segments.
# Each segment is assumed to be separated by a blank line.
[truncated — 17987 more characters]
```
### src/app/test/page.tsx
```typescript
"use client";
import { useState, useRef, useEffect } from "react";
import MessageItem from "@/components/MessageItem";
import MessageInput from "@/components/MessageInput";
import EmptyState from "@/components/EmptyState";
import { formatTime } from "@/utils/formatTime";
import { parseTimeToSeconds } from "@/utils/parseTimeToSeconds";
import VideoControls from "@/components/VideoControls";
interface VideoFilters {
brightness: number;
contrast: number;
saturation: number;
rotation: number;
filter: string;
volume: number;
bassBoost?: number;
treble?: number;
}
export default function Home() {
const [video, setVideo] = useState(null);
const [videoUrl, setVideoUrl] = useState("");
const [startTime, setStartTime] = useState("00:00:00");
const [endTime, setEndTime] = useState("00:00:00");
const [processing, setProcessing] = useState(false);
const [duration, setDuration] = useState(0);
const [currentTime, setCurrentTime] = useState(0);
const [isPlaying, setIsPlaying] = useState(false);
const videoRef = useRef(null);
const [filters, setFilters] = useState({
brightness: 100,
contrast: 100,
saturation: 100,
rotation: 0,
filter: "none",
volume: 1,
});
const [messages, setMessages] = useState<
{ text: string; sender: "user" | "ai"; title?: string }[]
>([]);
const [newMessage, setNewMessage] = useState("");
const chatContainerRef = useRef(null);
const finishNowEventListenerRef = useRef<() => void>(() => {});
const [animationInProgress, setAnimationInProgress] =
useState(false);
const [animationFinished, setAnimationFinished] = useState(false);
const handleVideoUpload = (e: React.ChangeEvent) => {
const file = e.target.files?.[0];
if (file) {
if (videoUrl) {
URL.revokeObjectURL(videoUrl);
}
const newVideoUrl = URL.createObjectURL(file);
setVideo(file);
setVideoUrl(newVideoUrl);
setStartTime("00:00:00");
setEndTime("00:00:00");
}
};
const handleTimeUpdate = () => {
if (videoRef.current) {
setCurrentTime(videoRef.current.currentTime);
}
};
const handleLoadedMetadata = () => {
if (videoRef.current) {
setDuration(videoRef.current.duration);
setEndTime(formatTime(videoRef.current.duration));
}
};
useEffect(() => {
return () => {
if (videoUrl) {
URL.revokeObjectURL(videoUrl);
}
};
}, [videoUrl]);
const handleCutVideo = async (download: boolean = true) => {
if (!video) return;
setProcessing(true);
try {
const formData = new FormData();
formData.append("video", video);
formData.append(
"cuts",
JSON.stringify([
{
cutStartTime: 0,
cutEndTime: 2,
},
])
);
formData.append("filters", JSON.stringify(filters));
const response = await fetch("/api/cut-video", {
method: "POST",
body: formData,
});
if (response.ok) {
const blob = await response.blob();
const url = URL.createObjectURL(blob);
if (download) {
const a = document.createElement("a");
a.href = url;
a.download = "trimmed-video.mp4";
document.body.appendChild(a);
a.click();
URL.revokeObjectURL(url);
document.body.removeChild(a);
} else {
setVideo(
new File([blob], "trimmed-video.mp4", { type: "video/mp4" })
);
setVideoUrl(url);
}
}
} catch (error) {
console.error("Error cutting video:", error);
} finally {
setProcessing(false);
}
};
const handleRemoveVideo = () => {
if (videoUrl) {
URL.revokeObjectURL(videoUrl);
}
setVideo(null);
setVideoUrl("");
};
const handlePlayPause = () => {
if (animationInProgress) {
return;
}
if (videoRef.current) {
if (isPlaying) {
videoRef.current.pause();
} else {
videoRef.current.play();
}
setIsPlaying(!isPlaying);
}
};
const simulateTask = async () => {
if (videoRef.current) {
const currentEndTime = parseTimeToSeconds(endTime);
const newEndTime = Math.max(0, currentEndTime - 1);
setEndTime(formatTime(newEndTime));
handleCutVideo(false);
}
};
const handleSendMessage = async (e: React.FormEvent) => {
e.preventDefault();
if (!newMessage.trim()) return;
// Add user message
const userMessage = { text: newMessage, sender: "user" as const };
setMessages((prev) => [...prev, userMessage]);
setNewMessage("");
if (!animationInProgress && !animationFinished) {
setAnimationInProgress(true);
setIsPlaying(false);
} else if (animationInProgress) {
finishNowEventListenerRef.current();
}
const response = {
tasks: [
{
name: "generate_visuals",
description: "I will generate visuals now...",
},
{
name: "remove_unnecessary_audio",
description: "I will trim the video now...",
},
],
};
await new Promise((resolve) => setTimeout(resolve, 1000)); // Simulate API call
const tasks = response.tasks;
for (const task of tasks) {
const message = `${task.description}`; // TODO: Change to whatever looks best design wise
const aiMessage = {
text: message,
sender: "ai" as const,
title: "Editing...",
};
setMessages((prev) => [...prev, aiMessage]);
if (task.name === "generate_visuals") {
await simulateTask();
}
}
};
useEffect(() => {
// Scroll to bottom
if (chatContainerRef.current) {
chatContainerRef.current.scrollTo({
top: chatContainerRef.current.scrollHeight,
behavior: "smooth",
});
}
}, [messages]);
if (!video) {
return ;
}
return (
{/* Main Content */}
[truncated — 5329 more characters]
```
[32 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]