Project Info

Winner

AppLovin: Ad Intelligence Challenge

ADLOVIN

Inspiration

The inspiration came from real-world ad recommendation systems like AppLovin’s Axon, which rely on large datasets of ad creatives to predict performance. We wanted to explore how far we could push multimodal intelligence by building a system that understands ad content visually, textually, and acoustically. Our goal was to make ad data more insightful and useful for smarter recommendation models.

What it does

ADLOVIN Media Intelligence processes ad creatives, both images and videos, and extracts high-value multimodal features such as visual composition, text sentiment, and audio mood. These embeddings can then be used to generate creative performance insights or power downstream machine learning models for recommendation and ranking.

How we built it

We built a modular feature extraction pipeline in Python. For video ads, we used FFmpeg to extract keyframes and Librosa for audio signal analysis. Each frame was embedded using a pretrained vision transformer, and text elements were captured through EasyOCR and language models. We used ChromaDB to store and query embeddings at scale, and FastAPI to provide an interface for experimentation and testing.

Challenges we ran into

One major challenge was optimizing feature extraction for large videos while maintaining temporal coherence between visual and audio signals. Another was ensuring meaningful clustering in the vector database without redundant representations, especially after removing the deduplication algorithm. Balancing efficiency and fidelity across multiple modalities required careful design and parameter tuning. One of the major issues faced was the wifi. It made our life really hard. We had to find place to work outside of the venue.

Accomplishments we're proud of

We successfully built an end-to-end multimodal intelligence pipeline that can process a wide range of ad creatives and output interpretable embeddings. The system achieved reliable text and sentiment extraction from frames and robust feature representation using transformer-based models.

What we learned

We learned how to align multimodal embeddings for recommendation tasks, fine-tune vector similarity searches for creative clustering, and manage large-scale ad processing efficiently. We also gained experience combining computer vision, audio analysis, and natural language understanding into one cohesive system.

What's next

Media Intelligence Next, we plan to add a lightweight deduplication model, connect the embeddings to a performance dataset for supervised learning, and build an interactive dashboard to visualize ad intelligence insights in real time. We also aim to explore generative tools that can use embeddings to suggest creative improvements.

Ad Lovin Frontend

A React TypeScript frontend for testing the Flask API server.

🚀 Quick Start

1. Install Dependencies

npm install

2. Start the Development Server

npm run dev

The frontend will start on http://localhost:3000

3. Make Sure Your Flask Backend is Running

In a separate terminal, start your Flask backend:

cd ../your-backend-directory
python3 api_server.py

The backend should be running on http://localhost:5001

📋 Available Scripts

npm run dev - Start development server
npm run build - Build for production
npm run preview - Preview production build

🔗 API Connection

The frontend is configured to connect to your Flask API at http://localhost:5001/api.

If your API is running on a different port or host, update the API_BASE_URL in src/App.tsx:

const API_BASE_URL = "http://localhost:5001/api";

✨ Features

Human Detection: Upload an image and detect if humans are present
Audio Analysis: Upload a video and analyze the audio features including:
- Duration
- Gender estimation
- Mean pitch
- Spectral bandwidth
- Emotion detection

🛠️ Technology Stack

React 18 - UI framework
TypeScript - Type safety
Vite - Fast build tool and dev server
Modern CSS - Responsive styling

📝 Project Structure

.
├── src/
│   ├── App.tsx        # Main React component
│   ├── main.tsx       # React entry point
│   └── index.css      # Global styles
├── index.html         # HTML template
├── vite.config.ts     # Vite configuration
├── tsconfig.json      # TypeScript configuration
└── package.json       # Dependencies

🐛 Troubleshooting

Port Already in Use

If port 3000 is already in use, Vite will automatically use the next available port (3001, 3002, etc.)

API Connection Issues

Make sure your Flask backend is running
Check the API_BASE_URL in src/App.tsx matches your backend URL
Check browser console for CORS or connection errors

CORS Errors

If you see CORS errors in the browser console, make sure your Flask backend has CORS enabled for localhost:3000.

Analysis

Compare with all teams

View

Metric

Harshit
1

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

CSSIn code
HTMLIn code
JavaScriptIn code
ReactIn code
Tailwind CSSIn code
TypeScriptIn code
Hugging FaceClaimed
PythonClaimed

6 of 8 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

26 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.

Feature verification

1 Verified3 Code-supported6 Claimed only1 Blocked

React/TypeScript frontend for testing the APIVerified
A React TypeScript frontend for testing the Flask API server
Claimed on readmehigh confidence
- src/App.tsx:386 — Full React/TypeScript App component with file upload, preview, and results rendering
- package.json:2 — Project is set up as a Vite + React + TypeScript app named ad-lovin-frontend
Audio analysis for video ads (duration, gender, pitch, emotion)Code-supported
Upload a video and analyze audio features including duration, gender estimation, mean pitch, spectral bandwidth, emotion detection
Claimed on readmemedium confidence
- src/App.tsx:445 — Frontend calls POST /api/analyze-audio and renders duration_sec, gender_estimation, mean_pitch, emotion from the response, but no backend audio processing code (e.g. Librosa) exists in this clone
Image analysis (human detection, OCR, marketing signals, NSFW check)Code-supported
Upload an image and detect humans, extract text, and get marketing/brand signals
Claimed on readmemedium confidence
- src/App.tsx:436 — Frontend calls POST /api/analyze-image and renders human_present, extracted_text, brand_name_text, is_nsfw etc. from the response, but the backend implementing this endpoint is not present in this repo
Video frame/marketing analysis (brand, promo, CTA detection across frames)Code-supported
Video ads are processed to extract brand name, promo, CTA, and frame-level statistics
Claimed on readmemedium confidence
- src/App.tsx:449 — Frontend calls POST /api/analyze-video and renders consolidated_video_analysis and frame counts, but the video processing backend is not in this repo
Audio mood extraction via LibrosaClaimed only
FFmpeg extracts keyframes and Librosa performs audio signal analysis for mood
Claimed on Devposthigh confidence
ChromaDB vector storage and querying of embeddingsClaimed only
ChromaDB used to store and query embeddings at scale for clustering
Claimed on Devposthigh confidence
Downstream ML / recommendation ranking use of embeddingsClaimed only
Embeddings power downstream machine learning models for recommendation and ranking
Claimed on Devposthigh confidence
FastAPI experimentation interfaceClaimed only
FastAPI provides an interface for experimentation and testing
Claimed on Devposthigh confidence
Text sentiment extraction via language modelsClaimed only
Text elements captured through EasyOCR and language models, with sentiment extraction
Claimed on Devposthigh confidence
Visual composition / vision transformer embeddingsClaimed only
Each frame is embedded using a pretrained vision transformer to extract visual composition features
Claimed on Devposthigh confidence
Deduplication algorithm for embeddingsBlocked
Deduplication algorithm was removed and is planned to be re-added as a lightweight model
Claimed on Devposthigh confidence

An AI agent derived these features from the project’s Devpost page and readme, then searched the code for each one. Verified features are backed by cited code; claimed-only features had no supporting code, which is not by itself proof a feature is missing.