# Project export: AdJacent This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: AdJacent quickly scores video, image, and audio ads using signals—shots, motion, text/CTA, product, end-card, thumbnails, platform fit, sentiment—to rank creatives and guide edits. - Devpost: https://devpost.com/software/rad-ads - GitHub: https://github.com/qedsphere/ad-intelligence-challenge - Demo: https://docs.google.com/presentation/d/1gqU4AAzTmIAJAuaZPlkHWb7eG2QwmmI-LJNvAPx7mPM/edit?usp=sharing - Team: 3 GitHub contributor(s) — George Huang (10 commits), Hydroxyi (10 commits), ElShroomster (1 commits) ## Devpost submission (written by the team) ### Inspiration Creative teams guess which ads work; we'd much prefer fast, explainable signals across video, image, and audio to pick winners before spend. ### What it does Extracts features from images, videos, and audio. Merges video+audio per video; outputs JSON with interpret-able metrics (shots, motion, text/CTA timing, product reveal, end-card, thumbnails, faces, color, clutter, platform fit, loop-readiness, audio speech/music). ### How we built it We created a modular pipeline: image/: visual semantics, attention maps, OCR, optional LLM captioning. videos/: per-metric modules with parallel per-video execution and dependency waves. Sub-tasks are parallelize-able audio/: separation, ASR, speech/music/text analysis. Orchestrator that runs image, and in parallel runs video and audio, then merges by filename stem. ### Challenges we ran into Hitting sub‑second per-metric latency without GPUs. Aggregation can almost entirely be parallelized so the result will only take the longest of each metric, summed if dependencies exist. Avoiding redundant computation across steps (saliency/gray/MSER reuse). ### Accomplishments we're proud of End‑to‑end, explainable feature set with per‑metric breakdowns, parallel per‑video tasking and A/V concurrency, tweakable speed/quality knobs and weights, simple JSON outputs ready for ranking/ML. ### What we learned Lightweight CV proxies (saliency/MSER/flow) are strong baselines. Caching + downsampling matter more than we expected. Clear dependency waves simplify parallelism and debugging. ### What's next We still need to add lightweight logo/face detectors and text OCR keywords for CTAs. For full functionality, we should train a learned ranker on historical performance. We also need to add web UI for previews, breakdowns, and A/B comparisons. Integrating this with AppLovin would be the ultimate step! ## README (from the GitHub repository) # Ad Intelligence ## Setup 1. Run the setup script: ```bash ./setup.sh ``` 2. Activate the virtual environment: ```bash source .venv/bin/activate ``` The setup script will automatically download the ad assets and install all required dependencies. 3. Finally, install Tesseract: ``` brew install tesseract ``` ## TPP-Gaze Submodule Setup Required for the attention map features. 1. Install brew dependencies: ``` brew install wget ``` 2. Follow the setup instructions at https://github.com/phuselab/tppgaze: ``` git clone https://github.com/phuselab/tppgaze cd tppgaze pip install -r requirements.txt ``` 3. Within the adintelligence folder, run the following. ``` bash download_models.sh ``` ## To run Audio/Video/Image Metrics 1. Checkout the relevant branch. Images are on the main branch. ```bash git checkout [audio/videos/main] ``` 2. Activate the virtual environment and run the setup: ```bash source .venv/bin/activate [repeat steps above] ``` The setup script will automatically download the ad assets and install all required dependencies. 3. Read each branch's README to see how to run the scripts. Details on what is run will be described in the provided markdown files. ## Citation This project uses the **TPP-Gaze** model for gaze dynamics and scanpath prediction. If you use or build upon this component, please cite the following work: ```bibtex @inproceedings{damelio2025tppgaze, title = {TPP-Gaze: Modelling Gaze Dynamics in Space and Time with Neural Temporal Point Processes}, author = {D'Amelio, Alessandro and Cartella, Giuseppe and Cuculo, Vittorio and Lucchi, Manuele and Cornia, Marcella and Cucchiara, Rita and Boccignone, Giuseppe}, booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision}, year = {2025} } ```` For more information, visit the official repository: [https://github.com/phuselab/tppgaze](https://github.com/phuselab/tppgaze) ## Detected evidence (automated analysis) Indexed codebase: 26 recognized source files, 100 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 170) ``` .gitignore .gitmodules backend/main.py data/tpp data/tpp_ data/tppgaze_config.yaml extracted_features/features_index.csv extracted_features/i0001/i0001_attention_heatmap.npz extracted_features/i0001/i0001_attention_map.json extracted_features/i0001/i0001_llm_description.json extracted_features/i0001/i0001_ocr.txt extracted_features/i0001/i0001_summary.json extracted_features/i0001/i0001_summary.txt extracted_features/i0001/i0001_visual_semantics.json extracted_features/i0002/i0002_attention_heatmap.npz extracted_features/i0002/i0002_attention_map.json extracted_features/i0002/i0002_llm_description.json extracted_features/i0002/i0002_ocr.txt extracted_features/i0002/i0002_summary.json extracted_features/i0002/i0002_summary.txt extracted_features/i0002/i0002_visual_semantics.json extracted_features/i0003/i0003_attention_heatmap.npz extracted_features/i0003/i0003_attention_map.json extracted_features/i0003/i0003_llm_description.json extracted_features/i0003/i0003_ocr.txt extracted_features/i0003/i0003_summary.json extracted_features/i0003/i0003_summary.txt extracted_features/i0003/i0003_visual_semantics.json extracted_features/i0004/i0004_attention_heatmap.npz extracted_features/i0004/i0004_attention_map.json extracted_features/i0004/i0004_llm_description.json extracted_features/i0004/i0004_ocr.txt extracted_features/i0004/i0004_summary.json extracted_features/i0004/i0004_summary.txt extracted_features/i0004/i0004_visual_semantics.json extracted_features/i0005/i0005_attention_heatmap.npz extracted_features/i0005/i0005_attention_map.json extracted_features/i0005/i0005_llm_description.json extracted_features/i0005/i0005_ocr.txt extracted_features/i0005/i0005_summary.json extracted_features/i0005/i0005_summary.txt extracted_features/i0005/i0005_visual_semantics.json extracted_features/i0006/i0006_attention_heatmap.npz extracted_features/i0006/i0006_attention_map.json extracted_features/i0006/i0006_llm_description.json extracted_features/i0006/i0006_ocr.txt extracted_features/i0006/i0006_summary.json extracted_features/i0006/i0006_summary.txt extracted_features/i0006/i0006_visual_semantics.json extracted_features/i0007/i0007_attention_heatmap.npz extracted_features/i0007/i0007_attention_map.json extracted_features/i0007/i0007_llm_description.json extracted_features/i0007/i0007_ocr.txt extracted_features/i0007/i0007_summary.json extracted_features/i0007/i0007_summary.txt extracted_features/i0007/i0007_visual_semantics.json extracted_features/i0008/i0008_attention_heatmap.npz extracted_features/i0008/i0008_attention_map.json extracted_features/i0008/i0008_llm_description.json extracted_features/i0008/i0008_ocr.txt extracted_features/i0008/i0008_summary.json extracted_features/i0008/i0008_summary.txt extracted_features/i0008/i0008_visual_semantics.json extracted_features/i0009/i0009_attention_heatmap.npz extracted_features/i0009/i0009_attention_map.json extracted_features/i0009/i0009_llm_description.json extracted_features/i0009/i0009_ocr.txt extracted_features/i0009/i0009_summary.json extracted_features/i0009/i0009_summary.txt extracted_features/i0009/i0009_visual_semantics.json extracted_features/i0010/i0010_attention_heatmap.npz extracted_features/i0010/i0010_attention_map.json extracted_features/i0010/i0010_llm_description.json extracted_features/i0010/i0010_ocr.txt extracted_features/i0010/i0010_summary.json extracted_features/i0010/i0010_summary.txt extracted_features/i0010/i0010_visual_semantics.json extracted_features/i0011/i0011_attention_heatmap.npz extracted_features/i0011/i0011_attention_map.json extracted_features/i0011/i0011_llm_description.json extracted_features/i0011/i0011_ocr.txt extracted_features/i0011/i0011_summary.json extracted_features/i0011/i0011_summary.txt extracted_features/i0011/i0011_visual_semantics.json extracted_features/i0012/i0012_attention_heatmap.npz extracted_features/i0012/i0012_attention_map.json extracted_features/i0012/i0012_llm_description.json extracted_features/i0012/i0012_ocr.txt extracted_features/i0012/i0012_summary.json extracted_features/i0012/i0012_summary.txt extracted_features/i0012/i0012_visual_semantics.json extracted_features/i0013/i0013_attention_heatmap.npz extracted_features/i0013/i0013_attention_map.json extracted_features/i0013/i0013_llm_description.json extracted_features/i0013/i0013_ocr.txt extracted_features/i0013/i0013_summary.json extracted_features/i0013/i0013_summary.txt extracted_features/i0013/i0013_visual_semantics.json extracted_features/i0014/i0014_attention_heatmap.npz extracted_features/i0014/i0014_attention_map.json extracted_features/i0014/i0014_llm_description.json extracted_features/i0014/i0014_ocr.txt extracted_features/i0014/i0014_summary.json extracted_features/i0014/i0014_summary.txt extracted_features/i0014/i0014_visual_semantics.json extracted_features/i0015/i0015_attention_heatmap.npz extracted_features/i0015/i0015_attention_map.json extracted_features/i0015/i0015_llm_description.json extracted_features/i0015/i0015_ocr.txt extracted_features/i0015/i0015_summary.json extracted_features/i0015/i0015_summary.txt extracted_features/i0015/i0015_visual_semantics.json extracted_features/i0016/i0016_attention_heatmap.npz extracted_features/i0016/i0016_attention_map.json extracted_features/i0016/i0016_llm_description.json extracted_features/i0016/i0016_ocr.txt extracted_features/i0016/i0016_summary.json extracted_features/i0016/i0016_summary.txt extracted_features/i0016/i0016_visual_semantics.json extracted_features/i0017/i0017_attention_heatmap.npz [50 more files omitted for size] ``` ### Dependencies - frontend/package.json: @eslint/js@^9.36.0, @types/react@^19.1.16, @types/react-dom@^19.1.9, @vitejs/plugin-react@^5.0.4, axios@^1.12.2, eslint@^9.36.0, eslint-plugin-react-hooks@^5.2.0, eslint-plugin-react-refresh@^0.4.22, globals@^16.4.0, react@^19.1.1, react-dom@^19.1.1, vite@^7.1.7 - requirements.txt: easyocr@>=1.7.1, faiss-cpu@>=1.7.4, matplotlib@>=3.7.0, numpy@>=1.24.0, open-clip-torch@>=2.24.0, opencv-python@>=4.8.0, Pillow@>=10.0.0, pytesseract@>=0.3.10, pytest@>=8.0.0, torch@>=2.2.0, torchvision@>=0.17.0, tqdm@>=4.66.0, ultralytics@>=8.0.0 ### Recent commits (newest first) - Update README - simple working frontend demo - Ignore generated extraction outputs - commit before merge - Merge branch 'image' of https://github.com/qedsphere/ad-intelligence-challenge into image - finished extractor structure, implemented readmes for visual sematic and extractor - Remove tppgaze submodule and local artifacts - modified README - updated README - Merge branch 'image' of https://github.com/qedsphere/ad-intelligence-challenge into image - updated README - added text detection with ocr - seeding to derandomize - Wire up tppgaze submodule to phuselab/tppgaze (main) - Clean broken tppgaze entry - implemented TPP-gaze model to mimic user attention hotspots - finished visual semantics feature using faiss and clip - Working visual sematics feature with wordbank and borders issue - implemented image standardization to prepare for specific feature files - created files ## Key source files (fetched from GitHub, selected and truncated for size) ### Instructions.md ```markdown # Ad Intelligence Challenge ## Overview At the core of AppLovin's **Axon technology** is a real-time recommendation system that processes thousands of features about users, advertisers, and the ads they see — all to predict which creative, at this very moment, will drive the best result. This challenge is about diving into that foundation: **how can we extract smarter, more meaningful signals from ad creatives themselves?** --- ## Your Goal Create a model or prototype that processes ad creatives (images or videos) and extracts novel, high-value features or insights that could feed into a recommendation engine. ### Examples may include: - Tone or emotional sentiment - Type of product or content being advertised - Text, logo, or object recognition - Audio or visual embeddings - Any other creative or contextual signals that could inform ad performance prediction --- ## What We're Looking For ### 1. Signal Extraction Insight - Ability to extract diverse high-value signals from ad creatives - Signals should be distinct and minimally overlapping - Clear justification for why these signals could improve a recommendation or ranking model ### 2. Performance - Processing should be fast (< 5 minutes) and parallelizable - Clear architecture and rational behind design decisions ### 3. Robustness - Results should be repeatable and consistent - Should work on a wide variety of ad types - product ads, app ads, video creatives, and static images ### 4. Creativity - Demonstrate outside-the-box thinking in what features to extract and how to interpret them --- ## Judging Criteria (weighted) | Criteria | Weight | |----------|--------| | Signal Extraction Insight | 25% | | Performance | 25% | | Robustness | 25% | | Creativity | 25% | --- ## Details In this folder, `ads.zip` contains both sample image and video advertisements from advertisers across multiple industries, including automotive, fashion, consumer products, insurance, and more. **Important notes:** - Ads are either a `.png` or a `.mp4` - ⚠️ They do **not** have consistent sizing, frame rates, or resolutions - Some have clear specific products they promote, whereas others are traditional brand ads promoting the company in general - Each ad is given an ID, for example, `i0001.png` You should use these ads to test your feature extraction pipeline. --- ## What Makes a Good Feature? ### Distinctiveness If features are highly correlated, they add little new information. ### Predictive Power Even though this challenge doesn't include performance labels, imagine how your feature would correlate with engagement, clickthrough, or purchases. **Examples:** - The presence of a CTA button often drives higher engagement - Too little action at the beginning of an ad can cause a user to lose interest ### Scalability The feature should be fast and consistent to compute across millions of ads. - ✅ **Good:** "Average motion intensity per frame" can be batch processed - ❌ **Bad:** "Manually annot [truncated — 794 more characters] ``` ### image/AD_Extraction_README.md ```markdown # 🧠 Ad Intelligence — Image Feature Extraction Runner This module (`image/extraction.py`) runs the full **Ad Intelligence** batch extraction pipeline. It standardizes images, extracts visual and attention-based features, and produces human-readable artifacts for each image. --- ## 🚀 Overview The extraction process has **two main stages**, executed in parallel: 1. **Stage A – Standardization:** Loads each input image, resizes/pads it to 1024×1024 RGB float32 [0–1], and writes standardized versions. 2. **Stage B – Feature Extraction:** For each standardized image, runs a feature extractor suite: - **Visual Semantics:** YOLO (detections), CLIP (embeddings + scene labels), OCR, and product category. - **Attention Map:** TPPGaze simulated gaze heatmaps, peaks, and entropy. - **LLM Description (optional):** lightweight textual summary (placeholder). The system is **deadline- and timeout-aware**, allowing a global runtime budget and per-image timeouts. --- ## 🧩 CLI Usage ```bash python -m image.extraction --images_dir ads/images --out_dir extracted_features --size 1024 1024 --std_workers 6 --ext_workers auto --deadline_seconds 300 --per_image_timeout 120 [--force] ``` ### Parameters | Flag | Description | |------|--------------| | `--images_dir` | Folder containing input ad images. | | `--out_dir` | Where to save standardized images and extracted features. | | `--size` | Target width and height (default 1024×1024). | | `--std_workers` | Parallel workers for image standardization. | | `--ext_workers` | Parallel workers for feature extraction (`auto` = detect). | | `--deadline_seconds` | Global time budget for entire batch. | | `--per_image_timeout` | Optional timeout per image (0 = disabled). | | `--force` | Re-run even if outputs already exist. | --- ## ⚙️ How It Works ### **1. Stage A — Standardization** Each image is: - Opened and converted to RGB. - Resized and padded to maintain aspect ratio. - Saved as `_standardized.png`. - Metadata (size, bytes, format, etc.) recorded in a `Meta` object. Returns: A payload including the standardized PIL image, NumPy array, and metadata. ### **2. Stage B — Extraction** The standardized image flows immediately into extraction: #### a. Visual Semantics ```python vs_out = VS.extract(std_arr, image_id) ``` Extracts: - **YOLO detections** (`detections` list) - **CLIP scene labels** (`scene_labels`) - **Product category** (via CLIP or FAISS) - **OCR text** (`ocr`) Artifacts: ``` *_visual_semantics.json *_detections.png *_ocr.txt ``` #### b. Attention Map (TPPGaze) ```python am_out = AM.extract(std_arr, image_id) ``` Generates: - Heatmap (.npz) - Overlay visualization (.png) - Attention metrics (entropy, peaks, scanpaths) Artifacts: ``` *_attention_heatmap.npz *_attention_overlay.png *_attention_map.json ``` #### c. LLM Description (optional) If available, calls `LLM.extract(std_arr, image_id)` to produce: ``` *_llm_description.json ``` [truncated — 2455 more characters] ``` ### requirements.txt ``` # Core numerics & imaging numpy>=1.24.0 Pillow>=10.0.0 opencv-python>=4.8.0 # Visualization for the demo (palette/saliency overlays) matplotlib>=3.7.0 # Testing pytest>=8.0.0 # YOLOv8 object detection (boxes only) ultralytics>=8.0.0 # CLIP embeddings (open_clip) open-clip-torch>=2.24.0 # PyTorch (needed for CLIP; enables CUDA or Apple Silicon MPS if available) # Tip (Apple Silicon): these wheels include MPS support. Verify with: # python -c "import torch; print(hasattr(torch.backends,'mps') and torch.backends.mps.is_available())" torch>=2.2.0 torchvision>=0.17.0 # torchaudio is optional if you later add audio features # torchaudio>=2.2.0 # Optional: FAISS for open-vocabulary text label retrieval # NOTE: On macOS arm64 (Apple Silicon), pip wheels for faiss-cpu may be unavailable. # If install fails on M-series Macs, install via conda-forge: # conda install -c conda-forge faiss-cpu # or comment this line out; the code will gracefully fall back to the mini label bank. faiss-cpu>=1.7.4 # If you want to be cautious on macOS arm64, you can instead use: # faiss-cpu>=1.7.4; platform_system!="Darwin" or platform_machine=="x86_64" # Optional (handy for building a large text index offline) tqdm>=4.66.0 easyocr>=1.7.1 pytesseract>=0.3.10 # optional fallback if you install tesseract ``` ### frontend/package.json ``` { "name": "frontend", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "vite build", "lint": "eslint .", "preview": "vite preview" }, "dependencies": { "axios": "^1.12.2", "react": "^19.1.1", "react-dom": "^19.1.1" }, "devDependencies": { "@eslint/js": "^9.36.0", "@types/react": "^19.1.16", "@types/react-dom": "^19.1.9", "@vitejs/plugin-react": "^5.0.4", "eslint": "^9.36.0", "eslint-plugin-react-hooks": "^5.2.0", "eslint-plugin-react-refresh": "^0.4.22", "globals": "^16.4.0", "vite": "^7.1.7" } } ``` ### backend/main.py ```python from fastapi import FastAPI, UploadFile, File from fastapi.responses import FileResponse, JSONResponse from fastapi.middleware.cors import CORSMiddleware from pathlib import Path import tempfile, shutil, zipfile, time from image.extraction import Runner app = FastAPI(title="Ad Feature Extraction API") app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_methods=["*"], allow_headers=["*"], ) @app.get("/") def root(): return {"message": "Ad Intelligence Feature Extractor API"} @app.post("/extract") async def extract(files: list[UploadFile] = File(...)): start = time.time() tmp_in = tempfile.mkdtemp() tmp_out = tempfile.mkdtemp() # save uploaded images for f in files: dest = Path(tmp_in) / f.filename with open(dest, "wb") as out: shutil.copyfileobj(f.file, out) # run extraction runner = Runner( images_dir=tmp_in, out_dir=tmp_out, target_size=(1024, 1024), std_workers=4, ext_workers=0, # auto force=True, deadline_seconds=300, per_image_timeout=120, ) summaries = runner.run() # zip results zip_path = Path(tmp_out) / "extracted_features.zip" with zipfile.ZipFile(zip_path, "w", zipfile.ZIP_DEFLATED) as zipf: for f in Path(tmp_out).rglob("*"): zipf.write(f, f.relative_to(tmp_out)) elapsed = time.time() - start print(f"[API] Completed extraction of {len(summaries)} images in {elapsed:.1f}s") return FileResponse( path=zip_path, media_type="application/zip", filename="extracted_features.zip", ) ``` ### frontend/src/main.jsx ```javascript import { StrictMode } from 'react' import { createRoot } from 'react-dom/client' import './index.css' import App from './App.jsx' createRoot(document.getElementById('root')).render( , ) ``` ### frontend/src/App.jsx ```javascript import { useState } from "react"; import axios from "axios"; export default function App() { const [files, setFiles] = useState([]); const [downloading, setDownloading] = useState(false); const [downloadUrl, setDownloadUrl] = useState(null); const handleUpload = (e) => setFiles(Array.from(e.target.files)); const handleSubmit = async () => { if (!files.length) return; setDownloading(true); setDownloadUrl(null); const formData = new FormData(); files.forEach((f) => formData.append("files", f)); try { const res = await axios.post("http://localhost:8000/extract", formData, { responseType: "blob", onUploadProgress: (p) => console.log(`Upload ${Math.round((p.loaded / p.total) * 100)}%`), }); const blob = new Blob([res.data], { type: "application/zip" }); const url = URL.createObjectURL(blob); setDownloadUrl(url); } catch (err) { alert("Extraction failed: " + err); } finally { setDownloading(false); } }; return (

🧠 Ad Feature Extractor


{downloadUrl && (
⬇️ Download Extracted Features
)}
); } ``` ### setup.sh ```shell #!/bin/bash set -e # Create virtual environment if it doesn't exist if [ ! -d ".venv" ]; then echo -n "Creating virtual environment... " python3 -m venv .venv echo "DONE!" fi # Activate virtual environment source .venv/bin/activate echo -n "Verifying Python dependencies... " pip install -r requirements.txt echo "DONE!" if [ ! -d "ads" ]; then echo -n "Setting up ad dataset... " pip install -q gdown FILE_ID="1DDj8l59RyEoE2zeeYAqfg9wHiiZehUfg" OUTPUT_FILE="ads.zip" gdown "https://drive.google.com/uc?id=${FILE_ID}" -O "${OUTPUT_FILE}" unzip -q -o "${OUTPUT_FILE}" -d . rm -rf __MACOSX rm -rf "${OUTPUT_FILE}" echo "DONE!" fi echo "" echo "Setup complete!" ``` ### data/tppgaze_config.yaml ```yaml model_type: transformer input_size: [1024, 1024] seed: 123 ``` ### frontend/vite.config.js ```javascript import { defineConfig } from 'vite' import react from '@vitejs/plugin-react' // https://vite.dev/config/ export default defineConfig({ plugins: [react()], }) ``` [14 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]