# Project export: ADLOVIN

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 12.0
- Tagline: An AI prototype that processes ad creatives, both images and videos, to extract rich visual, textual, and audio insights that reveal brand intent and power next-generation recommendation engines.
- Devpost: https://devpost.com/software/adlovin-media-intelligence
- GitHub: https://github.com/SisuKah/calhacks2
- Demo: https://github.com/abtonmoy/calhacks12
- Video: https://www.youtube.com/embed/yLyytqlP2Ik?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (AppLovin: Ad Intelligence Challenge)
- Team: 1 GitHub contributor(s) — Harshit (1 commits)

## Devpost submission (written by the team)

### Inspiration

The inspiration came from real-world ad recommendation systems like AppLovin’s Axon, which rely on large datasets of ad creatives to predict performance. We wanted to explore how far we could push multimodal intelligence by building a system that understands ad content visually, textually, and acoustically. Our goal was to make ad data more insightful and useful for smarter recommendation models.

### What it does

ADLOVIN Media Intelligence processes ad creatives, both images and videos, and extracts high-value multimodal features such as visual composition, text sentiment, and audio mood. These embeddings can then be used to generate creative performance insights or power downstream machine learning models for recommendation and ranking.

### How we built it

We built a modular feature extraction pipeline in Python. For video ads, we used FFmpeg to extract keyframes and Librosa for audio signal analysis. Each frame was embedded using a pretrained vision transformer, and text elements were captured through EasyOCR and language models. We used ChromaDB to store and query embeddings at scale, and FastAPI to provide an interface for experimentation and testing.

### Challenges we ran into

One major challenge was optimizing feature extraction for large videos while maintaining temporal coherence between visual and audio signals. Another was ensuring meaningful clustering in the vector database without redundant representations, especially after removing the deduplication algorithm. Balancing efficiency and fidelity across multiple modalities required careful design and parameter tuning. One of the major issues faced was the wifi. It made our life really hard. We had to find place to work outside of the venue.

### Accomplishments we're proud of

We successfully built an end-to-end multimodal intelligence pipeline that can process a wide range of ad creatives and output interpretable embeddings. The system achieved reliable text and sentiment extraction from frames and robust feature representation using transformer-based models.

### What we learned

We learned how to align multimodal embeddings for recommendation tasks, fine-tune vector similarity searches for creative clustering, and manage large-scale ad processing efficiently. We also gained experience combining computer vision, audio analysis, and natural language understanding into one cohesive system.

### What's next

Media Intelligence Next, we plan to add a lightweight deduplication model, connect the embeddings to a performance dataset for supervised learning, and build an interactive dashboard to visualize ad intelligence insights in real time. We also aim to explore generative tools that can use embeddings to suggest creative improvements.

## README (from the GitHub repository)

# Ad Lovin Frontend

A React TypeScript frontend for testing the Flask API server.

## 🚀 Quick Start

### 1. Install Dependencies

```bash
npm install
```

### 2. Start the Development Server

```bash
npm run dev
```

The frontend will start on `http://localhost:3000`

### 3. Make Sure Your Flask Backend is Running

In a separate terminal, start your Flask backend:

```bash
cd ../your-backend-directory
python3 api_server.py
```

The backend should be running on `http://localhost:5001`

## 📋 Available Scripts

- `npm run dev` - Start development server
- `npm run build` - Build for production
- `npm run preview` - Preview production build

## 🔗 API Connection

The frontend is configured to connect to your Flask API at `http://localhost:5001/api`.

If your API is running on a different port or host, update the `API_BASE_URL` in `src/App.tsx`:

```typescript
const API_BASE_URL = "http://localhost:5001/api";
```

## ✨ Features

- **Human Detection**: Upload an image and detect if humans are present
- **Audio Analysis**: Upload a video and analyze the audio features including:
  - Duration
  - Gender estimation
  - Mean pitch
  - Spectral bandwidth
  - Emotion detection

## 🛠️ Technology Stack

- **React 18** - UI framework
- **TypeScript** - Type safety
- **Vite** - Fast build tool and dev server
- **Modern CSS** - Responsive styling

## 📝 Project Structure

```
.
├── src/
│   ├── App.tsx        # Main React component
│   ├── main.tsx       # React entry point
│   └── index.css      # Global styles
├── index.html         # HTML template
├── vite.config.ts     # Vite configuration
├── tsconfig.json      # TypeScript configuration
└── package.json       # Dependencies
```

## 🐛 Troubleshooting

### Port Already in Use

If port 3000 is already in use, Vite will automatically use the next available port (3001, 3002, etc.)

### API Connection Issues

- Make sure your Flask backend is running
- Check the API_BASE_URL in `src/App.tsx` matches your backend URL
- Check browser console for CORS or connection errors

### CORS Errors

If you see CORS errors in the browser console, make sure your Flask backend has CORS enabled for `localhost:3000`.



## Detected evidence (automated analysis)

Indexed codebase: 7 recognized source files, 26 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Hugging Face (technology) — claimed on Devpost, not found in the code
- Python (language) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (12 of 12)

```
.gitignore
index.html
package.json
postcss.config.mjs
README.md
src/App.tsx
src/index.css
src/main.tsx
tailwind.config.js
tsconfig.json
tsconfig.node.json
vite.config.ts
```

### Dependencies

- package.json: @tailwindcss/postcss@^4.1.16, @types/react@^18.2.0, @types/react-dom@^18.2.0, @vitejs/plugin-react@^4.2.0, autoprefixer@^10.4.20, postcss@^8.4.47, react@^18.2.0, react-dom@^18.2.0, tailwindcss@^4.1.16, typescript@^5.2.0, vite@^5.0.0

### Recent commits (newest first)

- added adult check
- Initial commit: Ad Lovin frontend with complete image analysis display

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "ad-lovin-frontend",
  "private": true,
  "version": "0.1.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "tsc && vite build",
    "preview": "vite preview"
  },
  "dependencies": {
    "@tailwindcss/postcss": "^4.1.16",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "tailwindcss": "^4.1.16"
  },
  "devDependencies": {
    "@types/react": "^18.2.0",
    "@types/react-dom": "^18.2.0",
    "@vitejs/plugin-react": "^4.2.0",
    "autoprefixer": "^10.4.20",
    "postcss": "^8.4.47",
    "typescript": "^5.2.0",
    "vite": "^5.0.0"
  }
}

```

### src/main.tsx

```typescript
import React from "react";
import ReactDOM from "react-dom/client";
import App from "./App";
import "./index.css";

ReactDOM.createRoot(document.getElementById("root")!).render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);


```

### src/App.tsx

```typescript
"use client";

import { ChangeEvent, useEffect, useMemo, useState } from "react";

type MediaType = "image" | "video" | "other";

type AnalysisRow = {
  label: string;
  value: string;
};

type AnalysisSection = {
  title: string;
  description: string;
  rows: AnalysisRow[];
};

interface ImageAnalysisResult {
  // Basic Info
  image_path: string;
  image_name: string;
  timestamp: string;

  // Human Detection
  human_present: number;
  num_people: number;

  // OCR
  extracted_text: string;

  // Marketing Signals
  brand_name_text: string;
  product_name: string;
  industry: string;
  promo_present: boolean;
  promo_text: string;
  promo_deadline: string;
  price_value: string;
  cta_present: boolean;
  cta_type: string;
  text_density: string;
  brand_text_contrast: string;

  // Content Safety
  is_nsfw: boolean;

  error?: string;
}

interface AudioAnalysisResult {
  duration_sec: number;
  gender_estimation: string;
  mean_pitch: number;
  spectral_bandwidth: number;
  emotion: string;
  error?: string;
}

interface VideoAnalysisResult {
  success: boolean;
  file_info?: {
    file_path: string;
    file_name: string;
    file_type: string;
    processed_at: string;
    processing_time_seconds: number;
  };
  analysis?: {
    frames_analyzed: number;
    frames_with_brands: number;
    frames_with_promos: number;
    frames_with_promo_codes: number;
    frames_with_humans: number;
  };
  consolidated_video_analysis?: {
    brand_name_text?: string;
    product_name?: string;
    industry?: string;
    promo_present?: boolean;
    promo_text?: string;
    promo_code?: string;
    promo_deadline?: string;
    discount_type?: string;
    price_value?: string;
    cta_present?: boolean;
    cta_type?: string;
    cta_text?: string;
    human_present?: boolean;
    emotion?: string;
  };
  error?: string;
}

const API_BASE_URL = "/api";

const formatFileSize = (sizeInBytes: number): string => {
  if (!Number.isFinite(sizeInBytes) || sizeInBytes < 0) {
    return "—";
  }

  if (sizeInBytes === 0) {
    return "0 B";
  }

  const units = ["B", "KB", "MB", "GB"];
  const exponent = Math.min(
    Math.floor(Math.log(sizeInBytes) / Math.log(1024)),
    units.length - 1
  );
  const size = sizeInBytes / Math.pow(1024, exponent);

  return `${size.toFixed(exponent === 0 ? 0 : 1)} ${units[exponent]}`;
};

const detectMediaType = (file: File | null): MediaType => {
  if (!file) {
    return "other";
  }

  const mimeType = file.type.toLowerCase();

  if (mimeType.startsWith("image/")) {
    return "image";
  }

  if (mimeType.startsWith("video/")) {
    return "video";
  }

  return "other";
};

const buildAnalysisFromResult = (
  mediaType: MediaType,
  file: File | null,
  imageResult?: ImageAnalysisResult | null,
  audioResult?: AudioAnalysisResult | null,
  videoResult?: VideoAnalysisResult | null
): AnalysisSection[] => {
  const sections: AnalysisSection[] = [
    {
      title: "Media Overview",
      description: "Metadata detected by the analysis pipeline.",
      rows: [
        {
          label: "Detected Media Type",
          value:
            mediaType === "image"
              ? "Image"
              : mediaType === "video"
              ? "Video"
              : "Unknown Media",
        },
        {
          label: "File Name",
          value: file?.name ?? "—",
        },
        {
          label: "File Size",
          value: file ? formatFileSize(file.size) : "—",
        },
      ],
    },
  ];

  // Handle image analysis
  if (mediaType === "image" && imageResult) {
    if (imageResult.error) {
      sections.push({
        title: "Error",
        description: "An error occurred during analysis.",
        rows: [{ label: "Error Message", value: imageResult.error }],
      });
    } else {
      // Content Safety Section
      sections.push({
        title: "Content Safety",
        description: "Content moderation and safety assessment.",
        rows: [
          {
            label: "Adult Content",
            value: imageResult.is_nsfw ? "Yes ⚠️" : "No ✓",
          },
        ],
      });

      // Human Detection Section
      sections.push({
        title: "Human Detection",
        description: "Human detection analysis results.",
        rows: [
          {
            label: "Humans Detected",
            value: imageResult.human_present === 1 ? "Yes" : "No",
          },
          {
            label: "Number of People",
            value: imageResult.num_people.toString(),
          },
        ],
      });

      // OCR Results Section
      if (imageResult.extracted_text) {
        sections.push({
          title: "Text Extraction (OCR)",
          description: "All text extracted from the image.",
          rows: [
            {
              label: "Extracted Text",
              value: imageResult.extracted_text || "None",
            },
          ],
        });
      }

      // Marketing Signals Section
      const marketingRows: AnalysisRow[] = [];

      // Always show core marketing fields
      marketingRows.push({
        label: "Brand Name",
        value: imageResult.brand_name_text || "Not detected",
      });

      marketingRows.push({
        label: "Product Name",
        value: imageResult.product_name || "Not detected",
      });

      marketingRows.push({
        label: "Industry",
        value: imageResult.industry || "Not detected",
      });

      // Always show promo_present status
      marketingRows.push({
        label: "Promo Present",
        value: imageResult.promo_present ? "Yes" : "No",
      });

      // Show additional promo details if promo is present
      if (imageResult.promo_present && imageResult.promo_text) {
        marketingRows.push({
          label: "Promo Text",
          value: imageResult.promo_text,
        });
      }

      if (imageResult.promo_present && imageResult.promo_deadline) {
        marketingRows.push({
          label: "Promo Deadline",
          value: imageResult.promo_deadline,
     
[truncated — 16607 more characters]
```

### tailwind.config.js

```javascript
export default {
  content: ["./index.html", "./src/**/*.{js,ts,jsx,tsx}"],
};

```

### vite.config.ts

```typescript
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [react()],
  server: {
    port: 3000,
    proxy: {
      "/api": {
        target: "http://127.0.0.1:5001",
        changeOrigin: true,
      },
    },
  },
});

```

### index.html

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Ad Lovin Media Intelligence</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/main.tsx"></script>
  </body>
</html>

```

### src/index.css

```css
@import "tailwindcss";

:root {
  --background: #ffffff;
  --foreground: #171717;
}

@media (prefers-color-scheme: dark) {
  :root {
    --background: #0a0a0a;
    --foreground: #ededed;
  }
}

body {
  background: var(--background);
  color: var(--foreground);
  font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif;
}

```