# Project export: ClipStudy.ai This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: A video search engine to learn fast through videos. - Devpost: https://devpost.com/software/clipstudy-ai - GitHub: https://github.com/NP2241/ClipStudy - Video: https://www.youtube.com/embed/LmWCRNTCSKo?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — mc08 (24 commits), Neil Pendyala (18 commits), EddieTsai (4 commits), aliciaganzeng (4 commits) ## Devpost submission (written by the team) ### Inspiration Many students struggle to efficiently review specific concepts in long lecture recordings. Even with YouTube’s auto-generated chapter labels, users still have to manually scroll through these sections, and these labels often miss hyperspecific or nuanced topics, making it hard to quickly access exactly what’s needed. ### What it does ClipStudy allows users to paste in a YouTube video link and use natural language to ask for the specific concept they want to review. Our backend then analyzes the video transcript, finds the relevant timestamps, and automatically jumps to that point in our embedded video. ### How we built it We built ClipStudy.ai by developing a local script to fetch and clean YouTube video transcripts using yt-dlp and custom SRT parsing logic. Once the transcript is processed, we leverage Claude 4 Sonnet to analyze the transcript and identify segments relevant to a user’s query, using a prompt-based approach for granular concept detection. The backend coordinates transcript extraction and segment analysis, while the frontend presents an interface for users to submit links and review the identified video segments. All intermediate data (transcripts and segment metadata) are managed locally for efficiency and privacy. ### Challenges we ran into Curating and cleaning diverse YouTube transcripts with varying quality and formats Fine-tuning prompt engineering with Claude 4 Sonnet to accurately detect nuanced educational concepts Balancing fast response times with thorough transcript analysis for a smooth user experience ### Accomplishments we're proud of Successfully automated the end-to-end process from transcript extraction to concept-based video navigation Developed a robust transcript cleaning pipeline that handles a wide range of YouTube lecture formats Achieved accurate, AI-powered detection of nuanced concepts and seamless video segment jumping Built a privacy-focused system that processes and stores all user data locally ### What we learned The power of large language models for granular concept detection in long-form educational videos The value of prompt engineering for improving AI accuracy and relevance ### What's next Expand support to additional video platforms beyond YouTube, such as Vimeo, Zoom recordings, and educational portals. Integrate with popular notetaking and productivity apps such as Notion. Add support for YouTube playlists and video-searching inside the system (backend code exists). ## README (from the GitHub repository) # ClipStudy ClipStudy.ai lets you paste a YouTube lecture link and instantly jump to any concept you search for- no more endless scrolling, just focused, AI-powered video review. ## Project Structure - `backend/` - Backend services and API - `frontend/` - Frontend application - `docs/` - Documentation ## Getting Started - Run backend `cd backend & flask run -p 3001` - Run frontend `cd frontend & pnpm build & pnpm start` - Go to [localhost:3000](localhost:3000) ### For docker deployment - prepare for the Anthropic api key - running following script on root ``` docker compose up --build -d ``` ## Detected evidence (automated analysis) Indexed codebase: 14 recognized source files, 293 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (42 of 42) ``` .gitignore backend/.DS_Store backend/.gitkeep backend/app.py backend/Dockerfile backend/test/analyze_playlist.py backend/transcript_extraction/decide_clip.py backend/transcript_extraction/temporary_files/playlist_analysis.json backend/transcript_extraction/temporary_files/raw_transcript_rfG8ce4nNh0.txt backend/transcript_extraction/temporary_files/transcript_AJxYCosjRH0_density_segments.json backend/transcript_extraction/temporary_files/transcript_AJxYCosjRH0_earth_segments.json backend/transcript_extraction/temporary_files/transcript_fTsN4C9Utwc_central_dogma_of_bio_segments.json backend/transcript_extraction/temporary_files/transcript_fTsN4C9Utwc_Dogma_central_de_bio_segments.json backend/transcript_extraction/temporary_files/transcript_heBErnN3ZPk_employment_segments.json backend/transcript_extraction/temporary_files/transcript_heBErnN3ZPk.txt backend/transcript_extraction/temporary_files/transcript_KLh-rj6CrnE_identification_segments.json backend/transcript_extraction/temporary_files/transcript_qYNweeDHiyU_ai_segments.json backend/transcript_extraction/temporary_files/transcript_qYNweeDHiyU_deep_learning_segments.json backend/transcript_extraction/temporary_files/transcript_qYNweeDHiyU_gen_ai_segments.json backend/transcript_extraction/temporary_files/transcript_qYNweeDHiyU_generative_ai_segments.json backend/transcript_extraction/temporary_files/transcript_qYNweeDHiyU_IGNORE_ALL_PREVIOUS__segments.json backend/transcript_extraction/temporary_files/transcript_rfG8ce4nNh0_area_under_the_curve_segments.json backend/transcript_extraction/temporary_files/transcript_rfG8ce4nNh0_Matrix_Multiplicatio_segments.json backend/transcript_extraction/temporary_files/transcript_rfG8ce4nNh0_test_segments.json backend/transcript_extraction/temporary_files/transcript_WUvTyaaNkzM_matrix_multiplicatio_segments.json backend/transcript_extraction/temporary_files/transcript_xglEjH0Ue8o_vocab_segments.json backend/transcript_extraction/transcript_fetch.py deploy.sh docker-compose.yaml docs/ARCHITECTURE.md frontend/.gitignore frontend/Dockerfile frontend/eslint.config.mjs frontend/next.config.ts frontend/package.json frontend/postcss.config.mjs frontend/README.md frontend/src/app/globals.css frontend/src/app/layout.tsx frontend/src/app/page.jsx frontend/tsconfig.json README.md ``` ### Dependencies - frontend/package.json: @eslint/eslintrc@^3, @radix-ui/react-icons@^1.3.2, @radix-ui/themes@^3.2.1, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@15.3.4, next@15.3.4, react@^19.0.0, react-dom@^19.0.0, react-resizable-panels@^3.0.3, react-youtube@^10.1.0, tailwindcss@^4, typescript@^5, usehooks-ts@^3.1.1 ### Recent commits (newest first) - timestamp lax - Merge branch 'main' of https://github.com/NP2241/ClipStudy - fix - modify README.md for docker - Merge branch 'main' of https://github.com/NP2241/ClipStudy - some deploy script changes - code freeze pt 2 - Update playlist analysis and transcript files - refactor: Clean up comments in Dockerfiles for clarity - Merge branch 'main' of https://github.com/NP2241/SmartLLMs - feat: Add Dockerfile and docker-compose for backend and frontend services - code freeze - final fix - Merge branch 'main' of https://github.com/NP2241/ClipStudy - last commit - Update analyze_playlist.py - Create analyze_playlist.py - Merge branch 'main' of https://github.com/NP2241/ClipStudy - w - Merge remote changes with deploy script ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/ARCHITECTURE.md ```markdown # SmartLLMs - Comprehensive Architecture Documentation ## Table of Contents 1. [Project Overview](#project-overview) 2. [System Architecture](#system-architecture) 3. [File Structure](#file-structure) 4. [Data Flow](#data-flow) 5. [API Endpoints](#api-endpoints) 6. [Core Components](#core-components) 7. [Dependencies](#dependencies) 8. [Configuration](#configuration) ## Project Overview SmartLLMs is a Python-based web application that extracts, cleans, and analyzes YouTube video transcripts to find relevant segments based on user queries. The system uses Anthropic's Claude 4 Sonnet for intelligent transcript analysis and provides a REST API for easy integration. ### Key Features - **YouTube Transcript Extraction**: Automatically downloads and cleans auto-generated YouTube subtitles - **Intelligent Analysis**: Uses Claude 4 Sonnet to identify relevant video segments - **REST API**: Flask-based API for easy integration with frontend applications - **Transcript Cleaning**: Removes overlaps and merges incomplete sentences for better readability ## System Architecture ``` ┌─────────────────┐ HTTP Request ┌─────────────────┐ │ Frontend/ │ ──────────────────► │ Flask API │ │ Client │ │ (app.py) │ └─────────────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ ┌─────────────────┐ │ YouTube │ ◄────────────────── │ decide_clip.py │ │ (yt-dlp) │ │ │ └─────────────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ ┌─────────────────┐ │ Anthropic │ ◄───────────────────│ Claude 4 │ │ Claude API │ │ Sonnet │ └─────────────────┘ └─────────────────┘ ``` ## File Structure ``` SmartLLMs/ ├── backend/ │ ├── app.py # Flask web server and API endpoints │ ├── __pycache__/ # Python cache files │ │ └── app.cpython-311.pyc │ ├── .DS_Store # macOS system file │ ├── .gitkeep # Git placeholder file │ └── transcript_extraction/ │ ├── transcript_fetch.py # YouTube transcript download and cleaning │ ├── decide_clip.py # Main analysis script using Claude │ └── temporary_files/ # Storage for transcripts and segments │ ├── transcript_rfG8ce4nNh0.txt │ ├── transcript_rfG8ce4nNh0_Matrix_Multiplicatio_segments.json │ └── transcript_AJxYCosjRH0_density_segments.json ├── frontend/ # Next.js React frontend application │ ├── src/ │ │ └── app/ # Next.js App Router │ │ ├── layout.tsx [truncated — 12720 more characters] ``` ### docker-compose.yaml ```yaml version: '3.8' services: backend: build: context: ./backend dockerfile: Dockerfile platform: linux/arm64 # modify this if you are not on an M1/M2 Mac ports: - "3001:3001" environment: - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY} volumes: - ./backend:/app - backend_temp:/app/transcript_extraction/temporary_files restart: unless-stopped frontend: build: context: ./frontend dockerfile: Dockerfile platform: linux/arm64 # modify this if you are not on an M1/M2 Mac ports: - "3000:3000" depends_on: - backend environment: - NEXT_PUBLIC_API_URL=http://backend:3001 restart: unless-stopped volumes: backend_temp: ``` ### frontend/Dockerfile ``` FROM node:18-alpine WORKDIR /app RUN npm install -g pnpm COPY package*.json ./ COPY pnpm-lock.yaml* ./ RUN pnpm install --frozen-lockfile COPY . . RUN pnpm build EXPOSE 3000 CMD ["pnpm", "start"] ``` ### backend/Dockerfile ``` FROM python:3.11-slim WORKDIR /app RUN apt-get update && apt-get install -y \ curl \ && rm -rf /var/lib/apt/lists/* RUN pip install --no-cache-dir yt-dlp COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . # Create temporary_files directory RUN mkdir -p transcript_extraction/temporary_files EXPOSE 3001 CMD ["python", "app.py"] ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@radix-ui/react-icons": "^1.3.2", "@radix-ui/themes": "^3.2.1", "next": "15.3.4", "react": "^19.0.0", "react-dom": "^19.0.0", "react-resizable-panels": "^3.0.3", "react-youtube": "^10.1.0", "usehooks-ts": "^3.1.1" }, "devDependencies": { "@eslint/eslintrc": "^3", "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "15.3.4", "tailwindcss": "^4", "typescript": "^5" } } ``` ### backend/app.py ```python from flask import Flask, jsonify, request from flask_cors import CORS import os import sys import subprocess import json from urllib.parse import unquote app = Flask(__name__) CORS(app) # Allow requests from frontend def construct_youtube_url(video_id: str) -> str: """ Construct a YouTube URL from a video ID. Args: video_id (str): The YouTube video ID Returns: str: The complete YouTube URL """ return f"https://www.youtube.com/watch?v={video_id}" @app.route("/api/hello") def hello(): return jsonify({"message": "Hello from Flask!"}) @app.route("/api/get/") def get_segments(video_id): """ Get relevant segments from a YouTube video based on a prompt. Args: video_id (str): YouTube video ID prompt (str): Search prompt for finding relevant segments (query parameter) Returns: JSON response with segments and metadata """ try: print(f"[DEBUG] Received request for video_id: {video_id}") prompt = request.args.get('prompt') if not prompt: print("[DEBUG] Missing prompt query parameter") return jsonify({ "error": "Missing 'prompt' query parameter", "usage": "Use /api/get/{video_id}?prompt=your search query" }), 400 print(f"[DEBUG] Prompt: {prompt}") youtube_url = construct_youtube_url(video_id) print(f"[DEBUG] Constructed YouTube URL: {youtube_url}") decide_clip_path = os.path.join(os.path.dirname(__file__), 'transcript_extraction', 'decide_clip.py') print(f"[DEBUG] Path to decide_clip.py: {decide_clip_path}") command = [sys.executable, decide_clip_path, youtube_url, prompt] print(f"[DEBUG] Running command: {' '.join(command)}") result = subprocess.run( command, capture_output=True, text=True, cwd=os.path.dirname(__file__) ) print(f"[DEBUG] Subprocess return code: {result.returncode}") print(f"[DEBUG] Subprocess stdout: {result.stdout}") print(f"[DEBUG] Subprocess stderr: {result.stderr}") if result.returncode != 0: return jsonify({ "error": "Failed to process video", "details": result.stderr, "stdout": result.stdout }), 500 segments_dir = os.path.join(os.path.dirname(__file__), 'transcript_extraction', 'temporary_files') prompt_safe = ''.join(c for c in prompt[:20] if c.isalnum() or c in (' ', '-', '_')).replace(' ', '_') segments_file = f"transcript_{video_id}_{prompt_safe}_segments.json" segments_path = os.path.join(segments_dir, segments_file) print(f"[DEBUG] Looking for segments file at: {segments_path}") if not os.path.exists(segments_path): print("[DEBUG] Segments file not found!") return jsonify({ "error": "Segments file not found", "expected_path": segments_path }), 404 with open(segments_path, 'r', encoding='utf-8') as f: segments_data = json.load(f) print(f"[DEBUG] Successfully loaded segments data") # Read the full transcript transcript_path = os.path.join(segments_dir, f'transcript_{video_id}.txt') full_transcript = "" if os.path.exists(transcript_path): with open(transcript_path, 'r', encoding='utf-8') as f: full_transcript = f.read() print(f"[DEBUG] Successfully loaded full transcript") else: print(f"[DEBUG] Transcript file not found at: {transcript_path}") # Add transcript and other metadata to the response response_data = { "video_id": video_id, "youtube_url": youtube_url, "transcript": full_transcript, **segments_data # Include all the existing segments data } return jsonify(response_data) except Exception as e: print(f"[DEBUG] Exception occurred: {str(e)}") return jsonify({ "error": "Internal server error", "details": str(e) }), 500 @app.route("/api/info/") def get_video_info(video_id): """ Get basic information about a YouTube video. Args: video_id (str): YouTube video ID Returns: JSON response with video information """ try: youtube_url = construct_youtube_url(video_id) transcript_path = os.path.join( os.path.dirname(__file__), 'transcript_extraction', 'temporary_files', f'transcript_{video_id}.txt' ) if os.path.exists(transcript_path): return jsonify({ "video_id": video_id, "youtube_url": youtube_url, "transcript_available": True, "transcript_path": transcript_path }) else: return jsonify({ "video_id": video_id, "youtube_url": youtube_url, "transcript_available": False, "message": "Transcript not found. Use /api/get/{video_id}?prompt=your query to fetch and analyze." }) except Exception as e: print(f"[DEBUG] Exception occurred: {str(e)}") return jsonify({ "error": "Internal server error", "details": str(e) }), 500 if __name__ == "__main__": app.run(debug=True, port=3001) ``` ### frontend/src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "./globals.css"; import "@radix-ui/themes/styles.css"; import { Theme } from "@radix-ui/themes"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "ClipStudy.ai", description: "Study efficiently", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} {/* */} ); } ``` ### frontend/src/app/page.jsx ```javascript 'use client'; import { ChevronLeftIcon, ChevronRightIcon, MagnifyingGlassIcon, PlusIcon, ViewVerticalIcon } from "@radix-ui/react-icons"; import { Box, Button, Card, Code, Dialog, Flex, Grid, Heading, Popover, ScrollArea, Separator, Strong, Text, TextField, VisuallyHidden } from "@radix-ui/themes"; import Image from "next/image"; import React, { useRef, useState } from "react"; import YouTube from "react-youtube"; import { useInterval } from "usehooks-ts"; import logo from '../../public/logo.png'; import screenshot from '../../public/screenshot.png'; function Logo() { // return Logo; return ( logo ClipStudy.ai ); } function NewVideoPopover({children, setVideo}) { const [url, setUrl] = useState(''); const [search, setSearch] = useState(''); const [loading, setLoading] = useState(false); const [open, setOpen] = useState(false); return ( setOpen(v)} > {children} New video YouTube URL setUrl(e.target.value)}> Search query setSearch(e.target.value)}> {/* */} {/* */} ); } const VIDEO_OPTIONS = { width: 640, height: 360, }; const relevanceColor = ['#000000', '#CE2C31', '#CC4E00', '#FFDC00', '#B0E64C', '#2A7E3B']; function parseTime(str) { const [hours, minutes, seconds, milliseconds] = str.split(/[,:]/); return ( parseInt(hours) * 3600 + parseInt(minutes) * 60 + parseInt(seconds) + parseInt(milliseconds) / 1000 ); } export default function Home() { const [state, setState] = useState({ sidebar: true, // newVideo: false, findVideo: false, openSegments: { }, // playing: false, }); const [video, setVideoRaw] = useState(null); const playerRef = useRef(null); // function newVideo() { // setState({...state, newVideo: true}); // } function findVideo() { setState({...state, findVideo: true}); } const [currentTime, setCurrentTime] = useState(0); const [videos, setVideos] = useState([]); function setVideo(newVideo) { setVideos([newVideo, ...videos]); setVideoRaw(newVideo); // test // console.log([newVideo, ...videos]); } useInterval(() => { // if (playerRef.current && playerRef.current.getPlayerState() !== 2) { // const currentTime = playerRef.current.getCurrentTime(); // setCurrentTime(currentTime); // let v = video.transcript_parsed.find(t => (t.starts <= currentTime && currentTime <= t.ends)); // if (v) { // // console.log(document.getElementById(`transcript${v.id}`)) // document.getElementById(`transcript${v.id}`)?.scrollIntoView(); // } // } if (playerRef.current) { const currentTime = playerRef.current.getCurrentTime(); setCurrentTime(currentTime); if (playerRef.current.getPlayerState() !== 2) { let v = video.transcript_parsed.find(t => (t.starts <= currentTime && currentTime <= t.ends)); if (v) { // console.log(document.getElementById(`transcript${v.id}`)) document.getElementById(`transcript${v.id}`)?.scrollIntoView(); } } } }, 0); const [search, setSearch] = useState(''); return ( <> {/* setState({...state, newVideo: v})}> New video */} setState({...state, findVideo: v})}> /dev/null pkill -f "pnpm.*dev" 2>/dev/null exit 0 } # Set up signal handlers trap cleanup SIGINT SIGTERM # Kill any existing processes on ports 3000 and 3001 echo "🧹 Cleaning up existing processes..." lsof -ti:3000 | xargs kill -9 2>/dev/null lsof -ti:3001 | xargs kill -9 2>/dev/null # Start backend server echo "🔧 Starting backend server (Flask) on port 3001..." cd backend python app.py & BACKEND_PID=$! cd .. # Wait a moment for backend to start sleep 3 # Check if backend is running if curl -s http://127.0.0.1:3001/api/hello > /dev/null; then echo "✅ Backend server is running!" else echo "❌ Backend server failed to start" cleanup fi # Start frontend server echo "🎨 Starting frontend server (Next.js) on port 3000..." cd frontend pnpm build pnpm start & FRONTEND_PID=$! cd .. # Wait a moment for frontend to start sleep 5 # Check if frontend is running if curl -s http://localhost:3000 > /dev/null; then echo "✅ Frontend server is running!" else echo "❌ Frontend server failed to start" cleanup fi # Display the URLs echo "" echo "🎉 SmartLLMs is now running!" echo "" echo "📱 Frontend (Main Application):" echo " http://localhost:3000" echo "" echo "🔧 Backend API Endpoints:" echo " Health Check: http://127.0.0.1:3001/api/hello" echo " Video Analysis: http://127.0.0.1:3001/api/get/{video_id}?prompt={query}" echo " Video Info: http://127.0.0.1:3001/api/info/{video_id}" echo "" echo "💡 Example API call:" echo " curl \"http://127.0.0.1:3001/api/get/rfG8ce4nNh0?prompt=matrix multiplication\"" echo "" echo "🛑 Press Ctrl+C to stop all servers" echo "" # Keep the script running wait ``` ### frontend/next.config.ts ```typescript import type { NextConfig } from "next"; const nextConfig: NextConfig = { /* config options here */ async rewrites() { return [ { source: '/api/:path*', destination: 'http://localhost:3001/api/:path*' // Proxy to Flask } ] } }; export default nextConfig; ``` [4 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]