# Project export: zoomin This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: An AI-powered control-F app for your lectures. Upload a recording, and our AI turns it into searchable concepts and an automatic highlight reel. - Devpost: https://devpost.com/software/zoomin - GitHub: https://github.com/arianna-caow/TreeHacks26 - Video: https://www.youtube.com/embed/eTJgswlV2o4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Iona Xia (16 commits), Arianna (15 commits), Clara Wang (11 commits), Erika Li (6 commits) ## Devpost submission (written by the team) ### Inspiration Millions of lectures are recorded every week on platforms like Zoom. While recording has become effortless, navigating is not. Long-form video assumes passive, linear viewing, but students don’t study linearly. They search, revisit, skim, and review. Research consistently shows that: Users prefer condensed and swimmable video formats Highlight extraction improves discoverability and engagement Structured summaries reduced cognitive load and improve retention Foundational benchmarks such as TVSum (CVPR 2015) and SumMe (ECCV 2014) formalized video summarization as a temporal importance-scoring problem. More recently, QVHighlights (NeurIPS 2021) introduced query-conditioned highlight detection, enabling video segments to be ranked based on user intent. However, much of this work focuses on visual saliency and frame-level features, which are approaches that are less aligned with concept-dense educational content. We extend this paradigm with a transcript-first, LLM-driven approach tailored specifically to recorded lectures. Instead of treating Zoom recordings as raw video streams, we treat them as structured knowledge sources. By leveraging transcription and large language models, we enable semantic search, query-driven highlight extraction, and automatic study-ready reels. ### What it does After a sports game, people watch the highlight reel, not the three hour broadcast. After lecture, students want the same: key concepts and moments that matter. zoomin is a web application that breaks down long form recordings into structured knowledge. Our workflow starts with our recording upload. zoomin analyzes the entire recording, splitting it into small windows to automatically generate highlight clips. Each clip includes a contextual learning dashboard with recommended Youtube videos and articles. Beyond highlights, our search tool lets users filter by keywords or ask questions to instantly surface relevant clips. When inspiration strikes, students can use our note taking feature to jot down any ideas, questions, and information onto an interactive post-it, stuck directly to the page. In short, zoomin converts passive Zoom recordings into an interactive, searchable, and study-ready knowledge base. How We Built It Secure Content Upload We import Zoom cloud recordings using Service-to-Service (S2S) token flow and also accept upload of MP4s. This allows users to pull recordings directly from Zoom without manual downloads, while keeping access scoped and secure. Generating Clip Highlights Non-Zoom audio is transcribed with OpenAI Whisper (ASR), and Zoom recordings use built-in transcript support. Then, we use a multi-modal importance scoring algorithm to identify the most important and useful segments of the videos. Our scoring algorithm takes in video transcripts, chat messages and volume levels, assigns importance scores to segments, and then uses smart coalescing to create highlight clips. Using ffmpeg, we extract the highlight clips and timestamps from the original video for display on the site. We apply overlap and redundancy controls to ensure highlights are distance and temporally accurate. The result is a curated set of concise, watchable clips from long recordings. Query-Conditioned Search Inspired by QVHighlights (NeurIPS 2021), we extend our framework to support search. Scanning the transcript windows, we compute query-text relevance then cut playable clips of the top candidates. We apply LLM semantic compression to produce structured summaries and descriptive titles for each highlight segment. This effectively transforms unstructured video into a searchable semantic knowledge base. Recommending Resources We envision zoomin as a comprehensive, dynamic learning companion that helps students go beyond simply watching recordings. As students, we know how difficult it can be to unpack all the concepts covered in dense lectures and find resources to reinforce or clarify understanding. In response, we incorporated a resource recommendation feature embedded inside of each highlight clip. When the user expands on a clip, the site displays a dashboard of information with insights about the key moments and topics covered. Most importantly, the related learning section underneath the clip displays curated YouTube videos as well as informational articles to give the student a clear, actionable starting point for more exploration. Notes We save notes in the browser on a per-recording basis, fit with rich formatting features. The persistent floating panel allows users to take notes seamlessly across summaries, clips and search results without interrupting the viewing experience. ### Challenges we ran into Because our app heavily relies on LLM’s to analyze long video transcripts, our biggest challenge we faced was handling rate limits by OpenAI. The video and clip summaries, clip titles, transcripts, and related resources generation features all relied on transforming our data using OpenAI APIs. As a response to severe rate limiting caused by too many requests, we added a minimum 3s gap between chat/completion calls, a delay between title generation, and retries with backoff for the educational-importance scoring. We also introduced a request chunking schema that enforced an 8k token length cap for videos with long transcripts. Due to the hackathon’s limited time frame, we struggled to prioritize features strategically and efficiently. To guide our development, we selected students as our target demographic, and honed in on making their experience as seamless as possible. Drawing from our personal experiences as current students, we concentrated on features that we wished we had as current students, and knew others would use. We took input from sponsors and other teams to gain a well-rounded understanding of our user pain points, and created a 20 minute demo Zoom lesson to thoroughly test our workflow. ### Accomplishments we're proud of We shipped something we’d genuinely use ourselves during finals week. We made search return nothing when nothing exists, an underrated feature. The first time we fed in a 50-minute Zoom podcast with Sam Altman and successfully extracted clips on curing cancer, reallocating human capital, and training GPT-2 Our AI now knows which parts of a lecture students actually care about (and which parts can safely be skipped). We compressed hours of lecture into minutes without losing meaning (helping every procrastinator out there) And most importantly: we made 2-hour recordings feel less like punishment and more like productivity. ### What we learned Pipeline correctness often mattered more than model complexity early in the process. In fact, a user-friendly product mostly relies on maintaining consistency across the platform, which requires careful attention to the little details. In real life, video recordings vary in structure, transcript quality, and metadata availability. Fallback modes and robust defaults are necessary but require careful consideration to maintain reliability and stability. ### What's next Our next step is to evolve from a highlight tool into a true video intelligence platform. On the product side, we plan to support collaborative workflows: shared workspaces, team-level knowledge organization, and persistent cloud-based notes. Imagine a social layer for zoomin, where classmates can share clips they found personally meaningful, help each other answer questions, and build a collective knowledge library. Longer term, we envision becoming the infrastructure layer for video understanding. As video becomes the dominant form for communication, we see the future belonging to platforms that are structured, searchable and truly understandable. Because the most important insights only emerge when you zoom in. ## README (from the GitHub repository) # Zoomin MVP Hackathon MVP that imports a Zoom recording (or uses a local demo fallback), computes multimodal importance by time window, generates clips, and supports natural-language-ish keyword search. ## Stack - Backend: FastAPI - Video clipping: ffmpeg - Transcript: Whisper (if installed), with local transcript fallback - Audio features: librosa (if installed), with fallback heuristics - Facial features: OpenFace (if configured), with fallback heuristics ## Run 1. Create env and install dependencies: - `python3 -m venv .venv` - `source .venv/bin/activate` - `pip install -r requirements.txt` 2. Install `ffmpeg` (system dependency required for clipping): - macOS: `brew install ffmpeg` - Ubuntu/Debian: `sudo apt-get update && sudo apt-get install -y ffmpeg` - Verify: `ffmpeg -version` and `ffprobe -version` 3. Configure `.env` from `.env.example`. 4. Start server: - `uvicorn app.main:app --reload --host 0.0.0.0 --port 8000` 5. Open `http://localhost:8000`. ## MVP flow 1. Click **Import Zoom Recordings**. 2. API tries Zoom latest cloud recording using `ZOOM_ACCESS_TOKEN`. 3. If unavailable, API uses generated local demo media. 4. Pipeline computes window-level multimodal features and importance. 5. Top windows become 3-5 clips and are listed in dashboard. 6. Use **Find a moment...** to search matching windows/clips. ## API - `POST /api/zoom/import` - `POST /api/upload` (one file only: `video` `.mp4`, optional `title`) - `GET /api/recordings` - `GET /api/recordings/{recording_id}` - `POST /api/search` - `GET /api/zoom/oauth/start` - `GET /api/zoom/oauth/callback?code=...` - `POST /api/clips/publish` - `GET /api/health` ## Notes - ffmpeg is required for real clip generation and thumbnails. - ffmpeg is a system binary and is not installed by `requirements.txt`. - Upload is strict by design: only one MP4 file is accepted, and invalid/non-video MP4s are rejected with HTTP 400. - Whisper/OpenFace are optional for this one-day MVP; code includes hooks and fallback behavior. - OpenFace: if the `FeatureExtraction` (or `OpenFace`) binary is on your PATH, it is used automatically. Otherwise set `OPENFACE_BINARY` in `.env` to the full path, or leave unset for heuristic face features. - The output data model for each window is represented by `TimeWindowFeature` in `app/models/schemas.py`. - Social publishing is implemented as a queue stub in `data/publish_queue.json` for YouTube/Instagram/TikTok/LinkedIn. ## Deploy on Render 1. Push this repo to GitHub. 2. In Render, create a **Blueprint** using `render.yaml`. 3. Ensure ffmpeg is available in the Render runtime/build image. 4. Set `ZOOM_ACCESS_TOKEN` (or keep blank to use demo fallback). 5. Deploy and open the generated URL. ## Detected evidence (automated analysis) Indexed codebase: 30 recognized source files, 325 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 161) ``` .dockerignore .gitignore app/__init__.py app/api/__init__.py app/api/routes.py app/core/__init__.py app/core/config.py app/main.py app/models/__init__.py app/models/schemas.py app/services/__init__.py app/services/embeddings.py app/services/pipeline.py app/services/resource_search.py app/services/section_search.py app/static/app.js app/static/clip-explore.html app/static/clip-explore.js app/static/clips.html app/static/clips.js app/static/index.html app/static/search.html app/static/search.js app/static/styles.css app/storage/__init__.py clips/__init__ data/clipfarmer_db.json data/demo_rec_001_chat.txt data/demo_rec_001_transcript.txt data/demo_rec_001_transcript.vtt data/embeddings/upload_gmt20260214_214842_recording_640x360_f0c424ec_windows.json data/embeddings/upload_videoplayback_9abaee40_windows.json data/embeddings/zoom_89844305927_clips.json data/embeddings/zoom_89844305927_windows.json data/publish_queue.json data/related_resources/upload_gmt20260214_214842_recording_640x360_7896a040.json data/related_resources/upload_gmt20260214_214842_recording_640x360_f0c424ec.json data/related_resources/upload_section_on_11212025_default_7085bb21.json data/related_resources/upload_section_on_11212025_default_clip_short_50664b73.json data/related_resources/upload_videoplayback_2_26ccaf5e/upload_videoplayback_2_26ccaf5e_clip_02.json data/related_resources/upload_videoplayback_2_2c6251db/upload_videoplayback_2_2c6251db_clip_01.json data/related_resources/upload_videoplayback_2_2c6251db/upload_videoplayback_2_2c6251db_clip_02.json data/related_resources/upload_videoplayback_2_2c6251db/upload_videoplayback_2_2c6251db_clip_03.json data/related_resources/upload_videoplayback_2_646c9817/upload_videoplayback_2_646c9817_clip_01.json data/related_resources/upload_videoplayback_2_7650541a/upload_videoplayback_2_7650541a_clip_01.json data/related_resources/upload_videoplayback_2_ba609de0/upload_videoplayback_2_ba609de0_clip_01.json data/related_resources/upload_videoplayback_2_d733d01b/upload_videoplayback_2_d733d01b_clip_02.json data/related_resources/upload_videoplayback_21bfb9e0.json data/related_resources/upload_videoplayback_21bfb9e0/upload_videoplayback_21bfb9e0_clip_01.json data/related_resources/upload_videoplayback_21bfb9e0/upload_videoplayback_21bfb9e0_clip_02.json data/related_resources/upload_videoplayback_387373cf/upload_videoplayback_387373cf_clip_02.json data/related_resources/upload_videoplayback_387373cf/upload_videoplayback_387373cf_clip_04.json data/related_resources/upload_videoplayback_b0632e4c/upload_videoplayback_b0632e4c_clip_01.json data/related_resources/upload_videoplayback_be2d7177/upload_videoplayback_be2d7177_clip_01.json data/related_resources/upload_videoplayback_d18f2877/upload_videoplayback_d18f2877_clip_01.json data/related_resources/zoom_83291784428/zoom_83291784428_clip_01.json data/related_resources/zoom_83291784428/zoom_83291784428_clip_02.json data/related_resources/zoom_89844305927.json data/related_resources/zoom_89844305927/zoom_89844305927_clip_01.json data/related_resources/zoom_89844305927/zoom_89844305927_clip_02.json data/summaries/clips/upload_gmt20260214_214842_recording_640x360_3e4dace6_upload_gmt20260214_214842_recording_640x360_3e4dace6_clip_01.txt data/summaries/clips/upload_gmt20260214_214842_recording_640x360_3e4dace6_upload_gmt20260214_214842_recording_640x360_3e4dace6_clip_02.txt data/summaries/clips/upload_gmt20260214_214842_recording_640x360_3e4dace6_upload_gmt20260214_214842_recording_640x360_3e4dace6_clip_03.txt data/summaries/clips/upload_gmt20260214_214842_recording_640x360_638c1747_upload_gmt20260214_214842_recording_640x360_638c1747_clip_01.txt data/summaries/clips/upload_videoplayback_be2d7177_upload_videoplayback_be2d7177_clip_01.txt data/summaries/clips/upload_videoplayback_be2d7177_upload_videoplayback_be2d7177_clip_02.txt data/summaries/clips/upload_videoplayback_be2d7177_upload_videoplayback_be2d7177_search_44a97133_01_000295_000445.txt data/summaries/clips/upload_videoplayback_be2d7177_upload_videoplayback_be2d7177_search_44a97133_02_000525_000675.txt data/summaries/clips/zoom_83291784428_zoom_83291784428_clip_01.txt data/summaries/clips/zoom_83291784428_zoom_83291784428_clip_02.txt data/summaries/clips/zoom_83291784428_zoom_83291784428_clip_03.txt data/summaries/clips/zoom_83291784428_zoom_83291784428_search_1038345e_01_002120_002270.txt data/summaries/clips/zoom_83291784428_zoom_83291784428_search_294062c7_01_000000_000150.txt data/summaries/clips/zoom_83291784428_zoom_83291784428_search_294062c7_02_000320_000470.txt data/summaries/clips/zoom_83291784428_zoom_83291784428_search_311a2743_01_002120_002270.txt data/summaries/clips/zoom_89844305927_zoom_89844305927_clip_01.txt data/summaries/clips/zoom_89844305927_zoom_89844305927_clip_02.txt data/summaries/clips/zoom_89844305927_zoom_89844305927_clip_03.txt data/summaries/clips/zoom_89844305927_zoom_89844305927_clip_04.txt data/summaries/clips/zoom_89844305927_zoom_89844305927_clip_05.txt data/summaries/clips/zoom_89844305927_zoom_89844305927_search_d6666df3_01_001345_001495.txt data/summaries/clips/zoom_89844305927_zoom_89844305927_search_d6666df3_02_002220_002370.txt data/summaries/upload_gmt20260214_214842_recording_640x360_1eba2fd2.txt data/summaries/upload_gmt20260214_214842_recording_640x360_3e4dace6.txt data/summaries/upload_gmt20260214_214842_recording_640x360_6c99a324.txt data/summaries/upload_gmt20260214_214842_recording_640x360_6e0394e4.txt data/summaries/upload_gmt20260214_214842_recording_640x360_7896a040.txt data/summaries/upload_gmt20260214_214842_recording_640x360_86d1cb00.txt data/summaries/upload_gmt20260214_214842_recording_640x360_f0c424ec.txt data/summaries/upload_section_on_11212025_default_7085bb21.txt data/summaries/upload_section_on_11212025_default_clip_short_4a0ce3e9.txt data/summaries/upload_section_on_11212025_default_clip_short_50664b73.txt data/summaries/upload_videoplayback_2_646c9817.txt data/summaries/upload_videoplayback_2_7650541a.txt data/summaries/upload_videoplayback_387373cf.txt data/summaries/upload_videoplayback_9abaee40.txt data/summaries/upload_videoplayback_be2d7177.txt data/summaries/zoom_83291784428.txt data/summaries/zoom_89844305927.txt data/topics/demo_rec_001.json data/topics/upload_gmt20260214_214842_recording_640x360_1eba2fd2.json data/topics/upload_gmt20260214_214842_recording_640x360_3e4dace6.json data/topics/upload_gmt20260214_214842_recording_640x360_6c99a324.json data/topics/upload_gmt20260214_214842_recording_640x360_6e0394e4.json data/topics/upload_gmt20260214_214842_recording_640x360_7896a040.json data/topics/upload_gmt20260214_214842_recording_640x360_86d1cb00.json data/topics/upload_gmt20260214_214842_recording_640x360_f0c424ec.json data/topics/upload_section_on_11212025_default_7085bb21.json data/topics/upload_section_on_11212025_default_clip_short_4a0ce3e9.json data/topics/upload_section_on_11212025_default_clip_short_50664b73.json data/topics/upload_videoplayback_2_646c9817.json data/topics/upload_videoplayback_2_7650541a.json data/topics/upload_videoplayback_387373cf.json data/topics/upload_videoplayback_9abaee40.json data/topics/upload_videoplayback_be2d7177.json data/topics/zoom_83291784428.json data/topics/zoom_89844305927.json data/upload_gmt20260214_214842_recording_640x360_13c16653_transcript.vtt data/upload_gmt20260214_214842_recording_640x360_1eba2fd2_transcript.vtt data/upload_gmt20260214_214842_recording_640x360_6c99a324_transcript.vtt [41 more files omitted for size] ``` ### Dependencies - requirements.txt: fastapi@==0.116.1, openai@>=1.0.0, openai-whisper@>=20231117, pydantic@==2.11.7, pydantic-settings@==2.10.1, python-multipart@==0.0.20, requests@==2.32.3, uvicorn[standard]@==0.35.0 ### Recent commits (newest first) - Delete OpenFace/cmake/modules directory - modify loading messages - fixed clip names - fix wording - summarize transcripts per clip - temp working v - fixed search queries - fixed heading indentation - remove data from git upload - merge - add share button - open ai based algorithm - Merge branch 'main' of https://github.com/arianna-caow/TreeHacks26 - add notes - Merge branch 'main' of https://github.com/arianna-caow/TreeHacks26 - Add AI summaries for highlights - removed view clips button - move related resource to clip page, change max clip length to 2 min, adjust how clips are coalesced - added captions to highlights - fix video clipping in search ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/sections-json-design.md ```markdown # Section JSON design for search When a recording is uploaded, it can be parsed into **sections** and stored as JSON. Search then runs over these section files so you can find moments by **text** and by **what’s happening** (topics, activity type, etc.). --- ## 1. Section JSON schema (one file per recording) Store one JSON file per recording, e.g. `data/sections/{recording_id}_sections.json`: ```json { "recording_id": "demo_rec_001", "source": "upload", "title": "Product kickoff", "duration_s": 180, "sections": [ { "section_id": "demo_rec_001_sec_000", "start_s": 0, "end_s": 25, "transcript_text": "Quick welcome everyone, today we will align on launch plans.", "summary": "Team kickoff and agenda for launch", "activity_type": "introduction", "topics": ["kickoff", "agenda", "launch"], "keywords": ["welcome", "align", "launch", "plans"], "speakers": [], "importance_score": 0.72, "clip_path": "/clips/demo_rec_001_clip_01.mp4", "thumbnail_path": "/thumbs/demo_rec_001_clip_01.jpg" }, { "section_id": "demo_rec_001_sec_001", "start_s": 25, "end_s": 52, "transcript_text": "The key point is deciding pricing at twenty nine dollars per seat...", "summary": "Pricing decision: $29/seat", "activity_type": "decision", "topics": ["pricing", "decision"], "keywords": ["key point", "pricing", "twenty nine", "seat"], "speakers": [], "importance_score": 0.89, "clip_path": "/clips/demo_rec_001_clip_01.mp4", "thumbnail_path": "/thumbs/demo_rec_001_clip_01.jpg" } ] } ``` **Fields:** | Field | Purpose | Search / “what’s happening” | |-------|--------|-----------------------------| | `transcript_text` | Raw words from STT | Full-text search (current behavior) | | `summary` | 1 short sentence describing the segment | Search by meaning, not only exact words | | `activity_type` | Kind of moment | Search by “decision”, “Q&A”, “demo”, “action items” | | `topics` | What the segment is about | Search by topic: “pricing”, “launch”, “timeline” | | `keywords` | Important terms | Boosts matches when query hits these | | `speakers` | Who’s talking (if you have diarization) | “Where did Sarah talk about X?” | | `importance_score` | Salience (existing pipeline) | Rank results by importance | So the JSON **can** summarize what people are doing **without relying only on raw text**: `activity_type` + `topics` + `summary` describe the segment; search can use all of them. --- ## 2. How to get “what’s happening” into the JSON (without only text) You don’t need an LLM on day one. You can start with rules and existing signals, then add an LLM later. **A. Reuse current pipeline** - Your existing windows already have: `transcript_text`, `keywords`, `emphasis_hits`, `importance_score`, and sometimes chat/audio/face. - When you write section JSON, map each window (or merged clip) to one section and set: - `transc [truncated — 4249 more characters] ``` ### requirements.txt ``` fastapi==0.116.1 uvicorn[standard]==0.35.0 pydantic==2.11.7 pydantic-settings==2.10.1 python-multipart==0.0.20 requests==2.32.3 openai>=1.0.0 # For transcript from uploaded/imported video (pip install openai-whisper; first run downloads ~140MB base model) openai-whisper>=20231117 ``` ### Dockerfile ``` # Use Python 3.11 slim image FROM python:3.11-slim # Set working directory WORKDIR /app # Set environment variables ENV PYTHONUNBUFFERED=1 \ PYTHONDONTWRITEBYTECODE=1 \ PIP_NO_CACHE_DIR=1 \ PIP_DISABLE_PIP_VERSION_CHECK=1 # Install system dependencies if needed (for ffmpeg, etc.) RUN apt-get update && apt-get install -y --no-install-recommends \ ffmpeg \ && rm -rf /var/lib/apt/lists/* # Copy requirements first for better caching COPY requirements.txt . # Install Python dependencies RUN pip install --upgrade pip && \ pip install -r requirements.txt # Copy application code COPY . . # Create necessary directories RUN mkdir -p clips thumbs data # Expose port (Render will set PORT env var) EXPOSE 8000 # Run the application CMD uvicorn app.main:app --host 0.0.0.0 --port ${PORT:-8000} ``` ### app/main.py ```python import logging from pathlib import Path from fastapi import FastAPI from fastapi.responses import FileResponse from fastapi.staticfiles import StaticFiles from app.api.routes import router, openai_status from app.core.config import get_settings logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s %(message)s") logger = logging.getLogger(__name__) settings = get_settings() app = FastAPI(title=settings.app_name) @app.on_event("startup") def _log_openai_status() -> None: try: status = openai_status() configured = status.get("configured", False) valid = status.get("valid") err = status.get("error") if configured and valid: logger.info("OpenAI: configured and valid (topics: gpt-4o-mini, embeddings: text-embedding-3-small)") elif configured: logger.warning("OpenAI: key set but validation failed — %s", err or "unknown") else: logger.info("OpenAI: not configured (%s)", err or "OPENAI_API_KEY not set") except Exception as e: logger.warning("OpenAI: could not check status — %s", e) app.include_router(router, prefix="/api") app.mount("/static", StaticFiles(directory="app/static"), name="static") app.mount(f"/{settings.clip_output_dir}", StaticFiles(directory=settings.clip_output_dir), name="clips") app.mount(f"/{settings.thumb_output_dir}", StaticFiles(directory=settings.thumb_output_dir), name="thumbs") app.mount(f"/{settings.data_dir}", StaticFiles(directory=settings.data_dir), name="data") @app.get("/") def index() -> FileResponse: return FileResponse(Path("app/static/index.html")) @app.get("/clips") def clips_page() -> FileResponse: return FileResponse(Path("app/static/clips.html")) @app.get("/search") def search_page() -> FileResponse: return FileResponse(Path("app/static/search.html")) @app.get("/clip/explore") def clip_explore_page() -> FileResponse: return FileResponse(Path("app/static/clip-explore.html")) ``` ### app/static/app.js ```javascript const statusEl = document.getElementById("status"); const importBtn = document.getElementById("import-btn"); const uploadForm = document.getElementById("upload-form"); const uploadBtn = document.getElementById("upload-btn"); const uploadVideoInput = document.getElementById("upload-video"); const uploadZone = document.getElementById("upload-zone"); const uploadZoneText = document.getElementById("upload-zone-text"); const uploadPreview = document.getElementById("upload-preview"); const uploadFilename = document.getElementById("upload-filename"); let uploadPreviewObjectURL = null; /** Derive a friendly recording name from the file name (strip extension, clean separators). */ function estimateRecordingName(filename) { if (!filename) return ""; let name = filename.replace(/\.(mp4|MP4|Mp4)$/i, "").trim(); name = name.replace(/[-_]+/g, " ").replace(/\s+/g, " ").trim(); return name || "Recording"; } function showUploadedFile(file) { if (!file || !uploadZoneText || !uploadPreview || !uploadFilename) return; if (uploadPreviewObjectURL) { URL.revokeObjectURL(uploadPreviewObjectURL); uploadPreviewObjectURL = null; } uploadZoneText.hidden = true; uploadFilename.hidden = false; const estimatedName = estimateRecordingName(file.name); uploadFilename.value = estimatedName; uploadPreview.hidden = false; uploadPreviewObjectURL = URL.createObjectURL(file); uploadPreview.src = uploadPreviewObjectURL; // Seek to a representative frame (skip black/blank start) so the preview thumbnail looks accurate uploadPreview.addEventListener( "loadedmetadata", function seekPreview() { const d = uploadPreview.duration; if (d > 0 && isFinite(d)) { uploadPreview.currentTime = Math.min(2, d * 0.05); } uploadPreview.removeEventListener("loadedmetadata", seekPreview); }, { once: true } ); uploadPreview.load(); uploadZone?.classList.add("upload-zone-has-file"); } function clearUploadDisplay() { if (uploadPreviewObjectURL) { URL.revokeObjectURL(uploadPreviewObjectURL); uploadPreviewObjectURL = null; } if (uploadPreview) { uploadPreview.src = ""; uploadPreview.hidden = true; } if (uploadFilename) { uploadFilename.value = ""; uploadFilename.hidden = true; } if (uploadZoneText) uploadZoneText.hidden = false; uploadZone?.classList.remove("upload-zone-has-file"); } function setupUploadZone() { if (!uploadVideoInput || !uploadZone) return; uploadVideoInput.addEventListener("change", () => { const file = uploadVideoInput.files?.[0]; if (file) showUploadedFile(file); else clearUploadDisplay(); }); uploadZone.addEventListener("dragover", (e) => { e.preventDefault(); e.stopPropagation(); uploadZone.classList.add("upload-zone-dragover"); }); uploadZone.addEventListener("dragleave", (e) => { e.preventDefault(); e.stopPropagation(); uploadZone.classList.remove("upload-zone-dragover"); }); uploadZone.addEventListener("drop", (e) => { e.preventDefault(); e.stopPropagation(); uploadZone.classList.remove("upload-zone-dragover"); const file = e.dataTransfer?.files?.[0]; if (file && file.type === "video/mp4") { const dt = new DataTransfer(); dt.items.add(file); uploadVideoInput.files = dt.files; showUploadedFile(file); } }); uploadFilename?.addEventListener("click", (e) => e.stopPropagation()); } async function importRecordings() { // Store original button text and change to "Importing..." const originalButtonText = importBtn.textContent; importBtn.textContent = "Importing..."; importBtn.disabled = true; statusEl.textContent = ""; try { const res = await fetch("/api/zoom/import", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ latest_only: true }), }); if (!res.ok) { throw new Error("Import request failed"); } const data = await res.json(); if (data.recordings.length) { const recordingId = data.recordings[0].recording_id; window.location.href = "/clips?recording_id=" + encodeURIComponent(recordingId); return; } else { statusEl.textContent = "No recordings found. Try uploading a video instead."; importBtn.textContent = originalButtonText; } } catch (err) { statusEl.textContent = `Error: ${err.message}`; importBtn.textContent = originalButtonText; } finally { importBtn.disabled = false; // Restore button text if not already restored (in case of error) if (importBtn.textContent === "Importing...") { importBtn.textContent = originalButtonText; } } } async function uploadRecording(event) { event.preventDefault(); const video = uploadVideoInput?.files?.[0]; if (!video) { statusEl.textContent = "Pick an MP4 video to upload."; return; } const formData = new FormData(); formData.append("video", video); const title = uploadFilename?.value?.trim(); if (title) { formData.append("title", title); } // Store original button text and change to "Uploading..." const originalButtonText = uploadBtn.textContent; uploadBtn.textContent = "Uploading..."; uploadBtn.disabled = true; importBtn.disabled = true; statusEl.textContent = ""; try { const res = await fetch("/api/upload", { method: "POST", body: formData }); const data = await res.json().catch(() => ({})); if (!res.ok) { const detail = data.detail; const msg = Array.isArray(detail) ? detail.map((d) => d.msg || d).join("; ") : (detail || "Upload failed"); throw new Error(msg); } const recordingId = data.recording?.recording_id; uploadForm.reset(); clearUploadDisplay(); if (recordingId) { window.location.href = "/clips?recording_id=" + encodeURIComponent(recordingId); return; } statusEl.textContent = ""; } catch (err) { statusEl.textContent = "Error: " + err.message; upl [truncated — 450 more characters] ``` ### render.yaml ```yaml services: - type: web name: zoomin env: docker plan: free dockerfilePath: Dockerfile envVars: - key: PYTHON_VERSION value: 3.11.10 - key: APP_ENV value: prod - key: WINDOW_SECONDS value: 15 - key: IMPORTANCE_THRESHOLD value: 0.55 - key: TOP_CLIP_COUNT value: 5 ``` ### highlights/__init__.py ```python """Highlights module for identifying important moments in transcripts.""" ``` ### highlights/scoring.py ```python """Scoring functions for transcript segments and windows.""" import re from typing import Dict, List, Set from highlights.srt_parser import Segment # Keyword/phrase patterns with weights DECISIONS_COMMITMENTS = { "decision", "decided", "confirmed", "we will", "we'll", "ship", "launch", "final", "commit", "committed", "agreed", "approve", "approved" } KEY_CONCEPTS_DEFINITIONS = { "key point", "the point is", "the idea is", "in other words", "means", "definition", "essentially", "basically", "the concept", "what this means" } ACTIONS_DEADLINES = { "action item", "owner", "assign", "timeline", "by ", "deadline", "next week", "next friday", "due date", "follow up", "todo", "task", "assignee" } RISKS_BLOCKERS = { "risk", "blocker", "concern", "legal", "need approval", "issue", "problem", "challenge", "obstacle", "warning", "caution" } QA_CONFUSION = { "can you explain", "i don't understand", "what is", "why", "how does", "clarify", "question", "confused", "unclear" } IMPACT_WORDS = { "impact", "affect", "because", "therefore", "so that", "consequence", "result", "outcome", "effect" } GREETINGS_CLOSINGS = { "welcome", "thanks", "thank you", "bye", "goodbye", "hello", "hi", "see you", "talk later" } # Regex for numbers or timeframes NUMBER_OR_TIMEFRAME_RE = re.compile( r'\b(\d+|one|two|three|four|five|six|seven|eight|nine|ten|' r'week|weeks|month|months|year|years|day|days|monday|tuesday|' r'wednesday|thursday|friday|saturday|sunday|january|february|' r'march|april|may|june|july|august|september|october|november|' r'december|morning|afternoon|evening|tonight|tomorrow|yesterday)\b', re.IGNORECASE ) def segment_score(seg: Segment) -> float: """ Score a segment based on keyword/phrase patterns and heuristics. Scoring system: - Decisions/commitments: +4 - Key concepts/definitions: +4 - Actions/deadlines: +4 - Risks/blockers: +3 - Q&A/confusion: +3 - Density boosts: +2 if contains number or timeframe +1 if contains impact words +1 if text length > 80 - Penalties: -2 for greetings/closings (unless combined with action items) Args: seg: Segment to score Returns: Score as float (can be negative) """ text_lower = seg.text.lower() score = 0.0 # Category scoring if any(phrase in text_lower for phrase in DECISIONS_COMMITMENTS): score += 4.0 if any(phrase in text_lower for phrase in KEY_CONCEPTS_DEFINITIONS): score += 4.0 if any(phrase in text_lower for phrase in ACTIONS_DEADLINES): score += 4.0 if any(phrase in text_lower for phrase in RISKS_BLOCKERS): score += 3.0 if any(phrase in text_lower for phrase in QA_CONFUSION): score += 3.0 # Density boosts if NUMBER_OR_TIMEFRAME_RE.search(text_lower): score += 2.0 if any(word in text_lower for word in IMPACT_WORDS): score += 1.0 if len(seg.text) > 80: score += 1.0 # Penalties is_greeting_or_closing = any(phrase in text_lower for phrase in GREETINGS_CLOSINGS) has_action_item = any(phrase in text_lower for phrase in ACTIONS_DEADLINES) if is_greeting_or_closing and not has_action_item: score -= 2.0 return score def window_features(segs: List[Segment]) -> Dict[str, any]: """ Extract features for a list of segments within a window. Args: segs: List of Segment objects. Returns: A dictionary of features including counts of keywords, unique speakers, etc. """ text_lower = " ".join(s.text for s in segs).lower() features: Dict[str, any] = { "decision_count": sum(1 for phrase in DECISIONS_COMMITMENTS if phrase in text_lower), "key_concept_count": sum(1 for phrase in KEY_CONCEPTS_DEFINITIONS if phrase in text_lower), "action_count": sum(1 for phrase in ACTIONS_DEADLINES if phrase in text_lower), "risk_count": sum(1 for phrase in RISKS_BLOCKERS if phrase in text_lower), "qa_count": sum(1 for phrase in QA_CONFUSION if phrase in text_lower), "unique_speakers": set(s.speaker for s in segs if s.speaker), "total_text_length": sum(len(s.text) for s in segs), } return features ``` ### highlights/srt_parser.py ```python """SRT/VTT parser with time conversion utilities.""" import re from dataclasses import dataclass from typing import List, Optional @dataclass class Segment: """Represents a single transcript segment with timing and text.""" start_ms: int end_ms: int start_ts: str end_ts: str speaker: Optional[str] text: str def __post_init__(self) -> None: """Validate that start < end.""" if self.start_ms >= self.end_ms: raise ValueError(f"Invalid segment: start ({self.start_ms}) >= end ({self.end_ms})") def ts_to_ms(timestamp: str) -> int: """ Convert timestamp string "HH:MM:SS.mmm" to milliseconds. Args: timestamp: Time string in format HH:MM:SS.mmm or HH:MM:SS,mmm Returns: Milliseconds as integer Raises: ValueError: If timestamp format is invalid """ # Handle both . and , as decimal separator timestamp = timestamp.replace(',', '.') # Match HH:MM:SS.mmm or HH:MM:SS pattern = r'(\d{2}):(\d{2}):(\d{2})(?:\.(\d{1,3}))?' match = re.match(pattern, timestamp) if not match: raise ValueError(f"Invalid timestamp format: {timestamp}") hours = int(match.group(1)) minutes = int(match.group(2)) seconds = int(match.group(3)) milliseconds = int(match.group(4) or '0') # Pad milliseconds to 3 digits if needed if milliseconds < 100 and match.group(4) and len(match.group(4)) == 1: milliseconds *= 100 elif milliseconds < 1000 and match.group(4) and len(match.group(4)) == 2: milliseconds *= 10 return (hours * 3600 + minutes * 60 + seconds) * 1000 + milliseconds def ms_to_ts(ms: int) -> str: """ Convert milliseconds to timestamp string "HH:MM:SS.mmm". Args: ms: Milliseconds as integer Returns: Time string in format HH:MM:SS.mmm """ seconds = ms // 1000 milliseconds = ms % 1000 minutes, seconds = divmod(seconds, 60) hours, minutes = divmod(minutes, 60) return f"{hours:02d}:{minutes:02d}:{seconds:02d}.{milliseconds:03d}" def parse_srt(content: str) -> List[Segment]: """ Parse SRT-like transcript content into a list of Segment objects. Handles various malformations, including missing cue numbers, extra newlines, and speaker labels. Args: content: Raw string content of the SRT/VTT file. Returns: A list of Segment objects. """ segments: List[Segment] = [] lines = content.strip().split('\n') i = 0 while i < len(lines): line = lines[i].strip() # Skip empty lines and cue numbers if not line or line.isdigit(): i += 1 continue # Try to parse timestamp line if '-->' in line: parts = [p.strip() for p in line.split('-->')] if len(parts) == 2: try: start_ts_str = parts[0].replace(',', '.') end_ts_str = parts[1].replace(',', '.') start_ms = ts_to_ms(start_ts_str) end_ms = ts_to_ms(end_ts_str) # Advance to text lines i += 1 text_lines: List[str] = [] while i < len(lines) and lines[i].strip() and '-->' not in lines[i]: text_lines.append(lines[i].strip()) i += 1 full_text = " ".join(text_lines).strip() speaker: Optional[str] = None # Extract speaker if present (e.g., "Speaker Name: Text") speaker_match = re.match(r"^([A-Za-z][A-Za-z0-9 .'-]{0,40}):\s*(.*)", full_text) if speaker_match: speaker = speaker_match.group(1) full_text = speaker_match.group(2).strip() if full_text: segments.append( Segment( start_ms=start_ms, end_ms=end_ms, start_ts=ms_to_ts(start_ms), end_ts=ms_to_ts(end_ms), speaker=speaker, text=full_text ) ) continue # Continue outer loop after processing a segment except ValueError: # If timestamp parsing fails, treat as text and continue pass i += 1 return segments ``` ### highlights/windowing.py ```python """Window generation and selection for highlight candidates.""" from dataclasses import dataclass, field from typing import Dict, List, Optional, Set from highlights.srt_parser import Segment, ms_to_ts from highlights.scoring import ( segment_score, window_features, DECISIONS_COMMITMENTS, KEY_CONCEPTS_DEFINITIONS, ACTIONS_DEADLINES, RISKS_BLOCKERS, QA_CONFUSION, IMPACT_WORDS, ) @dataclass class Window: """Represents a candidate highlight window.""" id: str start_ms: int end_ms: int start_ts: str end_ts: str duration_ms: int score: float segments: List[Segment] text: str meta: Dict[str, any] = field(default_factory=dict) def generate_candidates( segments: List[Segment], min_s: int = 10, max_s: int = 30, top_k: int = 50 ) -> List[Window]: """ Generate candidate highlight windows from segments. Algorithm: 1. For each start index i, expand j forward until duration >= min_s 2. After reaching min duration, keep expanding while: - duration <= max_s AND - for short windows (< 45s): not adding 2 low-score segments in a row - for long windows (>= 45s): not adding any low-score segment (entire run must be interesting) 3. Compute window_score = sum(segment_score) + synergy bonuses: +2 if window contains (decision OR key concept) AND (action/deadline OR impact words OR risk) +1 if >= 2 unique speakers 4. Build text by joining segment texts with spaces 5. Keep top_k windows by score Args: segments: List of segments to generate windows from min_s: Minimum window duration in seconds max_s: Maximum window duration in seconds top_k: Number of top candidates to return Returns: List of candidate Window objects, sorted by score. """ candidates: List[Window] = [] min_ms = min_s * 1000 max_ms = max_s * 1000 # For long windows, require every segment to be interesting (stop at first low-score). # Below this duration we allow 2 low-score in a row for flexibility. long_window_threshold_ms = 45 * 1000 if not segments: return [] for i in range(len(segments)): current_segments: List[Segment] = [] current_score = 0.0 low_score_count = 0 window_start_ms = segments[i].start_ms for j in range(i, len(segments)): seg = segments[j] current_segments.append(seg) seg_score = segment_score(seg) current_score += seg_score if seg_score <= 0: low_score_count += 1 else: low_score_count = 0 # Reset if a good segment is found window_end_ms = seg.end_ms duration_ms = window_end_ms - window_start_ms # If we haven't reached min duration, keep expanding if duration_ms < min_ms: continue # Stop expanding: over max, or too many low-score segments. # For long windows (e.g. >= 45s), require entire run to be interesting (stop at 1 low-score). stop_for_low = low_score_count >= 2 if duration_ms < long_window_threshold_ms else low_score_count >= 1 if duration_ms > max_ms or stop_for_low: # If we just passed min_ms and this segment made it too long/low, # try to create a window from previous segments if they meet criteria if len(current_segments) > 1: prev_window_segments = current_segments[:-1] prev_window_end_ms = prev_window_segments[-1].end_ms prev_duration_ms = prev_window_end_ms - window_start_ms if min_ms <= prev_duration_ms <= max_ms: # Re-calculate score and features for the valid sub-window sub_window_score = sum(segment_score(s) for s in prev_window_segments) sub_window_features = window_features(prev_window_segments) text = ' '.join(s.text for s in prev_window_segments).strip() theme = _determine_theme(sub_window_features) synergy_bonus = _calculate_synergy_bonus(sub_window_features) candidates.append( Window( id=f"window_{i}_{j-1}", start_ms=window_start_ms, end_ms=prev_window_end_ms, start_ts=ms_to_ts(window_start_ms), end_ts=ms_to_ts(prev_window_end_ms), duration_ms=prev_duration_ms, score=sub_window_score + synergy_bonus, segments=prev_window_segments, text=text, meta={"theme": theme, "features": sub_window_features} ) ) break # Stop expanding for this start segment # Create a candidate window features = window_features(current_segments) theme = _determine_theme(features) synergy_bonus = _calculate_synergy_bonus(features) text = ' '.join(s.text for s in current_segments).strip() candidates.append( Window( id=f"window_{i}_{j}", start_ms=window_start_ms, end_ms=window_end_ms, start_ts=ms_to_ts(window_start_ms), end_ts=ms_to_ts(window_end_ms), duration_ms=duration_ms, score=current_score + synergy_bonus, segments=current_segments, text=text, meta={"theme": theme, "features": features} ) ) # Sort by scor [truncated — 3246 more characters] ``` [14 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]