# Project export: LectureFlow This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Teaching on Zoom for the AI era. - Devpost: https://devpost.com/software/lectureflow - GitHub: https://github.com/ChristmasSun/TreeHacks2026 - Team: 4 GitHub contributor(s) — ChristmasSun (41 commits), Theo Chapman (28 commits), Claude Opus 4.5 (22 commits), lolismek (1 commits) ## Devpost submission (written by the team) ### Inspiration Every university student knows the feeling: you're lost in a 300-person lecture, too afraid to raise your hand, and office hours are packed. Professors want to help every student individually, but there simply aren't enough hours in the day. We asked ourselves — what if every student could have a 1-on-1 session with their professor, powered by AI, right inside Zoom? Google Drive with demo video and more materials about this project: https://drive.google.com/drive/folders/1erRwy80BjeV3pBOeuafkrem6VlTP2hqX?usp=sharing ### What it does LectureFlow takes a professor's live lecture and turns it into a full, personalized learning experience — all delivered through Zoom, the platform students and professors already use. The pipeline works like this: Lecture In — A professor delivers their lecture over Zoom as they normally would. LectureFlow captures the audio and video via Zoom's Real-Time Media Streams (RTMS) API, transcribes it, and uses an LLM to break the content into discrete conceptual scenes. Animated Explainers Out — For each concept, the system automatically generates a 3Blue1Brown-style Manim animation with voice-cloned narration from the professor. These aren't generic — they're visual explanations tailored to the specific ideas covered in that day's lecture. Quizzes Linked to Concepts — From those same concepts, the system generates multiple-choice quiz questions. Each question is tied back to the explainer video that covers it. Students type /quiz in Zoom Team Chat and get an adaptive quiz — when they answer wrong, they watch the specific 2-minute animated explanation for that concept before moving on. AI Professor in Every Breakout Room — Using HeyGen's Interactive Avatar SDK, we clone the professor's face and voice into a lifelike AI avatar deployed via LiveKit streaming. LectureFlow creates Zoom breakout rooms and places a personalized AI tutor in each one. The avatar has full context of that day's lecture material — updated in real-time from RTMS transcripts — uses Socratic questioning, and adapts to each student's pace. Because it looks and sounds like their actual professor, students engage with it naturally rather than treating it like a generic chatbot. Real-Time Professor Dashboard — The professor monitors everything from an Electron app: live transcripts from the lecture, quiz performance across the class, and which concepts students are struggling with. Live Student Sentiment — By running expression recognition models on periodically sampled video frames from Zoom's RTMS feed, LectureFlow gives professors a real-time, aggregated view of classroom engagement — surfacing confusion, disengagement, or frustration as it happens, not after the fact. The key insight is that everything flows from the same source material — that day's lecture. The lecture produces the concepts, the concepts produce the animations, the animations back the quizzes, the quizzes inform the tutoring, and the professor sees it all in one place. For demo purposes at TreeHacks, we use YouTube lecture links as input since we can't host a live lecture on the spot — but the pipeline is the same regardless of whether the audio comes from a live Zoom session or a recording. ### How we built it LectureFlow is four microservices working together: Manim Video Pipeline (Python + asyncio) — Ingests lecture audio, transcribes with Whisper, uses an LLM to plan conceptual scenes, generates Manim animation code for each, renders in parallel, and stitches them with voice-cloned narration (ElevenLabs/PocketTTS). Quiz questions are generated from the same scene plan, each linked back to its explainer video. Python Backend (FastAPI) — The orchestration layer. Manages session lifecycle, creates HeyGen interactive avatar sessions with real-time lecture context, serves adaptive quizzes through a Zoom Team Chat chatbot, processes video frames for student sentiment analysis, and stores everything in a SQLAlchemy database. Zoom RTMS Bridge (Node.js on Render) — A Render-hosted service that receives Zoom webhooks, opens Real-Time Media Stream connections to live meetings, and forwards transcript chunks, chat messages, and video frames to the Python backend over HTTP and WebSocket. It also triggers quiz DMs to students when they first speak in the meeting. Render was essential here — it gave us a publicly reachable endpoint for Zoom webhooks, solving the NAT traversal problem, while relaying events over WebSocket to our local backend. Electron App (React + TypeScript) — The professor's control center. Create sessions, monitor live transcripts in real-time, trigger quizzes, interact with HeyGen avatars, and review class-wide analytics — all in a frosted-glass UI. ### Challenges we ran into Manim Code Generation — LLMs frequently generate Manim code that doesn't compile. We built a retry system with error feedback loops, parallel rendering with semaphores, and fallback strategies to keep the pipeline robust. Zoom RTMS Integration — Getting real-time audio, video, and transcripts out of Zoom required navigating the RTMS WebSocket protocol — handling media handshakes with S2S OAuth signatures, parsing mixed audio/video/transcript/chat message types, and maintaining persistent connections with keep-alive heartbeats every 100ms. HeyGen Avatar Context Injection — Making the HeyGen avatar feel like a real extension of the professor meant continuously injecting lecture context from RTMS transcripts into the avatar's knowledge base mid-session, so it could reference what was just taught moments ago rather than relying on static prompts. Unifying the Pipeline — The hardest design problem was making the concept graph the single source of truth. Scene planning, video generation, quiz questions, and tutoring context all needed to reference the same conceptual breakdown of the lecture, so changes upstream propagate cleanly. Interactive Team Chat Quizzes — Building a stateful, per-student quiz experience inside Zoom Team Chat meant managing quiz sessions across webhook-driven interactions — tracking progress, handling button clicks for A/B/C/D answers, generating follow-up questions on wrong answers, and linking back to the exact explainer video for each concept. ### Accomplishments we're proud of A single pipeline that takes one lecture and produces animated explainers, targeted quizzes, and personalized AI tutoring — all interconnected Real-time lecture understanding via Zoom RTMS that feeds transcripts directly into HeyGen avatar context, so the AI tutor always knows what was just taught — not a static bot, but one that evolves with the lecture A fully automated system that generates 3Blue1Brown-quality animated videos with voice-cloned narration from any lecture An interactive quiz system inside Zoom Team Chat that plays the exact explainer video for concepts a student gets wrong Live student sentiment analysis from RTMS video frames, giving professors real-time engagement visibility ~11,000 lines of production-quality code across Python, TypeScript, and JavaScript — built in one weekend ### What we learned Zoom's API ecosystem is remarkably deep — RTMS for live media, Team Chat for interactive bots, S2S OAuth for server integration — and once you understand how the pieces fit together, it enables workflows that wouldn't be possible on any other platform. HeyGen's Interactive Avatar SDK made something possible that we couldn't have built from scratch in a weekend: a real-time, lifelike video avatar that students actually want to talk to. The ability to inject context mid-session was the key to making it feel like a real tutor, not a canned demo. Render gave us instant public deployment for our RTMS bridge — no infrastructure headaches, just a webhook URL that worked. For a hackathon, that speed matters. LLM-generated code (especially Manim) needs robust validation and retry mechanisms; you can't trust it to compile on the first try. The biggest leverage in an AI education tool isn't any single feature — it's making them share the same conceptual backbone so the experience feels cohesive, not like five disconnected tools. ### What's next RAG-Powered Context Engine — Letting professors upload syllabi, textbooks, and slides so the AI tutor can reference specific course materials during conversations. On-the-Fly Quiz Generation — Generating quizzes from live lecture transcripts in real-time, closing the loop even further. Multi-Language Support — Translating explainer videos and avatar responses for international students. Production Deployment — Dockerized horizontal scaling with PostgreSQL and a polished onboarding flow for professors. ## README (from the GitHub repository) # AI Professor - Scalable Personalized Education > Transform any lecture into an interactive AI-powered learning experience with avatar tutors, auto-generated quizzes, and 3Blue1Brown-style explainer videos. [![Built at TreeHacks 2026](https://img.shields.io/badge/Built%20at-TreeHacks%202026-blue)]() [![Phase 1](https://img.shields.io/badge/Phase%201-Complete-success)]() [![Phase 2](https://img.shields.io/badge/Phase%202-Complete-success)]() [![Phase 3](https://img.shields.io/badge/Phase%203-Complete-success)]() [![Quiz Bot](https://img.shields.io/badge/Quiz%20Bot-Complete-success)]() [![Manim Pipeline](https://img.shields.io/badge/Manim%20Pipeline-Complete-success)]() ## The Problem Professors can't give personalized 1-on-1 attention to hundreds of students. Office hours are limited, and students often struggle with concepts without immediate help. ## Our Solution An AI-powered professor toolkit that: 1. **Clones the professor** as an AI avatar that can tutor students individually 2. **Generates animated explainers** from any lecture (3Blue1Brown style) 3. **Creates interactive quizzes** delivered via Zoom Team Chat 4. **Provides real-time analytics** on student understanding --- ## Features ### 1. HeyGen AI Avatar Tutoring - Professor's likeness cloned as interactive AI avatar - Real-time lip-sync and natural conversation - Joins Zoom breakout rooms to tutor students 1-on-1 - Context-aware responses using lecture transcripts ### 2. Manim Video Generation Pipeline Turn any YouTube lecture into animated educational content: ``` YouTube URL → Transcribe → Scene Split → Manim Animations → Voice Clone → Final Video ``` - **Whisper transcription** via Dedalus API - **LLM scene planning** - intelligently splits lectures into concept-based scenes - **Auto-generated Manim code** - creates 3Blue1Brown-style animations - **Voice cloning** with PocketTTS - maintains the professor's voice - **Parallel rendering** - generates multiple scenes concurrently ### 3. Interactive Quiz System - **Zoom Team Chat Chatbot** - students type `/makequiz` to start - **Auto-generated questions** from lecture concepts using Cerebras LLM - **Interactive button cards** - A/B/C/D answer buttons - **Video on wrong answer** - plays the relevant Manim explainer scene - **Progress tracking** - scores and concepts to review ### 4. Real-Time Meeting Integration - **Zoom RTMS** (Real-Time Media Streams) for live transcription - **WebSocket architecture** - Render service broadcasts to local dashboard - **Live transcript accumulation** per meeting - **Demeanor/engagement analysis** (extensible) ### 5. Professor Dashboard - Frosted glass Electron UI - One-click session start - Real-time student analytics - Quiz trigger buttons - Meeting management --- ## Architecture ``` ┌─────────────────────────────────────────────────────────────────────────────────┐ │ PROFESSOR DASHBOARD │ │ (Electron + React + Tailwind) │ └───────────────────────────────────┬─────────────────────────────────────────────┘ │ WebSocket ▼ ┌─────────────────────────────────────────────────────────────────────────────────┐ │ PYTHON BACKEND │ │ (FastAPI + SQLite) │ │ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐ │ │ │ Zoom API │ │ HeyGen API │ │ Cerebras LLM│ │ Quiz Session Manager │ │ │ │ (meetings) │ │ (avatars) │ │ (generation)│ │ (state per student) │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────────────────┘ │ └───────────────────────────────────┬─────────────────────────────────────────────┘ │ ┌───────────────────────────┼───────────────────────────┐ ▼ ▼ ▼ ┌───────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Zoom Meeting │ │ Render (RTMS) │ │ Zoom Team Chat │ │ │◄───────►│ Node.js │◄───────►│ Chatbot │ │ - Breakouts │ RTMS │ - Webhooks │ WS │ - /makequiz │ │ - Avatars │ WS │ - Transcripts │ │ - Buttons │ └───────────────┘ └─────────────────┘ └─────────────────┘ ``` --- ## Tech Stack | Layer | Technology | |-------|------------| | **Frontend** | Electron, React, TypeScript, Tailwind CSS | | **Backend** | Python 3.11+, FastAPI, SQLAlchemy, asyncio | | **RTMS Service** | Node.js, Express, WebSocket | | **Video Pipeline** | Manim, FFmpeg, PocketTTS | | **Database** | SQLite (dev), PostgreSQL (prod) | | **Deployment** | Render (cloud), uv (Python pkg mgmt) | ### APIs & Services | Service | Purpose | |---------|---------| | **Zoom REST API** | Meeting creation, breakout rooms, user management | | **Zoom RTMS** | Real-time audio/video/transcript streams | | **Zoom Team Chat** | Chatbot for interactive quizzes | | **HeyGen** | AI avatar generation and streaming | | **Deepgram** | Speech-to-text, text-to-speech | | **Cerebras** | Fast LLM inference (Llama 3.3 70B) | | **Dedalus** | Whisper API for transcription | | **PocketTTS** | Voice cloning for narration | | **HuggingFace** | Model hosting for TTS | --- ## Project Structure ``` TreeHacks2026/ ├── src/ # Manim video pipeline │ ├── pipeline.py # Main orchestration │ ├── downloader.py # YouTube audio download │ ├── transcribe.py # Whisper transcription │ ├── scene_splitter.py # LLM-based scene planning │ ├── clip_generator.py # Manim code generation │ ├── voice.py # TTS with voice cloning │ └── stitcher.py # Final video assembly │ ├── backend/ # Python backend │ ├── app.py # FastAPI main app │ ├── run_chatbot_client.py # Quiz WebSocket client │ ├── services/ │ │ ├── render_ws_client.py # Connects to Render WebSocket │ │ ├── chatbot_ws_handler.py # Handles /makequiz commands │ │ ├── quiz_generator.py # LLM quiz generation │ │ ├── quiz_session_manager.py # Per-student quiz state │ │ ├── zoom_chatbot_service.py # Zoom API message sending │ │ ├── heygen_controller.py # Avatar management │ │ ├── session_orchestrator.py # Meeting lifecycle │ │ └── llm_service.py # Cerebras/OpenAI wrapper │ └── models/ # SQLAlchemy models │ ├── rtms-zoom-official/ # Render-deployed Node.js service │ ├── index.js # Express + webhook handlers │ ├── frontendWss.js # WebSocket broadcasting │ └── library/ # RTMS SDK wrappers │ ├── frontend/ # Electron app (if separate) │ ├── outputs/ # Generated content (gitignored) │ └── {topic-name}/ │ ├── audio.mp3 │ ├── transcript.txt │ ├── scene_plan.json │ ├── quiz_questions.json │ └── videos/ │ ├── prompts/ # LLM prompt templates │ └── docs/ # Documentation ├── QUIZ_INTEGRATION.md # Quiz system guide ├── QUICKSTART.md └── DEPLOYMENT.md ``` --- ## Quick Start ### Prerequisites - Python 3.11+ - Node.js 18+ - uv (Python package manager) - FFmpeg - LaTeX (for Manim) ### 1. Clone and Install ```bash git clone https://github.com/ChristmasSun/TreeHacks2026.git cd TreeHacks2026 # Python dependencies uv sync # Node dependencies (for RTMS service) cd rtms-zoom-official && npm install && cd .. ``` ### 2. Co [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 175 recognized source files, 1096 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (120 of 332) ``` .env.example .gitignore .python-version backend/__init__.py backend/.env.example backend/app.py backend/integrations/__init__.py backend/integrations/deepgram_adapter.py backend/integrations/heygen_api_adapter.py backend/models/__init__.py backend/models/database.py backend/models/models.py backend/run_chatbot_client.py backend/scripts/demo_breakout.py backend/scripts/init_db.py backend/scripts/seed_data.py backend/services/__init__.py backend/services/chatbot_ws_handler.py backend/services/demeanor_service.py backend/services/expression_service.py backend/services/heygen_controller.py backend/services/heygen_lite_client.py backend/services/llm_service.py backend/services/pocket_tts_service.py backend/services/quiz_generator.py backend/services/quiz_session_manager.py backend/services/render_ws_client.py backend/services/rtms_transcription_service.py backend/services/session_orchestrator.py backend/services/speculative_llm.py backend/services/tts_service.py backend/services/tutor_session.py backend/services/vad_service.py backend/services/zoom_chatbot_service.py backend/static/professor-dashboard.html bun.lock CLAUDE.md demo-ng/animation-code/scene_000_attempt_1_error.txt demo-ng/animation-code/scene_000_attempt_1.py demo-ng/animation-code/scene_000_attempt_2.py demo-ng/animation-code/scene_001_attempt_1.py demo-ng/animation-code/scene_002_attempt_1_error.txt demo-ng/animation-code/scene_002_attempt_1.py demo-ng/animation-code/scene_002_attempt_2.py demo-ng/animation-code/scene_003_attempt_1.py demo-ng/animation-code/scene_004_attempt_1.py demo-ng/animation-code/scene_005_attempt_1.py demo-ng/animation-code/scene_006_attempt_1.py demo-ng/animation-code/scene_007_attempt_1.py demo-ng/animation-code/scene_008_attempt_1.py demo-ng/animation-code/scene_009_attempt_1_error.txt demo-ng/animation-code/scene_009_attempt_1.py demo-ng/animation-code/scene_009_attempt_2_error.txt demo-ng/animation-code/scene_009_attempt_2.py demo-ng/animation-code/scene_009_attempt_3.py demo-ng/animation-code/scene_010_attempt_1.py demo-ng/animation-code/scene_011_attempt_1.py demo-ng/animation-code/scene_012_attempt_1.py demo-ng/animation-code/scene_013_attempt_1.py demo-ng/animation-code/scene_014_attempt_1_error.txt demo-ng/animation-code/scene_014_attempt_1.py demo-ng/animation-code/scene_014_attempt_2.py demo-ng/audio.opus demo-ng/narration_scripts.json demo-ng/render_results.json demo-ng/scene_plan.json demo-ng/transcript.txt demo-ng/videos/Tex/00254b12d5956f4f.tex demo-ng/videos/Tex/0119e98ea375c4a5.tex demo-ng/videos/Tex/04ec29325f935bea.tex demo-ng/videos/Tex/09dc9cada7e939b0.tex demo-ng/videos/Tex/0a15dfcc46a53849.tex demo-ng/videos/Tex/0a65a7c3232fcd14.tex demo-ng/videos/Tex/0ce1028f715753f4.tex demo-ng/videos/Tex/0da99a3eb93cad91.tex demo-ng/videos/Tex/0f7c101631d8c0b4.tex demo-ng/videos/Tex/119d9bb5b5832143.tex demo-ng/videos/Tex/125126c829e77202.tex demo-ng/videos/Tex/13c0bde68d158f3a.tex demo-ng/videos/Tex/1489d5ce401c4324.tex demo-ng/videos/Tex/17efa82f38064ae2.tex demo-ng/videos/Tex/1ee93250046f4b8f.tex demo-ng/videos/Tex/1fa33da528630661.tex demo-ng/videos/Tex/21226a86b6f71711.tex demo-ng/videos/Tex/236b5de694afb373.tex demo-ng/videos/Tex/25026fbeea8720bf.tex demo-ng/videos/Tex/26549b5cda311bcb.tex demo-ng/videos/Tex/267acf69c7a63a41.tex demo-ng/videos/Tex/2ba0c0da047d75f5.tex demo-ng/videos/Tex/2c08093caf846594.tex demo-ng/videos/Tex/2c5751777e30b28b.tex demo-ng/videos/Tex/2e593a68c512f764.tex demo-ng/videos/Tex/2f30c5849c7f5106.tex demo-ng/videos/Tex/383d1fb9b7d92e82.tex demo-ng/videos/Tex/3ca2a7c70ae751c8.tex demo-ng/videos/Tex/3cc7b8de8f5544f2.tex demo-ng/videos/Tex/3e2d297a94d544d9.tex demo-ng/videos/Tex/3ecdda5b14fcaacb.tex demo-ng/videos/Tex/3ef5510985301300.tex demo-ng/videos/Tex/47fba03eee4f1127.tex demo-ng/videos/Tex/4a2c485865cccf7d.tex demo-ng/videos/Tex/4abf9b8012dbb8e4.tex demo-ng/videos/Tex/4bb8e69e29206c1d.tex demo-ng/videos/Tex/4d3f2ef5737559c2.tex demo-ng/videos/Tex/4d4ab04218b2ff28.tex demo-ng/videos/Tex/53c159c2e272df4c.tex demo-ng/videos/Tex/55c29e6bfa3fed61.tex demo-ng/videos/Tex/5a4491881615af6d.tex demo-ng/videos/Tex/5cc4c7db91b87654.tex demo-ng/videos/Tex/605c54b6835fe9a8.tex demo-ng/videos/Tex/62d7ba7a5588d532.tex demo-ng/videos/Tex/65aa4e50fc992ba7.tex demo-ng/videos/Tex/68fc92f787070001.tex demo-ng/videos/Tex/70a691901ea22862.tex demo-ng/videos/Tex/71239e7c3a1d7b4f.tex demo-ng/videos/Tex/78735f376b07973b.tex demo-ng/videos/Tex/79b39221bc2a05ad.tex demo-ng/videos/Tex/7ac28d29a12f4efa.tex demo-ng/videos/Tex/7bfe00c37f281dc8.tex demo-ng/videos/Tex/7c0d0c07a8a939e7.tex [212 more files omitted for size] ``` ### Dependencies - expression-dashboard/pyproject.toml: fastapi@>=0.115.0, fer@>=22.5.1, numpy@>=2.0.0, opencv-python-headless@>=4.10.0, pillow@>=11.0.0, python-multipart@>=0.0.18, setuptools@<81, uvicorn@>=0.34.0 - package.json: @heygen/streaming-avatar@^2.1.0, @types/node@^20.11.0, @types/react@^18.2.48, @types/react-dom@^18.2.18, @types/ws@^8.5.10, @vitejs/plugin-react@^4.2.1, autoprefixer@^10.4.17, concurrently@^8.2.2, electron@^40.4.1, electron-builder@^24.9.1, electron-vite@^2.0.0, eslint@^8.56.0, livekit-client@^2.1.5, postcss@^8.4.33, react@^18.2.0, react-dom@^18.2.0, tailwindcss@^3.4.1, typescript@^5.3.3, vite@^5.0.12, ws@^8.19.0, zustand@^4.5.0 - pyproject.toml: aiofiles@>=25.1.0, aiosqlite@>=0.22.1, dedalus_labs, deepgram-sdk@>=5.3.2, fastapi@>=0.129.0, greenlet@>=3.3.1, httpx, manim@==0.19.2, openai@>=2.21.0, pocket-tts@>=1.0.3, python-dotenv@>=1.2.1, scipy, silero-vad@>=6.2.0, sqlalchemy@>=2.0.46, torchaudio@>=2.10.0, torchcodec@>=0.10.0, uvicorn@>=0.40.0, websockets@>=16.0, yt-dlp@>=2026.2.4 - rtms-zoom-official/library/javascript/rtmsManager/package.json: node-fetch@^3.3.2, ws@^8.16.0 - rtms-zoom-official/library/javascript/webhookManager/package.json: express@^4.21.2 - rtms-zoom-official/library/javascript/webSocketManager/package.json: node-fetch@^3.3.2, ws@^8.16.0 - rtms-zoom-official/library/python/requirements.txt: python-dotenv@>=1.0.0, websockets@>=11.0 - rtms-zoom-official/package.json: @deepgram/sdk@^4.11.0, dotenv@^16.5.0, ejs@^3.1.10, express@^4.21.2, node-fetch@^3.3.2, openai@^5.10.2, uuid@^11.1.0, ws@^8.16.0 - zoom-bot-service/package.json: @types/cors@^2.8.17, @types/express@^4.17.21, @types/node@^20.11.5, @types/uuid@^9.0.7, @types/ws@^8.5.10, @zoom/meetingsdk@^3.7.0, axios@^1.13.5, cors@^2.8.5, dotenv@^16.3.1, electron@^40.4.1, express@^4.18.2, jsonwebtoken@^9.0.3, jsrsasign@^11.1.0, livekit-client@^2.17.1, playwright@^1.58.2, puppeteer@^21.7.0, ts-node@^10.9.2, typescript@^5.3.3, uuid@^9.0.1, ws@^8.16.0 ### Recent commits (newest first) - added wavs back - fixed ng voice - Fix connection issue - demo ng in, updated ui - fixed backend not connecting - Merge Demeanor system - Merge pull request #10 from ChristmasSun/theo-voice - Merge remote-tracking branch 'origin/main' into theo-voice - fix: patch critical security vulnerabilities before merge - working perfectly, welcome wayne - actually good live call? - modified UI and added 1.30 timestamp for ytb videos - heygen livekit migration in progress - fix: Use JPG codec for expression analysis (OpenCV compatible) - fix: Use H.264 codec for video instead of JPG - full rewrite ready for demo - debug: Add video frame logging to troubleshoot - fix: Use Optional type hint for Python 3.9 compatibility - fix: Route video frames through WebSocket to local Python backend - feat: Add demeanor/expression analysis system ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown ## Python (Manim pipeline) Use uv for all package management and running programs ``` ### RTMS_SETUP.md ```markdown # Zoom RTMS + HeyGen Live Transcription Setup This integration captures live meeting transcripts via Zoom RTMS and feeds them as context to HeyGen avatars. ## Architecture ``` Zoom Meeting ↓ (webhook: meeting.rtms_started) RTMS Service (rtms-service/) ↓ (WebSocket connection to Zoom RTMS) Live Transcript Stream ↓ (HTTP POST to Python backend) Python Backend (backend/) ↓ (context update) HeyGen Avatar → Responds with awareness of conversation ``` ## Quick Start ### 1. Configure Environment Copy the environment template: ```bash cd rtms-service cp .env.example .env ``` Edit `.env` with your credentials: ```bash # Required: Zoom credentials from your Zoom App ZOOM_CLIENT_ID=your_client_id ZOOM_CLIENT_SECRET=your_client_secret ZOOM_SECRET_TOKEN=your_webhook_secret_token # Python backend URL PYTHON_BACKEND_URL=http://localhost:8000 # Optional: Deepgram for TTS DEEPGRAM_API_KEY=your_deepgram_key ``` ### 2. Start Services **Terminal 1 - Python Backend:** ```bash cd backend python -m uvicorn app:app --reload --port 8000 ``` **Terminal 2 - RTMS Service:** ```bash cd rtms-service npm start ``` **Terminal 3 - Expose Webhook (for development):** ```bash ngrok http 3002 ``` ### 3. Configure Zoom App 1. Go to [Zoom App Marketplace](https://marketplace.zoom.us/develop/create) 2. Create or edit your **General App** 3. In **Features** → **Event Subscriptions**: - Add webhook URL: `https://your-ngrok-url.ngrok.io/webhook` - Subscribe to events: - `meeting.rtms_started` - `meeting.rtms_stopped` 4. In **Scopes**, add: - `meeting:read:meeting_audio` - `meeting:read:meeting_transcript` ### 4. Enable RTMS for Meetings RTMS must be enabled at the account level. Contact Zoom or post in [Zoom Developer Forum](https://devforum.zoom.us/) to request access. ## API Endpoints ### RTMS Service (Port 3002) | Endpoint | Method | Description | |----------|--------|-------------| | `/health` | GET | Health check | | `/stats` | GET | Active RTMS sessions | | `/webhook` | POST | Zoom webhook events | | `/ws` | WS | Frontend WebSocket | ### Python Backend (Port 8000) | Endpoint | Method | Description | |----------|--------|-------------| | `/api/rtms/session-start` | POST | RTMS session started | | `/api/rtms/session-stop` | POST | RTMS session stopped | | `/api/rtms/transcript` | POST | Receive transcript chunk | | `/api/rtms/session/{uuid}/transcripts` | GET | Get recent transcripts | ## How It Works ### 1. Meeting Starts with RTMS ``` Zoom → Webhook: meeting.rtms_started ↓ RTMS Service receives { meeting_uuid, rtms_stream_id, server_urls } ↓ Creates RTMSClient, connects to Zoom's WebSocket servers ``` ### 2. Transcripts Flow ``` Zoom RTMS WebSocket → msg_type: 17 (TRANSCRIPT) ↓ RTMSClient emits 'transcript' event ↓ HeyGenBridge.forwardTranscript() → POST /api/rtms/transcript ↓ Python backend stores in RTMSTranscriptionService ↓ Updates HeyGen avatar context ``` [truncated — 2233 more characters] ``` ### pyproject.toml ``` [project] name = "theo-manim" version = "0.1.0" description = "Generate 3Blue1Brown-style Manim videos from lecture transcripts" requires-python = ">=3.12" dependencies = [ "aiofiles>=25.1.0", "aiosqlite>=0.22.1", "dedalus_labs", "deepgram-sdk>=5.3.2", "fastapi>=0.129.0", "greenlet>=3.3.1", "httpx", "manim==0.19.2", "openai>=2.21.0", "pocket-tts>=1.0.3", "python-dotenv>=1.2.1", "scipy", "silero-vad>=6.2.0", "sqlalchemy>=2.0.46", "torchaudio>=2.10.0", "torchcodec>=0.10.0", "uvicorn>=0.40.0", "websockets>=16.0", "yt-dlp>=2026.2.4", ] ``` ### package.json ``` { "name": "ai-professor-breakout-system", "version": "1.0.0", "description": "AI-powered breakout room system with HeyGen professor clones", "main": "out/main/index.js", "scripts": { "dev": "concurrently \"npm run dev:electron\" \"npm run dev:backend\"", "dev:electron": "electron-vite dev", "dev:backend": "cd backend && uv run python -m uvicorn app:app --reload --host 0.0.0.0 --port 8000", "build": "electron-vite build", "build:electron": "electron-builder", "preview": "electron-vite preview", "typecheck": "tsc --noEmit", "lint": "eslint src --ext .ts,.tsx", "avatar": "electron scripts/avatar.js" }, "dependencies": { "@heygen/streaming-avatar": "^2.1.0", "livekit-client": "^2.1.5", "react": "^18.2.0", "react-dom": "^18.2.0", "ws": "^8.19.0", "zustand": "^4.5.0" }, "devDependencies": { "@types/node": "^20.11.0", "@types/react": "^18.2.48", "@types/react-dom": "^18.2.18", "@types/ws": "^8.5.10", "@vitejs/plugin-react": "^4.2.1", "autoprefixer": "^10.4.17", "concurrently": "^8.2.2", "electron": "^40.4.1", "electron-builder": "^24.9.1", "electron-vite": "^2.0.0", "eslint": "^8.56.0", "postcss": "^8.4.33", "tailwindcss": "^3.4.1", "typescript": "^5.3.3", "vite": "^5.0.12" }, "author": "TreeHacks2026", "license": "MIT" } ``` ### expression-dashboard/pyproject.toml ``` [project] name = "expression-dashboard" version = "0.1.0" description = "Facial expression analysis dashboard for Zoom meetings" requires-python = ">=3.12" dependencies = [ "fastapi>=0.115.0", "uvicorn>=0.34.0", "fer>=22.5.1", "setuptools<81", "opencv-python-headless>=4.10.0", "pillow>=11.0.0", "numpy>=2.0.0", "python-multipart>=0.0.18", ] ``` ### rtms-zoom-official/package.json ``` { "name": "rtms-sample", "version": "1.0.0", "type": "module", "scripts": { "start": "node index.js" }, "engines": { "node": ">=20.3.0" }, "dependencies": { "@deepgram/sdk": "^4.11.0", "dotenv": "^16.5.0", "ejs": "^3.1.10", "express": "^4.21.2", "node-fetch": "^3.3.2", "openai": "^5.10.2", "uuid": "^11.1.0", "ws": "^8.16.0" } } ``` ### zoom-bot-service/package.json ``` { "name": "zoom-bot-service", "version": "1.0.0", "description": "Node.js service for managing Zoom Meeting SDK bots", "main": "dist/index.js", "scripts": { "dev": "ts-node src/index.ts", "build": "tsc", "start": "node dist/index.js", "bot-service": "node src/bot-service.js", "test-deploy": "node test-bot-deploy.js", "heygen-simple": "node heygen-simple.js", "heygen-screenshare": "node heygen-screenshare.js", "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [ "zoom", "bot", "sdk", "meeting" ], "author": "TreeHacks2026", "license": "MIT", "dependencies": { "@zoom/meetingsdk": "^3.7.0", "axios": "^1.13.5", "cors": "^2.8.5", "dotenv": "^16.3.1", "electron": "^40.4.1", "express": "^4.18.2", "jsonwebtoken": "^9.0.3", "jsrsasign": "^11.1.0", "livekit-client": "^2.17.1", "playwright": "^1.58.2", "puppeteer": "^21.7.0", "uuid": "^9.0.1", "ws": "^8.16.0" }, "devDependencies": { "@types/cors": "^2.8.17", "@types/express": "^4.17.21", "@types/node": "^20.11.5", "@types/uuid": "^9.0.7", "@types/ws": "^8.5.10", "ts-node": "^10.9.2", "typescript": "^5.3.3" } } ``` ### rtms-zoom-official/library/python/requirements.txt ``` websockets>=11.0 python-dotenv>=1.0.0 ``` ### rtms-zoom-official/library/javascript/commonHelpers/package.json ``` { "type": "module" } ``` ### rtms-zoom-official/library/javascript/webhookManager/package.json ``` { "name": "webhook-manager", "version": "1.0.0", "type": "module", "dependencies": { "express": "^4.21.2" } } ``` [166 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]