# Project export: RecallMe This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: A smart AI lecture chatbot that follows along in class, lets you ask questions live, gives immediate answers, and keeps track of context across lectures and uploaded recordings. - Devpost: https://devpost.com/software/temp-031caq - GitHub: https://github.com/nightowl23/calhacks-2025 - Video: https://www.youtube.com/embed/DPN2NaGX0sA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Letta: Build Your First Stateful AI Agent with Letta Cloud) - Team: 2 GitHub contributor(s) — Ruthwika Gajjala (9 commits), nightowl23 (6 commits) ## Devpost submission (written by the team) ### Inspiration As undergrad students sitting in lecture halls with 200+ students, it becomes incredibly easy to zone out, lose track of important concepts, or feel too intimidated to ask questions in real time. We wanted to change that. RecallMe was born from the idea that learning should be active, engaging, and personalized—not passive and easily forgotten. With live transcription and real-time interaction, RecallMe keeps students focused, curious, and connected to the lecture from start to finish. Instead of waiting for the professor to finish speaking or rewatching hours of recordings later, students can now engage instantly, ask questions the moment confusion strikes, and never miss a key idea again. What it Does RecallMe captures lecture audio in real time, transcribes it instantly, and generates dynamic, structured notes that students can interact with as the lecture is happening. But it doesn’t stop there—every lecture is stored as long-term memory inside our Letta agent, meaning students can return days or even months later and ask questions about previous lectures without having to dig through hours of videos. RecallMe also supports video uploads, where a user can upload a lecture recording, have it transcribed and stored in memory, and then interact with it using a larger context window. It’s like talking to an AI that truly remembers your classes—across time, topics, and video lectures. Using our dynamic database within Letta, we can easily recall information with each video_id. How We Built It We used a variety of tools and sponsors at Calhacks to build RecallMe! With the use of Letta, Reka, and LiveKit, we created RecallMe to achieve low-latency transcription, persistent memory storage, and context-aware interaction. LiveKit handles real-time audio streaming from the lecture, where audio is segmented into PCM chunks and streamed to our backend. These chunks are asynchronously processed and forwarded to two separate Letta agents via API calls. The first agent performs instant summarization and lecture note generation using its reasoning models, while the second agent is dedicated to long-term memory persistence. This memory agent stores each chunk’s transcription into structured core memory blocks, indexed by unique lecture or video IDs to emulate scalable vectorized memory retrieval. For non-live content, such as uploaded lecture videos, we use Reka’s speech-to-text API to batch-process the video stream into high-accuracy transcripts. Since Reka and Letta do not have native interoperability, we built a custom middleware pipeline to transform Reka’s output into Letta-compatible memory schemas before pushing them into the same memory graph used for live sessions. All stored memory is queryable using semantic retrieval, meaning a user can ask context-dependent questions and the system fetches the relevant lecture fragments from long-term storage. The final architecture enables seamless interaction across live lectures, stored sessions, and uploaded videos—allowing users to converse with an AI that maintains temporal continuity, lecture-specific context, and cross-session recall. Challenges None of us had ever used AI agents before, and definitely not all three platforms—Letta, Reka, and LiveKit—at once. Our first challenge was figuring out how to structure memory inside Letta so that lecture information could be stored, retrieved, and referenced over time. Next, we had to integrate Reka and Letta, despite the fact that no direct integration exists. We built a workaround that allowed us to transcribe videos with Reka and send that data into Letta’s memory. LiveKit brought its own challenges, as we had to process audio in real-time chunks and make sure the transcription was fast and accurate enough to be useful during live lectures. But the hardest part was creating one seamless system that could handle live transcription, memory storage, interactive querying, and video processing—while making it feel natural and effortless to the user. ### Accomplishments we're proud of We’re proud of our hard work in combining frameworks from three different sponsors into a unified platform, leveraging AI agents, speech-to-text models, and real-time chatbot functionality under a strict development timeline. We formed strong relationships with peers, mentors, and sponsor representatives, gaining valuable insights into agentic technology, startup ecosystems, and the growing scope for innovation in today’s AI-driven world. ### What we learned We learned how to integrate AI agents and speech-to-text models to develop a real-time, context-aware chatbot capable of processing and responding to live lecture content. Throughout the process, we gained hands-on experience working with three complex sponsor technologies—Letta, LiveKit, and Reka—and learned how to design effective pipelines that connect them seamlessly. We also strengthened our skills in asynchronous event handling, API orchestration, and real-time data streaming, while managing development under a tight hackathon timeline. Beyond the technical side, we learned how to collaborate efficiently under pressure, divide tasks strategically, and iterate quickly to transform an ambitious concept into a functional prototype. ### What's next Looking ahead, we aim to expand our platform’s intelligence, scalability, and accessibility: Multi-Modal Agent for Interactive Quizzes: We plan to integrate a multi-modal AI agent capable of generating real-time, interactive quizzes based on lecture content, helping students actively test their understanding. Vectorized Database for Efficient Querying: Implementing a vector database will enable faster and more accurate semantic search, improving how the chatbot retrieves and relates lecture information. Multilingual Transcription and Support: To make our platform accessible to a broader audience, we plan to introduce multilingual transcription and translation, allowing students worldwide to benefit from localized lecture understanding. These advancements will strengthen the platform’s educational value and push it closer to a fully intelligent, globally accessible lecture companion. ## README (from the GitHub repository) # Lecture Platform An AI-powered platform for recording, uploading, transcribing, and chatting with lecture content. Combines video upload, live audio recording, and intelligent AI chat using Reka Vision, Letta AI, and LiveKit. ## 🌟 Features - **📹 Video Upload**: Upload pre-recorded lecture videos via URL or file (up to 500MB) - **🎙️ Live Recording**: Record live audio lectures with real-time transcription - **🤖 AI Chat**: Chat with your lectures using persistent AI memory - **📝 Auto Transcription**: Automatic transcription of all video and audio content - **🧠 Context Retention**: AI remembers context across all your lectures - **💬 Chatbot Widget**: Floating chatbot for quick questions ## 🏗️ Architecture ``` ┌─────────────────────────────────────────────────────────────┐ │ Lecture Platform │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────────┐ ┌──────────────────┐ │ │ │ Flask Backend │ │ Next.js Frontend │ │ │ │ (Port 8112) │◄────────┤ (Port 3000) │ │ │ └──────────────────┘ └──────────────────┘ │ │ │ │ │ │ │ │ │ │ ▼ ▼ │ │ ┌──────────────┐ ┌──────────────┐ │ │ │ Reka Vision │ │ LiveKit │ │ │ │ API │ │ Server │ │ │ └──────────────┘ └──────────────┘ │ │ │ │ │ ▼ │ │ ┌──────────────┐ │ │ │ Letta AI │ │ │ │ (Memory) │ │ │ └──────────────┘ │ └─────────────────────────────────────────────────────────────┘ ``` ## 🚀 Quick Start ### Prerequisites - Python 3.11+ - Node.js 18+ - Reka API Key ([Get one here](https://link.reka.ai/free)) - Letta API credentials - LiveKit credentials ### 1. Clone the Repository ```bash git clone cd calhacks-2025 ``` ### 2. Set Up Flask Backend ```bash # Navigate to reka directory cd reka/src # Install dependencies pip install -r requirements.txt # Create .env file with your credentials cat > .env << EOF API_KEY=your_reka_api_key BASE_URL=https://api.reka.ai LETTA_TOKEN=your_letta_token LETTA_AGENT_ID=your_letta_agent_id EOF # Run the Flask server python app.py ``` The Flask backend will be available at `http://localhost:8112` ### 3. Set Up LiveKit Frontend ```bash # Navigate to agents-playground directory cd ../../agents-playground # Install dependencies npm install # Create .env.local file with your LiveKit credentials cat > .env.local << EOF NEXT_PUBLIC_LIVEKIT_URL=your_livekit_url LIVEKIT_API_KEY=your_livekit_api_key LIVEKIT_API_SECRET=your_livekit_api_secret EOF # Run the Next.js development server npm run dev ``` The LiveKit frontend will be available at `http://localhost:3000` ## 📖 Usage Guide ### Accessing the Platform Open your browser to `http://localhost:8112/form` to access the Lecture Library. ### Uploading Videos 1. Click the **"📹 Upload Video"** card 2. Enter a name for your lecture 3. Choose your upload method: - **Video URL**: Paste a direct link to a video file - **Upload File**: Select a video file from your computer 4. Click "Upload" 5. Wait for processing and automatic transcription 6. Click on the video card to view, watch, and chat with the content ### Recording Live Audio 1. Click the **"🎙️ Record Audio"** card 2. You'll be redirected to the LiveKit interface 3. Click "Record" to start capturing audio 4. Speak your lecture content 5. Click "Stop Recording" when finished 6. Real-time transcription will be displayed 7. Click "Library" (top-left) to return to your lecture library ### Chatting with Lectures **Per-Lecture Chat:** - Open any video from your library - Use the question input box to ask about that specific lecture - View answers powered by Reka Vision **Cross-Lecture Chat:** - Click the chatbot widget (🤖 icon, bottom-right corner) - Ask questions about any of your lectures - The AI has context from all transcribed content via Letta memory ## 🔧 Configuration ### Flask Backend (Port 8112) Configure in `reka/src/.env`: ```env API_KEY=your_reka_api_key # Reka Vision API key BASE_URL=https://api.reka.ai # Reka API base URL LETTA_TOKEN=your_letta_token # Letta authentication token LETTA_AGENT_ID=your_agent_id # Letta agent identifier ``` ### LiveKit Frontend (Port 3000) Configure in `agents-playground/.env.local`: ```env NEXT_PUBLIC_LIVEKIT_URL=wss://your-livekit-server.livekit.cloud LIVEKIT_API_KEY=your_api_key LIVEKIT_API_SECRET=your_api_secret ``` ## 🗂️ Project Structure ``` calhacks-2025/ ├── reka/ # Flask backend │ └── src/ │ ├── app.py # Main Flask application │ ├── templates/ # HTML templates │ │ ├── form.html # Lecture library page │ │ ├── video.html # Individual video page │ │ ├── index.html # Home page │ │ └── chatbot_widget.html # Chatbot widget │ └── static/ # CSS and static assets │ ├── agents-playground/ # Next.js frontend │ └── src/ │ ├── pages/ │ │ ├── index.tsx # Live recording page │ │ └── upload-video.tsx # Video upload page │ └── components/ │ └── playground/ │ └── PlaygroundHeader.tsx # Navigation header │ ├── INTEGRATION_GUIDE.md # Detailed integration guide └── README.md # This file ``` ## 🛠️ Tech Stack ### Backend - **Flask**: Web framework for API and video management - **Reka Vision**: AI-powered video transcription and analysis - **Letta AI**: Persistent memory and intelligent chat - **Python 3.11+**: Backend programming language ### Frontend - **Next.js**: React framework for LiveKit interface - **React 18**: UI library - **TypeScript**: Type-safe development - **LiveKit**: Real-time audio/video communication ### APIs & Services - **Reka Vision API**: Video analysis and transcription - **Letta API**: AI memory and chat - **LiveKit Cloud**: Real-time audio streaming ## 🐛 Troubleshooting ### Videos Not Appearing - Verify Flask server is running on port 8112 - Check Reka API credentials in `.env` - Review browser console for errors ### Recording Issues - Ensure LiveKit server is running on port 3000 - Verify LiveKit credentials in `.env.local` - Grant browser microphone permissions ### Transcription Failures - Confirm Reka API key is valid and has quota - Check video/audio quality and format - Review Flask server logs for error details ### Navigation Problems - Run both servers simultaneously - Clear browser cache - Check that ports 3000 and 8112 are not blocked by firewall ## 📊 API Endpoints ### Flask Backend (`http://localhost:8112`) | Endpoint | Method | Description | |----------|--------|-------------| | `/form` | GET | Lecture library page | | `/video/` | GET | Individual video page | | `/api/upload_video` | POST | Upload video via URL | | `/api/upload_video_file` | POST | Upload video file | | `/api/delete_video` | DELETE | Delete a video | | `/api/ask_question` | POST | Ask question about video | | `/api/chatbot/message` | POST | Chat with Letta AI | | `/api/transcription_status` | GET | Get transcription status | ### LiveKit Frontend (`http://localhost:3000`) | Route | Description | |-------|-------------| | `/` | Live audio recording interface | | `/upload-video` | Video upload interface | ## 🎓 Use Cas [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 2268 recognized source files, 18361 KB. - C (language) — detected in the code - CSS (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository - AI coding agent: Codex — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (120 of 2513) ``` .env abby/.dockerignore abby/.env.example abby/.github/workflows/ruff.yml abby/.github/workflows/template-check.yml abby/.github/workflows/tests.yml abby/.gitignore abby/AGENTS.md abby/CLAUDE.md abby/Dockerfile abby/GEMINI.md abby/livekit.toml abby/pyproject.toml abby/README.md abby/src/__init__.py abby/src/agent.py abby/tests/test_agent.py abby/token_server.py abby/uv.lock agent_id.txt agents-playground/.env.example agents-playground/.eslintrc.json agents-playground/.github/ISSUE_TEMPLATE/bug_report.yaml agents-playground/.github/ISSUE_TEMPLATE/config.yml agents-playground/.github/ISSUE_TEMPLATE/feature_request.yaml agents-playground/.gitignore agents-playground/LICENSE agents-playground/next.config.js agents-playground/NOTICE agents-playground/package.json agents-playground/postcss.config.js agents-playground/README.md agents-playground/renovate.json agents-playground/SIMPLE_WAVEFORM_USAGE.md agents-playground/src/cloud/CloudConnect.tsx agents-playground/src/cloud/README.md agents-playground/src/cloud/useCloud.tsx agents-playground/src/components/audio/DotWaveform.tsx agents-playground/src/components/audio/SimpleWaveform.tsx agents-playground/src/components/audio/WaveformMeter.tsx agents-playground/src/components/button/Button.tsx agents-playground/src/components/button/LoadingSVG.tsx agents-playground/src/components/chat/ChatMessage.tsx agents-playground/src/components/chat/ChatMessageInput.tsx agents-playground/src/components/chat/ChatTile.tsx agents-playground/src/components/colorPicker/ColorPicker.tsx agents-playground/src/components/config/AttributeRow.tsx agents-playground/src/components/config/AttributesInspector.tsx agents-playground/src/components/config/AudioInputTile.tsx agents-playground/src/components/config/ConfigurationPanelItem.tsx agents-playground/src/components/config/NameValueRow.tsx agents-playground/src/components/playground/icons.tsx agents-playground/src/components/playground/Playground.tsx agents-playground/src/components/playground/PlaygroundDeviceSelector.tsx agents-playground/src/components/playground/PlaygroundHeader.tsx agents-playground/src/components/playground/PlaygroundTile.tsx agents-playground/src/components/playground/RpcPanel.tsx agents-playground/src/components/playground/SettingsDropdown.tsx agents-playground/src/components/PlaygroundConnect.tsx agents-playground/src/components/toast/PlaygroundToast.tsx agents-playground/src/components/toast/ToasterProvider.tsx agents-playground/src/hooks/useConfig.tsx agents-playground/src/hooks/useConnection.tsx agents-playground/src/hooks/useTrackVolume.tsx agents-playground/src/hooks/useWindowResize.ts agents-playground/src/lib/tailwindTheme.preval.ts agents-playground/src/lib/types.ts agents-playground/src/lib/util.ts agents-playground/src/pages/_app.tsx agents-playground/src/pages/_document.tsx agents-playground/src/pages/api/token.ts agents-playground/src/pages/index.tsx agents-playground/src/pages/upload-video.tsx agents-playground/src/styles/globals.css agents-playground/src/transcriptions/TranscriptionTile.tsx agents-playground/tailwind.config.js agents-playground/tsconfig.json agents-playground/WAVEFORM_USAGE.md agents/CSE 120_agent_id.txt agents/Data Structures_agent_id.txt agents/Database Systems_agent_id.txt agents/Database_Systems_agent_id.txt agents/Machine Learning 101_agent_id.txt CHANGES_SUMMARY.md chatbot_widget.py embed_widget.html example_website.html INTEGRATION_GUIDE.md lecture/bin/activate lecture/bin/activate.csh lecture/bin/activate.fish lecture/bin/Activate.ps1 lecture/bin/dotenv lecture/bin/flask lecture/bin/httpx lecture/bin/normalizer lecture/bin/pip lecture/bin/pip3 lecture/bin/pip3.11 lecture/bin/python lecture/bin/python3 lecture/bin/python3.11 lecture/lib/python3.11/site-packages/_distutils_hack/__init__.py lecture/lib/python3.11/site-packages/_distutils_hack/override.py lecture/lib/python3.11/site-packages/annotated_types-0.7.0.dist-info/INSTALLER lecture/lib/python3.11/site-packages/annotated_types-0.7.0.dist-info/licenses/LICENSE lecture/lib/python3.11/site-packages/annotated_types-0.7.0.dist-info/METADATA lecture/lib/python3.11/site-packages/annotated_types-0.7.0.dist-info/RECORD lecture/lib/python3.11/site-packages/annotated_types-0.7.0.dist-info/WHEEL lecture/lib/python3.11/site-packages/annotated_types/__init__.py lecture/lib/python3.11/site-packages/annotated_types/py.typed lecture/lib/python3.11/site-packages/annotated_types/test_cases.py lecture/lib/python3.11/site-packages/anyio-4.11.0.dist-info/entry_points.txt lecture/lib/python3.11/site-packages/anyio-4.11.0.dist-info/INSTALLER lecture/lib/python3.11/site-packages/anyio-4.11.0.dist-info/licenses/LICENSE lecture/lib/python3.11/site-packages/anyio-4.11.0.dist-info/METADATA lecture/lib/python3.11/site-packages/anyio-4.11.0.dist-info/RECORD lecture/lib/python3.11/site-packages/anyio-4.11.0.dist-info/top_level.txt lecture/lib/python3.11/site-packages/anyio-4.11.0.dist-info/WHEEL lecture/lib/python3.11/site-packages/anyio/__init__.py [2393 more files omitted for size] ``` ### Dependencies - abby/pyproject.toml: flask, flask-cors, letta-client, livekit-agents[silero,turn-detector]@~=1.2, livekit-api, livekit-plugins-assemblyai, livekit-plugins-noise-cancellation@~=0.2, python-dotenv - agents-playground/package.json: @livekit/components-react@^2.9.3, @livekit/components-styles@^1.1.5, @radix-ui/react-dropdown-menu@^2.1.2, @types/js-yaml@^4.0.9, @types/lodash@^4.17.13, @types/node@^20.17.9, @types/react@^18.3.14, @types/react-dom@^18.3.3, autoprefixer@^10.4.20, cookies-next@^4.3.0, eslint@^8.57.1, eslint-config-next@14.2.26, framer-motion@^10.18.0, js-yaml@^4.1.0, livekit-client@^2.9.5, livekit-server-sdk@^2.13.0, lodash@^4.17.21, next@^14.2.20, next-plugin-preval@^1.2.6, postcss@^8.4.49, prettier@^3.4.2, qrcode.react@^4.1.0, react@^18.3.1, react-audio-spectrum@^0.1.5, react-dom@^18.3.1, react-markdown@^10.1.0, tailwindcss@^3.4.16, typescript@^5.7.2 - reka/requirements.txt: Flask@==3.0.0, Flask-CORS@==4.0.0, letta-client@>= 0.1.320, markdown@>=3.4.0, python-dotenv@==1.1.1, requests@==2.31.0, Werkzeug@==3.0.1 - reka/workshop/requirements.txt: Flask@==3.0.0, markdown@>=3.4.0, python-dotenv@==1.1.1, requests@==2.31.0, Werkzeug@==3.0.1 - requirements.txt: Flask@==2.3.3, letta-client, markdown@==3.5.1, python-dotenv@==1.0.0, requests@==2.31.0 ### Recent commits (newest first) - further ui fixes - minor fixes - working mvp - Merge origin/letta into main; keep README from main - Merge pull request #1 from nightowl23/calhacks-clean - integrated chat - debug statements - letta in reka - feat: complete calhacks project with audio visualizer and agent integration - letta in reka - letta in reka - letta in reka - first push - letta in reka - Update agents-playground submodule with audio visualizer changes - lifekit code - trying to push - first push - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### PROGRESS.md ```markdown # Lecture Note-Taking App Progress ## ✅ Completed ### 1. LiveKit Agents Playground - UI Customization - **Cloned** LiveKit Agents Playground repo - **Added** Live Lecture / Upload Video toggle buttons in header - **Modified** PlaygroundHeader.tsx to support mode switching - **Integrated** mode state management in Playground.tsx ### 2. Python Agent (STT) - **Location:** `abby/tests/test_agent.py` - **Features:** - Deepgram STT integration working - Transcription receiving and publishing - Agent connects to LiveKit Cloud successfully ### 3. Token Server - **Location:** `abby/token_server.py` - **Features:** - Flask server generating JWT tokens - Runs on port 3000 - Integrates with LiveKit Cloud ## 🔄 In Progress / Next Steps ### Letta Integration (Not Started) 1. Install Letta Python SDK 2. Modify test_agent.py to: - Send transcripts to Letta - Get formatted notes from Letta - Publish notes back to frontend 3. Update frontend to display Letta notes instead of raw transcripts ## 📁 Project Structure ``` calhacks-2025/ ├── abby/ # Python backend │ ├── tests/test_agent.py # LiveKit STT agent │ ├── token_server.py # JWT token server │ ├── .env.local # LiveKit credentials │ └── pyproject.toml # Python dependencies │ ├── agents-playground/ # Frontend (Modified) │ ├── src/ │ │ ├── components/ │ │ │ └── playground/ │ │ │ ├── PlaygroundHeader.tsx # ✏️ MODIFIED - Added toggle │ │ │ └── Playground.tsx # ✏️ MODIFIED - Mode management │ │ └── pages/ │ └── .env.local # LiveKit config │ └── stt-frontend/ # Old React app (can delete) ``` ## 🚀 How to Run ### Terminal 1: Token Server ```bash cd abby uv run token_server.py ``` ### Terminal 2: Python Agent ```bash cd abby uv run tests/test_agent.py dev ``` ### Terminal 3: Frontend ```bash cd agents-playground npm run dev ``` Open: http://localhost:3000 ## 🎯 Current Goal **Build Live Lecture Mode:** - User clicks "Live Lecture" toggle - User clicks record - Audio → STT → Letta (formats as notes) → Display ## 📝 Notes - Hosted playground works: https://agents-playground.livekit.io/ - Local playground needs `.env.local` configured - Deepgram STT plugin installed and working - Next: Add Letta for transcript → notes conversion ``` ### INTEGRATION_GUIDE.md ```markdown # Lecture Platform - Integration Guide ## Overview This guide explains how to run and use the unified Lecture Platform, which combines video upload, live audio recording, and AI-powered chat functionality. ## Architecture The platform consists of two main components: 1. **Flask Backend (reka)**: Handles video uploads, transcription via Reka Vision, and chat via Letta AI 2. **Next.js Frontend (agents-playground)**: Provides live audio recording via LiveKit ## Prerequisites - Python 3.11+ - Node.js 18+ - Reka API Key - Letta API credentials - LiveKit credentials (for live recording) ## Setup Instructions ### 1. Flask Backend Setup Navigate to the reka directory: ```bash cd reka/src ``` Install dependencies: ```bash pip install -r requirements.txt ``` Create a `.env` file with your credentials: ```env API_KEY=your_reka_api_key BASE_URL=https://api.reka.ai LETTA_TOKEN=your_letta_token LETTA_AGENT_ID=your_letta_agent_id ``` Run the Flask server: ```bash python app.py ``` The Flask app will be available at: `http://localhost:8112` ### 2. LiveKit Frontend Setup Navigate to the agents-playground directory: ```bash cd agents-playground ``` Install dependencies: ```bash npm install ``` Create a `.env.local` file with your LiveKit credentials: ```env NEXT_PUBLIC_LIVEKIT_URL=your_livekit_url LIVEKIT_API_KEY=your_livekit_api_key LIVEKIT_API_SECRET=your_livekit_api_secret ``` Run the Next.js development server: ```bash npm run dev ``` The LiveKit app will be available at: `http://localhost:3000` ## Using the Platform ### Starting Point: Lecture Library 1. Open your browser to `http://localhost:8112/form` 2. You'll see your lecture library with two action cards: - **📹 Upload Video**: Upload pre-recorded lecture videos - **🎙️ Record Audio**: Record live audio lectures ### Option 1: Upload Video 1. Click the **"Upload Video"** card 2. Enter a name for your lecture 3. Choose between: - **Video URL**: Paste a URL to a video file - **Upload File**: Upload a video file from your computer (up to 500MB) 4. Click "Upload" 5. The video will be processed and automatically transcribed 6. Once complete, click on the video to: - Watch the video - View the transcript - Chat with the content using AI ### Option 2: Record Audio 1. Click the **"Record Audio"** card 2. You'll be taken to the LiveKit recording interface 3. Click "Record" to start recording audio 4. Speak your lecture content 5. Click "Stop Recording" when finished 6. The audio will be transcribed in real-time 7. Click the "Library" button in the top-left to return to your lecture library ### Chatting with Lectures Both uploaded videos and recorded audio are stored in Letta AI's memory: - On each video/lecture page, you can ask questions - The chatbot widget (bottom-right corner) allows you to chat across all your lectures - The AI remembers context from all your transcribed content ## Navigation Between Components ### From Lecture Library → Record Audio - Click the "🎙 [truncated — 2412 more characters] ``` ### requirements.txt ``` Flask==2.3.3 requests==2.31.0 python-dotenv==1.0.0 letta-client markdown==3.5.1 ``` ### reka/requirements.txt ``` Flask==3.0.0 Flask-CORS==4.0.0 Werkzeug==3.0.1 python-dotenv==1.1.1 requests==2.31.0 markdown>=3.4.0 letta-client >= 0.1.320 ``` ### reka/Dockerfile ``` # Use Python 3.12 slim image as base FROM python:3.12-slim # Set working directory in the container WORKDIR /app # Recommended Python runtime flags ENV PYTHONDONTWRITEBYTECODE=1 \ PYTHONUNBUFFERED=1 # Copy requirements file COPY requirements.txt . # Install Python dependencies RUN pip install --no-cache-dir -r requirements.txt # Copy application files COPY src/ src/ # Expose port 5000 EXPOSE 5000 # Optional build-time injection of environment variables (not recommended for secrets) # Usage: docker build --build-arg API_KEY=... --build-arg BASE_URL=... -t roast-my-life . ARG API_KEY ARG BASE_URL # Make them available at runtime (can be overridden by docker run -e / --env-file) ENV API_KEY=${API_KEY} \ BASE_URL=${BASE_URL} # Set environment variables ENV FLASK_APP=src/app.py ENV FLASK_ENV=production # Run the application CMD ["python", "src/app.py"] ``` ### abby/pyproject.toml ``` [build-system] requires = ["setuptools>=61.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "agent-starter-python" version = "1.0.0" description = "Simple voice AI assistant built with LiveKit Agents for Python" requires-python = ">=3.9" dependencies = [ "livekit-agents[silero,turn-detector]~=1.2", "livekit-plugins-noise-cancellation~=0.2", "livekit-plugins-assemblyai", "python-dotenv", "flask", "flask-cors", "livekit-api", "letta-client", ] [dependency-groups] dev = [ "pytest", "pytest-asyncio", "ruff", ] [tool.setuptools.packages.find] where = ["src"] [tool.setuptools.package-dir] "" = "src" [tool.pytest.ini_options] asyncio_mode = "auto" asyncio_default_fixture_loop_scope = "function" [tool.ruff] line-length = 88 target-version = "py39" [tool.ruff.lint] select = ["E", "F", "W", "I", "N", "B", "A", "C4", "UP", "SIM", "RUF"] ignore = ["E501"] # Line too long (handled by formatter) [tool.ruff.format] quote-style = "double" indent-style = "space" ``` ### agents-playground/package.json ``` { "name": "agents-playground", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint", "format": "prettier --write ." }, "dependencies": { "@livekit/components-react": "^2.9.3", "@livekit/components-styles": "^1.1.5", "@radix-ui/react-dropdown-menu": "^2.1.2", "cookies-next": "^4.3.0", "framer-motion": "^10.18.0", "js-yaml": "^4.1.0", "livekit-client": "^2.9.5", "livekit-server-sdk": "^2.13.0", "lodash": "^4.17.21", "next": "^14.2.20", "next-plugin-preval": "^1.2.6", "qrcode.react": "^4.1.0", "react": "^18.3.1", "react-audio-spectrum": "^0.1.5", "react-dom": "^18.3.1", "react-markdown": "^10.1.0" }, "devDependencies": { "@types/js-yaml": "^4.0.9", "@types/lodash": "^4.17.13", "@types/node": "^20.17.9", "@types/react": "^18.3.14", "@types/react-dom": "^18.3.3", "autoprefixer": "^10.4.20", "eslint": "^8.57.1", "eslint-config-next": "14.2.26", "postcss": "^8.4.49", "prettier": "^3.4.2", "tailwindcss": "^3.4.16", "typescript": "^5.7.2" } } ``` ### abby/Dockerfile ``` # syntax=docker/dockerfile:1 # Use the official UV Python base image with Python 3.13 on Debian Bookworm # UV is a fast Python package manager that provides better performance than pip # We use the slim variant to keep the image size smaller while still having essential tools ARG PYTHON_VERSION=3.13 FROM ghcr.io/astral-sh/uv:python${PYTHON_VERSION}-bookworm-slim AS base # Keeps Python from buffering stdout and stderr to avoid situations where # the application crashes without emitting any logs due to buffering. ENV PYTHONUNBUFFERED=1 # Create a non-privileged user that the app will run under. # See https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#user ARG UID=10001 RUN adduser \ --disabled-password \ --gecos "" \ --home "/app" \ --shell "/sbin/nologin" \ --uid "${UID}" \ appuser # Install build dependencies required for Python packages with native extensions # gcc: C compiler needed for building Python packages with C extensions # python3-dev: Python development headers needed for compilation # We clean up the apt cache after installation to keep the image size down RUN apt-get update && apt-get install -y \ gcc \ g++ \ python3-dev \ && rm -rf /var/lib/apt/lists/* # Create a new directory for our application code # And set it as the working directory WORKDIR /app # Copy just the dependency files first, for more efficient layer caching COPY pyproject.toml uv.lock ./ RUN mkdir -p src # Install Python dependencies using UV's lock file # --locked ensures we use exact versions from uv.lock for reproducible builds # This creates a virtual environment and installs all dependencies # Ensure your uv.lock file is checked in for consistency across environments RUN uv sync --locked # Copy all remaining application files into the container # This includes source code, configuration files, and dependency specifications # (Excludes files specified in .dockerignore) COPY . . # Change ownership of all app files to the non-privileged user # This ensures the application can read/write files as needed RUN chown -R appuser:appuser /app # Switch to the non-privileged user for all subsequent operations # This improves security by not running as root USER appuser # Pre-download any ML models or files the agent needs # This ensures the container is ready to run immediately without downloading # dependencies at runtime, which improves startup time and reliability RUN uv run src/agent.py download-files # Run the application using UV # UV will activate the virtual environment and run the agent. # The "start" command tells the worker to connect to LiveKit and begin waiting for jobs. CMD ["uv", "run", "src/agent.py", "start"] ``` ### reka/workshop/requirements.txt ``` Flask==3.0.0 Werkzeug==3.0.1 python-dotenv==1.1.1 requests==2.31.0 markdown>=3.4.0 ``` ### reka/workshop/app.py ```python import os import time from typing import Any, Dict, List from dotenv import load_dotenv from flask import Flask, render_template, request, jsonify import requests app = Flask(__name__) load_dotenv() api_key = os.environ.get('API_KEY') base_url = os.environ.get('BASE_URL') # Endpoint for the external chat/vision agent API. If needed this can be # overridden via env; otherwise we default to using base_url + /qa/chat REKA_VIDEO_QA_ENDPOINT = os.environ.get( 'REKA_VIDEO_QA_ENDPOINT', f"{base_url.rstrip('/')}/qa/chat" ) # Simple in-memory cache for videos to avoid hitting the API on every request. _VIDEO_CACHE: Dict[str, Any] = { "timestamp": 0.0, "ttl": 60.0, "results": [] } def fetch_videos() -> List[Dict[str, Any]]: """ STEP: Listing videos dynamically — fetch_videos() Paste the full fetch_videos() implementation from the README here. Until you paste it, we return an empty list so the page renders. """ return [] def call_reka_vision_qa(video_id: str) -> Dict[str, Any]: """ STEP: Roast — Vision QA call (backend) Paste the full call_reka_vision_qa() implementation from the README here. """ return {"error": "Not implemented. Paste call_reka_vision_qa() from README."} def simple_markdown_to_html(md: str) -> str: """ STEP: Roast — markdown to HTML Paste the full simple_markdown_to_html() implementation from the README here. """ return "" @app.route('/') def home() -> str: """ Render the home page with welcome text. Returns: str: Rendered HTML template for the home page. """ return render_template('index.html') @app.route('/form') def form_page() -> str: """ Render the form page with dynamic video selection grid. Returns: str: Rendered HTML template for the form page. """ videos = fetch_videos() # Transform videos to a simplified structure for the template. template_videos = [] for v in videos: meta = v.get("metadata", {}) template_videos.append({ "id": v.get("video_id"), "name": meta.get("title") or meta.get("video_name") or "Untitled", # fallback "thumbnail": meta.get("thumbnail") or "/static/images/image1.jpg", "url": v.get("url") or meta.get("url") or "", }) return render_template('form.html', videos=template_videos) @app.route('/api/upload_video', methods=['POST']) def upload_video() -> Dict[str, Any]: """ STEP: Add your own videos (Upload API route) Paste the full /api/upload_video route implementation from the README here. """ return jsonify({"success": False, "error": "Not implemented. Paste /api/upload_video from README."}), 501 @app.route('/api/process', methods=['POST']) def process_video() -> Dict[str, Any]: """ STEP: Roast API route (/api/process) Paste the full /api/process route implementation from the README here. """ return jsonify({"success": False, "error": "Not implemented. Paste /api/process from README."}), 501 if __name__ == '__main__': app.run(debug=True, host='0.0.0.0', port=5000) ``` [2170 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]