# Project export: ProfAI
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: Cal Hacks 12.0
- Tagline: Helping educators be more understood
- Devpost: https://devpost.com/software/profai-ml932w
- GitHub: https://github.com/asrayg/calhacks12.0-profAI
- Video: https://www.youtube.com/embed/FPCA3JE578Y?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 3 GitHub contributor(s) — Asray Gopa (17 commits), Rahul Sudev (3 commits), Arthur Lemus (1 commits)
## Devpost submission (written by the team)
### Overview
ProfAI
### Inspiration
We've all been in university classes that have been taught by professors with thicker accents that can be hard to comprehend. Also, it gets hard to understand the content they're teaching, even though they're brilliant people from all around the world. So we built ProfAI to bridge the gap between the world's best educators and university students.
### What it does
This application provides real-time accent translation for the classroom. It captures a professor's audio and instantly streams it to students in a clear American accent, ensuring they can follow the lecture live without comprehension difficulties. Our platform also enhances studying with powerful integrated tools. It automatically transcribes the professor's speech into detailed notes in real-time. Furthermore, it features a "rubber ducky mode," a smart voice agent that has context from your course materials on Canvas, YouTube, and Google Drive. You can talk to this agent while studying to get answers, explanations, or even have it perform web searches for you.
### How we built it
We built this project as a real-time audio-processing pipeline. Here’s a breakdown of the technology stack: Our application is built with a Python Flask backend and a React frontend. The core real-time audio pipeline uses LiveKit to manage the STT (Speech-to-Text) and TTS (Text-to-Speech) workflow, enabling instant accent translation. The live transcription feature is powered by printing the direct output from the STT module. For the "rubber ducky" assistant, we built a conversational AI bot, also using LiveKit, and expanded its knowledge base using Llama to pull context from course materials for accurate, on-demand answers.
### Challenges we ran into
Our primary challenge was latency. Building a direct STT-to-TTS pipeline fast enough for a live lecture was difficult, creating race condition issues that required significant optimization. This was compounded by network variability (WiFi issues) that we could not completely solve. We also faced an architectural hurdle in adapting our system from a default conversational chatbot model to an efficient, live transcription service. Finally, we had to find a TTS engine that balanced speed with voice naturalness to ensure the output sounded human and was suitable for long lectures.
### Accomplishments we're proud of
We are incredibly proud that we built a functional, end-to-end, real-time pipeline. Hearing a voice go in with one accent and come out in another, almost instantly, was a huge "wow" moment for our team. We successfully built a tool that directly solves a problem we and our friends personally experience in our education.
### What we learned
This project was a deep dive into the world of real-time audio processing and AI. We learned how difficult it is to manage audio streams, buffering, and network latency. We also gained valuable experience integrating multiple complex AI services, learning the trade-offs between speed, quality, and cost for each. More than anything, we learned how to quickly prototype an idea and turn it into a working product that could have a real-world impact.
### What's next
for Prof AI Our immediate goal is to make the application production-ready and bring it to market. Following the launch, we plan to integrate our technology with popular video conferencing platforms and develop a physical microphone that can perform the same live translation in an in-person classroom. We will also expand beyond the education sector, adapting the tool for new use cases like live sales calls, corporate interviews, and virtual networking events.
## README (from the GitHub repository)
## ProfAI
### Inspiration
We've all been in university classes that have been taught by professors with thicker accents that can be hard to comprehend. Also, it gets hard to understand the content they're teaching, even though they're brilliant people from all around the world. So we built ProfAI to bridge the gap between the world's best educators and university students.
### What it does
This application provides real-time accent translation for the classroom. It captures a professor's audio and instantly streams it to students in a clear American accent, ensuring they can follow the lecture live without comprehension difficulties.
Our platform also enhances studying with powerful integrated tools. It automatically transcribes the professor's speech into detailed notes in real-time. Furthermore, it features a "rubber ducky mode," a smart voice agent that has context from your course materials on Canvas, YouTube, and Google Drive. You can talk to this agent while studying to get answers, explanations, or even have it perform web searches for you.
### How we built it
We built this project as a real-time audio-processing pipeline. Here’s a breakdown of the technology stack:
Our application is built with a Python Flask backend and a React frontend. The core real-time audio pipeline uses LiveKit to manage the STT (Speech-to-Text) and TTS (Text-to-Speech) workflow, enabling instant accent translation. The live transcription feature is powered by printing the direct output from the STT module. For the "rubber ducky" assistant, we built a conversational AI bot, also using LiveKit, and expanded its knowledge base using Llama to pull context from course materials for accurate, on-demand answers.
### Challenges we ran into
Our primary challenge was latency. Building a direct STT-to-TTS pipeline fast enough for a live lecture was difficult, creating race condition issues that required significant optimization. This was compounded by network variability (WiFi issues) that we could not completely solve.
We also faced an architectural hurdle in adapting our system from a default conversational chatbot model to an efficient, live transcription service. Finally, we had to find a TTS engine that balanced speed with voice naturalness to ensure the output sounded human and was suitable for long lectures.
### Accomplishments that we're proud of
We are incredibly proud that we built a *functional, end-to-end, real-time* pipeline. Hearing a voice go in with one accent and come out in another, almost instantly, was a huge "wow" moment for our team. We successfully built a tool that directly solves a problem we and our friends personally experience in our education.
### What we learned
This project was a deep dive into the world of real-time audio processing and AI. We learned how difficult it is to manage audio streams, buffering, and network latency. We also gained valuable experience integrating multiple complex AI services, learning the trade-offs between speed, quality, and cost for each. More than anything, we learned how to quickly prototype an idea and turn it into a working product that could have a real-world impact.
### What's next for `Prof AI`
Our immediate goal is to make the application production-ready and bring it to market.
Following the launch, we plan to integrate our technology with popular video conferencing platforms and develop a physical microphone that can perform the same live translation in an in-person classroom. We will also expand beyond the education sector, adapting the tool for new use cases like live sales calls, corporate interviews, and virtual networking events.
## Detected evidence (automated analysis)
Indexed codebase: 30 recognized source files, 71 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (46 of 46)
```
.DS_Store
backend/.dockerignore
backend/.gitignore
backend/.python-version
backend/agent_duck.py
backend/agent.py
backend/Dockerfile
backend/LICENSE
backend/livekit.toml
backend/module-2.md
backend/module-3.md
backend/module-4.md
backend/module-5.md
backend/module-6.md
backend/pyproject.toml
backend/README.md
backend/rubberDuck.py
backend/run_tutor.py
backend/token_server.py
backend/uv.lock
postcss.config.js
QUICK_START.md
README
RUNNING_GUIDE.md
tailwind.config.js
video/.env.video
video/frame_ocr.py
video/video_to_frames.py
VOICE_TUTOR_SETUP.md
zoom-clone/.gitignore
zoom-clone/eslint.config.js
zoom-clone/index.html
zoom-clone/package.json
zoom-clone/postcss.config.cjs
zoom-clone/postcss.config.js
zoom-clone/README.md
zoom-clone/src/App.css
zoom-clone/src/App.jsx
zoom-clone/src/index.css
zoom-clone/src/LiveKitIntegration.jsx
zoom-clone/src/main.jsx
zoom-clone/src/ProfAISidePanel.jsx
zoom-clone/tailwind.config.cjs
zoom-clone/tailwind.config.js
zoom-clone/TranscriptPlayer
zoom-clone/vite.config.js
```
### Dependencies
- backend/pyproject.toml: livekit-agents[deepgram,openai,silero]@~=1.2, livekit-plugins-noise-cancellation@~=0.2, python-dotenv@>=1.1.1
- zoom-clone/package.json: @eslint/js@^9.36.0, @tailwindcss/postcss@^4.1.16, @types/react@^19.1.16, @types/react-dom@^19.1.9, @vitejs/plugin-react@^5.0.4, autoprefixer@^10.4.21, eslint@^9.36.0, eslint-plugin-react-hooks@^5.2.0, eslint-plugin-react-refresh@^0.4.22, globals@^16.4.0, postcss@^8.5.6, react@^19.1.1, react-dom@^19.1.1, tailwindcss@^4.1.16, vite@^7.1.7
### Recent commits (newest first)
- test
- Readme
- Blurred out duck
- Merge pull request #6 from asrayg/duck-changes
- Added tts for duck
- Merge pull request #5 from asrayg/video
- ylooo
- Merge pull request #4 from asrayg/demo
- AI NOtes and Meeting Notes with Integration
- w4hjw
- One file
- video draft
- Merge branch 'stt-tts-pipeline'
- Got a working mvp
- Merge pull request #3 from asrayg/backend
- Merge pull request #2 from asrayg/Zoom
- All frontend UI stuff done!
- So far all the stuff except the duck
- Side Panel Added
- Basic setup
## Key source files (fetched from GitHub, selected and truncated for size)
### QUICK_START.md
```markdown
# 🚀 Quick Start
## ✅ Current Status
- ✅ Backend agent is running and connected to LiveKit Cloud!
- ⏳ Need to start token server
- ⏳ Need to start frontend
## Run These Commands:
### Terminal 1 (Already Running ✅)
```bash
cd backend
uv run python agent.py dev
```
**Status:** Connected to wss://profai-q7jis5n7.livekit.cloud
---
### Terminal 2: Start Token Server
```bash
cd backend
uv run python token_server.py
```
This will run on http://localhost:7860
---
### Terminal 3: Start Frontend
```bash
cd zoom-clone
npm run dev
```
Open browser to http://localhost:5173
---
## 🎯 How to Use
1. Open http://localhost:5173 in your browser
2. **Toggle ON** the switch in the ProfAI panel (right side)
3. The video audio will be sent to the agent
4. Agent transcribes and echoes it back
5. Watch transcript appear in real-time!
## 🔧 Your LiveKit Setup
- Using: **LiveKit Cloud**
- URL: `wss://profai-q7jis5n7.livekit.cloud`
- Region: US Central
That's it! 🎉
```
### RUNNING_GUIDE.md
```markdown
# Running Guide: Dual-Mode AI Assistant
Your app now has **TWO modes**:
## 🎥 Mode 1: Video Audio Transcription (Default)
- Video plays automatically
- `agent.py` listens to video audio
- Agent echoes back what it hears from the video
## 🦆 Mode 2: Voice Assistant (Duck Clicked)
- Video pauses
- `rubberDuck.py` listens to YOUR microphone
- Duck responds intelligently as an OCAML programming assistant
---
## Setup Instructions
### 1. Create `.env` file in `backend/` directory:
```bash
# LiveKit Cloud
LIVEKIT_URL=wss://profai-q7jis5n7.livekit.cloud
LIVEKIT_API_KEY=your_livekit_api_key
LIVEKIT_API_SECRET=your_livekit_api_secret
# OpenAI (for GPT-4.1-mini LLM and Whisper STT)
OPENAI_API_KEY=your_openai_api_key
# Cartesia (for TTS)
CARTESIA_API_KEY=your_cartesia_api_key
```
### 2. Get API Keys:
- **LiveKit**: https://cloud.livekit.io/projects/profai-q7jis5n7/settings/keys
- **OpenAI**: https://platform.openai.com/api-keys (used for both LLM and Whisper STT)
- **Cartesia**: https://play.cartesia.ai/
---
## Running the App
### Option A: Test Video Transcription Mode
**Terminal 1** - Token Server:
```bash
cd backend
uv run python token_server.py
```
**Terminal 2** - Video Agent:
```bash
cd backend
uv run python agent.py dev
```
**Terminal 3** - Frontend:
```bash
cd zoom-clone
npm run dev
```
**Test**: Open http://localhost:5173 and the agent will echo what it hears from the video.
---
### Option B: Test Voice Assistant Mode (Talk to Duck)
**Terminal 1** - Token Server:
```bash
cd backend
uv run python token_server.py
```
**Terminal 2** - RubberDuck Agent:
```bash
cd backend
uv run python rubberDuck.py dev
```
**Terminal 3** - Frontend:
```bash
cd zoom-clone
npm run dev
```
**Test**:
1. Open http://localhost:5173
2. Click the duck 🦆
3. Video pauses and duck says "I'm listening! Ask me anything..."
4. Allow microphone access
5. Talk to the duck and it will respond!
---
## How It Works
### Video Mode (Duck OFF):
```
Video Audio → LiveKit Room "video-transcription" → agent.py → Echoes back
```
### Voice Mode (Duck ON):
```
Your Microphone → LiveKit Room "voice-assistant" → rubberDuck.py → Intelligent Response
```
The frontend automatically:
- Connects to `"video-transcription"` room when duck is OFF
- Connects to `"voice-assistant"` room when duck is ON
- Pauses/plays video accordingly
- Switches between video audio capture and microphone capture
---
## Troubleshooting
### "No module named 'livekit'"
Run: `cd backend && uv sync`
### "Failed to connect to LiveKit"
1. Check token server is running on port 7860
2. Verify your `.env` file has correct LiveKit credentials
### "Agent not responding"
Make sure you're running the RIGHT agent for the mode:
- Video mode needs `agent.py`
- Duck mode needs `rubberDuck.py`
### "Microphone not working"
1. Click the duck to activate microphone mode
2. Browser will ask for microphone permission - click Allow
3. Check browser console for errors
### Both modes at once?
If you want BOTH mode
[truncated — 827 more characters]
```
### backend/pyproject.toml
```
[project]
name = "voice-agent-workshop"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
"livekit-agents[deepgram,openai,silero]~=1.2",
"livekit-plugins-noise-cancellation~=0.2",
"python-dotenv>=1.1.1",
]
```
### zoom-clone/package.json
```
{
"name": "zoom-clone",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"lint": "eslint .",
"preview": "vite preview"
},
"dependencies": {
"react": "^19.1.1",
"react-dom": "^19.1.1"
},
"devDependencies": {
"@eslint/js": "^9.36.0",
"@tailwindcss/postcss": "^4.1.16",
"@types/react": "^19.1.16",
"@types/react-dom": "^19.1.9",
"@vitejs/plugin-react": "^5.0.4",
"autoprefixer": "^10.4.21",
"eslint": "^9.36.0",
"eslint-plugin-react-hooks": "^5.2.0",
"eslint-plugin-react-refresh": "^0.4.22",
"globals": "^16.4.0",
"postcss": "^8.5.6",
"tailwindcss": "^4.1.16",
"vite": "^7.1.7"
}
}
```
### backend/Dockerfile
```
# This is an example Dockerfile that builds a minimal container for running LK Agents
# For more information on the build process, see https://docs.livekit.io/agents/ops/deployment/builds/
# syntax=docker/dockerfile:1
# Use the official UV Python base image with Python 3.13 on Debian Bookworm
# UV is a fast Python package manager that provides better performance than pip
# We use the slim variant to keep the image size smaller while still having essential tools
ARG PYTHON_VERSION=3.13
FROM ghcr.io/astral-sh/uv:python${PYTHON_VERSION}-bookworm-slim AS base
# Keeps Python from buffering stdout and stderr to avoid situations where
# the application crashes without emitting any logs due to buffering.
ENV PYTHONUNBUFFERED=1
# Create a non-privileged user that the app will run under.
# See https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#user
ARG UID=10001
RUN adduser \
--disabled-password \
--gecos "" \
--home "/app" \
--shell "/sbin/nologin" \
--uid "${UID}" \
appuser
# Install build dependencies required for Python packages with native extensions
# gcc: C compiler needed for building Python packages with C extensions
# g++: C++ compiler needed for building Python packages with C++ extensions
# python3-dev: Python development headers needed for compilation
# We clean up the apt cache after installation to keep the image size down
RUN apt-get update && apt-get install -y \
gcc \
g++ \
python3-dev \
&& rm -rf /var/lib/apt/lists/*
# Create a new directory for our application code
# And set it as the working directory
WORKDIR /app
# Copy just the dependency files first, for more efficient layer caching
COPY pyproject.toml uv.lock ./
RUN mkdir -p src
# Install Python dependencies using UV's lock file
# --locked ensures we use exact versions from uv.lock for reproducible builds
# This creates a virtual environment and installs all dependencies
# Ensure your uv.lock file is checked in for consistency across environments
RUN uv sync --locked
# Copy all remaining pplication files into the container
# This includes source code, configuration files, and dependency specifications
# (Excludes files specified in .dockerignore)
COPY . .
# Change ownership of all app files to the non-privileged user
# This ensures the application can read/write files as needed
RUN chown -R appuser:appuser /app
# Switch to the non-privileged user for all subsequent operations
# This improves security by not running as root
USER appuser
# Pre-download any ML models or files the agent needs
# This ensures the container is ready to run immediately without downloading
# dependencies at runtime, which improves startup time and reliability
RUN uv run "agent.py" download-files
# Run the application using UV
# UV will activate the virtual environment and run the agent.
# The "start" command tells the worker to connect to LiveKit and begin waiting for jobs.
CMD ["uv", "run", "agent.py", "start"]
```
### zoom-clone/src/main.jsx
```javascript
import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client'
import './index.css'
import App from './App.jsx'
createRoot(document.getElementById('root')).render(
,
)
```
### zoom-clone/src/App.jsx
```javascript
import meetingScreenshot from "./assets/zoom.png";
import ProfAISidePanel from "./ProfAISidePanel";
import duckImage from "./assets/ducks.png";
import React, { useState, useRef } from "react";
import coolVideo from "./assets/Untitled design (1).mp4";
export default function App() {
const [showDuckChat, setShowDuckChat] = useState(false);
const videoRef = useRef(null);
const speechRef = useRef(null);
const speechTimeoutRef = useRef(null);
const duckText = `A rule tree proof, also known as a derivation tree, is a visual format for proving that a statement is true`;
const handleDuckClick = () => {
const newDuckState = !showDuckChat;
setShowDuckChat(newDuckState);
if (newDuckState) {
// 🔇 Pause and mute video
if (videoRef.current) {
videoRef.current.pause();
videoRef.current.muted = true;
}
if (speechRef.current) window.speechSynthesis.cancel();
if (speechTimeoutRef.current) clearTimeout(speechTimeoutRef.current);
// Start duck speaking after 4 sec
speechTimeoutRef.current = setTimeout(() => {
if ("speechSynthesis" in window) {
const utterance = new SpeechSynthesisUtterance(duckText);
utterance.rate = 0.9;
utterance.pitch = 1.0;
utterance.volume = 1.0;
utterance.onend = () => console.log("🦆 Duck finished speaking!");
speechRef.current = utterance;
window.speechSynthesis.speak(utterance);
}
}, 4000);
} else {
// 🔊 Resume and unmute video
if (videoRef.current) {
videoRef.current.play();
videoRef.current.muted = false;
}
if (speechTimeoutRef.current) clearTimeout(speechTimeoutRef.current);
if (window.speechSynthesis) window.speechSynthesis.cancel();
}
};
return (
{/* Zoom-like Background */}
{/* 🌫️ Blur overlay when Duck is active */}
{showDuckChat && (
)}
{/* Main video */}
{/* ProfAI Panel */}
