# Project export: TA-DA: Intelligent Teaching Assistant This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Why are students Googling mid-lecture? TA-DA keeps them in the room: AI teaching assistant built into Zoom that detects confusion and generates concept cards instantly. No context loss. Just learning. - Devpost: https://devpost.com/software/ta-da-intelligent-teaching-assistant - GitHub: https://github.com/pb2323/TA-da - Demo: https://ta-da-frontend.onrender.com/ - Video: https://www.youtube.com/embed/GuiFFxdgUZA?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner ([Zoom] Education Track) - Team: 3 GitHub contributor(s) — reet-571099 (12 commits), pb2323 (6 commits), Cursor (6 commits) ## Devpost submission (written by the team) ### Inspiration We've all been in that Zoom lecture where the instructor asks "Any questions?" and silence follows. Not because everyone understands—but because no one wants to interrupt. Students tab out to Google concepts mid-lecture, losing context. Instructors teach blind, with no feedback loop until the exam reveals the damage. The problem isn't recording or transcription. Tools like Otter.ai capture words. But they don't close the learning loop. TA-DA was born from a simple question: What if the classroom could sense confusion in real time and intervene—without breaking the flow? We wanted to build a teaching assistant that lives where learning happens: inside Zoom, always listening, always ready, never interrupting unless you need it. ### What it does TA-DA is an AI-powered teaching assistant built natively into Zoom using the Real-Time Meeting SDK (RTMS). It runs as an in-meeting side panel—no browser tabs, no context switching. Core Features: Auto-Generated Concept Cards: As the instructor teaches, TA-DA autonomously detects topic boundaries and generates concept cards with explanations and examples—appearing in real time on the Learning Timeline. Auto-Generated Concept Cards: As the instructor teaches, TA-DA autonomously detects topic boundaries and generates concept cards with explanations and examples—appearing in real time on the Learning Timeline. Intelligent Q&A: Students ask questions directly in the side panel. TA-DA doesn't just answer—it retrieves relevant transcript context, explains the concept, and provides examples. Intelligent Q&A: Students ask questions directly in the side panel. TA-DA doesn't just answer—it retrieves relevant transcript context, explains the concept, and provides examples. Live Learning Analytics: Instructors see which concepts students marked as "Lost" or "Understood," enabling instant intervention. Live Learning Analytics: Instructors see which concepts students marked as "Lost" or "Understood," enabling instant intervention. Catch-Up Summaries: Joined late? Ask TA-DA for a summary and get a structured recap of everything covered so far. Catch-Up Summaries: Joined late? Ask TA-DA for a summary and get a structured recap of everything covered so far. No Tab Switching: Everything happens inside Zoom. No Googling mid-lecture. No context loss. Just seamless learning. No Tab Switching: Everything happens inside Zoom. No Googling mid-lecture. No context loss. Just seamless learning. ### How we built it We designed TA-DA as a real-time, multi-agent system with four key layers: 1️⃣ Zoom RTMS + SDK (Frontend) Built a native Zoom app using the Real-Time Meeting SDK for in-meeting side panels Integrated RTMS pipeline to stream live transcripts and meeting events Created a responsive UI with Learning Timeline, Q&A interface, and analytics dashboard Designed for Zoom Marketplace deployment—one-click install for educators 2️⃣ Node.js Backend (Orchestration) Express.js API to handle transcript ingestion, agent communication, and search queries WebSocket events for real-time updates between Zoom, Elastic, and Fetch.ai agents Proxy layer for concept card retrieval and agent conversation endpoints 3️⃣ Elasticsearch (Memory Layer) Two indices: ta-da-latest (transcript chunks) and ta-da-concept-cards (structured concepts) Semantic search with vector embeddings for intelligent retrieval Filters by meeting ID, timestamps, and concept tags The single source of truth for everything we know about the lecture 4️⃣ Fetch.ai Agents (Intelligence Layer) Concept Card Agent: Runs on Fetch.ai's Agentverse platform Monitors new transcript chunks via Redis state tracking Detects topic boundaries using LLM-powered segmentation Generates concept cards (title, short explanation, example) and indexes them to Elastic Autonomous 60-second intervals with pause/resume controls Concept Card Agent: Runs on Fetch.ai's Agentverse platform Monitors new transcript chunks via Redis state tracking Detects topic boundaries using LLM-powered segmentation Generates concept cards (title, short explanation, example) and indexes them to Elastic Autonomous 60-second intervals with pause/resume controls Agent Tools: query_elastic_chunks() – Retrieve transcript context create_concept_card() – Generate and store concept cards call_backend_agent_converse() – Provide summaries and answers Agent Tools: query_elastic_chunks() – Retrieve transcript context create_concept_card() – Generate and store concept cards call_backend_agent_converse() – Provide summaries and answers LLM Integration: OpenAI GPT-4 for concept extraction, summarization, and Q&A LLM Integration: OpenAI GPT-4 for concept extraction, summarization, and Q&A 5️⃣ Tech Stack Frontend: Zoom RTMS SDK, JavaScript, CSS Backend: Node.js, Express.js, WebSockets Memory: Elasticsearch (with semantic search) Agents: Fetch.ai Agentverse, Python LLM: OpenAI GPT-4 / xAI State: Redis (incremental processing, pause/resume) Deployment: Docker, cloud-ready architecture ### Challenges we ran into 1. Real-Time Agent Coordination Running autonomous agents while avoiding duplicate processing was tricky. We solved it with Redis-based state tracking: each agent stores the last processed chunk index per meeting and only processes new chunks incrementally. 2. Zoom RTMS Learning Curve The Real-Time Meeting SDK documentation was sparse. We spent hours debugging WebSocket connections, transcript streaming delays, and side panel rendering. Eventually, we built a robust event-driven architecture that handles live data gracefully. 3. Live Avatar Instability We built a TA avatar assistant with lip-sync and custom-configured voice for a human-like learning experience. The feature works but isn't production-stable yet—lip-sync timing drifts, and rendering performance degrades in low-bandwidth scenarios. It's one of our most exciting features, but needs optimization. 4. Concept Boundary Detection Determining when one "concept" ends and another begins is subjective. We iterated on LLM prompts, testing different segmentation strategies (sliding windows, topic modeling, explicit markers) before landing on a hybrid approach: fixed chunk windows + LLM-based topic extraction. 5. Elastic Search Latency Initially, our searches were slow because we were doing full-text queries on large transcript indices. We optimized by: Adding semantic vector search with embeddings Filtering by meeting ID and timestamps Pre-indexing concept cards separately from raw transcripts 6. Fetch.ai Agent Debugging Debugging agents running on Agentverse was non-trivial—logs were async, and testing required deploying to the cloud. We added verbose logging, local Redis mocking, and a manual trigger endpoint to iterate faster. ### Accomplishments we're proud of Built a fully functional AI teaching assistant in 36 hours—from idea to live demo Seamless Zoom integration with native side panel—no browser tabs required Autonomous agents that generate concept cards without human intervention Real-time learning analytics that give instructors actionable feedback Scalable architecture ready for thousands of concurrent classrooms Closed the learning loop: sense confusion → intervene → verify understanding We didn't just build a transcript tool. We built a learning intelligence system that actively improves education outcomes. ### What's next for TA-DA Short-term: Stabilize Live Avatar: Fix lip-sync timing, optimize rendering, and make the TA avatar production-ready for a truly immersive learning experience Voice-based Q&A: Students speak questions instead of typing Multi-language support: Real-time translation for global classrooms Instructor dashboards: Advanced analytics—confusion heatmaps, engagement scores, concept mastery Pre-class prep mode: Upload slides/notes before class; TA-DA cross-references during lecture Long-term: TA-DA as Substitute Teacher: Train TA-DA on a professor's teaching style, course materials, and past lectures—if the professor can't join, TA-DA delivers the lecture autonomously with the live avatar Adaptive learning paths: TA-DA generates personalized review sessions based on which concepts each student struggled with Cross-meeting memory: Agents remember past lectures, building a knowledge graph across an entire course Federated agents: Specialized agents for different subjects (Math Tutor, Code Debugger, Writing Coach) Zoom Marketplace launch: One-click install for universities, bootcamps, and corporate training The vision? Every classroom gets a teaching assistant that never sleeps, never judges, and scales infinitely. Education shouldn't be a guessing game. With TA-DA, it won't be. TA-DA: Teaching Assistant Done Autonomously The closed-loop learning system built for Zoom. ## README (from the GitHub repository) # TA-DA! **A Live Zoom Teaching Assistant that Detects Confusion, Fixes It in the Moment, and Verifies Understanding** Built at TreeHacks 2026. --- ## What is TA-DA? TA-DA! is a real-time AI Teaching Assistant built directly inside Zoom. It transforms passive lectures into an active learning loop: **Sense confusion → Intervene instantly → Verify understanding → Generate personalized next steps** Unlike traditional meeting tools that just transcribe or summarize, TA-DA actively changes what happens during class and closes the learning loop afterward. --- ## The Problem Live classes fail for predictable reasons: - Instructors don't know what isn't landing - Questions don't scale (duplicates, vague, shy students) - No verification of understanding - Students leave without clear next steps Most tools are passive. **TA-DA is active.** --- ## What It Does | Feature | Description | |--------|-------------| | **Live Concept Cards** | Auto-generated during lecture: concept name, short explanation, example, timestamp, and one quick check question. Builds a live "course memory." | | **Real-Time Confusion Heatmap** | Students signal Lost / Kinda / Got it / Question. TA-DA clusters signals, surfaces top confusing concepts, duplicate questions, and suggested interventions. | | **Instructor Intervention Engine** | On confusion spike, instructor can generate a 30-second re-explain (simpler analogy, reframed explanation, quick poll). Optional HeyGen Avatar reads it live. | | **Multi-Turn Diagnostic Tutor** | Agentic core: clarifying questions, targeted explanations, check questions, adaptation, and student learning profile updates. | | **Learning Contract (After Class)** | Per student: what they struggled with, what they nailed, 2 micro-practice actions (≤10 min), 3-question verification. Instructor gets concept-by-concept understanding %, top misconceptions, and suggested tweaks. | --- ## Architecture | Layer | Tech | Location | |-------|------|----------| | **Frontend** | Zoom App (in-meeting side panel), Next.js on Vercel | [frontend](frontend/) | | **Backend** | Render (API, WebSocket, background worker) | [backend](backend/) | | **Memory & retrieval** | Elastic Cloud, Jina embeddings, hybrid retrieval | [elastic](elastic/) | | **Multi-agent** | Fetch.ai (Moments, Confusion, Tutor, Contract agents) | [fetch-ai](fetch-ai/) | | **Research** | Perplexity Sonar API (sources, misconception detection) | [perplexity](perplexity/) | | **Avatar (optional)** | HeyGen Avatar API (re-explain mode) | [heygen](heygen/) | --- ## Sponsor Alignment TA-DA is designed to be award-eligible across: - **Zoom × Render** — In-meeting app + real-time backend - **Elastic** — End-to-end agentic system on Elasticsearch - **Fetch.ai** — Multi-agent workflow + monetization-ready - **HeyGen** — Real-time avatar integration - **Vercel** — Production-ready deployed app - **Perplexity Sonar** — Grounded research + citations --- ## Getting Started ### 1. Clone the repo ```bash git clone https://github.com/pb2323/TA-da cd TA-da ``` ### 2. Install dependencies ```bash npm install ``` ### 3. Environment variables Create `.env.local` with: ```env ZOOM_CLIENT_ID= ZOOM_CLIENT_SECRET= ELASTICSEARCH_URL= ELASTIC_API_KEY= RENDER_BACKEND_URL= SONAR_API_KEY= HEYGEN_API_KEY= FETCH_AGENTVERSE_KEY= ``` ### 4. Run locally ```bash npm run dev ``` ### 5. Deploy - **Frontend:** Deploy to Vercel - **Backend:** Deploy API + worker to Render - **Elastic:** Provision Elastic Cloud cluster; configure Jina embeddings inference endpoint --- ## Demo Script (2-minute flow) 1. Start Zoom mini-lecture 2. Concept cards appear live 3. Two students hit "Lost" → instructor dashboard updates 4. Click "Generate 30-sec re-explain" 5. Student asks question → TA-DA runs multi-turn diagnostic 6. End class → show Learning Contract --- ## Vision TA-DA turns Zoom from a video platform into a **Learning Operations System**: not just notes or summaries, but a closed-loop intelligence layer for real-time education. --- ## Repository structure - [frontend](frontend/) — Next.js app (Vercel) + Zoom in-meeting UI - [backend](backend/) — Render API, WebSocket server, workers - [elastic](elastic/) — Elastic Cloud, search, concept cards, embeddings - [fetch-ai](fetch-ai/) — Multi-agent layer (Agentverse) - [heygen](heygen/) — HeyGen avatar integration - [perplexity](perplexity/) — Perplexity Sonar research & verification - [docs](docs/) — Architecture, demo script, and design notes --- **Team** — Built at TreeHacks 2026. 36 hours. One mission: make learning adaptive in real time. ## Detected evidence (automated analysis) Indexed codebase: 43 recognized source files, 769 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - Redis (technology) — detected in the code - TypeScript (language) — detected in the code - LangChain (technology) — claimed on Devpost, not found in the code - AI coding agent: Cursor — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (65 of 65) ``` .DS_Store .gitignore backend/package.json backend/README.md backend/src/index.js docs/README.md elastic/INDEXING.md elastic/indices/README.md elastic/indices/ta-da-concept-card-agent-state.mapping.json elastic/indices/ta-da-concept-cards.mapping.json elastic/indices/ta-da-latest.mapping.json elastic/indices/ta-da-session-state.mapping.json elastic/indices/ta-da-sessions.mapping.json elastic/indices/ta-da-signals.mapping.json elastic/indices/ta-da-tutor-turns.mapping.json elastic/indices/ta-da-users.mapping.json elastic/mcp-config.example.json elastic/README.md elastic/scripts/converse-agent.sh elastic/scripts/create-agent-indices.py elastic/scripts/delete-all-docs.py elastic/scripts/index-test-chunk.py elastic/scripts/semantic-search.py fetch-ai/concept_card_agent_README.md fetch-ai/concept_card_agent.py fetch-ai/config.py fetch-ai/elastic_client.py fetch-ai/GIGMART_REFERENCE.md fetch-ai/llm_client.py fetch-ai/README.md fetch-ai/redis_client.py fetch-ai/requirements.txt frontend/README.md heygen/README.md perplexity/README.md README.md requirements-elastic.txt zoom-RTMS-SDK/.github/CODEOWNERS zoom-RTMS-SDK/.github/workflows/lint.yml zoom-RTMS-SDK/.gitignore zoom-RTMS-SDK/.prettierrc.json zoom-RTMS-SDK/CONTRIBUTING.md zoom-RTMS-SDK/create_index.js zoom-RTMS-SDK/eslint.config.mjs zoom-RTMS-SDK/index.ts zoom-RTMS-SDK/LICENSE zoom-RTMS-SDK/package.json zoom-RTMS-SDK/public/css/styles.css zoom-RTMS-SDK/public/index.html zoom-RTMS-SDK/public/instructor.html zoom-RTMS-SDK/public/js/app.js zoom-RTMS-SDK/public/js/instructor.js zoom-RTMS-SDK/public/js/logger.js zoom-RTMS-SDK/public/livekit-client.umd.js zoom-RTMS-SDK/README.md zoom-RTMS-SDK/screenshots/.gitkeep zoom-RTMS-SDK/src/inworld/evaluationGraph.ts zoom-RTMS-SDK/src/inworld/guidanceGraph.ts zoom-RTMS-SDK/src/inworld/inworldService.ts zoom-RTMS-SDK/src/inworld/visualEvalGraph.ts zoom-RTMS-SDK/src/rtms/websocketHandler.ts zoom-RTMS-SDK/src/types/rtms.d.ts zoom-RTMS-SDK/src/utils/applyHeaders.ts zoom-RTMS-SDK/src/utils/logging.ts zoom-RTMS-SDK/tsconfig.json ``` ### Dependencies - backend/package.json: @elastic/elasticsearch@^8.15.0, dotenv@^17.3.1, express@^4.21.0 - fetch-ai/requirements.txt: elasticsearch@>=8.0.0, openai@>=1.0.0, pydantic@>=2.5.0, python-dotenv@>=1.2.0, redis@>=5.0.0, requests@>=2.32.0, uagents@>=0.22.0, uagents-core@>=0.2.0 - zoom-RTMS-SDK/package.json: @elastic/elasticsearch@^8.11.0, @eslint/js@^9.0.0, @inworld/runtime@^0.8.0, @types/express@^4.17.21, @types/node@^20.0.0, @types/ws@^8.5.12, @zoom/rtms@^0.0.2, dotenv@^16.0.3, eslint@^9.0.0, eslint-config-prettier@^9.1.0, eslint-plugin-prettier@^5.1.3, express@^4.21.2, livekit-client@^2.17.1, nodemon@^3.1.10, prettier@^3.2.5, sharp@^0.34.4, tsx@^4.7.0, typescript@^5.3.3, typescript-eslint@^8.0.0, ws@^8.16.0 ### Recent commits (newest first) - avatar pause - avatar working - Merge branch 'main' of https://github.com/pb2323/TA-da - instructor dashboard - Backend GET /concept-cards; Learning Timeline from API, add-only-new - Concept card agent: Redis incremental processing, auto-run, summary, formatting - Update elastic indexing, concept card agent, and fetch-ai config/clients - small fix for prod - fix - Format agent responses and add animated loading dots - Clean up README.md by removing content - Add Ask Question chatbot, backend proxy, fetch-ai agent, and elastic mappings - not needed - Tabs instead of scrolling - Merge branch 'main' of https://github.com/pb2323/TA-da - better UI - Add backend API (semantic search, agent converse), elastic scripts, and indices - zoom sdk app enabled - chunks sent to elastic search - Elastic indexing: Python only, ELASTICSEARCH_URL + ELASTIC_API_KEY (no Cloud ID) ## Key source files (fetched from GitHub, selected and truncated for size) ### zoom-RTMS-SDK/CONTRIBUTING.md ```markdown # Contributing to Zoom Demeanor Evaluator Thank you for your interest in contributing to the Zoom Demeanor Evaluator! This document provides guidelines and instructions for contributing. ## Getting Started ### Prerequisites - Node.js (v20 or higher) - npm (v9 or higher) - Git - An Inworld AI account and API key - A Zoom App with RTMS access - Ngrok (for local development) ### Development Setup 1. **Fork the repository** on GitHub 2. **Clone your fork**: ```bash git clone https://github.com/YOUR_USERNAME/zoom-demeanor-evaluator-node.git cd zoom-demeanor-evaluator-node ``` 3. **Install dependencies**: ```bash npm install ``` 4. **Set up environment variables**: Create a `.env` file in the root directory: ```bash INWORLD_API_KEY=your_api_key_here ZM_RTMS_CLIENT=your_zoom_client_id ZM_RTMS_SECRET=your_zoom_client_secret ``` 5. **Verify the setup**: ```bash npm run build npm run lint npm run format:check ``` ## Development Workflow ### Making Changes 1. **Create a feature branch**: ```bash git checkout -b feature/your-feature-name # or git checkout -b fix/your-bug-fix ``` 2. **Make your changes** and test them locally 3. **Run code quality checks** before committing: ```bash npm run lint # Check for linting errors npm run lint:fix # Auto-fix linting issues npm run format # Format code with Prettier npm run format:check # Verify formatting npm run type-check # Check TypeScript types npm run build # Ensure code compiles ``` 4. **Commit your changes**: ```bash git add . git commit -m "Your descriptive commit message" ``` Write clear, descriptive commit messages that explain what and why you changed something. ## Code Style ### TypeScript/JavaScript - Use TypeScript strict mode for new TypeScript files - Provide explicit types for function parameters and return values - Avoid `any` types - use `unknown` or proper types instead - Follow the existing code style and patterns ### Formatting - Code is automatically formatted with Prettier - Run `npm run format` before committing - Maximum line length: 80 characters - Use single quotes for strings - Use semicolons ### Linting - ESLint is configured with TypeScript and JavaScript support - All linting errors must be resolved before submitting a PR - Run `npm run lint:fix` to auto-fix issues where possible ### File Structure ``` src/ ├── inworld/ # Inworld AI integration ├── rtms/ # Zoom RTMS integration └── utils/ # Helper utilities ``` ## Pull Request Process 1. **Update your fork**: ```bash git checkout main git pull upstream main git push origin main ``` 2. **Create your PR**: - Push your branch to your fork - Open a Pull Request on GitHub - Fill out the PR template (if available) - Link any related issues 3. **PR Requirements**: - All tests pass (if applicable) - Code f [truncated — 1001 more characters] ``` ### fetch-ai/concept_card_agent_README.md ```markdown # TA-DA Concept Card Agent ## Overview An autonomous agent that detects when lecture concepts are completed and generates concept cards from live transcripts. It uses the Fetch.ai uAgents framework with the Chat Protocol and integrates with Elasticsearch and an LLM (xAI Grok or OpenAI) for topic-boundary detection and concept extraction. ## Purpose Acts as the "Concept Card Agent" for TA-DA: it builds a live "course memory" by turning completed lecture segments into structured concept cards (title, short explanation, example, timestamp) that can be shown in the Zoom app and used by other agents (tutor, confusion heatmap, learning contract). ## Responsibilities ### Core Capabilities - **Topic completion detection** — Polls transcript chunks from `ta-da-latest`, compares current vs previous conversation state, and identifies when a new concept has started (topic boundary). When a boundary is detected, the previous segment is treated as a completed concept. - **Concept card creation** — For each completed segment, uses an LLM to extract a title, short explanation, and example, then indexes a document into `ta-da-concept-cards`. ### Tools | Tool | Purpose | |------|---------| | **detect_concept_completion** | Runs on a schedule; fetches recent chunks, loads prior state, calls LLM to detect topic boundaries; invokes create_concept_card when a boundary is found. | | **create_concept_card** | Takes a completed segment (chunk range), runs LLM extraction, and indexes to `ta-da-concept-cards`. | ## Input - **Transcript chunks** — From `ta-da-latest` (populated by the existing transcript indexing pipeline). Chunks have `meeting_id`, `chunk_index`, `text`, `start_time`, `end_time`. - **Meetings** — From `ta-da-sessions` (`meeting_id`). The agent processes all meetings and fetches chunks per meeting. - **Chat messages** — Via Chat Protocol: users can send natural language; the agent replies with a short description of its role. ## Output - **Concept cards** — Documents in `ta-da-concept-cards`: `meeting_id`, `concept_id`, `title`, `short_explain`, `example`, `timestamp`. - **Agent state** — Per-session state in `ta-da-concept-card-agent-state` (last chunk index, last topic summary) for the next comparison. ## Example Interactions (Chat) - "What do you do?" → Agent describes its role (concept detection and card creation). - "Run detection" / "Create a card" — Can be extended to trigger tools from chat. ## Technical Details ### Tech Stack - **uAgents Framework** — Fetch.ai agent runtime - **Chat Protocol** — ASI-1 compatible messaging (ChatMessage, ChatAcknowledgement) - **Elasticsearch** — Transcript chunks, sessions, concept cards, agent state - **LLM** — xAI Grok (default) or OpenAI for topic comparison and concept extraction - **Python** — Core implementation ### Deployment - **Transport** — HTTP server on configurable port (default 8010); optional mailbox for Agentverse - **Identity** — Seed phrase from `CONCEPT_CARD_AGENT_SEED` (or pla [truncated — 1642 more characters] ``` ### fetch-ai/requirements.txt ``` uagents>=0.22.0 uagents-core>=0.2.0 openai>=1.0.0 pydantic>=2.5.0 python-dotenv>=1.2.0 requests>=2.32.0 elasticsearch>=8.0.0 redis>=5.0.0 ``` ### backend/package.json ``` { "name": "ta-da-backend", "version": "1.0.0", "description": "TA-DA event router and API (Render)", "main": "src/index.js", "type": "module", "scripts": { "start": "node src/index.js", "dev": "node --watch src/index.js" }, "dependencies": { "@elastic/elasticsearch": "^8.15.0", "dotenv": "^17.3.1", "express": "^4.21.0" } } ``` ### zoom-RTMS-SDK/package.json ``` { "name": "zoom-demeanor-evaluator", "version": "1.0.0", "description": "Zoom App powered by Inworld AI Runtime for real-time communication evaluation and guidance", "main": "dist/index.js", "type": "module", "scripts": { "build": "tsc && cp -r public dist/", "start": "node dist/index.js", "dev": "nodemon --exec \"tsx index.ts\" --ext ts", "lint": "eslint .", "lint:fix": "eslint . --fix", "format": "prettier --write \"**/*.{ts,tsx,js,jsx,json,md}\"", "format:check": "prettier --check \"**/*.{ts,tsx,js,jsx,json,md}\"", "type-check": "tsc --noEmit || echo 'No TypeScript files to check'" }, "keywords": [ "inworld", "ai", "zoom", "demeanor-evaluator" ], "author": "Inworld AI", "license": "MIT", "dependencies": { "@elastic/elasticsearch": "^8.11.0", "@inworld/runtime": "^0.8.0", "@zoom/rtms": "^0.0.2", "dotenv": "^16.0.3", "express": "^4.21.2", "livekit-client": "^2.17.1", "sharp": "^0.34.4", "ws": "^8.16.0" }, "devDependencies": { "@eslint/js": "^9.0.0", "@types/express": "^4.17.21", "@types/node": "^20.0.0", "@types/ws": "^8.5.12", "eslint": "^9.0.0", "eslint-config-prettier": "^9.1.0", "eslint-plugin-prettier": "^5.1.3", "nodemon": "^3.1.10", "prettier": "^3.2.5", "tsx": "^4.7.0", "typescript": "^5.3.3", "typescript-eslint": "^8.0.0" }, "engines": { "node": ">=20.0.0", "npm": ">=9.0.0" } } ``` ### zoom-RTMS-SDK/index.ts ```typescript import express, { type Request, type Response } from 'express'; import path from 'path'; import { fileURLToPath } from 'url'; import { createServer, type Server } from 'http'; import dotenv from 'dotenv'; import WebSocketHandler from './src/rtms/websocketHandler.js'; import applyHeaders from './src/utils/applyHeaders.js'; import { Logger } from './src/utils/logging.js'; const logger = new Logger('Server'); dotenv.config(); interface RTMSConfig { clientId: string; clientSecret: string; } const RTMS_CONFIG: RTMSConfig = { clientId: process.env.ZM_RTMS_CLIENT || '', clientSecret: process.env.ZM_RTMS_SECRET || '', }; // Get the directory name for ES modules const __filename = fileURLToPath(import.meta.url); const __dirname = path.dirname(__filename); // Create Express app const app = express(); const PORT = process.env.PORT || 3000; // Parse JSON bodies for webhooks app.use(express.json()); app.use(express.urlencoded({ extended: true })); // Apply OWASP security headers to all requests app.use((_req, res, next) => { applyHeaders(res); next(); }); // Serve static files from public folder app.use(express.static(path.join(__dirname, 'public'))); // Root route - serve the HTML page app.get('/', (_req: Request, res: Response) => { logger.debug('Serving HTML page for GET /'); res.sendFile(path.join(__dirname, 'public', 'index.html')); }); // Create HTTP server const server: Server = createServer(app); // Initialize WebSocket handler with server and RTMS config const wsHandler = new WebSocketHandler(server, RTMS_CONFIG); // Webhook endpoint for RTMS events (CRITICAL for Zoom app to work) app.post('/webhook', (req: Request, res: Response) => { logger.debug('Received webhook POST to /webhook'); const webhookData = req.body; wsHandler.handleWebhookEvent(webhookData); res.status(200).send('OK'); }); // Proxy to TA-DA backend agent converse API (avoids CORS, centralizes backend URL) const TA_DA_BACKEND_URL = process.env.TA_DA_BACKEND_URL || 'http://localhost:3000'; app.post('/api/agent/converse', async (req: Request, res: Response) => { try { const r = await fetch(`${TA_DA_BACKEND_URL}/agent/converse`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(req.body || {}), }); const data = await r.json().catch(() => ({})); res.status(r.status).json(data); } catch (err) { const msg = err instanceof Error ? err.message : 'converse proxy failed'; logger.error('Agent converse proxy error:', msg); res.status(502).json({ error: msg }); } }); // Proxy to TA-DA backend LiveAvatar token endpoint app.post('/api/liveavatar/token', async (req: Request, res: Response) => { try { const r = await fetch(`${TA_DA_BACKEND_URL}/api/liveavatar/token`, { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(req.body || {}), }); const data = await r.json().catch(() => ({})); res.status(r.status).json(data); } catch (err) { const msg = err instanceof Error ? err.message : 'liveavatar token proxy failed'; logger.error('LiveAvatar token proxy error:', msg); res.status(502).json({ error: msg }); } }); // Proxy to LiveAvatar sessions/start endpoint (avoids CSP issues) app.post('/api/liveavatar/session/start', async (req: Request, res: Response) => { try { const { session_token } = req.body || {}; if (!session_token) { return res.status(400).json({ error: 'session_token required' }); } const r = await fetch('https://api.liveavatar.com/v1/sessions/start', { method: 'POST', headers: { Authorization: `Bearer ${session_token}`, 'Content-Type': 'application/json', Accept: 'application/json', }, }); const data = await r.json().catch(() => ({})); res.status(r.status).json(data); } catch (err) { const msg = err instanceof Error ? err.message : 'liveavatar session start proxy failed'; logger.error('LiveAvatar session start proxy error:', msg); res.status(502).json({ error: msg }); } }); // Proxy to TA-DA backend concept cards (ta-da-concept-cards index) app.get('/api/concept-cards', async (req: Request, res: Response) => { try { const meetingId = req.query.meeting_id as string | undefined; const size = req.query.size as string | undefined; const params = new URLSearchParams(); if (meetingId) params.set('meeting_id', meetingId); if (size) params.set('size', size); const query = params.toString(); const url = `${TA_DA_BACKEND_URL}/concept-cards${query ? `?${query}` : ''}`; const r = await fetch(url); const data = await r.json().catch(() => ({})); res.status(r.status).json(data); } catch (err) { const msg = err instanceof Error ? err.message : 'concept-cards proxy failed'; logger.error('Concept cards proxy error:', msg); res.status(502).json({ error: msg }); } }); // Start the HTTP server server.listen(PORT, () => { logger.success(`🚀 Server running on port ${PORT}`); logger.info(`📊 Elasticsearch: ${process.env.ELASTIC_URL ? 'Configured' : 'Not configured'}`); logger.info(`📝 Index: ${process.env.ES_INDEX || 'ta-da-latest'}`); }); // Graceful shutdown handlers process.on('SIGINT', async () => { logger.info('Received SIGINT, shutting down gracefully...'); await shutdown(); }); process.on('SIGTERM', async () => { logger.info('Received SIGTERM, shutting down gracefully...'); await shutdown(); }); process.on('uncaughtException', async (error: Error) => { logger.error('Uncaught Exception:', error); await shutdown(); }); process.on( 'unhandledRejection', async (reason: unknown, promise: Promise) => { logger.error('Unhandled Rejection at:', promise, 'reason:', reason); await shutdown(); } ); // Shutdown function let isShuttingDown = false; async function shutdown(): Promise { if (isShuttingDown) { return; } isShuttingDown = true [truncated — 703 more characters] ``` ### backend/src/index.js ```javascript /** * TA-DA Backend (Render) * - POST /events: Zoom App UI sends events here; we route to Tutor or ClassOps agent (or update Elastic for CONCEPT_SET). */ import { config } from "dotenv"; config(); import express from "express"; import { Client } from "@elastic/elasticsearch"; const app = express(); app.use(express.json()); const ELASTICSEARCH_URL = process.env.ELASTICSEARCH_URL || "https://localhost:9200"; const ELASTIC_API_KEY = process.env.ELASTIC_API_KEY || ""; const KIBANA_URL = process.env.KIBANA_URL || ELASTICSEARCH_URL.replace(".es.", ".kb.").replace(/:443$/, ""); const TUTOR_AGENT_URL = process.env.TUTOR_AGENT_URL || "http://localhost:5001"; const CLASSOPS_AGENT_URL = process.env.CLASSOPS_AGENT_URL || "http://localhost:5002"; const INDEX_SESSIONS = "ta-da-sessions"; const INDEX_TA_DA_LATEST = "ta-da-latest"; const INDEX_CONCEPT_CARDS = "ta-da-concept-cards"; function getElastic() { if (!ELASTIC_API_KEY) throw new Error("ELASTIC_API_KEY required for CONCEPT_SET"); return new Client({ node: ELASTICSEARCH_URL, auth: { apiKey: ELASTIC_API_KEY } }); } app.get("/health", (req, res) => { res.json({ status: "ok", service: "ta-da-backend" }); }); /** * Semantic search over ta-da-latest transcript chunks. * For Zoom app or backend: find chunks by natural-language query (semantic search on content field). * * POST /search/semantic * Body: { "query": string (required), "meeting_id"?: string, "size"?: number (default 10) } * Returns: { "hits": [ { "_id", "_score", "meeting_id", "chunk_index", "text", "start_time", "end_time", ... } ] } */ app.post("/search/semantic", async (req, res) => { const { query, meeting_id, size = 10 } = req.body || {}; if (!query || typeof query !== "string") { return res.status(400).json({ error: "body must include query (string)" }); } try { const es = getElastic(); const retriever = { standard: { query: { semantic: { field: "content", query: query.trim() } }, ...(meeting_id && { filter: { term: { meeting_id } } }), }, }; const response = await es.search({ index: INDEX_TA_DA_LATEST, size: Math.min(Math.max(1, Number(size) || 10), 100), retriever, }); const hits = (response.hits?.hits || []).map((h) => ({ _id: h._id, _score: h._score, ...h._source, })); return res.json({ hits }); } catch (e) { console.error("search/semantic error", e); return res.status(500).json({ error: e.message || "search failed" }); } }); /** * Fetch all concept cards from ta-da-concept-cards index. * * GET /concept-cards * Query: meeting_id (optional) - filter by meeting_id * size (optional) - max hits (default 100, max 1000) * Returns: { hits: [ { _id, _score, meeting_id, concept_id, title, short_explain, example, timestamp }, ... ] } */ app.get("/concept-cards", async (req, res) => { try { const es = getElastic(); const meeting_id = req.query.meeting_id; const size = Math.min(Math.max(1, parseInt(req.query.size, 10) || 100), 1000); const query = meeting_id ? { bool: { filter: [{ term: { meeting_id } }] } } : { match_all: {} }; const response = await es.search({ index: INDEX_CONCEPT_CARDS, query, size, sort: [{ timestamp: { unmapped_type: "date", order: "asc" } }], }); const hits = (response.hits?.hits || []).map((h) => ({ _id: h._id, _score: h._score, ...h._source, })); return res.json({ total: response.hits?.total?.value ?? hits.length, hits, }); } catch (e) { console.error("concept-cards error", e); return res.status(500).json({ error: e.message || "concept-cards fetch failed" }); } }); /** * Elastic Agent Builder converse: ask the TA-DA agent a question. * Proxies to Kibana Agent Builder API and returns the full response. * * POST /agent/converse * Body: { "input": string (required), "agent_id"?: string (default: tada-agent) } * Returns: converse API response (conversation_id, steps, response.message, etc.) */ app.post("/agent/converse", async (req, res) => { const { input, agent_id = "tada-agent" } = req.body || {}; if (!input || typeof input !== "string") { return res.status(400).json({ error: "body must include input (string)" }); } try { if (!ELASTIC_API_KEY) { return res.status(500).json({ error: "ELASTIC_API_KEY required for agent converse" }); } const r = await fetch(`${KIBANA_URL}/api/agent_builder/converse`, { method: "POST", headers: { Authorization: `ApiKey ${ELASTIC_API_KEY}`, "kbn-xsrf": "true", "Content-Type": "application/json", }, body: JSON.stringify({ input: input.trim(), agent_id }), }); const data = await r.json().catch(() => ({})); if (!r.ok) { return res.status(r.status).json(data); } return res.json(data); } catch (e) { console.error("agent/converse error", e); return res.status(500).json({ error: e.message || "converse failed" }); } }); /** * Get LiveAvatar session token * POST /api/liveavatar/token * Returns: { session_token, session_id } */ app.post("/api/liveavatar/token", async (req, res) => { try { const apiKey = process.env.LIVEAVATAR_API_KEY; const avatarId = process.env.LIVEAVATAR_AVATAR_ID; const voiceId = process.env.LIVEAVATAR_VOICE_ID; if (!apiKey || !avatarId || !voiceId) { return res.status(400).json({ error: "LiveAvatar configuration missing", details: "LIVEAVATAR_API_KEY, LIVEAVATAR_AVATAR_ID, and LIVEAVATAR_VOICE_ID must be set in .env" }); } const response = await fetch( "https://api.liveavatar.com/v1/sessions/token", { method: "POST", headers: { "X-API-KEY": apiKey, "Content-Type": "application/json", Accept: "application/json", }, body: JSON.stringify({ mode: "FULL", avatar_id: avatarId [truncated — 3805 more characters] ``` ### zoom-RTMS-SDK/create_index.js ```javascript import dotenv from 'dotenv'; import { Client as ESClient } from '@elastic/elasticsearch'; dotenv.config(); const ES_INDEX = process.env.ES_INDEX || 'ta-da-latest'; const ELASTIC_URL = process.env.ELASTIC_URL || process.env.ELASTICSEARCH_URL; if (!ELASTIC_URL) { console.error('❌ Set ELASTIC_URL to your Elasticsearch node'); console.error('Example: ELASTIC_URL=http://localhost:9200'); process.exit(1); } const esOpts = { node: ELASTIC_URL }; if (process.env.ELASTIC_API_KEY) { esOpts.auth = { apiKey: process.env.ELASTIC_API_KEY }; } const es = new ESClient(esOpts); async function createIndex() { try { // Test connection await es.ping(); console.log('✅ Connected to Elasticsearch'); // Check if index exists const exists = await es.indices.exists({ index: ES_INDEX }); if (exists) { console.log(`ℹ️ Index ${ES_INDEX} already exists`); return; } // Create index with ta-da-latest schema await es.indices.create({ index: ES_INDEX, body: { mappings: { properties: { meeting_id: { type: 'keyword' }, chunk_index: { type: 'integer' }, text: { type: 'text' }, start_time: { type: 'float' }, end_time: { type: 'float' }, speaker_id: { type: 'keyword' }, meeting_start_time: { type: 'date' }, received_at: { type: 'date' }, source: { type: 'keyword' }, embedding: { type: 'dense_vector', dims: 1536 } } } } }); console.log(`✅ Created index ${ES_INDEX}`); } catch (err) { console.error('❌ Error:', err.message); process.exit(1); } finally { await es.close(); } } createIndex(); ``` ### fetch-ai/config.py ```python """ Configuration for the Concept Card Agent. Uses environment variables (with .env via python-dotenv). """ import os from pathlib import Path from dotenv import load_dotenv # Load .env from repo root (override=True so .env.local wins over shell env vars) _env_path = Path(__file__).resolve().parent.parent / ".env.local" load_dotenv(_env_path, override=True) from pydantic import BaseModel, Field class ElasticConfig(BaseModel): url: str = Field( default_factory=lambda: os.getenv("ELASTICSEARCH_URL", "https://localhost:9200") ) api_key: str = Field(default_factory=lambda: os.getenv("ELASTIC_API_KEY", "")) class BackendConfig(BaseModel): url: str = Field( default_factory=lambda: os.getenv("RENDER_BACKEND_URL", "http://localhost:3000") ) class RedisConfig(BaseModel): url: str = Field( default_factory=lambda: os.getenv("REDIS_URL", "redis://localhost:6379") ) db: int = Field( default_factory=lambda: int(os.getenv("REDIS_DB", "0")) ) class LLMConfig(BaseModel): # XAI_API_KEY for xAI/Grok; OPENAI_API_KEY for OpenAI (XAI takes precedence) api_key: str = Field( default_factory=lambda: os.getenv("XAI_API_KEY") or os.getenv("OPENAI_API_KEY", "") ) base_url: str = Field( default_factory=lambda: os.getenv("LLM_BASE_URL", "") # https://api.x.ai/v1 for xAI ) model: str = Field( default_factory=lambda: os.getenv( "LLM_MODEL", os.getenv("OPENAI_MODEL", "gpt-4o-mini"), ) ) class AgentConfig(BaseModel): seed: str = Field(default_factory=lambda: os.getenv("CONCEPT_CARD_AGENT_SEED", "")) port: int = Field(default_factory=lambda: int(os.getenv("CONCEPT_CARD_AGENT_PORT", "8010"))) poll_interval_sec: float = Field( default_factory=lambda: float(os.getenv("CONCEPT_CARD_AGENT_POLL_INTERVAL", "45")) ) auto_run_interval_sec: float = Field( default_factory=lambda: float(os.getenv("CONCEPT_CARD_AUTO_RUN_INTERVAL", "60")) ) class ConceptCardAgentConfig(BaseModel): elastic: ElasticConfig = Field(default_factory=ElasticConfig) backend: BackendConfig = Field(default_factory=BackendConfig) redis: RedisConfig = Field(default_factory=RedisConfig) llm: LLMConfig = Field(default_factory=LLMConfig) agent: AgentConfig = Field(default_factory=AgentConfig) def get_config() -> ConceptCardAgentConfig: return ConceptCardAgentConfig() ``` ### fetch-ai/redis_client.py ```python """ Redis client for the Concept Card Agent. Tracks the last processed chunk index per meeting for incremental concept card creation. """ from typing import Optional import redis class RedisClient: def __init__(self, url: str, db: int = 0): """Initialize Redis client. Set url to empty string to disable Redis.""" self.client = redis.from_url(url, db=db, decode_responses=True) if url else None def _ensure_client(self): if not self.client: raise RuntimeError("Redis client not configured (REDIS_URL required)") def get_last_chunk_index(self, meeting_id: str) -> int: """ Get the last processed chunk index for a meeting. Returns -1 if no state exists (meaning all chunks are new). """ self._ensure_client() key = f"concept_cards:last_chunk:{meeting_id}" value = self.client.get(key) return int(value) if value is not None else -1 def set_last_chunk_index(self, meeting_id: str, chunk_index: int, ttl: Optional[int] = None): """ Set the last processed chunk index for a meeting. Args: meeting_id: Meeting identifier chunk_index: Last chunk index that was processed ttl: Optional TTL in seconds (e.g. 86400 for 24h, 604800 for 7 days) """ self._ensure_client() key = f"concept_cards:last_chunk:{meeting_id}" self.client.set(key, chunk_index) if ttl: self.client.expire(key, ttl) def reset_last_chunk_index(self, meeting_id: str): """Delete the last chunk index for a meeting (force reprocess all chunks).""" self._ensure_client() key = f"concept_cards:last_chunk:{meeting_id}" self.client.delete(key) def ping(self) -> bool: """Check if Redis is reachable. Returns True if connected, False otherwise.""" if not self.client: return False try: return self.client.ping() except Exception: return False def is_paused(self) -> bool: """Check if automatic concept card generation is paused globally.""" self._ensure_client() key = "concept_cards:paused" return self.client.get(key) == "1" def pause(self): """Pause automatic concept card generation.""" self._ensure_client() key = "concept_cards:paused" self.client.set(key, "1") def resume(self): """Resume automatic concept card generation.""" self._ensure_client() key = "concept_cards:paused" self.client.delete(key) ``` [24 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]