# Project export: zoomED

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2026
- Tagline: zoomED turns live Zoom classes into real-time engagement intelligence, so instructors can detect attention drops early and respond with AI-powered interventions
- Devpost: https://devpost.com/software/zoomed-pnklzv
- GitHub: https://github.com/tiaL-ops/zoomED
- Video: https://www.youtube.com/embed/NwFoOfsZWXI?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 4 GitHub contributor(s) — nishia1 (17 commits), enamacahiya (10 commits), izabelladesign (4 commits), Cursor (3 commits)

## Devpost submission (written by the team)

### Inspiration

Online classes make it hard to read the room, participate, and retain engagement. Instructors often realize students were lost only after a quiz or exam. Our team built zoomED to give teachers the same intuition they have in person: live awareness of attention, participation, and confusion signals during class.

### What it does

zoomED monitors live Zoom sessions and turns raw meeting activity into actionable teaching insights. It: Tracks attention trends using computer vision (gaze-based attention scoring) Streams chat and participation events in real time Detects engagement drops and highlights at-risk moments Uses AI agents to generate engagement summaries for teachers, gentle nudges to come back to class for students, and adaptive quiz/poll suggestions based on real-time transcripts Displays everything in a live instructor dashboard for immediate intervention

### How we built it

We built a multi-service real-time system: Zoom client app with Zoom Meeting SDK + integration with MediaPipe Face Mesh for attention signals WebSocket event pipeline to stream attention/chat/participation data Node.js/Express backend to aggregate live meeting state Multi-agent AI layer (Anthropic-powered) for summarization, nudges, and quiz generation React + Vite dashboard to visualize engagement and recommendations in real time JWT authentication endpoint for secure Zoom SDK session access

### Challenges we ran into

Synchronizing multiple noisy real-time signals (CV + chat + participation) into one reliable engagement view Zoom RTMS access and setup issues, even when working with an ex-Zoom engineer onsite Keeping WebSocket streams stable and low-latency across services Tuning attention scoring so it’s useful without being overly sensitive Designing AI outputs to be actionable for instructors, not just descriptive Managing the complexity of running four local services during rapid demo iteration

### Accomplishments we're proud of

End-to-end live pipeline from Zoom session -> engagement signal -> AI recommendation -> instructor dashboard Real-time attention event streaming working during live calls Multi-agent architecture that produces different types of classroom interventions A practical demo that feels immediately useful for educators, not just technically impressive Modular architecture that can scale to richer analytics and interventions While tailored towards the education sector, can be expanded into the workplace as well (as said by TreeHacks mentors, thank you for your insights!)

### What we learned

Real-time educational feedback is as much a product-design problem as an AI problem Combining multimodal signals gives better engagement insight than any single metric Fast iteration loops (instrumentation + observability) are critical for live systems Building for trust, transparency, and instructor control is essential in edtech AI

### What's next

Personalize interventions by class style, subject, and learner profile Add longitudinal analytics across sessions (weekly trends, concept-level struggle maps) Improve model calibration and fairness across diverse camera and classroom conditions Pilot with real instructors and measure outcomes like participation lift and retention gains Linkage to "away" feature to allow students to not be badgered with notifications and questions when away from their devices

## README (from the GitHub repository)

# zoomED - AI-Powered Engagement System

**TreeHacks 2026** — Real-time student engagement monitoring with multi-agent AI, computer vision attention tracking, and adaptive intervention.

## Quick Start

**Windows:**
```bash
start-all.bat
```

**Mac/Linux:**
```bash
chmod +x start-all.sh
./start-all.sh
```

Then open:
- **Zoom App:** http://localhost:8080 (join meeting)
- **Teacher Dashboard:** http://localhost:5173/report (view engagement analytics)

Note: TO run this, you must have your own API keys 
*Claude API key: [Get yours here](https://console.anthropic.com/settings/keys)*
---

## Functionality

This system monitors student engagement during Zoom meetings and uses **multi-agent AI** to adaptively respond:

### **Real-Time Monitoring**
- **Computer Vision:** MediaPipe-based gaze tracking detects when students look away
- **Chat Analysis:** Monitors participation in meeting chat
- **Attendance:** Tracks join/leave events

### **Multi-Agent Decision System**
Three specialized Claude agents work together every 10 minutes:

1. **Engagement Summarizer** — Analyzes overall class engagement, identifies struggling students
2. **Nudge Agent** — Sends supportive, personalized check-ins to low-engagement students (rate-limited, non-intrusive)
3. **Quiz Generator** — Creates adaptive questions when a student loses focus 3+ times, prompting re-engagement with material

### **Teacher Dashboard**
- Live engagement metrics and trends over time
- Per-student attention scores
- Timeline showing when engagement was high vs. low
- Recent AI interventions (nudges, quizzes sent)

### **Student Experience**
- **Focus tracking opt-in** (camera-based attention detection)
- **Gentle nudges** when attention drifts ("Quick check-in" popup)
- **Material-based quizzes** appear in sidebar after repeated disengagement
- **Focus game** for quick mental resets

---

## Architecture

### **Event-Driven System**
- Frontend sends events (chat, attention, join/leave) → WebSocket server
- Server accumulates events per meeting in memory
- Every 10 minutes, agents analyze accumulated events and make decisions
- Decisions broadcast back to connected clients via WebSocket

### **Tech Stack**
- **Frontend:** React + Vite
- **Backend:** Node.js + Express + WebSocket
- **AI:** Claude 3.5 Sonnet (multi-agent orchestration)
- **Computer Vision:** MediaPipe Face Mesh
- **Zoom Integration:** Zoom Meeting SDK

### **Key Components**
```
server/
  ├── index.js          # Main server: WebSocket, agents, API endpoints
  ├── agents.js         # Multi-agent logic (summarizer, nudge, quiz)
  └── leaderboard.js    # Quiz scoring system

client/
  ├── src/
      ├── Home.jsx      # Landing page with live engagement feed
      └── Report.jsx    # Teacher dashboard (analytics + controls)

zoomapp/
  ├── app.js            # Zoom SDK integration + gaze tracking
  └── index.html        # Meeting UI with engagement sidebar
```

---

## How It Works

### **1. Join Meeting**
Student opens http://localhost:8080, enters meeting ID, and joins as attendee. Host can enable focus tracking (camera-based attention detection).

### **2. Activity Tracking**
- Every chat message → `CHAT_MESSAGE` event
- Every 5 seconds → `ATTENTION_SCORE` event (if focus tracking on)
- Join/leave → `participant_joined` / `participant_left` events

### **3. Agent Analysis (Every 10 Minutes)**
```python
# Pseudocode
engagement_data = summarize_engagement(events)
low_engagement_users = identify_struggling_students(engagement_data)

for user in low_engagement_users:
    if should_send_nudge(user):
        send_nudge(user, personalized_message)
    
    if user.look_away_count >= 3:
        quiz = generate_quiz_on_material()
        send_quiz(user, quiz)
```

### **4. Real-Time Intervention**
- Students see nudges as popups: *"Quick check-in: Looks like your attention drifted. Want to try a focus game?"*
- Quizzes appear in sidebar: *"Agent question (on material)"*
- Teacher sees everything on the dashboard timeline

---

## 🔧 Setup (First Time Only)

**Prerequisites:** Node 18+, Zoom Meeting SDK credentials

### 1. Clone repo
```bash
git clone <your-repo>
cd treehackswinner2026
```

### 2. Backend setup
```bash
cd server
npm install
cp .env.example .env
# Edit .env: add CLAUDE_API_KEY from https://console.anthropic.com/settings/keys
```

### 3. Frontend setup
```bash
cd client
npm install
```

### 4. Zoom auth endpoint
```bash
cd zoomapp/meetingsdk-auth-endpoint-sample
npm install
# .env already has ZOOM_MEETING_SDK_KEY and ZOOM_MEETING_SDK_SECRET env variables, but you need to generate keys and add them in
```

After setup, just run **`start-all.bat`** (Windows) or **`./start-all.sh`** (Mac).

---

##  Testing & Development

### **Automated Flow**
1. Run `start-all.bat` (or `.sh`)
2. Join meeting at http://localhost:8080
3. Chat messages and attention are tracked automatically
4. Agents run every 10 minutes
5. View teacher dashboard at http://localhost:5173/report

### **Manual Agent Trigger**
To test agents immediately without waiting:
```bash
curl -X POST http://localhost:3000/api/tick \
  -H "Content-Type: application/json" \
  -d '{"meetingId":"YOUR_MEETING_ID"}'
```

### **Mock Events (No Zoom Meeting)**
```bash
# Simulate a chat message
curl -X POST http://localhost:3000/api/events \
  -H "Content-Type: application/json" \
  -d '{"meetingId":"default","type":"CHAT_MESSAGE","userId":"u1","displayName":"Alex"}'

# Simulate attention score
curl -X POST http://localhost:3000/api/events \
  -H "Content-Type: application/json" \
  -d '{"meetingId":"default","type":"ATTENTION_SCORE","userId":"u2","displayName":"Sam","cv_attention_score":0.5}'
```

Then view results at http://localhost:5173/report?meetingId=default

### **Adjust Agent Timing**
In `server/index.js`, change:
```javascript
const SUMMARY_INTERVAL_MS = 10 * 60 * 1000;  // 10 minutes
// To:
const SUMMARY_INTERVAL_MS = 60 * 1000;  // 1 minute (for testing)
```

<<<<<<< HEAD
=======
* go back to **Report**, click **Update summary** — you should see a summary (class engagement, per-user, students losing focus) and last decision.

## 4. test the 10-minute popup

* stay on report with meeting ID `default`.  
* ensure you’ve sent at least one event and run `/api/tick` (or wait for the server’s 10‑minute timer)
* the page connects via WebSocket (you’ll see **● Live** when connected)
* every 10 minutes the server runs the summarizer and pushes `SUMMARY_UPDATE`; a **popup** should appear with the latest summary. (to test without waiting, temporarily change `SUMMARY_INTERVAL_MS` in `server/index.js` to e.g. `60 * 1000` for 1 minute.)

## 5. test gaze feeding into meeting state (optional)

* In the **zoomapp**, enable focus tracking in a meeting; attention is sent to the server for that meeting.
* On the **Report** (or wait for the 10‑minute periodic summary) you’ll see attention in the summary.

## 6. test the nudge agent (refocus popup for attendees)

* send events that create at least one low-engagement user (e.g. `userId: "u2"` with no chat and low/no attention), then run **POST /api/tick** with `meetingId: "default"`
* on the **Report** page, enter a **Preview as attendee (userId)** value that matches someone who got a nudge (e.g. `u2` or `Sam`), and stay on the page with WebSocket connected
* after the next agent run (or run `/api/tick` again), the server broadcasts `NUDGE` for that user; the Report page shows a **“Quick check-in” popup** with the supportive message (this is what the attendee would see in the Zoom app)
* the Report page also shows **Engagement over time** and **Recent nudges sent** so the teacher can see when engagement was high vs low and what nudges were sent

## 7. test chat from Zoom app (live meeting chat → engagement summarizer)

Chat messages sent during a Zoom meeting are forwarded to the server and used by the engagement summarizer (alongside polls and attention scores).

1. **Start the Zoom app stack:**
   ```bash
   # Terminal 1 – auth endpoint (port 4000)

[README truncated for size]

## Detected evidence (automated analysis)

Indexed codebase: 23 recognized source files, 225 KB.
- Anthropic (technology) — detected in the code
- CSS (language) — detected in the code
- Express (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- React (technology) — detected in the code
- Node.js (technology) — claimed on Devpost, not found in the code
- AI coding agent: Cursor — evidence: commit authorship or trailers

## Codebase structure (from repository index)

### Files (32 of 32)

```
.DS_Store
.gitignore
API_REFERENCE.md
client/.gitignore
client/index.html
client/package.json
client/src/App.jsx
client/src/components/Home.jsx
client/src/components/Report.jsx
client/src/index.css
client/src/main.jsx
client/vite.config.js
README.md
server/.gitignore
server/agents.js
server/index.js
server/leaderboard.js
server/package.json
server/summary.txt
server/test-wiring.js
start-all.bat
start-all.sh
test-knowledge-graph.sh
VERIFY_WIRING.sh
zoom-panel/src/index.css
zoom-panel/src/ZoomPanelApp.jsx
zoomapp/.gitignore
zoomapp/app.js
zoomapp/CAPTION_INTEGRATION.js
zoomapp/client-view.js
zoomapp/COMPLETE_INTEGRATION.js
zoomapp/index.html
```

### Dependencies

- client/package.json: @vitejs/plugin-react@^5.1.4, react@^19.2.4, react-dom@^19.2.4, react-router-dom@^7.13.0, vite@^7.3.1
- server/package.json: @anthropic-ai/sdk@^0.74.0, @types/cors@^2.8.19, @types/express@^5.0.6, @types/ws@^8.18.1, cors@^2.8.6, dotenv@^17.3.1, express@^5.2.1, ws@^8.19.0

### Recent commits (newest first)

- resolving minor bug in teacher report dash
- Remove integration checklist from API_REFERENCE.md
- Delete QUICK_REFERENCE.md
- Delete QUIZ_GENERATION_GUIDE.md
- Delete WIRING_COMPLETE.md
- Delete WIRING_DIAGRAM.md
- Delete WIRING_VERIFICATION_INDEX.md
- Revise README for TreeHacks 2026 project
- Merge pull request #7 from tiaL-ops/feature/demo-followups-ui-tuning
- Tune demo UX for nudges and dashboard controls.
- moving all content to main
- fixed teacher button, eye detection button in zoom web app
- Improve live transcript-driven question flow and sidebar behavior.
- merged final report webapp with engagement sidebar branch
- WIP before merging engagement sidebar branch
- WIP before syncing with remote
- knowledge graph backend agent implemented &
- transcript added, quiz generation connected
- Engagement sidebar, nudge popup, and material quiz agent
- Delete server/test-orchestrator.js

## Key source files (fetched from GitHub, selected and truncated for size)

### API_REFERENCE.md

```markdown
# API Reference: Complete Wiring

## Overview

All endpoints needed for caption → quiz generation pipeline:

```
Zoom Captions
    ↓
POST /api/transcript     ← Captions arrive here
    ↓
POST /api/topic          ← Optional: Set lesson topic
    ↓
POST /api/events         ← Student engagement events
    ↓
POST /api/orchestrate    ← MAIN: Generates nudges + quizzes
    ↓
GET /api/report          ← View meeting state
```

---

## Endpoints

### 1. POST /api/transcript
**Receives Zoom live captions (real-time)**

**Request:**
```bash
curl -X POST http://localhost:3000/api/transcript \
  -H "Content-Type: application/json" \
  -d '{
    "meetingId": "meeting-123",
    "userId": "instructor-1",           (optional)
    "displayName": "Dr. Smith",         (optional)
    "text": "Newton'\''s first law states that...",
    "topic": "Newton'\''s Laws of Motion",
    "timestamp": 1707974400000          (optional, auto-added)
  }'
```

**Parameters:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| meetingId | string | ✓ | Unique meeting identifier |
| userId | string | | User ID of speaker (default: 'instructor') |
| displayName | string | | Display name (default: 'Instructor') |
| text | string | ✓ | The caption text |
| topic | string | | Current lesson topic |
| timestamp | number | | When caption occurred (default: now) |

**Response:**
```json
{
  "ok": true,
  "snippetCount": 42,
  "topic": "Newton's Laws of Motion"
}
```

**Usage in Zoom App:**
```javascript
// zoomapp/app.js
fetch('http://localhost:3000/api/transcript', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    meetingId: context.meetingID,
    displayName: 'Instructor',
    text: payload.caption,
    topic: 'Current Lesson Topic'
  })
});
```

---

### 2. POST /api/topic
**Set or update the current lesson topic**

**Request:**
```bash
curl -X POST http://localhost:3000/api/topic \
  -H "Content-Type: application/json" \
  -d '{
    "meetingId": "meeting-123",
    "topic": "Newton'\''s Laws of Motion"
  }'
```

**Parameters:**
| Name | Type | Required | Description |
|------|------|----------|-------------|
| meetingId | string | ✓ | Meeting ID |
| topic | string | ✓ | Lesson topic (used for quiz generation) |

**Response:**
```json
{
  "ok": true,
  "topic": "Newton's Laws of Motion",
  "meetingId": "meeting-123"
}
```

**When to Use:**
- Before class starts, set the topic
- When instructor switches topics mid-class
- To ensure quiz questions match the lesson

---

### 3. POST /api/events
**Ingest student engagement events**

**Request:**
```bash
curl -X POST http://localhost:3000/api/events \
  -H "Content-Type: application/json" \
  -d '{
    "meetingId": "meeting-123",
    "userId": "student-1",
    "displayName": "Alice",
    "type": "ATTENTION_SCORE",
    "cv_attention_score": 0.2
  }'
```

**Common Event Types:**

| Type | Payload | Description |
|------|---------|-------------|
| ATTENTION_SCORE | `
[truncated — 7169 more characters]
```

### client/package.json

```
{
  "name": "client",
  "version": "1.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "preview": "vite preview",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "description": "",
  "dependencies": {
    "@vitejs/plugin-react": "^5.1.4",
    "react": "^19.2.4",
    "react-dom": "^19.2.4",
    "react-router-dom": "^7.13.0",
    "vite": "^7.3.1"
  }
}

```

### server/package.json

```
{
  "name": "server",
  "version": "1.0.0",
  "description": "node/express backend including websocket\r implements all server, web socket, and multi-agent system logic",
  "type": "module",
  "main": "index.js",
  "scripts": {
    "start": "node index.js",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@anthropic-ai/sdk": "^0.74.0",
    "cors": "^2.8.6",
    "dotenv": "^17.3.1",
    "express": "^5.2.1",
    "ws": "^8.19.0"
  },
  "devDependencies": {
    "@types/cors": "^2.8.19",
    "@types/express": "^5.0.6",
    "@types/ws": "^8.18.1"
  }
}

```

### client/src/main.jsx

```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import App from './App';
import './index.css';

ReactDOM.createRoot(document.getElementById('root')).render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);

```

### client/src/App.jsx

```javascript
import React from 'react';
import { BrowserRouter as Router, Routes, Route, Link } from 'react-router-dom';
import Report from './components/Report';

function App() {
  return (
    <Router>
      <div style={{ minHeight: '100vh', background: '#f1f5f9' }}>
        <nav style={{
          padding: '18px 40px',
          background: 'linear-gradient(135deg, #1e293b 0%, #334155 100%)',
          color: 'white',
          boxShadow: '0 2px 16px rgba(0,0,0,0.1)',
          display: 'flex',
          alignItems: 'center',
          gap: '28px',
        }}>
          <Link to="/" style={{ color: 'white', textDecoration: 'none', fontWeight: 700, fontSize: '20px' }}>Engage</Link>
        </nav>

        <Routes>
          <Route path="/" element={<Report />} />
          <Route path="/report" element={<Report />} />
        </Routes>
      </div>
    </Router>
  );
}

export default App;

```

### server/index.js

```javascript
import http from 'http';
import express from 'express';
import cors from 'cors';
import path from 'path';
import { fileURLToPath } from 'url';
import fs from 'fs/promises';
import { WebSocketServer } from 'ws';
import { quizPollAgent, transcribingAgent, engagementSummarizerAgent, meetingCoordinatorAgent, nudgeAgent, orchestrateEngagementSystem } from './agents.js';
import { updateLeaderboard } from './leaderboard.js';
import dotenv from 'dotenv';

dotenv.config();

if (!process.env.CLAUDE_API_KEY) {
  console.warn("CLAUDE_API_KEY not set. Copy server/.env.example to server/.env and add your key. Get one at https://console.anthropic.com/settings/keys");
}

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const app = express();
const PORT = 3000;

const WINDOW_MS = 5 * 60 * 1000; // 5 min for snapshot
const NUDGE_COOLDOWN_MS = 4 * 60 * 1000; // don't nudge same user more than once per 4 min
const lastNudgeByUser = new Map(); // key: meetingId:userId, value: timestamp

/** Build snapshot { users, recentPolls, recentTranscriptSnippets, recentQuestions } from meeting.events for agents */
function buildSnapshot(meeting) {
  const now = Date.now();
  const events = (meeting.events || []).filter((e) => e.ts && now - e.ts <= WINDOW_MS);
  const usersMap = new Map();
  for (const e of events) {
    const uid = e.userId ?? 'anonymous';
    const displayName = e.displayName ?? uid;
    if (!usersMap.has(uid)) {
      usersMap.set(uid, {
        userId: uid,
        displayName,
        signals: {
          polls_answered: 0,
          polls_missed: 0,
          chat_messages: 0,
          avg_response_latency_ms: 0,
          cv_attention_score: null,
          video_on: true,
          _latencies: [],
          _gaze: [],
        },
      });
    }
    const u = usersMap.get(uid);
    if (e.type === 'QUIZ_ANSWER' || e.type === 'QUIZ_RESPONSE') {
      u.signals.polls_answered += 1;
      if (e.responseTimeMs != null) u.signals._latencies.push(e.responseTimeMs);
    } else if (e.type === 'CHAT_MESSAGE' || e.type === 'CHAT') {
      u.signals.chat_messages += 1;
    } else if (e.type === 'ATTENTION_SCORE' || e.type === 'GAZE') {
      const score = e.cv_attention_score ?? e.avgGaze ?? e.gazeScore;
      if (score != null) u.signals._gaze.push(Number(score));
    }
    usersMap.set(uid, u);
  }
  const users = Array.from(usersMap.values()).map((u) => {
    const lat = u.signals._latencies;
    u.signals.avg_response_latency_ms = lat.length ? lat.reduce((a, b) => a + b, 0) / lat.length : 0;
    const g = u.signals._gaze;
    u.signals.cv_attention_score = g.length ? g.reduce((a, b) => a + b, 0) / g.length : null;
    delete u.signals._latencies;
    delete u.signals._gaze;
    return u;
  });
  const recentPolls = events.filter((e) => e.type === 'QUIZ_ANSWER' || e.type === 'QUIZ_RESPONSE');
  const recentTranscriptSnippets = meeting.recentTranscriptSnippets || [];
  const recentQuestions = events.filter((e) => e.type === 'QUESTION');
  return { users, recentPolls, recentTranscriptSnippets, recentQuestions };
}

// ----- Meeting state & WebSocket (multi-agent pipeline) -----
const meetingState = {};
const wss = new WebSocketServer({ noServer: true });
const socketsByMeeting = new Map();

function broadcast(meetingId, msg) {
  const set = socketsByMeeting.get(meetingId);
  if (!set) return;
  const data = JSON.stringify(msg);
  for (const ws of set) {
    if (ws.readyState === ws.OPEN) ws.send(data);
  }
}

function canNudgeUser(meetingId, userId) {
  const key = `${meetingId}:${userId}`;
  const last = lastNudgeByUser.get(key);
  if (!last) return true;
  return Date.now() - last >= NUDGE_COOLDOWN_MS;
}

function recordNudgeSent(meetingId, userId) {
  lastNudgeByUser.set(`${meetingId}:${userId}`, Date.now());
}

/** True after 3+ refocus nudges have been sent; then the coordinator agent decides when to show a poll question. */
function shouldEscalateToPoll(meeting) {
  const nudgeCount = (meeting.recentNudges || []).length;
  return nudgeCount >= 3;
}

/** 1) Summarize. 2) Nudge first (give leeway). 3) Only if sustained low engagement, run coordinator and maybe generate poll. */
async function runAgentsForMeeting(meetingId) {
  const meeting = meetingState[meetingId];
  if (!meeting) return { error: 'no meeting' };
  const snapshot = buildSnapshot(meeting);
  const summary = await engagementSummarizerAgent(meeting);
  meeting.lastSummary = summary;
  meeting.engagementHistory = meeting.engagementHistory || [];
  meeting.engagementHistory.push({
    at: new Date().toISOString(),
    class_engagement: summary.class_engagement,
    cold_students: summary.cold_students || [],
    summary: summary.summary,
  });
  const keep = 50;
  if (meeting.engagementHistory.length > keep) meeting.engagementHistory = meeting.engagementHistory.slice(-keep);

  // Step 1: Nudge first (give leeway—maybe away, restroom, parent). No punishment.
  try {
    const nudgeResult = await nudgeAgent(summary, { meetingType: 'education' });
    const nudges = nudgeResult.nudges || [];
    meeting.recentNudges = meeting.recentNudges || [];
    for (const n of nudges) {
      if (!canNudgeUser(meetingId, n.userId)) continue;
      recordNudgeSent(meetingId, n.userId);
      meeting.recentNudges.push({ ...n, at: new Date().toISOString() });
      if (meeting.recentNudges.length > 30) meeting.recentNudges = meeting.recentNudges.slice(-30);
      broadcast(meetingId, { type: 'NUDGE', payload: { userId: n.userId, displayName: n.displayName, message: n.message, reason: n.reason } });
    }
  } catch (e) {
    console.error('Nudge agent error:', e);
  }

  // Step 2: Only if engagement has been low for a sustained period, escalate to poll/coordinator
  let decision = null;
  if (shouldEscalateToPoll(meeting)) {
    decision = await meetingCoordinatorAgent(summary, snapshot);
    meeting.lastDecision = decision;
    if (decision.action === 'GENERATE_POLL') {
      const snippet = (snapshot.recentT
[truncated — 30737 more characters]
```

### zoomapp/app.js

```javascript
// Initialize Zoom Meeting SDK
ZoomMtg.setZoomJSLib("https://source.zoom.us/3.8.10/lib", "/av");
ZoomMtg.preLoadWasm();
ZoomMtg.prepareWebSDK();

// Auth endpoint (runs on port 4000)
const authEndpoint = "http://localhost:4000";
const leaveUrl = window.location.origin;
let eyeTracker = null;
let lastAttentionLogMs = 0;
let unfocusedSinceMs = null;
let lastFocusPopupMs = 0;
let meetingWs = null;
const FOCUS_POPUP_COOLDOWN_MS = 7000;   // 7 sec between popups (was 12)
const UNFOCUSED_TRIGGER_MS = 4000;     // 4 sec looking away before popup (notes, pen, etc. should not trigger)
const NUDGE_POPUP_AUTO_QUESTION_MS = 18000;  // 18 sec: if user doesn't pick, trigger question agent
const LOOK_AWAY_COUNT_FOR_QUIZ = 3;    // after 3 look-aways, pop sidebar and trigger material quiz
const QUIZ_COOLDOWN_MS = 2 * 60 * 1000; // after answering, don't show new questions for 2 min (until next disengagement)
const NUDGE_GRACE_AFTER_QUIZ_MS = 60 * 1000; // 60s grace: no nudge right after finishing questions
let focusPopupShowCount = 0;           // how many times we've shown the focus popup this "session"
let sidebarQuizCooldownUntil = 0;     // ignore POLL_SUGGESTION until this time (so questions don't loop)
let questionRoundActive = false;      // while answering sidebar questions, pause nudge flow
let nudgeGraceUntil = 0;              // absolute timestamp until nudges are paused
const SERVER_WS_PORT = 3000;
const ATTENTION_POST_INTERVAL_MS = 5000;  // throttle attention events to server
let currentMeetingId = null;
let currentUserId = null;
let currentUserDisplayName = null;
let nudgePopupTimeoutId = null;
let lastAttentionPostMs = 0;
let hostRoleWatcherIntervalId = null;

function postEventToServer(event) {
  if (!currentMeetingId) return;
  var scheme = location.protocol === "https:" ? "https:" : "http:";
  var host = location.hostname || "localhost";
  fetch(scheme + "//" + host + ":" + SERVER_WS_PORT + "/api/events", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ ...event, meetingId: currentMeetingId, ts: Date.now() }),
  }).catch(function () {});
}

function forwardChatToServer(chatData) {
  var sender = chatData && chatData.sender;
  var userId = sender && (sender.userId != null ? String(sender.userId) : sender.participantId);
  var displayName = (sender && (sender.name || sender.displayName)) || "Unknown";
  if (!userId) userId = "unknown";
  postEventToServer({
    type: "CHAT_MESSAGE",
    userId: userId,
    displayName: displayName,
  });
}

function pickTranscriptTextFromObject(obj) {
  if (!obj || typeof obj !== "object") return "";
  var candidates = [
    obj.messageContent,
    obj.msg,
    obj.text,
    obj.content,
    obj.captionMessage,
    obj.message,
    obj.caption,
    obj.transcript,
    obj.body,
  ];
  for (var i = 0; i < candidates.length; i += 1) {
    var c = candidates[i];
    if (typeof c === "string" && c.trim()) return c.trim();
  }
  if (Array.isArray(obj.lines)) {
    var joined = obj.lines
      .map(function (l) { return typeof l === "string" ? l : (l && (l.text || l.content || l.msg)) || ""; })
      .filter(Boolean)
      .join(" ")
      .trim();
    if (joined) return joined;
  }
  if (obj.payload && typeof obj.payload === "object") {
    var nested = pickTranscriptTextFromObject(obj.payload);
    if (nested) return nested;
  }
  return "";
}

// Forward live transcription to server so poll/question agent gets real-time context (Zoom: host must enable "save closed captions" / live transcript)
function forwardLiveTranscriptionToServer(data) {
  if (!currentMeetingId) return;
  var text = "";
  if (typeof data === "string") text = data.trim();
  if (!text) text = pickTranscriptTextFromObject(data);
  if (!text) return;
  var speaker = (data && (data.speakerName || data.userName || data.speaker || data.displayName)) ? String(data.speakerName || data.userName || data.speaker || data.displayName) : "";
  var scheme = location.protocol === "https:" ? "https:" : "http:";
  var host = location.hostname || "localhost";
  fetch(scheme + "//" + host + ":" + SERVER_WS_PORT + "/api/meetings/" + encodeURIComponent(currentMeetingId) + "/transcript", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ lines: [{ text: text, speaker: speaker || undefined, ts: data.timeStamp || Date.now() }] }),
  }).catch(function () {});
}

function showError(message) {
  const errorDiv = document.getElementById("error-message");
  errorDiv.textContent = message;
  setTimeout(() => {
    errorDiv.textContent = "";
  }, 5000);
}

function connectMeetingWebSocket(meetingNumber) {
  if (meetingWs && meetingWs.readyState === WebSocket.OPEN) return;
  const scheme = location.protocol === "https:" ? "wss:" : "ws:";
  const host = location.hostname || "localhost";
  const url = scheme + "//" + host + ":" + SERVER_WS_PORT + "/ws?meetingId=" + encodeURIComponent(String(meetingNumber));
  try {
    meetingWs = new WebSocket(url);
    meetingWs.onmessage = function (event) {
      try {
        const msg = JSON.parse(event.data);
        if (msg.type === "POLL_SUGGESTION" && msg.payload) {
          if (msg.payload.reason !== "Look-away material quiz") return; // sidebar questions should only run for disengagement flow
          if (Date.now() < sidebarQuizCooldownUntil) return; // don't show new questions until next disengagement round
          const poll = msg.payload.poll;
          const hint = msg.payload.hint;
          if (poll && poll.questions && poll.questions.length > 0) {
            showPollInSidebar(poll);
            clearSidebarHint();
          } else if (hint) {
            showSidebarHint(hint);
          }
        }
      } catch (e) {}
    };
    meetingWs.onclose = function () {
      meetingWs = null;
    };
  } catch (e) {
    console.warn("Meeting WebSocket failed:", e);
  }
}

function closeMeetingWebSocket() {
  if (meetingWs) {
    try {
      meetingWs.close();
    } ca
[truncated — 33151 more characters]
```

### start-all.sh

```shell
#!/bin/bash
# Start all services for Zoom Engagement Tool
# Run this from the project root directory

echo
echo "================================"
echo " Starting All Services"
echo "================================"
echo

# Check if Node.js is installed
if ! command -v node &> /dev/null; then
    echo "ERROR: Node.js is not installed or not in PATH"
    exit 1
fi

echo "Starting Backend Server (port 3000)..."
osascript -e 'tell app "Terminal" to do script "cd '"$(pwd)"'/server && node index.js"' &
sleep 2

echo "Starting Zoom Auth Endpoint (port 4000)..."
osascript -e 'tell app "Terminal" to do script "cd '"$(pwd)"'/zoomapp/meetingsdk-auth-endpoint-sample && npm start"' &
sleep 2

echo "Starting Zoom App (port 8080)..."
osascript -e 'tell app "Terminal" to do script "cd '"$(pwd)"'/zoomapp && npx serve -p 8080"' &
sleep 2

echo "Starting Client Frontend (port 5173)..."
osascript -e 'tell app "Terminal" to do script "cd '"$(pwd)"'/client && npm run dev"' &

echo
echo "================================"
echo " All Services Started!"
echo "================================"
echo
echo "Services running:"
echo "  Backend:       http://localhost:3000"
echo "  Zoom Auth:     http://localhost:4000"
echo "  Zoom App:      http://localhost:8080"
echo "  Client:        http://localhost:5173"
echo
echo "Each service is running in its own Terminal window"
echo "Close individual Terminal windows to stop services"
echo

```

### VERIFY_WIRING.sh

```shell
#!/bin/bash
# 
# FINAL VERIFICATION CHECKLIST
# Run this to confirm everything is correctly wired
# 

echo "=============================================================================="
echo "FINAL VERIFICATION: Caption → Quiz Generation Pipeline"
echo "=============================================================================="
echo ""

# Colors
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

checks_passed=0
checks_failed=0

check_file() {
  local file=$1
  local description=$2
  
  if [ -f "$file" ]; then
    echo -e "${GREEN}✓${NC} $description"
    ((checks_passed++))
  else
    echo -e "${RED}✗${NC} $description"
    echo "  Missing: $file"
    ((checks_failed++))
  fi
}

check_endpoint() {
  local endpoint=$1
  local description=$2
  
  echo "  Checking $endpoint..."
  # Just checking if server responds, not full validation
  if curl -s -X POST http://localhost:3000$endpoint -H "Content-Type: application/json" -d '{"meetingId":"test"}' > /dev/null 2>&1; then
    echo -e "  ${GREEN}✓${NC} $description"
    ((checks_passed++))
  else
    echo -e "  ${YELLOW}⚠${NC} $description (server may not be running)"
    ((checks_failed++))
  fi
}

# ============================================================================
# CHECK: Files Exist
# ============================================================================
echo "STEP 1: Checking required files..."
echo "=================================="
echo ""

echo "Backend Files:"
check_file "server/index.js" "Backend server with endpoints"
check_file "server/agents.js" "Multi-agent system"
check_file "server/test-wiring.js" "End-to-end test"

echo ""
echo "Zoom App Files:"
check_file "zoomapp/CAPTION_INTEGRATION.js" "Zoom caption listener"
check_file "zoomapp/COMPLETE_INTEGRATION.js" "Full orchestration setup"
check_file "zoomapp/CAPTION_SETUP.md" "Caption setup guide"

echo ""
echo "Documentation Files:"
check_file "WIRING_COMPLETE.md" "Wiring complete guide"
check_file "QUIZ_GENERATION_GUIDE.md" "Quiz generation guide"
check_file "WIRING_DIAGRAM.md" "Visual wiring diagram"
check_file "API_REFERENCE.md" "API endpoint reference"

echo ""

# ============================================================================
# CHECK: Code Contains Key Strings
# ============================================================================
echo "STEP 2: Checking code contains key components..."
echo "================================================="
echo ""

contains_string() {
  local file=$1
  local string=$2
  local description=$3
  
  if grep -q "$string" "$file" 2>/dev/null; then
    echo -e "${GREEN}✓${NC} $description"
    ((checks_passed++))
  else
    echo -e "${RED}✗${NC} $description"
    echo "  Not found in: $file"
    ((checks_failed++))
  fi
}

echo "Backend Integration:"
contains_string "server/index.js" "app.post.*api/transcript" "POST /api/transcript endpoint exists"
contains_string "server/index.js" "app.post.*api/topic" "POST /api/topic endpoint exists"
contains_string "server/index.js" "app.post.*api/orchestrate" "POST /api/orchestrate endpoint exists"
contains_string "server/index.js" "orchestrateEngagementSystem" "Uses orchestrateEngagementSystem"

echo ""
echo "Agent System:"
contains_string "server/agents.js" "orchestrateEngagementSystem" "Orchestrator function exists"
contains_string "server/agents.js" "executeParticipantChain" "Participant chain executor exists"
contains_string "server/agents.js" "classContext.recentTranscript" "Quiz agent uses transcript"
contains_string "server/agents.js" "classContext.currentTopic" "Quiz agent uses topic"

echo ""
echo "Zoom App Integration:"
contains_string "zoomapp/CAPTION_INTEGRATION.js" "setupLiveCaptionListener" "Caption listener setup exists"
contains_string "zoomapp/CAPTION_INTEGRATION.js" "api/transcript" "Sends to /api/transcript"
contains_string "zoomapp/COMPLETE_INTEGRATION.js" "orchestrateEngagementSystem\|api/orchestrate" "Orchestration setup exists"

echo ""

# ============================================================================
# SUMMARY
# ============================================================================
echo "=============================================================================="
echo "VERIFICATION SUMMARY"
echo "=============================================================================="
echo ""
echo -e "Checks passed: ${GREEN}$checks_passed${NC}"
echo -e "Checks failed: ${RED}$checks_failed${NC}"
echo ""

total=$((checks_passed + checks_failed))
percentage=$((checks_passed * 100 / total))

if [ $checks_failed -eq 0 ]; then
  echo -e "${GREEN}🎉 ALL CHECKS PASSED!${NC}"
  echo ""
  echo "System is correctly wired for:"
  echo "  ✓ Zoom caption capture"
  echo "  ✓ Caption storage"
  echo "  ✓ Multi-agent orchestration"
  echo "  ✓ Content-based quiz generation"
  echo "  ✓ Personalized nudges"
  echo "  ✓ Real-time broadcasting"
  echo ""
  echo "Next steps:"
  echo "  1. Start backend: npm start"
  echo "  2. Import setupLiveCaptionListener() in Zoom app"
  echo "  3. Test: node server/test-wiring.js"
  echo "  4. Run live class with captions enabled"
  echo ""
else
  echo -e "${YELLOW}⚠ Some checks failed${NC}"
  echo "Please ensure all required files are in place"
fi

echo ""
echo "=============================================================================="
echo ""

# ============================================================================
# QUICK TEST (if server running)
# ============================================================================
if command -v curl &> /dev/null; then
  echo "OPTIONAL: Quick API Test (if server running)"
  echo "============================================="
  echo ""
  
  MEETING_ID="test-$(date +%s)"
  
  echo "Testing POST /api/transcript..."
  RESPONSE=$(curl -s -X POST http://localhost:3000/api/transcript \
    -H "Content-Type: application/json" \
    -d "{\"meetingId\":\"$MEETING_ID\", \"text\":\"Test caption\", \"topic\":\"Test Topic\"}" 2>/dev/null)
[truncated — 336 more characters]
```

### test-knowledge-graph.sh

```shell
#!/bin/bash
# Knowledge Graph Testing Script
# Run this to verify the knowledge graph system is working

set -e  # Exit on error

BASE_URL="http://localhost:3000"
MEETING_ID="test-kg-$(date +%s)"
STUDENT_1="student-alice"
STUDENT_2="student-bob"

echo "======================================================================"
echo "KNOWLEDGE GRAPH SYSTEM TEST"
echo "======================================================================"
echo ""
echo "Meeting ID: $MEETING_ID"
echo ""

# Color codes
GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color

test_count=0
passed_count=0

function test_endpoint() {
  local name=$1
  local method=$2
  local endpoint=$3
  local data=$4
  
  ((test_count++))
  echo -n "Test $test_count: $name... "
  
  if [ "$method" = "GET" ]; then
    response=$(curl -s -X GET "$BASE_URL$endpoint")
  else
    response=$(curl -s -X $method "$BASE_URL$endpoint" \
      -H "Content-Type: application/json" \
      -d "$data")
  fi
  
  # Check if response contains error
  if echo "$response" | grep -q "error"; then
    echo -e "${RED}FAILED${NC}"
    echo "Response: $response"
    echo ""
    return 1
  else
    echo -e "${GREEN}PASS${NC}"
    ((passed_count++))
    return 0
  fi
}

# Test 1: Send initial transcript
echo -e "${YELLOW}Phase 1: Sending Transcripts${NC}"
echo ""

test_endpoint "Send transcript 1" "POST" "/api/transcript" \
  "{
    \"meetingId\": \"$MEETING_ID\",
    \"text\": \"Today we're learning about Newton's Second Law. Force equals mass times acceleration. The equation is F equals m times a.\",
    \"topic\": \"Physics: Newton's Laws\",
    \"displayName\": \"Instructor\"
  }"

test_endpoint "Send transcript 2" "POST" "/api/transcript" \
  "{
    \"meetingId\": \"$MEETING_ID\",
    \"text\": \"Let's think about what happens when we double the force. The acceleration also doubles.\",
    \"displayName\": \"Instructor\"
  }"

test_endpoint "Send transcript 3" "POST" "/api/transcript" \
  "{
    \"meetingId\": \"$MEETING_ID\",
    \"text\": \"Different masses experience different accelerations. A lighter object accelerates faster.\",
    \"displayName\": \"Instructor\"
  }"

echo ""
echo -e "${YELLOW}Phase 2: Simulate Student Events${NC}"
echo ""

# Simulate engagement events for student 1 (high engagement)
test_endpoint "Send chat event for Alice" "POST" "/api/events" \
  "{
    \"meetingId\": \"$MEETING_ID\",
    \"type\": \"CHAT\",
    \"userId\": \"$STUDENT_1\",
    \"displayName\": \"Alice\",
    \"text\": \"Great explanation!\"
  }"

# Simulate low engagement for student 2
test_endpoint "Send gaze event for Bob (low attention)" "POST" "/api/analyze-gaze" \
  "{
    \"meetingId\": \"$MEETING_ID\",
    \"userId\": \"$STUDENT_2\",
    \"displayName\": \"Bob\",
    \"avgGaze\": 0.25
  }"

echo ""
echo -e "${YELLOW}Phase 3: Trigger Multi-Agent Orchestration${NC}"
echo ""

# This is where knowledge graph gets extracted
response=$(curl -s -X POST "$BASE_URL/api/orchestrate" \
  -H "Content-Type: application/json" \
  -d "{\"meetingId\": \"$MEETING_ID\"}")

if echo "$response" | grep -q "knowledgeGraph"; then
  echo -e "${GREEN}✓ Orchestration triggered - knowledge graph generated!${NC}"
  ((test_count++))
  ((passed_count++))
else
  echo -e "${RED}✗ Orchestration failed${NC}"
  echo "Response: $response"
  ((test_count++))
fi

echo ""
echo -e "${YELLOW}Phase 4: Retrieve Class-Level Knowledge Graph${NC}"
echo ""

# Get the class knowledge graph
response=$(curl -s -X GET "$BASE_URL/api/knowledge-graph/$MEETING_ID")

if echo "$response" | grep -q "key_points"; then
  echo -e "${GREEN}✓ Class knowledge graph retrieved${NC}"
  ((test_count++))
  ((passed_count++))
  
  # Count key points
  key_point_count=$(echo "$response" | grep -o '"id"' | wc -l)
  echo "  Concepts identified: $key_point_count"
  
  # Show concepts
  echo "  Topics covered:"
  echo "$response" | grep -o '"title":"[^"]*"' | head -5 | sed 's/"title":"//;s/"//g' | sed 's/^/    - /'
else
  echo -e "${RED}✗ Failed to retrieve knowledge graph${NC}"
  echo "Response: $response"
  ((test_count++))
fi

echo ""
echo -e "${YELLOW}Phase 5: Retrieve Per-Participant Progress${NC}"
echo ""

# Try to get progress for each student
for student in $STUDENT_1 $STUDENT_2; do
  response=$(curl -s -X GET "$BASE_URL/api/knowledge-graph/$MEETING_ID/$student")
  
  if echo "$response" | grep -q "userId"; then
    echo -e "${GREEN}✓ Progress retrieved for $student${NC}"
    ((test_count++))
    ((passed_count++))
    
    # Show their progress
    encountered=$(echo "$response" | grep -o '"encounteredConcepts":\[' | wc -l)
    if [ $encountered -gt 0 ]; then
      echo "  Encountered concepts:"
      echo "$response" | grep -o '"id":"[^"]*"' | head -3 | sed 's/"id":"//;s/"//g' | sed 's/^/    - /'
    fi
  else
    echo -e "${YELLOW}! No progress yet for $student (quizzes may not have been generated)${NC}"
    ((test_count++))
    # Not counting as pass/fail - depends on engagement thresholds
  fi
done

echo ""
echo -e "${YELLOW}Phase 6: Update Concept Mastery${NC}"
echo ""

# Update mastery for a student
# First get concept IDs from the graph
response=$(curl -s -X GET "$BASE_URL/api/knowledge-graph/$MEETING_ID")

# Extract first concept ID (if exists)
concept_id=$(echo "$response" | grep -o '"id":"kp[0-9]*"' | head -1 | sed 's/"id":"//;s/"//g')

if [ ! -z "$concept_id" ]; then
  echo "  Updating mastery for concept: $concept_id"
  
  response=$(curl -s -X POST "$BASE_URL/api/knowledge-graph/$MEETING_ID/$STUDENT_1/update-mastery" \
    -H "Content-Type: application/json" \
    -d "{\"conceptId\": \"$concept_id\", \"mastered\": true}")
  
  if echo "$response" | grep -q "success"; then
    echo -e "${GREEN}✓ Concept mastery updated${NC}"
    ((test_count++))
    ((passed_count++))
  else
    echo -e "${RED}✗ Failed to update mastery${NC}"
    echo "Response: $response"
    ((test_count++))
  fi
else
  echo -e "${YELLOW}! Could not find concept ID to test with${NC}"
  ((test_count++))
fi

echo
[truncated — 1365 more characters]
```

[14 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]