# Project export: Omni This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: A layer of intelligence for meetings. - Devpost: https://devpost.com/software/rtms - GitHub: https://github.com/Ishan-Sinha123/treehacks_26 - Team: 5 GitHub contributor(s) — Gaurang Pendharkar (34 commits), Kenneth Yang (19 commits), Ishaan Awasthi (18 commits), Kenneth Yang (2 commits), Claude Opus 4.6 (1 commits) ## Devpost submission (written by the team) ### Overview For decades, we've seen how meetings are a cornerstone of nearly every industry. Coordination among colleagues is how efforts scale from a single CEO to tens of thousands of employees. Yet despite their importance, most meetings remain passive experiences: time-consuming, stressful, and frequently redundant. Critical ideas get buried across sprawling agendas, context from past discussions is forgotten, and participants struggle to stay aligned as conversations diverge and change context. We saw an opportunity to change that. We wanted to build meeting software that goes beyond simply connecting people. Something that actively understands the flow of conversation, recalls what matters from previous sessions, and helps every voice contribute meaningfully, whether it's a small study group, a company-wide webinar, or a fast-paced team standup. Zoom has become a critical component of our modern internet and business infrastructure, and having a native solution that addresses these problems will have a great impact on society. Our solution is a Zoom-integrated meeting copilot that transforms live conversations into actionable intelligence. It captures and transcribes meetings in real time, summarizes topics as they emerge, and lets participants query anything that's been said, all from within the Zoom interface. Beyond the current meeting, it connects to the collective knowledge of your organization: when a new topic surfaces, it consults AI agents representing team members outside the room and surfaces only the most surprising or actionable perspectives. The result is a meeting experience where context is never lost, insights arrive in real time, and every discussion is informed by the full breadth of your organization's thinking. Constant iteration! Our tech stack combines real-time streaming, semantic search, and AI inference. Zoom App SDK + Immersive View: The app runs natively inside the Zoom client as a sidebar and immersive overlay. Hosted and served from Render. Zoom App SDK + Immersive View: The app runs natively inside the Zoom client as a sidebar and immersive overlay. Hosted and served from Render. Zoom RTMS SDK: Live meeting audio is transcribed in real time and sent via Zoom's WebSocket API, producing a continuous stream of speaker-attributed transcript segments. Zoom RTMS SDK: Live meeting audio is transcribed in real time and sent via Zoom's WebSocket API, producing a continuous stream of speaker-attributed transcript segments. Elasticsearch (on Render): Powers transcript storage, indexing, and retrieval using Jina AI embeddings for semantic vector search. The Kibana API is used for agent interactions and Elastic Workflows for agentic pipelines. Elasticsearch (on Render): Powers transcript storage, indexing, and retrieval using Jina AI embeddings for semantic vector search. The Kibana API is used for agent interactions and Elastic Workflows for agentic pipelines. AI Inference (Claude): Claude handles real-time summarization, topic detection, and natural language querying via Elasticsearch's OpenInference API. AI Inference (Claude): Claude handles real-time summarization, topic detection, and natural language querying via Elasticsearch's OpenInference API. Render: Manages all backend services, AI inference, and storage infrastructure orchestration, and serves rendered interfaces to the Zoom client. It is the operational backbone of our system. Render: Manages all backend services, AI inference, and storage infrastructure orchestration, and serves rendered interfaces to the Zoom client. It is the operational backbone of our system. Learning how to develop for new platforms (to us) like Zoom Apps, RTMS, and ElasticSearch. Learning how to develop for new platforms (to us) like Zoom Apps, RTMS, and ElasticSearch. Acquiring the right permissions necessary for development. Acquiring the right permissions necessary for development. Connecting our front-end components (Zoom App) with the back end (RTMS, Elastic Search, Render). Connecting our front-end components (Zoom App) with the back end (RTMS, Elastic Search, Render). Making sure the goals we wanted to achieve could not already be easily done by existing commercial tools. We looked at tools like Zoom's own meeting AI summarizer, Otter AI, and Granola AI. A key differentiating factor we wanted to bring was our ability to use Zoom's real-time media streams to bring participants separated contexts and summarizations to users. Making sure the goals we wanted to achieve could not already be easily done by existing commercial tools. We looked at tools like Zoom's own meeting AI summarizer, Otter AI, and Granola AI. A key differentiating factor we wanted to bring was our ability to use Zoom's real-time media streams to bring participants separated contexts and summarizations to users. Building out high-volume data pipelines via free subscriptions etc. (not without its challenges!) Building out high-volume data pipelines via free subscriptions etc. (not without its challenges!) Developing a low-latency Realtime Media Stream (RTMS) to inference pipeline. Extremely quick in providing topic summaries to meeting attendees. Developing a low-latency Realtime Media Stream (RTMS) to inference pipeline. Extremely quick in providing topic summaries to meeting attendees. Interactive immersive view and integration with the native client. Interactive immersive view and integration with the native client. Render is the host system for our entire application. Render is the host system for our entire application. Open-source implementation. Open-source implementation. How to build engaging applications for Zoom. How to build engaging applications for Zoom. Manage and feed data to power LLM agents, and how to use them to enable novel workflows. Manage and feed data to power LLM agents, and how to use them to enable novel workflows. Working together under pressure to deliver features that both we and Zoom were excited about. Working together under pressure to deliver features that both we and Zoom were excited about. We would love to have our technology stack shipped in Zoom's experience. We believe real-time in-meeting context summarization and agentic knowledge retrieval are key innovations in the space of online meeting infrastructure and should be widely adopted. We would love to have our technology stack shipped in Zoom's experience. We believe real-time in-meeting context summarization and agentic knowledge retrieval are key innovations in the space of online meeting infrastructure and should be widely adopted. Releasing a guide on best practices for Zoom's developer tools, especially Realtime Media Streams. Releasing a guide on best practices for Zoom's developer tools, especially Realtime Media Streams. Building agent information siloes (for organizations where agents might be limited in what they can share, depending on who asks). Building agent information siloes (for organizations where agents might be limited in what they can share, depending on who asks). ## README (from the GitHub repository) # Zoom Apps JavaScript Sample This Zoom App Sample uses Node.js + Express to build a simple Hello World Zoom App. ## Prerequisites 1. [Node JS](https://nodejs.org/en/) 2. [Ngrok](https://ngrok.com/docs/getting-started) 3. [Zoom Account](https://support.zoom.us/hc/en-us/articles/207278726-Plan-Types-) 4. [Zoom App Credentials](#config:-app-credentials) (Instructions below) 1. Client ID 2. Client Secret 3. Redirect URI ## Getting started Open your terminal: ```bash # Clone down this repository git clone git@github.com:zoom/zoomapps-sample-js.git # navigate into the cloned project directory cd zoomapps-sample-js # run NPM to install the app dependencies npm install # initialize your ngrok session ngrok http 3000 ``` ### Create your Zoom App In your web browser, navigate to [Zoom Developer Portal](https://developers.zoom.us/) and register/log into your developer account. Click the "Build App" button at the top and choose to "Zoom Apps" application. 1. Name your app 2. Choose whether to list your app on the marketplace or not 3. Click "Create" For more information, you can follow [this guide](https://dev.to/zoom/introducing-zoom-apps-33he) check out [this video series](https://www.youtube.com/playlist?list=PLKpRxBfeD1kGN-0QgQ6XtSwtxI3GQM16R) on how to create and configure these sample Zoom Apps. ### Config: App Credentials In your terminal where you launched `ngrok`, find the `Forwarding` value and copy/paste that into the "Home URL" and " Redirect URL for OAuth" fields. ``` Home URL: https://xxxxx.ngrok.io Redirect URL for OAuth: https://xxxxx.ngrok.io/auth ``` > NOTE: ngrok URLs under ngrok's Free plan are ephemeral, meaning they will only live for up to a couple hours at most, and will change every time you reinitialize the application. This will require you to update these fields every time you restart your ngrok service. #### OAuth allow list - `https://example.ngrok.io` #### Domain allow list - `appssdk.zoom.us` - `ngrok.io` ### Config: Information The following information is required to activate your application: - Basic Information - App name - Short description - Long description (entering a short message here is fine for now) - Developer Contact Information - Name - Email address > NOTE: if you intend to publish your application on the Zoom Apps Marketplace, more information will be required in this section before submitting. ### Config: App Features Under the Zoom App SDK section, click the `+ Add APIs` button and enable the following options from their respective sections: #### APIs - shareApp ### Scopes Ensure that the following scope is selected on the Scopes tab: - `zoomapp:inmeeting` ### Config `.env` When building for Development, open the `.env` file in your text editor and enter the following information from the App Credentials section you just configured: ```ini # Client ID for your Zoom App ZM_CLIENT_ID=[app_client_id] # Client Secret for your Zoom app ZM_CLIENT_SECRET=[app_client_secret] # Redirect URI set for your app in the Zoom Marketplace ZM_REDIRECT_URL=https://[xxxx-xx-xx-xxx-x].ngrok.io/auth ``` #### Zoom for Government If you are a [Zoom for Government (ZfG)](https://www.zoomgov.com/) customer you can use the `ZM_HOST` variable to change the base URL used for Zoom. This will allow you to adjust to the different Marketplace and API Base URLs used by ZfG customers. **Marketplace URL:** marketplace.*zoomgov.com* **API Base URL:** api.*zoomgov.com* ## Start the App ### Development Run the `dev` npm script to start in development mode using a Docker container. ```shell npm run dev ``` The `dev` script will: 1. Watch JS files and built to the dist/ folder 1. Watch Server files and build to the dist/ folder 1. Start the application ### Production When running your application in production no logs are sent to the console by default and the server is not restarted on file changes. We use the `NODE_ENV` environment variable here to tell the application to start in prodcution mode. ```shell # Mac/Linux NODE_ENV=production npm start # Windows set NODE_ENV=production && npm start ```` ## Usage To install the Zoom App, Navigate to the **Home URL** that you set in your browser and click the link to install. After you authorize the app, Zoom will automatically open the app within the client. ### Keeping secrets secret This application makes use of your Zoom App Client ID and Client Secret as well as a custom secret for signing session cookies. During development, the application will read from the .env file. ; In order to align with security best practices, this application does not read from the .env file in production mode. This means you'll want to set environment variables on the hosting platform that you' re using instead of within the .env file. This might include using a secret manager or a CI/CD pipeline. > :warning: **Never commit your .env file to version control:** The file likely contains Zoom App Credentials and Session Secrets ### Code Style This project uses [prettier](https://prettier.io/) and [eslint](https://eslint.org/) to enforce style and protect against coding errors along with a pre-commit git hook(s) via [husky](https://typicode.github.io/husky/#/) to ensure files pass checks prior to commit. ### Testing At this time there are no e2e or unit tests. ## Need help? If you're looking for help, try [Developer Support](https://devsupport.zoom.us) or our [Developer Forum](https://devforum.zoom.us). Priority support is also available with [Premier Developer Support](https://zoom.us/docs/en-us/developer-support-plans.html) plans. ### Documentation Make sure to review [our documentation](https://marketplace.zoom.us/docs/zoom-apps/introduction/) as a reference when building your Zoom Apps. ## Detected evidence (automated analysis) Indexed codebase: 53 recognized source files, 1094 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - JavaScript (language) — detected in the code - TypeScript (language) — detected in the code - Python (language) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (72 of 72) ``` .claude/settings.local.json .env.sample .eslintrc.json .gitignore .husky/.gitignore .husky/pre-commit .prettierrc app.js config.js elasticsearch/Dockerfile kibana/Dockerfile LICENSE.md package.json pnpm-workspace.yaml public/css/bulma.min.css public/css/style.css public/img/.gitkeep public/js/index.js README.md render.yaml rollup.config.js scripts/dev.js scripts/gen-secrets.cjs server/helpers/agent-manager.js server/helpers/cipher.js server/helpers/elasticsearch.js server/helpers/routing.js server/helpers/setup-inference.js server/helpers/summarizer.js server/helpers/transcript-buffer.js server/helpers/zoom-api.js server/routes/api.js server/routes/auth.js server/routes/index.js server/routes/rtms.js server/routes/webhook.js server/rtmsManager/ActiveConnectionManager.js server/rtmsManager/FrontendManager.js server/rtmsManager/FrontendWssManager.js server/rtmsManager/helpers/audio/MediaAudioFiller.js server/rtmsManager/helpers/video/black_frame.h264 server/rtmsManager/helpers/video/MediaVideoFiller.js server/rtmsManager/helpers/video/sps_pps_keyframe.h264 server/rtmsManager/index.js server/rtmsManager/mediaSocket.js server/rtmsManager/mediaSocketMessageHandler.js server/rtmsManager/package.json server/rtmsManager/processors/audioProcessor.js server/rtmsManager/processors/chatProcessor.js server/rtmsManager/processors/sharescreenProcessor.js server/rtmsManager/processors/transcriptProcessor.js server/rtmsManager/processors/videoProcessor.js server/rtmsManager/RTMSManager.js server/rtmsManager/RTMSMessageHandler.js server/rtmsManager/signalingSocket.js server/rtmsManager/signalingSocketMessageHandler.js server/rtmsManager/types/index.d.ts server/rtmsManager/utils/FileLogger.js server/rtmsManager/utils/redactSecrets.js server/rtmsManager/utils/RTMSConfigHelper.js server/rtmsManager/utils/RTMSError.js server/rtmsManager/utils/rtmsEventLookupHelper.js server/rtmsManager/utils/RTMSFlagHelper.js server/rtmsManager/utils/rtmsMediaParams.js server/rtmsManager/utils/signatureHelper.js server/server.js server/session.js server/views/error.pug server/views/index.pug server/views/layout.pug server/views/mixins/chat-interface.pug server/views/mixins/prompt-install.pug ``` ### Dependencies - package.json: @babel/cli@^7.17.6, @babel/core@^7.17.9, @babel/plugin-transform-runtime@^7.17.0, @babel/preset-env@^7.16.11, @elastic/elasticsearch@^9.2.0, @rollup/plugin-babel@^5.3.1, @rollup/plugin-commonjs@^21.0.3, @rollup/plugin-node-resolve@^13.1.3, @rollup/plugin-url@^6.1.0, @zoom/appssdk@^0.16.26, axios@^1.7.9, compression@^1.7.4, concurrently@^7.1.0, cookie-parser@~1.4.6, cookie-session@^2.0.0, core-js@^3.21.1, debug@~4.3.3, dotenv@^16.0.0, envsub@^4.0.7, eslint@^8.9.0, eslint-config-prettier@^8.4.0, express@^4.21.1, express-validator@^6.14.0, helmet@^5.0.2, http-errors@^2.0.0, husky@^7.0.4, lint-staged@^12.3.4, morgan@~1.10.0, nodemon@^3.0.1, prettier@^2.5.1, pug@^3.0.2, rollup@^2.70.1, rollup-plugin-copy@^3.4.0, rollup-plugin-delete@^2.0.0, rollup-plugin-terser@^7.0.2, ws@^8.18.0 - server/rtmsManager/package.json: node-fetch@^3.3.2, ws@^8.16.0 ### Recent commits (newest first) - Agentic network activated - Upgrade ES client to 9.2.0 and fix body→document/top-level migration - Add agent-to-agent communication network via Kibana Agent Builder - fixes - semantic fix - k - . - users - / - easier fix - fix - Add verbose client-side logging for poll debugging - Add verbose logging to webhook and speakers endpoint - Persist meeting ID→UUID mapping to ES meetings index - fix docker shit - fix security thing oops my bad - Merge pull request #8 from Ishan-Sinha123/hooking_up - Add logging across meeting ID pipeline for debugging - Fix meeting ID translation across all endpoints - Fix meeting ID field name: payload.object.meeting_id not .id ## Key source files (fetched from GitHub, selected and truncated for size) ### LICENSE.md ```markdown MIT License Copyright (c) 2022 Zoom Video Communications, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ``` ### package.json ``` { "name": "zoomapps-sample-js", "version": "1.0.5", "type": "module", "exports": "./server", "private": true, "scripts": { "prestart": "npm run build", "start": "node app.js", "dev": "node scripts/dev.js", "build": "rollup -c", "prepare": "npx husky install" }, "dependencies": { "@babel/cli": "^7.17.6", "@babel/core": "^7.17.9", "@babel/plugin-transform-runtime": "^7.17.0", "@babel/preset-env": "^7.16.11", "@elastic/elasticsearch": "^9.2.0", "@rollup/plugin-babel": "^5.3.1", "@rollup/plugin-commonjs": "^21.0.3", "@rollup/plugin-node-resolve": "^13.1.3", "@rollup/plugin-url": "^6.1.0", "@zoom/appssdk": "^0.16.26", "axios": "^1.7.9", "compression": "^1.7.4", "cookie-parser": "~1.4.6", "cookie-session": "^2.0.0", "core-js": "^3.21.1", "debug": "~4.3.3", "express": "^4.21.1", "express-validator": "^6.14.0", "helmet": "^5.0.2", "http-errors": "^2.0.0", "morgan": "~1.10.0", "pug": "^3.0.2", "rollup": "^2.70.1", "rollup-plugin-copy": "^3.4.0", "rollup-plugin-delete": "^2.0.0", "rollup-plugin-terser": "^7.0.2", "ws": "^8.18.0" }, "devDependencies": { "concurrently": "^7.1.0", "dotenv": "^16.0.0", "envsub": "^4.0.7", "eslint": "^8.9.0", "eslint-config-prettier": "^8.4.0", "husky": "^7.0.4", "lint-staged": "^12.3.4", "nodemon": "^3.0.1", "prettier": "^2.5.1" }, "nodemonConfig": { "quiet": true, "watch": [ "server/**/*.js", "app.js", "config.js", ".env" ] }, "lint-staged": { "*.js": [ "eslint --cache --fix", "prettier --write" ] } } ``` ### kibana/Dockerfile ``` FROM docker.elastic.co/kibana/kibana:9.2.0 EXPOSE 5601 ``` ### elasticsearch/Dockerfile ``` FROM docker.elastic.co/elasticsearch/elasticsearch:9.2.0 # Disable X-Pack security for hackathon simplicity ENV xpack.security.enabled=false ENV xpack.license.self_generated.type=trial ENV discovery.type=single-node # Expose ports EXPOSE 9200 9300 # Health check HEALTHCHECK --interval=30s --timeout=3s --retries=3 \ CMD curl -f http://localhost:9200/_cluster/health || exit 1 ``` ### server/rtmsManager/package.json ``` { "name": "rtms-manager-dev", "version": "2.0.0", "description": "Zoom RTMS SDK Library - Developer-friendly API for real-time media streaming", "main": "index.js", "types": "types/index.d.ts", "type": "module", "exports": { ".": { "types": "./types/index.d.ts", "import": "./index.js" } }, "keywords": [ "zoom", "rtms", "real-time", "media", "streaming", "audio", "video", "transcript", "meeting", "webinar" ], "dependencies": { "ws": "^8.16.0", "node-fetch": "^3.3.2" } } ``` ### app.js ```javascript import express from 'express'; import axios from 'axios'; import compression from 'compression'; import cookieParser from 'cookie-parser'; import debug from 'debug'; import helmet from 'helmet'; import logger from 'morgan'; import { dirname } from 'path'; import { fileURLToPath, URL } from 'url'; import { start } from './server/server.js'; import indexRoutes from './server/routes/index.js'; import authRoutes from './server/routes/auth.js'; import apiRoutes from './server/routes/api.js'; import webhookRoutes from './server/routes/webhook.js'; import { initializeRTMSWebSocket } from './server/routes/rtms.js'; import { appName, port, redirectUri } from './config.js'; const __dirname = dirname(fileURLToPath(import.meta.url)); /* App Config */ const app = express(); const dbg = debug(`${appName}:app`); const redirectHost = new URL(redirectUri).host; // views and assets const staticDir = `${__dirname}/dist`; const viewDir = `${__dirname}/server/views`; app.set('view engine', 'pug'); app.set('views', viewDir); app.locals.basedir = staticDir; // HTTP app.set('port', port); app.set('trust proxy', true); // log Axios requests and responses const logFunc = (r) => { if (process.env.NODE_ENV !== 'production') { let { method, status, url, baseURL, config } = r; const endp = url || config?.url; const base = baseURL || config?.baseURL; let str = new URL(endp, base).href; if (method) str = `${method.toUpperCase()} ${str}`; if (status) str = `${status} ${str}`; debug(`${appName}:axios`)(str); } return r; }; axios.interceptors.request.use(logFunc); axios.interceptors.response.use(logFunc); /* Middleware */ const headers = { frameguard: { action: 'sameorigin', }, hsts: { maxAge: 31536000, }, referrerPolicy: 'same-origin', crossOriginEmbedderPolicy: false, contentSecurityPolicy: { directives: { 'default-src': 'self', styleSrc: ["'self'"], scriptSrc: ["'self'", 'https://appssdk.zoom.us/sdk.min.js'], imgSrc: ["'self'", `https://${redirectHost}`], 'connect-src': ["'self'", 'ws:', 'wss:'], // Allow WebSocket connections 'base-uri': 'self', 'form-action': 'self', }, }, }; app.use(helmet(headers)); app.use(express.json()); app.use(compression()); app.use(cookieParser()); app.use(express.urlencoded({ extended: false })); app.use(logger('dev', { stream: { write: (msg) => dbg(msg) } })); // serve our app folder app.use(express.static(staticDir)); /* Routing */ app.use('/', indexRoutes); app.use('/auth', authRoutes); app.use('/api', apiRoutes); // NEW: API routes app.use('/webhook', webhookRoutes); // Zoom RTMS webhooks // eslint-disable-next-line no-unused-vars app.use((err, req, res, next) => { const status = err.status || 500; const title = `Error ${err.status}`; // set locals, only providing error in development res.locals.message = err.message; res.locals.error = req.app.get('env') === 'development' ? err : {}; if (res.locals.error) dbg(`${title} %s`, err.stack); // render the error page res.status(status); res.render('error'); }); // redirect users to the home page if they get a 404 route app.get('*', (req, res) => res.redirect('/')); // start serving start(app, port) .then((server) => { // Initialize RTMS WebSocket after server starts initializeRTMSWebSocket(server); console.log('✅ RTMS WebSocket ready'); }) .catch(async (e) => { console.error(e); process.exit(1); }); export default app; ``` ### server/server.js ```javascript import http from 'http'; import debug from 'debug'; import { appName } from '../config.js'; import { testConnection, initializeIndices } from './helpers/elasticsearch.js'; import { setupInferenceEndpoints } from './helpers/setup-inference.js'; // Agent network (parked — requires working Kibana Agent Builder) // import { initializeTools } from './helpers/agent-manager.js'; const dbg = debug(`${appName}:http`); /** * Start the HTTP server * @param app - Express app to attach to * @param {String|number} port - local TCP port to serve from */ export async function start(app, port) { // Initialize Elasticsearch before starting server console.log('🔄 Connecting to Elasticsearch...'); const connected = await testConnection(); if (connected) { // Inference endpoints must exist before creating indices // (transcript_chunks needs jina_embeddings for semantic_text field) try { await setupInferenceEndpoints(); } catch (err) { console.error('⚠️ Inference endpoint setup failed:', err.message); } console.log('🔄 Initializing Elasticsearch indices...'); await initializeIndices(); console.log('✅ Elasticsearch ready'); // Agent network (parked — uncomment when Kibana Agent Builder is ready) // try { // await initializeTools(); // } catch (err) { // console.warn('⚠️ Agent network setup failed:', err.message); // } } else { console.warn( '⚠️ Elasticsearch not available - some features will be disabled' ); } // Create HTTP server const server = http.createServer(app); // let the user know when we're serving server.on('listening', () => { const addr = server.address(); const bind = typeof addr === 'string' ? `pipe ${addr}` : `http://localhost:${addr.port}`; dbg(`Listening on ${bind}`); }); server.on('error', async (error) => { if (error?.syscall !== 'listen') throw error; const bind = typeof port === 'string' ? `Pipe ${port}` : `Port ${port}`; // handle specific listen errors with friendly messages switch (error?.code) { case 'EACCES': throw new Error(`${bind} requires elevated privileges`); case 'EADDRINUSE': throw new Error(`${bind} is already in use`); default: throw error; } }); // Listen on provided port, on all network interfaces return server.listen(port); } ``` ### server/rtmsManager/index.js ```javascript /** * RTMSManager v2.0.0 - Developer-friendly Zoom RTMS SDK * * @example * ```javascript * import { RTMSManager } from 'rtms-manager-dev'; * * await RTMSManager.init({ * credentials: { * meeting: { clientId: '...', clientSecret: '...', secretToken: '...' } * }, * mediaTypes: RTMSManager.MEDIA.AUDIO | RTMSManager.MEDIA.TRANSCRIPT * }); * * RTMSManager.on('audio', ({ buffer, userName }) => { * console.log(`Audio from ${userName}: ${buffer.length} bytes`); * }); * * RTMSManager.on('error', (error) => { * console.error(error.toString()); // Pretty-printed error with causes and fixes * }); * ``` */ // Main class export { RTMSManager } from './RTMSManager.js'; // Error class export { RTMSError, ZOOM_STATUS_CODES, SDK_ERROR_CODES, } from './utils/RTMSError.js'; // Utilities export { FileLogger } from './utils/FileLogger.js'; export { RTMSConfigHelper } from './utils/RTMSConfigHelper.js'; export { RTMS_MEDIA_PARAMS } from './utils/rtmsMediaParams.js'; export { RTMSFlagHelper, TYPE_FLAGS } from './utils/RTMSFlagHelper.js'; // Lookup helpers export { getRtmsSessionState, getRtmsStreamState, getRtmsStopReason, getRtmsStatusCode, getHandshakeResponse, } from './utils/rtmsEventLookupHelper.js'; // Connection manager export { ActiveConnectionManager } from './ActiveConnectionManager.js'; // Message handlers export { RTMSMessageHandler } from './RTMSMessageHandler.js'; // Default export export { RTMSManager as default } from './RTMSManager.js'; ``` ### server/routes/index.js ```javascript import express from 'express'; import { handleError, sanitize } from '../helpers/routing.js'; import { contextHeader, getAppContext } from '../helpers/cipher.js'; import { getInstallURL } from '../helpers/zoom-api.js'; import { semanticSearch, esClient } from '../helpers/elasticsearch.js'; // Agent network (parked — requires Kibana 9.2 + working Agent Builder) // import { handleUserChat } from '../helpers/agent-manager.js'; import session from '../session.js'; const router = express.Router(); /* * Home Page - Zoom App Launch handler * this route is used when a user navigates to the deep link */ function isContextExpired(context) { const currentTime = Date.now(); return context.exp && context.exp < currentTime; } router.get('/', async (req, res, next) => { try { sanitize(req); const header = req.header(contextHeader); const context = header && getAppContext(header); if (!context) { return res.render('index', { isZoom: false, title: `Hello Browser`, }); } // Check if the context is valid and not expired if (isContextExpired(context)) { return res .status(401) .json({ error: 'Invalid or expired context' }); } return res.render('index', { isZoom: true, title: 'Context Assistant', chatHistory: [], chatTarget: 'Meeting', }); } catch (e) { next(handleError(e)); } }); /* * Install Route - Install the Zoom App from the Zoom Marketplace * this route is used when a user installs the app from the Zoom Client */ router.get('/install', session, async (req, res) => { const { url, state, verifier } = getInstallURL(); req.session.state = state; req.session.verifier = verifier; res.redirect(url.href); }); /* * Chat Route - Semantic search + Anthropic inference * (Agent network version parked — swap back when Kibana is ready) */ router.post('/chat', async (req, res, next) => { try { sanitize(req); const { message } = req.body; if (!message || typeof message !== 'string' || message.trim() === '') { return res .status(400) .json({ success: false, error: 'Message is required' }); } // 1. Semantic search across all transcript chunks let relevantChunks = []; try { relevantChunks = await semanticSearch(message, null, null, 5); } catch (err) { console.warn('Semantic search failed during chat:', err.message); } const chunksText = relevantChunks .map( (c) => `[${c.speaker_names?.join(', ') || 'Unknown'}]: ${c.text}` ) .join('\n---\n') .substring(0, 4000); // 2. Build prompt with context + question const prompt = `You are an AI assistant helping a meeting participant understand what's being discussed in their current meeting. Answer questions based on the transcript excerpts provided. ${ chunksText ? `Relevant transcript excerpts:\n${chunksText}\n\n` : 'No transcript data available yet.\n\n' }Question: ${message} Answer concisely and helpfully based on the meeting context provided. If there is no relevant context, say so.`; // 3. Call Anthropic via ES inference try { const completion = await esClient.transport.request({ method: 'POST', path: '/_inference/completion/anthropic_completion', body: { input: prompt }, }); const answer = completion.completion?.[0]?.result || 'No response generated.'; return res.json({ success: true, response: answer }); } catch (inferenceError) { console.warn('Inference endpoint error:', inferenceError.message); if (relevantChunks.length > 0) { const fallback = relevantChunks .map( (c) => `${c.speaker_names?.join(', ') || 'Unknown'}: "${ c.text }"` ) .join('\n\n'); return res.json({ success: true, response: `I found relevant transcript excerpts but the AI inference endpoint is not available. Here's what was said:\n\n${fallback}`, }); } return res.json({ success: true, response: 'The AI inference endpoint is not configured yet. Please set up the Elasticsearch Anthropic inference endpoint.', }); } } catch (e) { next(handleError(e)); } }); export default router; ``` ### public/js/index.js ```javascript import zoomSdk from '@zoom/appssdk'; // State let participants = []; let inImmersiveView = false; let isChatProcessing = false; let meetingId = null; let screenName = null; let pollInterval = null; // Get components const mainContent = document.getElementById('main'); const toggleButton = document.getElementById('toggle-view'); const immersiveContainer = document.getElementById('immersive'); const chatInput = document.getElementById('chat-input'); const chatSubmit = document.getElementById('chat-submit'); const chatMessages = document.getElementById('chat-messages'); const participant1Name = document.getElementById('participant-1-name'); const participant2Name = document.getElementById('participant-2-name'); const participant3Name = document.getElementById('participant-3-name'); const participant4Name = document.getElementById('participant-4-name'); const participantNames = [ participant1Name, participant2Name, participant3Name, participant4Name, ]; // Context summary elements (immersive view) const participantSummaries = [ document.getElementById('participant-1-summary'), document.getElementById('participant-2-summary'), document.getElementById('participant-3-summary'), document.getElementById('participant-4-summary'), ]; // Context buttons and containers const showContextButtons = [ document.getElementById('show-participant-1-context'), document.getElementById('show-participant-2-context'), document.getElementById('show-participant-3-context'), document.getElementById('show-participant-4-context'), ]; const hideContextButtons = [ document.getElementById('hide-participant-1-context'), document.getElementById('hide-participant-2-context'), document.getElementById('hide-participant-3-context'), document.getElementById('hide-participant-4-context'), ]; const contextContainers = [ document.getElementById('user-1-context'), document.getElementById('user-2-context'), document.getElementById('user-3-context'), document.getElementById('user-4-context'), ]; /** * Initialize context toggle buttons */ function initializeContextToggle() { // Set up show button listeners for (let i = 0; i < showContextButtons.length; i++) { if (showContextButtons[i] && contextContainers[i]) { showContextButtons[i].addEventListener('click', () => { contextContainers[i].classList.remove('is-hidden'); showContextButtons[i].classList.add('is-hidden'); }); } } // Set up hide button listeners for (let i = 0; i < hideContextButtons.length; i++) { if (hideContextButtons[i] && contextContainers[i]) { hideContextButtons[i].addEventListener('click', () => { contextContainers[i].classList.add('is-hidden'); showContextButtons[i].classList.remove('is-hidden'); }); } } } /** * Poll speakers from backend and update UI */ async function pollSpeakers() { if (!meetingId) return; try { const response = await fetch(`/api/meeting/${meetingId}/speakers`); if (!response.ok) return; const data = await response.json(); const speakers = data.speakers || []; if (speakers.length === 0) return; // Update immersive view participant summaries // Match speakers to participant slots by name for (let i = 0; i < participantNames.length; i++) { const slotName = participantNames[i]?.textContent?.trim(); if (!slotName || slotName === `Participant ${i + 1}`) continue; const matched = speakers.find( (s) => s.speaker_name && slotName .toLowerCase() .includes(s.speaker_name.toLowerCase()) ); if (matched && participantSummaries[i]) { participantSummaries[i].textContent = matched.context_summary || 'No summary yet.'; } } // Update timeline with topics from all speakers updateTimeline(speakers); } catch (error) { console.error('Failed to poll speakers:', error); } } /** * Update the timeline with real topics from speakers */ function updateTimeline(speakers) { const timeline = document.getElementById('timeline'); if (!timeline) return; // Collect all topics from all speakers const allTopics = []; for (const speaker of speakers) { if (speaker.topics && Array.isArray(speaker.topics)) { for (const topic of speaker.topics) { if (!allTopics.includes(topic)) { allTopics.push(topic); } } } } if (allTopics.length === 0) return; // Clear existing timeline items (keep the spacer div at the end) while (timeline.children.length > 1) { timeline.removeChild(timeline.firstChild); } // Add topic boxes const spacer = timeline.lastElementChild; for (const topic of allTopics) { const box = document.createElement('div'); box.className = 'box content ml-3'; const p = document.createElement('p'); p.textContent = topic; box.appendChild(p); timeline.insertBefore(box, spacer); } } /** * Start polling for speaker data */ function startPolling() { if (pollInterval) clearInterval(pollInterval); // Poll immediately, then every 5 seconds pollSpeakers(); pollInterval = setInterval(pollSpeakers, 5000); } // Initialize app (async () => { try { const configResponse = await zoomSdk.config({ capabilities: [ 'startRTMS', 'stopRTMS', 'runRenderingContext', 'closeRenderingContext', 'drawParticipant', 'clearParticipant', 'getMeetingParticipants', [truncated — 9073 more characters] ``` [44 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]