Project Info
Inspiration
Sen no Rikyu, the founder of the Japanese tea ceremony, explained its philosophy with this phrase: 一期一会 (ichi-go ichi-e): one time – one meeting. Life is about seizing every moment. Business pitches, lectures, and personal conversations are all fleeting—so make them count. The Problem: Remote meetings lack presence and depth—listeners get distracted, engagement is low, and tracking audience feedback is difficult. If you cannot perceive your audience's status, your meetings are left sub-par. Inspired by the Japanese tea ceremony philosophy, we set out to enhance real-time engagement in virtual meetings: The Solution: AI-driven avatars and immersive environments that make online meetings not just as good as in-person, but even better. Matcha and meditation apps like Otsuka ekkomi brought premium meditative experiences into the digital realm. We asked ourselves: How can we take this further and enable meaningful, premium interactions to be held online? We focused on the educational and healthcare benefits of AI companionship and metrics, and hence redefined virtual meetings to be more dynamic, engaging, and memorable. Thus, we came up with our real-time meeting analytics tools and responsive interactive experience we call Matcha Vibe. We create meetings that match your vibe. Everyone does Zoom meetings. We turn Zoom meetings into personalized experiences tailored to real-time consumer feedback. Why This Matters 90% of communication is non-verbal cues, which are hard to track online. We capture real-time engagement metrics to analyze each meeting participant’s mood, attention, and interaction levels, allowing businesses and educators to adjust dynamically. Seamless group experiences using Zoom’s API—turning meetings into engaging, responsive environments that feel as immersive as real-world interactions. What It Does Real-Time Speech & Gesture Analysis – The AI detects sentiment, engagement levels, and facial expressions to understand participant reactions. Personalized AI Presence – The avatar adapts dynamically depending on the meeting flow from analytics. From Mindfulness to Business Pitches – Whether it's a tea ceremony to guide relaxation or an AI-driven sales coach assessing audience attention, this system is open for personalized interactions for any situation. By integrating AI-driven adaptability, MatchaVibe transforms how we engage in virtual meetings, making online collaboration more immersive and impactful. How We Built It Leveraging Zoom’s API, we integrated real-time analytics to track user engagement, expressions, and speech sentiment. AI-powered avatars were developed using Groq for fast AI processing, ensuring smooth and intelligent avatar behavior. Dynamic backgrounds were created using Luma Apps, adapting the meeting atmosphere in real-time. Facial expression tracking was implemented to recognize stress levels and engagement, providing immediate visual or verbal responses. Groq’s multimodal capabilities powered seamless text, vision, and audio processing, making the AI avatar highly responsive across different forms of input. Gemini 2.0 Flash for sentiment analysis to interpret real-time user feedback. ElevenLabs for AI-driven conversational speech and text-to-speech for seamless user interactions. Luma AI transforms text, images, and video seamlessly to create immersive meeting environments and interactive backgrounds that react to real-time consumer metrics. OpenAI APIs enhance AI-generated conversation, sentiment-driven avatar responses, and dynamic adaptability. Challenges We Ran Into Balancing AI responsiveness with natural human interaction—we refined avatar expressions to feel intuitive rather than robotic. Processing real-time engagement metrics efficiently—Groq allowed us to minimize latency while running AI analytics. Ensuring accessibility and inclusivity—different cultures have different stress triggers and relaxation cues, requiring adaptive feedback mechanisms. Optimizing multimodal AI coordination—seamless real-time text, vision, and audio processing presented computational challenges. Integrating multiple AI models (Gemini, ElevenLabs, Groq) into a single pipeline while maintaining efficiency and reducing response lag. Accomplishments That We're Proud Of Created a fully interactive AI-driven avatar that dynamically reacts to user speech, emotions, and gestures in Zoom meetings. Successfully integrated real-time sentiment and behavior analysis to adjust meeting atmosphere and enhance engagement. Built an AI presence that goes beyond text and voice responses, setting a new standard for digital interaction. Utilized Groq’s multimodal processing to enhance responsiveness across multiple sensory inputs. Implemented Gemini’s sentiment analysis to personalize meeting engagement. Leveraged ElevenLabs AI voice processing to make virtual meetings more interactive and human-like. Enhanced real-time media interactions using Luma AI for seamless content adaptation. By enhancing collaboration with AI-driven insights, Matcha Vibe exemplifies how Zoom’s tools can push the boundaries of digital interaction and connect people more effectively. What’s Next for MatchaVibe? Expanding AI Emotion & Gesture Recognition – Detect stress, disengagement, and excitement with even greater accuracy. Integrating Voice Analysis for Deeper Insights – Use pitch and pacing analysis to understand user confidence and engagement. Customizable AI Avatars for Different Use Cases – Expand beyond tea ceremonies to education, business, and therapy. AI Coaching & Learning Tools – Guide users through public speaking, sales pitches, and mindfulness exercises using AI-driven feedback. Matcha Vibe is redefining virtual engagement, blending AI-driven analytics, personalized avatars, and real-time interactions to create truly immersive digital experiences. From tea ceremonies to boardrooms, we make online meetings more human.
Realtime Media Streams Mock Server -- V.1.0
Confidential under NDA - Do Not Distribute The information in this document is confidential and requires an NDA. It is intended only for partners in the Zoom RTMS Beta Developers program. Participation in the RTMS Beta Offering, including access to and use of these RTMS Beta Offering materials, is subject to Zoom's Beta Program - Terms of Use.
Repository: github.com/zoom/rtms-mock-server-sample
This is a mock server that simulates the WebSocket-based streaming of Zoom's Realtime Media Streams (RTMS). It provides a complete development and testing environment for client-server interactions, including media streaming, signaling, and webhook management.
Documentation: Realtime Media Streams - Beta Developer Documentation
Video guide: Testing the RTMS mock server
Sample client
A sample client (Express server) is available at ./client to help test this mock server. The client implements connection handling and provides a user interface for testing different media formats.
Sample client features:
- Webhook endpoint implementation
- WebSocket connection handling
- Media streaming controls
- Incoming real time data logs
Installation & setup
This app requires FFmpeg and Node.js version 14 or higher.
The app can be run locally by cloning and installing packages with npm or on Docker.
npm To setup with npm, install dependencies and run the app:
cd rtms-mock-server-sample
# Install dependencies
npm install
# Start the server
npm start
Docker To setup with Docker, run the following:
cd rtms-mock-server-sample
# Option 1: Using docker-compose (recommended)
docker-compose up -d
# Option 2: Manual docker commands
# Build Docker image
docker build -t rtms-mock-server .
# Run the container
docker run -d \
-p 9092:9092 \
-p 8081:8081 \
-v $(pwd)/data:/app/data \
--name rtms-mock-server \
rtms-mock-server
# View logs
docker logs -f rtms-mock-server
To stop the container:
docker stop rtms-mock-server
To restart the container:
docker start rtms-mock-server
Using the sample client
Start the server (npm or Docker) and open the mock server at http://localhost:9092. The sample client at ./client can now be used to consume media from the mock server.
In a new terminal, run the sample client:
node client/server.js
This opens up a server at localhost:8000. For webhook validation, the client will need to be exposed to the internet with a tunnel, like ngrok.
ngrok http 8000
The ngrok URL will be used to validate the webhook endpoint. Copy your URL and paste it into the webhook URL field on the mock server (http://localhost:9092). Click validate. In the RTMS server and client you'll see confirmation of the validation.
You can now start a meeting and start streaming media to the client.
Click Start Meeting and provide camera/microphone permissions. The client will start receiving media. Resume, Stop, and Start RTMS to control the media stream.
Media packets are sent to the client every 100ms. The client will log incoming packets to the console.
Creating your own client
To start, you'll need to create a webhook receiver to handle incoming meeting.rtms.started events when streams are available. You'll also need to validate the webhook URL. You can find the webhook verification token in rtms_credentials.json.
The mock server will send the following POST requests to your webhook endpoint:
{
"event": "endpoint.url_validation",
"payload": {
"plainToken": "abc123"
}
}
The webhook endpoint should respond with the following:
{
"plainToken": "abc123",
"encryptedToken": "encrypted_token_hash"
}
When the webhook is validated and the meeting starts, you'll receive a meeting.rtms.started webhook payload.
Meeting Started Webhook Payload:
{
"event": "meeting.rtms.started",
"payload": {
"operator_id": "user123",
"object": {
"meeting_uuid": "uuid",
"rtms_stream_id": "stream_id",
"server_urls": "server_urls"
}
}
}
Handling WebSocket Connections to receive RTMS data
Once you receive the server urls in the webhook payload, you need to open a websocket connection with the server url, and send a handshake request in the following format:
Handshake Request (Client → Mock Server):
{
"msg_type": "SIGNALING_HAND_SHAKE_REQ",
"protocol_version": 1,
"meeting_uuid": "meeting_uuid",
"rtms_stream_id": "stream_id",
"signature": "hmac_sha256_signature"
}
Note: The signature field should be generated using HMAC-SHA256 with the following pattern:
signature = HMAC-SHA256(
key: client_secret,
message: client_id + meeting_uuid + rtms_stream_id
)
The client_secret and other credentials can be found in data/rtms_credentials.json.
Handshake Response (Mock Server → Client):
The media urls are returned in the handshake response if the handshake is successful:
{
"msg_type": "SIGNALING_HAND_SHAKE_RESP",
"status": "STATUS_OK",
"media_urls": {
"all": "ws://localhost:8081/all",
"video": "ws://localhost:8081/video",
"audio": "ws://localhost:8081/audio",
"transcript": "ws://localhost:8081/transcript"
}
}
The signaling socket will also send you the following messages:
Keep-Alive Request (Mock Server → Client):
{
"msg_type": "KEEP_ALIVE_REQ",
"timestamp": 1234567890
}
Keep-Alive Response (Client → Mock Server):
{
"msg_type": "KEEP_ALIVE_RESP",
"timestamp": 1234567890
}
If you miss three consecutive keep-alive requests, the connection will be closed.
Session State Update:
{
"msg_type": "SESSION_STATE_UPDATE",
"state": "STARTED", // or "PAUSED", "RESUMED", "STOPPED"
"stop_reason": "reason",
"timestamp": 1234567890
}
Media WebSocket Messages:
When you open a websocket connection wtih the media URLs you need to send the following handshake request:
{
"msg_type": "DATA_HAND_SHAKE_REQ",
"protocol_version": 1,
"meeting_uuid": "meeting_uuid",
"rtms_stream_id": "stream_id",
"signature": "hmac_sha256_signature",
"payload_encryption": false
}
Note: The signature is generated using the same method as the signaling handshake.
The mock server will respond with the following message:
{
"msg_type": "DATA_HAND_SHAKE_RESP",
"status": "STATUS_OK"
}
The media websocket will send you the following messages depending on which media type you are subscribed to:
Video Data Format:
{
"msg_type": "MEDIA_DATA_VIDEO",
"content": {
"user_id": 0,
"data": "base64_encoded_video_frame",
"timestamp": 1234567890
}
}
Audio Data Format:
{
"msg_type": "MEDIA_DATA_AUDIO",
"content": {
"user_id": 0,
"data": "base64_encoded_audio_chunk",
"timestamp": 1234567890
}
}
Transcript Data Format:
{
"msg_type": "MEDIA_DATA_TRANSCRIPT",
"content": {
"user_id": 0,
"data": "transcribed text",
"timestamp": 1234567890
}
}
License
See LICENSE.md file for details.
Support
For questions or help needed, join us on the Realtime Media Streams category on the Zoom Developer Forum. If you need access to this, please reach out in your Zoom Team Chat channel.
Developer Forum thread: Realtime Media Streams Mock Server
Analysis
View
Metric
- 21
- 3
- 2
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- ExpressIn code
- Google GeminiIn code
- HTMLIn code
- JavaScriptIn code
- OpenAIIn code
- DockerClaimed
- Node.jsClaimed
6 of 8 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
- CursorConfig
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
196 KB
Source files
25
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
treehacks2025/mockRTMSserver
32 files · 246 KB · @ d1d3f47
Structure
API & routing
9 files · 28%Request entry points: routes, handlers and controllers.
Application logic
3 files · 9%Domain rules, services and shared utilities.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- JavaScript81%
- HTML8%
- CSS6%
- Markdown5%
- YAML0%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
package.json
npm · 9- @google/generative-ai
- cors
- dotenv
- express
- form-data
- multer
- node-fetch
- openai
- ws
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
This project’s features have not been analysed yet.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.