# Project export: Tommy Chat This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Next Gen Talking Tom: An AI moderator that keeps multiplayer game chats fun and safe. Easy plug-and-play API for developers.Tom handles toxicity, engages players, and adds personality to every session - Devpost: https://devpost.com/software/tommy-chat - GitHub: https://github.com/Rushabh-Runwal/calhacks - Video: https://www.youtube.com/embed/playlist?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) β€” Rushabh Runwal (26 commits), dishajadav12 (8 commits), Udayan Atreya (3 commits) ## Devpost submission (written by the team) ### Inspiration Multiplayer gaming is all about connection and fun, but toxic chat can ruin the experience. We've all seen itβ€”arguments escalate, new players feel unwelcome, and moderators can't scale to every game session. We wanted to create an AI companion that could keep gaming communities positive, engaging, and safe without the need for expensive human moderation. Inspired by the nostalgic Talking Tom character, we built an AI moderator that's not just functionalβ€”it's genuinely fun to interact with. ### What it does Talking Tom Chat is an AI-powered chat moderator that game developers can integrate into their multiplayer games. Tom acts as a friendly companion who: Moderates conversations - Detects and gently defuses tension before conflicts escalate Engages players - Jumps into food discussions, tells personalized stories featuring all players, and playfully echoes messages Welcomes newcomers - Greets new players and helps them feel included Facilitates decisions - Helps groups make choices when discussions get stuck Speaks naturally - Uses Fish Audio TTS to deliver emotion-rich voice responses that match the conversation's mood Tom knows when to speak and when to stay silent, making him feel like a natural part of the conversation rather than an intrusive bot. ### How we built it Tech Stack: Backend: Python FastAPI with Socket.IO for real-time WebSocket communication AI Brain: Anthropic's Claude AI (Sonnet 4.5) with a carefully crafted system prompt that defines Tom's personality and behavior Voice: Fish Audio TTS API for natural, emotion-tagged speech synthesis Frontend: Next.js with TypeScript, React, and Tailwind CSS for a responsive, modern UI Deployment: Backend on Render.com, Frontend on Vercel Architecture: Real-time chat messages flow through Socket.IO Claude AI analyzes context and decides whether Tom should respond If responding, Claude generates speech with emotion tags (e.g., (happy), (excited), (calm)) Fish Audio converts the tagged text into natural-sounding voice Audio streams back to all players in the room Key Features: Priority-based response system (13 different triggers) Context-aware conversation analysis Emotion-tagged speech for authentic voice delivery Real-time multiplayer room management Cross-platform responsive design ### Challenges we ran into Getting the AI to "shut up" - Our biggest challenge was teaching Tom when NOT to talk. Early versions were too chatty. We solved this with a sophisticated priority system and a [[NO_OUTPUT]] token that tells Tom to stay silent. Getting the AI to "shut up" - Our biggest challenge was teaching Tom when NOT to talk. Early versions were too chatty. We solved this with a sophisticated priority system and a [[NO_OUTPUT]] token that tells Tom to stay silent. Fish Audio integration - Working with emotion-tagged TTS was new territory. We had to carefully map conversation contexts to appropriate emotion tags and ensure the tags were properly formatted according to Fish Audio's strict requirements. Fish Audio integration - Working with emotion-tagged TTS was new territory. We had to carefully map conversation contexts to appropriate emotion tags and ensure the tags were properly formatted according to Fish Audio's strict requirements. Real-time audio streaming - Serving audio files efficiently while maintaining low latency across Socket.IO connections required careful backend optimization and proper CORS configuration. Real-time audio streaming - Serving audio files efficiently while maintaining low latency across Socket.IO connections required careful backend optimization and proper CORS configuration. Deployment complexity - Managing separate frontend and backend deployments with proper environment variables, WebSocket support, and CORS policies across Vercel and Render took careful configuration. Deployment complexity - Managing separate frontend and backend deployments with proper environment variables, WebSocket support, and CORS policies across Vercel and Render took careful configuration. Prompt engineering - Crafting a system prompt that balanced personality, helpfulness, and silence was an iterative process. We went through many versions to get Tom's voice just right. Prompt engineering - Crafting a system prompt that balanced personality, helpfulness, and silence was an iterative process. We went through many versions to get Tom's voice just right. ### Accomplishments we're proud of Created a genuinely useful tool - Tom isn't just a gimmick; he actually improves group chat dynamics Achieved natural voice interactions - The emotion-tagged speech makes Tom feel alive and responsive Built a production-ready API - The backend is scalable and ready for real game integrations Designed an intuitive UX - Players can jump into rooms and start chatting immediately Implemented smart context awareness - Tom understands ongoing conversations and responds appropriately Successfully deployed - Full stack application running smoothly in production ### What we learned AI needs guardrails - Sophisticated gating logic is essential for AI companions to feel natural rather than annoying Voice changes everything - Adding emotion-rich TTS transformed Tom from a text bot into a character people actually enjoy Real-time is hard - WebSocket management, connection handling, and audio streaming require careful architecture Prompt engineering is an art - Small changes in system prompts can dramatically affect AI behavior User experience matters - Even powerful AI needs a great interface to be useful Deployment strategy - Separating frontend and backend into dedicated platforms (Vercel + Render) simplifies scaling and maintenance ### What's next Short-term: Game integrations - Partner with indie game developers to test Tom in real gaming environments Enhanced personality modes - Let developers customize Tom's personality for different game genres Analytics dashboard - Give developers insights into chat health and Tom's engagement metrics Multi-language support - Expand beyond English to reach global gaming communities Long-term: Voice commands - Let players interact with Tom using voice input Custom training - Allow developers to fine-tune Tom for their specific community culture Sentiment analysis - Provide real-time toxicity scores and proactive moderation Plugin marketplace - Create an ecosystem where developers can add custom Tom behaviors Mobile SDK - Native iOS/Android libraries for mobile game integration Advanced storytelling - Dynamic quest generation and interactive narratives featuring players Vision: Transform Tommy Chat into the go-to AI companion for every multiplayer game, making online gaming communities safer, more fun, and more engagingβ€”one conversation at a time. Try it live: https://tommy-chat.vercel.app/ GitHub: https://github.com/Rushabh-Runwal/calhacks/tree/main Backend API: https://backend-talking-tom.onrender.com ## README (from the GitHub repository) # 🐱 Talking Tom Chat A real-time, room-based group chat where you and friends can talk to an AI β€œTalking Tom” via text or voice. Messages are delivered instantly over Socket.IO. Voice notes are auto-transcribed and Tom replies with natural text-to-speech audio. Tom follows prompt-guided behavior: he speaks when addressed or helpful, avoids interrupting active threads, nudges the room if it’s quiet, and applies simple moderation. - Low-latency, multiplayer rooms - Text + voice messaging (VAD recording, ASR transcription, TTS replies) - Context-aware AI persona with speaking rules and cooldowns - Works on desktop and mobile ## Features - πŸ’¬ Real-time multiplayer chat (Socket.IO) - πŸŽ™οΈ Voice input with automatic speech detection - πŸ”Š AI voice responses (Fish Audio TTS) + transcription (ASR) - πŸ€– JanitorAI-powered character with context-aware logic - πŸ‘₯ Room-based conversations with shareable codes - πŸ›‘οΈ Simple moderation prompts and anti-interruption behavior - πŸ“¦ Audio caching and HTTP streaming for generated speech ## Tech Stack - Backend: Python 3.10+, FastAPI, python-socketio, Uvicorn - Frontend: Next.js, React, TypeScript, Tailwind CSS - AI: JanitorAI - Audio: Fish Audio SDK (TTS + ASR), FFmpeg ## Prerequisites - Python 3.10+ - Node.js 18+ - FFmpeg installed and on PATH - Docker (optional, for Compose) ## Quick Start ### Option 1: Docker Compose (recommended) ```powershell # In repo root Copy-Item backend/.env.example backend/.env Copy-Item frontend/.env.example frontend/.env.local # Edit both env files with your API keys and settings docker-compose up --build ``` - Frontend: http://localhost:3000 - Backend API: http://localhost:8000 - WebSocket: ws://localhost:8000/socket.io/ ### Option 2: Manual setup Backend (Windows PowerShell): ```powershell cd backend Copy-Item .env.example .env # edit with your API keys python -m venv .venv . .\.venv\Scripts\Activate.ps1 pip install -r requirements.txt python start_server.py ``` Frontend: ```powershell cd frontend Copy-Item .\.env.example .\.env.local # edit if needed npm install npm run dev ``` - Frontend: http://localhost:3000 - Backend API: http://localhost:8000 - WebSocket: ws://localhost:8000/socket.io/ ## Usage 1) Enter a username, then create or join a room via code. 2) Send text or hold-to-record a voice message. 3) Tom replies in text and usually with synthesized voice. 4) Say his name to engage directly; he avoids interrupting active 1:1 threads. ## API Overview HTTP - GET `/` β†’ Health/info - GET `/rooms` β†’ List rooms - POST `/rooms/create` β†’ Create a room - GET `/audio/{filename}` β†’ Stream cached TTS audio Socket.IO (ws://localhost:8000/socket.io/) - Client β†’ Server: - `joinRoom` { roomId, username } - `sendMessage` { roomId, username, content } - `sendVoiceMessage` { roomId, username, audio: base64 } - Server β†’ Client: - `userJoined`, `userLeft`, `roomUsers` - `roomMessages`, `newMessage` - `error` ## Tom’s Behavior (prompt-guided) - Responds when addressed or when helpful to the group - Avoids interrupting ongoing 1:1 exchanges - Optional β€œsilence nudge” after quiet periods - Brief, friendly voice style; minimal fillers; no cat sounds in moderation - Simple moderation: immediate warning on harassment; repeat β†’ removal Tune timings, cooldowns, and wording in the prompt/config (e.g., `config/ai_prompt.py`) ## Environment Variables Backend (`backend/.env`) ```env HOST=0.0.0.0 PORT=8000 FISH_API_KEY=your_fish_audio_api_key JANITOR_API_KEY=your_janitor_ai_api_key ``` Frontend (`frontend/.env.local`) ```env NEXT_PUBLIC_API_URL=http://localhost:8000 ``` ## Project Structure ``` . β”œβ”€β”€ backend/ # FastAPI + Socket.IO server β”‚ β”œβ”€β”€ config/ # AI prompt and settings β”‚ β”œβ”€β”€ models/ # Pydantic models β”‚ β”œβ”€β”€ services/ # JanitorAI + Fish Audio integrations β”‚ β”œβ”€β”€ audio_cache/ # Generated audio files β”‚ β”œβ”€β”€ main.py # App entry β”‚ β”œβ”€β”€ start_server.py # Uvicorn launcher β”‚ └── requirements.txt β”‚ β”œβ”€β”€ frontend/ # Next.js app β”‚ β”œβ”€β”€ src/ # app/, components/, types/ β”‚ β”œβ”€β”€ public/ β”‚ └── package.json β”‚ └── docker-compose.yml ``` ## Troubleshooting - Missing API keys: set `FISH_API_KEY` and `JANITOR_API_KEY` in `backend/.env`. - FFmpeg not found: install and add to PATH, then restart terminal. - CORS/Socket errors: confirm `NEXT_PUBLIC_API_URL` points to your backend URL. - No audio playback: ensure `audio_cache/` is writable and files are being created. ## License MIT # Backend URL: https://backend-tommy.onrender.com ## Detected evidence (automated analysis) Indexed codebase: 27 recognized source files, 94 KB. - CSS (language) β€” detected in the code - FastAPI (technology) β€” detected in the code - Next.js (technology) β€” detected in the code - Python (language) β€” detected in the code - React (technology) β€” detected in the code - Tailwind CSS (technology) β€” detected in the code - TypeScript (language) β€” detected in the code ## Codebase structure (from repository index) ### Files (62 of 62) ``` .gitignore backend/.env.example backend/.gitignore backend/config/ai_prompt.py backend/Dockerfile backend/main.py backend/models/chat.py backend/pyproject.toml backend/README.md backend/render.yaml backend/services/fish_audio.py backend/services/janitor_ai.py backend/start_server.py backend/uv.lock DEPLOY.md docker-compose.yml frontend/.env.example frontend/Dockerfile frontend/eslint.config.mjs frontend/next-env.d.ts frontend/next.config.ts frontend/package.json frontend/postcss.config.mjs frontend/public/vad/ort-wasm-simd-threaded.asyncify.mjs frontend/public/vad/ort-wasm-simd-threaded.asyncify.wasm frontend/public/vad/ort-wasm-simd-threaded.jsep.mjs frontend/public/vad/ort-wasm-simd-threaded.jsep.wasm frontend/public/vad/ort-wasm-simd-threaded.mjs frontend/public/vad/ort-wasm-simd-threaded.wasm frontend/public/vad/ort.all.bundle.min.mjs frontend/public/vad/ort.all.min.mjs frontend/public/vad/ort.all.mjs frontend/public/vad/ort.bundle.min.mjs frontend/public/vad/ort.min.mjs frontend/public/vad/ort.mjs frontend/public/vad/ort.node.min.mjs frontend/public/vad/ort.wasm.bundle.min.mjs frontend/public/vad/ort.wasm.min.mjs frontend/public/vad/ort.wasm.mjs frontend/public/vad/ort.webgl.min.mjs frontend/public/vad/ort.webgl.mjs frontend/public/vad/ort.webgpu.bundle.min.mjs frontend/public/vad/ort.webgpu.min.mjs frontend/public/vad/ort.webgpu.mjs frontend/public/vad/silero_vad_legacy.onnx frontend/public/vad/silero_vad_v5.onnx frontend/README.md frontend/src/app/globals.css frontend/src/app/layout.tsx frontend/src/app/page.tsx frontend/src/app/room/[roomId]/page.tsx frontend/src/components/Background.tsx frontend/src/components/ChatInput.tsx frontend/src/components/ChatMessage.tsx frontend/src/components/ContinuousRecorder.tsx frontend/src/types/chat.ts frontend/start_frontend.sh frontend/tailwind.config.ts frontend/tsconfig.json README.md start-backend.sh start-frontend.sh ``` ### Dependencies - backend/pyproject.toml: black@>=23.0.0, fastapi@>=0.111.0, ffmpeg-python@>=0.2.0, fish-audio-sdk@>=0.1.13, httpx@>=0.27.0, isort@>=5.12.0, pytest@>=7.4.0, pytest-asyncio@>=0.21.0, python-dotenv@>=1.0.1, python-socketio@>=5.11.2, uvicorn[standard]@>=0.29.0 - frontend/package.json: @ricky0123/vad-web@^0.0.28, @types/node@^20, @types/react@^19, @types/react-dom@^19, autoprefixer@^10.4.16, eslint@^9, eslint-config-next@16.0.0, nanoid@^5.1.6, next@16.0.0, postcss@^8.4.32, react@19.2.0, react-dom@19.2.0, socket.io@^4.8.1, socket.io-client@^4.8.1, tailwindcss@^3.4.0, typescript@^5 ### Recent commits (newest first) - Update backend URL reference - Fix: Readme - Add food talk feature and update AI prompt priorities - Replace emoji with Tom face image and update responsive sizing - update image - Simplify Socket.IO connection and fix environment variable - Simplify Socket.IO connection and fix environment variable - Merge pull request #2 from Rushabh-Runwal/feature/final - Add Background component with layered SVG images for enhanced visuals - Fix audio URL to use backend URL in production - Fix Socket.IO connection to use NEXT_PUBLIC_API_URL in production - Remove standalone output config for Vercel deployment - Remove vercel.json - configure via Vercel dashboard instead - Fix vercel.json: Remove redundant build command - Add uv.lock for reproducible builds and fix Dockerfile - Add deployment configs for Vercel (frontend) + Render (backend) - Clean up: Remove unnecessary files and simplify README - Restructure project: Separate frontend and backend into dedicated folders - Fix Socket.IO connection: Update frontend to connect to port 8000 and fix backend to use socket_app - Merge branch 'python-version' of https://github.com/Rushabh-Runwal/calhacks into python-version ## Key source files (fetched from GitHub, selected and truncated for size) ### DEPLOY.md ```markdown # Deployment Guide: Vercel + Render ## Overview - **Frontend**: Deployed on Vercel (automatic deployments from GitHub) - **Backend**: Deployed on Render (Docker-based deployment) --- ## Part 1: Deploy Backend to Render ### Step 1: Sign up for Render 1. Go to https://render.com 2. Sign up with your GitHub account ### Step 2: Create a New Web Service 1. Click **"New +"** β†’ **"Web Service"** 2. Connect your GitHub repository: `Rushabh-Runwal/calhacks` 3. Configure the service: - **Name**: `multiplayer-ai-backend` - **Region**: Choose closest to your users - **Branch**: `main` - **Root Directory**: `backend` - **Runtime**: `Docker` - **Docker Command**: Leave empty (uses Dockerfile CMD) ### Step 3: Add Environment Variables Click "Advanced" and add these environment variables: - `HOST` = `0.0.0.0` - `PORT` = `8000` - `FISH_API_KEY` = `b34cff9e6fed4b8cb414b3ed4356014d` - `JANITOR_API_KEY` = `calhacks2047` ### Step 4: Deploy 1. Click **"Create Web Service"** 2. Wait for the build to complete (5-10 minutes) 3. Once deployed, copy your backend URL (e.g., `https://multiplayer-ai-backend.onrender.com`) **Note**: Render free tier may spin down after inactivity. First request might be slow. --- ## Part 2: Deploy Frontend to Vercel ### Step 1: Sign up for Vercel 1. Go to https://vercel.com 2. Sign up with your GitHub account ### Step 2: Import Project 1. Click **"Add New..."** β†’ **"Project"** 2. Import `Rushabh-Runwal/calhacks` repository 3. Configure: - **Framework Preset**: Next.js - **Root Directory**: `frontend` - **Build Command**: `npm run build` - **Output Directory**: `.next` - **Install Command**: `npm install` ### Step 3: Add Environment Variable In the "Environment Variables" section: - **Key**: `NEXT_PUBLIC_API_URL` - **Value**: `https://your-backend-url.onrender.com` (from Step 4 above) - **Environment**: Production, Preview, and Development ### Step 4: Deploy 1. Click **"Deploy"** 2. Wait for deployment (2-3 minutes) 3. Your app will be live at `https://your-project.vercel.app` ### Step 5: Set Custom Domain (Optional) 1. Go to your project settings on Vercel 2. Click **"Domains"** 3. Add your custom domain and follow DNS instructions --- ## Part 3: Update Backend CORS After deploying frontend, update your backend CORS settings to allow your Vercel domain. In `backend/main.py`, update: ```python sio = socketio.AsyncServer( async_mode="asgi", cors_allowed_origins=[ "https://your-project.vercel.app", "http://localhost:3000", "*" # Remove this in production ] ) ``` Then commit and push to trigger a redeploy on Render. --- ## Verification ### Backend Health Check Visit: `https://your-backend-url.onrender.com/` You should see: ```json { "message": "Talking Tom Chat API", "status": "running" } ``` ### Frontend Visit: `https://your-project.vercel.app` You should see the chat interface. ### Test Socket.IO Connection 1. Open browser console on your fronte [truncated β€” 2456 more characters] ``` ### docker-compose.yml ```yaml version: '3.8' services: backend: build: context: ./backend dockerfile: Dockerfile ports: - "8000:8000" environment: - HOST=0.0.0.0 - PORT=8000 - FISH_API_KEY=${FISH_API_KEY} - JANITOR_API_KEY=${JANITOR_API_KEY} volumes: - ./backend/audio_cache:/app/audio_cache restart: unless-stopped healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/"] interval: 30s timeout: 10s retries: 3 frontend: build: context: ./frontend dockerfile: Dockerfile args: - NEXT_PUBLIC_API_URL=http://localhost:8000 ports: - "3000:3000" environment: - NEXT_PUBLIC_API_URL=http://backend:8000 depends_on: - backend restart: unless-stopped volumes: audio_cache: ``` ### backend/Dockerfile ``` # Backend Dockerfile for Render FROM python:3.12-slim WORKDIR /app # Install system dependencies RUN apt-get update && apt-get install -y \ ffmpeg \ curl \ && rm -rf /var/lib/apt/lists/* # Install uv COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv # Copy dependency files COPY pyproject.toml uv.lock ./ # Install dependencies RUN uv sync --no-dev # Copy application code COPY . . # Create audio cache directory RUN mkdir -p audio_cache # Expose port EXPOSE 8000 # Start server CMD ["uv", "run", "python", "start_server.py"] ``` ### frontend/package.json ``` { "name": "multiplayer-ai-chat", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@ricky0123/vad-web": "^0.0.28", "nanoid": "^5.1.6", "next": "16.0.0", "react": "19.2.0", "react-dom": "19.2.0", "socket.io": "^4.8.1", "socket.io-client": "^4.8.1" }, "devDependencies": { "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "autoprefixer": "^10.4.16", "eslint": "^9", "eslint-config-next": "16.0.0", "postcss": "^8.4.32", "tailwindcss": "^3.4.0", "typescript": "^5" } } ``` ### backend/pyproject.toml ``` [tool.uv] dev-dependencies = [] [tool.uv.sources] "fastapi-poe" = { "git" = "https://github.com/fastapi-poe/fastapi-poe.git" } [project] name = "multiplayer-ai-chat-python" version = "0.1.0" description = "A multiplayer chat application with an AI character." dependencies = [ "fastapi>=0.111.0", "python-socketio>=5.11.2", "uvicorn[standard]>=0.29.0", "python-dotenv>=1.0.1", "fish-audio-sdk>=0.1.13", "ffmpeg-python>=0.2.0", "httpx>=0.27.0" ] requires-python = ">=3.10" [project.optional-dependencies] dev = [ "pytest>=7.4.0", "pytest-asyncio>=0.21.0", "black>=23.0.0", "isort>=5.12.0", ] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["."] [tool.black] line-length = 88 target-version = ['py311'] [tool.isort] profile = "black" line_length = 88 ``` ### frontend/Dockerfile ``` # Frontend Dockerfile (for Docker deployments) FROM node:20-alpine AS base # Install dependencies only when needed FROM base AS deps RUN apk add --no-cache libc6-compat WORKDIR /app COPY package.json package-lock.json* ./ RUN npm ci # Rebuild the source code only when needed FROM base AS builder WORKDIR /app COPY --from=deps /app/node_modules ./node_modules COPY . . # Set environment variables for build ARG NEXT_PUBLIC_API_URL ENV NEXT_PUBLIC_API_URL=$NEXT_PUBLIC_API_URL RUN npm run build # Production image, copy all the files and run next FROM base AS runner WORKDIR /app ENV NODE_ENV=production RUN addgroup --system --gid 1001 nodejs RUN adduser --system --uid 1001 nextjs COPY --from=builder /app/public ./public # Set the correct permission for prerender cache RUN mkdir .next RUN chown nextjs:nodejs .next # Automatically leverage output traces to reduce image size COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./ COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static USER nextjs EXPOSE 3000 ENV PORT=3000 ENV HOSTNAME="0.0.0.0" CMD ["node", "server.js"] ``` ### backend/main.py ```python import os import secrets import asyncio import base64 from datetime import datetime from typing import Dict, List from fastapi import FastAPI, HTTPException from fastapi.staticfiles import StaticFiles from fastapi.responses import FileResponse import socketio from models.chat import Message, User, Room from services.fish_audio import FishAudioClient from services.janitor_ai import JanitorAIClient import uuid app = FastAPI(title="Talking Tom Chat API", version="1.0.0") sio = socketio.AsyncServer(async_mode="asgi", cors_allowed_origins="*") # Serve static audio files app.mount("/audio", StaticFiles(directory="audio_cache"), name="audio") socket_app = socketio.ASGIApp(sio, app) # Initialize clients fish_audio_client = FishAudioClient() janitor_ai_client = JanitorAIClient() rooms: Dict[str, Room] = {} def generate_room_id() -> str: return secrets.token_urlsafe(4).upper()[:6] def generate_message_id() -> str: return secrets.token_hex(12) @sio.event async def connect(sid, environ): print(f"Client connected: {sid}") @sio.event async def disconnect(sid): print(f"Client disconnected: {sid}") rooms_to_update = list(rooms.items()) for room_id, room in rooms_to_update: room.users = [user for user in room.users if user.id != sid] if not room.users: del rooms[room_id] else: await sio.emit("userLeft", {"userId": sid}, room=room_id) @sio.event async def joinRoom(sid, data): try: room_id = data.get("roomId") username = data.get("username") if not room_id or not username: await sio.emit("error", {"message": "Room ID and username are required"}, room=sid) return if room_id not in rooms: rooms[room_id] = Room(id=room_id) user = User(id=sid, username=username, room_id=room_id) rooms[room_id].users.append(user) await sio.enter_room(sid, room_id) await sio.emit("userJoined", user.dict(), room=room_id, skip_sid=sid) await sio.emit("roomUsers", [user.dict() for user in rooms[room_id].users], room=sid) await sio.emit("roomMessages", [msg.dict() for msg in rooms[room_id].messages], room=sid) print(f"User {username} joined room {room_id}") except Exception as e: print(f"Error in joinRoom: {e}") await sio.emit("error", {"message": "Failed to join room"}, room=sid) @sio.event async def sendMessage(sid, data): try: content = data.get("content", "").strip() username = data.get("username") room_id = data.get("roomId") if not content or not username or not room_id: return user_message = Message( id=generate_message_id(), content=content, username=username, timestamp=int(datetime.now().timestamp() * 1000), is_ai=False ) if room_id in rooms: rooms[room_id].messages.append(user_message) await sio.emit("newMessage", user_message.dict(), room=room_id) await generate_ai_response(room_id, username) except Exception as e: print(f"Error in sendMessage: {e}") @sio.event async def sendVoiceMessage(sid, data): try: room_id = data.get("roomId") username = data.get("username") audio_base64 = data.get("audio") if not room_id or not username or not audio_base64: await sio.emit("error", {"message": "Missing required data for voice message"}, room=sid) return if room_id not in rooms: await sio.emit("error", {"message": "Room not found"}, room=sid) return try: audio_bytes = base64.b64decode(audio_base64) except Exception as e: await sio.emit("error", {"message": "Invalid audio data format"}, room=sid) return transcribed_text = await fish_audio_client.transcribe_audio(audio_bytes) if not transcribed_text: await sio.emit("error", {"message": "Could not transcribe audio. Please try speaking more clearly."}, room=sid) return user_message = Message( id=generate_message_id(), content=transcribed_text, username=username, timestamp=int(datetime.now().timestamp() * 1000), is_ai=False, is_voice=True ) rooms[room_id].messages.append(user_message) await sio.emit("newMessage", user_message.dict(), room=room_id) await generate_ai_response(room_id, username) except Exception as e: print(f"Error in sendVoiceMessage: {e}") await sio.emit("error", {"message": "Failed to process voice message. Please try again."}, room=sid) def should_tom_respond(message_content: str, username: str) -> bool: """Check if Tom should respond based on the prompt's gating rules.""" content_lower = message_content.lower() # Rule 1: Direct mention of "Tom" if "tom" in content_lower: return True # Rule 2: Addressed to everyone (covers greetings like "hey all") if any(phrase in content_lower for phrase in ["guys", "everyone", "all"]): return True # Rule 3: A question is asked if "?" in message_content: return True return False async def generate_ai_response(room_id: str, username: str): """Generates a response from the AI and broadcasts it to the room.""" print(f"[{room_id}] Generating AI response for {username}") room = rooms.get(room_id) if not room: return # 1. Build conversation context context = janitor_ai_client.build_conversation_context(room.messages) # 2. Get AI text response ai_text_response = await janitor_ai_client.get_ai_response(context) # 3. Handle silence if no [truncated β€” 3347 more characters] ``` ### frontend/src/app/layout.tsx ```typescript import type { Metadata } from "next"; import "./globals.css"; import Image from "next/image"; export const metadata: Metadata = { title: "Tommy Chat", description: "Multiplayer AI chat with Tommy - voice to voice conversations!", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return (
{/* Sun (orbiting) */}
Sun Rays
{children} ); } ``` ### frontend/src/app/page.tsx ```typescript 'use client'; import { useState } from 'react'; import { useRouter } from 'next/navigation'; import { nanoid } from 'nanoid'; import Image from 'next/image'; export default function Home() { const [roomId, setRoomId] = useState(''); const [username, setUsername] = useState(''); const router = useRouter(); const createRoom = () => { const newRoomId = nanoid(6).toUpperCase(); if (username.trim()) { router.push(`/room/${newRoomId}?username=${encodeURIComponent(username)}`); } else { alert('Please enter a username'); } }; const joinRoom = () => { if (roomId.trim() && username.trim()) { router.push(`/room/${roomId.toUpperCase()}?username=${encodeURIComponent(username)}`); } else { alert('Please enter both room ID and username'); } }; return (
{/* Tom Character Header */}
Tom's face

Tommy Chat

Meow! Let's chat together!

{/* Username Input */}
setUsername(e.target.value)} placeholder="Enter your name, friend!" className="tom-input w-full px-4 py-2 text-lg" maxLength={20} />
{/* Create Room */}

Create New Room

{/* Join Room */}

Join Existing Room

setRoomId(e.target.value.toUpperCase())} placeholder="Enter room code" className="tom-input w-full px-4 py-2 text-lg uppercase font-mono" maxLength={6} />

Tom will respond to every message!

Share room codes with friends

Have fun chatting together!

); } ``` ### frontend/src/app/room/[roomId]/page.tsx ```typescript 'use client'; import { useEffect, useState, useRef } from 'react'; import { useParams, useSearchParams } from 'next/navigation'; import { io, Socket } from 'socket.io-client'; import { Message, User } from '@/types/chat'; import ChatMessage from '@/components/ChatMessage'; import ChatInput from '@/components/ChatInput'; import ContinuousRecorder from '@/components/ContinuousRecorder'; import Image from 'next/image'; export default function RoomPage() { const params = useParams(); const searchParams = useSearchParams(); const roomId = params.roomId as string; const username = searchParams.get('username') || ''; const [socket, setSocket] = useState(null); const [messages, setMessages] = useState([]); const [users, setUsers] = useState([]); const [isConnected, setIsConnected] = useState(false); const [error, setError] = useState(''); const messagesEndRef = useRef(null); useEffect(() => { if (!username) { setError('Username is required'); return; } // Initialize socket connection const backendUrl = process.env.NEXT_PUBLIC_API_URL || process.env.NEXT_PUBLIC_BACKEND_URL || 'http://localhost:8000'; const newSocket = io(backendUrl); setSocket(newSocket); // Connection events newSocket.on('connect', () => { setIsConnected(true); setError(''); // Join room after connection newSocket.emit('joinRoom', { roomId, username }); }); newSocket.on('disconnect', () => { setIsConnected(false); }); // Message events newSocket.on('newMessage', (message: Message) => { setMessages(prev => [...prev, message]); }); newSocket.on('userJoined', (user: User) => { setUsers(prev => { if (!prev.find(u => u.id === user.id)) { return [...prev, user]; } return prev; }); }); newSocket.on('userLeft', (userId: string) => { setUsers(prev => prev.filter(user => user.id !== userId)); }); newSocket.on('roomUsers', (roomUsers: User[]) => { setUsers(roomUsers); }); newSocket.on('roomMessages', (roomMessages: Message[]) => { setMessages(roomMessages); }); newSocket.on('error', (errorMessage: string) => { setError(errorMessage); }); return () => { newSocket.close(); }; }, [roomId, username]); // Auto-scroll to bottom when new messages arrive useEffect(() => { messagesEndRef.current?.scrollIntoView({ behavior: 'smooth' }); }, [messages]); const sendMessage = (content: string) => { if (socket && content.trim()) { socket.emit('sendMessage', { content: content.trim(), username, roomId }); } }; const handleVoiceRecording = async (audioBlob: Blob) => { if (!socket || !roomId || !username) { return; } // Convert blob to base64 const reader = new FileReader(); reader.onloadend = () => { const base64Audio = reader.result?.toString().split(',')[1]; if (base64Audio) { socket.emit('sendVoiceMessage', { roomId, username, audio: base64Audio }); } }; reader.readAsDataURL(audioBlob); }; const copyRoomCode = () => { navigator.clipboard.writeText(roomId); // You could add a toast notification here }; if (error) { return (

Error

{error}

); } return (
{/* Header */}
Tom's face

Room: {roomId}