# Project export: EloquenceAI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2025 - Tagline: Eloquence.AI is your AI-powered speaking coach to get instant, engaging feedback on speech, delivery, and more in a gamified manner. - Devpost: https://devpost.com/software/eloquentai - GitHub: https://github.com/bdonyan/berkhack - Video: https://www.youtube.com/embed/BaxKwoBtULM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Brandon Yan (9 commits), briandabuilder (7 commits), Arav Sachdeva (5 commits) ## Devpost submission (written by the team) ### Inspiration All of us in our team constantly face situations, whether in college or at work, where we need to speak clearly and confidently. Whether it’s presenting in class, sharing research findings, or articulating ideas in meetings, strong communication is key. As students conducting technical research at our universities, we often struggle to explain complex ideas in a way that’s both thorough and concise. Thus, we built Eloquence.AI to solve this real need: a tool that helps us refine our delivery and feel confident when speaking under pressure. What Sets Us Apart What sets Eloquence.AI apart is its multimodal intelligence—it doesn't just analyze your words, but your tone, facial expressions, and body language. Unlike existing tools, we incorporate emotion detection, gesture tracking, and eye contact analysis to give users a complete view of their communication style. By combining AI-powered insight with a gamified experience, Eloquence.AI transforms the traditionally dull, anxiety-inducing task of public speaking into a motivating, actionable, and even fun journey. It’s more than feedback—it’s a personalized coach that helps you grow with every speech. What We Learned We explored how to combine speech-to-text models with LLMs to generate useful and personalized feedback. We also learned how to use computer vision to analyze body language, eye contact, and facial expression—turning raw video data into actionable insights. Most importantly, we learned how to tune AI systems to give feedback that feels human and helpful, not robotic. How We Built It We used OpenAI's Whisper to transcribe recorded speeches and GPT-4o to analyze clarity, pacing, filler words, and argument strength. We integrated computer vision tools like MediaPipe to track eye contact and gestures, giving a more complete view of a speaker’s delivery. Our frontend was built with React and Tailwind for a clean, user-friendly interface, and the backend runs on FastAPI. We also developed our own machine learning model, trained, and tested it on our own. Classroom Integration & Gamified Learning Eloquence.AI can be used in classrooms to make public speaking practice more interactive and fun—similar to how Kahoot transforms quizzes. Students can log in with individual accounts, complete speech challenges, and earn Elo points based on performance. A live leaderboard allows educators to track progress, encourage improvement, and spark friendly competition. This gamified system turns communication training into a dynamic experience, helping students build confidence while enjoying the process. Challenges We Faced One major challenge was syncing multimodal inputs, video, audio, and text, and presenting feedback in a seamless way. We also had to carefully craft prompts for GPT-4o to ensure feedback was specific and motivating. Balancing technical depth with usability under time pressure was tough, but it pushed us to make smart design decisions and prioritize the core user experience. !(/Users/arav/Downloads/designdiagram.png) ## README (from the GitHub repository) # 🗣️ Eloquence.AI — Real-Time Public Speaking Feedback & Elo Trainer ## 🎯 Mission Build a web application that helps users improve their public speaking through **real-time multimodal feedback**. It simulates audience reactions and uses **AI agents** to analyze **voice, facial expressions, and body language**, providing actionable critiques and performance scoring (ELO-style). ## 🏗️ Project Structure ``` eloquence-ai/ ├── voice-ai/ # Person A: Voice + Speech AI ├── vision-ai/ # Person B: Facial Expression + Gesture Recognition ├── game-ui/ # Person C: Game Loop, Elo Scoring, UI ├── shared/ # Shared utilities and schemas └── docs/ # Documentation and integration guides ``` ## 🧩 MVP Modules ### 🔊 Voice + Speech AI (Person A) - Real-time transcription using OpenAI Whisper/Vapi API - Tone analysis and filler word detection - Speech pace and rhythm analysis - AI-generated structured feedback using Claude/GPT-4 - **Output**: JSON with speech stats + feedback + emotion score ### 🎥 Facial Expression + Gesture Recognition (Person B) - Webcam-based computer vision analysis - Eye contact and gaze estimation - Facial emotion detection (confidence, engagement) - Body posture and hand gesture recognition - **Output**: JSON with timestamps, gestures/emotions, posture score ### 🎮 Game Loop, Elo Scoring, UI (Person C) - Interactive frontend with mock audience reactions - Real-time feedback display panel - Elo-style ranking system - Performance history and progress tracking - **Output**: Frontend MVP with UI integration and scoring logic ## 🛠️ Tech Stack ### AI & ML - **Voice**: OpenAI Whisper, Vapi API - **LLM**: Claude 3.5 Sonnet, GPT-4o - **Computer Vision**: MediaPipe, TensorFlow.js, OpenCV - **Vector DB**: Pinecone/Weaviate (optional) ### Frontend & Backend - **Framework**: Next.js + Tailwind CSS - **Real-time**: WebRTC, WebSocket - **Agent Orchestration**: LangGraph (optional) - **Fast Inference**: Groq (optional) ## 🚀 Quick Start 1. **Clone and setup modules:** ```bash git clone cd eloquence-ai ``` 2. **Setup each module:** ```bash # Voice AI Module cd voice-ai && npm install # Vision AI Module cd ../vision-ai && npm install # Game UI Module cd ../game-ui && npm install ``` 3. **Environment Setup:** ```bash # Copy example env files cp voice-ai/env.example voice-ai/.env cp vision-ai/env.example vision-ai/.env cp game-ui/env.example game-ui/.env ``` 4. **Start development:** ```bash # Terminal 1: Voice AI cd voice-ai && npm run dev # Terminal 2: Vision AI cd vision-ai && npm run dev # Terminal 3: Game UI cd game-ui && npm run dev ``` ## 🎯 Prize Track Targets - ✅ **Creativity**: Multimodal, Elo-style gamified public speaking - ✅ **Productivity**: Makes users better communicators - ✅ **Voice AI (Vapi)**: Real-time voice transcription - ✅ **Multimodal Agent (Unify)**: Vision + voice + LLM - ✅ **Claude (Anthropic)**: Empathy-based feedback - ✅ **LLM Agent (Nobel Era)**: Feedback + rebuttal generation ## 📋 Development Checklist - [ ] Each teammate clones repo + sets up subfolder for their module - [ ] Start local JSON schema for feedback format (voice + visual) - [ ] Schedule mid-build integration checkpoint - [ ] Implement real-time data flow between modules - [ ] Create unified scoring system - [ ] Add audience reaction simulation - [ ] Implement Elo ranking algorithm - [ ] Add performance analytics dashboard ## 🤝 Integration Points ### Data Flow ``` Voice AI → JSON Feedback → Game UI Vision AI → JSON Feedback → Game UI Game UI → Elo Score → Performance History ``` ### Shared Schemas - Speech feedback format - Visual feedback format - Performance scoring metrics - Elo ranking data structure ## Detected evidence (automated analysis) Indexed codebase: 45 recognized source files, 295 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - Express (technology) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Firebase (technology) — claimed on Devpost, not found in the code - Flask (technology) — claimed on Devpost, not found in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - Node.js (technology) — claimed on Devpost, not found in the code - PostgreSQL (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 141) ``` .gitignore docs/INTEGRATION.md docs/SETUP.md game-ui/env.example game-ui/next-env.d.ts game-ui/next.config.js game-ui/package.json game-ui/postcss.config.js game-ui/src/components/AnalysisDashboard.tsx game-ui/src/components/DailyMissions.tsx game-ui/src/components/DetailedAnalysis.tsx game-ui/src/components/FeedbackPanel.tsx game-ui/src/components/PerformanceMetrics.tsx game-ui/src/components/SessionControls.tsx game-ui/src/components/TranscriptViewer.tsx game-ui/src/hooks/useMediaStream.ts game-ui/src/pages/_app.tsx game-ui/src/pages/_error.tsx game-ui/src/pages/404.tsx game-ui/src/pages/dashboard.tsx game-ui/src/pages/home.tsx game-ui/src/pages/index.tsx game-ui/src/pages/login.tsx game-ui/src/pages/signup.tsx game-ui/src/stores/authStore.ts game-ui/src/stores/gameStore.ts game-ui/src/styles/globals.css game-ui/src/utils/EloScoringSystem.ts game-ui/src/utils/mlDemo.ts game-ui/src/utils/runMLDemo.ts game-ui/src/utils/testVisualData.ts game-ui/src/utils/VisualDataGenerator.ts game-ui/tailwind.config.js game-ui/tsconfig.json game-ui/tsconfig.tsbuildinfo health-check.sh README.md scripts/setup.sh shared/schemas.ts start-all.sh stop-all.sh vision-ai/env.example vision-ai/package.json vision-ai/src/index.ts vision-ai/src/services/GestureRecognizer.ts vision-ai/src/services/VisionAnalyzer.ts vision-ai/src/services/WebSocketManager.ts vision-ai/tsconfig.json voice-ai/env.example voice-ai/package.json voice-ai/src/index.ts voice-ai/src/services/FeedbackGenerator.ts voice-ai/src/services/HumeAnalyzer.ts voice-ai/src/services/SpeechAnalyzer.ts voice-ai/src/services/WebSocketManager.ts voice-ai/transcripts/session-1750581101530-2025-06-22T08-31-45-177Z.txt voice-ai/transcripts/session-1750581219060-2025-06-22T08-33-47-598Z.txt voice-ai/transcripts/session-1750581671220-2025-06-22T08-41-19-736Z.txt voice-ai/transcripts/session-1750582637066-2025-06-22T08-57-40-108Z.txt voice-ai/transcripts/session-1750582709193-2025-06-22T08-58-33-358Z.txt voice-ai/transcripts/session-1750582726204-2025-06-22T08-58-52-711Z.txt voice-ai/transcripts/session-1750582734649-2025-06-22T08-59-00-843Z.txt voice-ai/transcripts/session-1750582742857-2025-06-22T08-59-07-455Z.txt voice-ai/transcripts/session-1750582748294-2025-06-22T08-59-20-833Z.txt voice-ai/transcripts/session-1750582770548-2025-06-22T08-59-34-140Z.txt voice-ai/transcripts/session-1750582775486-2025-06-22T08-59-38-631Z.txt voice-ai/transcripts/session-1750582779268-2025-06-22T08-59-42-347Z.txt voice-ai/transcripts/session-1750582786895-2025-06-22T09-00-01-588Z.txt voice-ai/transcripts/session-1750582803720-2025-06-22T09-00-13-779Z.txt voice-ai/transcripts/session-1750582848643-2025-06-22T09-01-15-356Z.txt voice-ai/transcripts/session-1750582993363-2025-06-22T09-03-25-098Z.txt voice-ai/transcripts/session-1750583006416-2025-06-22T09-03-34-053Z.txt voice-ai/transcripts/session-1750583466074-2025-06-22T09-11-21-877Z.txt voice-ai/transcripts/session-1750583497102-2025-06-22T09-11-54-130Z.txt voice-ai/transcripts/session-1750583522738-2025-06-22T09-12-05-598Z.txt voice-ai/transcripts/session-1750583586598-2025-06-22T09-13-15-031Z.txt voice-ai/transcripts/session-1750583596746-2025-06-22T09-13-24-046Z.txt voice-ai/transcripts/session-1750583604963-2025-06-22T09-13-28-421Z.txt voice-ai/transcripts/session-1750583616289-2025-06-22T09-13-40-221Z.txt voice-ai/transcripts/session-1750583621657-2025-06-22T09-13-45-941Z.txt voice-ai/transcripts/session-1750583734586-2025-06-22T09-15-51-293Z.txt voice-ai/transcripts/session-1750584243794-2025-06-22T09-24-10-920Z.txt voice-ai/transcripts/session-1750584515753-2025-06-22T09-29-02-854Z.txt voice-ai/transcripts/session-1750584544617-2025-06-22T09-29-16-739Z.txt voice-ai/transcripts/session-1750584570590-2025-06-22T09-29-39-455Z.txt voice-ai/transcripts/session-1750584641174-2025-06-22T09-30-48-579Z.txt voice-ai/transcripts/session-1750584658078-2025-06-22T09-31-04-960Z.txt voice-ai/transcripts/session-1750584688408-2025-06-22T09-31-34-912Z.txt voice-ai/transcripts/session-1750584711347-2025-06-22T09-31-58-403Z.txt voice-ai/transcripts/session-1750584755137-2025-06-22T09-32-41-743Z.txt voice-ai/transcripts/session-1750584929331-2025-06-22T09-35-37-918Z.txt voice-ai/transcripts/session-1750584951713-2025-06-22T09-36-01-142Z.txt voice-ai/transcripts/session-1750585000206-2025-06-22T09-36-49-566Z.txt voice-ai/transcripts/session-1750585023401-2025-06-22T09-37-11-355Z.txt voice-ai/transcripts/session-1750585046465-2025-06-22T09-37-37-126Z.txt voice-ai/transcripts/session-1750585072726-2025-06-22T09-38-06-518Z.txt voice-ai/transcripts/session-1750585180591-2025-06-22T09-39-59-538Z.txt voice-ai/transcripts/session-1750585238095-2025-06-22T09-40-55-742Z.txt voice-ai/transcripts/session-1750585421027-2025-06-22T09-43-51-288Z.txt voice-ai/transcripts/session-1750585449217-2025-06-22T09-44-15-751Z.txt voice-ai/transcripts/session-1750585495982-2025-06-22T09-44-58-887Z.txt voice-ai/transcripts/session-1750585541349-2025-06-22T09-45-48-960Z.txt voice-ai/transcripts/session-1750586042022-2025-06-22T09-54-09-227Z.txt voice-ai/transcripts/session-1750586444896-2025-06-22T10-00-51-912Z.txt voice-ai/transcripts/session-1750586466494-2025-06-22T10-01-17-416Z.txt voice-ai/transcripts/session-1750586501303-2025-06-22T10-01-52-634Z.txt voice-ai/transcripts/session-1750589004425-2025-06-22T10-43-32-369Z.txt voice-ai/transcripts/session-1750589099079-2025-06-22T10-45-10-632Z.txt voice-ai/transcripts/session-1750589135710-2025-06-22T10-46-02-421Z.txt voice-ai/transcripts/session-1750610644916-2025-06-22T16-44-21-626Z.txt voice-ai/transcripts/session-1750612428506-2025-06-22T17-13-53-681Z.txt voice-ai/transcripts/session-1750612712735-2025-06-22T17-18-36-896Z.txt voice-ai/transcripts/session-1750612861832-2025-06-22T17-21-08-176Z.txt voice-ai/transcripts/session-1750613082995-2025-06-22T17-24-48-236Z.txt voice-ai/transcripts/session-1750613129349-2025-06-22T17-25-36-310Z.txt voice-ai/transcripts/session-1750613145542-2025-06-22T17-25-52-682Z.txt voice-ai/transcripts/session-1750613173402-2025-06-22T17-26-25-286Z.txt voice-ai/transcripts/session-1750613214456-2025-06-22T17-27-01-689Z.txt voice-ai/transcripts/session-1750613544829-2025-06-22T17-32-38-680Z.txt voice-ai/transcripts/session-1750613584230-2025-06-22T17-33-14-759Z.txt [21 more files omitted for size] ``` ### Dependencies - game-ui/package.json: @types/node@^20.10.0, @types/react@^18.2.38, @types/react-dom@^18.2.17, @types/uuid@^9.0.7, autoprefixer@^10.4.16, eslint@^8.54.0, eslint-config-next@^14.0.0, framer-motion@^10.16.4, lucide-react@^0.292.0, next@^14.0.0, postcss@^8.4.31, prettier@^3.1.0, prettier-plugin-tailwindcss@^0.5.7, react@^18.2.0, react-dom@^18.2.0, react-hot-toast@^2.5.2, recharts@^2.8.0, socket.io-client@^4.7.4, tailwindcss@^3.3.0, ts-node@^10.9.2, typescript@^5.3.2, uuid@^9.0.1, zustand@^4.4.6 - vision-ai/package.json: @types/cors@^2.8.13, @types/express@^4.17.17, @types/jest@^29.5.4, @types/node@^20.5.0, cors@^2.8.5, dotenv@^16.3.1, express@^4.18.2, helmet@^7.0.0, jest@^29.6.2, socket.io@^4.7.2, ts-node-dev@^2.0.0, typescript@^5.1.6 - voice-ai/package.json: @anthropic-ai/sdk@^0.20.7, @types/cors@^2.8.17, @types/express@^4.17.21, @types/fluent-ffmpeg@^2.1.24, @types/jest@^29.5.8, @types/node@^20.10.0, @types/uuid@^9.0.7, @types/ws@^8.5.10, compromise@^14.10.0, cors@^2.8.5, dotenv@^16.3.1, eslint@^8.54.0, express@^4.18.2, fluent-ffmpeg@^2.1.2, hume@^0.11.4, jest@^29.7.0, natural@^6.10.4, node-record-lpcm16@^1.0.1, openai@^4.20.1, openai-whisper@^1.0.2, prettier@^3.1.0, socket.io@^4.7.4, ts-jest@^29.1.1, ts-node-dev@^2.0.0, typescript@^5.3.2, uuid@^9.0.1, wav@^1.0.2, ws@^8.14.2 ### Recent commits (newest first) - Update README.md - File version done - current version - Add authentication system and homepage with presentation skills information - Add auth store with Zustand for user management - Create login and signup pages with form validation - Add homepage with compelling presentation skills content - Create dashboard page for authenticated users - Add 404 page for better error handling - Fix TypeScript errors in AnalysisDashboard and VisualDataGenerator - Update app structure to support authentication flow - Add enhanced ML model with improved accuracy and resolve merge conflicts - Remove all merge conflict markers and unify logic in index.tsx - Finalize merge: resolve all remaining conflict markers and unify logic in index.tsx - Resolve all merge conflicts in index.tsx and unify improved logic - Resolve merge conflict in index.tsx after pulling from main - score fixed, use gpt for the scoring - scores display but not super accurate - latest displayed score - fixed elo rating and avg duration stats - done - fixed error - Resolved merge conflicts using local version - fixed filler word and clarity analysis. also WPM - working wpm - wpm - insights done ## Key source files (fetched from GitHub, selected and truncated for size) ### docs/SETUP.md ```markdown # 🚀 Eloquence.AI Setup Guide This guide will help you set up the complete Eloquence.AI project with all three modules: Voice AI, Vision AI, and Game UI. ## 📋 Prerequisites - **Node.js** (v18 or higher) - **npm** or **yarn** - **Git** - **Webcam** and **microphone** for testing - **API Keys** for OpenAI and Anthropic ## 🔑 Required API Keys ### OpenAI API Key 1. Go to [OpenAI Platform](https://platform.openai.com/) 2. Create an account or sign in 3. Navigate to API Keys section 4. Create a new API key 5. Copy the key (starts with `sk-`) ### Anthropic API Key (Claude) 1. Go to [Anthropic Console](https://console.anthropic.com/) 2. Create an account or sign in 3. Navigate to API Keys section 4. Create a new API key 5. Copy the key (starts with `sk-ant-`) ## 🏗️ Project Structure ``` eloquence-ai/ ├── voice-ai/ # Person A: Voice + Speech AI ├── vision-ai/ # Person B: Facial Expression + Gesture Recognition ├── game-ui/ # Person C: Game Loop, Elo Scoring, UI ├── shared/ # Shared utilities and schemas └── docs/ # Documentation and integration guides ``` ## 🎯 Module Setup ### 1. Voice AI Module (Person A) ```bash # Navigate to voice-ai directory cd voice-ai # Install dependencies npm install # Copy environment template cp env.example .env # Edit .env file with your API keys nano .env ``` **Required environment variables:** ```env OPENAI_API_KEY=your_openai_api_key_here ANTHROPIC_API_KEY=your_anthropic_api_key_here PORT=3001 ``` **Start the Voice AI module:** ```bash npm run dev ``` The Voice AI module will be available at `http://localhost:3001` ### 2. Vision AI Module (Person B) ```bash # Navigate to vision-ai directory cd vision-ai # Install dependencies npm install # Copy environment template cp env.example .env # Edit .env file nano .env ``` **Required environment variables:** ```env PORT=3002 NODE_ENV=development ``` **Start the Vision AI module:** ```bash npm run dev ``` The Vision AI module will be available at `http://localhost:3002` ### 3. Game UI Module (Person C) ```bash # Navigate to game-ui directory cd game-ui # Install dependencies npm install # Copy environment template cp env.example .env # Edit .env file nano .env ``` **Required environment variables:** ```env NEXT_PUBLIC_VOICE_AI_URL=http://localhost:3001 NEXT_PUBLIC_VISION_AI_URL=http://localhost:3002 ``` **Start the Game UI module:** ```bash npm run dev ``` The Game UI will be available at `http://localhost:3003` ## 🔄 Running All Modules You'll need three terminal windows to run all modules simultaneously: **Terminal 1 - Voice AI:** ```bash cd voice-ai npm run dev ``` **Terminal 2 - Vision AI:** ```bash cd vision-ai npm run dev ``` **Terminal 3 - Game UI:** ```bash cd game-ui npm run dev ``` ## 🧪 Testing the Setup ### 1. Health Checks Test that all modules are running correctly: ```bash # Voice AI health check curl http://localhost:3001/health # Vision AI health check curl http:/ [truncated — 2846 more characters] ``` ### docs/INTEGRATION.md ```markdown # 🔗 Eloquence.AI Integration Guide This guide explains how the three modules of Eloquence.AI communicate and work together to provide real-time public speaking feedback. ## 🏗️ Architecture Overview ``` ┌─────────────────┐ WebSocket ┌─────────────────┐ │ Game UI │ ◄──────────────► │ Voice AI │ │ (Port 3003) │ │ (Port 3001) │ └─────────────────┘ └─────────────────┘ │ │ │ WebSocket │ ▼ │ ┌─────────────────┐ │ │ Vision AI │ ◄────────────────────────┘ │ (Port 3002) │ └─────────────────┘ ``` ## 📡 Communication Protocols ### 1. WebSocket Messages All modules communicate via WebSocket using standardized message formats defined in `shared/schemas.ts`. **Message Types:** - `speech_feedback` - Voice analysis results - `visual_feedback` - Visual analysis results - `performance_update` - Combined performance metrics - `audience_reaction` - Simulated audience responses - `elo_update` - Rating changes ### 2. HTTP API Endpoints **Voice AI Endpoints:** ```typescript POST /analyze-speech { audioData: Buffer, sessionId: string } POST /stream-speech { audioChunk: Buffer, sessionId: string, isFinal: boolean } ``` **Vision AI Endpoints:** ```typescript POST /analyze-frame { imageData: string, // base64 encoded sessionId: string } POST /stream-video { frameData: string, sessionId: string, isKeyFrame: boolean } ``` ## 🔄 Data Flow ### 1. Session Initialization ```mermaid sequenceDiagram participant UI as Game UI participant Voice as Voice AI participant Vision as Vision AI UI->>Voice: start-session (sessionId) UI->>Vision: start-vision-session (sessionId) Voice-->>UI: session-started Vision-->>UI: session-started ``` ### 2. Real-time Feedback Loop ```mermaid sequenceDiagram participant User as User participant UI as Game UI participant Voice as Voice AI participant Vision as Vision AI User->>UI: Start speaking UI->>Voice: audio-chunk UI->>Vision: video-frame Voice-->>UI: speech-feedback Vision-->>UI: vision-feedback UI->>UI: Update combined score UI->>User: Display feedback ``` ### 3. Session Completion ```mermaid sequenceDiagram participant UI as Game UI participant Voice as Voice AI participant Vision as Vision AI UI->>Voice: end-session UI->>Vision: end-session UI->>UI: Calculate final score UI->>UI: Update Elo rating UI->>UI: Save session history ``` ## 📊 Data Schemas ### Speech Feedback Schema ```typescript interface SpeechFeedback { timestamp: number; transcript: string; confidence: number; tone: { emotion: 'confident' | 'nervous' | 'enthusiastic' | 'monotone' | 'engaging'; score: number; }; pace: { wordsPerMinute: number; pauses: number; score: number; }; [truncated — 7772 more characters] ``` ### vision-ai/package.json ``` { "name": "vision-ai", "version": "1.0.0", "description": "Facial Expression + Gesture Recognition Module", "main": "dist/index.js", "scripts": { "dev": "ts-node-dev --project tsconfig.json --respawn --transpile-only src/index.ts", "build": "tsc", "start": "node dist/index.js", "test": "jest" }, "dependencies": { "express": "^4.18.2", "socket.io": "^4.7.2", "cors": "^2.8.5", "helmet": "^7.0.0", "dotenv": "^16.3.1" }, "devDependencies": { "@types/express": "^4.17.17", "@types/node": "^20.5.0", "@types/cors": "^2.8.13", "typescript": "^5.1.6", "ts-node-dev": "^2.0.0", "jest": "^29.6.2", "@types/jest": "^29.5.4" }, "keywords": ["computer-vision", "facial-recognition", "gesture-detection"], "author": "Eloquence.AI Team", "license": "MIT" } ``` ### game-ui/package.json ``` { "name": "eloquence-game-ui", "version": "1.0.0", "description": "Game UI module for Eloquence.AI - Interactive frontend with Elo scoring and audience reactions", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint", "type-check": "tsc --noEmit", "ml-demo": "npx ts-node --project tsconfig.json src/utils/runMLDemo.ts" }, "dependencies": { "autoprefixer": "^10.4.16", "framer-motion": "^10.16.4", "lucide-react": "^0.292.0", "next": "^14.0.0", "postcss": "^8.4.31", "react": "^18.2.0", "react-dom": "^18.2.0", "react-hot-toast": "^2.5.2", "recharts": "^2.8.0", "socket.io-client": "^4.7.4", "tailwindcss": "^3.3.0", "uuid": "^9.0.1", "zustand": "^4.4.6" }, "devDependencies": { "@types/node": "^20.10.0", "@types/react": "^18.2.38", "@types/react-dom": "^18.2.17", "@types/uuid": "^9.0.7", "eslint": "^8.54.0", "eslint-config-next": "^14.0.0", "prettier": "^3.1.0", "prettier-plugin-tailwindcss": "^0.5.7", "ts-node": "^10.9.2", "typescript": "^5.3.2" }, "keywords": [ "nextjs", "react", "public-speaking", "elo-scoring", "gamification", "real-time-feedback" ], "author": "Eloquence.AI Team", "license": "MIT" } ``` ### voice-ai/package.json ``` { "name": "eloquence-voice-ai", "version": "1.0.0", "description": "Voice AI module for Eloquence.AI - Real-time speech analysis and feedback", "main": "dist/index.js", "scripts": { "dev": "ts-node-dev --respawn --transpile-only src/index.ts", "build": "tsc", "start": "node dist/index.js", "test": "jest", "lint": "eslint src/**/*.ts", "format": "prettier --write src/**/*.ts" }, "dependencies": { "@anthropic-ai/sdk": "^0.20.7", "compromise": "^14.10.0", "cors": "^2.8.5", "dotenv": "^16.3.1", "express": "^4.18.2", "fluent-ffmpeg": "^2.1.2", "hume": "^0.11.4", "natural": "^6.10.4", "node-record-lpcm16": "^1.0.1", "openai": "^4.20.1", "openai-whisper": "^1.0.2", "socket.io": "^4.7.4", "uuid": "^9.0.1", "wav": "^1.0.2", "ws": "^8.14.2" }, "devDependencies": { "@types/cors": "^2.8.17", "@types/express": "^4.17.21", "@types/fluent-ffmpeg": "^2.1.24", "@types/jest": "^29.5.8", "@types/node": "^20.10.0", "@types/uuid": "^9.0.7", "@types/ws": "^8.5.10", "eslint": "^8.54.0", "jest": "^29.7.0", "prettier": "^3.1.0", "ts-jest": "^29.1.1", "ts-node-dev": "^2.0.0", "typescript": "^5.3.2" }, "keywords": [ "speech-recognition", "ai", "public-speaking", "real-time", "feedback" ], "author": "Eloquence.AI Team", "license": "MIT" } ``` ### vision-ai/src/index.ts ```typescript import express from 'express'; import cors from 'cors'; import { createServer } from 'http'; import { Server } from 'socket.io'; import dotenv from 'dotenv'; import { VisionAnalyzer } from './services/VisionAnalyzer'; import { GestureRecognizer } from './services/GestureRecognizer'; import { WebSocketManager } from './services/WebSocketManager'; import { VisualFeedback } from '../../shared/schemas'; dotenv.config(); const app = express(); const server = createServer(app); const io = new Server(server, { cors: { origin: ["http://localhost:3000", "http://localhost:3003"], methods: ["GET", "POST"] } }); const PORT = process.env.PORT || 3002; // Middleware app.use(cors()); app.use(express.json({ limit: '50mb' })); app.use(express.urlencoded({ extended: true })); // Initialize services const visionAnalyzer = new VisionAnalyzer(); const gestureRecognizer = new GestureRecognizer(); const wsManager = new WebSocketManager(io); // Health check endpoint app.get('/health', (req, res) => { res.json({ status: 'healthy', module: 'vision-ai', timestamp: new Date().toISOString() }); }); // Frame analysis endpoint app.post('/analyze-frame', async (req, res) => { try { const { imageData, sessionId } = req.body; if (!imageData) { return res.status(400).json({ error: 'Image data is required' }); } // Analyze facial expressions and body language const feedback = await visionAnalyzer.analyzeFrame(imageData); // Recognize gestures const gestures = gestureRecognizer.detectGestures(imageData); feedback.gestures.detected = gestures; // Send real-time feedback via WebSocket wsManager.broadcastVisualFeedback(feedback); res.json(feedback); } catch (error) { console.error('Frame analysis error:', error); res.status(500).json({ error: 'Frame analysis failed' }); } }); // Real-time video stream endpoint app.post('/stream-video', async (req, res) => { try { const { frameData, sessionId, isKeyFrame } = req.body; if (!frameData) { return res.status(400).json({ error: 'Frame data is required' }); } // Process frame (only analyze key frames for performance) if (isKeyFrame) { const feedback = await visionAnalyzer.analyzeFrame(frameData); const gestures = gestureRecognizer.detectGestures(frameData); feedback.gestures.detected = gestures; // Broadcast to connected clients wsManager.broadcastVisualFeedback(feedback); res.json({ success: true, feedback }); } else { res.json({ success: true, skipped: true }); } } catch (error) { console.error('Video stream processing error:', error); res.status(500).json({ error: 'Video stream processing failed' }); } }); // WebSocket connection logic is now managed by WebSocketManager // No need for a separate io.on('connection', ...) block here // Start server server.listen(PORT, () => { console.log(`📹 Vision AI module running on port ${PORT}`); console.log(`📡 WebSocket server ready for connections`); console.log(`🔗 Health check: http://localhost:${PORT}/health`); }); // Graceful shutdown process.on('SIGTERM', () => { console.log('SIGTERM received, shutting down gracefully'); server.close(() => { console.log('Vision AI server closed'); process.exit(0); }); }); ``` ### voice-ai/src/index.ts ```typescript import express from 'express'; import cors from 'cors'; import { createServer } from 'http'; import { Server } from 'socket.io'; import dotenv from 'dotenv'; import { SpeechAnalyzer } from './services/SpeechAnalyzer'; import { FeedbackGenerator } from './services/FeedbackGenerator'; import { WebSocketManager } from './services/WebSocketManager'; import { HumeAnalyzer } from './services/HumeAnalyzer'; import { SpeechFeedback, VisualFeedback } from '../../shared/schemas'; dotenv.config(); const app = express(); const server = createServer(app); const io = new Server(server, { cors: { origin: ["http://localhost:3000", "http://localhost:3003"], methods: ["GET", "POST"] } }); const PORT = process.env.PORT || 3001; // Middleware app.use(cors()); app.use(express.json({ limit: '50mb' })); app.use(express.urlencoded({ extended: true })); // Initialize services const speechAnalyzer = new SpeechAnalyzer(); const feedbackGenerator = new FeedbackGenerator(); const wsManager = new WebSocketManager(io); const humeAnalyzer = new HumeAnalyzer(); // Test Hume AI connection on startup humeAnalyzer.testConnection().then(success => { if (success) { console.log('✅ Hume AI connection test successful'); } else { console.log('⚠️ Hume AI connection test failed - using fallback mode'); } }); // Health check endpoint app.get('/health', (req, res) => { res.json({ status: 'healthy', module: 'voice-ai', timestamp: new Date().toISOString() }); }); // Get transcript for a specific session app.get('/transcript/:sessionId', async (req, res) => { try { const { sessionId } = req.params; const transcript = await speechAnalyzer.getTranscript(sessionId); if (transcript) { res.json({ transcript }); } else { res.status(404).json({ error: 'Transcript not found' }); } } catch (error) { console.error('Get transcript error:', error); res.status(500).json({ error: 'Failed to get transcript' }); } }); // Get all transcripts app.get('/transcripts', async (req, res) => { try { const transcripts = await speechAnalyzer.getAllTranscripts(); res.json({ transcripts }); } catch (error) { console.error('Get all transcripts error:', error); res.status(500).json({ error: 'Failed to get transcripts' }); } }); // Speech analysis endpoint app.post('/analyze-speech', async (req, res) => { try { const { audioData, sessionId, duration } = req.body; if (!audioData) { return res.status(400).json({ error: 'Audio data is required' }); } // Convert base64 data URL to buffer const base64String = audioData.split(';base64,').pop(); if (!base64String) { return res.status(400).json({ error: 'Invalid audio data format' }); } const audioBuffer = Buffer.from(base64String, 'base64'); // Analyze speech const analysis = await speechAnalyzer.analyzeAudio(audioBuffer, sessionId, duration); // Generate feedback const feedback = await feedbackGenerator.generateFeedback(analysis); // Send real-time feedback via WebSocket wsManager.broadcastFeedback({ type: 'speech_feedback', data: feedback, timestamp: Date.now() }); res.json(feedback); } catch (error: any) { console.error('Speech analysis error:', error); if (error.status) { // This indicates an API error from OpenAI return res.status(error.status).json({ error: `Failed to process speech. OpenAI API returned status ${error.status}.`, details: error.message }); } res.status(500).json({ error: 'An unexpected error occurred during speech analysis.' }); } }); // Real-time speech streaming endpoint app.post('/stream-speech', async (req, res) => { try { const { audioChunk, sessionId, isFinal } = req.body; if (!audioChunk) { return res.status(400).json({ error: 'Audio chunk is required' }); } // Process audio chunk const analysis = await speechAnalyzer.processChunk(audioChunk, sessionId, isFinal); if (analysis && isFinal) { // Generate feedback for complete utterance const feedback = await feedbackGenerator.generateFeedback(analysis); // Broadcast to connected clients wsManager.broadcastFeedback({ type: 'speech_feedback', data: feedback, timestamp: Date.now() }); } res.json({ success: true, analysis }); } catch (error) { console.error('Stream processing error:', error); res.status(500).json({ error: 'Stream processing failed' }); } }); // Hume AI expression analysis endpoint app.post('/analyze-expression', async (req, res) => { try { const { audioData, videoData, sessionId } = req.body; if (!audioData) { return res.status(400).json({ error: 'Audio data is required' }); } // Convert base64 data to buffer const audioBase64 = audioData.split(';base64,').pop(); if (!audioBase64) { return res.status(400).json({ error: 'Invalid audio data format' }); } const audioBuffer = Buffer.from(audioBase64, 'base64'); let videoBuffer: Buffer | undefined; if (videoData) { const videoBase64 = videoData.split(';base64,').pop(); if (videoBase64) { videoBuffer = Buffer.from(videoBase64, 'base64'); } } // Analyze expressions using Hume AI const expressionResult = await humeAnalyzer.analyzeExpression(audioBuffer, videoBuffer); const summary = humeAnalyzer.getExpressionSummary(expressionResult); // Convert Hume AI results to VisualFeedback format const visualFeedback: VisualFeedback = { timestamp: Date.now(), eyeContact: { percentage: summary.engagement * 100, duration: 0, // Will be calculated from video analysis score: summary.engagement * 100 }, facialExpression: { emotion: summary.dominantEmotion as any, confidence: summary.confidence * 100, score: summa [truncated — 2694 more characters] ``` ### game-ui/src/pages/index.tsx ```typescript import { useEffect } from 'react'; import { useRouter } from 'next/router'; import { useAuthStore } from '../stores/authStore'; import { DailyMissions } from '@/components/DailyMissions'; import { useGameStore } from '../stores/gameStore'; import { motion } from 'framer-motion'; export default function Home() { const router = useRouter(); const { isAuthenticated } = useAuthStore(); const { isSessionActive, speechFeedback, visualFeedback, audienceReaction, currentEloRating, sessionHistory, settings, startSession: startSessionState, endSession: endSessionState, setSpeechFeedback, setVisualFeedback, setAudienceReaction, } = useGameStore(); const { xp, streak } = useGameStore(); const incrementXP = useGameStore((state) => () => state.xp = state.xp + 10); const incrementStreak = useGameStore((state) => () => state.streak = state.streak + 1); // Debug: Log XP and streak to the console console.log('XP:', xp, 'Streak:', streak); useEffect(() => { if (isAuthenticated) { router.push('/dashboard'); } else { router.push('/home'); } }, [isAuthenticated, router]); return (
{/* Header */}

🗣️ Eloquence.AI

Master public speaking with real-time AI feedback

{/* XP and Streak Tracker */}
{xp} XP
🔥 {streak} day streak
{/* Debug buttons for testing XP and streak */}
{/* Daily Missions Panel */}
); } ``` ### stop-all.sh ```shell #!/bin/bash # Eloquence.AI Stop Script # This script stops all running modules echo "🛑 Stopping Eloquence.AI modules..." # Kill processes on ports 3001, 3002, 3003 for port in 3001 3002 3003; do PID=$(lsof -ti:$port) if [ ! -z "$PID" ]; then echo "Stopping process on port $port (PID: $PID)" kill -9 $PID else echo "No process running on port $port" fi done echo "All modules stopped!" ``` ### health-check.sh ```shell #!/bin/bash # Eloquence.AI Health Check Script # This script checks if all modules are running properly echo "🏥 Checking Eloquence.AI module health..." # Colors GREEN='\033[0;32m' RED='\033[0;31m' NC='\033[0m' # Check Voice AI echo "Checking Voice AI (port 3001)..." if curl -s http://localhost:3001/health > /dev/null; then echo -e "${GREEN}✓ Voice AI is healthy${NC}" else echo -e "${RED}✗ Voice AI is not responding${NC}" fi # Check Vision AI echo "Checking Vision AI (port 3002)..." if curl -s http://localhost:3002/health > /dev/null; then echo -e "${GREEN}✓ Vision AI is healthy${NC}" else echo -e "${RED}✗ Vision AI is not responding${NC}" fi # Check Game UI echo "Checking Game UI (port 3003)..." if curl -s http://localhost:3003 > /dev/null; then echo -e "${GREEN}✓ Game UI is healthy${NC}" else echo -e "${RED}✗ Game UI is not responding${NC}" fi echo "Health check complete!" ``` [36 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]