Project Info

EloquenceAI

Inspiration

All of us in our team constantly face situations, whether in college or at work, where we need to speak clearly and confidently. Whether it’s presenting in class, sharing research findings, or articulating ideas in meetings, strong communication is key. As students conducting technical research at our universities, we often struggle to explain complex ideas in a way that’s both thorough and concise. Thus, we built Eloquence.AI to solve this real need: a tool that helps us refine our delivery and feel confident when speaking under pressure. What Sets Us Apart What sets Eloquence.AI apart is its multimodal intelligence—it doesn't just analyze your words, but your tone, facial expressions, and body language. Unlike existing tools, we incorporate emotion detection, gesture tracking, and eye contact analysis to give users a complete view of their communication style. By combining AI-powered insight with a gamified experience, Eloquence.AI transforms the traditionally dull, anxiety-inducing task of public speaking into a motivating, actionable, and even fun journey. It’s more than feedback—it’s a personalized coach that helps you grow with every speech. What We Learned We explored how to combine speech-to-text models with LLMs to generate useful and personalized feedback. We also learned how to use computer vision to analyze body language, eye contact, and facial expression—turning raw video data into actionable insights. Most importantly, we learned how to tune AI systems to give feedback that feels human and helpful, not robotic. How We Built It We used OpenAI's Whisper to transcribe recorded speeches and GPT-4o to analyze clarity, pacing, filler words, and argument strength. We integrated computer vision tools like MediaPipe to track eye contact and gestures, giving a more complete view of a speaker’s delivery. Our frontend was built with React and Tailwind for a clean, user-friendly interface, and the backend runs on FastAPI. We also developed our own machine learning model, trained, and tested it on our own. Classroom Integration & Gamified Learning Eloquence.AI can be used in classrooms to make public speaking practice more interactive and fun—similar to how Kahoot transforms quizzes. Students can log in with individual accounts, complete speech challenges, and earn Elo points based on performance. A live leaderboard allows educators to track progress, encourage improvement, and spark friendly competition. This gamified system turns communication training into a dynamic experience, helping students build confidence while enjoying the process. Challenges We Faced One major challenge was syncing multimodal inputs, video, audio, and text, and presenting feedback in a seamless way. We also had to carefully craft prompts for GPT-4o to ensure feedback was specific and motivating. Balancing technical depth with usability under time pressure was tough, but it pushed us to make smart design decisions and prioritize the core user experience. !(/Users/arav/Downloads/designdiagram.png)

🗣️ Eloquence.AI — Real-Time Public Speaking Feedback & Elo Trainer

🎯 Mission

Build a web application that helps users improve their public speaking through real-time multimodal feedback. It simulates audience reactions and uses AI agents to analyze voice, facial expressions, and body language, providing actionable critiques and performance scoring (ELO-style).

🏗️ Project Structure

eloquence-ai/
├── voice-ai/           # Person A: Voice + Speech AI
├── vision-ai/          # Person B: Facial Expression + Gesture Recognition  
├── game-ui/            # Person C: Game Loop, Elo Scoring, UI
├── shared/             # Shared utilities and schemas
└── docs/              # Documentation and integration guides

🧩 MVP Modules

🔊 Voice + Speech AI (Person A)

Real-time transcription using OpenAI Whisper/Vapi API
Tone analysis and filler word detection
Speech pace and rhythm analysis
AI-generated structured feedback using Claude/GPT-4
Output: JSON with speech stats + feedback + emotion score

🎥 Facial Expression + Gesture Recognition (Person B)

Webcam-based computer vision analysis
Eye contact and gaze estimation
Facial emotion detection (confidence, engagement)
Body posture and hand gesture recognition
Output: JSON with timestamps, gestures/emotions, posture score

🎮 Game Loop, Elo Scoring, UI (Person C)

Interactive frontend with mock audience reactions
Real-time feedback display panel
Elo-style ranking system
Performance history and progress tracking
Output: Frontend MVP with UI integration and scoring logic

🛠️ Tech Stack

AI & ML

Voice: OpenAI Whisper, Vapi API
LLM: Claude 3.5 Sonnet, GPT-4o
Computer Vision: MediaPipe, TensorFlow.js, OpenCV
Vector DB: Pinecone/Weaviate (optional)

Frontend & Backend

Framework: Next.js + Tailwind CSS
Real-time: WebRTC, WebSocket
Agent Orchestration: LangGraph (optional)
Fast Inference: Groq (optional)

🚀 Quick Start

Clone and setup modules:
```
git clone <repo-url>
cd eloquence-ai
```

Setup each module:

# Voice AI Module
cd voice-ai && npm install

# Vision AI Module  
cd ../vision-ai && npm install

# Game UI Module
cd ../game-ui && npm install

Environment Setup:

# Copy example env files
cp voice-ai/env.example voice-ai/.env
cp vision-ai/env.example vision-ai/.env
cp game-ui/env.example game-ui/.env

Start development:

# Terminal 1: Voice AI
cd voice-ai && npm run dev

# Terminal 2: Vision AI
cd vision-ai && npm run dev

# Terminal 3: Game UI
cd game-ui && npm run dev

🎯 Prize Track Targets

✅ Creativity: Multimodal, Elo-style gamified public speaking
✅ Productivity: Makes users better communicators
✅ Voice AI (Vapi): Real-time voice transcription
✅ Multimodal Agent (Unify): Vision + voice + LLM
✅ Claude (Anthropic): Empathy-based feedback
✅ LLM Agent (Nobel Era): Feedback + rebuttal generation

📋 Development Checklist

Each teammate clones repo + sets up subfolder for their module
Start local JSON schema for feedback format (voice + visual)
Schedule mid-build integration checkpoint
Implement real-time data flow between modules
Create unified scoring system
Add audience reaction simulation
Implement Elo ranking algorithm
Add performance analytics dashboard

🤝 Integration Points

Data Flow

Voice AI → JSON Feedback → Game UI
Vision AI → JSON Feedback → Game UI
Game UI → Elo Score → Performance History

Shared Schemas

Speech feedback format
Visual feedback format
Performance scoring metrics
Elo ranking data structure

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

AnthropicIn code
CSSIn code
ExpressIn code
JavaScriptIn code
Next.jsIn code
OpenAIIn code
ReactIn code
Tailwind CSSIn code
TypeScriptIn code
FirebaseClaimed
FlaskClaimed
Google GeminiClaimed
Node.jsClaimed
PostgreSQLClaimed

9 of 14 appear in the indexed code. 5 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.

AI coding agents

No AI coding agent signals were found in this repository.

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

295 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.