# Project export: Speechful This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Want to learn public speaking but too nervous to get on stage? Speechful is the coach you need! - Devpost: https://devpost.com/software/speechful-th5368 - GitHub: https://github.com/onKTun/speechful-cal-hacks-12.0 - Video: https://www.youtube.com/embed/VaLGTY42k2o?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — onKTun (25 commits), Ethan Tran (18 commits), Ylann Bouis (4 commits), Arthur Wang (2 commits) ## Devpost submission (written by the team) ### Inspiration Public speaking can be intimidating, and practicing alone often lacks meaningful feedback. We wanted to create a tool that provides real-time, personalized coaching, helping users improve confidence, clarity, and engagement. ### What it does Speechful is a public speaking coaching app that analyzes your speech in real-time and gives personalized feedback. It evaluates factors like sentiment, confidence, and speech familiarity, helping users refine both content and delivery. ### How we built it Frontend: React + TypeScript for a clean and responsive interface where users can record speeches and see feedback. Backend: Node.js + Express to handle API requests, process data, and return actionable insights. Speech Analysis: Claude API and Lava API for sentiment analysis, scoring the speaker’s confidence and engagement. Voice detection AI APIs to track how well the user is familiar with their speech (intonation, pauses, and fluency). Claude API and Lava API for sentiment analysis, scoring the speaker’s confidence and engagement. Voice detection AI APIs to track how well the user is familiar with their speech (intonation, pauses, and fluency). Live Feedback: Combines the above metrics to provide personalized advice on pacing, clarity, and emotional impact. ### Challenges we ran into Integrating multiple AI APIs and ensuring their outputs were consistent and interpretable. Real-time processing: delivering feedback without noticeable lag. Designing actionable feedback, presenting insights in a way that’s easy for users to act on without overwhelming them. ### Accomplishments we're proud of Successfully integrated Claude and Lava APIs for sentiment scoring and personalized feedback. Built a working prototype that tracks speech familiarity and provides actionable coaching. Designed a real-time dashboard that gives intuitive, immediate insights during practice sessions. ### What we learned Combining multiple AI APIs can create rich, multidimensional feedback for users. Real-time processing requires careful optimization to maintain responsiveness. Personalized feedback is far more effective than generic tips as users respond better when advice is tailored to their own performance. ### What's next Add tone and emotion detection to give more nuanced feedback. Introduce progress tracking over multiple sessions to help users see improvement over time. Expand multilingual support for non-English speakers. Build a mobile version for on-the-go practice. ## README (from the GitHub repository) # Speechful: Cal Hacks 12.0 ## Inspiration Public speaking can be intimidating, and practicing alone often lacks meaningful feedback. We wanted to create a tool that provides real-time, personalized coaching, helping users improve confidence, clarity, and engagement. ## What it does Speechful is a public speaking coaching app that analyzes your speech in real-time and gives personalized feedback. It evaluates factors like sentiment, confidence, and speech familiarity, helping users refine both content and delivery. ## How we built it - Frontend: React + TypeScript for a clean and responsive interface where users can record speeches and see feedback. - Backend: Node.js + Express to handle API requests, process data, and return actionable insights. - Speech Analysis: - Claude API and Lava API for sentiment analysis, scoring the speaker’s confidence and engagement. - Voice detection AI APIs to track how well the user is familiar with their speech (intonation, pauses, and fluency). - Live Feedback: Combines the above metrics to provide personalized advice on pacing, clarity, and emotional impact. ## Challenges we ran into - Integrating multiple AI APIs and ensuring their outputs were consistent and interpretable. - Real-time processing: delivering feedback without noticeable lag. - Designing actionable feedback, presenting insights in a way that’s easy for users to act on without overwhelming them. ## Accomplishments that we're proud of - Successfully integrated Claude and Lava APIs for sentiment scoring and personalized feedback. - Built a working prototype that tracks speech familiarity and provides actionable coaching. - Designed a real-time dashboard that gives intuitive, immediate insights during practice sessions. ## What we learned - Combining multiple AI APIs can create rich, multidimensional feedback for users. - Real-time processing requires careful optimization to maintain responsiveness. - Personalized feedback is far more effective than generic tips as users respond better when advice is tailored to their own performance. ## What's next for Speechful - Add tone and emotion detection to give more nuanced feedback. - Introduce progress tracking over multiple sessions to help users see improvement over time. - Expand multilingual support for non-English speakers. - Build a mobile version for on-the-go practice. ## Running Speechful Locally To run Speechful on your local machine, follow these steps: 1. **Start the server** ```bash cd server npm install node app.js 2. **Start the client** ```bash cd client npm install npm run dev 3. **Open the app** Once both server & client are up, open your browser and navigate to URL shown. ## Detected evidence (automated analysis) Indexed codebase: 44 recognized source files, 169 KB. - CSS (language) — detected in the code - Express (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Anthropic (technology) — claimed on Devpost, not found in the code - Node.js (technology) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code - AI coding agent: GitHub Copilot — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (50 of 50) ``` .github/copilot-instructions.md .gitignore AI_FEEDBACK_FIXES.md client/eslint.config.js client/index.html client/package.json client/src/App.tsx client/src/components/Navigation.tsx client/src/components/SpeechCoach/Captions.tsx client/src/components/SpeechCoach/Controls.tsx client/src/components/SpeechCoach/EmojiOverlay.tsx client/src/components/SpeechCoach/FeedbackScreen.tsx client/src/components/SpeechCoach/FreeSessionSplash.tsx client/src/components/SpeechCoach/HIGHLIGHTING_TRANSCRIPT_USAGE.md client/src/components/SpeechCoach/HighlightingTranscript.tsx client/src/components/SpeechCoach/LearningPage.tsx client/src/components/SpeechCoach/REHEARSAL_RECORDING_FLOW.md client/src/components/SpeechCoach/RehearsalPage.tsx client/src/components/SpeechCoach/SentimentDisplay.tsx client/src/components/SpeechCoach/SessionScreen.tsx client/src/components/SpeechCoach/SetupScreen.tsx client/src/components/SpeechCoach/STOP_BUTTON_FLOW.md client/src/components/SpeechCoach/SuggestionDisplay.tsx client/src/components/SpeechCoach/Timer.tsx client/src/components/SpeechCoach/TranscriptDisplay.tsx client/src/components/SpeechCoach/VisualSuggestion.tsx client/src/components/SpeechCoach/WebcamDisplay.tsx client/src/hooks/useSentimentCapture.ts client/src/hooks/useVisualSentiment.ts client/src/Landing.tsx client/src/learning/LearningPage.tsx client/src/learning/utils/processTranscript.ts client/src/main.tsx client/src/RehearsalCapture.tsx client/src/tailwind.css client/src/ThemeContext.tsx client/src/types/index.ts client/src/WebcamCapture.tsx client/src/webcamCaptureFeedback.tsx client/tsconfig.app.json client/tsconfig.json client/tsconfig.node.json client/vite.config.ts FEEDBACK_DISPLAY_FIXES.md FIXES_APPLIED.md README.md server/app.js server/package.json server/SETUP_GUIDE.md VIDEO_FEEDBACK_FIXES.md ``` ### Dependencies - client/package.json: @eslint/js@^9.36.0, @tailwindcss/vite@^4.1.16, @types/node@^24.6.0, @types/react@^19.1.16, @types/react-dom@^19.1.9, @vitejs/plugin-react@^5.0.4, eslint@^9.36.0, eslint-plugin-react-hooks@^5.2.0, eslint-plugin-react-refresh@^0.4.22, globals@^16.4.0, lucide-react@^0.548.0, react@^19.1.1, react-dom@^19.1.1, react-router@^7.9.4, react-router-dom@^7.9.4, react-webcam@^7.2.0, string-comparison@^1.3.0, tailwindcss@^4.1.16, typescript@~5.9.3, typescript-eslint@^8.45.0, vite@^7.1.7 - server/package.json: @deepgram/sdk@^4.11.2, cors@^2.8.5, dotenv@^17.2.3, express@^5.1.0, multer@^2.0.2, ws@^8.18.3 ### Recent commits (newest first) - Merge branch 'main' of https://github.com/onKTun/speechful-cal-hacks-12.0 - DONE - clarify readme - official readme - added back visual suggestions - Merge branch 'Front-end' - push - fixed difficulty switch - improvements to tracking - Testing - Merge branch 'main' into Front-end - Testing - Merge branch 'main' of https://github.com/onKTun/speechful-cal-hacks-12.0 - pause/play fix - slight improvements in transcription and tracking - Merge pull request #6 from onKTun/suggestions - Merge branch 'main' into suggestions - adding edits for suggestions - cleanup some files - basic transcript tracking ## Key source files (fetched from GitHub, selected and truncated for size) ### FEEDBACK_DISPLAY_FIXES.md ```markdown # Feedback Display Fixes ## Changes Made ### Improved Feedback Text Formatting **Problem**: AI feedback was displaying as a single paragraph, making it hard to read bullet points and structure. **Solution**: Enhanced the feedback display to properly format: - **Bullet points** (detects `-`, `•`, or numbered lists) - **Headers** (detects capitalized text ending with `:`) - **Proper spacing** between paragraphs **Implementation**: ```typescript // Splits feedback by newlines and formats each line feedback.split('\n').map((line, index) => { const isBullet = line.trim().startsWith('-') || line.trim().startsWith('•') || line.trim().match(/^\d+\./); const isHeader = line.trim().match(/^[A-Z][^.!?]*:$/); // Render with appropriate styling }) ``` **Visual Improvements**: - Bullet points get a purple bullet (•) marker - Headers are bold and colored purple - Proper line spacing between paragraphs - Fallback for empty feedback --- ### Enhanced Error Handling **Added**: - Validation that feedback is a string before displaying - Better error messages - Debug logging to track feedback flow - Fallback display when no feedback is available **Console Logs Added**: ```javascript console.log("Setting feedback:", data.result.substring(0, 100) + "..."); ``` --- ### Expected Feedback Format The AI should return feedback in this format: ``` What you do well: • You maintain good eye contact with the camera, which helps create a connection with your audience. Areas to Improve: • Your facial expressions could be more varied and engaging. • You're fidgeting slightly, which can be distracting. • Consider using more hand gestures to emphasize key points. ``` --- ## Testing 1. Record a session (at least 10 seconds) 2. Stop the recording 3. Check the feedback screen: - ✅ Feedback should display with proper formatting - ✅ Bullet points should have purple markers - ✅ Headers should be bold and purple - ✅ Text should be readable and well-spaced ### Debug Checklist If feedback doesn't display: 1. Check browser console for: - `"=== FETCHING AI FEEDBACK ==="` - `"AI feedback received:"` - `"Setting feedback:"` 2. Check server console for: - `"Received averaged ratings:"` - `"AI API response:"` 3. Verify feedback data structure: - Should be a string - Should contain bullet points or paragraphs --- ## Files Modified | File | Changes | |------|---------| | `client/src/components/SpeechCoach/FeedbackScreen.tsx` | Enhanced feedback formatting, added validation, improved error handling | --- ## Success Criteria ✅ Feedback displays with proper formatting ✅ Bullet points are visually distinct ✅ Headers are properly styled ✅ Text is readable and well-spaced ✅ Error messages are helpful ✅ Console logs provide debugging info ``` ### VIDEO_FEEDBACK_FIXES.md ```markdown # Video Recording & AI Feedback Fixes ## Date: Current Session ## Issues Fixed ### 1. ✅ Server Error Handling - API Endpoints Not Sending Responses **Problem**: Both `/sentiment/visual` and `/feedback` endpoints were not sending responses when errors occurred, causing the client to hang indefinitely. **Root Cause**: - Error handlers only logged errors but didn't send HTTP responses - Client was waiting for responses that never came **Solution**: Added proper error responses to both endpoints: ```javascript // Before: catch (err) { console.error(err); } // After: catch (err) { console.error(err); res.status(500).json({ error: 'Failed to analyze visual sentiment', details: err.message }); } ``` **Files Modified**: - `server/app.js` (lines 103-106, 247-250) --- ### 2. ✅ Video Recording Not Starting - Webcam Stream Timing Issue **Problem**: Video recording sometimes failed because the webcam stream wasn't ready when recording attempted to start. **Root Cause**: - Only checked once with a 100ms delay - No retry logic if stream wasn't ready - Silent failures made debugging difficult **Solution**: - Added retry logic with exponential backoff (up to 10 retries at 200ms intervals) - Enhanced error logging to show exactly what went wrong - Better stream readiness checking **Files Modified**: - `client/src/components/SpeechCoach/RehearsalPage.tsx` (lines 88-172) **Key Changes**: ```typescript // Retry logic to ensure webcam stream is ready let retryCount = 0; const maxRetries = 10; const retryInterval = 200; const tryStartRecording = () => { if (webcamRef.current?.stream) { console.log("Webcam stream ready, starting recording"); startRecording(); } else if (retryCount < maxRetries) { retryCount++; console.log(`Webcam stream not ready yet, retry ${retryCount}/${maxRetries}`); setTimeout(tryStartRecording, retryInterval); } else { console.error("Failed to start recording: webcam stream not available after retries"); } }; ``` --- ### 3. ✅ Client-Side Error Handling Improvements **Problem**: Client wasn't properly handling API errors, leading to unhelpful error messages. **Solution**: Enhanced error handling in multiple places: **Files Modified**: - `client/src/components/SpeechCoach/FeedbackScreen.tsx` (lines 45-70) - `client/src/hooks/useVisualSentiment.ts` (lines 36-58) **Improvements**: - Check HTTP response status before parsing JSON - Extract error details from API responses - Show specific error messages to users - Better console logging for debugging --- ## Testing Instructions ### 1. Set Up Environment Variables Create a `.env` file in the `server/` directory: ```env DEEPGRAM_API_KEY=your_deepgram_api_key_here LAVA_BASE_URL=your_lava_base_url_here MODEL_URL=your_model_url_here LAVA_FORWARD_TOKEN=your_lava_forward_token_here ``` ### 2. Start the Server ```bash cd server npm start ``` ### 3. Start the Client ```bash cd client npm run dev ``` ### 4. Test Recording 1. Navigate to Rehe [truncated — 2264 more characters] ``` ### server/package.json ``` { "name": "server", "version": "1.0.0", "main": "index.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC", "description": "", "dependencies": { "@deepgram/sdk": "^4.11.2", "cors": "^2.8.5", "dotenv": "^17.2.3", "express": "^5.1.0", "multer": "^2.0.2", "ws": "^8.18.3" } } ``` ### client/package.json ``` { "name": "speechful-cal-hacks-12.0", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "lint": "eslint .", "preview": "vite preview" }, "dependencies": { "@tailwindcss/vite": "^4.1.16", "lucide-react": "^0.548.0", "react": "^19.1.1", "react-dom": "^19.1.1", "react-router": "^7.9.4", "react-router-dom": "^7.9.4", "react-webcam": "^7.2.0", "string-comparison": "^1.3.0", "tailwindcss": "^4.1.16" }, "devDependencies": { "@eslint/js": "^9.36.0", "@types/node": "^24.6.0", "@types/react": "^19.1.16", "@types/react-dom": "^19.1.9", "@vitejs/plugin-react": "^5.0.4", "eslint": "^9.36.0", "eslint-plugin-react-hooks": "^5.2.0", "eslint-plugin-react-refresh": "^0.4.22", "globals": "^16.4.0", "typescript": "~5.9.3", "typescript-eslint": "^8.45.0", "vite": "^7.1.7" } } ``` ### server/app.js ```javascript const express = require("express"); const dotenv = require("dotenv"); const http = require("http"); const cors = require("cors"); const multer = require("multer"); const WebSocket = require("ws"); const { createClient, LiveTranscriptionEvents } = require("@deepgram/sdk"); dotenv.config(); const app = express(); app.use(cors()); app.use(express.json({ limit: '50mb' })); const server = http.createServer(app); // Wrap it in HTTP server const wss = new WebSocket.WebSocketServer({ server }); const PORT = 3000; const deepgramClient = createClient(process.env.DEEPGRAM_API_KEY); let keepAlive; // Configure multer for audio file uploads const upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: 10 * 1024 * 1024, // 10MB limit }, }); // Checks validness of response from Claude (ideally, this should always return true, but we never know) function getValues(str) { try { // Check if the response is a string and follows the expected format if (!str || !str.content || !str.content[0] || !str.content[0].text) { console.error("Invalid response structure:", str); return [-1, -1, -1]; } const text = str.content[0].text; // Try to match JSON in code blocks first let match = text.match(/```json\s*([\s\S]*?)\s*```/); let jsonString = match ? match[1] : text.trim(); const data = JSON.parse(jsonString); // Validate required fields exist and are numbers if (typeof data.facial_expression !== 'number' || typeof data.eye_contact !== 'number' || typeof data.focus !== 'number') { console.error("Missing or invalid fields in response:", data); return [-1, -1, -1]; } return [data.facial_expression, data.eye_contact, data.focus]; } catch (e) { console.error("Error parsing sentiment response:", e.message); console.log("Raw response text:", str?.content?.[0]?.text); return [-1, -1, -1]; } } app.post('/sentiment/visual', async (req, res) => { const { base64Image } = req.body; try { const url = `${process.env.LAVA_BASE_URL}/forward?u=${process.env.MODEL_URL}`; const headers = { 'Content-Type': 'application/json', 'x-api-key': process.env.LAVA_FORWARD_TOKEN, 'anthropic-version': '2023-06-01' }; const requestBody = { model: 'claude-haiku-4-5', max_tokens: 100, messages: [ { role: 'user', content: [ { type: 'image', source: { type: 'base64', media_type: 'image/webp', data: base64Image.split(',')[1] } }, { type: 'text', text: `For each of the following three following categories, give a score from 1 to 10 regarding how well the user is practicing good public speaking. A 1 means poor performance (i.e. doesn't look like they're presenting or they're not on the screen) and 10 is good (don't be afraid to give 10s commonly). This means like looking at the camera when talking (or at least looking near the camera; just as long as it's not far off), and not fidgeting around or looking distracted. Also, do not deduct points for a slightly blurry camera (deduct if really blurry) or lighting (or anything beyond the user's control). Be more extreme with your judgements (both positive and negative). 1) Facial expression - how friendly and engaged they appear 2) Eye contact - how well they look at the camera/audience 3) Focus - how still and attentive they are (not fidgeting) Format the output as a JSON: {facial_expression: ___, eye_contact: ___, focus: ___}. DO NOT SEND ANYTHING OTHER THAN THE JSON AND NO FORMATTING OUTSIDE OF THE OUTERMOST BRACES.` }, ], }, ], system: 'You are a helpful assistant.' }; const response = await fetch(url, { method: 'POST', headers: headers, body: JSON.stringify(requestBody) }); const data = await response.json(); const result = getValues(data); console.log(result); //const result = data.choices[0].message.content; // if (!isValidClaudeJsonResponse(data.content[0].text)) { // data.content[0].text = '```json\n{"facial_expression": 5, "eye_contact": 5, "focus": 5}\n```' // } res.json({ result }); } catch (err) { console.error(err); res.status(500).json({ error: 'Failed to analyze visual sentiment', details: err.message }); } }); // Handle stt websocket/deppgram connection const setupDeepgram = (ws) => { const deepgram = deepgramClient.listen.live({ smart_format: false, interim_results:true, utterance_end_ms: 1000, vad_events: true, endpointing: 300, model: "nova-3", }); if (keepAlive) clearInterval(keepAlive); keepAlive = setInterval(() => { console.log("deepgram: keepalive"); deepgram.keepAlive(); }, 10 * 1000); deepgram.addListener(LiveTranscriptionEvents.Open, async () => { console.log("deepgram: connected"); deepgram.addListener(LiveTranscriptionEvents.Transcript, (data) => { console.log( "deepgram: transcript received: " + data.channel.alternatives[0].transcript ); console.log("ws: transcript sent to client"); ws.se [truncated — 6292 more characters] ``` ### client/src/main.tsx ```typescript import { StrictMode } from 'react' import { createRoot } from 'react-dom/client' import './tailwind.css' import App from './App.tsx' import { BrowserRouter } from 'react-router-dom' import { ThemeProvider } from './ThemeContext.tsx' createRoot(document.getElementById('root')!).render( , ) ``` ### client/src/App.tsx ```typescript import Landing from './Landing.tsx' import { FreeSessionSplash } from './components/SpeechCoach/FreeSessionSplash' import RehearsalPage from './components/SpeechCoach/RehearsalPage' import LearningPage from './components/SpeechCoach/LearningPage' import { Routes, Route } from 'react-router-dom' import LearningPageDev from './learning/LearningPage.tsx' function App() { return ( } /> } /> } /> } /> } /> ) } export default App ``` ### client/src/types/index.ts ```typescript export type Mode = "rehearsal" | "learning"; export type Difficulty = "easy" | "medium" | "hard"; export interface SetupConfig { transcript: string; mode: Mode; difficulty: Difficulty; } export interface SessionState { isStarted: boolean; isPaused: boolean; elapsedTime: number; showTranscript: boolean; } ``` ### client/vite.config.ts ```typescript import { defineConfig } from 'vite' import react from '@vitejs/plugin-react' import tailwindcss from '@tailwindcss/vite' // https://vite.dev/config/ export default defineConfig({ plugins: [react(), tailwindcss()], }) ``` ### client/index.html ```html speechful-cal-hacks-12.0
``` [35 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]