# Project export: StageReady This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Ever felt too anxious to ask for feedback on your presentation? Our app analyzes facial expressions and prosody, giving you non-judgmental, rubric-based feedback to help you improve your confidence. - Devpost: https://devpost.com/software/oratix - GitHub: https://github.com/1meza/StageReady - Demo: https://gamma.app/docs/StageReady--ugx0euzygj7cyu2 - Video: https://www.youtube.com/embed/X6ClOymHh9o?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — NFerreira98 (5 commits), Meza, Isaiah (1 commits) ## Devpost submission (written by the team) ### Inspiration Our inspiration for this project was our personal experience of struggling to prepare for public speaking. Whether through previous hackathons, class projects, or poster presentations, we all have had trouble practicing our speeches. The anxiety of public speaking makes it difficult to want to practice with other people and has been a common obstacle preventing us from improving our speech skills. We learned that many people are too shy to seek to rehearse their speeches with others. This motivated us to create an app that provides non-judgmental feedback based on facial expressions and speech patterns, empowering individuals to practice and perfect their public speaking skills in a human-free, non-judgemental environment. ### What it does Our project starts with a web app where the user can record their presentation in front of their camera. After their presentation concludes, a Q&A takes place with hume's EVI (Empathic Voice Interface) within the context of their speech and a custom prompt. Additionally, we used Deepgram's voice platform to transcribe the user's speech. Groq then parses this speech to rate the user's speaking ability and provide feedback on what they can do to improve. ### How we built it The frontend of our web application was created using react.js as well as shadcn, a component library. The backend of our application uses hume ai as the main technology. We use the face model in their expression measurement tool to gather values for different emotions/expressions the user exhibits in their speech. These values are then evaluated by Groq which provides a rating for the user's presentation in a friendly, yet helpful manner. The other component of our project is the Q&A portion where the user is asked questions based on the topic they discussed. ### Challenges we ran into The biggest challenge we ran into was trying to seamlessly integrate three major technologies into the project; those being hume ai, Deepgram, and Google Gemini. Learning all of these technologies on the spot was challenge enough, but finding a way to connect them all was another level of difficult. It was also a challenge testing the code as the wait times for getting output was rather slow and quickly added up, which meant we had to sparingly re-run our code. ### Accomplishments we're proud of We're proud of successfully integrating Hume AI and Deepgram, offering actionable feedback on both emotional expressions and speech content. Bringing together facial expression analysis and a post-speech Q&A with an empathetic voice interface (EVI) is something we believe will make practicing speeches much more interactive and insightful for users. We're also proud of our ability to transcribe speech in real time and use Google Gemini's NLP to analyze and score it for clarity, structure, and impact. These components, working together, form the basis of a comprehensive feedback system that helps users improve their presentation skills with each use. ### What we learned One major thing we learned was using GitHub to collaborate with backend and frontend developers simultaenously. With one teammate new to hackathons, it was crucial to learn version control for easy management of our codebase. Despite not ultimately using it, we learned about natural language processing (NLP) with Google Gemini. One more thing we learned was how to integrate an AI model into our project with hume ai's face model. This experience emphasized the critical role that emotional awareness plays in effective communication and taught us to appreciate the nuances that come with empathic interactions. ### What's next for Oratix Moving forward, we plan to enhance our application by refining the feedback mechanism. We want to introduce real-time analysis of their face expressions as well as prosody, so they can adjust accordingly during the presentation rather than after. Another feature we want to add is combining EVI with a live avatar. This will simulate interacting with a human who might nod along or acknowledge statements from the presenter throughout their talk. ## README (from the GitHub repository) # StageReady ## Inspiration Our inspiration for this project was our personal experience of struggling to prepare for public speaking. Whether through previous hackathons, class projects, or poster presentations, we all have had trouble practicing our speeches. The anxiety of public speaking makes it difficult to want to practice with other people and has been a common obstacle preventing us from improving our speech skills. We learned that many people are too shy to seek to rehearse their speeches with others. This motivated us to create an app that provides non-judgmental feedback based on facial expressions and speech patterns, empowering individuals to practice and perfect their public speaking skills in a human-free, non-judgemental environment. ## What it does Our project starts with a web app where the user can record their presentation in front of their camera. After their presentation concludes, a Q&A takes place with **hume's EVI** (Empathic Voice Interface) within the context of their speech and a custom prompt. Additionally, we used **Deepgram's voice platform** to transcribe the user's speech. **Groq** then parses this speech to rate the user's speaking ability and provide feedback on what they can do to improve. ## How we built it The frontend of our web application was created using react.js as well as shadcn, a component library. The backend of our application uses hume ai as the main technology. We use the face model in their expression measurement tool to gather values for different emotions/expressions the user exhibits in their speech. These values are then evaluated by Groq which provides a rating for the user's presentation in a friendly, yet helpful manner. The other component of our project is the Q&A portion where the user is asked questions based on the topic they discussed. ## Challenges we ran into The biggest challenge we ran into was trying to seamlessly integrate three major technologies into the project; those being hume ai, Deepgram, and Google Gemini. Learning all of these technologies on the spot was challenge enough, but finding a way to connect them all was another level of difficult. It was also a challenge testing the code as the wait times for getting output was rather slow and quickly added up, which meant we had to sparingly re-run our code. ## Accomplishments that we're proud of We're proud of successfully integrating Hume AI and Deepgram, offering actionable feedback on both emotional expressions and speech content. Bringing together facial expression analysis and a post-speech Q&A with an empathetic voice interface (EVI) is something we believe will make practicing speeches much more interactive and insightful for users. We're also proud of our ability to transcribe speech in real time and use Google Gemini's NLP to analyze and score it for clarity, structure, and impact. These components, working together, form the basis of a comprehensive feedback system that helps users improve their presentation skills with each use. ## What we learned One major thing we learned was using GitHub to collaborate with backend and frontend developers simultaenously. With one teammate new to hackathons, it was crucial to learn version control for easy management of our codebase. Despite not ultimately using it, we learned about natural language processing (NLP) with Google Gemini. One more thing we learned was how to integrate an AI model into our project with hume ai's face model. This experience emphasized the critical role that emotional awareness plays in effective communication and taught us to appreciate the nuances that come with empathic interactions. ## What's next for StageReady Moving forward, we plan to enhance our application by refining the feedback mechanism. We want to introduce real-time analysis of their face expressions as well as prosody, so they can adjust accordingly during the presentation rather than after. Another feature we want to add is combining EVI with a live avatar. This will simulate interacting with a human who might nod along or acknowledge statements from the presenter throughout their talk. ### Demo [![Video Title](https://img.youtube.com/vi/X6ClOymHh9o/0.jpg)](https://youtu.be/X6ClOymHh9o) ## Detected evidence (automated analysis) Indexed codebase: 28 recognized source files, 73 KB. - CSS (language) — detected in the code - Flask (technology) — detected in the code - JavaScript (language) — detected in the code - LangChain (technology) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code ## Codebase structure (from repository index) ### Files (40 of 40) ``` client/.eslintrc.json client/.gitignore client/components.json client/jsconfig.json client/next.config.mjs client/package.json client/postcss.config.mjs client/README.md client/src/app/globals.css client/src/app/layout.js client/src/app/page.js client/src/components/ClientComponent.jsx client/src/components/rubric-screen.jsx client/src/components/speech-ai-settings-modal.jsx client/src/components/ui/button.jsx client/src/components/ui/card.jsx client/src/components/ui/dialog.jsx client/src/components/ui/input.jsx client/src/components/ui/label.jsx client/src/components/ui/progress.jsx client/src/components/ui/switch.jsx client/src/components/ui/textarea.jsx client/src/components/video-call-collapsible-transcript.jsx client/src/components/webcamPage.jsx client/src/context/TranscriptContext.js client/src/lib/utils.js client/tailwind.config.js README.md server/.gitignore server/analyze_expression.py server/app.py server/CalHacks11/server/presentations/job_predictions.json server/job_predictions.json server/prompt.py server/requirements.txt server/run_chatmodel.py server/server.txt server/top_emotions.py server/transcribe.py server/video_utils.py ``` ### Dependencies - client/package.json: @humeai/voice-react@^0.1.16, @radix-ui/react-dialog@^1.1.2, @radix-ui/react-icons@^1.3.0, @radix-ui/react-label@^2.1.0, @radix-ui/react-progress@^1.1.0, @radix-ui/react-slot@^1.1.0, @radix-ui/react-switch@^1.1.1, canvas-confetti@^1.9.3, class-variance-authority@^0.7.0, clsx@^2.1.1, eslint@^8, eslint-config-next@14.2.15, lucide-react@^0.453.0, next@14.2.15, postcss@^8, react@^18, react-dom@^18, react-media-recorder@^1.7.1, react-record-webcam@^1.1.5, react-webcam@^7.2.0, tailwind-merge@^2.5.4, tailwindcss@^3.4.1, tailwindcss-animate@^1.0.7 - server/requirements.txt: deepgram-sdk, flask, flask-cors, google-generativeai, hume, langchain, langchain_core, langchain_groq, moviepy, python-dotenv, requests ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Update README.md - Merge pull request #1 from 1meza/nicholas - Created readme - Added project ## Key source files (fetched from GitHub, selected and truncated for size) ### server/requirements.txt ``` # requirements.txt flask requests hume python-dotenv moviepy flask-cors deepgram-sdk google-generativeai langchain langchain_core langchain_groq ``` ### client/package.json ``` { "name": "client", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@humeai/voice-react": "^0.1.16", "@radix-ui/react-dialog": "^1.1.2", "@radix-ui/react-icons": "^1.3.0", "@radix-ui/react-label": "^2.1.0", "@radix-ui/react-progress": "^1.1.0", "@radix-ui/react-slot": "^1.1.0", "@radix-ui/react-switch": "^1.1.1", "canvas-confetti": "^1.9.3", "class-variance-authority": "^0.7.0", "clsx": "^2.1.1", "lucide-react": "^0.453.0", "next": "14.2.15", "react": "^18", "react-dom": "^18", "react-media-recorder": "^1.7.1", "react-record-webcam": "^1.1.5", "react-webcam": "^7.2.0", "tailwind-merge": "^2.5.4", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "eslint": "^8", "eslint-config-next": "14.2.15", "postcss": "^8", "tailwindcss": "^3.4.1" } } ``` ### server/app.py ```python from flask import Flask, request, jsonify import os import asyncio from analyze_expression import analyze_expression from flask_cors import CORS from transcribe import transcribe_file import json from run_chatmodel import run_chatmodel app = Flask(__name__) # Configure CORS CORS(app, resources={ r"/*": { "origins": ["http://localhost:3000"], "methods": ["GET", "POST", "OPTIONS"], "allow_headers": ["Content-Type"] } }) @app.route('/invoke', methods=['POST']) def invoke(): if request.method == 'POST': data = request.json if data and "speech" in data: result = run_chatmodel(data["speech"]) return jsonify(result), 200 return jsonify({"message": "Missing 'speech' field in request"}), 400 @app.route('/test', methods=['GET', 'POST']) def test(): if request.method == 'POST': data = request.json # Use this for JSON payloads if data and "test" in data: return jsonify({"message": data["test"]}), 200 return jsonify({"message": "Missing 'test' field in request"}), 400 return jsonify({"message": "Server is up and running"}), 200 @app.route('/upload', methods=['POST']) def upload_video(): try: if 'video' not in request.files: print("No video file in request") return jsonify({"error": "No video file provided"}), 400 video = request.files['video'] if video.filename == '': print("No selected file") return jsonify({"error": "No selected file"}), 400 # Create directory if it doesn't exist presentations_dir = os.path.join(os.path.dirname(__file__), 'presentations') os.makedirs(presentations_dir, exist_ok=True) # Generate unique filename from datetime import datetime timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") safe_filename = f"video_{timestamp}.mp4" video_path = os.path.join(presentations_dir, safe_filename) # Save the file try: video.save(video_path) print(f"Video saved to: {video_path}") except Exception as e: print(f"Error saving file: {str(e)}") return jsonify({"error": f"Error saving file: {str(e)}"}), 500 # File analysis try: result = asyncio.run(analyze_expression(video_path)) print("Analysis result:", result) return jsonify({ "message": "Video uploaded and analyzed successfully", "path": video_path, "analysis": result, }), 200 except Exception as e: print(f"Error during analysis: {str(e)}") return jsonify({"error": f"Error during analysis: {str(e)}"}), 500 except Exception as e: print(f"Unexpected error: {str(e)}") return jsonify({"error": f"Unexpected error: {str(e)}"}), 500 @app.route('/transcribe', methods=['POST']) def transcribe(): try: if 'video' not in request.files: print("No video file in request") return jsonify({"error": "No video file provided"}), 400 video = request.files['video'] if video.filename == '': print("No selected file") return jsonify({"error": "No selected file"}), 400 # Create directory if it doesn't exist presentations_dir = os.path.join(os.path.dirname(__file__), 'presentations') os.makedirs(presentations_dir, exist_ok=True) # Generate unique filename from datetime import datetime timestamp = datetime.now().strftime("%Y%m%d_%H%M%S") safe_filename = f"video_{timestamp}.mp4" video_path = os.path.join(presentations_dir, safe_filename) # Save the file try: video.save(video_path) print(f"Video saved to: {video_path}") except Exception as e: print(f"Error saving file: {str(e)}") return jsonify({"error": f"Error saving file: {str(e)}"}), 500 # File analysis try: transcribe_string = asyncio.run(transcribe_file(video_path)) transcribe_result = json.loads(transcribe_string) print("Transcription result:", transcribe_result) return jsonify({ "transcription": transcribe_result }), 200 except Exception as e: print(f"Error during analysis: {str(e)}") return jsonify({"error": f"Error during analysis: {str(e)}"}), 500 except Exception as e: print(f"Unexpected error: {str(e)}") return jsonify({"error": f"Unexpected error: {str(e)}"}), 500 if __name__ == '__main__': app.run(debug=True, port=5000) ``` ### client/src/app/layout.js ```javascript import localFont from "next/font/local"; import "./globals.css"; const geistSans = localFont({ src: "./fonts/GeistVF.woff", variable: "--font-geist-sans", weight: "100 900", }); const geistMono = localFont({ src: "./fonts/GeistMonoVF.woff", variable: "--font-geist-mono", weight: "100 900", }); export const metadata = { title: "Create Next App", description: "Generated by create next app", }; export default function RootLayout({ children }) { return ( {children} ); } ``` ### client/src/app/page.js ```javascript "use client"; import { useState } from "react"; import dynamic from "next/dynamic"; import { SpeechAiSettingsModal } from "@/components/speech-ai-settings-modal"; import { fetchAccessToken } from "hume"; // import { TranscriptContext } from "@/context/transcriptContext"; const WebcamAppComponent = dynamic(() => import("../components/WebcamPage"), { ssr: false, // Disable server-side rendering for this component }); export default async function Home() { const [isSettingsOpen, setIsSettingsOpen] = useState(true); // const [transcript, setTranscript] = useState(""); const accessToken = await fetchAccessToken({ apiKey: process.env.NEXT_PUBLIC_API_KEY, secretKey: process.env.NEXT_PUBLIC_SECRET_KEY, }); console.log(await accessToken); if (!accessToken) { throw new Error(); } return (
{/* */}
); } ``` ### server/prompt.py ```python promptDictionary = { "welcome": "Welcome to the server! Please enter your name: ", } ``` ### server/video_utils.py ```python import moviepy.editor as mp from tempfile import NamedTemporaryFile def split_video_into_segments(video_path, segment_duration=5): video = mp.VideoFileClip(video_path) duration = int(video.duration) segments = [] for start in range(0, duration, segment_duration): end = min(start + segment_duration, duration) segment = video.subclip(start, end) temp_file = NamedTemporaryFile(delete=False, suffix=".mp4") segment.write_videofile(temp_file.name, codec="libx264", audio_codec="aac") segments.append(temp_file.name) return segments ``` ### server/transcribe.py ```python import os import asyncio from dotenv import load_dotenv from deepgram import ( DeepgramClient, PrerecordedOptions, FileSource, ) load_dotenv() # Path to the audio file AUDIO_FILE = "presentations/TestSpeech.mp4" API_KEY = os.getenv("DEEPGRAM_API_KEY") async def transcribe_file(file_path): try: # STEP 1 Create a Deepgram client using the API key deepgram = DeepgramClient(API_KEY) with open(file_path, "rb") as file: buffer_data = file.read() payload: FileSource = { "buffer": buffer_data, } #STEP 2: Configure Deepgram options for audio analysis options = PrerecordedOptions( model="nova-2", smart_format=True, ) # STEP 3: Call the transcribe_file method with the text payload and options response = deepgram.listen.prerecorded.v("1").transcribe_file(payload, options, timeout = 300) # STEP 4: Print the response results = response.to_json(indent=4) return results except Exception as e: print(f"Exception: {e}") ``` ### client/tailwind.config.js ```javascript /** @type {import('tailwindcss').Config} */ module.exports = { darkMode: ["class"], content: [ "./src/pages/**/*.{js,ts,jsx,tsx,mdx}", "./src/components/**/*.{js,ts,jsx,tsx,mdx}", "./src/app/**/*.{js,ts,jsx,tsx,mdx}", ], theme: { extend: { colors: { background: 'hsl(var(--background))', foreground: 'hsl(var(--foreground))', card: { DEFAULT: 'hsl(var(--card))', foreground: 'hsl(var(--card-foreground))' }, popover: { DEFAULT: 'hsl(var(--popover))', foreground: 'hsl(var(--popover-foreground))' }, primary: { DEFAULT: 'hsl(var(--primary))', foreground: 'hsl(var(--primary-foreground))' }, secondary: { DEFAULT: 'hsl(var(--secondary))', foreground: 'hsl(var(--secondary-foreground))' }, muted: { DEFAULT: 'hsl(var(--muted))', foreground: 'hsl(var(--muted-foreground))' }, accent: { DEFAULT: 'hsl(var(--accent))', foreground: 'hsl(var(--accent-foreground))' }, destructive: { DEFAULT: 'hsl(var(--destructive))', foreground: 'hsl(var(--destructive-foreground))' }, border: 'hsl(var(--border))', input: 'hsl(var(--input))', ring: 'hsl(var(--ring))', chart: { '1': 'hsl(var(--chart-1))', '2': 'hsl(var(--chart-2))', '3': 'hsl(var(--chart-3))', '4': 'hsl(var(--chart-4))', '5': 'hsl(var(--chart-5))' } }, borderRadius: { lg: 'var(--radius)', md: 'calc(var(--radius) - 2px)', sm: 'calc(var(--radius) - 4px)' } } }, plugins: [require("tailwindcss-animate")], }; ``` ### server/analyze_expression.py ```python import asyncio import os import time import json from hume import AsyncHumeClient from hume.expression_measurement.batch import Face, Models from hume.expression_measurement.batch.types import InferenceBaseRequest, EmotionScore, FacePrediction from dotenv import load_dotenv from typing import List load_dotenv() HUME_API_KEY = os.getenv("HUME_API_KEY") if not HUME_API_KEY: raise ValueError("HUME_API_KEY is not set. Please check your .env file.") async def analyze_expression(filepath: str): # Initialize an authenticated client client = AsyncHumeClient(api_key=HUME_API_KEY) results = [] with open(filepath, mode="rb") as file: # Create configurations for each model you would like to use (blank = default) face_config = Face() # Create a Models object models_chosen = Models(face=face_config) # Create a stringified object containing the configuration stringified_configs = InferenceBaseRequest(models=models_chosen) # Start an inference job and get the job_id job_id = await client.expression_measurement.batch.start_inference_job_from_local_file( json=stringified_configs, file=[file] ) # Poll for job completion while True: job_details = await client.expression_measurement.batch.get_job_details(job_id) if job_details.state.status == "COMPLETED": break elif job_details.state.status == "FAILED": raise Exception(f"Job {job_id} failed with message: {job_details.state.message}") time.sleep(5) # Wait for 5 seconds before polling again # Get the result of the job job_predictions = await client.expression_measurement.batch.get_job_predictions(job_id) # Convert job_predictions to a serializable format job_predictions_serializable = [convert_to_dict(prediction) for prediction in job_predictions] # Define the directory where you want to save the JSON file output_dir = os.path.join("CalHacks11", "server", "presentations") os.makedirs(output_dir, exist_ok=True) # Create the directory if it doesn't exist # Define the full path for the JSON file output_file_path = os.path.join(output_dir, "job_predictions.json") # Store the result in a JSON file with open(output_file_path, 'w') as json_file: json.dump(job_predictions_serializable, json_file, indent=4) for prediction in job_predictions: results.extend(prediction.results.predictions) # Extend the results list with job predictions # Process the results to generate feedback feedback = generate_feedback(results) return feedback def convert_to_dict(obj): """ Convert an object to a dictionary. """ if isinstance(obj, list): return [convert_to_dict(item) for item in obj] elif hasattr(obj, "__dict__"): return {key: convert_to_dict(value) for key, value in obj.__dict__.items()} else: return obj def sort_emotions(emotions: List[EmotionScore]): return sorted(emotions, key=lambda e: e.score, reverse=True) def generate_feedback(predictions): feedback = [] for prediction in predictions: for grouped_prediction in prediction.models.face.grouped_predictions: for face_prediction in grouped_prediction.predictions: sorted_emotions = sort_emotions(face_prediction.emotions) feedback.append(f"Frame {face_prediction.frame}:") for emotion in sorted_emotions: feedback.append(f" {emotion.name}: {emotion.score:.2f}") return "\n".join(feedback) if __name__ == "__main__": # Example usage filepaths = [ "example_video.mp4" ] result = asyncio.run(analyze_expression(filepaths[0])) print(result) ``` [18 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]