# Project export: SharkProof This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Ever felt the sweaty palms, the racing heartbeat, and bubbling nervousness before a life-changing interview. You're in luck because SharkProof is here. Never fear interviews again. - Devpost: https://devpost.com/software/sharkproof - GitHub: https://github.com/EW0824/CalHacks24 - Video: https://www.youtube.com/embed/n0bIJ221ung?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Vikram Penumarti (21 commits), Edmond (14 commits), aaravurg (2 commits), Harsh-Karia (1 commits) ## Devpost submission (written by the team) ### Inspiration As students, we have had to face countless interviews, whether it be for school clubs, research positions, or those coveted internships. A majority of these interviews are held virtually, through zoom or automated platforms like HireVue. These interviews and the days leading up to them are often filled with nervousness and anxiety. Often it is also hard to schedule mock interviews with engineers in positions that are similar to those that would conduct interviews. Mostly, students end up winging this process and achieving poor results. Having worked in early stage startups before, we saw a similar trend in early stage founders who were applying to incubators like YCombinator and TechStars or even raising their seed and Series A rounds. The resemblance of the problem allowed us to think even more about the problems that founders face with the biggest being a fear of the actual investors themselves. So we decided to solve this problem through repetition and muscle memory. ### What it does SharkProof is an all in one tool to perfect your interview skills and prepare for your best interview performance. It has two main target audiences, students and founders. Currently, we have focused the tool primarily for founders, especially those founders who have upcoming pitches or interviews with VCs and incubators. Once a founders logs on to the platform, they are allowed to choose a persona that will interview them. This persona can be another founder, such as Elon Musk, or a potential VC like Peter Thiel. Essentially it will let you choose or create the persona of the person that is actually going to be interviewing you in the real world. Once the persona has been selected, you will enter the interview room. In the interview room, the persona you have selected will interview you based on their personality (Mark Cuban might ask for more financial details while Oprah might ask for more impact related information). On top of this, the interviewer’s voice will be matched to their actual voice, allowing you to truly imagine that you are speaking to the actual person. And once your interview is completed, you will receive feedback detailing exactly what you need to work on and what you did well on. And when we say detailed, we truly mean detailed. On top of an overall interview score out of 100, you will be told what emotions you are depicting the most throughout your interview as well as emotions for specific questions. And the same will apply for your hand gestures and facial expressions. Suppose you are touching your hair too often or talking while not making eye contact, you will know exactly when that occurred and how to fix it. ### How we built it We built this using technologies from Hume AI, Cartesia, Google’s Gemini Model, Whisper from OpenAI and Groq. For the backend, we built it using a Flask backend with Python. We used Flask because it allows for us to setup the backend in Python which was ideal since we were making many LLM and API calls to AI models. For the frontend, we decided to go with React because of its component reusability features, its integration abilities with Flask, and its efficiency due to the Virtual DOM features. The two foundational models we use are Hume AI’s EVI (voice-to-voice) model and the Facial Expressions Model. The EVI model takes in the user’s audio input and converts it into an audio and text embedding. It then uses Hume's emotion mapping technology and identifies scores for prevalence of 48 different emotions in the interview. We do this in a sentence by sentence method so your emotions are accurately tracked across the interview. This is then used by Gemini within the Hume model configuration to understand and create a response as well as new follow-up questions, which are outputted as the voice of the interviewer. Our second model, the Facial Expressions Model is used to identify the emotions of the face of the interviewee and then mapped onto the correct portions of the audio so that we can make sure that the voice and the facial emotions are in line. Next we sent all of this data to the backend as soon as the interview is completed and use a Llama 3.1 model that we feed with a custom interview scoring algorithms based off of weights we decide as well as a holistic response quality analysis by the LLM model and output feedback and an interview score. This is also then displayed graphically for better understanding. ### Challenges we ran into One of the biggest challenges we ran into was coordinating and measuring the data from the Hume models. This was because the voice-to-voice model would give emotion outputs in the sentence to sentence interval while the facial expressions model would only accept videos in a 5 second batch. So recursively splitting and processing the video in 5 seconds, which we also did using Whisper and mapping that onto the voice data was quite difficult because there was no simple way to achieve that. This led to some completed dictionaries nested within dictionaries and many calls in the backend to process the data accurately. Another challenge was to come up with a custom algorithm to determine which emotions and which facial expressions should have what sort of weights in our model and how much of an effect they should have for our overall score’s equation. And lastly, we had a little bit of trouble with making sure the web sockets were routing our traffic correctly and opening and closing when we wanted them to. ### Accomplishments we're proud of Some major accomplishments that we were proud of were actually integrating AI personas within our application with voices that are super realistic. We were also really proud of taking on the challenge of essentially merging the two Hume models which was a major challenge that even the Hume team is currently working on. And lastly, we thoroughly enjoyed making a project that we ourselves would use to improve our interview preparation skills in the future. ### What we learned We learned the importance of having a deep understanding of the model architecture and the input and output sequences of models for projects that are closely related to maximizing the model’s potential. We also learned about handling binary file data from frontend to backend and vice versa. We got the chance to delve into web socket programming and understand the importance of clear communications between the data types that are handled in the backend ### What's next Something we really wanted to implement but didn’t get the chance to do was create AI deepfakes of the interviewers. This would offer a complete persona of the interviewer, essentially building muscle memory for the founder which could be tapped into when they go into the actual interview. One other feature we wanted to incorporate was the ability for students to add their resume and job description, allowing for the interviewer to be pre-aware of their skills and experience level and ask questions based on those inputs. That would lead to the ideal interview simulation environment. ## README (from the GitHub repository) # CalHacks24 ## Detected evidence (automated analysis) Indexed codebase: 28 recognized source files, 175 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (37 of 37) ``` .gitignore .gitmodules Backend/.gitignore Backend/aarav_requirements.txt Backend/requirements.txt Backend/src/EVI/__init__.py Backend/src/EVI/authenticator.py Backend/src/EVI/connection.py Backend/src/EVI/devices.py Backend/src/EVI/evi.py Backend/src/server.py Backend/src/transcription.py Backend/src/utils.py EVI/old/evi_sdk.ipynb EVI/old/evi_sdk.py EVI/src/authenticator.py EVI/src/connection.py EVI/src/devices.py EVI/src/main.py Frontend/.gitignore Frontend/index.html Frontend/package.json Frontend/postcss.config.js Frontend/public/zuck.avif Frontend/src/components/ChooseModal.jsx Frontend/src/components/Modal.jsx Frontend/src/index.css Frontend/src/main.jsx Frontend/src/pages/EVI.ts Frontend/src/pages/Homepage.jsx Frontend/src/pages/Interview.jsx Frontend/src/pages/Report.jsx Frontend/src/pages/Routing.jsx Frontend/tailwind.config.js Frontend/tsconfig.json Frontend/vite.config.js README.md ``` ### Dependencies - Backend/requirements.txt: cartesia@==1.0.14, Flask-Cors@==5.0.0, flask[async]@==3.0.3, hume@==0.7.2, moviepy@==1.0.3, numpy, openai-whisper@==20240930, python-dotenv - Frontend/package.json: @cartesia/cartesia-js@^1.1.0, @heroicons/react@^1.0.6, @types/react@^18.3.10, @types/react-dom@^18.3.0, @vitejs/plugin-react@^4.3.2, autoprefixer@^10.4.20, axios@^1.7.7, chart.js@^4.4.5, globals@^15.9.0, hume@^0.9.1, postcss@^8.4.47, react@^18.3.1, react-chartjs-2@^5.2.0, react-dom@^18.3.1, react-modal@^3.16.1, react-router-dom@^6.27.0, react-webcam@^7.2.0, tailwindcss@^3.4.14, typescript@^5.6.3, vite@^5.4.8 ### Recent commits (newest first) - demo version - changes - changes - data trasnfer - some more code - working version - groq - Send timestamped emotions information to the backend after conversation finishes - changes - Receive and process emotions data from Hume EVI - Merge branch 'main' of github.com:EW0824/CalHacks24 - wo - questions - hume debugging - Remove Hume EVI from backend - Set up Cartesia - qiuestions - Get Hume EVI to work on the frontend with typescript api - Add ts support and Elon image - frontedn changes ## Key source files (fetched from GitHub, selected and truncated for size) ### Backend/requirements.txt ``` flask[async]==3.0.3 Flask-Cors==5.0.0 openai-whisper==20240930 cartesia==1.0.14 hume==0.7.2 moviepy==1.0.3 numpy python-dotenv ``` ### Frontend/package.json ``` { "name": "calhacks24", "private": true, "version": "0.0.0", "type": "module", "scripts": { "dev": "vite", "build": "vite build", "lint": "eslint .", "preview": "vite preview" }, "dependencies": { "@cartesia/cartesia-js": "^1.1.0", "@heroicons/react": "^1.0.6", "axios": "^1.7.7", "chart.js": "^4.4.5", "hume": "^0.9.1", "react": "^18.3.1", "react-chartjs-2": "^5.2.0", "react-dom": "^18.3.1", "react-modal": "^3.16.1", "react-router-dom": "^6.27.0", "react-webcam": "^7.2.0" }, "devDependencies": { "@types/react": "^18.3.10", "@types/react-dom": "^18.3.0", "@vitejs/plugin-react": "^4.3.2", "autoprefixer": "^10.4.20", "globals": "^15.9.0", "postcss": "^8.4.47", "tailwindcss": "^3.4.14", "typescript": "^5.6.3", "vite": "^5.4.8" } } ``` ### Frontend/src/main.jsx ```javascript import { StrictMode } from "react"; import { createRoot } from "react-dom/client"; import App from "./pages/Routing.jsx"; import "./index.css"; import Modal from "react-modal"; Modal.setAppElement("#root"); createRoot(document.getElementById("root")).render( , ); ``` ### EVI/src/main.py ```python # main.py import asyncio import os from authenticator import Authenticator from connection import Connection from devices import AudioDevices from dotenv import load_dotenv from pyaudio import PyAudio, paInt16 # Audio format and parameters FORMAT = paInt16 CHANNELS = 1 SAMPLE_WIDTH = 2 # PyAudio.get_sample_size(pyaudio, format=paInt16) CHUNK_SIZE = 1024 async def main(): """ Main asynchronous function to set up audio devices, authenticate, and connect to the Hume AI websocket. """ # Initialize PyAudio instance pyaudio = PyAudio() # # List available audio input and output devices # input_devices, output_devices = AudioDevices.list_audio_devices(pyaudio) # # Choose the audio input device and get its sample rate # input_device_index, input_device_sample_rate = AudioDevices.choose_device( # input_devices, "input" # ) # # Choose the audio output device # output_device_index = AudioDevices.choose_device(output_devices, "output") # Instead of listing and choosing, just get default input_device_index, output_device_index, input_device_sample_rate = AudioDevices.get_default_input_output_devices(pyaudio) # Open the audio stream with the selected parameters audio_stream = pyaudio.open( format=FORMAT, channels=CHANNELS, frames_per_buffer=CHUNK_SIZE, rate=input_device_sample_rate, input=True, output=True, input_device_index=input_device_index, output_device_index=output_device_index, ) # Fetch the access token for authentication access_token, config_id = get_access_token_and_config_id() # Construct the websocket URL with the access token socket_url = ( "wss://api.hume.ai/v0/assistant/chat?" f"access_token={access_token}&config_id={config_id}" ) # Connect to the websocket and start the audio stream await Connection.connect( socket_url, audio_stream, input_device_sample_rate, SAMPLE_WIDTH, CHANNELS, CHUNK_SIZE, ) # Close the PyAudio stream and terminate PyAudio audio_stream.stop_stream() audio_stream.close() pyaudio.terminate() def get_access_token_and_config_id() -> str: """ Load API credentials from environment variables and fetch an access token. Returns: str: The access token. Raises: SystemExit: If API key or Secret key are not set. """ load_dotenv() # Attempt to retrieve API key and Secret key from environment variables HUME_API_KEY = os.getenv("HUME_API_KEY") HUME_SECRET_KEY = os.getenv("HUME_SECRET_KEY") HUME_CONFIG_ID = os.getenv("HUME_CONFIG_ID", None) # Ensure API key and Secret key are set if HUME_API_KEY is None or HUME_SECRET_KEY is None: print( "Error: HUME_API_KEY and HUME_SECRET_KEY must be set either in a .env file or as environment variables." ) exit() # Ensure the config ID is set if HUME_CONFIG_ID is None: print( "Error: HUME_CONFIG_ID must be set either in a .env file or as an environment variable." ) exit() # Create an instance of Authenticator with the API key and Secret key authenticator = Authenticator(HUME_API_KEY, HUME_SECRET_KEY) # Fetch the access token access_token = authenticator.fetch_access_token() return access_token, HUME_CONFIG_ID if __name__ == "__main__": """ Entry point for the script. Runs the main asynchronous function. """ asyncio.run(main()) ``` ### Backend/src/server.py ```python import io import os import tempfile import asyncio from flask import Flask, request, jsonify from datetime import datetime from dotenv import load_dotenv from hume import AsyncHumeClient from hume.expression_measurement.batch import Face, Models from hume.expression_measurement.batch.types import InferenceBaseRequest import json from transcription import extract_video_audio from groq import Groq GROQ_KEY = os.getenv("GROQ_KEY") API_KEY = os.getenv("API_KEY") load_dotenv() ######## # SETUP ######## app = Flask(__name__) def get_top_3_facs(predictions): top_facs_per_clip = [] # List to hold results for all clips for pred in predictions: # Each file for prediction in pred.results.predictions: # Per file facs_map = {} for grouped_preds in prediction.models.face.grouped_predictions: for pred in grouped_preds.predictions: # Per frame per file facs = pred.facs for facs_score in facs: # Store the score along with the file name if facs_score.name not in facs_map: facs_map[facs_score.name] = ( facs_score.score, prediction.file, ) else: # Update the score, but keep the same file name existing_score, _ = facs_map[facs_score.name] facs_map[facs_score.name] = ( existing_score + facs_score.score, prediction.file, ) # Average the scores by the number of frames (assuming 15 frames) for key in facs_map.keys(): score, file = facs_map[key] facs_map[key] = (score / 15.0, file) # Get the top 3 FACS scores sorted by score top_3_facs = sorted(facs_map.items(), key=lambda x: x[1][0], reverse=True)[ :3 ] # Store the top FACS scores for this clip top_facs_per_clip.extend( (facs_name, score, file) for facs_name, (score, file) in top_3_facs ) return top_facs_per_clip async def process_videos_hume(client, files): face_config = Face(facs={}) models_chosen = Models(face=face_config) stringified_configs = InferenceBaseRequest(models=models_chosen) job_id = ( await client.expression_measurement.batch.start_inference_job_from_local_file( json=stringified_configs, file=files ) ) await poll_for_completion(client, job_id) job_predictions = await client.expression_measurement.batch.get_job_predictions( id=job_id ) top_facs_scores = get_top_3_facs(job_predictions) return top_facs_scores async def poll_for_completion(client: AsyncHumeClient, job_id, timeout=120): """ Polls for the completion of a job with a specified timeout (in seconds). Uses asyncio.wait_for to enforce a maximum waiting time. """ try: # Wait for the job to complete or until the timeout is reached await asyncio.wait_for(poll_until_complete(client, job_id), timeout=timeout) except asyncio.TimeoutError: # Notify if the polling operation has timed out print(f"Polling timed out after {timeout} seconds.") async def poll_until_complete(client: AsyncHumeClient, job_id): """ Continuously polls the job status until it is completed, failed, or an unexpected status is encountered. Implements exponential backoff to reduce the frequency of requests over time. """ last_status = None delay = 1 # Start with a 1-second delay while True: # Wait for the specified delay before making the next status check await asyncio.sleep(delay) # Retrieve the current job details job_details = await client.expression_measurement.batch.get_job_details(job_id) status = job_details.state.status # If the status has changed since the last check, print the new status if status != last_status: print(f"Status changed: {status}") last_status = status if status == "COMPLETED": # Job has completed successfully print("\nJob completed successfully:") # Convert timestamps from milliseconds to datetime objects created_time = datetime.fromtimestamp( job_details.state.created_timestamp_ms / 1000 ) started_time = datetime.fromtimestamp( job_details.state.started_timestamp_ms / 1000 ) ended_time = datetime.fromtimestamp( job_details.state.ended_timestamp_ms / 1000 ) # Print job details neatly print(f" Created at: {created_time}") print(f" Started at: {started_time}") print(f" Ended at: {ended_time}") print(f" Number of errors: {job_details.state.num_errors}") print(f" Number of predictions: {job_details.state.num_predictions}") break elif status == "FAILED": # Job has failed print("\nJob failed:") # Convert timestamps from milliseconds to datetime objects created_time = datetime.fromtimestamp( job_details.state.created_timestamp_ms / 1000 ) started_time = datetime.fromtimestamp( job_details.state.started_timestamp_ms / 1000 ) ended_time = datetime.fromtimestamp( job_details.state.ended_timestamp_ms / 1000 ) # Print error details neatly print(f" Created at: {created_time}") print(f" Started at: {started_time}") print(f" Ended at: {ended_time}") print(f" E [truncated — 11689 more characters] ``` ### Frontend/postcss.config.js ```javascript export default { plugins: { tailwindcss: {}, autoprefixer: {}, }, }; ``` ### Frontend/tailwind.config.js ```javascript /** @type {import('tailwindcss').Config} */ export default { content: ["./src/**/*.{js,jsx,ts,tsx}"], theme: { extend: {}, }, plugins: [], }; ``` ### Frontend/vite.config.js ```javascript import { defineConfig } from 'vite' import react from '@vitejs/plugin-react' // https://vitejs.dev/config/ export default defineConfig({ plugins: [react()], }) ``` ### Frontend/index.html ```html SharkProof
``` ### Frontend/src/index.css ```css @tailwind base; @tailwind components; @tailwind utilities; :root { background-color: black; scroll-behavior: smooth; } ``` [17 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]