# Project export: Floo
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: Cal Hacks 11.0
- Tagline: We're building an AI-powered web app that simulates interviews, assesses performance using speech-to-text, and provides personalized feedback driven by conversation history and resume data!
- Devpost: https://devpost.com/software/floo-s1tc53
- GitHub: https://github.com/dangzdylan/floo.git
- Team: 3 GitHub contributor(s) — Arnav Khinvasara (23 commits), slapblackjack (15 commits), Alvin Tan (11 commits)
## Devpost submission (written by the team)
### Overview
The inspiration for Floo came from the need for personalized interview preparation. Many candidates struggle with behavioral interviews and lack the resources to practice effectively. We aimed to create a solution that tailors the interview experience to individual users based on their unique backgrounds and qualifications. Floo is an AI-driven platform that simulates behavioral interviews. It provides users with realistic interview scenarios and gives personalized feedback. By analyzing the user’s past experiences and resumes, Floo recommends the best responses, helping users build confidence and improve their interviewing skills. We developed Floo using a combination APIs including Hume, OpenAI, and Deepgram. The backend is powered by Flask, which manages user data and interacts with a database storing users’ past experiences and resumes. The front end is built with React, creating a seamless and interactive user experience. One of the main challenges was ensuring that the AI accurately interpreted user experiences and provided meaningful feedback. We also faced difficulties in designing an intuitive user interface that effectively communicated the AI’s recommendations. Integrating the database with the AI model posed additional technical challenges. We are proud to have successfully developed an AI agent to conduct realistic behavioral interviews. We go beyond a simple AI system that would only ask questions and give surface-level advice. Driven by the user's personal experiences derived from past responses and resume details, it curates advice specifically for the user, maximizing its impact on their learning experience. Through this project, we learned the importance of user-centered design and the value of iterative testing. We gained hands-on experience with speech/text APIs as well as, deepening our understanding of how to leverage AI in practical applications. Collaborating as a team taught us the significance of communication and adaptability in problem-solving. Moving forward, we plan to enhance Floo’s capabilities by incorporating more advanced AI algorithms for better feedback and recommendations. We aim to expand our database to include a wider range of industries and roles, providing users with a more comprehensive practice experience. Additionally, we want to explore partnerships with career services and educational institutions to reach a broader audience.
## README (from the GitHub repository)
# floo
## Detected evidence (automated analysis)
Indexed codebase: 30 recognized source files, 53 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Firebase (technology) — claimed on Devpost, not found in the code
- Flask (technology) — claimed on Devpost, not found in the code
- OpenAI (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (46 of 46)
```
.DS_Store
.github/workflows/firebase-hosting-pull-request.yml
backend/.DS_Store
backend/.firebaserc
backend/.gitignore
backend/data.json
backend/deepgram_test.py
backend/firebase.json
backend/flaskbase.py
backend/hume_funcs.py
backend/interview_assesment.py
backend/public/404.html
backend/public/index.html
backend/resume.txt
backend/resume1.py
backend/speech_text_message.json
backend/text_to_speech.py
backend/transcript.txt
frontend/.DS_Store
frontend/.gitignore
frontend/package.json
frontend/public/.DS_Store
frontend/public/index.html
frontend/public/manifest.json
frontend/public/robots.txt
frontend/README.md
frontend/src/.DS_Store
frontend/src/App.css
frontend/src/App.js
frontend/src/App.test.js
frontend/src/components/AudioRecorder.css
frontend/src/components/AudioRecorder.jsx
frontend/src/components/InterviewScreen.css
frontend/src/components/InterviewScreen.jsx
frontend/src/components/InterviewSelection.css
frontend/src/components/InterviewSelection.jsx
frontend/src/components/Onboarding.css
frontend/src/components/Onboarding.jsx
frontend/src/components/SynopsisScreen.css
frontend/src/components/SynopsisScreen.jsx
frontend/src/components/TextSpeechFunction.jsx
frontend/src/index.css
frontend/src/index.js
frontend/src/reportWebVitals.js
frontend/src/setupTests.js
README.md
```
### Dependencies
- frontend/package.json: @deepgram/sdk@^3.8.1, @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, axios@^1.7.7, dotenv@^16.4.5, react@^18.3.1, react-dom@^18.3.1, react-scripts@^5.0.1, web-vitals@^2.1.4
### Recent commits (newest first)
- Update README.md
- more front end
- merg conf fix
- fuck
- audio finally works
- Merge branch 'ownBranch'
- Merge branch 'main' of https://github.com/dangzdylan/floo
- almost dun
- added perform_interview
- gg
- 3rd page uiux
- Merge branch 'main' of https://github.com/dangzdylan/floo
- added data storage for assessment
- Merge branch 'main' of https://github.com/dangzdylan/floo
- text to speech for each interview prob
- Merge branch 'ownBranch'
- all flows together
- fix mergconf
- added text to speech
- clean up
## Key source files (fetched from GitHub, selected and truncated for size)
### frontend/package.json
```
{
"name": "floo",
"version": "0.1.0",
"private": true,
"proxy": "http://localhost:5000",
"dependencies": {
"@deepgram/sdk": "^3.8.1",
"@testing-library/jest-dom": "^5.17.0",
"@testing-library/react": "^13.4.0",
"@testing-library/user-event": "^13.5.0",
"axios": "^1.7.7",
"dotenv": "^16.4.5",
"react": "^18.3.1",
"react-dom": "^18.3.1",
"react-scripts": "^5.0.1",
"web-vitals": "^2.1.4"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
```
### frontend/src/index.js
```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';
const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
);
// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();
```
### frontend/src/App.js
```javascript
import './App.css';
import React, { useState } from 'react';
import InterviewSelection from './components/InterviewSelection';
import Onboarding from './components/Onboarding';
import InterviewScreen from './components/InterviewScreen';
import SynopsisScreen from './components/SynopsisScreen';
//import AudioRecorder from './components/AudioRecorder';
function App() {
const [onboardingVisible, setOnboardingVisible] = useState(true)
const [interviewSelectionVisible, setInterviewSelectionVisible] = useState(false)
const [interviewScreenVisible, setInterviewScreenVisible] = useState(false)
const [interviewLength, setInterviewLength] = useState(0);
const [synopsisScreenVisible, setSynopsisScreenVisible] = useState(false)
const afterOnBoardHandler = () => {
setOnboardingVisible(false)
setInterviewSelectionVisible(true)
}
const afterSelectionHandler = (length) => {
setInterviewLength(length)
setInterviewSelectionVisible(false)
setInterviewScreenVisible(true)
}
const afterInterviewHandler = () => {
setInterviewScreenVisible(false)
setSynopsisScreenVisible(true)
}
const afterSynopsisHandler = () => {
setSynopsisScreenVisible(false)
setOnboardingVisible(true)
}
return (
{onboardingVisible && ( afterOnBoardHandler()}/>)}
{interviewSelectionVisible && ( afterSelectionHandler(length)}/>)}
{interviewScreenVisible && ()}
{synopsisScreenVisible && ()}
);
}
export default App;
```
### backend/resume1.py
```python
import PyPDF2
def pdf_to_text(pdf_path):
with open(pdf_path, 'rb') as pdf_file:
pdf_reader = PyPDF2.PdfReader(pdf_file)
text = ''
for page_num in range(len(pdf_reader.pages)):
page = pdf_reader.pages[page_num]
text += page.extract_text()
with open("resume.txt", 'w', encoding='utf-8') as txt_file:
txt_file.write(text)
```
### backend/text_to_speech.py
```python
import os
from dotenv import load_dotenv
from deepgram import (
DeepgramClient,
SpeakOptions,
)
load_dotenv()
def text_to_speech(text):
try:
SPEAK_OPTIONS = {"text": text}
filename = "text_to_speech.wav"
# STEP 1: Create a Deepgram client using the API key from environment variables
deepgram = DeepgramClient(api_key=os.getenv("NEW_DG_KEY"))
# STEP 2: Configure the options (such as model choice, audio configuration, etc.)
options = SpeakOptions(
model="aura-asteria-en",
encoding="linear16",
container="wav"
)
# STEP 3: Call the save method on the speak property
response = deepgram.speak.v("1").save(filename, SPEAK_OPTIONS, options)
print(response.to_json(indent=4))
return filename
except Exception as e:
print(f"Exception: {e}")
```
### backend/deepgram_test.py
```python
import os
from dotenv import load_dotenv
from deepgram import DeepgramClient, PrerecordedOptions
import json
# The API key we created in step 3
load_dotenv()
DEEPGRAM_API_KEY = os.getenv('DEEPGRAM_API_KEY')
def run_deepgram(PATH_TO_FILE):
deepgram = DeepgramClient(DEEPGRAM_API_KEY)
try:
with open(PATH_TO_FILE, 'rb') as buffer_data:
payload = { 'buffer': buffer_data }
options = PrerecordedOptions(
smart_format=True, model="nova-2", language="en-US"
)
response = deepgram.listen.prerecorded.v('1').transcribe_file(payload, options)
speech_string = response['results']['channels'][0].alternatives[0].transcript
print(speech_string)
with open('speech_text_message.json', 'w') as json_file:
json.dump({"message": speech_string}, json_file, indent=4)
except FileNotFoundError:
print(f"Error: The file {PATH_TO_FILE} does not exist.")
except Exception as e:
print(f"An error occurred: {e}")
if __name__ == '__main__':
main()
```
### backend/hume_funcs.py
```python
import asyncio
import os
from datetime import datetime
from dotenv import load_dotenv
from typing import List
from hume import AsyncHumeClient
from hume.expression_measurement.batch import Face, Models
from hume.expression_measurement.batch.types import UnionPredictResult
load_dotenv()
HUME_API_KEY = os.getenv("HUME_API_KEY")
def top_emotions(file):
client = AsyncHumeClient(api_key=HUME_API_KEY)
local_filepaths = [
open("backend/uploaded_audio.wav", mode="rb")
]
# Create configurations for each model you would like to use (blank = default)
face_config = Face()
# Create a Models object
models_chosen = Models(face=face_config)
# Create a stringified object containing the configuration
stringified_configs = InferenceBaseRequest(models=models_chosen)
# Start an inference job and print the job_id
job_id = client.expression_measurement.batch.start_inference_job_from_local_file(
json=stringified_configs, file=local_filepaths
)
print(job_id)
job_predictions = client.expression_measurement.batch.get_job_predictions(
id=job_id
)
print(job_predictions)
top_emotions("backend/uploaded_audio.wav")
```
### backend/flaskbase.py
```python
import os
from flask import Flask, request, send_file, jsonify
from werkzeug.utils import secure_filename
import PyPDF2
from openai import OpenAI
from resume1 import pdf_to_text
from dotenv import load_dotenv
from flask_cors import CORS
from deepgram_test import run_deepgram
from text_to_speech import text_to_speech
from interview_assesment import *
# Load the OpenAI API key from environment variables
load_dotenv()
openai_api_key = os.getenv("OPENAI_API_KEY")
# Initialize the OpenAI client with the API key
client = OpenAI(api_key=openai_api_key)
app = Flask(__name__)
CORS(app)
"""
interview = []
questions = []
length = []
first_question = False
@app.route("/interview/setup", methods=["POST"])
def interview_setup():
length.append(request.json.get("length"))
interview.append(
{
"role": "system", "content": ""
You are acting as an interviewer that asks behavioral questions.
The user is applying for a software engineering role.
Your job is to ask general behavioral questions that a typical interview would target.
After each question, you will ask a follow up question about any details important to the job field. This encourages the user to elaborate on his answers and be more through.
Make sure to be enthuisatic and show that you are an attentive listener!
The first question you ask will always be "Tell me about yourself".
""
}
)
@app.route("/interview", methods=["POST", "GET"])
def interview_question():
length[0] -= 1
if (first_question[0]):
answer = request.json.get("message") # Extract the user's answer
interview.append({"role": "user", "content": answer})
first_question[0] = True
interview.append({"role": "system", "content": "Give a kind short response if the user has answered a question. Then ask another one."})
interview.append("role": "user", "content": "Next question")
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[interview]
)
# Access the content of the message
message = completion.choices[0].message.content
questions.append(message)
return jsonify({"response": message})
@app.route("/interview/followup")
def followup_question():
answer = request.json.get("message") # Extract the user's answer
interview.append(
{"role": "user", "content": answer},
{"role": "system", "content": "You will recieve an answer to the question. Respond with a follow up question that encourages the user to elaborate on vague details."}
)
completion = client.chat.completions.create(
model="gpt-4o-mini",
messages=[interview]
)
message = completion.choices[0].message.content
questions.append(message)
if (length[0] > 0):
return jsonify({"response": message, "continue": True})
else:
return jsonify({"response": message, "continue": False})
@app.route("/interview/save")
def interview_save():
answer = request.json.get("message") # Extract the user's answer
interview.append(
{"role": "user", "content": answer}
)
return jsonify({"reponse": "Thank you"})
"""
@app.route("/resumeParser", methods=["POST"])
def resumeParser():
if 'file' not in request.files:
return "No file part", 400
file = request.files['file']
if file.filename == '':
return "No selected file", 400
# Secure the filename
filename = secure_filename(file.filename)
# Save the file temporarily
filepath = f'./{filename}'
file.save(filepath)
pdf_to_text(filepath)
return "File saved successfully"
@app.route('/upload_audio', methods=['POST'])
def upload_audio():
file = request.files['audio']
file.save('uploaded_audio.wav')
run_deepgram('uploaded_audio.wav')
return "File uploaded successfully", 200
@app.route('/text_to_speech', methods=['POST'])
def handle_text_to_speech():
data = request.json
text = data.get('text', '')
# Generate the speech file
audio_file = text_to_speech(text)
if audio_file:
return send_file(audio_file, mimetype="audio/wav")
else:
return jsonify({"error": "Failed to generate audio"}), 500
@app.route('/get_feedback', methods=['GET'])
def get_feedback():
return "./data.json"
behavioralQuestions = [
"Tell me about a time when you faced a difficult technical problem. How did you approach it?",
"Describe a situation where you had to work with a challenging team member. How did you handle it?",
"Give an example of a project where you had to quickly learn a new technology or framework.",
"Tell me about a time when you received critical feedback. How did you respond and improve?",
"Describe a situation where you had to balance competing priorities or deadlines. How did you manage it?",
"Can you share an example of a time you took ownership of a task or project?",
"Tell me about a time when you had to collaborate with a non-technical team member. How did you ensure effective communication?",
"Describe a situation where you had to make a decision with incomplete information. How did you handle it?",
"Tell me about a project where things didn’t go as planned. What did you do to address the issues?",
"Describe a time when you had to work under pressure. How did you maintain productivity?",
"Tell me about a time when you had to debug a complex issue. How did you approach the problem?",
"Can you share an example of a time when you helped a team member resolve a technical issue?",
"Tell me about a time you went above and beyond in a project. What motivated you?",
"Describe a situation where you had to adapt to significant changes in a project. How did you adjust?",
"Tell me about a time when you contributed to improving a process or workflow. What was the outcome?"
]
@app.route('/perform_interview', method=['
[truncated — 184 more characters]
```
### backend/interview_assesment.py
```python
from openai import OpenAI
from flask import Flask
from flaskbase import *
import json
import re
load_dotenv(dotenv_path='/backend')
openai_api_key = os.getenv(OPENAI_API_KEY)
def transcript_saver(transcript):
f = open("transcript.txt", "a")
for message in transcript:
f.write(message["role"] + ": " + message["content"])
f.close()
def assessment_start(transcript_path, resume_path):
# Takes in path to transcript txt file and resume txt file
# Returns feedback text
client = OpenAI(api_key=openai_api_key)
# Create files for assistant
transcript = client.files.create(
file=open(transcript_path, "rb"),
purpose="assistants"
)
resume = client.files.create(
file=open(resume_path, "rb"),
purpose="assistants"
)
assistant = client.beta.assistants.create(
model="gpt-4o-mini",
instructions="Output feedback according to the two given files.",
tools=[{"type": "code_interpreter"}, {"type": "file_search"}]
)
# Upload files and add to vector store
transcript_vector = client.beta.vector_stores.create(name="Transcript")
resume_vector = client.beta.vector_stores.create(name="Resume")
file_paths = [transcript_path]
file_streams = [open(path, "rb") for path in file_paths]
assistant = client.beta.assistants.update(
assistant_id=assistant.id,
tool_resources={"file_search": {"vector_store_ids": [transcript_vector.id]}},
)
client.beta.vector_stores.file_batches.upload_and_poll(
vector_store_id=transcript_vector.id, files=file_streams
)
file_paths = [resume_path]
file_streams = [open(path, "rb") for path in file_paths]
client.beta.vector_stores.file_batches.upload_and_poll(
vector_store_id=resume_vector.id, files=file_streams
)
# Create thread
thread = client.beta.threads.create(
messages=[
{
"role": "user",
"content": """
I have attatched a transcript file. Now I will attatch a resume
file.
""",
"attachments": [
{
"file_id": transcript.id,
"tools": [{"type": "code_interpreter"}, {"type": "file_search"}]
}
]
},
{
"role": "user",
"content": """
You are given both an interview transcript and a resume.
Tell the user feedback on the user responses. How could they improve?
Use specific examples. Be concise and do not write much.
""",
"attachments": [
{
"file_id": resume.id,
"tools": [{"type": "code_interpreter"}, {"type": "file_search"}]
}
]
}
]
)
run0 = client.beta.threads.runs.create_and_poll(
thread_id=thread.id,
assistant_id=assistant.id
)
messages = list(client.beta.threads.messages.list(thread_id=thread.id, run_id=run0.id))
message2 = client.beta.threads.messages.create(
thread.id,
role="user",
content="""
For every answer given by the user in the interview, output an improved answer.
Do not output anything else. Be very concise and do not write a lot for every answer.
""",
)
run1 = client.beta.threads.runs.create_and_poll(
thread_id=thread.id,
assistant_id=assistant.id
)
messages2 = list(client.beta.threads.messages.list(thread_id=thread.id, run_id=run1.id))
message3 = client.beta.threads.messages.create(
thread.id,
role="user",
content="""
Output a single number, nothing else, representing the rating of the user's interview performance.
""",
)
run2 = client.beta.threads.runs.create_and_poll(
thread_id=thread.id,
assistant_id=assistant.id
)
messages3 = list(client.beta.threads.messages.list(thread_id=thread.id, run_id=run2.id))
# delete files, thread, assistant
client.beta.threads.delete(thread.id)
client.beta.assistants.delete(assistant.id)
client.files.delete(transcript.id)
client.files.delete(resume.id)
with open(transcript_path, "r") as text_file:
transcript_string = text_file.read()
y = {"interview": transcript_string,
"better_answers": messages2[0].content[0].text.value,
"feedback": messages[0].content[0].text.value,
"score": messages3[0].content[0].text.value
}
with open("backend/data.json",'r+') as file:
file_data = json.load(file)
file_data["previous_interviews"].append(y)
file.seek(0)
json.dump(file_data, file, indent = 4)
return y
def obtain_rating(transcript_path, resume_path):
# Takes in path to transcript txt file and resume txt file
# Returns feedback text
client = OpenAI(api_key=openai_api_key)
# Create files for assistant
transcript = client.files.create(
file=open(transcript_path, "rb"),
purpose="assistants"
)
resume = client.files.create(
file=open(resume_path, "rb"),
purpose="assistants"
)
assistant = client.beta.assistants.create(
model="gpt-4o-mini",
instructions="Output a single number from one to ten according to the given interview transcript.",
tools=[{"type": "code_interpreter"}, {"type": "file_search"}]
)
# Upload files and add to vector store
transcript_vector = client.beta.vector_stores.create(name="Transcript")
resume_vector = client.beta.vector_stores.create(name="Resume")
file_paths = [transcript_path]
file_streams = [open(path, "rb") for path in file_paths]
assistant = client.beta.assistants.update(
assistant_id=assistant.id,
tool_r
[truncated — 1495 more characters]
```
### frontend/src/setupTests.js
```javascript
// jest-dom adds custom jest matchers for asserting on DOM nodes.
// allows you to do things like:
// expect(element).toHaveTextContent(/react/i)
// learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom';
```
[18 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]