# Project export: Ai's Two Cents This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: An Application with the goal of providing improvement recommendations to presentators through an emotional analysis of an audience in relationship to speaker data by HumeAI. - Devpost: https://devpost.com/software/ai-s-two-cents - GitHub: https://github.com/ezsinehan/calHacks2023-Ais-Two-Cents - Video: https://www.youtube.com/embed/F27D5-REpOg?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Sinehan Ezhilmuthu (5 commits) ## Devpost submission (written by the team) ### Inspiration At UC Merced, alongside an upper-classman student, I recently started helping with his open-source polling software which had placed a heavy focus on data, as it is the purpose of the poll, at a very granular scale, for instance, it may collect data checking if placing a poll choice in a different spot would influence which poll choice the user picked. When I first saw HumeAI, I realized the amount of data this could provide to a speaker if he had the emotional data of an audience. ### What it does The software would compare the speaker's speech and the audience's emotional data collected by HumeAI, which would allow the speaker to receive feedback specific feedback on what they can change by processing the collected data using some other software, for this test example we use OpenAI's GPT model. How I built it I built the test example, using a react frontend to present the sample data, a Python flask backend to connect to the APIs, parse the data, and send data to the frontend, using HumeAI's API to get the audience's emotional data, as well as using OpenAI GPT-3 to formulate specific recommendations to the user. ### Challenges we ran into Immediately off the bat, this being my first hackathon and the first time I've attempted to build something so on the spot like this, I lacked the understanding of the importance of planning. I spent the first whole day, trying different ways to reuse the code provided by HumeAI, as I originally wanted to create a real-time application that provided real-time recommendations to the presenter. After realizing that I wasn't going to be able to use the code provided by Hume, I finally decided to switch over to a post-action recommendation system, since I did not know how to implement a WebSocket as I only understood what even was on the second day. Now after spending the first day, wasting time due to improper planning, I waste the majority of the second day due to improper planning, I decided to start working on my project by implementing a file upload system to my front end which would take an mp4 file to analyze through the API. After spending a whopping three hours due to my lack of knowledge on how to save the file from the front end to the backend I refused to learn it and was looking for someone else who did the same thing, so I could just replicate them. After being unable to do that for three hours, I finally decided to learn enough flask to complete my task which took another 2 hours. After a total of 5 hours, I used Flask for the first time to send the mp4 file from the front end to the backend server, and then decided not the use the file uploader in the end since I later found out I was unable to send any bigger files through for some reason. Now that I had an mp4 file to work with, I needed to focus on implementing the HumeAI API to process the mp4 file to get data for the audience's emotions, which I knew I couldn't find someone online who was doing this, so I understood the spk to the best of my abilities and implemented it with not many hiccups. Now, that I had the data, so from here I had to parse through 3000+ lines of data and simply the data to a point where I pass through the gpt model which was easier than I expected with the help of online resources. Now that I had the simplified data which was a mere 50 lines, I manually added the sample speech to the data. This is another mistake I made, I rushed trying to get a working model, that I didn't get all the data I needed originally, this mistake is again related to improper planning alongside not understand how hard it would be to go back and do this. Then I implemented the OpenAI gpt API, sent the data, formulated a prompt, and got specific recommendations on what the speaker could change. Linked the data the GPT model presents to the front-end which took long again due to lack of flask knowledge. At this point, I realized I didn't have the time or energy to keep going, so I put the sample video, speech, top emotions, and recommendations all on the front end and fell asleep at my workstation. I was able to continue developing my love for computer science. Accomplishments that I'm proud of I had a group member who showed up late after hacking started and after the teammate mixer, which is fine but then he bailed on me since he found another group with three people. I ended up finding another person who also had no group but after only the first day, he quit on me since there was no clear end in sight and everything we did failed. I had to push myself to keep going and I had fun doing it. I preserved, there were times were I doubted myself rethinking whole career choices, but I know that even the greats had doubted their choices so I kept going and kept learning. I learned so much and my number one goal for coming to this hackathon was making a submission as I heard from my more experienced friend that many people don’t even submit. What I learned Planning, it is actually the most important step in the development process and will save you so much time by preventing doing unneeded things. Don’t postone tasks unless you absolutely have to, finish tasks thoroughly before moving on as it requires more work to come back and finish later than it does now, you might not end up coming back to it Work on Github and document process there as well for more accurate development experience and information of past code if needed Don’t replicate similar code in your project, use it to understand better what you need to do since replicating code for your use case will probably take the same amount of time if not more and give you better code along with better understanding. ### What's next Currently, I’ve only processed sample data of the audience and the speech transcribed, in a very non-catered way, I want to improve how the data is processed as well as process the data in a more specific way focusing on more important aspects, like specific emotions and emotions at specific frames or seconds, and have better way to group data than averages of all emotions individuality. I love HumeAI, and I want to take this project as far as I possible can and further. https://docs.google.com/document/d/1fi9nQwI1Z-3NfR8Y7knmnVS2Sg_RjzrEgIzEIi8bNl8/edit?usp=sharing ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 18 recognized source files, 19 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Flask (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (29 of 29) ``` .DS_Store backend/gettingData/humesdk.py backend/gettingData/parsingPredictions/dataMerger.py backend/gettingData/parsingPredictions/emotionSpecifier.py backend/gettingData/parsingPredictions/merged_data.json backend/gettingData/parsingPredictions/parsed_data.json backend/gettingData/parsingPredictions/specificPredictions.json backend/gettingData/parsingPredictions/specificPredictions.py backend/gettingData/predictions.json backend/gettingResponses/gptresponse.py backend/gettingResponses/recommendations.json backend/gettingResponses/server.py backend/uploads/text.json frontend/.gitignore frontend/package.json frontend/public/index.html frontend/public/manifest.json frontend/public/robots.txt frontend/README.md frontend/src/App.css frontend/src/FileUpload.js frontend/src/index.js frontend/src/RecommendationsComponent.js frontend/src/reportWebVitals.js frontend/src/RunGptResponseAndRefresh.js frontend/src/SampleText.js frontend/src/setupTests.js frontend/src/VideoPlayer.js README.md ``` ### Dependencies - frontend/package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, proxy@http://localhost:5000, react@^18.2.0, react-dom@^18.2.0, react-scripts@5.0.1, web-vitals@^2.1.4 ### Recent commits (newest first) - final commit, only one use case - show the sample data and processed thru openai - able to process data and respond with recs - first set of sample data implementing gptapi - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### frontend/package.json ``` { "name": "client", "version": "0.1.0", "private": true, "dependencies": { "@testing-library/jest-dom": "^5.17.0", "@testing-library/react": "^13.4.0", "@testing-library/user-event": "^13.5.0", "proxy": "http://localhost:5000", "react": "^18.2.0", "react-dom": "^18.2.0", "react-scripts": "5.0.1", "web-vitals": "^2.1.4" }, "scripts": { "start": "react-scripts start", "build": "react-scripts build", "test": "react-scripts test", "eject": "react-scripts eject" }, "eslintConfig": { "extends": [ "react-app", "react-app/jest" ] }, "browserslist": { "production": [ ">0.2%", "not dead", "not op_mini all" ], "development": [ "last 1 chrome version", "last 1 firefox version", "last 1 safari version" ] } } ``` ### frontend/src/index.js ```javascript import "./App.css"; import React from "react"; import ReactDOM from "react-dom/client"; import FileUpload from "./FileUpload"; import reportWebVitals from "./reportWebVitals"; import VideoPlayer from "./VideoPlayer"; import RecommendationsComponent from "./RecommendationsComponent"; import SampleText from "./SampleText"; const root = ReactDOM.createRoot(document.getElementById("root")); root.render(
{/* */}
); // If you want to start measuring performance in your app, pass a function // to log results (for example: reportWebVitals(console.log)) // or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals reportWebVitals(); ``` ### backend/gettingResponses/server.py ```python from flask import Flask, flash, request, redirect, url_for, jsonify from werkzeug.utils import secure_filename import os import json from flask_cors import CORS import subprocess import sys # Initialize Flask app and set up CORS app = Flask(__name__) CORS(app) # Configuration settings app.config['UPLOAD_FOLDER'] = './uploads' VIDEO_FILENAME = 'video.mp4' # The fixed filename for the uploaded video @app.route('/', methods=['GET', 'POST']) def upload_file(): if request.method == 'POST': # check if the post request has the file part if 'file' not in request.files: flash('No file part') return redirect(request.url) file = request.files['file'] # If the user does not select a file, the browser submits an empty file without a filename. if file.filename == '': flash('No selected file') return redirect(request.url) if file: # Check if "video.mp4" already exists and delete it if os.path.exists(os.path.join(app.config['UPLOAD_FOLDER'], VIDEO_FILENAME)): os.remove(os.path.join(app.config['UPLOAD_FOLDER'], VIDEO_FILENAME)) # Save the uploaded file as "video.mp4" file.save(os.path.join(app.config['UPLOAD_FOLDER'], VIDEO_FILENAME)) return redirect(url_for('download_file', name=VIDEO_FILENAME)) @app.route('/get_recommendations', methods=['GET']) def get_recommendations(): try: with open('./recommendations.json', 'r') as f: recommendations = json.load(f) print("API hit, read file successfully", recommendations) # Debugging line return jsonify(recommendations) except Exception as e: print(f"Error: {e}") # Debugging line return jsonify({"error": "Failed to read recommendations"}), 500 @app.route('/run_gpt_response', methods=['POST']) def run_gpt_response(): try: # Run gptresponse.py subprocess.run([sys.executable, '/Users/sinehanezhilmuthu/Desktop/csShit/stev2/backend/gettingResponses/gptresponse.py'], check=True) return jsonify({"status": "success"}), 200 except subprocess.CalledProcessError as e: print(f"Error: {e}") return jsonify({"status": "error"}), 500 if __name__ == "__main__": app.run() ``` ### frontend/src/App.css ```css video { float: right; margin-right: 50px; } ``` ### frontend/src/setupTests.js ```javascript // jest-dom adds custom jest matchers for asserting on DOM nodes. // allows you to do things like: // expect(element).toHaveTextContent(/react/i) // learn more: https://github.com/testing-library/jest-dom import '@testing-library/jest-dom'; ``` ### frontend/src/reportWebVitals.js ```javascript const reportWebVitals = onPerfEntry => { if (onPerfEntry && onPerfEntry instanceof Function) { import('web-vitals').then(({ getCLS, getFID, getFCP, getLCP, getTTFB }) => { getCLS(onPerfEntry); getFID(onPerfEntry); getFCP(onPerfEntry); getLCP(onPerfEntry); getTTFB(onPerfEntry); }); } }; export default reportWebVitals; ``` ### frontend/src/VideoPlayer.js ```javascript import React from "react"; import video from "./video.mp4"; class VideoPlayer extends React.Component { render() { return (
); } } export default VideoPlayer; ``` ### backend/gettingData/humesdk.py ```python from hume import HumeBatchClient from hume.models.config import FaceConfig import json client = HumeBatchClient("skiGzxJnlS86cGiGHNtAGrYbABaQEyLv7ru73BeGG8pCRiC4") # Replace 'file_path.jpg' with the actual path to your local image file. files = ["../uploads/video.mp4"] config = FaceConfig() job = client.submit_job([], [config], files=files) print(job) print("Running...") job.await_complete() predictions = job.get_predictions() # Save predictions to a JSON file with open("predictions.json", "w") as json_file: json.dump(predictions, json_file, indent=4) print("Predictions saved to 'predictions.json'") ``` ### frontend/src/RunGptResponseAndRefresh.js ```javascript async function runGptResponseAndRefresh() { // Run gptresponse.py through Flask const response = await fetch("http://127.0.0.1:5000/run_gpt_response", { method: "POST", }); const data = await response.json(); if (data.status === "success") { // Refresh recommendations fetch("http://127.0.0.1:5000/get_recommendations") .then((response) => response.json()) .then((data) => { setRecommendations(data.recommendations); }); } else { console.error("Failed to run gptresponse.py"); } } // Inside your component's return statement return (
{/* ... rest of your component */}
); ``` ### frontend/src/SampleText.js ```javascript import React from "react"; function SampleText() { return (

Sample Speech

From 0:01-0:02: You can study for half of a hour.

From 0:02-0:04: It doesn't take an half hour break to recharge your batteries

Sample Top Emotions presented by Audience

@0:00 Calmness Concentration Interest

@0:01 Calmness Concentration Interest

@0:02 Calmness Concentration Interest

@0:03 Calmness Concentration Confusion

@0:04 Calmness Concentration Confusion

); } export default SampleText; ``` [7 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]