# Project export: Speak Your Mind

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: We conduct presentation analysis--grading emotions, gesture-use, & textual content generated from a speech--then generate pointers to improve on targeted categories w/ audience specific tips.
- Devpost: https://devpost.com/software/speak-your-mind
- GitHub: https://github.com/ArjunAshok17/speak-your-mind
- Team: 3 GitHub contributor(s) — Arjun Ashok (20 commits), Tej Sidhu (6 commits), atri5 (3 commits)

## Devpost submission (written by the team)

### Inspiration

My group and I often struggled to gauge audience perceptions to our projects and as a result, we created this project to do just that.

### What we learned

We learned a lot about API calls and what factors make up a good speech

### What's next

We currently are just in our first iteration. We have plans to expand the audience types and give more detailed analytics.

## README (from the GitHub repository)

# Presentation Analyzer [Speak Your Mind] #
>We can grade emotions, gesture-use, & textual content generated from a video of
a speech and then generate pointers to improve on each >of these categories.

- Emotions of the Presenter are mirrored by the audience; therefore, gauging how 
the presenter is speaking (stress, anxiety, etc.) ⇒ Hume AI can do this as well
- Spoken Content needs to be parsed into something we can process textually ⇒ 
some sort of speech-to-text API for converting the spoken language into a speech
    
    - Google likely has some APIs for thisTextual Content can be analyzed 
    for correctness, delivery, ⇒ LLM from Grammarly or GPT or Claude for 
    understanding and grading the content of the speech for accuracy
- Together AI has an open LLM to use (w/ LLaMa), but we have to ensure 
that payment isn’t crazy
    - https://developer.grammarly.com/docs/writing-suggestions
  
- Body Language is an essential part of the speaker’s appearance on stage 
and therefore reception in the audience ⇒ analyzing hand gestures, 
posture, eye contact, etc. would be powerful to users
    - Lol no idea how to do this, should be CV libraries that can accomplish this but how tf
    - May have to be a part of next steps

- We can study the qualitative aspects that good presenters possess, 
creating heuristics for analyzing the metrics we gauge
    - Rank certain emotions as +/- based on the qualitative study
    - Qualify certain textual content [words, phrases, slang, etc.] as +/
    based on great speeches’ content
    - Classify body language/movements as +/- based on speed of movement, 
    position, etc.
    - Gauge speed, stutter, & clarity of speech and associate with +/- i
    mpact on speech quality

## Next Steps
- Audience Analysis: we could analyze audience emotion to gauge how they 
react to the speeches given (i.e. we can record the live audience in an 
actual speech; Ted Talks, etc. are good resources for this)

- Eventually, we can leverage this to predict how an audience would 
respond to a given speech, thereby providing a more thorough analysis of 
audience reception to the speech content, body language, and emotions of 
the presenter’s specific video

## Elevator Pitch
### Names
- Speak Your Mind
- SymPresent / PresentSym
- GhostWriter

### Use-Cases
- For those with learning disabilities, social anxiety, social disorders, 
etc. our product helps overcome these limitations through targeted 
practice
    - Not just any kind of practice, but practice that helps YOU get 
    better
    - Can be fine-tuned by the person, helping anyone get tips that 
    concentrate on their specific weaknesses
- Similarly, for anyone who’s afraid of speaking, this is a tool that 
    helps break down some of the stress of public speaking through 
    practice
    - Again, fine-tuned to the person so we can better suggest pointers, 
    etc.
    - It’s like a speech coach that knows everything about speaking

- For companies who want to analyze consumer responses to product 
roll-outs Apple, etc. launch products, they may want real-time analysis 
for the audience reactions and which products excite the people most 
interested in their product
- For audience analysis as a whole, we can eventually roll-out features 
for specific audiences (fan-bases, general audience, etc. can all be 
analyzed to create datasets based on their propensity for certain 
behaviors)
    - Again, can be used to see how audiences would react to product 
    announcements based on the speech, textual, and emotional content of 
    an advertisement

- For research purposes, it’ll be useful to have a way to collect data 
about audiences and behavior on a large scale when influenced by a single 
(or multiple) speaker(s)
    - Can conduct case studies on the impact of certain emotions and 
    gestures on audience retention, etc. ⇒ political science, sociology 
    research
    - Can use this dataset to avoid heuristics when grading the users, 
    instead relying on real-world data

    - We can justify the user of heuristics for now since we our audience 
    members ourselves and we conducted a qualitative study, but in the 
    future quantifiable proof would be preferred

## Plan
### Pipeline
Web App


Backend
Frontend



## Detected evidence (automated analysis)

Indexed codebase: 32 recognized source files, 44 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code

## Codebase structure (from repository index)

### Files (55 of 55)

```
app/.dev
app/.gitignore
app/app.py
app/grouped-expressions.json
app/hume_api.py
app/llm_feedback.py
app/package.json
app/predictions.json
app/process_response.py
app/prompt-template.txt
app/public/index.html
app/public/manifest.json
app/public/robots.txt
app/README.md
app/src/App.css
app/src/App.js
app/src/App.test.js
app/src/Components/Banner.css
app/src/Components/Banner.js
app/src/Components/DisplayGauges.js
app/src/Components/PostForm.js
app/src/index.css
app/src/index.js
app/src/reportWebVitals.js
app/src/setupTests.js
case-studies/.DS_Store
case-studies/example-video/hume-response-1.json
case-studies/example-video/hume-response-2.json
case-studies/example-video/hume-response-3.json
case-studies/example-video/hume-response-4.json
case-studies/presentation-analysis/approach.md
case-studies/presentation-analysis/grouped-expressions.json
case-studies/presentation-analysis/hume-expressions.json
case-studies/presentation-analysis/qual_analysis.py
case-studies/presentation-analysis/quant-tracker.csv
case-studies/presentation-analysis/quant-tracker.json
case-studies/README.md
database/README.md
facial-recog/README.md
gen-feedback/llm_feedback.py
gen-feedback/prompt-template.txt
gen-feedback/README.md
hume-api/get_job_details.py
hume-api/get_job_predictions.py
hume-api/grouped-expressions.json
hume-api/hume_api_template.py
hume-api/hume_api.py
hume-api/predictions.json
hume-api/process_response.py
hume-api/README.md
hume-api/wholething.py
LICENSE
package.json
predictions.json
README.md
```

### Dependencies

- app/package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, axios@^1.6.0, react@^18.2.0, react-dom@^18.2.0, react-gauge-chart@^0.4.1, react-scripts@5.0.1, web-vitals@^2.1.4
- package.json: react-gauge-chart@^0.4.1

### Recent commits (newest first)

- removing irrelevant files
- wrapping up some of the previous work
- wrapping up some of the previous work
- finalized the pipeline start to finish, testing still needs to happen
- Added some prompts for prompt engineering
- analysis of the hume api return is done, we need to modify api to call it
- final edits
- generating metrics in progress, almost done aggregating
- Added some default values and gauges and made the website look nicer
- more on the metrics generation
- completed gen_hume_analysis function
- created the framework for the hume-api to work with local files
- processing response in progress, functionality for dataset generation up and running
- created the framework for the hume-api to work with local files
- processing response in progress, functionality for dataset generation up and running
- Created an api endpoint and connected the UI to this endpoint.
- processing response utility for the hume response
- created job production getter function, accepts
- dispatcher functionality being added, some sys paths needed
- finalized hume-api-template

## Key source files (fetched from GitHub, selected and truncated for size)

### case-studies/presentation-analysis/approach.md

```markdown
# Approach for Qualitative & Quantitative Studies
The approach we'll use for this case study is as follows:

0. Grouping: for every expression that's potentially returned by HUME, we should 
group them into more human-readable traits (i.e. Nervousness => lip biting, 
facial tics, hands on face, etc.)
    1. With each of these groups, we can better communicate with users and 
    generate responses for pointers (w/ the LLM) more accurately

1. Qualitative: watch 10 Ted Talks, 10 presentations from famous speakers, and 
10 other instances of influencers [or celebrities] and how they speak. For each 
speech, record whether the specific characteristic worked in or against their 
favor, and whether it was noticeably present or not
    1. At the end, observe patterns among speakers and write down what groups of 
    behaviors helped their presentation the best

2. Quantitative: using the data, we can create correlation matrices to observe 
what fields are similar, modify the groupings as necessary, and figure out what 
groups quantitatively are negatively impacting the performance of a speech, and 
which groups of features are positively impacting
    1. The idea is to put some numbers to the approach we take, even if the 
    numbers are based on some level of qualitative/survey data
```

### package.json

```
{
  "dependencies": {
    "react-gauge-chart": "^0.4.1"
  }
}

```

### app/package.json

```
{
  "name": "calhacks-ui",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@testing-library/jest-dom": "^5.17.0",
    "@testing-library/react": "^13.4.0",
    "@testing-library/user-event": "^13.5.0",
    "axios": "^1.6.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-gauge-chart": "^0.4.1",
    "react-scripts": "5.0.1",
    "web-vitals": "^2.1.4"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

```

### app/app.py

```python
"""
    Defines the functionality for the backend [namely the endpoints being 
    called in the front-end].
"""


# ----------------- Environment Setup ----------------- #
import sys
import os

# current_dir = os.path.dirname(os.path.abspath(__file__))
# parent_dir = os.path.dirname(current_dir)
# sys.path.append(parent_dir + "/hume-api/")
# sys.path.append(parent_dir + "/facial-recog/")
# sys.path.append(parent_dir + "/gen-feedback/")
# sys.path.append(parent_dir + "/app/")

import json
from flask import Flask, request, jsonify
from flask_restful import Resource, Api, reqparse
from flask_cors import CORS
# import pandas as pd
import ast
from enum import Enum
app = Flask(__name__)
CORS(app)
api = Api(app)

from hume_api import *
from llm_feedback import *


# ----------------- Endpoint Wrappers ----------------- #
"""
    Classes for interpreting inputs & outputs.
"""
class Content_Type(Enum):
    SPEAKER = 0
    AUDIENCE = 1
    SPEAKERAUDIENCE = 2


"""
    Class to wrap all methods.
"""
class SYM_Processor(Resource):

    """
        Receives POST response, wraps all necessary calls:

        Dispatcher :: handles the full API request, will return the full text 
                    analysis + pointers, and a JSON with timestamps of each 
                    event/emotion/pointer
        
                :: We’ll obtain input from the web app in the form of a video, 
                    string prompt, and analysis type [speaker, audience, speaker 
                    + audience] ⇒ JSON
                
                        :: This analysis type can later be inferred from number of 
                        faces, lighting, etc. but for simplicity and iteration 
                        sake we can do this for now
        
                :: Throughout this process, we’ll be generating a dictionary of 
                    timestamps [int]: sentiment-analysis [dictionary] that’ll 
                    later be a key output point
                    
                :: We’ll dispatch the facial recognition, gather results, and 
                    separate the input into audience and speaker frames
                        :: For each one of those frames, we’ll dispatch the given 
                        Hume Analysis wrapper that then provides the final 
                        verdicts for each frame
                        :: This will be a JSON/dictionary that associates key 
                        emotions/thoughts with the frame
    """
    def post(self) -> tuple:
        # unpack args & check quality
        request_data = request.json
        if not request_data or 'request_content' not in request_data or 'target' not in request_data:
            return jsonify({
                "status": "error",
                "message": "Invalid request data. Please provide 'request_content' and 'target'."
            }), 400
        
        request_content = request_data["request_content"]
        target = request_data["target"]

        print(request_content + "\n")
        print(target + "\n")

        # dispatcher
        dispatch_params = {
           "API_KEY": os.getenv("HUME_API_KEY"),
           "VIDEO_URL": request_content["content-type"],
           "CONTENT_TYPE": target
        }
        
        aggregate_expressions = gen_hume_analysis(dispatch_params)
        prompt = "get better at moving the audience, I think the subject area is uninteresting to a lot of people at first glance so the better I communicate its importance and sway the audience, the better!"
        user_feedback = gen_llm_feedback(aggregate_expressions, prompt)

        return {
            "appreciation": aggregate_expressions["Appreciation"],
            "impact": aggregate_expressions["Impact"],
            "confidence": aggregate_expressions["Confidence"],
            "engagement": aggregate_expressions["Engagement"],
            "feedback": user_feedback
        }, 200



api.add_resource(SYM_Processor, "/sym-processor")


# ----------------- Feature Iteration & Testing ----------------- #
if __name__ == "__main__":
    # run app
    app.run()


```

### app/src/App.js

```javascript
import logo from './logo.svg';
import './App.css';
import PostForm from './Components/PostForm';
import Banner from './Components/Banner';

function App() {
  return (
    <div className="App">
      <Banner />
      <PostForm/>
    </div>
  );
}

export default App;

```

### app/src/index.js

```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);

// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();

```

### hume-api/get_job_predictions.py

```python



import requests

    
def getJobPrediction(jobKey){
    url = "https://api.hume.ai/v0/batch/jobs/%s/predictions" % jobKey
    headers = {
        
        "accept": "application/json; charset=utf-8",
        "X-Hume-Api-Key": "9BAoszAhvQSgWLIttRJHlBJRHavk4NWOzfZQUTrDSATB5RFu"
    }

    response = requests.get(url, headers=headers)
}
```

### hume-api/wholething.py

```python
from hume import HumeBatchClient
from hume.models.config import FaceConfig
from hume.models.config import ProsodyConfig

client = HumeBatchClient("9BAoszAhvQSgWLIttRJHlBJRHavk4NWOzfZQUTrDSATB5RFu")
file = ["examplevideo.mp4"]
configs = [FaceConfig(fps_pred = 1, prob_threshold = 0.9, identify_faces=True,), ProsodyConfig()]
job = client.submit_job([],configs = configs, files=file)

job.await_complete()
job.download_predictions("predictions.json")
```

### app/llm_feedback.py

```python
"""
    Generates feedback using the prompt engineering as specified by the metrics 
    we generate.
"""


# ----------------- Environment Setup ----------------- #
import json
import os
import openai
openai.api_key = os.getenv("OPEN_AI_API_KEY")


# ----------------- Processing Functionality ----------------- #
"""
    Wraps the entire functionality for generating feedback from a key set of 
    metrics.
"""
def gen_llm_feedback(metrics: dict, prompt: str):
    # load prompt
    with open("prompt-template.txt", "r") as f:
        guidelines_prompt = f.read()
    
    # generate prompt
    prompt = guidelines_prompt.format(
        metrics_to_string(metrics),
        prompt
    )
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "user", "content": prompt}
        ]
    )


"""
    Generates a string representation of the metrics.
"""
def metrics_to_string(metrics: dict) -> str:
    final_output = ""

    for metric, value in metrics.items():
        final_output += f"a final score of {value} for {metric}, "
    
    return final_output

```

### gen-feedback/llm_feedback.py

```python
"""
    Generates feedback using the prompt engineering as specified by the metrics 
    we generate.
"""


# ----------------- Environment Setup ----------------- #
import json
import os
import openai
openai.api_key = os.getenv("OPEN_AI_API_KEY")


# ----------------- Processing Functionality ----------------- #
"""
    Wraps the entire functionality for generating feedback from a key set of 
    metrics.
"""
def gen_llm_feedback(metrics: dict, prompt: str):
    # load prompt
    with open("prompt-template.txt", "r") as f:
        guidelines_prompt = f.read()
    
    # generate prompt
    prompt = guidelines_prompt.format(
        metrics_to_string(metrics),
        prompt
    )
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "user", "content": prompt}
        ]
    )


"""
    Generates a string representation of the metrics.
"""
def metrics_to_string(metrics: dict) -> str:
    final_output = ""

    for metric, value in metrics.items():
        final_output += f"a final score of {value} for {metric}, "
    
    return final_output

```

[16 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]