# Project export: Moods and Metrics

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2025
- Tagline: Hyper-analyze your mood with a simple click of a button. See your emotions clearly through audio, video, and transcription sentiment analysis to take control of your mental well-being!
- Devpost: https://devpost.com/software/moodmap-ai
- GitHub: https://github.com/corey-shen/stress-level-audio-analysis
- Video: https://www.youtube.com/embed/3kKoX5IYdJQ?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Otsuka Valuenex: ekkomi® Advanced Technologies for User Feedback Award ($500 Amazon + $100 ekkomi® matcha gift card per team member))
- Team: 4 GitHub contributor(s) — Corey Shen (15 commits), Ryota (7 commits), Dhruv Shah (3 commits), Parth Joshi (2 commits)

## Devpost submission (written by the team)

### Inspiration

In a world where stress and anxiety are increasingly prevalent, we were inspired to create a tool that not only helps individuals understand their emotional states but also empowers them to take control of their mental well-being. Mood and Metrics was born out of a desire to bridge the gap between technology and emotional health, offering a way for people to easily visualize and comprehend their emotional levels through AI-driven insights.

### What it does

Mood and Metrics offers three analysis modalities: Audio Analysis: Utilizes a locally deployed audio sentiment analysis model to capture arousal, valence, and dominance values to map them onto a stress score scale Video Analysis: Sends a video clip to Gemini, and evaluates calm/stress levels and in-depth text reasoning analysis through facial expression data Transcription Analysis: Performs sentiment analysis on video transcription data, evaluating calm/stress levels and in-depth text reasoning analysis For all modalities, data is visualized in 2D or 3D

### How we built it

Utilized a state-of-the-art fine-tuned wav2vec2 transformer architecture (https://arxiv.org/abs/2203.07378) for audio sentiment analysis via Hugging Face and Pytorch Leveraged three.js to create an interactable 3D graph to map valence, dominance, and arousal Mapped emotion using a theoretical approach based on a tri-dimensional model of core affect and emotion concepts (https://www.redalyc.org/pdf/3111/311126297005.pdf) Integrated Google’s Gemini AI API for video-based and transcription-based mood analysis. Developed a React frontend with an interactive UI

### Challenges we ran into

Locally installing transformer model via Hugging Face and Pytorch, and allowing GPU cuda acceleration Extrapolating accurate stress score from arousal, valence, and dominance values Getting the axis and rotation from the camera perspective to match for 3D visualizations Constructing a pipeline to send video footage to Gemini via an API call Creating a clean frontend to visualize graphs using data from the backend server

### Accomplishments we're proud of

Successfully integrating AI-driven audio, video, and transcription sentiment analysis Creating an engaging and informational visualization of emotional/mood states Achieving reliable stress detection for educational, healthcare, and meditational use

### What we learned

The nuances of valence, dominance, and arousal in audio sentiment analysis Balancing technicality and simplicity in data visualization

### What's next

for Moods and Weights Enhancing AI models with more modalities such as heartbeat, EKG signals, etc. Collecting accurate breathing audio to stress score data and training the model from scratch Adding real-time video analysis without needing pre-recorded clips Expanding features with personalized stress relief suggestions Deploying a mobile version for on-the-go, fully local emotional tracking

## README (from the GitHub repository)

## Inspiration
In a world where stress and anxiety are increasingly prevalent, we were inspired to create a tool that not only helps individuals understand their emotional states, but also empowers them to take control of their mental well-being. Mood and Metrics was born out of a desire to bridge the gap between technology and emotional health, offering a way for people to easily visualize and comprehend their emotional levels through AI-driven insights.

## What it does 
Mood and Metrics offers three analysis modalities:
- Audio Analysis: Utilizes a locally deployed audio sentiment analysis model to capture arousal, valence, and dominance values to map them onto a stress score scale
- Video Analysis: Sends a video clip to Gemini, and evaluates calm/stress levels and in-depth text reasoning analysis through facial expression data
- Transcription Analysis: Performs sentiment analysis on video transcription data, evaluating calm/stress levels and in-depth text reasoning analysis
- For all modalities, data is visualized in 2D or 3D

## How we built it
- Utilized a state-of-the-art fine-tuned wav2vec2 transformer architecture (https://arxiv.org/abs/2203.07378) for audio sentiment analysis via Hugging Face and Pytorch
- Leveraged three.js to create an interactable 3D graph to map valence, dominance, and arousal
- Mapped emotion using a theoretical approach based on a tri-dimensional
model of core affect and emotion concepts (https://www.redalyc.org/pdf/3111/311126297005.pdf)
- Integrated Google’s Gemini AI API for video-based and transcription-based mood analysis.
- Developed a React frontend with an interactive UI

## Challenges we ran into
- Locally installing transformer model via Hugging Face and Pytorch, and allowing GPU cuda acceleration
- Extrapolating accurate stress score from arousal, valence, and dominance values
- Getting the axis and rotation from the camera perspective to match for 3D visualizations
- Constructing a pipeline to send video footage to Gemini via an API call
- Creating a clean frontend to visualize graphs using data from the backend server

## Accomplishments that we're proud of
- Successfully integrating AI-driven audio, video, and transcription sentiment analysis
- Creating an engaging and informational visualization of emotional/mood states
- Achieving reliable stress detection for educational, healthcare, and meditational use

## What we learned 
- The nuances of valence, dominance, and arousal in audio sentiment analysis
- Balancing technicality and simplicity in data visualization

## What's next for Moods and Weights
- Enhancing AI models with more modalities such as heartbeat, EKG signals, etc.
- Collecting accurate breathing audio to stress score data and training the model from scratch
- Adding real-time video analysis without needing pre-recorded clips
- Expanding features with personalized stress relief suggestions
- Deploying a mobile version for on-the-go, fully local emotional tracking

## Detected evidence (automated analysis)

Indexed codebase: 18 recognized source files, 49 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- FastAPI (technology) — claimed on Devpost, not found in the code
- Google Gemini (technology) — claimed on Devpost, not found in the code
- Hugging Face (technology) — claimed on Devpost, not found in the code
- PyTorch (technology) — claimed on Devpost, not found in the code
- Tailwind CSS (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (23 of 23)

```
.DS_Store
client/.gitignore
client/package.json
client/public/index.html
client/public/manifest.json
client/public/robots.txt
client/README.md
client/src/App.css
client/src/App.js
client/src/App.test.js
client/src/components/AudioProcessing.js
client/src/components/AudioRecorder.js
client/src/components/EmotionGraph2D.js
client/src/components/LandingPage.css
client/src/components/LandingPage.js
client/src/components/Scene3D.js
client/src/index.css
client/src/index.js
client/src/reportWebVitals.js
client/src/setupTests.js
README.md
server/audio_measure.py
server/main.py
```

### Dependencies

- client/package.json: @react-three/drei@^9.80.1, @react-three/fiber@^8.13.5, @testing-library/dom@^10.4.0, @testing-library/jest-dom@^6.6.3, @testing-library/react@^16.2.0, @testing-library/user-event@^13.5.0, axios@^1.7.9, d3@^7.9.0, react@^18.2.0, react-dom@^18.2.0, react-router-dom@^7.1.5, react-scripts@5.0.1, three@^0.173.0, web-vitals@^2.1.4

### Recent commits (newest first)

- Merge pull request #2 from corey-shen/landing-page
- Audio processing to home page linking
- updated the tag line
- there is no whitespace around the landing page, the AudioProcessing component is not affected
- Fixed file structure
- created the landing page, it is affecting other components, needs fixing
- made 3D graph container for better visualization
- added rotation fixed bugs for 3D visualization
- modified package
- Fixed Scene3D
- Graph description + successful merge
- Added 3D visualization
- Change UI/recording button
- Merge pull request #1 from corey-shen/2D-graph
- Centered everything, including expandable graphs
- Expandable graph feature
- 2D graphs lined up vertically, discrete values connected
- GPU accelerated inference for audio_measure.py
- Working MVP
- added audio_measure.py

## Key source files (fetched from GitHub, selected and truncated for size)

### client/package.json

```
{
  "name": "client",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@react-three/drei": "^9.80.1",
    "@react-three/fiber": "^8.13.5",
    "@testing-library/dom": "^10.4.0",
    "@testing-library/jest-dom": "^6.6.3",
    "@testing-library/react": "^16.2.0",
    "@testing-library/user-event": "^13.5.0",
    "axios": "^1.7.9",
    "d3": "^7.9.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-router-dom": "^7.1.5",
    "react-scripts": "5.0.1",
    "three": "^0.173.0",
    "web-vitals": "^2.1.4"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

```

### server/main.py

```python
from fastapi import FastAPI, UploadFile, File, HTTPException
from fastapi.responses import JSONResponse
from fastapi.middleware.cors import CORSMiddleware
from pydub import AudioSegment
import io
import os
import subprocess

app = FastAPI()

# CORS Configuration
app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

MAX_FILE_SIZE = 25 * 1024 * 1024  # 25MB
SUPPORTED_TYPES = {'audio/wav', 'audio/x-wav', 'audio/mpeg', 'audio/ogg'}

def process_audio(file_path: str):
    try:
        result = subprocess.run(
            ["python", "./audio_measure.py", "--audio_path", file_path],
            capture_output=True,
            text=True,
            check=True
        )
        return result.stdout
    except subprocess.CalledProcessError as e:
        error_msg = f"Processing failed: {e.stderr.strip()}"
        print(error_msg)
        raise HTTPException(status_code=500, detail=error_msg)

@app.post("/process_audio")
async def process_audio_endpoint(file: UploadFile = File(...)):
    try:
        # Validate file type
        if file.content_type not in SUPPORTED_TYPES:
            raise HTTPException(400, "Unsupported file type")

        # Read and validate file size
        file_bytes = await file.read()
        if len(file_bytes) > MAX_FILE_SIZE:
            raise HTTPException(413, "File too large. Max 25MB")

        # Convert to WAV format
        audio = AudioSegment.from_file(io.BytesIO(file_bytes))
        wav_buffer = io.BytesIO()
        audio.export(wav_buffer, format="wav")
        
        # Save processed file
        save_dir = "processed_audio"
        os.makedirs(save_dir, exist_ok=True)
        file_path = os.path.join(save_dir, f"processed_{file.filename}.wav")
        
        with open(file_path, "wb") as f:
            f.write(wav_buffer.getvalue())

        # Process audio
        results = process_audio(file_path)
        
        return JSONResponse(content={"results": results, "filename": file.filename})

    except Exception as e:
        print(f"Processing error: {str(e)}")
        raise HTTPException(500, "Audio processing failed") from e

if __name__ == '__main__':
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)


```

### client/src/App.js

```javascript
import React from "react";
import './App.css';
import { BrowserRouter, Routes, Route } from "react-router-dom";
import LandingPage from "./components/LandingPage.js";
import AudioProcessing from "./components/AudioProcessing.js";

function App() {
  return (
    <div className="App">
        <BrowserRouter>
            <Routes>
                <Route path="/" element={<LandingPage />} />
                <Route path="/audio" element={<AudioProcessing />} />
            </Routes>
        </BrowserRouter>
    </div>
  );
}

export default App;
```

### client/src/index.js

```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';

import { BrowserRouter, Routes, Route } from "react-router-dom";


const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);

// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();

```

### server/audio_measure.py

```python
import numpy as np
import torch
import torch.nn as nn
from transformers import Wav2Vec2Processor
from transformers.models.wav2vec2.modeling_wav2vec2 import (
    Wav2Vec2Model,
    Wav2Vec2PreTrainedModel,
)
import soundfile as sf
import argparse
import librosa
from tqdm import tqdm


class RegressionHead(nn.Module):
    r"""Classification head."""

    def __init__(self, config):

        super().__init__()

        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.dropout = nn.Dropout(config.final_dropout)
        self.out_proj = nn.Linear(config.hidden_size, config.num_labels)

    def forward(self, features, **kwargs):

        x = features
        x = self.dropout(x)
        x = self.dense(x)
        x = torch.tanh(x)
        x = self.dropout(x)
        x = self.out_proj(x)

        return x


class EmotionModel(Wav2Vec2PreTrainedModel):
    r"""Speech emotion classifier."""

    def __init__(self, config):

        super().__init__(config)

        self.config = config
        self.wav2vec2 = Wav2Vec2Model(config)
        self.classifier = RegressionHead(config)
        self.init_weights()

    def forward(
            self,
            input_values,
    ):

        outputs = self.wav2vec2(input_values)
        hidden_states = outputs[0]
        hidden_states = torch.mean(hidden_states, dim=1)
        logits = self.classifier(hidden_states)

        return hidden_states, logits

# load model from hub
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
if torch.cuda.is_available():
    print(f"Using GPU: {torch.cuda.get_device_name()}")
else:
    print("Using CPU")
model_name = 'audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim'
processor = Wav2Vec2Processor.from_pretrained(model_name)
model = EmotionModel.from_pretrained(model_name).to(device)

def process_func(
    x: np.ndarray,
    sampling_rate: int,
    embeddings: bool = False,
) -> np.ndarray:
    r"""Predict emotions or extract embeddings from raw audio signal."""

    # run through processor to normalize signal
    # always returns a batch, so we just get the first entry
    # then we put it on the device
    y = processor(x, sampling_rate=sampling_rate)
    y = y['input_values'][0]
    y = y.reshape(1, -1)
    y = torch.from_numpy(y).to(device)

    # run through model
    with torch.no_grad():
        y = model(y)[0 if embeddings else 1]

    # convert to numpy
    y = y.detach().cpu().numpy()

    return y

def process_audio(audio, num_chunks, sampling_rate):
    """Process audio by splitting into equal chunks
    
    Args:
        audio (numpy.ndarray): Input audio data
        num_chunks (int): Number of equal-sized chunks to split audio into
        sampling_rate (int): Sampling rate of audio 
        
    Returns:
        list: List of audio chunks
    """
    # Calculate samples per chunk based on total length
    chunk_samples = len(audio) // num_chunks
    chunks = []
    
    print("Chunking audio...")
    for i in tqdm(range(num_chunks)):
        start = i * chunk_samples
        end = start + chunk_samples
        chunk = audio[start:end]
        chunks.append(chunk)
    
    return chunks

def process_file(audio_path, num_chunks, alpha, beta, gamma):
    """Process an audio file and return array of stress values
    
    Args:
        audio_path (str): Path to audio file
        num_chunks (int): Number of equal-sized chunks to split audio into
        alpha (float): Weight for arousal in stress calculation
        beta (float): Weight for valence in stress calculation
        gamma (float): Emphasis factor for stress calculation
        
    Returns:
        dict: Dictionary containing arrays of emotion values
    """
    print("Loading audio file...")
    # Load audio file using soundfile with tqdm
    with sf.SoundFile(audio_path) as f:
        frames = len(f)
        channels = f.channels
        with tqdm(total=frames, desc="Reading audio") as pbar:
            audio = np.zeros(frames, dtype=np.float32)  # Pre-allocate array for one channel
            pos = 0
            block_size = 10000
            while pos < frames:
                chunk = f.read(block_size)
                if not len(chunk):
                    break
                # If stereo, take the first channel
                if channels > 1:
                    chunk = chunk[:, 0]
                audio[pos:pos + len(chunk)] = chunk  # Direct assignment to array
                pos += len(chunk)
                pbar.update(len(chunk))
    
    sr = f.samplerate
    print(f"Original sample rate: {sr}")
    if sr != 16000:
        print("Resampling audio to 16kHz...")
        audio = librosa.resample(audio, orig_sr=sr, target_sr=16000)
        sr = 16000
    print(f"New sample rate: {sr}")
    
    chunks = process_audio(audio, num_chunks, sr)

    arousal_values = []
    dominance_values = []
    valence_values = []
    stress_values = []
    three_d_values = []
    
    print("Processing chunks...")
    for chunk in tqdm(chunks):
        results = process_func(chunk.reshape(1, -1), sr)[0]
        results = np.clip(results, 0, 1)   # clip
        stress = (alpha * results[0] + beta * (1 - results[2])) / (alpha + beta)
        # Apply gamma
        stress = stress ** gamma
        # Get the data values
        arousal = round(float(results[0]), 4)
        dominance = round(float(results[1]), 4)
        valence = round(float(results[2]), 4)
        stress = round(float(stress), 4)

        arousal_values.append(arousal)
        dominance_values.append(dominance)
        valence_values.append(valence)
        stress_values.append(stress)
        three_d_values.append((dominance, valence, arousal))
    
    dictionary = {}

    dictionary["arousal"] = arousal_values
    dictionary["dominance"] = dominance_values
    dictionary["valence"] = valence_values
    dictionary["stress"] = stress_values
    dictionary["three_d"] = three_d_values

    return dictionary

def main():
    parser = argparse.Ar
[truncated — 1179 more characters]
```

### client/src/setupTests.js

```javascript
// jest-dom adds custom jest matchers for asserting on DOM nodes.
// allows you to do things like:
// expect(element).toHaveTextContent(/react/i)
// learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom';

```

### client/src/App.test.js

```javascript
import { render, screen } from '@testing-library/react';
import App from './App';

test('renders learn react link', () => {
  render(<App />);
  const linkElement = screen.getByText(/learn react/i);
  expect(linkElement).toBeInTheDocument();
});

```

### client/src/reportWebVitals.js

```javascript
const reportWebVitals = onPerfEntry => {
  if (onPerfEntry && onPerfEntry instanceof Function) {
    import('web-vitals').then(({ getCLS, getFID, getFCP, getLCP, getTTFB }) => {
      getCLS(onPerfEntry);
      getFID(onPerfEntry);
      getFCP(onPerfEntry);
      getLCP(onPerfEntry);
      getTTFB(onPerfEntry);
    });
  }
};

export default reportWebVitals;

```

### client/src/index.css

```css
body {
  margin: 0;
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen',
    'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',
    sans-serif;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
}

code {
  font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New',
    monospace;
}

```

### client/public/index.html

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="theme-color" content="#000000" />
    <meta
      name="description"
      content="Web site created using create-react-app"
    />
    <link rel="apple-touch-icon" href="%PUBLIC_URL%/logo192.png" />
    <!--
      manifest.json provides metadata used when your web app is installed on a
      user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
    -->
    <link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
    <!--
      Notice the use of %PUBLIC_URL% in the tags above.
      It will be replaced with the URL of the `public` folder during the build.
      Only files inside the `public` folder can be referenced from the HTML.

      Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
      work correctly both with client-side routing and a non-root public URL.
      Learn how to configure a non-root public URL by running `npm run build`.
    -->
    <title>React App</title>
  </head>
  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
    <!--
      This HTML file is a template.
      If you open it directly in the browser, you will see an empty page.

      You can add webfonts, meta tags, or analytics to this file.
      The build step will place the bundled scripts into the <body> tag.

      To begin the development, run `npm start` or `yarn start`.
      To create a production bundle, use `npm run build` or `yarn build`.
    -->
  </body>
</html>

```

[7 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]