# Project export: Mood Theremin

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: Transform emotions into music - your unique melody!
- Devpost: https://devpost.com/software/mood-theremin
- GitHub: https://github.com/Sepulchre49/CalHacks23
- Video: https://www.youtube.com/embed/IgHx2koYhAo?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 4 GitHub contributor(s) — Tri Pham (24 commits), Caleb (15 commits), Ryland Hu (6 commits), Yuri1220 (2 commits)

## Devpost submission (written by the team)

### Inspiration

The theremin, an electronic musical instrument invented in the 1920s, is played without physical contact—pitch and volume are controlled by moving hands near two antennas. Its eerie sound has been featured in various music genres and movie soundtracks.

### What it does

Mood Theremin translates your facial expressions into music - no musical theory needed! Smile, frown, or be surprised – your emotions become beautiful chords and melodies in real time. Explore your feelings and get inspired through music!

### How we built it

Our project integrates facial expression analysis, emotion sentiment analysis, machine learning, and audio synthesis to generate music based on users' facial expressions. Through video input, we captured user's facial expressions, analyzed emotional sentiments using the HUME streaming API, and fed this data into a Scikit-Learn K-Nearest Neighbors (KNN) regressor model. We trained the model real-time and it translates facial expressions into frequencies for a 7th chord and a corresponding volume level based on the intensity levels of users. Then, the chords are played using the Web Audio API, creating a unique melody that mirrors user's emotions.

### Challenges we ran into

Integrating different components seamlessly, ensuring smooth data flow. Fine-tuning the machine learning model to accurately capture emotional nuances.

### Accomplishments we're proud of

Getting this to work from a concept in 24 hours! Integrating multiple elements, emotion analysis, machine learning models, and music generation. Generating an immersive 3D user interface for a rich interactive experience.

### What we learned

Enhanced our respective skillsets in integrating diverse technologies into a cohesive project. Deepened understanding of facial expression analysis and emotion sentiment interpretation.

### What's next

Multi-user Support: Adapting the model for multiple users, considering complex emotional interactions. Emotion Sentiment Refinement: Fine-tuning emotion sentiment analysis for greater accuracy. Diversifying Audio Output: Add different classes of instruments (piano, string ensemble, etc.) User Experience Improvements: Enhancing the interface and feedback for a seamless user experience. Customization options: Offer the user options to customize certain chords corresponding to certain emotions.

## README (from the GitHub repository)

# Mood Theremin

**Description:**

The theremin is an electronic musical instrument invented in the 1920s. It's played without physical contact—pitch and volume are controlled by moving hands near two antennas. The proximity of the right hand to the pitch antenna determines the pitch, while the left hand's distance from the volume antenna controls the volume. Known for its eerie sound, the theremin has been used in various music genres and movie soundtracks.

Inspired by the theremin, we created Mood Theremin in [Cal Hack 10.0](https://www.calhacks.io/). This is a unique project that combines facial expression analysis, emotion sentiment analysis, machine learning, and audio synthesis to generate music based on users' facial expressions. By utilizing the HUME streaming API for emotion sentiment analysis and a Scikit-Learn K-Nearest Neighbors (KNN) regressor model, we translate users' facial expressions into chords and corresponding volume levels. The generated chords are then played using the Web Audio API, creating a musical experience that mirrors users' emotions.

**How It Works:**

1. **Facial Expression Analysis:** Users' facial expressions are captured through video input.
2. **Emotion Sentiment Analysis:** The HUME streaming API is used to analyze the emotion sentiments expressed in the users' facial expressions.
3. **Machine Learning:** The emotion sentiment data is fed into a Scikit-Learn KNN regressor model to predict chords and volume levels based on emotions detected.
4. **Audio Synthesis:** The predicted chords are played using the Web Audio API, creating music in real-time.

**Requirements:**

- Python (3.6+)
- Scikit-Learn Package
- HUME streaming API access
- Web Audio API compatible web browser


**Usage:**

1. Obtain HUME streaming API credentials and configure the API access.
2. Run the facial expression analysis component to capture users' facial expressions via video input.
3. Process the facial expression data using the HUME streaming API to obtain emotion sentiment analysis results.
4. Feed the emotion sentiment data into the trained Scikit-Learn KNN regressor model to predict chords and volume levels.
5. Utilize the Web Audio API to play the generated chords with corresponding volume levels, creating emotion-driven music.

**Contribution:**

- Shujing Hu: Developed the machine learning model and fine-tuned the mapping algorithm.
- Tri Pham: Designed the front-end interactive 3-D experience using Three.js



**Acknowledgements:**

- [HUME streaming API](https://dev.hume.ai/docs/streaming-api) for emotion sentiment analysis.
- Scikit-Learn and Web Audio API communities for their valuable tools and resources.

Feel free to customize, expand, and explore the Emotion-Driven Music Generator project to create interactive and emotionally engaging musical experiences!


# TODO
- Add your contributions!!
- Flask and JSON (add more description?)


## Detected evidence (automated analysis)

Indexed codebase: 8 recognized source files, 36 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- Flask (technology) — claimed on Devpost, not found in the code
- PyTorch (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (11 of 11)

```
.gitignore
Conversion_model_neural_network.py
Conversion_model.py
expression
package.json
README.md
server.py
static/blobUtils.js
static/script.js
static/style.css
templates/index.html
```

### Dependencies

- package.json: lil-gui@^0.17.0, three@^0.148.0, vite@^4.0.4

### Recent commits (newest first)

- Theremin
- Merge branch 'main' of https://github.com/Sepulchre49/CalHacks23 into main
- Remove TODO comments
- Last min changes
- Edit linear gradient and add hover for header
- Edit linear gradient
- Remove background and edit linear gradient
- Remove mute button
- Change linear gradient
- Merge branch 'main' of github.com:Sepulchre49/CalHacks23
- Linear gradient
- Directory cleanup
- Fixed imports
- Remove scroll bar, center mute button
- Add TODO comment
- Change style of h1
- Change h1 tag to Mood Theramin
- Add mute button with styling
- Add linear gradient background
- Link to external css file

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "threejs-journey-exercise",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "vite build"
  },
  "devDependencies": {
    "vite": "^4.0.4"
  },
  "dependencies": {
    "lil-gui": "^0.17.0",
    "three": "^0.148.0"
  }
}

```

### server.py

```python
from flask import Flask, request, render_template 
from Conversion_model import calculate_frequencies

app = Flask(__name__)

@app.route("/")
def index():
    return render_template("index.html")

@app.route("/freq", methods=["POST"])
def freq():
    emotion_data = request.get_json()
    if len(emotion_data) != 0:
        outputs = calculate_frequencies(emotion_data).tolist()
    else:
        outputs = "No face detected"
    return outputs


if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0',port=3000, ssl_context="adhoc")
```

### Conversion_model.py

```python
'''
The final KNN regressor working model to predict frequencies and volumes based on the emotion vectors.
'''
from sklearn.neighbors import KNeighborsRegressor
from sklearn.decomposition import PCA
import numpy as np
import random

random.seed(46)


def calculate_frequencies(X_final):
    '''
    # Setup feature matrix X and response Y
    '''

    # temporary mapping
    emotion_chord_mapping = {
        'Admiration': [60, 64, 67, 71],  # Cmaj7 (C Major 7th chord)
        'Adoration': [57, 60, 64, 67],  # Amin7 (A minor 7th chord)
        'Aesthetic Appreciation': [62, 66, 69, 73],  # Dmaj7 (D Major 7th chord)
        'Amusement': [59, 62, 65, 69],  # Bmin7 (B minor 7th chord)
        'Anger': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
        'Annoyance': [63, 67, 70, 74],  # Emaj7 (E Major 7th chord)
        'Anxiety': [58, 62, 65, 69],  # Bmin7 (B minor 7th chord)
        'Awe': [64, 67, 71, 74],  # Emin7 (E minor 7th chord)
        'Awkwardness': [59, 62, 66, 69],  # Bmin7 (B minor 7th chord)
        'Boredom': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
        'Calmness': [60, 64, 67, 71],  # Cmaj7 (C Major 7th chord)
        'Concentration': [62, 66, 69, 73],  # Dmaj7 (D Major 7th chord)
        'Confusion': [59, 62, 66, 70],  # Bmin7 (B minor 7th chord)
        'Contemplation': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
        'Contempt': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Contentment': [60, 64, 67, 71],  # Cmaj7 (C Major 7th chord)
        'Craving': [57, 60, 64, 67],  # Amin7 (A minor 7th chord)
        'Determination': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Disappointment': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
        'Disapproval': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Disgust': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
        'Distress': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
        'Doubt': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Ecstasy': [64, 67, 71, 74],  # Emin7 (E minor 7th chord)
        'Embarrassment': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
        'Empathic Pain': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Enthusiasm': [64, 67, 71, 74],  # Emin7 (E minor 7th chord)
        'Entrancement': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Envy': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
        'Excitement': [64, 67, 71, 74],  # Emin7 (E minor 7th chord)
        'Fear': [61, 64, 67, 71],  # C#min7 (C# minor 7th chord)
        'Gratitude': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
        'Guilt': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
        'Horror': [61, 64, 67, 71],  # C#min7 (C# minor 7th chord)
        'Interest': [64, 67, 71, 74],  # Emin7 (E minor 7th chord)
        'Joy': [62, 66, 69, 73],  # Dmaj7 (D Major 7th chord)
        'Love': [57, 60, 64, 67],  # Amin7 (A minor 7th chord)
        'Nostalgia': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
        'Pain': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
        'Pride': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Realization': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
        'Relief': [61, 64, 67, 71],  # C#min7 (C# minor 7th chord)
        'Romance': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Sadness': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
        'Sarcasm': [62, 65, 68, 71],  # Dmin7 (D minor 7th chord)
        'Satisfaction': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Desire': [57, 60, 64, 67],  # Amin7 (A minor 7th chord)
        'Shame': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
        'Surprise (negative)': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Surprise (positive)': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Sympathy': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
        'Tiredness': [59, 62, 65, 69],  # Bmin7 (B minor 7th chord)
        'Triumph': [61, 64, 67, 71]  # C#min7 (C# minor 7th chord)
    }

    emotions = list(emotion_chord_mapping.keys())

    # convertion from note (MIDI) to frequency
    def note_to_frequency(note_number):
        A4_frequency = 440  # Frequency of A4 note in Hz
        return A4_frequency * (2 ** ((note_number - 69) / 12.0))

    # Normalize a sample (x0 -> (X_i, y_i))
    def normalize(x0):
        # take a high dimensional vector and normalize the highest score to be the absolute volume
        maxscore = max(x0, key=lambda x: x['score'])
        s = 0
        X_scores = []
        for tmp in x0:
            for key, item in tmp.items():
                if key == "score":
                    s += item
                    # record the scores only as the training features
                    X_scores.append(item)
        y = emotion_chord_mapping[maxscore["name"]][:]
        y.append(maxscore["score"] / s + maxscore["score"] / 100)
        return y, X_scores

    # generate more samples (x0s) using emotiosn and uniform distribution
    def generate_samples(num_samples=100, num_emotions=len(emotions)):
        samples = []
        for _ in range(num_samples):
            sample = []
            for _ in range(num_emotions):
                emotion = random.choice(emotions)
                score = random.uniform(0, 1)
                sample.append({'name': emotion, 'score': score})
            samples.append(sample)
        return samples


    X = [[{'name': 'Admiration', 'score': 0.06379243731498718}, {'name': 'Adoration', 'score': 0.07222934812307358}, {'name': 'Aesthetic Appreciation', 'score': 0.02808445133268833}, {'name': 'Amusement', 'score': 0.027589013800024986}, {'name': 'Anger', 'score': 0.0120259253308177}, {'name': 'Anxiety', 'score': 0.004923961125314236}, {'name': 'Awe', 'score': 0.025031352415680885}, {'name': 'Awkwardness', 'score': 0.061385106295347214}, {'name': 'Boredom', 'score': 0.05333968624472618}, {'name': 'Calmness', 'score': 0.135557159781456}, {'name': 'Concentration', 'score': 
[truncated — 6784 more characters]
```

### Conversion_model_neural_network.py

```python
'''
Attempt of application of Pytorch as the model to predict frequencies and volumes based on the emotion vectors.
'''
from sklearn.model_selection import train_test_split
import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import numpy as np
import random

random.seed(46)

'''
Setup feature matrix X and response Y
'''

# temporary mapping
emotion_chord_mapping = {
    'Admiration': [60, 64, 67, 71],  # Cmaj7 (C Major 7th chord)
    'Adoration': [57, 60, 64, 67],  # Amin7 (A minor 7th chord)
    'Aesthetic Appreciation': [62, 66, 69, 73],  # Dmaj7 (D Major 7th chord)
    'Amusement': [59, 62, 65, 69],  # Bmin7 (B minor 7th chord)
    'Anger': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
    'Annoyance': [63, 67, 70, 74],  # Emaj7 (E Major 7th chord)
    'Anxiety': [58, 62, 65, 69],  # Bmin7 (B minor 7th chord)
    'Awe': [64, 67, 71, 74],  # Emin7 (E minor 7th chord)
    'Awkwardness': [59, 62, 66, 69],  # Bmin7 (B minor 7th chord)
    'Boredom': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
    'Calmness': [60, 64, 67, 71],  # Cmaj7 (C Major 7th chord)
    'Concentration': [62, 66, 69, 73],  # Dmaj7 (D Major 7th chord)
    'Confusion': [59, 62, 66, 70],  # Bmin7 (B minor 7th chord)
    'Contemplation': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
    'Contempt': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Contentment': [60, 64, 67, 71],  # Cmaj7 (C Major 7th chord)
    'Craving': [57, 60, 64, 67],  # Amin7 (A minor 7th chord)
    'Determination': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Disappointment': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
    'Disapproval': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Disgust': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
    'Distress': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
    'Doubt': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Ecstasy': [64, 67, 71, 74],  # Emin7 (E minor 7th chord)
    'Embarrassment': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
    'Empathic Pain': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Enthusiasm': [64, 67, 71, 74],  # Emin7 (E minor 7th chord)
    'Entrancement': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Envy': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
    'Excitement': [64, 67, 71, 74],  # Emin7 (E minor 7th chord)
    'Fear': [61, 64, 67, 71],  # C#min7 (C# minor 7th chord)
    'Gratitude': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
    'Guilt': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
    'Horror': [61, 64, 67, 71],  # C#min7 (C# minor 7th chord)
    'Interest': [64, 67, 71, 74],  # Emin7 (E minor 7th chord)
    'Joy': [62, 66, 69, 73],  # Dmaj7 (D Major 7th chord)
    'Love': [57, 60, 64, 67],  # Amin7 (A minor 7th chord)
    'Nostalgia': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
    'Pain': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
    'Pride': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Realization': [61, 64, 68, 71],  # C#min7 (C# minor 7th chord)
    'Relief': [61, 64, 67, 71],  # C#min7 (C# minor 7th chord)
    'Romance': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Sadness': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
    'Sarcasm': [62, 65, 68, 71],  # Dmin7 (D minor 7th chord)
    'Satisfaction': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Desire': [57, 60, 64, 67],  # Amin7 (A minor 7th chord)
    'Shame': [60, 63, 67, 70],  # Cmin7 (C minor 7th chord)
    'Surprise (negative)': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Surprise (positive)': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Sympathy': [62, 65, 69, 72],  # Dmin7 (D minor 7th chord)
    'Tiredness': [59, 62, 65, 69],  # Bmin7 (B minor 7th chord)
    'Triumph': [61, 64, 67, 71]  # C#min7 (C# minor 7th chord)
}

emotions = list(emotion_chord_mapping.keys())

# conversion from note (MIDI) to frequency
def note_to_frequency(note_number):
    A4_frequency = 440  # Frequency of A4 note in Hz
    return A4_frequency * (2 ** ((note_number - 69) / 12.0))

# Normalize a sample (x0 -> (X_i, y_i))
def normalize(x0):
    # take a high dimensional vector and normalize the highest score to be the absolute volume
    maxscore = max(x0, key=lambda x: x['score'])
    s = 0
    X_scores = []
    for tmp in x0:
        for key, item in tmp.items():
            if key == "score":
                s += item
                # record the scores only as the training features
                X_scores.append(item)
    y = emotion_chord_mapping[maxscore["name"]][:]
    y.append(maxscore["score"] / s + maxscore["score"] / 100)
    return y, X_scores

# generate more samples (x0s) using emotiosn and uniform distribution
def generate_samples(num_samples=100, num_emotions=len(emotions)):
    samples = []
    for _ in range(num_samples):
        sample = []
        for _ in range(num_emotions):
            emotion = random.choice(emotions)
            score = random.uniform(0, 1)
            sample.append({'name': emotion, 'score': score})
        samples.append(sample)
    return samples


X = [[{'name': 'Admiration', 'score': 0.06379243731498718}, {'name': 'Adoration', 'score': 0.07222934812307358}, {'name': 'Aesthetic Appreciation', 'score': 0.02808445133268833}, {'name': 'Amusement', 'score': 0.027589013800024986}, {'name': 'Anger', 'score': 0.0120259253308177}, {'name': 'Annoyance', 'score': 0.025653120130300522}, {'name': 'Anxiety', 'score': 0.004923961125314236}, {'name': 'Awe', 'score': 0.025031352415680885}, {'name': 'Awkwardness', 'score': 0.061385106295347214}, {'name': 'Boredom', 'score': 0.05333968624472618}, {'name': 'Calmness', 'score': 0.135557159781456}, {'name': 'Concentration', 'score': 0.010018930770456791}, {'name': 'Confusion', 'score': 0.09115109592676163}, {'name': 'Contemplation', 'score': 0.020809845998883247}, {'name': 'Contempt', 'score': 0.030744805932044983}, {'name': 'Contentment', 'score': 0.060751479119062424},
[truncated — 8920 more characters]
```

### static/blobUtils.js

```javascript
function blobToBase64(blob) {
  return new Promise((resolve, reject) => {
    const reader = new FileReader();
    reader.onloadend = () => {
      if (reader.result) {
        const result = reader.result.toString() ;
        resolve(result.split(",")[1]);
      }
    };
    reader.readAsDataURL(blob);
  });
}

function canvasToImageBlob(canvas, format = "image/png") {
  return new Promise((resolve, reject) => {
    const handleBlob = (blob) => {
      if (blob) {
        resolve(blob);
      } else {
        reject("Could not parse blob");
      }
    };
    canvas.toBlob(handleBlob, format, 1);
  });
}

```

### templates/index.html

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Video Processing</title>
    <link rel="stylesheet" href="/static/style.css" />
    <script type="text/javascript" src="/static/blobUtils.js"></script>
    <script type="text/javascript" defer src="/static/script.js"></script>
  </head>
  <body>
    <header>
      <h1>Mood Theremin</h1>
    </header>

    <video id="myVidPlayer" muted autoplay></video>

    <div class="mycanvas">
      <canvas></canvas>
    </div>
  </body>
</html>

```

### static/style.css

```css
* {
  margin: 0;
  padding: 0;
  box-sizing: border-box;
}

body {
  overflow: hidden; /* This will remove the scrollbar */
  background: linear-gradient(
    to bottom,
    #000000 0%,
    #181248 45%,
    #2f042f 75%,
    #130113 100%
  );

  height: 100vh;
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
}

html,
h1 {
  font-family: "Roboto", sans-serif;
  color: #ffffff;
  text-transform: uppercase;
  font-size: 2.0em;
  font-weight: 700;
  letter-spacing: 2.5px;
  text-align: center;
  padding: 30px 0px 0px 0px;
  transition: all 0.69s ease; /* This will ensure smooth transitions */
  transform: scale(1);
  /* border-radius: 8px; */
  /* box-shadow: 0px 3px 15px rgba(0, 0, 0, 0.2); */
}

h1:hover {
  transform: scale(1.1); /* This will make the text slightly larger on hover */
  /* text-shadow: 3px 3px 5px rgba(0, 0, 0, 0.4); Adds a shadow for depth */
  /* color: #c72bc7; Changes color on hover to match your gradient */
}

h6 {
  margin: 0;
  padding: 0;
  color: #ffffff;
  margin-bottom: 1rem;
}

header h1 {
  color: #ffffff;
  margin-bottom: 2rem;
}

.mycanvas {
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
}

canvas {
  width: calc(100vh * 0.75);
  height: 75vh;
  background-color: black;
  border: 5px solid white;
}

video#myVidPlayer {
  width: 100%;
  max-width: 800px;
  margin: 2rem 0;
}

```

### static/script.js

```javascript
const canvas = document.querySelector("canvas");
const context = canvas.getContext("2d");
const video = document.querySelector("#myVidPlayer");
const socket = new WebSocket(
  "wss://api.hume.ai/v0/stream/models?apiKey=IgSfnRwS4whHOTGMhxwxh5m6CGoKMj8zu1RrzwwoaErcjZkY"
);
const actx = new AudioContext();
const gain = new GainNode(actx);
const oscs = [new OscillatorNode(actx), new OscillatorNode(actx), new OscillatorNode(actx), new OscillatorNode(actx)];
oscs.forEach(osc => {
    osc.start();
    osc.connect(gain);
});
gain.connect(actx.destination);

socket.onopen = e => console.log("Successfully connected to HUME");
socket.onerror = e => console.error(`Error connecting to HUME api: ${e}`);
socket.onmessage = async e => {
    actx.resume();
    const data = JSON.parse(e.data);
    //console.log(JSON.stringify(data));
    //const emotions = data["face"]["predictions"][0]["emotions"].sort((a,b) => b.score - a.score);
    const faces = data["face"];
    if (faces["code"] == "W0103") {
        console.log("No faces detected");
    } else {
        const emotions = faces["predictions"][0]["emotions"];
        const result = await fetch("/freq", {
            "method": "POST",
            "headers": {
                "Content-Type": "application/json"
            },
            "body": JSON.stringify(emotions)
        });
        results = await result.json();
        results = results[0]
        //console.log(results);
        gain.gain.exponentialRampToValueAtTime(results[4], 0.3);
        console.log(gain.gain.value)
        for (let i =0; i<4; i++) {
            oscs[i].frequency.exponentialRampToValueAtTime(results[i],0.3);
        }
        console.log(oscs);
    }
}

var w, h;
canvas.style.display = "none";

async function snapshot() {
  context.fillRect(0, 0, w, h);
  context.drawImage(video, 0, 0, w, h);
  const snapBlob = await canvasToImageBlob(canvas);
  const blob64 = await blobToBase64(snapBlob);
  const requestData = JSON.stringify({
    data: blob64,
    models: {
      face: {},
    },
  });

  if (socket.readyState === WebSocket.OPEN) {
    socket.send(requestData);
  }
}

window.navigator.mediaDevices
  .getUserMedia({ video: true, audio: true })
  .then((stream) => {
    video.srcObject = stream;
    video.onloadedmetadata = (e) => {
      video.play();

      w = video.videoWidth;
      h = video.videoHeight;

      canvas.width = w;
      canvas.height = h;
      snapshot();
    };
  })
  .catch((error) => {
    alert("You have to enable the microphone and the camera");
  });
setInterval(() => snapshot(), 1000);


/**
 * Mute Button
 */
const muteButton = document.querySelector("#muteButton");

// Initially set to muted
let isMuted = true;

muteButton.addEventListener("click", function() {
  if (isMuted) {
    muteButton.innerText = "Unmute";
    isMuted = false;
  } else {
    muteButton.innerText = "Mute";
    isMuted = true;
  }
});

```