# Project export: Evatone

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 11.0
- Tagline: Evatone partners with online call services like Zoom to provide real-time emotion detection for facial expression identification. This increases accessibility for those with Autism and alexithymia.
- Devpost: https://devpost.com/software/evatone
- GitHub: https://github.com/jesslyng33/evatone
- Demo: https://steamforstudents.wixsite.com/evatone
- Video: https://www.youtube.com/embed/M1QpEgcqUp0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — jesslyng33 (11 commits), Anushka Bora (2 commits)

## Devpost submission (written by the team)

### Inspiration

All three members of the team have had personal experiences with people on the autism spectrum, and seen firsthand their struggles with deciphering emotions. With the recent coronavirus pandemic, the use of online conference programs such as Zoom have exacerbated the issue of reading emotional cues. This is not only a problem for the 75+ million people struggling with ASD worldwide, but also for people with alexithymia, a condition that affects individuals’ ability to understand emotion (which makes up 10% of the world population). Our team decided to solve this problem through the development of Evatone, a tool that provides assistance in emotion identification on video conferencing platforms.

### What it does

Our tool is emotion detection software specifically incorporated for video conferencing. It allows users to see what major expressions they elicit as they speak and identify major emotions expressed by the other participants in the video conference. This appears as labels on video participants’ faces, allowing people with Autism and alexithymia to quickly gauge the emotions of others in the meeting room.

### How we built it

We created a live webcam feed using Typescript (part of our front end) that takes in video input. At set intervals, we send a frame from the video as a JPG image to the Python backend using Flask REST APIs. In our backend, we used the Hume Expression Measurement streaming API, along with a web socket to maintain an open connection, to analyze the facial expression of the frames in real-time, and detect the emotion. We parsed the output of Hume’s list of emotions detected to include only the emotion with the highest score (as this represented the dominant emotion), and then sent this data back to the front end (Typescript) to display it on our live webcam feed. Using cv2’s face detection model paired with div elements part of our html, we then overlay a red box over the face detected in the webcam with the dominant emotion label.

### Challenges we ran into

One challenge we ran into was ensuring that the live webcam feed was running the entire time our application was running. Originally, we built out both the live webcam feed and the backend with Python, using Python’s cv2 library to capture the frames. However, we found that the cv2 webcam frames conflicted with how we were sending frames to Hume’s API. Thus, we transitioned to using Typescript for the webcam and Python for the backend, which allowed us to run the webcam feed the whole time while simultaneously sending frames to our model.

### Accomplishments we're proud of

We are proud of being able to connect the front end and back end of our code, seamlessly presenting our backend output and data in a more presentable way and with a much better user interface, using front-end code. We are also proud that we were able to learn how to use Hume’s API. On Saturday, we spent hours at Hume’s table debugging our code and learning how web sockets work to allow real-time continuous detection of emotion in facial expressions. We’re really proud that, while we ran into challenges with using Hume’s streaming API, we pushed through, asked questions, and got our final output!

### What we learned

We learned how to work with APIs, as we navigated the Hume API to incorporate it into our code for facial emotion detection. We learned about web sockets and how they allow for a continuous connection. We also learned how to code in TypeScript and how to use Flask as a framework for connecting HTML/CSS/JavaScript with our backend Python code using POST and GET.

### What's next

We envision Evatone to be incorporated into online meeting platforms like Zoom, Google Meet, and Microsoft Teams. Furthermore, we see Evatone increasing accessibility for people with autism across the online sphere. We have thought about making a Chrome extension that can scan for any face currently on screen, whether that belongs to a Zoom call or a YouTube video, and similarly detect the emotion of the face, helping people with autism navigate online social interactions with ease.

## README (from the GitHub repository)

# Evatone
With our society's transition to an online world, it has become even harder for people with Autism to navigate people’s emotions in online conversations. Thus, Evatone partners with online call services like Zoom, Google Meet, and Microsoft Teams to provide a real-time emotion detection tool to help with facial expression identification. This increases accessibility on their platforms for those with autism spectrum disorder (ASD), emotional dysregulation, and alexithymia.

## To run our code:
1) Ensure that the typescript file has been compiled  to javascript by running "tsc camera.ts --outDir static/js" in terminal
2) Run the app.py file by running "python app.py" in terminal. This will open up a local host with a demo of our product.


## Detected evidence (automated analysis)

Indexed codebase: 14 recognized source files, 25 KB.
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- TypeScript (language) — detected in the code
- CSS (language) — claimed on Devpost, not found in the code
- Flask (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (14 of 14)

```
app.py
camera.ts
environment.yml
README.md
static/js/camera.js
templates/index.html
test archive/fifthtest.py
test archive/fourthtest.py
test archive/main.py
test archive/secondtest.py
test archive/sixthtest.py
test archive/test.py
test archive/testing.py
test archive/thirdtest.py
```

### Dependencies

No dependency index available.

### Recent commits (newest first)

- update readme
- final - little styling changes
- style
- Merge branch 'main' of https://github.com/jesslyng33/calhacks
- mvp
- re-updatd html and deletd workspace file
- html front end yayay!!!!
- emotion detection working
- setting up front end
- a LOT - got max emotion to print out for x frames!!
- environment
- set up camera & square box
- Initial commit

## Key source files (fetched from GitHub, selected and truncated for size)

### app.py

```python
from flask import Flask, render_template, request, jsonify
import os
import cv2
import asyncio
from hume import AsyncHumeClient
from hume.expression_measurement.stream import Config
from hume.expression_measurement.stream.socket_client import StreamConnectOptions
from hume.expression_measurement.stream.types import StreamFace

app = Flask(__name__)

UPLOAD_FOLDER = 'temp'
os.makedirs(UPLOAD_FOLDER, exist_ok=True)

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/process-frame', methods=['POST'])
def process_frame():
    if 'image' not in request.files:
        return jsonify({'error': 'No image file uploaded'}), 400

    image_file = request.files['image']
    image_path = os.path.join(UPLOAD_FOLDER, image_file.filename)
    image_file.save(image_path)

    image = cv2.imread(image_path)
    face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')
    gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    faces = face_cascade.detectMultiScale(gray_image, scaleFactor=1.1, minNeighbors=5)
    
    largest_emotion, largest_score = asyncio.run(process_with_hume(image_path))

    face_data = []  # Prepare list to store face info
    for (x, y, w, h) in faces:
        # Include coordinates and the emotion in the data sent back
        face_data.append({
            'x': int(x), 
            'y': int(y), 
            'width': int(w), 
            'height': int(h),
            'emotion': largest_emotion
        })

    return jsonify({
        'status': 'success',
        'message': 'Frame processed',
        'emotion': largest_emotion,  # Max emotion in frame
        'score': largest_score,
        'faces': face_data  # Send back coordinates and emotions for each face
    })

async def process_with_hume(image_path):
    client = AsyncHumeClient(api_key="4pI4JMESdAFcX7YWA28TP4Fk7GFr6Oiw2zVmh7AizVV8lhP5")
    model_config = Config(face=StreamFace())
    stream_options = StreamConnectOptions(config=model_config)

    async with client.expression_measurement.stream.connect(options=stream_options) as socket:
        result = await socket.send_file(image_path)

        if result.face and result.face.predictions:
            largest_score = result.face.predictions[0].emotions[0].score
            largest_emotion = result.face.predictions[0].emotions[0].name

            for i in range(1, len(result.face.predictions[0].emotions)):
                if result.face.predictions[0].emotions[i].score > largest_score:
                    largest_score = result.face.predictions[0].emotions[i].score
                    largest_emotion = result.face.predictions[0].emotions[i].name

            return largest_emotion, largest_score
        else:
            return None, 0.0

if __name__ == '__main__':
    app.run(debug=True)
```

### test archive/main.py

```python
# IMPORTS
import cv2

# CAMERA

cap = cv2.VideoCapture(0) # open connection to camera

# pre-trained face detection model
face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')

# turn on camera until loop is broken
while True:
    ret, frame = cap.read() # capture frame by frame
    cv2.imshow('Camera Feed', frame) # display the frame
    
    # convert the frame to grayscale (needed for face detection)
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

    # detect faces in the frame
    faces = face_cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=5, minSize=(30, 30))

    # draw a rectangle around each detected face
    for (x, y, w, h) in faces:
        cv2.rectangle(frame, (x, y), (x+w, y+h), (255, 0, 0), 2)

    # display the frame with rectangles around faces
    cv2.imshow('Camera Feed', frame)
    
    if cv2.waitKey(1) & 0xFF == ord('q'): # stop camera if q is pressed
        break

# release camera and close all windows
cap.release()
cv2.destroyAllWindows()
```

### environment.yml

```yaml
name: calhacks

channels:
  - defaults

dependencies:
  - python=3.9
  - pip
  
  - pip:
    - requests
    - opencv-python
    - hume
```

### camera.ts

```typescript
const videoElement: HTMLVideoElement | null = document.getElementById('videoElement') as HTMLVideoElement;
const canvas: HTMLCanvasElement | null = document.createElement('canvas');
const emotionElement: HTMLElement | null = document.querySelector('h1');
const rectanglesContainer: HTMLElement | null = document.getElementById('rectangles-container');
const interval = 1000; // Capture frame every 1 second

if (navigator.mediaDevices && navigator.mediaDevices.getUserMedia) {
  navigator.mediaDevices.getUserMedia({ video: true })
    .then((stream) => {
      if (videoElement) {
        videoElement.srcObject = stream;
        videoElement.play();
      }

      setInterval(() => {
        if (videoElement && canvas) {
          captureAndSendFrame();
        }
      }, interval);
    })
    .catch((err) => {
      console.error("Error accessing webcam: ", err);
    });
} else {
  console.error("getUserMedia not supported by this browser.");
}

function captureAndSendFrame() {
    if (!videoElement || !canvas) return;
  
    canvas.width = videoElement.videoWidth;
    canvas.height = videoElement.videoHeight;
  
    const context = canvas.getContext('2d');
    if (context) {
      context.drawImage(videoElement, 0, 0, canvas.width, canvas.height);
  
      canvas.toBlob((blob) => {
        if (blob) {
          const formData = new FormData();
          formData.append('image', blob, 'frame.jpg');
  
          fetch('/process-frame', {
            method: 'POST',
            body: formData
          })
          .then((response) => response.json())
          .then((data) => {
            if (data.emotion && data.score && emotionElement) {
              emotionElement.textContent = `Detected Emotion: ${data.emotion}, Score: ${data.score.toFixed(4)}`;
            }
  
            if (data.faces && rectanglesContainer) {
              // Clear old rectangles and labels
              rectanglesContainer.innerHTML = '';
  
              // Draw new rectangles and labels
              data.faces.forEach((face: { x: number, y: number, width: number, height: number, emotion: string }) => {
                // Create the rectangle div
                const rect = document.createElement('div');
                rect.classList.add('face-rectangle');
                rect.style.left = `${face.x}px`;
                rect.style.top = `${face.y}px`;
                rect.style.width = `${face.width}px`;
                rect.style.height = `${face.height}px`;
  
                // Create a label div for the emotion
                const label = document.createElement('div');
                label.classList.add('face-label');
                label.textContent = face.emotion;
                label.style.left = `${face.x}px`;
                label.style.top = `${face.y - 20}px`;  // Position it above the rectangle
  
                // Append the rectangle and label to the container
                rectanglesContainer.appendChild(rect);
                rectanglesContainer.appendChild(label);
              });
            }
          })
          .catch((error) => {
            console.error('Error sending frame to backend:', error);
          });
        }
      }, 'image/jpeg');
    }
  }  
```

### test archive/sixthtest.py

```python
# IMPORTS
import asyncio
from hume import AsyncHumeClient
from hume.expression_measurement.stream import Config
from hume.expression_measurement.stream.socket_client import StreamConnectOptions
from hume.expression_measurement.stream.types import StreamFace
import cv2
import os
import sys

# CAMERA

cap = cv2.VideoCapture(0) # open connection to camera



# turn on camera until loop is broken
while True:
    
    ret, frame = cap.read() # capture frame by frame
    cv2.imshow('Camera Feed', frame) # display the frame
    cv2.imwrite(f'frame.jpg', frame)
    
    if cv2.waitKey(1) & 0xFF == ord('q'): # stop camera if q is pressed
        break

# release camera and close all windows
cap.release()
cv2.destroyAllWindows()
```

### test archive/testing.py

```python
import asyncio
from hume import AsyncHumeClient
from hume.expression_measurement.stream import Config
from hume.expression_measurement.stream.socket_client import StreamConnectOptions
from hume.expression_measurement.stream.types import StreamLanguage

samples = [
    "Mary had a little lamb,",
    "Its fleece was white as snow."
    "Everywhere the child went,"
    "The little lamb was sure to go."
]

async def main():
    client = AsyncHumeClient(api_key="4pI4JMESdAFcX7YWA28TP4Fk7GFr6Oiw2zVmh7AizVV8lhP5")

    model_config = Config(language=StreamLanguage())

    stream_options = StreamConnectOptions(config=model_config)

    async with client.expression_measurement.stream.connect(options=stream_options) as socket:
        for sample in samples:
            result = await socket.send_text(sample)
            print(result.language.predictions[0].emotions)

if __name__ == "__main__":
    asyncio.run(main())
```

### test archive/thirdtest.py

```python
import asyncio
from hume import AsyncHumeClient
from hume.expression_measurement.stream import Config
from hume.expression_measurement.stream.socket_client import StreamConnectOptions
from hume.expression_measurement.stream.types import StreamFace
import cv2
import os
import sys
import base64

def capture_frames(frames, video_source=0):
    cap = cv2.VideoCapture(video_source)

    frame_count = 0
    while True:
        ret, frame = cap.read()
        if not ret:
            break

        # Save the frame as a file (e.g., frame_1.jpg, frame_2.jpg, etc.)
        cv2.imwrite(f'frame_{frame_count}.jpg', frame)
        frame_count += 1

        # Exit after capturing 10 frames, for example
        if frame_count >= frames:
            break

    cap.release()
    cv2.destroyAllWindows()

async def main():
    capture_frames(1)
    
    # get root folder
    root_folder = os.path.abspath(os.path.join('..'))
    if root_folder not in sys.path:
        sys.path.append(root_folder)

    file_path = os.path.join(root_folder, 'calhacks\\frame_0.jpg')
    
    client = AsyncHumeClient(api_key="4pI4JMESdAFcX7YWA28TP4Fk7GFr6Oiw2zVmh7AizVV8lhP5")
    model_config = Config(face=StreamFace())
    stream_options = StreamConnectOptions(config=model_config)
    async with client.expression_measurement.stream.connect(options=stream_options) as socket:
        result = await socket.send_file(file_path)
        print(result)

if __name__ == "__main__":
    asyncio.run(main())


```

### templates/index.html

```html
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Camera with TypeScript</title>
  <style>
    body {
      margin: 0;
      padding: 0;
      display: flex;
      flex-direction: column;
      justify-content: center;
      align-items: center;
      height: 100vh;
      background: linear-gradient(to right, #fcbd9d, #7d6f63, #8015e8, #f3a78f);
    }

    h1 {
      text-align: center;
      margin-bottom: 20px;
      font-size: 2em;
      color: #f7dfb7;
    }

    .video-container {
      position: relative;
      width: 640px;
      height: 480px;
      border: 2px solid #000;
      border-radius: 10px;
      box-shadow: 0px 4px 8px rgba(0, 0, 0, 0.2);
    }

    video {
      width: 100%;
      height: 100%;
    }

    .face-rectangle {
      position: absolute;
      border: 2px solid red;
      pointer-events: none; /* So the user can't interact with the rectangles */
    }

    .face-label {
        position: absolute;
        font-size: 14px;
        background-color: rgba(255, 255, 255, 0.7); /* Semi-transparent background */
        padding: 2px 4px;
        border-radius: 3px;
        color: black;
        text-align: center;
        pointer-events: none; /* So the user can't interact with the label */
    }
  </style>
</head>
<body>
  <h1> </h1>
  <div class="video-container">
    <video id="videoElement" autoplay></video>
    <div id="rectangles-container"></div> <!-- This will hold face rectangles -->
  </div>
  <script src="{{ url_for('static', filename='js/camera.js') }}"></script>
</body>
</html>
```

### test archive/test.py

```python
import requests
import time
import asyncio

async def main():
    # Step 1: Start inference job (POST /v0/batch/jobs)
    response = requests.post(
        "https://api.hume.ai/v0/batch/jobs",
        headers={
            "X-Hume-Api-Key": "4pI4JMESdAFcX7YWA28TP4Fk7GFr6Oiw2zVmh7AizVV8lhP5",
            "Content-Type": "application/json"
        },
        json={
            "urls": [
                "https://hume-tutorials.s3.amazonaws.com/faces.zip"
            ],
            "notify": True,
            "models": {
                "face": {}
            },
            "transcription": {}
        },
    )

    job_id = response.json().get('job_id')
    if not job_id:
        print(f"Failed to create job: {response.text}")
        return

    print(f"Job created successfully. Job ID: {job_id}")

    # Step 2: Poll for job completion
    while True:
        response2 = requests.get(
            f"https://api.hume.ai/v0/batch/jobs/{job_id}/predictions",
            headers={
                "X-Hume-Api-Key": "4pI4JMESdAFcX7YWA28TP4Fk7GFr6Oiw2zVmh7AizVV8lhP5"
            },
        )

        # If the job is still in progress, wait and retry
        if response2.status_code == 400 and "Job is in progress." in response2.text:
            print("Job is still in progress. Retrying in 5 seconds...")
            time.sleep(5)  # Wait for 5 seconds before polling again
        elif response2.status_code == 200:
            # If the job is complete, print the predictions
            print("Job completed successfully!")
            print(response2.json())
            break
        else:
            # Handle other possible errors
            print(f"Error: {response2.status_code}, {response2.text}")
            break

if __name__ == "__main__":
    asyncio.run(main())

```

### test archive/fourthtest.py

```python
import asyncio
from hume import AsyncHumeClient
from hume.expression_measurement.stream import Config
from hume.expression_measurement.stream.socket_client import StreamConnectOptions
from hume.expression_measurement.stream.types import StreamFace
import cv2
import os
import sys
import base64

def capture_frames(frames, video_source=0):
    cap = cv2.VideoCapture(video_source)

    frame_count = 0
    while True:
        ret, frame = cap.read()
        if not ret:
            break

        # Save the frame as a file (e.g., frame_1.jpg, frame_2.jpg, etc.)
        cv2.imwrite(f'frame_{frame_count}.jpg', frame)
        frame_count += 1

        # Exit after capturing 10 frames, for example
        if frame_count >= frames:
            break

    cap.release()
    cv2.destroyAllWindows()

async def main():
    frames = 5
    
    capture_frames(frames)
    
    # get root folder
    root_folder = os.path.abspath(os.path.join('..'))
    if root_folder not in sys.path:
        sys.path.append(root_folder)
        
    for i in range(0, frames):
        file_path = os.path.join(root_folder, f'calhacks\\frame_{i}.jpg')
        
        client = AsyncHumeClient(api_key="4pI4JMESdAFcX7YWA28TP4Fk7GFr6Oiw2zVmh7AizVV8lhP5")
        model_config = Config(face=StreamFace())
        stream_options = StreamConnectOptions(config=model_config)
        async with client.expression_measurement.stream.connect(options=stream_options) as socket:
            result = await socket.send_file(file_path)
            # Extract the top emotion
            largest_score = result.face.predictions[0].emotions[0].score
            largest_emotion = result.face.predictions[0].emotions[0].name
            for i in range(1, len(result.face.predictions[0].emotions)):
                if result.face.predictions[0].emotions[i].score > largest_score:
                    largest_score = result.face.predictions[0].emotions[i].score
                    largest_emotion = result.face.predictions[0].emotions[i].name
            print(largest_emotion)

if __name__ == "__main__":
    asyncio.run(main())
```

[3 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]