Problem The demanding nature of healthcare work often leads to burnout, impacting both the individuals themselves and the quality of patient care. Unfortunately, burnout frequently goes unnoticed until it reaches a critical stage, affecting not only the mental health of the professionals but also potentially jeopardizing patient safety and outcomes. Our Solution The app not only efficiently detects emotional states but identifies early indicators of burnout among medical professionals through a custom neural network. It offers a user-friendly interface that presents weekly logs of burnout risk levels and a user friendly web dashboard of their associated emotional trends. This app builds upon Hume's video emotion detection technology to predict and prevent burnout among healthcare workers. Existing approaches use frequent surveillance to profile medical practitioners' mental states, jeopardizing privacy. Our application protects privacy by recording infrequent but regular interactions with patients, with full knowledge of those involved.

How we built it

At the core of our project was Hume's emotion detection API's ability to recognize 52 diverse, expressive nuances in facial expressions. We used the results of HumeBatchClient and HumeStreamClient's facial expression model to train a custom ML model to detect burnout on a carefully curated training dataset. We collected images labeled based on levels of burnout from open health research 1 2, friends, Hume's sample images, and synthetic data to train a regression model in scikit-learn. The model took as inputs the 52 emotions extracted from video frames as well as their manually selected labels, and optimized to learn the relationship between these emotions and eventual burnout. Our project wouldn't be possible without CockroachDB's serverless database and their intuitive SQL Shell, which allowed us to define and work with a PostgreSQL schema. We communicate with the database from the Flask backend as well as the python script used to collect video and query the Hume batch API endpoints.

Challenges we ran into

One of the biggest challenges was the amount of iteration our project went through before a finalized idea. We took great care to find a socially relevant, unique problem with an original solution. We struggled to determine the ground truth for what counts as someone showing signs of being "burnt out". We found it effective to use some synthetic data generated by dall-e 3 along with manually labeled images.

Accomplishments we're proud of

Successfully trained a classification model and deployed it on Hume's custom model platform using a self-labeled dataset for identifying potential burnout signs.

What we learned

Explored HumeStreamClient for WebSocket integration in Flask. We learned about nuances of WebSockets vs other HTTP connections. None of us have ever worked with front-end before. We successfully made a working flask, html, and Javascript website to demo our machine learning app.

What's next

We want to incorporate spoken data to improve our prediction and data visualization. Conversational Emotion Recognition (CER) is a task to predict the emotion of dialogue in the context of a conversation. We want to improve the accuracy of our model by expanding on training data, parameters, and compute. Add a centralized alert system that creates actionable suggestions based on likely burnout for workers.