Project Info
Inspiration
The theremin, an electronic musical instrument invented in the 1920s, is played without physical contact—pitch and volume are controlled by moving hands near two antennas. Its eerie sound has been featured in various music genres and movie soundtracks.
What it does
Mood Theremin translates your facial expressions into music - no musical theory needed! Smile, frown, or be surprised – your emotions become beautiful chords and melodies in real time. Explore your feelings and get inspired through music!
How we built it
Our project integrates facial expression analysis, emotion sentiment analysis, machine learning, and audio synthesis to generate music based on users' facial expressions. Through video input, we captured user's facial expressions, analyzed emotional sentiments using the HUME streaming API, and fed this data into a Scikit-Learn K-Nearest Neighbors (KNN) regressor model. We trained the model real-time and it translates facial expressions into frequencies for a 7th chord and a corresponding volume level based on the intensity levels of users. Then, the chords are played using the Web Audio API, creating a unique melody that mirrors user's emotions.
Challenges we ran into
Integrating different components seamlessly, ensuring smooth data flow. Fine-tuning the machine learning model to accurately capture emotional nuances.
Accomplishments we're proud of
Getting this to work from a concept in 24 hours! Integrating multiple elements, emotion analysis, machine learning models, and music generation. Generating an immersive 3D user interface for a rich interactive experience.
What we learned
Enhanced our respective skillsets in integrating diverse technologies into a cohesive project. Deepened understanding of facial expression analysis and emotion sentiment interpretation.
What's next
Multi-user Support: Adapting the model for multiple users, considering complex emotional interactions. Emotion Sentiment Refinement: Fine-tuning emotion sentiment analysis for greater accuracy. Diversifying Audio Output: Add different classes of instruments (piano, string ensemble, etc.) User Experience Improvements: Enhancing the interface and feedback for a seamless user experience. Customization options: Offer the user options to customize certain chords corresponding to certain emotions.
Mood Theremin
Description:
The theremin is an electronic musical instrument invented in the 1920s. It's played without physical contact—pitch and volume are controlled by moving hands near two antennas. The proximity of the right hand to the pitch antenna determines the pitch, while the left hand's distance from the volume antenna controls the volume. Known for its eerie sound, the theremin has been used in various music genres and movie soundtracks.
Inspired by the theremin, we created Mood Theremin in Cal Hack 10.0. This is a unique project that combines facial expression analysis, emotion sentiment analysis, machine learning, and audio synthesis to generate music based on users' facial expressions. By utilizing the HUME streaming API for emotion sentiment analysis and a Scikit-Learn K-Nearest Neighbors (KNN) regressor model, we translate users' facial expressions into chords and corresponding volume levels. The generated chords are then played using the Web Audio API, creating a musical experience that mirrors users' emotions.
How It Works:
- Facial Expression Analysis: Users' facial expressions are captured through video input.
- Emotion Sentiment Analysis: The HUME streaming API is used to analyze the emotion sentiments expressed in the users' facial expressions.
- Machine Learning: The emotion sentiment data is fed into a Scikit-Learn KNN regressor model to predict chords and volume levels based on emotions detected.
- Audio Synthesis: The predicted chords are played using the Web Audio API, creating music in real-time.
Requirements:
- Python (3.6+)
- Scikit-Learn Package
- HUME streaming API access
- Web Audio API compatible web browser
Usage:
- Obtain HUME streaming API credentials and configure the API access.
- Run the facial expression analysis component to capture users' facial expressions via video input.
- Process the facial expression data using the HUME streaming API to obtain emotion sentiment analysis results.
- Feed the emotion sentiment data into the trained Scikit-Learn KNN regressor model to predict chords and volume levels.
- Utilize the Web Audio API to play the generated chords with corresponding volume levels, creating emotion-driven music.
Contribution:
- Shujing Hu: Developed the machine learning model and fine-tuned the mapping algorithm.
- Tri Pham: Designed the front-end interactive 3-D experience using Three.js
Acknowledgements:
- HUME streaming API for emotion sentiment analysis.
- Scikit-Learn and Web Audio API communities for their valuable tools and resources.
Feel free to customize, expand, and explore the Emotion-Driven Music Generator project to create interactive and emotionally engaging musical experiences!
TODO
- Add your contributions!!
- Flask and JSON (add more description?)
Analysis
View
Metric
- 24
- 15
- 6
- 2
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- HTMLIn code
- JavaScriptIn code
- PythonIn code
- FlaskClaimed
- PyTorchClaimed
4 of 6 appear in the indexed code. 2 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
36 KB
Source files
8
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
caleb-fringer/Mood-Theremin
12 files · 55 KB · @ f086e5b
Structure
Interface
1 file · 8%Screens, components and styles rendered to the user.
Application logic
4 files · 33%Domain rules, services and shared utilities.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- Python77%
- JavaScript10%
- Markdown8%
- CSS4%
- HTML2%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
package.json
npm · 3- lil-gui
- three
- +1 more
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
This project’s features have not been analysed yet.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.