Project Info
This project did not submit a demo video on Devpost.
Inspiration
Our inspiration for Serenade comes from the high-pressure environment we found ourselves in as college students. There were countless instances, especially during midterms and exams, where we felt overwhelmed, stressed out, and emotionally worn out. In these moments, we found peace in music – from calming classical tracks to the latest pop songs—after all, who hasn't listened to their favorite song at a stressful time? We began to realize the immense therapeutic potential of music and how it could be harnessed towards improving mental health. Meanwhile, we were also becoming deeply intrigued by the capabilities of AI and machine learning. We decided to combine these two interests and thus, Serenade was born. Anyone struggling with mental health can use Serenade, particularly those without access to costly therapy. Serenade can also supplement therapy for a quick at-home remedy during downtime.
What it does
Serenade uses AI-driven programs to implement its unique music therapy approach. It's designed to take input from a user's audio, effectively analyzing speech prosody to understand the user’s emotions and needs. Additionally, it integrates with Spotify to discover the user's favorite songs, delivering a personalized therapeutic experience. Serenade synergizes these elements to craft a custom-made therapeutic audio tailored to the user's needs.
How we built it
We used Convex and Clerk to sign in to Spotify and get the user’s top recommended song (because it’s probably a song that the user likes). We created an immersive emoji picker to gauge the user’s mood in a natural way. GPT-3.5 combines these signals and instructs MusicGen, an open-source music generation model by Meta, to generate specially-formulated snippets of audio for therapy. We also take a clip from the user’s top song (from YouTube) to make MusicGen generate something that reminds the user of their favorite music.
Challenges we ran into
Implementing all the novel features of our app into a coherent, user-friendly flow was quite the challenge. We had to carefully integrate systems such as GPT-3.5, MusicGen, Replicate hosting, Convex, and Spotify authentication. Each feature had its own unique requirements that needed to be met for seamless functioning.Finding the most effective prompts for our music generation - tailored to specific emotions - was another considerable task: what kind of music do "happy" people want vs "sad" people? What about specific use cases?
Accomplishments we're proud of
The use of the Convex model exposed us to new techniques for good API architecture.We were able to develop a more structured, reliable, and secure API system for Serenade. Another major milestone was incorporating social OAuth that simplified the sign-in process significantly, offering users a seamless, effortless entry into our service. We learned of the various services that Convex and Hume.ai can provide, and enjoyed integrating these products into our webapp. For Convex, we used its features to provide an authentication method for users and host their profiles. We learned about techniques for good API architecture through the Convex model, as well as how to easily do social OAuth for easy sign-in with providers like Spotify.
What's next
Beyond CalHacks, we plan on integrating Hume.ai's streaming facial recognition technology that operates seamlessly in real-time. Utilizing this advanced technology, the ability to read and analyze facial expressions will add another layer of emotional discernment to our platform. It will significantly augment the way we detect nuanced moods and emotions that may not be entirely captured through voice alone. By combining audio and facial cues, Serenade aims to create a more personalized and enriching therapeutic journey for the user.
Serenade
Serenade is a music therapy tool powered by generative AI, designed to cater to a wide range of audiences including children, adolescents, adults, and the elderly with various needs. It is particularly beneficial for individuals with mental health needs, developmental and learning disabilities, Alzheimer's disease and other aging related conditions, substance abuse problems, brain injuries, physical disabilities, and acute and chronic pain, including mothers in labor.
Problem
Serenade addresses a multitude of issues:
- Emotional and psychological issues: It helps individuals manage stress, anxiety, depression, and other emotional challenges by providing an outlet for emotional expression and promoting relaxation and emotional regulation.
- Cognitive difficulties: For people with cognitive impairments or neurological conditions, Serenade can help improve cognitive functions such as memory, attention, and problem-solving skills.
- Physical rehabilitation: It is used in physical rehabilitation settings to enhance motor skills, coordination, and movement for individuals with physical disabilities or injuries.
- Social and communication skills: Serenade can be beneficial for improving social interactions and communication skills, particularly for individuals with autism spectrum disorder, developmental disorders, or social anxiety.
- Pain management: It can be used to reduce the perception of pain and discomfort, making it a valuable tool in medical settings.
- Overall well-being: Serenade can enhance the overall quality of life by promoting relaxation, enhancing mood, and fostering a sense of connection and belonging.
Solution
Serenade employs music therapy, a therapeutic approach that uses music to address physical, emotional, cognitive, and social needs of individuals. It involves the use of music as a medium to help individuals improve their overall well-being and quality of life. Powered by generative AI, Serenade creates a personalized musical experience that caters to the unique needs of each individual, providing a powerful and effective therapeutic tool.
Analysis
View
Metric
- 20
- 20
- 19
- 1
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- CSSIn code
- JavaScriptIn code
- Next.jsIn code
- OpenAIIn code
- ReactIn code
- Tailwind CSSIn code
- TypeScriptIn code
7 of 7 appear in the indexed code.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
39 KB
Source files
26
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
kdrag0n/serenade-calhacks23
38 files · 418 KB · @ bd94da2
Structure
Interface
5 files · 13%Screens, components and styles rendered to the user.
API & routing
3 files · 8%Request entry points: routes, handlers and controllers.
Application logic
13 files · 34%Domain rules, services and shared utilities.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- TypeScript74%
- JavaScript11%
- Markdown10%
- CSS6%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
web/package.json
npm · 31- @aws-sdk/client-s3
- @aws-sdk/s3-request-presigner
- @clerk/clerk-react
- @clerk/nextjs
- @types/styled-components
- axios
- class-variance-authority
- clsx
- convex
- dotenv
- lucide-react
- next
- openai
- react
- react-audio-voice-recorder
- react-dom
- replicate
- styled-components
- +13 more
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
This project’s features have not been analysed yet.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.