# Project export: Serenade This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: Personalized, AI-powered therapeutic experiences to promote mental, emotional, physical wellbeing - Devpost: https://devpost.com/software/serenade - GitHub: https://github.com/kdrag0n/serenade-calhacks23 - Team: 4 GitHub contributor(s) — Danny Lin (20 commits), mayuridev (20 commits), imedra (19 commits), brian5121 (1 commits) ## Devpost submission (written by the team) ### Inspiration Our inspiration for Serenade comes from the high-pressure environment we found ourselves in as college students. There were countless instances, especially during midterms and exams, where we felt overwhelmed, stressed out, and emotionally worn out. In these moments, we found peace in music – from calming classical tracks to the latest pop songs—after all, who hasn't listened to their favorite song at a stressful time? We began to realize the immense therapeutic potential of music and how it could be harnessed towards improving mental health. Meanwhile, we were also becoming deeply intrigued by the capabilities of AI and machine learning. We decided to combine these two interests and thus, Serenade was born. Anyone struggling with mental health can use Serenade, particularly those without access to costly therapy. Serenade can also supplement therapy for a quick at-home remedy during downtime. ### What it does Serenade uses AI-driven programs to implement its unique music therapy approach. It's designed to take input from a user's audio, effectively analyzing speech prosody to understand the user’s emotions and needs. Additionally, it integrates with Spotify to discover the user's favorite songs, delivering a personalized therapeutic experience. Serenade synergizes these elements to craft a custom-made therapeutic audio tailored to the user's needs. ### How we built it We used Convex and Clerk to sign in to Spotify and get the user’s top recommended song (because it’s probably a song that the user likes). We created an immersive emoji picker to gauge the user’s mood in a natural way. GPT-3.5 combines these signals and instructs MusicGen, an open-source music generation model by Meta, to generate specially-formulated snippets of audio for therapy. We also take a clip from the user’s top song (from YouTube) to make MusicGen generate something that reminds the user of their favorite music. ### Challenges we ran into Implementing all the novel features of our app into a coherent, user-friendly flow was quite the challenge. We had to carefully integrate systems such as GPT-3.5, MusicGen, Replicate hosting, Convex, and Spotify authentication. Each feature had its own unique requirements that needed to be met for seamless functioning.Finding the most effective prompts for our music generation - tailored to specific emotions - was another considerable task: what kind of music do "happy" people want vs "sad" people? What about specific use cases? ### Accomplishments we're proud of The use of the Convex model exposed us to new techniques for good API architecture.We were able to develop a more structured, reliable, and secure API system for Serenade. Another major milestone was incorporating social OAuth that simplified the sign-in process significantly, offering users a seamless, effortless entry into our service. We learned of the various services that Convex and Hume.ai can provide, and enjoyed integrating these products into our webapp. For Convex, we used its features to provide an authentication method for users and host their profiles. We learned about techniques for good API architecture through the Convex model, as well as how to easily do social OAuth for easy sign-in with providers like Spotify. ### What's next Beyond CalHacks, we plan on integrating Hume.ai's streaming facial recognition technology that operates seamlessly in real-time. Utilizing this advanced technology, the ability to read and analyze facial expressions will add another layer of emotional discernment to our platform. It will significantly augment the way we detect nuanced moods and emotions that may not be entirely captured through voice alone. By combining audio and facial cues, Serenade aims to create a more personalized and enriching therapeutic journey for the user. ## README (from the GitHub repository) **Serenade** Serenade is a music therapy tool powered by generative AI, designed to cater to a wide range of audiences including children, adolescents, adults, and the elderly with various needs. It is particularly beneficial for individuals with mental health needs, developmental and learning disabilities, Alzheimer's disease and other aging related conditions, substance abuse problems, brain injuries, physical disabilities, and acute and chronic pain, including mothers in labor. **Problem** Serenade addresses a multitude of issues: - Emotional and psychological issues: It helps individuals manage stress, anxiety, depression, and other emotional challenges by providing an outlet for emotional expression and promoting relaxation and emotional regulation. - Cognitive difficulties: For people with cognitive impairments or neurological conditions, Serenade can help improve cognitive functions such as memory, attention, and problem-solving skills. - Physical rehabilitation: It is used in physical rehabilitation settings to enhance motor skills, coordination, and movement for individuals with physical disabilities or injuries. - Social and communication skills: Serenade can be beneficial for improving social interactions and communication skills, particularly for individuals with autism spectrum disorder, developmental disorders, or social anxiety. - Pain management: It can be used to reduce the perception of pain and discomfort, making it a valuable tool in medical settings. - Overall well-being: Serenade can enhance the overall quality of life by promoting relaxation, enhancing mood, and fostering a sense of connection and belonging. **Solution** Serenade employs music therapy, a therapeutic approach that uses music to address physical, emotional, cognitive, and social needs of individuals. It involves the use of music as a medium to help individuals improve their overall well-being and quality of life. Powered by generative AI, Serenade creates a personalized musical experience that caters to the unique needs of each individual, providing a powerful and effective therapeutic tool. ## Detected evidence (automated analysis) Indexed codebase: 26 recognized source files, 39 KB. - CSS (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (32 of 32) ``` README.md web/.env.local web/.eslintrc.json web/.gitignore web/components.json web/convex/_generated/api.d.ts web/convex/_generated/api.js web/convex/_generated/dataModel.d.ts web/convex/_generated/server.d.ts web/convex/_generated/server.js web/convex/auth.config.js web/convex/http.ts web/convex/messages.ts web/convex/spotifyAuth.ts web/convex/storeYoutubeFile.ts web/next.config.js web/package.json web/postcss.config.js web/README.md web/src/lib/utils.ts web/src/pages/_app.tsx web/src/pages/_document.tsx web/src/pages/api/generate_clip.ts web/src/pages/api/getHume.ts web/src/pages/api/getTracks.ts web/src/pages/auth.tsx web/src/pages/index.tsx web/src/styles/globals.css web/src/util_server.ts web/src/util.ts web/tailwind.config.ts web/tsconfig.json ``` ### Dependencies - web/package.json: @aws-sdk/client-s3@^3.438.0, @aws-sdk/s3-request-presigner@^3.438.0, @clerk/clerk-react@^4.27.1, @clerk/nextjs@^4.26.1, @types/axios@^0.14.0, @types/node@^20, @types/react@^18, @types/react-dom@^18, @types/styled-components@^5.1.29, autoprefixer@^10, axios@^1.6.0, class-variance-authority@^0.7.0, clsx@^2.0.0, convex@^1.5.1, daisyui@^3.9.4, dotenv@^16.3.1, eslint@^8, eslint-config-next@14.0.0, lucide-react@^0.290.0, next@14.0.0, openai@^4.14.1, postcss@^8, react@^18, react-audio-voice-recorder@^2.2.0, react-dom@^18, replicate@^0.20.1, styled-components@^6.1.0, tailwind-merge@^1.14.0, tailwindcss@^3, tailwindcss-animate@^1.0.7, typescript@^5 ### Recent commits (newest first) - Merge branch 'main' of https://github.com/kdrag0n/serenade-calhacks23 - gen - fix prompt - mp3 - ui - rmo - z - fixes - added more ui flow p2 - added more ui flow - further hume integ - Merge branch 'main' of https://github.com/kdrag0n/serenade-calhacks23 - started hume integ - Create README.md - fix url - ui - r2 - logo - convex - mtation ## Key source files (fetched from GitHub, selected and truncated for size) ### web/package.json ``` { "name": "web", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@aws-sdk/client-s3": "^3.438.0", "@aws-sdk/s3-request-presigner": "^3.438.0", "@clerk/clerk-react": "^4.27.1", "@clerk/nextjs": "^4.26.1", "@types/styled-components": "^5.1.29", "axios": "^1.6.0", "class-variance-authority": "^0.7.0", "clsx": "^2.0.0", "convex": "^1.5.1", "dotenv": "^16.3.1", "lucide-react": "^0.290.0", "next": "14.0.0", "openai": "^4.14.1", "react": "^18", "react-audio-voice-recorder": "^2.2.0", "react-dom": "^18", "replicate": "^0.20.1", "styled-components": "^6.1.0", "tailwind-merge": "^1.14.0", "tailwindcss-animate": "^1.0.7" }, "devDependencies": { "@types/axios": "^0.14.0", "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10", "daisyui": "^3.9.4", "eslint": "^8", "eslint-config-next": "14.0.0", "postcss": "^8", "tailwindcss": "^3", "typescript": "^5" } } ``` ### web/convex/_generated/server.js ```javascript /* eslint-disable */ /** * Generated utilities for implementing server-side Convex query and mutation functions. * * THIS CODE IS AUTOMATICALLY GENERATED. * * Generated by convex@1.5.1. * To regenerate, run `npx convex dev`. * @module */ import { actionGeneric, httpActionGeneric, queryGeneric, mutationGeneric, internalActionGeneric, internalMutationGeneric, internalQueryGeneric, } from "convex/server"; /** * Define a query in this Convex app's public API. * * This function will be allowed to read your Convex database and will be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const query = queryGeneric; /** * Define a query that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to read from your Convex database. It will not be accessible from the client. * * @param func - The query function. It receives a {@link QueryCtx} as its first argument. * @returns The wrapped query. Include this as an `export` to name it and make it accessible. */ export const internalQuery = internalQueryGeneric; /** * Define a mutation in this Convex app's public API. * * This function will be allowed to modify your Convex database and will be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const mutation = mutationGeneric; /** * Define a mutation that is only accessible from other Convex functions (but not from the client). * * This function will be allowed to modify your Convex database. It will not be accessible from the client. * * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument. * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible. */ export const internalMutation = internalMutationGeneric; /** * Define an action in this Convex app's public API. * * An action is a function which can execute any JavaScript code, including non-deterministic * code and code with side-effects, like calling third-party services. * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive. * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}. * * @param func - The action. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped action. Include this as an `export` to name it and make it accessible. */ export const action = actionGeneric; /** * Define an action that is only accessible from other Convex functions (but not from the client). * * @param func - The function. It receives an {@link ActionCtx} as its first argument. * @returns The wrapped function. Include this as an `export` to name it and make it accessible. */ export const internalAction = internalActionGeneric; /** * Define a Convex HTTP action. * * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object * as its second. * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`. */ export const httpAction = httpActionGeneric; ``` ### web/src/pages/index.tsx ```typescript import { Inter } from 'next/font/google'; import { use, useEffect, useRef, useState } from "react"; import styled, { keyframes } from 'styled-components' import { useRouter } from 'next/router'; import { useConvexAuth } from "convex/react"; import { SignOutButton } from '@clerk/clerk-react'; import { useAuth } from '@clerk/nextjs' import { Loader2 } from 'lucide-react'; import clsx from 'clsx'; import { fetchJson } from '@/util'; import Image from 'next/image'; const inter = Inter({ subsets: ['latin'] }); const enterDown = keyframes` 0% { opacity: 0; transform: translateY(-15px); } 100% { opacity: 1; transform: translateY(0px); } ` const enterUp = keyframes` 0% { opacity: 0; transform: perspective(500px) translateY(15px) rotateX(3deg); } 100% { opacity: 1; transform: perspective(500px) translateY(0px) rotateX(0deg); } ` interface AnimateInProps { stagger?: number } const AnimateIn = styled.div` animation: ${enterDown} 750ms ease both; animation-delay: ${p => (p.stagger ?? 0) * 150}ms; ` const ProductAnimateIn = styled.div` animation: ${enterUp} 1000ms ease both; animation-delay: ${p => (p.stagger ?? 0) * 150}ms; ` function GeneratingOverlay() { return

Happy times soon…

} function Mood({ spotData }: { spotData: string[] | null }) { const emojiBank = '😀,😁,🥹,😅,😂,🥲,😊,🙂,😌,🥰,😋,😝,🤪,😎,🤩,🥳,😒,😞,😔,😟,😖,😩,🥺,😢,😭,😤,😡,🤬,🤯,😳,😱,😰,😓,🤗,🤭,😐,😬,😯,😧,😮,😴,😪,😮,‍🤐,🤧,😷,🤒,🤕,🤠'; let { userId } = useAuth() let [mood, setMood] = useState(null) let [generating, setGenerating] = useState(false) let [audioUrl, setAudioUrl] = useState(null) let audioRef = useRef(null) let done = !!audioUrl async function generateClip() { // request setGenerating(true) let output = await fetchJson(`/api/generate_clip?userId=${encodeURIComponent(userId!)}&emoji=${encodeURIComponent(mood!)}`, { method: 'POST', }) setGenerating(false) console.log('api', output) // play from blob setAudioUrl(output.audioUrl) } return done ? <>

Relax and enjoy 🎶

{audioUrl &&
:
{mood !== null ? <>

You like this song, don’t you?

 {spotData &&

{spotData[0]}

{spotData[1]}

} {!spotData && } : <>

Hi.

How are you feeling?
{emojiBank.split(',').map((emoji, i) => ( ))}
} {audioUrl &&
} export default function Home() { const { isLoading, isAuthenticated } = useConvexAuth(); const { userId, isLoaded } = useAuth(); const router = useRouter(); const [spotData, setSpotData] = useState(null); const fetchSpotifyTopTrack = async () => { try { const response = await fetch(`/api/getTracks?userId=${userId}`); const data = await response.json(); const title = data.items[0].name; const artist = data.items[0].artists[0].name; const album = data.items[0].album.name; const link = data.items[0].external_urls.spotify; return [title, artist, link] } catch (error) { console.error("Error fetching data from external resource:", error); return null; } }; useEffect(() => { if (isAuthenticated === false && isLoading === false) { router.push('/auth'); } else if (isAuthenticated === true && userId) { fetchSpotifyTopTrack().then((sData) => { setSpotData(sData); }) } }, [isAuthenticated, isLoading, isLoaded, router, userId]); return (
logo {isAuthenticated ? <> : } {/* TODO fix react error */}
) } ``` ### web/postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, } ``` ### web/next.config.js ```javascript /** @type {import('next').NextConfig} */ const nextConfig = { reactStrictMode: true, compiler: { // Enables the styled-components SWC transform styledComponents: true }, } require('dotenv').config(); module.exports = nextConfig ``` ### web/tailwind.config.ts ```typescript import type { Config } from 'tailwindcss' const config: Config = { darkMode: ["class"], content: [ './src/pages/**/*.{js,ts,jsx,tsx,mdx}', './src/components/**/*.{js,ts,jsx,tsx,mdx}', './src/app/**/*.{js,ts,jsx,tsx,mdx}', ], theme: { container: { center: true, padding: "2rem", screens: { "2xl": "1400px", }, }, extend: { backgroundImage: { 'gradient-radial': 'radial-gradient(var(--tw-gradient-stops))', 'gradient-conic': 'conic-gradient(from 180deg at 50% 50%, var(--tw-gradient-stops))', }, colors: { border: "hsl(var(--border))", input: "hsl(var(--input))", ring: "hsl(var(--ring))", background: "hsl(var(--background))", foreground: "hsl(var(--foreground))", primary: { DEFAULT: "hsl(var(--primary))", foreground: "hsl(var(--primary-foreground))", }, secondary: { DEFAULT: "hsl(var(--secondary))", foreground: "hsl(var(--secondary-foreground))", }, destructive: { DEFAULT: "hsl(var(--destructive))", foreground: "hsl(var(--destructive-foreground))", }, muted: { DEFAULT: "hsl(var(--muted))", foreground: "hsl(var(--muted-foreground))", }, accent: { DEFAULT: "hsl(var(--accent))", foreground: "hsl(var(--accent-foreground))", }, popover: { DEFAULT: "hsl(var(--popover))", foreground: "hsl(var(--popover-foreground))", }, card: { DEFAULT: "hsl(var(--card))", foreground: "hsl(var(--card-foreground))", }, }, borderRadius: { lg: "var(--radius)", md: "calc(var(--radius) - 2px)", sm: "calc(var(--radius) - 4px)", }, keyframes: { "accordion-down": { from: { height: '0' }, to: { height: "var(--radix-accordion-content-height)" }, }, "accordion-up": { from: { height: "var(--radix-accordion-content-height)" }, to: { height: '0' }, }, }, animation: { "accordion-down": "accordion-down 0.2s ease-out", "accordion-up": "accordion-up 0.2s ease-out", }, }, }, plugins: [require('daisyui'), require("tailwindcss-animate")], } export default config ``` ### web/convex/auth.config.js ```javascript export default { providers: [ { domain: "https://helpful-gobbler-81.clerk.accounts.dev", applicationID: "convex", }, ] }; ``` ### web/convex/messages.ts ```typescript import { mutation } from "./_generated/server"; export const generateUploadUrl = mutation(async (ctx) => { return await ctx.storage.generateUploadUrl(); }); ``` ### web/src/util_server.ts ```typescript import { exec, execFile } from 'child_process' import { promisify } from 'util' const execFileAsync = promisify(execFile) export async function runProcess(exe: string, args: string[]) { let { stdout, stderr } = await execFileAsync(exe, args) return stdout + stderr } ``` ### web/convex/http.ts ```typescript import { httpRouter } from "convex/server"; import { spotifyAuth } from "./spotifyAuth"; import { storeYoutubeFile } from "./storeYoutubeFile"; const http = httpRouter(); http.route({ path: "/getAuth", method: "POST", handler: spotifyAuth, }); http.route({ path: "/storeYoutubeFile", method: "POST", handler: storeYoutubeFile, }); // Convex expects the router to be the default export of `convex/http.js`. export default http; ``` [15 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]