# Project export: Rythm Moodify

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: Cal Hacks 10.0
- Tagline: Mood-Driven Melodies: Emotion-Driven AI Reflecting Your Day’s Mood through Vector-Based Song Recommendations.
- Devpost: https://devpost.com/software/rythm-moodify
- GitHub: https://github.com/triquenguyen/rythm-moodify
- Team: 2 GitHub contributor(s) — Trique Nguyen (4 commits), Jerry L (1 commits)

## Devpost submission (written by the team)

### Inspiration

There were two main inspirations for this project. Many people suggest that AI tools like ChatGPT are very good at suggesting new music to listen to based on past experiences and genres. There was also another similar project that was ideated in Cal Hacks 9.0 which does music recommendation based on heartbeat rhythms using Zepp OS's watch sensors. This time we decided to do a similar song recommendation system but based on a Journal entry.

### What it does

Recommend songs based on your mood as it is presented in your journal entries. Tech Stack Frontend: React, Tailwind, Auth0, SpotifyAPI Backend: Convex, FastAPI, Javascript, Python, OpenAI API, Genius API Database: CockroachDB, Milvus

### Challenges we ran into

Getting together.ai to properly clean our lyric texts. Creating a Milvus database without an index. Getting the Spotify API to work properly together with authentication and Convex.

### Accomplishments we're proud of

Finishing our app and having a working product that we ourselves would actually use.

### What we learned

How to generate and store vector embeddings in a vector database; how to integrate different APIs, such as the Spotify API, into Convex.

### What's next

Integrating a user's playlist for personalized recommendation. Ability to pick and choose generated choices. Integrating real-time recording feedback

## README (from the GitHub repository)

No README available.

## Detected evidence (automated analysis)

Indexed codebase: 27 recognized source files, 50 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- FastAPI (technology) — claimed on Devpost, not found in the code
- OpenAI (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (32 of 32)

```
client/.eslintrc.cjs
client/.gitignore
client/convex/_generated/api.d.ts
client/convex/_generated/api.js
client/convex/_generated/dataModel.d.ts
client/convex/_generated/server.d.ts
client/convex/_generated/server.js
client/convex/auth.config.js
client/convex/files.js
client/convex/getEmotions.js
client/convex/getPredictions.js
client/convex/getRecordPredictions.js
client/convex/README.md
client/convex/tsconfig.json
client/index.html
client/main.py
client/package.json
client/postcss.config.js
client/README.md
client/src/App.css
client/src/App.jsx
client/src/components/LoginBtn.jsx
client/src/components/LogoutBtn.jsx
client/src/components/Navbar.jsx
client/src/components/RecordPredictions.jsx
client/src/components/Transcript.jsx
client/src/index.css
client/src/main.jsx
client/src/storage/hume-test.m4a
client/src/utils/index.js
client/tailwind.config.js
client/vite.config.js
```

### Dependencies

- client/package.json: @auth0/auth0-react@^2.2.1, @types/react@^18.2.15, @types/react-dom@^18.2.7, @vitejs/plugin-react@^4.0.3, @xixixao/uploadstuff@^0.0.5, autoprefixer@^10.4.16, convex@^1.5.1, eslint@^8.45.0, eslint-plugin-react@^7.32.2, eslint-plugin-react-hooks@^4.6.0, eslint-plugin-react-refresh@^0.4.3, node-fetch@^2.7.0, postcss@^8.4.31, react@^18.2.0, react-dom@^18.2.0, tailwindcss@^3.3.5, vite@^4.4.5

### Recent commits (newest first)

- Add files via upload
- transcript fixed
- finished transcript
- finished authentication
- first commit

## Key source files (fetched from GitHub, selected and truncated for size)

### client/package.json

```
{
  "name": "client",
  "private": true,
  "version": "0.0.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "lint": "eslint . --ext js,jsx --report-unused-disable-directives --max-warnings 0",
    "preview": "vite preview"
  },
  "dependencies": {
    "@auth0/auth0-react": "^2.2.1",
    "@xixixao/uploadstuff": "^0.0.5",
    "convex": "^1.5.1",
    "node-fetch": "^2.7.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0"
  },
  "devDependencies": {
    "@types/react": "^18.2.15",
    "@types/react-dom": "^18.2.7",
    "@vitejs/plugin-react": "^4.0.3",
    "autoprefixer": "^10.4.16",
    "eslint": "^8.45.0",
    "eslint-plugin-react": "^7.32.2",
    "eslint-plugin-react-hooks": "^4.6.0",
    "eslint-plugin-react-refresh": "^0.4.3",
    "postcss": "^8.4.31",
    "tailwindcss": "^3.3.5",
    "vite": "^4.4.5"
  }
}

```

### client/main.py

```python
import requests
from lyricsgenius import Genius
import psycopg
import time
import asyncio
import traceback
import json
import dirtyjson
import random
from retrying import retry
import os

from hume import HumeStreamClient
from hume.models.config import LanguageConfig

url = "https://accounts.spotify.com/api/token"
headers = {
    "Content-Type": "application/x-www-form-urlencoded",
}
data = {
    "grant_type": "client_credentials",
    "client_id": "6667498822ba4ec8bbf0cb719ec7c69c",
    "client_secret": "5d1d0284dfc2423b82d8e7aa6f33454d",
}

response = requests.post(url, headers=headers, data=data)
response = response.json()

token = response['access_token']
token = 'BQASR8_ZWvMVhkf66eNh220DzwrdRLoGjI9DcEOVHxe3EA_VezwwfL0Zu8wJEJ6zJ0huHuSHZkmuM7r9WH-xX8_Q7RTWSCPi0wk45yg1gA0Zo3JY8pg'

data = requests.get('https://api.spotify.com/v1/playlists/1xJHno7SmdVtZAtXbdbDZp/tracks?market=US',headers={"Authorization":'Bearer '+token})

data = data.json()

total = int(data.get('total'))

songs = list()

def get_top_three_scores(dict_list):
    sorted_list = sorted(dict_list, key=lambda x: x['score'], reverse=True)
    return sorted_list[:3]

async def emotion(text_sample):
    try:
        client = HumeStreamClient("k3KreyCvTSHbCDTgAqcGC95K2nGkfOMAGDfGZiYxq0PmNFZL")
        config = LanguageConfig(granularity="sentence")
        async with client.connect([config]) as socket:
                result = await socket.send_text(text_sample)
                emotions = result["language"]["predictions"][0]["emotions"]
                print(f"\n{text_sample}")

                return get_top_three_scores(emotions)
                
    except Exception as e:
        print("Problem with emotion")
        print(e)

async def main():

    conn = psycopg.connect('postgresql://trique:Z-TRgb3ZtnqnSTYYKq91bw@garden-hermit-3692.g95.cockroachlabs.cloud:26257/rythm?sslmode=verify-full')

    for x in range(12):

        name = data.get('items')[x].get('track').get('name')
        artist = data.get('items')[x].get('track').get('artists')[0].get('name')
        
        with conn.cursor() as cur:
            cur.execute('SELECT name FROM rythm.songs WHERE name = %s;', (name,))
            result = cur.fetchone()
            if result is not None:
                continue

        genius = Genius('5aMAFiETHLqQc_AQR4nzGdPwJalnqddmUzKZMEptzUx0xu0CLU1yoQpjEL3-_MLL')
        genius.verbose = False # Turn off status messages
        genius.remove_section_headers = True # Remove section headers (e.g. [Chorus]) from lyrics when searching
        genius.skip_non_songs = False # Include hits thought to be non-songs (e.g. track lists)
        genius.excluded_terms = ["(Remix)", "(Live)"] # Exclude songs with these words in their title
            
        song = genius.search_song(name, artist)
        if song is None:
            continue
        

        e = await emotion(song.lyrics)
        songs.append({'name':name,'artist':artist, 'lyrics':song.lyrics, 'emotion':e})

    for s in songs:

        emotionstring = ""

        try:
            for e in s['emotion']:
                print("Emotion",e)
                emotionstring += e['name'] + " " + str(e['score']) + " "
        except KeyError as e:
            print(f"KeyError: {e}")
        except TypeError as e:
            print(f"TypeError: {e}")
        except Exception as e:
            print(f"Exception: {e}")
        else:
            time.sleep(2)
            print("Emotion String", emotionstring)
        try:
            with conn.cursor() as cur:
                cur.execute("INSERT INTO rythm.songs (name, artist, lyrics, emotion) VALUES (%s, %s, %s, %s)",(s['name'], s['artist'], s['lyrics'], emotionstring))  
            conn.commit()
        except Exception as e:
            print(s["name"])
            print(e)

    conn.close()
    exit()
    def submit_job(s) -> str:
        url = "https://api.together.xyz/inference"
        prompt_list = [
            "[INST] Assume the role of the artist, with the lyrics obtained through web scraping. \nEliminate any sections that do not appear to belong to the song\n " + s['lyrics'] + "\n Provide your answer as a ONLY JSON object in the format: `{'lyrics': ''}` \n Be sure characters like apostrophe are correctly escaped \n Output should be parse-able in Python [/INST]",
            "[INST] Act as the artist, with the lyrics sourced from online web scraping. \nRemove parts that don't seem to be a part of the song\n " + s['lyrics'] + "\n Provide your answer as a ONLY JSON object in the format: `{'lyrics': ''}` \n Be sure characters like apostrophe are correctly escaped \n Output should be parse-able in Python [/INST]",
            "[INST] Assume the identity of the artist, with lyrics acquired through web scraping. \nDiscard any portions that do not seem to be part of the song\n " + s['lyrics'] + "\n Provide your answer as a ONLY JSON object in the format: `{'lyrics': ''}` \n Be sure characters like apostrophe are correctly escaped \n Output should be parse-able in Python [/INST]",
            "[INST] Pretend to be the artist, with the lyrics retrieved from online web scraping. \nEliminate any segments that don't seem to belong to the song\n " + s['lyrics'] + "\n Provide your answer as a ONLY JSON object in the format: `{'lyrics': ''}` \n Be sure characters like apostrophe are correctly escaped \n Output should be parse-able in Python [/INST]",
            "[INST] Take on the persona of the artist, with the lyrics web scraped from online sources. \nRemove any sections that appear to be extraneous to the song\n " + s['lyrics'] + "\n Provide your answer as a ONLY JSON object in the format: `{'lyrics': ''}` \n Be sure characters like apostrophe are correctly escaped \n Output should be parse-able in Python [/INST]"
        ]
        payload = {
            "model": "togethercomputer/llama-2-70b-chat",
            "prompt": prompt_list[random.randint(0,len(prompt_list)-1)],
            "max_tokens": 2000,
            "temperature":
[truncated — 5449 more characters]
```

### client/src/App.jsx

```javascript
import { useState } from 'react'
import './App.css'
import LoginBtn from './components/LoginBtn'

import { Authenticated, Unauthenticated, AuthLoading } from "convex/react";
import Navbar from './components/Navbar';
import Transcript from './components/Transcript';
import RecordPredictions from './components/RecordPredictions';

function App() {
  return (
    <div className="bg-[#022A3B] w-[100vw] h-[100vh]">
      <Navbar />

      <Transcript />

      <RecordPredictions />
    </div>
  );
}
export default App


```

### client/src/main.jsx

```javascript
import React from "react";
import ReactDOM from "react-dom/client";
import App from "./App";
import "./index.css";
import { ConvexReactClient } from "convex/react";
import { ConvexProviderWithAuth0 } from "convex/react-auth0";
import { Auth0Provider } from "@auth0/auth0-react";

const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL);

ReactDOM.createRoot(document.getElementById("root")).render(
  <React.StrictMode>
    <Auth0Provider
      domain="dev-olog88c7d4x1phbb.us.auth0.com"
      clientId="mGqeZDZOexm765AzzrCyU5ISI9mkQFW1"
      authorizationParams={{
        redirect_uri: window.location.origin,
      }}
      useRefreshTokens={true}
      cacheLocation="localstorage"
    >
      <ConvexProviderWithAuth0 client={convex}>
        <App />
      </ConvexProviderWithAuth0>
    </Auth0Provider>
  </React.StrictMode>
);
```

### client/src/utils/index.js

```javascript
const BASE_URL = 'https://api.hume.ai/v0/batch/jobs';

// 1. Set your API Key
const HUME_API_KEY = 'miqSHRUTrMYw3C8Mg6cXWtu5kRfO51TrGnF1MgiJ8q2LAbRU';

// 2. Specify which Language
const language = 'en';

// 3. Specify Language Model configuration
const languageModelConfig = {};

// 4. Copy and paste the text you'd like processed here
const rawTextInput = '';

async function processRawText(
  rawText,
  language,
  languageModelConfig
) {
  const MAX_RETRIES = 5; // adjust the number of retries here
  const INITIAL_DELAY_MS = 1000; // starting with 1 second delay

  let delay = INITIAL_DELAY_MS;
  const jobId = await startJob(rawText, language, languageModelConfig);

  // poll with exponential backoff
  for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
    const status = await getJobStatus(jobId);

    if (status === 'COMPLETED') {
      console.log('Status is COMPLETED!');
      const predictions = await getPredictions(jobId);
      console.log(JSON.stringify(predictions));
      return;
    }
    console.log(`Status is ${status}. Retrying in ${delay / 1000} seconds...`);
    await sleep(delay);
    delay *= 2; // exponential backoff
  }

  console.error('Max retries reached. Giving up.');
}

async function startJob(
  rawText,
  language,
  languageModelConfig
) {
  const body = JSON.stringify({
    text: [rawText],
    models: { language: languageModelConfig },
    transcription: { language },
  });
  const options = { ...buildHumeRequestOptions('POST'), body };
  const response = await fetch(BASE_URL, options);
  if (!response.ok) {
    throw new Error(`Failed to start job: ${response.statusText}`);
  }
  const json = await response.json();

  return json.job_id;
}

async function getJobStatus(jobId) {
  const options = buildHumeRequestOptions('GET');
  const response = await fetch(`${BASE_URL}/${jobId}`, options);
  if (!response.ok) {
    throw new Error(`Failed to fetch job status: ${response.statusText}`);
  }
  const json = await response.json();

  return json.state.status;
}


function buildHumeRequestOptions(method) {
  const headers = new Headers();
  headers.append('X-Hume-Api-Key', HUME_API_KEY);
  headers.append('Content-Type', 'application/json');

  return { method, headers };
}

/**
 * Helper function to support exponential backoff implementation when polling for job status.
 */
function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}
```

### client/convex/_generated/server.js

```javascript
/* eslint-disable */
/**
 * Generated utilities for implementing server-side Convex query and mutation functions.
 *
 * THIS CODE IS AUTOMATICALLY GENERATED.
 *
 * Generated by convex@1.5.1.
 * To regenerate, run `npx convex dev`.
 * @module
 */

import {
  actionGeneric,
  httpActionGeneric,
  queryGeneric,
  mutationGeneric,
  internalActionGeneric,
  internalMutationGeneric,
  internalQueryGeneric,
} from "convex/server";

/**
 * Define a query in this Convex app's public API.
 *
 * This function will be allowed to read your Convex database and will be accessible from the client.
 *
 * @param func - The query function. It receives a {@link QueryCtx} as its first argument.
 * @returns The wrapped query. Include this as an `export` to name it and make it accessible.
 */
export const query = queryGeneric;

/**
 * Define a query that is only accessible from other Convex functions (but not from the client).
 *
 * This function will be allowed to read from your Convex database. It will not be accessible from the client.
 *
 * @param func - The query function. It receives a {@link QueryCtx} as its first argument.
 * @returns The wrapped query. Include this as an `export` to name it and make it accessible.
 */
export const internalQuery = internalQueryGeneric;

/**
 * Define a mutation in this Convex app's public API.
 *
 * This function will be allowed to modify your Convex database and will be accessible from the client.
 *
 * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument.
 * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible.
 */
export const mutation = mutationGeneric;

/**
 * Define a mutation that is only accessible from other Convex functions (but not from the client).
 *
 * This function will be allowed to modify your Convex database. It will not be accessible from the client.
 *
 * @param func - The mutation function. It receives a {@link MutationCtx} as its first argument.
 * @returns The wrapped mutation. Include this as an `export` to name it and make it accessible.
 */
export const internalMutation = internalMutationGeneric;

/**
 * Define an action in this Convex app's public API.
 *
 * An action is a function which can execute any JavaScript code, including non-deterministic
 * code and code with side-effects, like calling third-party services.
 * They can be run in Convex's JavaScript environment or in Node.js using the "use node" directive.
 * They can interact with the database indirectly by calling queries and mutations using the {@link ActionCtx}.
 *
 * @param func - The action. It receives an {@link ActionCtx} as its first argument.
 * @returns The wrapped action. Include this as an `export` to name it and make it accessible.
 */
export const action = actionGeneric;

/**
 * Define an action that is only accessible from other Convex functions (but not from the client).
 *
 * @param func - The function. It receives an {@link ActionCtx} as its first argument.
 * @returns The wrapped function. Include this as an `export` to name it and make it accessible.
 */
export const internalAction = internalActionGeneric;

/**
 * Define a Convex HTTP action.
 *
 * @param func - The function. It receives an {@link ActionCtx} as its first argument, and a `Request` object
 * as its second.
 * @returns The wrapped endpoint function. Route a URL path to this function in `convex/http.js`.
 */
export const httpAction = httpActionGeneric;

```

### client/postcss.config.js

```javascript
export default {
  plugins: {
    tailwindcss: {},
    autoprefixer: {},
  },
}

```

### client/tailwind.config.js

```javascript
/** @type {import('tailwindcss').Config} */
export default {
  content: [
    "./index.html",
    "./src/**/*.{js,ts,jsx,tsx}",
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}
```

### client/vite.config.js

```javascript
import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'
// import cors from 'vite-plugin-cors';


// https://vitejs.dev/config/
export default defineConfig({
  plugins: [react()],
  server: {
    cors: {
      origin: false
    },
  }
})

```

### client/index.html

```html
<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Vite + React</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/main.jsx"></script>
  </body>
</html>

```

[16 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]