# Project export: Mind.me This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 10.0 - Tagline: Mind.me serves as your personal emotional guide, utilizing data from your Zepp watch to interpret your emotional state and provide thorough GenAI tailored suggestions to improve your status - Devpost: https://devpost.com/software/mind-me - GitHub: https://github.com/jacopo-minniti/mind.me - Demo: https://calhacks-fe.vercel.app/ - Video: https://player.vimeo.com/video/879184558?byline=0&portrait=0&title=0#t= - Team: 1 GitHub contributor(s) — Jacopo Minniti (14 commits) ## Devpost submission (written by the team) ### Inspiration With Mind.me, emotions are finally on your side. In today’s fast-paced world, mental health often takes a backseat. However, it’s crucial to remember that our emotional well-being is just as important as our physical health. The inspiration behind Mind.me is the understanding that managing mental health should be a priority, not an afterthought. ### What it does Mind.me is an innovative system designed to enhance users' emotional awareness and empower them to manage their emotional well-being effectively. It operates seamlessly in the background, collecting data from various sources like heart rate, blood pressure, and voice recordings. This data is analyzed to track your mood throughout the day and uncover trends and patterns. The insights gained are then used to generate personalized, detailed suggestions and advice to help you manage your emotions better. ### How we built it Mind.me is a comprehensive system that includes a front-end application built using the Zepp Framework and a user-exclusive webpage. The webpage provides detailed explanations of the user's emotional status, refined prompts, and trends and graphs for understanding their long-term emotional trajectory. The system leverages MindsDB for swift integration of the MySQL database in the backend and machine learning models. It utilizes GPT3.5 Turbo to construct human-like, detailed suggestions. The system uses a total of 57 predictors, including health data captured through the Zepp Watch sensors and emotions mapped throughout the day via the user's voice and discourse. To convert voice into emotions, Mind.me employs Hume API, which uses state-of-the-art deep learning models specifically designed for this purpose. All data is stored in a MySQL database in the backend for optimal performance. ### What's next While Mind.me has already many features to help users manage their emotional well-being, there's still a long way to go. This application has the potential to change the lives of millions of people around the world. Future development will focus on improvements through third-party integrations and more complex data analytics. ## README (from the GitHub repository) ![Xthin Header (1)](https://github.com/jacopo-minniti/mind.me/assets/115539886/faf2fc57-c6f2-4501-bad4-03519f6a1814) [ Project Winner of two prizes at Calhacks 2023 ] # mind.me ### Your emotions, finally on your side. For the connected repo of the webpage, visit: https://github.com/Tig-ranK/calhacks-fe For the deployed webpage, visit: https://calhacks-fe.vercel.app/ ## What is mind.me Mind.me is a pioneering integrated system designed to enhance users’ emotional awareness and empower them to manage and improve their emotional well-being in both the short and long term. By collecting data from recordings, heart rate, blood pressure, and other scientifically-based predictors, it tracks your mood throughout the day, 24 hours a day. It uncovers trends and patterns which are then used to generate personalized, detailed, and specific suggestions and advice. These recommendations are interactive and genuinely useful, thanks to numerous integrations with third-party apps. ![Presentation UI](https://github.com/jacopo-minniti/mind.me/assets/115539886/7b1ddacf-de79-44bf-b207-90c0548bcf30) Using Mind.me is as simple as wearing a watch. It operates in the background and never requires any input from you - it simply provides output. The data we collect from you remains yours, and we use it solely for your benefit to make predictions. ## How it is built ![FINALXOVERVIEW](https://github.com/jacopo-minniti/mind.me/assets/115539886/a3b49bbb-035a-4f87-b5a5-86c69b4c0e9b) Mind.me is a comprehensive system that includes a front-end application, constructed using the Zepp Framework directly on the watch, and a user-exclusive webpage. This webpage provides detailed explanations of the user's emotional status, more refined prompts, and trends and graphs that allow users to understand their long-term emotional trajectory. Powered by MindsDB, Mind.me enables swift integration of the MySQL database in the backend and machine learning models. It leverages GPT3.5 Turbo, one of the most sophisticated generative AI tools, to construct the most human-like, detailed suggestions. The system utilizes a total of 57 predictors, which include (i) health data captured through the Zepp Watch sensors (e.g., heart rate), and (ii) emotions mapped throughout the day via the user's voice and discourse. To convert voice into emotions, Mind.me employs Hume API, which utilizes highly sophisticated, state-of-the-art deep learning models specifically designed for this purpose. All data is stored in a MySQL database in the backend. To ensure optimal performance, .wav files are stored separately on Google Cloud Storage. Python scripts are deployed as cloud functions through Google Clouds, and a custom API connects the webpage to the database. ## What's next The path is still long. This application has the potential to change the lives of millions of people around the World. The improvements, through third-party integrations and more complex data analytics, will constitute the biggest objective of its future development. ## Detected evidence (automated analysis) Indexed codebase: 6 recognized source files, 17 KB. - Python (language) — detected in the code - JavaScript (language) — claimed on Devpost, not found in the code - React (technology) — claimed on Devpost, not found in the code - SQL (language) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (8 of 8) ``` .gitignore .vscode/settings.json database_connection.py google_storage_helpers.py hume_helpers.py main.py predictions.py README.md ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - Update google_storage_helpers.py - Update database_connection.py - Update hume_helpers.py - Delete prediction.py - Update and rename minds_db.py to predictions.py - Update main.py - Delete api_try.py - Update README.md - First commit - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### main.py ```python # In main.py, most of other scripts are executed or used as helpers. # The objective is: # 1. Retrieve audio data from the stroage bucket # 2. Run the audio data through Hume to get emotions predictios # 3. Store the emotions predictions in the database from hume import HumeBatchClient from hume.models.config import ProsodyConfig from google.cloud import storage import database_connection as db import hume_helpers as hh import google_storage_helpers as gsh from api_keys import HUME_API_KEY, GPT3_API_KEY, GOOGLE_APPLICATION_CREDENTIALS hume_client = HumeBatchClient(HUME_API_KEY) # Google Cloud Storage was chosen as the storage solution for this project, because: # 1. It is a cloud-based storage solution, which means it is accessible from anywhere # 2. It is a cost-effective solution, because it is a pay-as-you-go service # 3. It is a highly scalable solution # 4. The data did not need any particular ordering or structure, so a NoSQL was great. # storage_client = storage.Client(project=GOOGLE_APPLICATION_CREDENTIALS) print('Clients initialized') # The configs, necessary for Hume operations, can speicfy multiple models, # that take as data different sources (audio, text, etc.). In our case, considering how # it would have been cumbersome (and not respectful of privacy at all), we chose to considere # exclusevely audio recordings. configs = [ProsodyConfig()] # in a real context, these would be retrieved from the database. # In the google cloud storage bucket, the audio files are stored in a folder structure like this: # - Hume Recordings # - user_id # - date # - audio_file # The files are uploaded to Cloud Storage from the Zepp watch, and they are about 3 minutes long each # The audio files are stored in the bucket in the following format: user_id_date_time # They are quite short, as batching more requests in one is faster than one giant file with the batch Hume API. # Examples of user_id (unique integer) and date (string in the format YYYY_MM_DD) user_id = 1 date = '2023_10_28' # Retrieve urls, and blob_ids from the storage bucket # On the database, we store just the public url, not the file itself, as it would be too expensive [blob_ids, urls] = gsh.blobs_to_update(storage_client, user_id, date) print('urls retrieved') print('Running predictions...') # Run predictions on the retrieved urls. # The predictions are returned in the format of a list of dictionaries. emotions_data = hh.get_emotions_data(hume_client, urls, configs) print('Emotional Data retrieved') # After having retrieved the emotions data, we can update the database with it. db.connect_and_execute({'emotions_data': emotions_data, 'user_id': user_id}) ``` ### google_storage_helpers.py ```python import os # Set the path to the credentials file # THis code needs to be executed just once, and then can be removed. credential_path = "/Users/USERNAME/.config/gcloud/application_default_credentials.json" os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credential_path # blobs are the individual audio files. # Each blob is divided in a ~ 5 second audio segment, but originally it is a ~ 3 min video file. # The blobs to update are determined so based on the time. # In a real world scenario, a cloud function checks when a day or so passed since last report, and analyes the yet not analyzed blobs. # Thanks to the highly organized structure, you can construct the path easly to get the files. def blobs_to_update(client, id, date): sub_bucket = f'{id}_{date}' # this is the constant, first bucket. # All recordings are stored in this bucket, but each user has its own sub-bucket. bucket = client.get_bucket('hume_recordings') blobs = list(bucket.list_blobs(prefix=sub_bucket)) # When there are other directopries, the first element returned is not a blob. blobs = blobs[1:] # List to store the public URLs of the blobs blob_urls = [] # List to store the names of the blobs blob_names = [] for blob in blobs: blob_names.append(blob.name) blob_urls.append(blob.public_url) return [blob_names, blob_urls] ``` ### database_connection.py ```python import mysql.connector from api_keys import HOST, DATABASE, USER, PASSWORD # Configuration to access a mysql database in local. The database will be used as a data source for MindsDB. db_config = { 'user': USER, 'password': PASSWORD, 'host': HOST, 'database': DATABASE, } # This function has two objectives: # 1. Connect to the database # 2. Execute a query # The function was built in a way that it can be used to execute different types of query. # In our case example, we are using it to execute an INSERT query. def connect_and_execute(args): conn = None try: conn = mysql.connector.connect(**db_config) if conn.is_connected(): print('Connected to MySQL database') cursor = conn.cursor() # we get the specific query statement and the values to fill it. # executemany() and commit() permit to execute bath queries, whjich considering the vast amount of short audio segments (i.e. rows) # that we have, it is great. [query, values] = add_emotions_query( args['emotions_data'], args['user_id']) cursor.executemany(query, values) conn.commit() # baisc conncetion error handling except ConnectionError as e: print(f'Connection Failed: {e}') except Exception as e: print(f'Exception during query execution: {e}') finally: if conn is not None and conn.is_connected(): conn.close() # the following function returns the query statement and the corresponding values. def add_emotions_query(emotions_data, user_id): values = [] # the emotion data, which is a list of dictionaries, is converted into a list of tuples. # others parameters not measured by hume are addedd (e.g. heart rate, pressure, breath) and initialized to 'semi-null' values for data in emotions_data: values.append(([user_id, 0.0, 0.0, 0.0] + list(data.values()) + ['no'])) values = tuple(values) # To maintain the simmerity also the query will consider these other non-emotions factors. column_names = ["user_id", "Heart Rate BPM", "Pressure", "Breath"] + \ list(emotions_data[0].keys()) + ["Suggestions/Advices"] query = f"INSERT INTO emotions_table ({', '.join(['`' + col + '`' for col in column_names])}) VALUES " # While it may not be the most elegant of choices, this code adds a placeholder for each value in the query statement. # The number of placeholders is equal to the number of columns in the table. # It is costant so can be hardcoded. query += '(%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' return [query, values] ``` ### hume_helpers.py ```python from hume import HumeBatchClient from hume.models.config import ProsodyConfig # the following function uses the Hume API to get the predictions # each emotions is a value from 0 to 1; the higher, the more intense the emotion. # It is important to note how the values of all the emotions do not sum up to 1, as one can be at the same time highly calm and happy, for example. # The emotions are 48 in total. def get_predictions(client, urls, configs): job = client.submit_job(configs=configs, urls=urls) # timeout is increased signficicantly. COnsidering this operation is to be executed once per day (for each user), # there is no excessive need for speed, and it is more important that the transaction is completed. # notice this is an async block job.await_complete(timeout=800) results = job.get_predictions() predictions = [] for result in results: # This long line is to get the actual predictions from the results. # The structure of the json file retrieved is highlt complex and convulted. # Considering that most of the data is useless for our purposes, we reduce the data to be sent to the database considerably. # Fundamentally, we are only interested in the final dictionary with the emotions mapped. predictions.append(result["results"]['predictions'][0]['models'] ['prosody']['grouped_predictions'][0]['predictions']) return predictions # The following functions are used to get the specific components of the predictions list. # Remember as audio files are segnmented in ~ 5 second audio files, so the list will grow linearly with longer videos. # Returns a list of maps. # Each map is a dictionary with the emotions as keys and the values as the score of the emotion def get_emotions(predictions): total_emotions = [] for prediction in predictions: emotions = [] for emotion in prediction['emotions']: emotions.append(emotion['score']) total_emotions.append(emotions) return total_emotions # Returns a list of lists. # Each list is [begin, end] timestamp for the specific segment of the audio file. def get_timestamp(predictions): timestamps = [] for prediction in predictions: begin = prediction['time']['begin'] end = prediction['time']['end'] timestamps.append([begin, end]) return timestamps # Returns a list of strings. # Each string is the transcript of the specific segment of the audio file. def get_transcript(predictions): transcripts = [] for prediction in predictions: transcripts.append(prediction['text']) return transcripts # Returns the cohmprensive data. # The others function are needed in specific contexts, while this returns the data in a format that can be easily used for the database. # It returns everything we need to uploda from Hume. def get_emotions_data(client, urls, configs): predictions = get_predictions(client, urls, configs) total_emotions = [] for i in range((len(predictions))): file_prediction = predictions[i] for j in range(len(file_prediction)): segment_prediction = file_prediction[j] # create a dictionary to store the data of the single file. emotions_data = {} # add the individual parts emotions_data['transcript'] = segment_prediction['text'] emotions_data['begin'] = segment_prediction['time']['begin'] emotions_data['end'] = segment_prediction['time']['end'] for emotion in predictions[i][j]['emotions']: emotions_data[emotion['name']] = emotion['score'] total_emotions.append(emotions_data) return total_emotions ``` ### predictions.py ```python import mindsdb_sdk from api_keys import GPT3_API_KEY server = mindsdb_sdk.connect() # connects to the specified host and port server = mindsdb_sdk.connect('http://127.0.0.1:47334') # gets the server istance. emotions_db = server.get_database('emotions_db') print('Running') # check if the project exists # if it does, it access it thoug the get_project('name') method # otherwise it creates it with the create_project('name') method project_exists = False project = '' # list all the projects istances in the MindsDB server. for pg in server.list_projects(): if (pg.name == 'emotions_pj'): project_exists = True break if (project_exists): project = server.get_project('emotions_pj') else: project = server.create_project('emotions_pj') print('Create Project') # for the sake of simplicity, in our mysql database, we have only one table with all the features to make the desired predictions. # the table emotions_table has in total 57 columns. emotions_table = emotions_db.get_table('emotions_table') # the same procedure is adopted for the model model_exists = False model = '' for md in project.list_models(): if (md.name == 'babbage-002'): model_exists = True break if (model_exists): model = project.get_model('babbage-002') else: model = project.create_model( name='babbage-002', # the column we want to predict is the suggestion/advice. # Using the ones already produced by us (integrated learning) and the 50+ predictors of emotions and health records # from Zepp watch, we are going to predict how the user can improve his emotional status, though presionalized suggestions # and integrations with other apps. In its final form, Mind.me finds pattern and trends, and functions as a personal coach # which finds new, interactive, personalized ways to improve the user's emotional status. predict='suggestion/advice', # The engine was created though the console. engine='emotions_engine', options={ 'model_name': 'babbage-002', # The api key permits to use GPT3.5 Turbo 'api_key': GPT3_API_KEY, # the prompt template is used to train the model. In particular considering we do not have access to GPT4, # we generated a very precise and long prompt, to be sure the model does not produce inaccurate or inconsistent results. 'prompt_template': 'You are a sentiment classifier. Your role is to help users take back control of their emotions. You will inform them of trends, suggest them possible reasons behind some emotional status, and advice on strategies, based on their personality, to improve their emotional status. These advices should be as specific as possible. To make predictions you will use any other column in the table as input. You will analyze the column {{transcript}}, and all the columns with a score about emotions. That score is a number from 0 to 1 which indicates how much the text in the column is related to that emotion. Above all, you will use the column {{Suggestions/Advices}} as input. This is your main source. These suggestions are your labels, how to predict future entries. Your will return {{Suggestions/Advices}} with the appropriate response. If the value of the column is no, it needs to be updated. Do not produce one word answers. Your feedback has to be 3-4 sentences long. Do not create different suggestions for each row, but instead use small batchs of rows to create one suggestion and replicte it for some rows. The output is always a string, between 3-4 sentences. Your returning value is {{Suggestions/Advices}}.', # to avoid useless, not-premeditated expenses, the token limit is set to 500. 'max_tokens': 500 } ) print('Create Model') # Possible predictions. These are not useful for the actual product, and are used only for testing purposes. table_limit = emotions_table.limit(1) print('Making prediction on the first 1 rows of the table') ret = model.predict(table_limit) print(ret['suggestion/advice']) ```