# Project export: SereneAI
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: Cal Hacks 11.0
- Tagline: Serene AI – Your personal wellness coach, powered by voice interaction and seamlessly integrated with Gmail and Google Calendar to turn personalized recommendations into proactive actions.
- Devpost: https://devpost.com/software/sereneai
- GitHub: https://github.com/jw-source/SereneAI
- Team: 3 GitHub contributor(s) — Jet Wu (21 commits), cyprien (17 commits), sp12356 (11 commits)
## Devpost submission (written by the team)
### Inspiration
In today’s fast-paced world, balancing mental well-being with productivity is a challenge for many. We were inspired by the idea of creating a tool that merges emotional intelligence with proactive productivity management. Our goal was to help users manage stress, monitor their emotional health, and stay on top of tasks in a personalized and seamless way. Serene AI was born from the desire to provide users with a voice-powered, emotionally aware companion that not only listens but also takes meaningful actions to support their well-being.
### What it does
Serene AI is a voice-to-voice web application that assists users in managing both their mental well-being and productivity. It listens to users' thoughts, emotions, and concerns, offering thoughtful responses that include personalized advice and proactive actions. Through integrations with Gmail and Google Calendar, Serene AI helps users manage their schedules and tasks while simultaneously tracking emotional trends over time. It transforms well-being recommendations into actionable steps to improve overall wellness and productivity.
### How we built it
Serene AI was developed using a robust, multi-layered tech stack: Frontend: We created the interface with React and used Chart.js for visualizing emotional trends and mood scores over time. Backend: Flask was utilized to manage API interactions and host the custom AI model, which is powered by Groq, with our embedding models from OpenAI. Voice-to-Voice Interaction: Speech-to-text and text-to-speech capabilities were enabled using the VAPI API, allowing seamless voice interaction. AI & Sentiment Analysis: We integrated Hume AI for advanced emotional sentiment tracking and analysis, while ChromaDB stores conversation histories and emotional data. Productivity Management: A multi-agent framework was implemented to integrate Google Calendar and Gmail, enabling Serene to manage events, send emails, and adjust schedules based on the user’s emotional state and preferences.
### Challenges we ran into
One of the major challenges was creating agents that could interpret natural language queries to control various APIs, such as Google Calendar and Gmail. Fortunately, Groq's function-calling models simplified this process, making it easier to handle complex requests and improve the system's functionality.
### Accomplishments we're proud of
We’re proud of creating a fully functional, voice-powered wellness coach that combines mental well-being support with productivity tools. A major achievement was building a custom LLM that understands and responds to users' emotions in real-time. The multi-agent framework, which allows Serene to proactively manage calendars and emails based on emotional input, is another significant accomplishment. Additionally, the ability to visualize emotional trends and provide users with actionable well-being insights over time is a feature we’re particularly excited about.
### What we learned
Throughout the development of Serene AI, we deepened our understanding of how AI and emotional intelligence can intersect. We learned the importance of balancing personalization with user autonomy—ensuring that Serene’s suggestions are not only helpful but also actionable. We also enhanced our skills in integrating multi-agent systems, refining our ability to combine AI-driven emotional insights with practical productivity tools.
### What's next
for Serene AI Looking ahead, we plan to: Expand integrations with additional productivity tools, such as task management apps. Improve our sentiment analysis with more sophisticated real-time mood prediction models. Incorporate guided meditations, breathing exercises, and other wellness activities based on emotional feedback. Add support for multiple languages to make Serene AI accessible to a global audience. With these enhancements, Serene AI aims to become an even more effective personal wellness coach for users everywhere.
## README (from the GitHub repository)
# Serene AI - Personal Wellness Coach
Serene AI is a voice-to-voice web application designed to help users manage their mental well-being and productivity through personalized interactions. Serene listens to users' thoughts, emotions, and concerns, and responds with thoughtful advice and actions to reduce stress, manage schedules, and track emotional trends over time.
Developed for CalHacks 2024 at UC Berkeley, Serene AI integrates a custom AI model with emotional sentiment tracking and productivity management tools.
## Features
- **Voice-to-Voice Interaction**: Users can speak directly to Serene using the VAPI API, and Serene responds with personalized spoken dialogue.
- **Custom LLM**: Powered by a custom Large Language Model (LLM) using Groq and OpenAI, Serene generates context-aware responses based on user input.
- **Multi-Agent Framework**: Serene AI integrates with Google Calendar and Gmail, allowing it to:
- Retrieve and manage events and emails.
- Create or modify calendar events and send emails.
- Relieve stress by adjusting schedules based on user preferences and emotional state.
- **Sentiment Tracking**: Tracks user sentiment over time, adjusting responses to improve personalization based on emotional patterns.
- **Trends and Insights**: Users can view their emotional trends over time, with visualizations of:
- Mood scores/magnitudes across conversations.
- Top emotions felt and their frequency.
- Well-being insights over a month.
## Tech Stack
### Frontend:
- **React**: Web app interface.
- **Chart.js**: Visualizing trends and emotional data.
- **VAPI API**: Speech-to-text and text-to-speech interactions.
### Backend:
- **Flask**: To expose the custom LLM and manage API interactions.
- **Python**: For AI agent creation and overall backend logic.
### AI & Data:
- **Custom LLM**: Built using Groq and OpenAI for processing user input and generating responses.
- **Hume AI**: For advanced emotional sentiment analysis.
- **ChromaDB**: For storing conversation history, creating a knowledge base, and holding sentiment data.
- **Multi-Agent Functionality**: Allows us to control and edit your calendar or inbox directly through text and voice
## Setup and Installation
### Prerequisites
- Python 3.8 or higher
- Node.js (v14 or later)
- API keys for VAPI API, Google Calendar, Gmail, Hume AI, and OpenAI.
### Installation
1. **Clone the repository**:
```bash
git clone https://github.com/yourusername/serene-ai.git
cd SereneAI
```
2. **Install backend dependencies** (Flask, AI models, and other Python packages):
```bash
pip install -r backend/requirements.txt
```
3. **Install frontend dependencies** (React app):
```bash
cd frontend
npm install
```
4. **Set up environment variables for API keys** (create a `.env` file in both `backend` and `frontend`):
```bash
VAPI_API_KEY=your_vapi_api_key
GOOGLE_CALENDAR_API_KEY=your_google_calendar_api_key
GMAIL_API_KEY=your_gmail_api_key
HUME_AI_API_KEY=your_hume_ai_api_key
OPENAI_API_KEY=your_openai_api_key
```
### Running the Application
1. **Start the Flask backend**:
```bash
cd backend
flask run
```
2. **Start the React frontend**:
```bash
cd frontend
npm start
```
3. **Access the app in your browser**:
```
http://localhost:3000
```
## Usage
- **Conversation**: Speak directly to Serene about your current feelings, stress levels, or concerns. Serene will analyze the input and respond with advice or tips. For example, you can say:
- "I have an interview tomorrow, and I feel anxious. Can you help me calm down?"
- "I have a presentation next week, but I get really nervous when public speaking. What can I do to prepare?"
- "I’ve been feeling overwhelmed lately. Do you have any tips to reduce my stress?"
- "I haven’t been sleeping well. Can you help me with some relaxation techniques before bed?"
- "I’m feeling a bit down today. Can we talk about ways to lift my mood?"
- **Productivity**: Ask Serene to manage your calendar or emails. For example, you can say:
- "What's on my schedule for today?"
- "Cancel my meeting at 3 PM."
- "Send an email to [recipient] about rescheduling."
- **Trends**: Track your emotional trends over time on the trends page, where you can view:
- Monthly mood scores.
- Most frequent emotions.
- Sentiment analysis over conversations.
## Roadmap
- Integration with additional productivity tools (task management apps, to-do lists).
- Enhanced mood prediction and real-time emotional analysis using deep learning models.
- Expanding language support for voice interactions.
- Incorporation of wellness activities like guided meditations and breathing exercises based on emotional state.
## License
This project is licensed under the MIT License - see the LICENSE file for details.
## Acknowledgments
- **VAPI API** for enabling seamless voice interaction.
- **Hume AI** for emotional sentiment analysis.
- **Google Cloud** for Calendar and Gmail API services.
- **OpenAI & Groq** for powering the LLMs and embedding models behind Serene AI.
- **ChromaDB** for enabling seamless memory storage and creation
## Detected evidence (automated analysis)
Indexed codebase: 21 recognized source files, 53 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
## Codebase structure (from repository index)
### Files (26 of 26)
```
calltollm.py
frontend/backend/app.py
frontend/backend/calendar_functions.py
frontend/backend/chroma.py
frontend/backend/mail_functions.py
frontend/package.json
frontend/public/index.html
frontend/public/manifest.json
frontend/public/robots.txt
frontend/src/App.css
frontend/src/App.jsx
frontend/src/App.test.js
frontend/src/components/ActiveCallDetail.jsx
frontend/src/components/base/Button.jsx
frontend/src/components/call/AssistantSpeechIndicator.jsx
frontend/src/components/call/VolumeLevel.jsx
frontend/src/components/TrendsPage.jsx
frontend/src/index.css
frontend/src/index.js
frontend/src/reportWebVitals.js
frontend/src/setupTests.js
frontend/src/utils/index.js
givetopemotions.py
package.json
README.md
requirements.txt
```
### Dependencies
- frontend/package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, @vapi-ai/web@^2.1.4, chart.js@^4.4.5, react@^18.3.1, react-chartjs-2@^5.2.0, react-dom@^18.3.1, react-router-dom@^6.27.0, react-scripts@^5.0.1, react-spinners@^0.14.1, web-vitals@^2.1.4
- package.json: chart.js@^4.4.5, react-chartjs-2@^5.2.0
- requirements.txt: beautiful-date, gcsa, groq, python-dotenv, simplegmail
### Recent commits (newest first)
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- cool stuff
- Final
- reset formatting
- reformatting
- Update and rename README.md to misc/README.md
- Delete calendar_functions.py
- Delete main.py
- Delete chroma.py
- Delete mail_functions.py
- Merge branch 'main' of https://github.com/jw-source/SereneAI
- solving old push
## Key source files (fetched from GitHub, selected and truncated for size)
### requirements.txt
```
gcsa
beautiful-date
simplegmail
python-dotenv
groq
```
### package.json
```
{
"dependencies": {
"chart.js": "^4.4.5",
"react-chartjs-2": "^5.2.0"
}
}
```
### frontend/package.json
```
{
"name": "demo",
"version": "0.1.0",
"private": true,
"dependencies": {
"@testing-library/jest-dom": "^5.17.0",
"@testing-library/react": "^13.4.0",
"@testing-library/user-event": "^13.5.0",
"@vapi-ai/web": "^2.1.4",
"chart.js": "^4.4.5",
"react": "^18.3.1",
"react-chartjs-2": "^5.2.0",
"react-dom": "^18.3.1",
"react-router-dom": "^6.27.0",
"react-scripts": "^5.0.1",
"react-spinners": "^0.14.1",
"web-vitals": "^2.1.4"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": [
"react-app",
"react-app/jest"
]
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
}
}
```
### frontend/src/index.js
```javascript
import "./index.css";
import App from "./App";
import React from "react";
import ReactDOM from "react-dom/client";
// render application
const root = ReactDOM.createRoot(document.getElementById("root"));
root.render(
);
```
### frontend/src/App.jsx
```javascript
import { useEffect, useState } from "react";
import { BrowserRouter as Router, Route, Routes, Link } from "react-router-dom";
import ActiveCallDetail from "./components/ActiveCallDetail";
import Button from "./components/base/Button";
import Vapi from "@vapi-ai/web";
import { isPublicKeyMissingError } from "./utils";
import TrendsPage from "./components/TrendsPage"; // Import the new page
// Put your Vapi Public Key below.
const vapi = new Vapi("def07e68-c687-4d95-80dd-a171cc01d29c");
const App = () => {
const [connecting, setConnecting] = useState(false);
const [connected, setConnected] = useState(false);
const [userTranscript, setUserTranscript] = useState("");
const [assistantTranscript, setAssistantTranscript] = useState("");
const [assistantIsSpeaking, setAssistantIsSpeaking] = useState(false);
const [volumeLevel, setVolumeLevel] = useState(0);
const { showPublicKeyInvalidMessage, setShowPublicKeyInvalidMessage } = usePublicKeyInvalid();
// hook into Vapi events
useEffect(() => {
vapi.on("call-start", () => {
setConnecting(false);
setConnected(true);
setShowPublicKeyInvalidMessage(false);
});
vapi.on("call-end", () => {
setConnecting(false);
setConnected(false);
setShowPublicKeyInvalidMessage(false);
});
vapi.on("speech-start", () => {
setAssistantIsSpeaking(true);
});
vapi.on("speech-end", () => {
setAssistantIsSpeaking(false);
});
vapi.on("volume-level", (level) => {
setVolumeLevel(level);
});
vapi.on('message', (message) => {
console.log("Received message:", message);
if (message.role === 'user') {
setUserTranscript((prev) => prev + message.transcript + "\n");
} else if (message.role === 'assistant') {
setAssistantTranscript((prev) => prev + message.transcript + "\n");
}
});
vapi.on("error", (error) => {
console.error(error);
setConnecting(false);
if (isPublicKeyMissingError({ vapiError: error })) {
setShowPublicKeyInvalidMessage(true);
}
});
// we only want this to fire on mount
// eslint-disable-next-line react-hooks/exhaustive-deps
}, []);
// call start handler
const startCallInline = () => {
setConnecting(true);
vapi.start("5983bd53-7f57-4364-a161-ca212bf2c55d");
};
const endCall = () => {
console.log(userTranscript)
console.log(assistantTranscript)
vapi.stop();
};
return (
Is your Vapi Public Key missing? (recheck your code)
);
};
const ReturnToDocsLink = () => {
return (
Trends
);
};
export default App;
```
### frontend/backend/app.py
```python
import os
import json
from flask import Flask, request, Response
from groq import Groq
from dotenv import load_dotenv
from calendar_functions import llm_edit_calendar
from mail_functions import llm_edit_gmail
from datetime import datetime
from chroma import add_chat_history, search_chat_history, add_calendar_history, add_mail_history, search_calendar_history, search_mail_history
# Initialize the Flask app
app = Flask(__name__)
# Load environment variables
load_dotenv()
# Initialize the Groq client
client = Groq(api_key=os.environ.get("GROQ_API_KEY"))
ROUTING_MODEL = "llama3-70b-8192"
TOOL_USE_MODEL = "llama3-groq-70b-8192-tool-use-preview"
GENERAL_MODEL = "llama3-70b-8192"
def basic_query(messages):
print(messages)
user_message = " ".join([messages[-1]['content'] for msg in messages if msg['role'] == 'user'])
response = client.chat.completions.create(
model=GENERAL_MODEL,
messages=[
{"role": "system", "content": f"You are my female therapist and friend named SereneAI. THIS IS VERY IMPORTANT, the current datetime is {datetime.now()}. Please ask insightful and meaningful questions to help guide the user. Emphasize to the user you have access to manage their calendar and email. Do not make up calendar or email information that is not provided. Suggest calendar or email actions as you see fit as it relates to the conversation, your role as a therapist, and the user's needs."},
{"role": "user", "content": f"User: {user_message}. This is the relevant past conversation history: {search_chat_history(user_message)}. This is the relevant past calendar history: {search_calendar_history(user_message)}. This is the relevant past email history: {search_mail_history(user_message)}"},
],
)
result = response.choices[0].message.content.strip()
add_chat_history(result)
return result
def route_query(messages):
routing_prompt = messages[-1]['content']
routing_messages = [
{"role": "system", "content": "You are a llm routing assistant. Determine if tools are needed based on the user query. If a calendar assistant is needed, return Calendar. If a mail assistant is needed, return Mail."},
{"role": "user", "content": routing_prompt},
]
response = client.chat.completions.create(
model=ROUTING_MODEL,
messages=routing_messages,
)
routing_decision = response.choices[0].message.content.strip()
if "Calendar" in routing_decision:
print("Calendar Called")
result = llm_edit_calendar(routing_prompt)
if result != None:
add_calendar_history(result)
elif "Mail" in routing_decision:
print("Mail Called")
result = llm_edit_gmail(routing_prompt)
if result != None:
add_mail_history(result)
else:
result = basic_query(messages)
return result
def generate_therapist_response(messages):
try:
# Process the messages and generate a response
print(messages)
response = route_query(messages)
# Create a generator to stream the response
def event_stream():
try:
# Split the response into words and yield them as separate chunks
words = response.split()
for word in words:
chunk = {
"choices": [
{
"delta": {"content": word + " "},
"finish_reason": None,
"index": 0
}
],
"created": int(datetime.now().timestamp()),
"model": GENERAL_MODEL,
"object": "chat.completion.chunk"
}
yield f"data: {json.dumps(chunk)}\n\n"
# Send the final chunk
final_chunk = {
"choices": [
{
"delta": {},
"finish_reason": "stop",
"index": 0
}
],
"created": int(datetime.now().timestamp()),
"model": GENERAL_MODEL,
"object": "chat.completion.chunk"
}
yield f"data: {json.dumps(final_chunk)}\n\n"
yield "data: [DONE]\n\n"
except Exception as e:
yield f"data: {json.dumps({'error': str(e)})}\n\n"
return event_stream()
except Exception as e:
return f"Error: {str(e)}"
@app.route("/chat/completions", methods=["POST"])
def generate_response():
data = request.get_json()
try:
messages = data.get("messages", [])
# Generate the therapist response stream
response_stream = generate_therapist_response(messages)
print(response_stream)
# Return the streamed response as text/event-stream
return Response(response_stream, mimetype="text/event-stream")
except Exception as e:
return Response(f"data: {json.dumps({'error': str(e)})}\n\n", mimetype="text/event-stream")
if __name__ == "__main__":
app.run(debug=True, port=5000) # Run the Flask app
```
### frontend/src/utils/index.js
```javascript
export const isPublicKeyMissingError = ({ vapiError }) => {
return !!vapiError && vapiError.error.statusCode === 403 && vapiError.error.error === "Forbidden";
};
```
### givetopemotions.py
```python
import asyncio
from hume import AsyncHumeClient
from hume.expression_measurement.stream import Config
from hume.expression_measurement.stream.socket_client import StreamConnectOptions
from hume.expression_measurement.stream.types import StreamLanguage
samples = [
"hello there i am feeling a little tiny bit sad",
"yes no yes no"
]
async def main():
client = AsyncHumeClient(api_key="3v4DChnAqtBLBnQzgD8VROM8nUmCLhhVYfw5V5EWXVrlP1ub")
model_config = Config(language=StreamLanguage())
stream_options = StreamConnectOptions(config=model_config)
async with client.expression_measurement.stream.connect(options=stream_options) as socket:
for sample in samples:
result = await socket.send_text(sample)
print(result.language.predictions[0].emotions)
if __name__ == "__main__":
asyncio.run(main())
```
### calltollm.py
```python
import os
from groq import Groq
from dotenv import load_dotenv
# Initialize the Groq client
load_dotenv()
client = Groq(
api_key=os.environ.get("GROQ_API_KEY"),
)
def generate_therapist_response(text, emotions):
# Define the therapist-like prompt structure
prompt = f"Generate a response a therapist would give to '{text}', considering the user is feeling the following emotions with their respective magnitudes: '{emotions}'"
try:
# Call the Groq API using a suitable model (replace 'llama3-8b-8192' with your preferred model)
response = client.chat.completions.create(
model="llama3-8b-8192", # You can adjust the model depending on the performance you need
messages=[
{"role": "system", "content": "You are a helpful and empathetic therapist."},
{"role": "user", "content": prompt},
],
max_tokens=200 # Adjust based on how long you want the response
)
# Extract the generated response
return response.choices[0].message.content
except Exception as e:
return f"Error: {str(e)}"
if __name__ == "__main__":
therapist_response = generate_therapist_response(text, top_n_emotions)
print(therapist_response)
```
### frontend/src/setupTests.js
```javascript
// jest-dom adds custom jest matchers for asserting on DOM nodes.
// allows you to do things like:
// expect(element).toHaveTextContent(/react/i)
// learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom';
```
[13 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]