# Project export: Wander This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: Dementia is a brutal disease that strips people of independence and breaks down relationships with family. We want to empower caregivers to live full lives while supporting those they love. - Devpost: https://devpost.com/software/forgetable - GitHub: https://github.com/JonesMays/wander - Demo: https://youtu.be/Kpq2egWsZLo - Video: https://www.youtube.com/embed/jJuIbd9ZXTM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Birdbh (16 commits), Jones Mays II (4 commits) ## Devpost submission (written by the team) ### Inspiration Three years ago, Jone's grandmother, Kathy, was diagnosed with dementia. Since then, her symptoms have progressively worsened. She has begun wandering away from home and family gatherings more frequently, and her ability to sleep well at night has significantly declined. Witnessing these challenges inspired us to focus on the often-overlooked struggles of caregivers—typically family members—who dedicate themselves to supporting loved ones with dementia. Our mission is to empower these caregivers by providing them with the tools and resources they need to deliver compassionate, high-quality care while still maintaining their own well-being and fulfilling their personal lives. Together, we can make a difference in the lives of both caregivers and those they care for. ### What it does The Wander Bracelet is a cutting-edge cyber-physical system designed to enhance the safety and well-being of individuals with dementia while providing peace of mind to their caregivers. This innovative device offers a suite of powerful features: it tracks the wearer’s GPS location in real-time, monitors their movements, and alerts caregivers instantly if the wearer leaves a designated safe area, such as their home. Leveraging advanced AI technology, the bracelet can gently guide wearers back home by creating personalized audio messages—based on 1-minute recordings by a familiar voice—to help reorient them. The Wander Bracelet also integrates seamlessly with SMS, enabling caregivers to receive location updates, activity reports, and notifications without needing to download additional apps. For added safety, the bracelet is equipped with lights and vibration motors to alert both the wearer and those nearby in case of wandering incidents. Additionally, the SMS text agent provides caregivers with access to advanced analytics, offering valuable insights into the wearer’s patterns and behaviors. With its combination of real-time tracking, AI-driven guidance, and user-friendly communication tools, the Wander Bracelet empowers caregivers to provide compassionate care while ensuring the safety and independence of their loved ones. Iteration and Feedback After researching the problem, we found that 66% of Alzheimer’s patients experience wandering at some point. This period of restlessness and wandering is most severe during sunset, a phenomenon known as sundowning. From there, we spoke with and surveyed over 20 caregivers who had family members with the disease to determine if they faced the same challenges as Jones. Based on our research and the pain points highlighted by potential users, we created low-fidelity prototypes. Finally, we built our final prototype and observed people interacting with it to evaluate its effectiveness. Core Components The Wander Bracelet system is a sophisticated integration of two core components: the physical hardware and the cyber software. Hardware The physical system is a robust hardware platform developed in C++ and powered by an ESP32 microcontroller. It features a suite of sensors and actuators, including vibrational motors, waterproof LED strips, and a GPS module, all meticulously soldered and assembled into a cohesive, wearable ecosystem. This hardware foundation ensures reliable performance and durability, making it ideal for everyday use by elderly patients. Software On the cyber side, the system leverages advanced AI agents to manage data flow and automate incident response. By analyzing GPS movement data and user inputs, the Wander Bracelet integrates cutting-edge platforms like OpenAI, ElevenLabs, and Twilio to intelligently guide and redirect patients who may wander. For example, it can use AI-generated audio prompts—personalized by loved ones—to gently steer individuals back to safety. Cyber-Physical Layer Connecting these two layers is a seamless communication framework built on MQTT and Firebase, enabling real-time data transmission, collection, and response between the device and the cloud. This ensures that caregivers receive instant updates and actionable insights, while the system autonomously manages critical situations. Together, the Wander Bracelet’s hardware and software create a powerful, user-friendly solution that enhances safety, provides peace of mind, and empowers caregivers to deliver exceptional care. ### Challenges we ran into During the development of our project, we encountered two primary challenges that tested our creativity and focus. Hardware Limitations and Time Constraints One of the biggest hurdles we faced was the size of available technology and the time constraints of the hackathon. Working with bulky, modular components limited the form factor of our product, making it less sleek and user-friendly than we envisioned. However, throughout the design cycle, we gathered valuable feedback from TreeHacks participants—including fellow hackers, mentors, and sponsors—which helped us iterate over several wearable versions to determine the best form factor. These insights will guide future iterations to create a more compact and wearable design. Scope Management The problem we aimed to solve is complex and impacts millions of people worldwide. Narrowing down the scope to create a meaningful and impactful project within the hackathon timeframe was a significant challenge. We had to carefully prioritize which features to implement, ensuring that our solution remained focused and achievable while still addressing a critical aspect of the problem. This experience taught us the importance of balancing ambition with practicality to deliver a functional and impactful prototype. These challenges not only pushed us to innovate but also reinforced the value of user feedback and strategic planning in creating solutions that truly make a difference. ### Accomplishments we're proud of We are thrilled to showcase the groundbreaking AI integration we’ve achieved with Eleven Labs, which has enabled us to implement a suite of innovative features, including Automatic Voice Cloning, Conversational AI, and AI Agents. These cutting-edge tools have allowed us to create a project that sits at the exciting intersection of AI and elder care, offering a truly transformative approach to supporting individuals with dementia and their caregivers. Beyond voice-based AI, our application leverages powerful machine learning algorithms to analyze and translate sleep and activity data. By utilizing scikit-learn, we’ve developed advanced models trained on sleep pattern datasets from Stanford University, enabling us to uncover meaningful insights into dementia progression and care patterns. This complex data is then distilled into an intuitive, user-friendly interface, empowering caregivers with actionable insights to improve the quality of care. Our work represents a significant step forward in combining AI innovation with compassionate care, and we’re excited about the potential to make a lasting impact on the lives of millions. ### What we learned One of our greatest takeaways was the immense value of a cross-functional team. Despite our diverse backgrounds and areas of expertise, each team member’s willingness to contribute and unique knowledge base enabled us to tackle the wide range of challenges that come with developing a cyber-physical product. From technical complexities to business viability and design excellence, our team’s multi-disciplinary approach was instrumental in creating a product that is not only functional but also innovative and user-centric. This collaboration allowed us to deliver a solution with impactful features and a seamless integration of hardware, software, and AI—proving that diverse perspectives are key to building something truly extraordinary. ### What's next The Wander team brings together a diverse mix of expertise in business strategy, design consulting, and industrial systems engineering. With our combined experience and shared passion for innovation, we are confident that the Wander system has the potential to extend far beyond Stanford and create a meaningful impact on aging populations worldwide. Our unique blend of skills and dedication positions us to deliver a solution that not only addresses the challenges of dementia care today but also evolves to meet the needs of tomorrow’s global aging community. ## README (from the GitHub repository) # 🌳 TreeHacks Submission: Wander ![Banner Image](wanderrender.png) --- ## 🚀 **About the Project** Wander is a smart wearable system designed to enhance the safety and independence of elderly individuals with dementia. Built during TreeHacks 2023, our solution addresses the growing challenges of dementia care by empowering caregivers and ensuring patient safety. --- ## 💡 **Inspiration** With over 55 million people worldwide living with dementia, caregivers often face immense challenges in ensuring the safety and well-being of their loved ones. Inspired by Jones grandmother, we created Wander to provide a seamless, AI-driven solution that reduces caregiver stress and enhances patient safety. --- ## 🛠️ **How It Works** - **Real-Time GPS Tracking**: The Wander Bracelet tracks the wearer’s location in real-time using an ESP32 microcontroller and GT-U7 GPS module - **AI-Powered Guidance**: Leveraging OpenAI and ElevenLabs, the system plays personalized audio prompts to gently guide users back to safety. - **Caregiver Notifications**: Integrated with Twilio, caregivers receive instant SMS alerts and activity reports. - **Seamless Connectivity**: Data flows between the hardware and software layers via MQTT and Firebase, ensuring real-time updates and analytics. --- ## 🎥 **Demo Video** [![Demo Video](https://img.youtube.com/vi/jJuIbd9ZXTM/0.jpg)](https://youtu.be/jJuIbd9ZXTM) *Click the image above to watch the demo!* --- ## 🛠️ **Tech Stack** ![C++](https://img.shields.io/badge/C++-00599C?style=for-the-badge&logo=c%2B%2B&logoColor=white) ![ESP32](https://img.shields.io/badge/ESP32-000000?style=for-the-badge&logo=espressif&logoColor=white) ![Firebase](https://img.shields.io/badge/Firebase-FFCA28?style=for-the-badge&logo=firebase&logoColor=black) ![OpenAI](https://img.shields.io/badge/OpenAI-412991?style=for-the-badge&logo=openai&logoColor=white) ![Twilio](https://img.shields.io/badge/Twilio-F22F46?style=for-the-badge&logo=twilio&logoColor=white) --- ## 🧠 **Challenges We Faced** - **Hardware-Software Integration**: Connecting the ESP32 microcontroller with Firebase and MQTT required extensive debugging and optimization. - **AI Personalization**: Generating realistic and comforting audio prompts using ElevenLabs posed initial challenges in achieving the right tone and clarity. --- ## � **Accomplishments We’re Proud Of** - Successfully built a functional hardware prototype in under 36 hours. - Integrated three different AI platforms (OpenAI, ElevenLabs, Twilio) to create a seamless user experience. - Designed a user-friendly interface for caregivers to monitor their loved ones effortlessly. --- ## 📚 **What We Learned** - The importance of iterative design and rapid prototyping. - How to effectively integrate hardware and software systems in a limited timeframe. - The power of AI in creating personalized, empathetic solutions. --- ## 🚀 **What’s Next for Wander** - Expand the system to include fall detection and health monitoring features. - Conduct user testing with caregivers and dementia patients to refine the design. - Explore partnerships with healthcare organizations to bring the solution to market. --- ## 👥 **Team** | Name | Role | Fun Fact | |-----------------|--------------------------|-----------------------------------| | Heanan Bird | Hardware Engineer | Loves building robots in free time | | Jones Mays | AI/ML Specialist | Once coded for 48 hours straight | | Anna Wang | UX Designer | Avid painter and illustrator | --- ## 🙏 **Acknowledgments** - A huge shoutout to the **TreeHacks organizers** for putting together an incredible event! - Special thanks to ElevenLabs for their guidance and support. - We used OpenAI to bring our project to life. --- ## 📄 **License** This project is licensed under the MIT License --- ✨ **Thanks for checking out our project! We’d love to hear your feedback.** ## Detected evidence (automated analysis) Indexed codebase: 7 recognized source files, 34 KB. - Python (language) — detected in the code - Firebase (technology) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (11 of 11) ``` arduino code/gps.c++ Chat.py Config.py Eleven.py Functions.py jade_instance_key.pem ml.py README.md sketch_feb15a/h.ino sketch_feb15a/sketch_feb15a.ino SMS.py ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - Update README.md - Add files via upload - Add files via upload - Rename 20250215_133720.jpg to Team_Photo.jpg - Add files via upload - Update README.md - Update README.md - Add files via upload - Update README.md - Create README.md - gps addition - ml addition - Add files via upload - Add files via upload ## Key source files (fetched from GitHub, selected and truncated for size) ### Config.py ```python import os from openai import OpenAI from twilio.rest import Client import datetime import firebase_admin from firebase_admin import credentials, storage, firestore from elevenlabs import ElevenLabs # This files contains all the import key stuff # Twilio credentials (make sure these are set as environment variables for security) account_sid = os.environ.get('TWILIO_ACCOUNT_SID', 'xxxxxxxxxxxx') auth_token = os.environ.get('TWILIO_AUTH_TOKEN', 'xxxxxxxxxx') twilio_client = Client(account_sid, auth_token) # OpenAI API and assistant key openAI_client = OpenAI(api_key="xxxx") Assistant_ID = "xxxx" #Eleven labs api key eleven_labs = ElevenLabs( api_key="xxxxxxxxx", ) # Firebase credentials for using the platform #cred = credentials.Certificate('/home/ec2-user/jade_AI/jadeai-77973-firebase-adminsdk-jlw5m-59e0f5a501.json') cred = credentials.Certificate('/Users/jonesmaysii/Desktop/XcodeP/Opal_AI/jadeai-77973-firebase-adminsdk-jlw5m-59e0f5a501.json') firebase_admin.initialize_app(cred, { 'storageBucket': 'jadeai-77973' # Ensure this is your correct project ID }) db = firestore.client() # Firebase database structure chat_history = { 'participants': ['user1'], 'messages': [ {'role': 'user', 'content': 'Hello!', 'timestamp': datetime.datetime.now().isoformat()}, ], 'fileIDs': [], 'hasSentShareMessage': False, 'hasSentReminder': False, 'hasSubscribed': True, 'createdAt': datetime.datetime.now().isoformat(), 'textStop': False, 'voiceID': '' } # url to contact card for the number vcard_url = 'https://firebasestorage.googleapis.com/v0/b/jadeai-77973/o/Jade%20AI.vcf?alt=media' ``` ### ml.py ```python import numpy as np import plotly.graph_objects as go from sklearn.preprocessing import PolynomialFeatures from sklearn.linear_model import LinearRegression '''HOW TO USE''' # 1. Import 'from ml.py import plot_sleep_scores' # 2. Call 'plot_sleep_scores(sleep_scores)' with a list of sleep scores # 2.1 The list should be like this sleep_scores = [0.2, 0.5, 0.7, 0.4, 0.6, 0.8, 0.3] # 3. The function will generate a plot and save it as 'sleep_scores.png' def describe_sleep_score(score): if score < 0.3: return "Poor" elif score < 0.6: return "Average" else: return "Good" def calculate_sleep_trend(sleep_scores): x = np.arange(len(sleep_scores)) y = np.array(sleep_scores) # Fit a polynomial regression line to determine the trend poly = PolynomialFeatures(degree=2) X_poly = poly.fit_transform(x.reshape(-1, 1)) model = LinearRegression() model.fit(X_poly, y) y_poly_pred = model.predict(X_poly) # Calculate the derivative of the polynomial fit to determine the trend trend = np.polyder(np.polyfit(x, y_poly_pred, 2)) if trend[0] > 0.01: return "Sleep is getting better", trend[0] elif trend[0] < -0.01: return "Sleep quality is declining", trend[0] else: return "Sleep is staying constant", trend[0] def plot_sleep_scores(sleep_scores): # Generate x values (e.g., days) x = np.arange(len(sleep_scores)) y = np.array(sleep_scores) # Fit a polynomial regression line poly = PolynomialFeatures(degree=2) X_poly = poly.fit_transform(x.reshape(-1, 1)) model = LinearRegression() model.fit(X_poly, y) y_poly_pred = model.predict(X_poly) # Calculate sleep trend trend_description, trend_rate = calculate_sleep_trend(sleep_scores) # Create a plot fig = go.Figure() fig.add_trace(go.Scatter(x=x, y=y, mode='markers', name='Actual Scores')) fig.add_trace(go.Scatter(x=x, y=y_poly_pred, mode='lines', name='Polynomial Fit')) # Add descriptions for each score fig.update_layout( title=f'Sleep Scores: {trend_description} (Rate: {trend_rate:.2f})', xaxis_title='Day', yaxis_title='Sleep Score', xaxis=dict( tickmode='array', tickvals=np.arange(7), ticktext=['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'] ) ) fig.write_image('sleep_scores.png') fig.show() ``` ### Chat.py ```python import datetime import io from firebase_admin import credentials, storage, firestore import mimetypes from openai import OpenAI from io import BytesIO import Config import datetime # Function to upload media to Firebase Storage def upload_file(media_content, media_url): content_type, _ = mimetypes.guess_type(media_url) if content_type is None: content_type = 'application/octet-stream' blob = Config.bucket.blob(f'userFiles/{media_url.split("/")[-1]}') # Ensure the file name is extracted correctly blob.upload_from_string(media_content, content_type=content_type) blob.make_public() print("Uploaded file to firebase") def save_chat(chat_id, chat_data): try: for message in chat_data['messages']: if isinstance(message['timestamp'], datetime.datetime): message['timestamp'] = message['timestamp'].isoformat() if isinstance(chat_data['createdAt'], datetime.datetime): chat_data['createdAt'] = chat_data['createdAt'].isoformat() Config.db.collection('Tree_Hacks').document(chat_id).set(chat_data) print('Chat successfully written!') except Exception as e: print(f'Error writing thread: {e}') # This ensures that the database has all the fields def ensure_chat_fields(chat_data): if 'hasSentShareMessage' not in chat_data: chat_data['hasSentShareMessage'] = False if 'hasSentReminder' not in chat_data: chat_data['hasSentReminder'] = False if 'hasSubscribed' not in chat_data: chat_data['hasSubscribed'] = True if 'fileIDs' not in chat_data: chat_data['fileIDs'] = [] if 'messages' not in chat_data: chat_data['messages'] = [] if 'voiceID' not in chat_data: chat_data['voiceID'] = '' if 'participant' not in chat_data: chat_data['participant'] = [] if 'createdAt' not in chat_data or not chat_data['createdAt']: chat_data['createdAt'] = datetime.datetime.now().isoformat() return chat_data def get_chat(chat_id): try: chat_ref = Config.db.collection('Tree_Hacks').document(chat_id) chat = chat_ref.get() if chat.exists: chat_data = chat.to_dict() Config.chat_history = ensure_chat_fields(chat_data) print("The data was loaded") return True else: print('No such chat!') return False except Exception as e: print(f'Error getting thread: {e}') return False ``` ### Eleven.py ```python # This file is all about the conversation with my macs speakers import asyncio import json import sounddevice as sd import numpy as np import websockets import wave import os from elevenlabs import ElevenLabs from elevenlabs.conversational_ai.conversation import Conversation import Config import requests from requests.auth import HTTPBasicAuth import uuid from pydub import AudioSegment from pydub.playback import play from elevenlabs import VoiceSettings from elevenlabs.client import ElevenLabs # Initialize ElevenLabs client ELEVEN_LABS_API_KEY = "sk_b93da5f80170383c63661ffdbe7c1923d4dc9a4ba53b8d4b" ELEVEN_LABS_AGENT_ID = "lBIULzeSraGV3KoxMsuT" # Audio parameters SAMPLE_RATE = 24000 # ElevenLabs default CHANNELS = 1 DURATION = 5 # seconds for each audio clip FORMAT = np.int16 # 16-bit PCM format async def record_audio(): """Captures microphone input and returns audio as numpy array""" print("🎤 Recording... Speak now!") audio_data = sd.rec( int(DURATION * SAMPLE_RATE), samplerate=SAMPLE_RATE, channels=CHANNELS, dtype=FORMAT ) sd.wait() print("✅ Recording finished!") return audio_data async def play_audio(audio_data): """Plays received AI-generated audio using computer speakers.""" print("🔊 Playing response...") sd.play(audio_data, samplerate=SAMPLE_RATE) sd.wait() print("✅ Playback finished!") async def elevenlabs_conversation(): """Handles real-time conversation using ElevenLabs.""" uri = f"wss://api.elevenlabs.io/v1/conversations/{ELEVEN_LABS_AGENT_ID}/stream" headers = [("Authorization", f"Bearer {ELEVEN_LABS_API_KEY}")] async with websockets.connect(uri, extra_headers=headers) as ws: print("🎙️ Connected to ElevenLabs AI Agent!") # Start conversation with a specific voice ID await ws.send(json.dumps({ "action": "start", "voice_id": Config.chat_history['voiceID'] # 👈 Pass your new voice ID here })) while True: # Record user input from the microphone user_audio = await record_audio() # Send user audio to ElevenLabs await ws.send(json.dumps({ "audio": user_audio.tobytes() })) # Receive and play AI response response = await ws.recv() response_data = json.loads(response) if "audio" in response_data: ai_audio = np.frombuffer(response_data["audio"], dtype=FORMAT) await play_audio(ai_audio) elif "status" in response_data and response_data["status"] == "disconnected": print("🔴 Disconnected from ElevenLabs AI") break #This function cleans audio def clean_audio_with_elevenlabs(mp3_file_path): """Removes background noise from the audio file using ElevenLabs' Audio Isolation API.""" url = "https://api.elevenlabs.io/v1/audio-isolation" with open(mp3_file_path, "rb") as audio_file: files = {"audio": audio_file} headers = {"xi-api-key": ELEVEN_LABS_API_KEY} response = requests.post(url, files=files, headers=headers) if response.status_code == 200: print("✅ Audio cleaned successfully!") cleaned_file_path = f"{mp3_file_path}_cleaned.mp3" # Save the cleaned audio with open(cleaned_file_path, "wb") as f: f.write(response.content) return cleaned_file_path # Return new cleaned file path else: print(f"❌ Error cleaning audio: {response.status_code} - {response.text}") return None # Return None if cleaning fails def text_to_speech_and_play(text: str): # Call ElevenLabs text-to-speech API response = Config.eleven_labs.text_to_speech.convert( voice_id=Config.chat_history['voiceID'], # Example voice ID output_format="mp3_22050_32", text=text, model_id="eleven_turbo_v2_5", # Low latency model voice_settings=VoiceSettings( stability=0.7, # More consistent voice similarity_boost=1.0, style=0.2, # Slight variation but not too much use_speaker_boost=False, ), ) # Save to a temporary file filename = f"{uuid.uuid4()}.mp3" with open(filename, "wb") as f: for chunk in response: if chunk: f.write(chunk) print(f"✅ Audio saved as {filename}") # Load and play the audio audio = AudioSegment.from_mp3(filename) play(audio) # Optional: Delete the file after playing os.remove(filename) ``` ### Functions.py ```python # This file is all about function calling with OpenAI import paho.mqtt.client as mqtt import asyncio import Eleven # Assuming this contains your ElevenLabs AI functions from elevenlabs import Voice, VoiceSettings, text_to_speech, stream import sounddevice as sd import numpy as np import Config import sounddevice as sd import numpy as np import tempfile import os import uuid from elevenlabs import VoiceSettings from elevenlabs.client import ElevenLabs import time import datetime from openai import OpenAI # MQTT Broker Details (Use HiveMQ for free) MQTT_BROKER = "broker.hivemq.com" # Public MQTT Broker MQTT_PORT = 1883 MQTT_TOPIC = "wander/commands" # This will detect paitent speech def detect_patient_speech(): """Detects if the patient is speaking. Returns True if speech is detected.""" print("🎤 Detecting patient speech...") audio_data = sd.rec(int(5 * 24000), samplerate=24000, channels=1, dtype=np.int16) # Record for 5 sec sd.wait() # Check if there is significant sound if np.max(np.abs(audio_data)) > 500: # Adjust threshold as needed return True return False # This will listen and transcipt the patinet def listen_to_patient(): """Records the patient's response and transcribes it to text.""" print("🎧 Listening for patient response...") audio_data = sd.rec(int(5 * 24000), samplerate=24000, channels=1, dtype=np.int16) sd.wait() # Convert recorded audio to text (mock function, replace with actual speech-to-text) transcribed_text = "Patient's spoken words" # Replace with actual transcription logic print(f"📝 Patient said: {transcribed_text}") return transcribed_text # get text def get_voice_reply(incoming_message): """Processes an incoming message and generates a response.""" Config.chat_history['messages'].append( {'role': 'user', 'content': incoming_message, 'timestamp': datetime.datetime.now().isoformat()} ) formatted_messages = [ {"role": message['role'], "content": message['content']} for message in Config.chat_history['messages'][-10:] ] # Create a thread with the incoming message thread = Config.openAI_client.beta.threads.create( messages=formatted_messages ) print(f"👉 Incoming Message: {incoming_message}") # Submit the thread to the assistant (as a new run) without function calls run = Config.openAI_client.beta.threads.runs.create( thread_id=thread.id, assistant_id=Config.Assistant_ID ) print(f"👉 Run Created: {run.id}") # Wait for run to complete while run.status != "completed": run = Config.openAI_client.beta.threads.runs.retrieve( thread_id=thread.id, run_id=run.id ) print(f"🏃 Run Status: {run.status}") time.sleep(1) print(f"🏁 Run Completed!") # Get the latest message from the thread message_response = Config.openAI_client.beta.threads.messages.list(thread_id=thread.id) messages = message_response.data # Get the content of the latest message latest_message = messages[0] print(f"💬 Response: {latest_message.content[0].text.value}") # Save the message to chat history Config.chat_history['messages'].append( {'role': 'assistant', 'content': latest_message.content[0].text.value, 'timestamp': datetime.datetime.now().isoformat()} ) print(Config.chat_history) return latest_message.content[0].text.value # Connect to MQTT broker def on_connect(client, userdata, flags, rc): if rc == 0: print("✅ Connected to MQTT Broker!") else: print(f"❌ Failed to connect, return code {rc}") mqtt_client = mqtt.Client() mqtt_client.connect(MQTT_BROKER, MQTT_PORT, 60) mqtt_client.on_connect = on_connect mqtt_client.loop_start() #If the user wanders out of the bounds tools_user_wandered_out = { "type": "function", "function": { "name": "user_wandered_out", "description": "This function should be called if the user wandered out or just types WWW", "parameters": {} } } import threading def user_wandered_out(): global reminder_active, reminder_thread reminder_active = True # Start reminders """Handles the case when the patient wanders out, sending periodic reminders and listening for responses.""" command = "alert" print(f"📡 Publishing MQTT message: {command}") # ✅ Debug log mqtt_client.publish(MQTT_TOPIC, command, retain=True) # ✅ Publish alert status def reminder_loop(): """Continuously sends reminders every minute unless interrupted by user speech.""" while reminder_active: # Generate AI response response_text = get_voice_reply("Give me a message to encourage my patient to come back to their home.") # Generate speech and play it Eleven.text_to_speech_and_play(response_text) # Check if the patient is speaking if detect_patient_speech(): print("🎤 Patient started speaking... Listening mode activated.") patient_response = False if patient_response: # Respond to the patient's speech response_text = get_voice_reply(patient_response) filename = Eleven.text_to_speech_and_play(response_text) if filename and os.path.exists(filename): audio = AudioSegment.from_mp3(filename) play(audio) os.remove(filename) # Clean up after playing else: print("❌ Error: Could not play the generated response audio file.") print("✅ Resuming reminders after conversation.") print("⏳ Waiting for 1 minute before next reminder...") time.sleep(60) # Wait 1 minute before sending another reminder # Start the reminder [truncated — 1547 more characters] ``` ### SMS.py ```python import threading from flask import Flask, request, render_template, jsonify import requests from twilio.twiml.messaging_response import MessagingResponse import time from openai import OpenAI import time import datetime from twilio.rest import Client import datetime import json import os import re import Chat import Config import Functions import paho.mqtt.client as mqtt from elevenlabs import ElevenLabs from pydub import AudioSegment from requests.auth import HTTPBasicAuth import Eleven def get_wander_reply(incoming_message): Config.chat_history['messages'].append( {'role': 'user', 'content': incoming_message, 'timestamp': datetime.datetime.now().isoformat()} ) formatted_messages = [ {"role": message['role'], "content": message['content']} for message in Config.chat_history['messages'][-10:] ] # Create a thread with the incoming message thread = Config.openAI_client.beta.threads.create( messages=formatted_messages ) print(f"👉 Incoming Message: {incoming_message}") # Submit the thread to the assistant (as a new run) run = Config.openAI_client.beta.threads.runs.create( thread_id=thread.id, assistant_id=Config.Assistant_ID, tools=[Functions.tools_user_wandered_out, Functions.tools_user_wandered_back,Functions.tools_user_data, {"type": "file_search"}] ) print(f"👉 Run Created: {run.id}") # Wait for run to complete while run.status != "completed": run = Config.openAI_client.beta.threads.runs.retrieve( thread_id=thread.id, run_id=run.id ) print(f"🏃 Run Status: {run.status}") time.sleep(1) if run.status == "requires_action": print("Function Calling") required_actions = run.required_action.submit_tool_outputs.model_dump() print(required_actions) tool_outputs = [] for action in required_actions["tool_calls"]: func_name = action['function']['name'] arguments = json.loads(action['function']['arguments']) # This is where they give me their location, tell them something, call 911, or let me speak with them if func_name == "user_wandered_out": output = Functions.user_wandered_out( ) tool_outputs.append({ "tool_call_id": action['id'], "output": output }) elif func_name == "user_wandered_back": output = Functions.user_wandered_back( ) tool_outputs.append({ "tool_call_id": action['id'], "output": output }) elif func_name == "user_data": output = Functions.user_data( ) tool_outputs.append({ "tool_call_id": action['id'], "output": output }) else: raise ValueError(f"Unknown function: {func_name}") print("Submitting outputs back to Wander...") Config.openAI_client.beta.threads.runs.submit_tool_outputs( thread_id=thread.id, run_id=run.id, tool_outputs=tool_outputs ) print(f"🏁 Run Completed!") # Get the latest message from the thread message_response = Config.openAI_client.beta.threads.messages.list(thread_id=thread.id) messages = message_response.data # Get the content of the latest message latest_message = messages[0] print(f"💬 Response: {latest_message.content[0].text.value}") # This saves the messga to chat history Config.chat_history['messages'].append({'role': 'assistant', 'content': latest_message.content[0].text.value, 'timestamp': datetime.datetime.now().isoformat()}) print(Config.chat_history) return latest_message.content[0].text.value def get_sms_reply(incoming_message, from_number, to_number): # This tells me what is being printed out print(incoming_message) print(from_number) print(to_number) if incoming_message and from_number and to_number: if Chat.get_chat(from_number): if "Wander STOP" in incoming_message: Config.chat_history['hasSubscribed'] = False Chat.save_chat(from_number, Config.chat_history) resp = MessagingResponse() resp.message("Aww, we'll miss you! If you ever want to keep a loving eye on your dear one again, just send us a text. We're always here for you! 😊") return str(resp) else: latest_message = get_wander_reply(incoming_message) # Respond with the latest message from OpenAI and save data Chat.save_chat(from_number, Config.chat_history) Config.twilio_client.messages.create( body=re.sub(r'【.*?】', '', latest_message), from_=to_number, to=from_number ) return latest_message else: #New wander user yayy!!!!!! Config.chat_history['participants'] = [] Config.chat_history['messages'] = [] Config.chat_history['createdAt'] = datetime.datetime.now().isoformat() welcome_message = ( "Welcome! 🎉 Your Wander AI bracelet is now connected. Here's how we help:\n" "1. Geofencing: Set safe zones.\n" "2. Alerts: Get notified if boundaries are crossed.\n" "3. Voice Guidance: Use your [truncated — 5953 more characters] ```