# Project export: Hoshi AI
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2024
- Tagline: Sidekick every elderly deserves
- Devpost: https://devpost.com/software/hoshi-ai
- GitHub: https://github.com/aliyanishfaq/Treehacks2024
- Demo: https://docs.google.com/presentation/d/1U00SGsm1kaYc8xD_xZbHbkQ7Yd9E9v1rYW7lE1gF58w/edit?usp=sharing
- Video: https://www.youtube.com/embed/playlist?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — Aliyan Ishfaq (7 commits), Aryan Siddiqui (3 commits)
## Devpost submission (written by the team)
### Inspiration
According to the US Census Bureau, 14 million elderly individuals live alone – sometimes because they have nowhere else to go or that they do not want to burden their family. At this stage in life, it is common for them to encounter various health-related issues. They often struggle to manage their health, including difficulties accessing timely, personalized medical advice. Furthermore, the digital divide exacerbates these issues, as many elderly are not comfortable with existing digital health solutions. Managing your health is one aspect; ensuring emergency assistance is another critical concern. 36 million falls are recorded yearly amongst the elderly and when paired with the rather common independent lifestyle, it can be dangerous. Traditionally, the responsibility to initiate emergency communication falls upon the individual; however, this is unreliable because they may be injured and not even have full information. If only there was a 24/7 companion that could offer not just comfort but real-time, actionable health guidance and emergency support. Imagine a solution that's digitally intuitive, seamlessly integrating into their daily life without overwhelming them, bridging the gap between technology and practical usability.
### What it does
A hardware AI companion designed specifically for the elderly living alone, addressing their unique health management and emergency assistance needs. Emergency support: Our AI companion not only offers advanced fall detection but also actively listens for verbal cues indicating discomfort or severe health issues. We asses the severity of the situation based on the user's verbal expressions. In cases of significant health concerns, the system either calls their designated caregivers or sends them a detailed text message, depending on the assessed urgency. Additionally, when a fall is detected, the system provides essential information about the user's appearance and exact location to facilitate quicker and more efficient identification upon the caregiver's arrival. Emergency support: Our AI companion not only offers advanced fall detection but also actively listens for verbal cues indicating discomfort or severe health issues. We asses the severity of the situation based on the user's verbal expressions. In cases of significant health concerns, the system either calls their designated caregivers or sends them a detailed text message, depending on the assessed urgency. Additionally, when a fall is detected, the system provides essential information about the user's appearance and exact location to facilitate quicker and more efficient identification upon the caregiver's arrival. Real-time, actionable health guidance: By integrating with electronic health records, it contextualizes its guidance with the user's medical history, ensuring that all suggestions are relevant, safe, and tailored to each individual's needs. This approach enables the elderly to make informed decisions about their daily health management. We prompt the user about reminders such as taking pills etc. too. We learn their behavior over time. Real-time, actionable health guidance: By integrating with electronic health records, it contextualizes its guidance with the user's medical history, ensuring that all suggestions are relevant, safe, and tailored to each individual's needs. This approach enables the elderly to make informed decisions about their daily health management. We prompt the user about reminders such as taking pills etc. too. We learn their behavior over time. Digitally intuitive hardware companion: Understanding that ease of use is crucial for technology adoption among the elderly, our hardware companion is designed to be as user-friendly as an "Alexa for old people." It features simple voice commands, tactile buttons for those less comfortable with voice interaction, and a large, easy-to-read display for visual prompts and reminders. Its design is intuitive, eliminating the barriers to digital technology use and making it accessible to everyone, regardless of their technological proficiency. Digitally intuitive hardware companion: Understanding that ease of use is crucial for technology adoption among the elderly, our hardware companion is designed to be as user-friendly as an "Alexa for old people." It features simple voice commands, tactile buttons for those less comfortable with voice interaction, and a large, easy-to-read display for visual prompts and reminders. Its design is intuitive, eliminating the barriers to digital technology use and making it accessible to everyone, regardless of their technological proficiency.
### How we built it
Our project harnesses a multi-faceted approach to enhance the safety and well-being of elderly individuals, integrating advanced technologies and methodologies: OpenAI's AI Agents: These agents facilitate dynamic conversational interactions, enabling tasks such as emergency communication and real-time news updates. Twilio API: Integration allows for seamless initiation of emergency calls to loved ones, ensuring rapid communication during critical situations. Google Maps API: Utilization to pinpoint the exact location of elderly individuals in emergencies, enabling swift assistance. GNews API: Incorporation provides up-to-date news updates on various topics, promoting engagement and connectivity. Whisper AI: Employed for fluid text-to-speech conversion, enhancing communication efficiency. Fusion: Utilized to craft immersive 3D models, enhancing the visual appeal of the user experience. Rag Technique and Langchain: Implemented for efficient analysis of electronic health records, facilitating access to vital medical details. OpenAI's Multi-Modal Capabilities for Fall Detection: By integrating multi-modal data sources such as audio, video, and sensor data, our system can accurately analyze and identify potential falls in real-time. This approach enables robust fall detection, not only based on visual cues but also by analyzing accompanying audio signals or sensor data. Through the synergistic integration of these technologies, our solution provides a comprehensive AI virtual assistant addressing the safety, well-being, and informational needs of elderly individuals, ensuring a holistic approach to their care and support.
### Challenges we ran into
We were limited by the hardware we had available to us. However, we were determined to create prototype as a proof of concept. We wanted to design an encasing that would hold one of our phones, which would serve as the visual and audio input. Using Fusion 360 and the PRL, we 3D printed hardware that houses the iPhone 14 Pro Max with cutouts for camera and mics. We underwent multiple iterations using feedback and ended up with a modern design inspired by stars. One of the most significant challenges we faced was ensuring the reliability of fall detection and emergency communication features. To get accurate results, the approach was to use Llava. However, given the time constraint, we used OpenAI’s model, passing video as frames. Audio streaming took us significant amount of time, especially on iOS. We are very glad that we were able to do this, as this greatly improves the user experience
### Accomplishments we're proud of
One of our proudest accomplishments has been our direct engagement with elderly individuals, actively seeking their feedback throughout the development process. This ensured that our solution resonated with their unique needs and preferences, ultimately leading to a more tailored and effective product. We also made strides in leveraging AI technology to handle diverse tasks, providing users with flexible and adaptive support. Our team experienced breakthrough moments, such as enabling AI-driven emergency calls and seamlessly integrating contextual medical records and location data of elderly individuals Additionally, overcoming obstacles in iOS development to successfully implement audio streaming was a notable achievement.
### What we learned
We were not familiar with real time audio and video streaming. We also learnt different RAG techniques.
### What's next
We are really excited by our mission. We would love to create a self-housed prototype that would be local to ensure full privacy and limited calls to APIs. Through extensive testing, we would be interested to build out the mental health offering of Hoshi. For example, use previous discussions to inform and prompt seniors to reminisce past stories in a meaningful way. We also would look to improve our range of detection to provide more comprehensive emergency services. • Seizures • Mobility Reduction • Prolonged Stillness
## README (from the GitHub repository)
No README available.
## Detected evidence (automated analysis)
Indexed codebase: 17 recognized source files, 58 KB.
- Flask (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- LangChain (technology) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
## Codebase structure (from repository index)
### Files (20 of 20)
```
.gitignore
app.py
components/agent.py
components/EHRVectorization.py
components/location.py
components/news.py
components/phone.py
components/transcribe.py
server/app.py
server/Dockerfile
server/requirements.txt
server/static/js/app.js
server/static/js/hark.bundle.js
server/templates/index.html
server/tools.py
static/js/app.js
static/js/hark.bundle.js
templates/index.html
tools.py
vision.py
```
### Dependencies
- server/requirements.txt: Flask, flask_cors, googlemaps, langchain, langchainhub, openai, python-dotenv, requests, twilio
### Recent commits (newest first)
- delete .env
- dockerfile changes
- dockerfile
- Merge branch 'main' of https://github.com/aliyanishfaq/Treehacks2024
- news.py
- Agent behavior
- EHR vectorization
- Aryan initial changes
- setup for location and phone
- initial setup
## Key source files (fetched from GitHub, selected and truncated for size)
### server/requirements.txt
```
Flask
python-dotenv
openai
googlemaps
twilio
requests
langchain
langchainhub
flask_cors
```
### server/Dockerfile
```
# Use an official Python runtime as a parent image
FROM python:3.9-slim
WORKDIR /python-docker
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
COPY . .
CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]
```
### app.py
```python
from dotenv import load_dotenv
from flask import Flask, render_template, request, jsonify
from openai import OpenAI
import os
from langchain import hub
from langchain_community.chat_models import ChatOpenAI
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_core.messages import AIMessage, HumanMessage
from tools import agent_tools
app = Flask(__name__)
client = OpenAI()
llm = ChatOpenAI(model="gpt-4")
prompt = hub.pull("hwchase17/openai-tools-agent")
agent = create_openai_tools_agent(llm, agent_tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=agent_tools, verbose=True)
chat_history = []
@app.route('/')
def index():
return render_template('index.html')
@app.route('/handle_question', methods=['POST'])
def handle_question():
audio = request.files['audio']
audio_path = "audio.webm"
audio.save(audio_path)
audio_file = open("audio.webm", "rb")
question = client.audio.transcriptions.create(
model="whisper-1",
file=audio_file,
response_format="text"
)
print(f"The question is: {question}")
chat_history.append(HumanMessage(content=question))
agent_input = {
"input": question,
"chat_history": chat_history,
}
response = agent_executor.invoke(agent_input)
output = response['output']
chat_history.append(AIMessage(content=output))
speech_path = "static/speech.mp3"
speech_response = client.audio.speech.create(
model="tts-1",
voice="nova",
input=output
)
speech_response.stream_to_file(speech_path)
return jsonify({'speech_url': speech_path})
if __name__ == '__main__':
app.run(debug=True)
```
### server/app.py
```python
from dotenv import load_dotenv
from flask import Flask, render_template, request, jsonify
from openai import OpenAI
import os
from langchain import hub
from langchain_community.chat_models import ChatOpenAI
from langchain.agents import AgentExecutor, create_openai_tools_agent
from langchain_core.messages import AIMessage, HumanMessage
#from tools import agent_tools
from dotenv import load_dotenv
from langchain.pydantic_v1 import BaseModel, Field
from langchain.tools import BaseTool, StructuredTool, tool
from twilio.rest import Client
import googlemaps
import os
import requests
from flask_cors import CORS, cross_origin
app = Flask(__name__)
CORS(app, supports_credentials=True, resources={r'/*': {'origins': '*'}})
client = OpenAI()
load_dotenv()
account_sid = os.getenv('TWILIO_ACCOUNT_SID')
auth_token = os.getenv('TWILIO_AUTH_TOKEN')
from_number = os.getenv('TWILIO_PHONE_NUMBER')
gmaps_key = os.getenv('GOOGLE_MAPS_KEY')
gmaps = googlemaps.Client(key=gmaps_key)
@tool
def get_location_of_individual_in_emergency():
"""
Returns the address of an individual in an emergency.
Returns:
str: A human-readable address representing the individual's location. If the location cannot be determined
or converted into an address, a default message indicating that no address was found is returned.
"""
url = "https://www.googleapis.com/geolocation/v1/geolocate"
params = {'key': gmaps_key}
headers = {'Content-Type': 'application/json'}
data = {
"homeMobileCountryCode": 310,
"homeMobileNetworkCode": 410,
"radioType": "gsm",
"carrier": "Vodafone",
"considerIp": "true"
}
response = requests.post(url, params=params, headers=headers, json=data)
if response.status_code == 200:
data = response.json()
else:
raise Exception(f"Error in geolocation request: {response.text}")
latitude = float(data['location']['lat'])
longitude = float(data['location']['lng'])
result = gmaps.reverse_geocode((latitude, longitude))
if result:
location = result[0].get('formatted_address', 'No address found')
else:
location = 'No address found'
return location
@tool
def send_emergency_call(message):
"""
Sends an emergency call to with a specified message using Twilio. You must explicitly state the location as context in the call. The phone number will always be the same, so don't worry about it.
Parameters:
message (str): The message to be read during the call.
Returns:
bool: True if the call was successfully initiated, False otherwise.
"""
try:
# Twilio credentials - ideally, fetch from environment variables or a secure config
# Fetch Twilio credentials from environment variables
account_sid = os.getenv('TWILIO_ACCOUNT_SID')
auth_token = os.getenv('TWILIO_AUTH_TOKEN')
from_number = os.getenv('TWILIO_PHONE_NUMBER')
# Initialize the Twilio client
client = Client(account_sid, auth_token)
# Create the call with TwiML to say the emergency message
call = client.calls.create(
twiml=f'{message}',
to="+14156361256",
from_='+18667643449' # Your Twilio number
)
return True
except Exception as e:
return False
agent_tools = []
agent_tools.append(get_location_of_individual_in_emergency)
agent_tools.append(send_emergency_call)
# https://python.langchain.com/docs/modules/agents/tools/custom_tools
llm = ChatOpenAI(model="gpt-4")
prompt = hub.pull("hwchase17/openai-tools-agent")
agent = create_openai_tools_agent(llm, agent_tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=agent_tools, verbose=True)
chat_history = []
@app.route('/')
def index():
print('Audio Input')
return render_template('index.html')
@app.route('/handle_question', methods=['POST'])
def handle_question():
audio = request.files['audio']
audio_path = "audio.webm"
audio.save(audio_path)
audio_file = open("audio.webm", "rb")
question = client.audio.transcriptions.create(
model="whisper-1",
file=audio_file,
response_format="text"
)
print(f"The question is: {question}")
chat_history.append(HumanMessage(content=question))
agent_input = {
"input": question,
"chat_history": chat_history,
}
response = agent_executor.invoke(agent_input)
output = response['output']
chat_history.append(AIMessage(content=output))
speech_path = "static/speech.mp3"
speech_response = client.audio.speech.create(
model="tts-1",
voice="nova",
input=output
)
speech_response.stream_to_file(speech_path)
return jsonify({'speech_url': speech_path})
if __name__ == '__main__':
app.run(debug=True)
```
### static/js/app.js
```javascript
// document.addEventListener('DOMContentLoaded', function() {
// // Check if the browser supports the required APIs
// if (!navigator.mediaDevices || !window.MediaRecorder) {
// alert('Your browser does not support the required media APIs.');
// return;
// }
// let mediaRecorder;
// let audioChunks = [];
// document.getElementById('startRecording').addEventListener('click', function() {
// navigator.mediaDevices.getUserMedia({ audio: true, video: false })
// .then(stream => {
// if (mediaRecorder) {
// // If a mediaRecorder instance already exists, stop any ongoing recording and release the previous stream
// if (mediaRecorder.state === 'recording') {
// mediaRecorder.stop();
// }
// stream.getTracks().forEach(track => track.stop());
// }
// mediaRecorder = new MediaRecorder(stream);
// audioChunks = [];
// mediaRecorder.addEventListener('dataavailable', event => {
// audioChunks.push(event.data);
// });
// mediaRecorder.addEventListener('stop', () => {
// const audioBlob = new Blob(audioChunks, { type: 'audio/webm' });
// const formData = new FormData();
// formData.append('audio', audioBlob);
// axios.post('http://127.0.0.1:5000/whisper', formData, {
// headers: {
// 'Content-Type': 'multipart/form-data'
// }
// })
// .then(response => {
// console.log('Server response:', response.data);
// // Here you can update the UI with the transcription result or any response from the server
// })
// .catch(error => console.error('Error sending audio to the server:', error));
// });
// // Use the Hark library to detect when the user starts and stops speaking
// const speechEvents = hark(stream, {});
// speechEvents.on('speaking', function() {
// console.log('Speaking started');
// mediaRecorder.start();
// });
// speechEvents.on('stopped_speaking', function() {
// console.log('Speaking stopped');
// if (mediaRecorder.state === 'recording') {
// mediaRecorder.stop();
// }
// });
// })
// .catch(error => {
// console.error('Error accessing the microphone:', error);
// alert('Could not access the microphone. Please ensure it is connected and permission is granted.');
// });
// });
// });
// Ensure the DOM is fully loaded before executing the script
document.addEventListener('DOMContentLoaded', function() {
let isProcessing = false; // Flag to indicate whether the backend is currently processing audio
let mediaRecorder; // Global variable to hold the MediaRecorder instance
// Initializes and starts the microphone recording based on speech detection
function initRecording() {
// Check if the browser supports necessary APIs and if not currently processing audio
if (navigator.mediaDevices && window.MediaRecorder && !isProcessing) {
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(stream => {
// Initialize hark with the audio stream and optional configurations
const options = {}; // Placeholder for hark options if needed
const speechEvents = hark(stream, options);
// Initialize MediaRecorder with the audio stream
mediaRecorder = new MediaRecorder(stream);
let audioChunks = []; // Array to store chunks of audio data
// Event listener for when audio data is available
mediaRecorder.ondataavailable = event => {
// Collect audio data chunks
audioChunks.push(event.data);
};
// Event listener for when recording stops
mediaRecorder.onstop = () => {
// Combine audio chunks into a single Blob
const audioBlob = new Blob(audioChunks, { type: 'audio/webm' });
// Send the audio blob to the server for processing
sendAudioToServer(audioBlob);
// Clear the audioChunks array for the next recording
audioChunks = [];
};
// Hark event when speech is detected
speechEvents.on('speaking', function() {
// Start recording if not currently processing audio
if (!isProcessing) {
mediaRecorder.start();
console.log('Started recording');
}
});
// Hark event when speech stops
speechEvents.on('stopped_speaking', function() {
// Stop recording if it's currently active
if (mediaRecorder.state === 'recording') {
mediaRecorder.stop();
console.log('Stopped recording');
// Indicate that audio is being processed
isProcessing = true;
}
});
})
.catch(error => {
// Handle errors, such as user denying microphone access
console.error('Error accessing the microphone:', error);
});
}
}
// Sends the recorded audio to the server and manages the res
[truncated — 1885 more characters]
```
### server/static/js/app.js
```javascript
// document.addEventListener('DOMContentLoaded', function() {
// // Check if the browser supports the required APIs
// if (!navigator.mediaDevices || !window.MediaRecorder) {
// alert('Your browser does not support the required media APIs.');
// return;
// }
// let mediaRecorder;
// let audioChunks = [];
// document.getElementById('startRecording').addEventListener('click', function() {
// navigator.mediaDevices.getUserMedia({ audio: true, video: false })
// .then(stream => {
// if (mediaRecorder) {
// // If a mediaRecorder instance already exists, stop any ongoing recording and release the previous stream
// if (mediaRecorder.state === 'recording') {
// mediaRecorder.stop();
// }
// stream.getTracks().forEach(track => track.stop());
// }
// mediaRecorder = new MediaRecorder(stream);
// audioChunks = [];
// mediaRecorder.addEventListener('dataavailable', event => {
// audioChunks.push(event.data);
// });
// mediaRecorder.addEventListener('stop', () => {
// const audioBlob = new Blob(audioChunks, { type: 'audio/webm' });
// const formData = new FormData();
// formData.append('audio', audioBlob);
// axios.post('http://127.0.0.1:5000/whisper', formData, {
// headers: {
// 'Content-Type': 'multipart/form-data'
// }
// })
// .then(response => {
// console.log('Server response:', response.data);
// // Here you can update the UI with the transcription result or any response from the server
// })
// .catch(error => console.error('Error sending audio to the server:', error));
// });
// // Use the Hark library to detect when the user starts and stops speaking
// const speechEvents = hark(stream, {});
// speechEvents.on('speaking', function() {
// console.log('Speaking started');
// mediaRecorder.start();
// });
// speechEvents.on('stopped_speaking', function() {
// console.log('Speaking stopped');
// if (mediaRecorder.state === 'recording') {
// mediaRecorder.stop();
// }
// });
// })
// .catch(error => {
// console.error('Error accessing the microphone:', error);
// alert('Could not access the microphone. Please ensure it is connected and permission is granted.');
// });
// });
// });
// Ensure the DOM is fully loaded before executing the script
document.addEventListener('DOMContentLoaded', function() {
let isProcessing = false; // Flag to indicate whether the backend is currently processing audio
let mediaRecorder; // Global variable to hold the MediaRecorder instance
// Initializes and starts the microphone recording based on speech detection
function initRecording() {
// Check if the browser supports necessary APIs and if not currently processing audio
if (navigator.mediaDevices && window.MediaRecorder && !isProcessing) {
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(stream => {
// Initialize hark with the audio stream and optional configurations
const options = {}; // Placeholder for hark options if needed
const speechEvents = hark(stream, options);
// Initialize MediaRecorder with the audio stream
mediaRecorder = new MediaRecorder(stream);
let audioChunks = []; // Array to store chunks of audio data
// Event listener for when audio data is available
mediaRecorder.ondataavailable = event => {
// Collect audio data chunks
audioChunks.push(event.data);
};
// Event listener for when recording stops
mediaRecorder.onstop = () => {
// Combine audio chunks into a single Blob
const audioBlob = new Blob(audioChunks, { type: 'audio/webm' });
// Send the audio blob to the server for processing
sendAudioToServer(audioBlob);
// Clear the audioChunks array for the next recording
audioChunks = [];
};
// Hark event when speech is detected
speechEvents.on('speaking', function() {
// Start recording if not currently processing audio
if (!isProcessing) {
mediaRecorder.start();
console.log('Started recording');
}
});
// Hark event when speech stops
speechEvents.on('stopped_speaking', function() {
// Stop recording if it's currently active
if (mediaRecorder.state === 'recording') {
mediaRecorder.stop();
console.log('Stopped recording');
// Indicate that audio is being processed
isProcessing = true;
}
});
})
.catch(error => {
// Handle errors, such as user denying microphone access
console.error('Error accessing the microphone:', error);
});
}
}
// Sends the recorded audio to the server and manages the res
[truncated — 1885 more characters]
```
### vision.py
```python
from IPython.display import display, Image, Audio
import cv2
import base64
import time
from openai import OpenAI
import os
import requests
client = OpenAI()
def generate_frame_descriptions(video_path):
video = cv2.VideoCapture(video_path)
base64Frames = []
while video.isOpened():
success, frame = video.read()
if not success:
break
_, buffer = cv2.imencode(".jpg", frame)
base64Frames.append(base64.b64encode(buffer).decode("utf-8"))
video.release()
PROMPT_MESSAGES = [
{
"role": "user",
"content": [
"You are given frames from a video. Your task is to describe the action in each frame, focusing on positioning and movement of individuals. Detail if the individual is standing, sitting, moving, or if there's an absence in a frame where they previously were.",
*map(lambda x: {"image": x, "resize": 768}, base64Frames[0::50]),
],
},
]
params = {
"model": "gpt-4-vision-preview",
"messages": PROMPT_MESSAGES,
"max_tokens": 200,
}
result = client.chat.completions.create(**params)
output = result.choices[0].message.content
return output
def determine_fall(frame_descriptions):
PROMPT = """
You will be provided with a series of descriptions for consecutive frames from a video. Your task is to analyze these descriptions to determine if they indicate that a fall has occurred. During any of the frame descriptions, if the person seems to begin moving downwards (indicated by change in posture), and then partially or totally disappears from the frame, it is to be considered a very likely indication that the individual has fallen.
You must either output yes or no. If you are unsure, output yes. You must not output any reasoning/explaination.
"""
output = gpt_4(PROMPT, frame_descriptions)
return output
def gpt_4(system, user, model="gpt-4-1106-preview"):
print("Using GPT-4")
messages = [
{
"role": "system",
"content": system
},
{
"role": "user",
"content": user
}
]
try:
response = client.chat.completions.create(
model=model,
messages=messages,
temperature=0,
)
gpt_response = response.choices[0].message.content.strip()
return gpt_response
except Exception as e:
print(f"Error in OpenAI API call: {e}")
return None
video_path = "sample_test_video.mp4"
frame_descriptions = generate_frame_descriptions(video_path)
fall_or_not = determine_fall(frame_descriptions)
print(fall_or_not)
```
### tools.py
```python
from dotenv import load_dotenv
from langchain.pydantic_v1 import BaseModel, Field
from langchain.tools import BaseTool, StructuredTool, tool
from twilio.rest import Client
import googlemaps
import os
import requests
load_dotenv()
account_sid = os.getenv('TWILIO_ACCOUNT_SID')
auth_token = os.getenv('TWILIO_AUTH_TOKEN')
from_number = os.getenv('TWILIO_PHONE_NUMBER')
gmaps_key = os.getenv('GOOGLE_MAPS_KEY')
gmaps = googlemaps.Client(key=gmaps_key)
@tool
def get_location_of_individual_in_emergency():
"""
Returns the address of an individual in an emergency.
Returns:
str: A human-readable address representing the individual's location. If the location cannot be determined
or converted into an address, a default message indicating that no address was found is returned.
"""
url = "https://www.googleapis.com/geolocation/v1/geolocate"
params = {'key': gmaps_key}
headers = {'Content-Type': 'application/json'}
data = {
"homeMobileCountryCode": 310,
"homeMobileNetworkCode": 410,
"radioType": "gsm",
"carrier": "Vodafone",
"considerIp": "true"
}
response = requests.post(url, params=params, headers=headers, json=data)
if response.status_code == 200:
data = response.json()
else:
raise Exception(f"Error in geolocation request: {response.text}")
latitude = float(data['location']['lat'])
longitude = float(data['location']['lng'])
result = gmaps.reverse_geocode((latitude, longitude))
if result:
location = result[0].get('formatted_address', 'No address found')
else:
location = 'No address found'
return location
@tool
def send_emergency_call(message):
"""
Sends an emergency call to with a specified message using Twilio. You must explicitly state the location as context in the call. The phone number will always be the same, so don't worry about it.
Parameters:
message (str): The message to be read during the call.
Returns:
bool: True if the call was successfully initiated, False otherwise.
"""
try:
# Twilio credentials - ideally, fetch from environment variables or a secure config
# Fetch Twilio credentials from environment variables
account_sid = os.getenv('TWILIO_ACCOUNT_SID')
auth_token = os.getenv('TWILIO_AUTH_TOKEN')
from_number = os.getenv('TWILIO_PHONE_NUMBER')
# Initialize the Twilio client
client = Client(account_sid, auth_token)
# Create the call with TwiML to say the emergency message
call = client.calls.create(
twiml=f'{message}',
to="+14156361256",
from_='+18667643449' # Your Twilio number
)
return True
except Exception as e:
return False
agent_tools = []
agent_tools.append(get_location_of_individual_in_emergency)
agent_tools.append(send_emergency_call)
# https://python.langchain.com/docs/modules/agents/tools/custom_tools
```
### components/news.py
```python
from gnews import GNews
def news(topic):
google_news = GNews()
news_data = google_news.get_news(topic)
headlines = [article['title'] for article in news_data]
for headline in headlines:
print(headline)
```
### templates/index.html
```html
Speech to Text Transcriber
Speech to Text Transcriber
```
[9 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]