# Project export: Dial AI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 11.0 - Tagline: Imagine someone's trying to kill you, and 911 puts you on hold when it matters most. Dial AI eliminates wait times by instantly answering, categorizing, and prioritizing emergency calls. - Devpost: https://devpost.com/software/dial-ai - GitHub: https://github.com/waquas9248/Dial-AI - Team: 3 GitHub contributor(s) — waquas9248 (11 commits), Qadeer2syed (4 commits), Shivank Singh Thakur (3 commits) ## Devpost submission (written by the team) ### Inspiration Alan Liang from Oakland faced a terrifying reality during an emergency. “I called 911, and the experience was horrible. I was placed on hold for 10 to 15 minutes. It took nearly 20 minutes to speak to a dispatcher.” The average wait time for 911 calls in Oakland is 54 seconds, but in emergencies, even a few seconds can mean the difference between life and death. No one should ever have to wait to be connected when every second could save a life. https://www.nbcbayarea.com/news/local/oakland-911-crisis/3266349/ ### What it does The average 911 wait time in cities like Oakland is 54 seconds, but during an emergency, every second counts. Dial AI is designed to support 911 centers when they’re understaffed or overwhelmed with calls. Our system uses a multi-agent approach: one AI agent engages with the caller in real-time instantly with 0 wait time, just like a live 911 operator, while another extracts critical information. Simultaneously, additional agents categorize and prioritize the details for the appropriate departments, ensuring that responders receive the information they need quickly. We provide a comprehensive dashboard to all department responders, where incidents are stack-ranked by severity, complete with the caller's location and all necessary details for immediate action. In times of high call volume or staffing shortages, traditional 911 lines can become overloaded. Dial AI ensures real-time communication, providing callers with instant responses and swift action. The system scales seamlessly based on call volume, ensuring there are never any delays due to the robust and adaptive design of the technology. Imagine a scenario like Alan Liang’s—someone in desperate need of help. Instead of being placed on hold for minutes, Dial AI engages immediately, ensuring their issue is addressed without delay. ### How we built it We carefully evaluated different models to ensure the best fit for our use case, understanding that when someone calls 911, they are already in distress. Selecting an empathetic voice model was crucial, and Vapi was the obvious choice for its ability to respond with human-like sensitivity. For real-time conversations, we integrated Vapi with Twilio through webhooks, allowing callers to speak directly into their phones as they would with a live operator. We built the system using Fetch AI to manage the agents, with AgentVerse hosting them. In this setup, two agents communicate with each other in real time. When a call is placed, the first agent retrieves the chat ID and real-time transcripts using Twilio’s webhook. These transcripts are passed through the agent, which uses OpenAI’s API (GPT-4) to extract key information, organize it, and prioritize the emergency based on urgency. This agent communicates with another agent to pass the extracted information to the dispatcher console. These agents work together, with one handling the live interactions and event classifications. The data from the first agent is formatted in JSON by the second agent and sent to a dashboard, where dispatchers can view emergencies stack-ranked by severity and location. By using Fetch AI, AgentVerse, Vapi, and OpenAI, we’ve created a system that ensures quick, real-time responses during high call volumes without delays, making the process efficient, empathetic, and reliable in critical moments. ### Challenges we ran into One of the main challenges we encountered was connecting Vapi with Twilio using webhooks. While obtaining transcripts after the conversation ended was straightforward, extracting real-time transcripts during the call was much more complex. This step was critical for ensuring that information could reach dispatchers as quickly as possible, so overcoming this learning curve was essential for speeding up emergency response. Another significant challenge was getting agents to communicate effectively within AgentVerse. Since this was entirely new for us, we spent a lot of time learning and refining how agents interact with each other in real-time to ensure a seamless flow of information and decision-making. ### Accomplishments we're proud of We’re incredibly proud to have built and validated a proof-of-concept AI emergency response agent that offers a truly lifelike experience, making conversational AI over the phone sound remarkably similar to speaking with a real 911 operator. Beyond that, we’ve developed a complete end-to-end solution that directly addresses a critical real-world issue—reducing high caller wait times while offering vital support to 911 operators. Our multi-agent model performed exceptionally well, and we managed to seamlessly integrate Twilio, Vapi, Streamlit and Fetch AI into a unified architecture. What excites us most is the real-time conversational AI on phones and how our agents work together to prioritize and process emergencies. This was our first experience working with voice technology, and we’re proud of how natural and effective the interactions feel, as well as how smoothly the agents communicate behind the scenes. ### What we learned We had never worked with voice synthesis or real-time conversational AI until this project, and we’re thrilled to have successfully implemented both. It was also a first for several team members to work with Streamlit and Webhooks, which added new technical challenges and learning opportunities. Through this project, we’ve gained a newfound respect for the vital work 911 dispatchers perform. It takes exceptional knowledge, skill, and empathy to handle constant emergency calls with such professionalism. While we’re proud of our accomplishments with Dial AI, this experience has deepened our appreciation for the challenging and compassionate work that dispatchers do every day. ### What's next This is just the first step toward creating an ideal world where every emergency call is answered instantly, with zero wait time. While we’re proud of what we’ve achieved, there’s still much to improve. Current models excel in clear-cut, black-and-white scenarios, but they struggle with more nuanced, gray-area situations. With advancements in frontier models, we expect Dial AI to become even better at making contextually aware decisions during calls. Additionally, we see room for improvement in both latency and empathy. As real-time conversational models evolve, Dial AI will continue to enhance its ability to respond faster and more compassionately. No seconds should be wasted—every second could save a life. That’s where the real impact of Dial AI lies. While we could have pursued a commercial project, like a customer service AI, our goal was to make a meaningful difference in people’s lives. We wanted to use advancements in technology to improve emergency response and potentially save lives. ## README (from the GitHub repository) # Dial AI ## 911 Emergency Call System with AI Agents ### Description We carefully evaluated different models to ensure the best fit for our use case, understanding that when someone calls 911, they are already in distress. Selecting a low latency voice model was crucial, and Vapi was the obvious choice for its ability to respond with human-like sensitivity. For real-time conversations, we integrated Vapi with Twilio through webhooks, allowing callers to speak directly into their phones as they would with a live operator. We built the system using Fetch AI to manage the agents, with AgentVerse hosting them. In this setup, two agents communicate with each other in real time. When a call is placed, the first agent retrieves the chat ID and real-time transcripts using Twilio’s webhook. These transcripts are passed through the agent, which uses OpenAI’s API (GPT-4) to extract key information, organize it, and prioritize the emergency based on urgency. This agent communicates with another agent to pass the extracted information to the dispatcher console. These agents work together, with one handling the live interactions and event classifications. The data from the first agent is formatted in JSON by the second agent and sent to a dashboard, where dispatchers can view emergencies stack-ranked by severity and location. ### Instructions to Run the Project 1. To initiate an instance of the project, please dial the following number: **+1(858)260-3506**. This initiates an instance emulating a 9-1-1 call. 2. After conversing with the AI agent, you can view the extracted and classified data in the dispatcher dashboard. 3. To view the dashboard (Run the following commands) : a) #Install Requirements: pip install -r requirements.txt b) #Start the Flask Server: python server.py c) #Expose the Flask Server using Ngrok: ngrok http 8080 d) #Run the Streamlit App: streamlit run dashboard.py ### Agents from the Agentverse - **Agent ID using Vapi as a voice assistant and OpenAI to process chats**: `agent1qgr7ah8a3fx7y6anwu98pkh3h099ntmkvynhxaczpt8gvwc3hq0q7jkmz7q` - **Agent ID using Hume as a voice assistant and OpenAI to process chats**: `agent1q0e07ywlx29q442zhn3qg5ycns79ex67e5c93hv5qxlves347j8hyc3wk8l` ## Detected evidence (automated analysis) Indexed codebase: 6 recognized source files, 20 KB. - Flask (technology) — detected in the code - Python (language) — detected in the code - Streamlit (technology) — detected in the code ## Codebase structure (from repository index) ### Files (9 of 9) ``` .devcontainer/devcontainer.json agents/transcript_agent_hume/agent.py agents/transcript_agent_vapi/agent.py agents/transcript_agent_vapi/simple_protocol.py dashboard.py data.json README.md requirements.txt server.py ``` ### Dependencies - requirements.txt: altair@==5.4.1, Flask@==3.0.3, folium@==0.17.0, geopy@==2.4.1, matplotlib@==3.9.2, pandas@==2.2.3, Requests@==2.32.3, streamlit@==1.39.0, streamlit_folium@==0.23.1 ### Recent commits (newest first) - add fixes - Update README.md - Update README.md - Update README.md - Update README.md - Added Dev Container Folder - Merge pull request #1 from waquas9248/dashboard - Add files via upload - Update README.md - Update README.md - Create agent.py - Create simple_protocol.py - Update agent.py - Create agent.py - Delete agents/transcript_agent_vapi - Create transcript_agent_vapi - Update README.md - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### requirements.txt ``` altair==5.4.1 Flask==3.0.3 folium==0.17.0 geopy==2.4.1 matplotlib==3.9.2 pandas==2.2.3 Requests==2.32.3 streamlit==1.39.0 streamlit_folium==0.23.1 ``` ### server.py ```python from flask import Flask, request, jsonify import json import os import logging # Set up logging logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s') app = Flask(__name__) # Path to the JSON file json_file_path = "data.json" # Initialize data file if it doesn't exist if not os.path.exists(json_file_path): initial_data = { "wildlife": [], "police": [], "water": [], "fire": [], "medical": [] } with open(json_file_path, 'w') as f: json.dump(initial_data, f) logging.info(f"Initialized data file at {json_file_path} with empty categories.") @app.route('/webhook', methods=['POST']) def webhook(): try: # Log HTTP request details logging.info(f"Received {request.method} request at {request.url}") logging.debug(f"Request headers: {request.headers}") logging.debug(f"Request data: {request.data}") data = request.json logging.debug(f"Parsed JSON data: {data}") # Read existing data with open(json_file_path, 'r') as f: existing_data = json.load(f) # Append new data to the appropriate categories for category, cases in data.items(): if category in existing_data: for case in cases: # Check if the case_number already exists in the category if case['case_number'] not in [existing_case['case_number'] for existing_case in existing_data[category]]: existing_data[category].append(case) logging.debug(f"Added new case to category '{category}': {case}") else: logging.debug(f"Duplicate case number '{case['case_number']}' found in category '{category}', not adding.") # Write updated data back to the file with open(json_file_path, 'w') as f: json.dump(existing_data, f) logging.info("Data updated successfully.") response = jsonify({"status": "success", "message": "Data received and updated."}) logging.info(f"Response: {response.get_json()}") return response, 200 except Exception as e: logging.exception("An error occurred while processing the webhook.") response = jsonify({"status": "error", "message": "An error occurred while processing the request."}) logging.info(f"Response: {response.get_json()}") return response, 500 if __name__ == '__main__': app.run(port=8080) ``` ### dashboard.py ```python import streamlit as st import pandas as pd from geopy.geocoders import Nominatim import time import folium from folium.plugins import MarkerCluster from streamlit_folium import folium_static import json import requests import matplotlib.pyplot as plt import altair as alt # Function to read JSON data from a file def read_json_data(file_path): with open(file_path, 'r') as f: return json.load(f) # Function to process JSON data def process_data(json_data): all_cases = [] for category, cases in json_data.items(): for case in cases: case['category'] = category all_cases.append(case) df = pd.DataFrame(all_cases) expected_columns = ['case_number', 'location', 'dispatch', 'situation', 'open_status', 'stack_rank', 'category'] for col in expected_columns: if col not in df.columns: st.error(f"Missing expected column: {col}") return None return df # Function to geocode locations def geocode_locations(locations): geolocator = Nominatim(user_agent="emergency_app") coords = [] for location in locations: try: loc = geolocator.geocode(location) if loc: coords.append((loc.latitude, loc.longitude)) else: coords.append((None, None)) except Exception as e: st.error(f"Error geocoding {location}: {e}") coords.append((None, None)) time.sleep(1) # To respect Nominatim's usage policy return coords # Color mapping for categories def get_color(category): color_map = { "wildlife": "green", "police": "blue", "water": "lightblue", "fire": "red" } return color_map.get(category, "gray") # Function to send POST request def send_post_request(data): url = "http://localhost:5000/webhook" headers = {"Content-Type": "application/json"} try: response = requests.post(url, data=json.dumps(data), headers=headers) if response.status_code == 200: st.success("POST request successful!") else: st.error(f"POST request failed with status code: {response.status_code}") except requests.exceptions.RequestException as e: st.error(f"An error occurred: {e}") # Streamlit app st.title("Emergency Call Insights") # Path to the JSON file json_file_path = "data.json" # Initialize session state to store data if 'data' not in st.session_state: st.session_state.data = {} # Create placeholders for components data_placeholder = st.empty() insights_placeholder = st.empty() map_placeholder = st.empty() # Streamlit loop to check for file updates while True: # Read the latest data new_data = read_json_data(json_file_path) # Check if the data has changed if new_data != st.session_state.data: st.session_state.data = new_data # Update data if the file has changed # Send POST request with new data # send_post_request(new_data) # Process the updated data with st.spinner("Processing data..."): df = process_data(st.session_state.data) if df is None or df.empty: st.warning("No data available to display.") continue # Skip the rest if there's no data # Update DataFrame with data_placeholder.container(): st.subheader("Emergency Call Data") st.dataframe(df) # Update Insights with insights_placeholder.container(): col1, col2 = st.columns(2) with col1: st.subheader("Open Status Distribution") open_status_counts = df['open_status'].value_counts() # Pie chart for open status fig, ax = plt.subplots() ax.pie(open_status_counts.values, labels=open_status_counts.index, autopct='%1.1f%%', startangle=90) ax.axis('equal') # Equal aspect ratio ensures that pie is drawn as a circle st.pyplot(fig) with col2: st.subheader("Task Priority by Category") # Bar chart for stack rank by category chart = alt.Chart(df).mark_bar().encode( x='category:N', y='stack_rank:Q', color='category:N', tooltip=['category', 'stack_rank', 'situation'] ).properties( width=400, height=300 ) st.altair_chart(chart, use_container_width=True) # Keep the existing category distribution chart category_counts = df['category'].value_counts() st.subheader("Emergency Category Distribution") st.bar_chart(category_counts) # Geocode locations with st.spinner("Geocoding locations..."): coords = geocode_locations(df['location'].tolist()) df['latitude'] = [coord[0] for coord in coords] df['longitude'] = [coord[1] for coord in coords] # Prepare map data with colors map_data = df[['latitude', 'longitude', 'category', 'situation', 'case_number']].dropna() map_data['color'] = map_data['category'].apply(get_color) # Update Map with map_placeholder.container(): st.subheader("Locations on Map") if not map_data.empty: with st.spinner("Generating map..."): # Create a map centered around the mean location of all valid coordinates m = folium.Map(location=[map_data['latitude'].mean(), map_data['longitude'].mean()], zoom_start=12) marker_cluster = MarkerCluster().add_to(m) # Add markers for each location for _, row in map_data.iterrows(): folium.Marker( [truncated — 560 more characters] ``` ### agents/transcript_agent_vapi/simple_protocol.py ```python from uagents import Context, Model, Protocol from ai_engine import UAgentResponse, UAgentResponseType simples = Protocol(name="simples", version="1.1") class Request(Model): message: str @simples.on_message(model=Request, replies={UAgentResponse}) async def handle_message(ctx: Context, sender: str, msg: Request): await ctx.send(sender, UAgentResponse(message="0", type=UAgentResponseType.FINAL)) ``` ### agents/transcript_agent_hume/agent.py ```python """ This agent can request data from a website and send an alert to your wallet if a condition is met. """ from uagents import Agent, Context class Request(Model): message: str agent = Agent() API_KEY = "" # Replace with your API key def fetch_top_chats(): url = "https://api.hume.ai/v0/evi/chats" params = { 'page_number': 0, 'page_size': 3, # Retrieve the top 3 chats 'ascending_order': False # Set descending order to get the latest chats first } headers = { 'X-Hume-Api-Key': API_KEY } response = requests.get(url, params=params, headers=headers) if response.status_code == 200: data = response.json() chat_ids = [chat["id"] for chat in data.get("chats_page", [])] return chat_ids else: print(f"Failed to fetch chats: {response.status}") return [] def process_chat_id(chat_id): # Example: define the second API call you want to perform with each chat ID url = f"https://api.hume.ai/v0/evi/chats/{chat_id}" params = { 'page_size' : 100, 'ascending_order': True # Set descending order to get the latest chats first } headers = { 'X-Hume-Api-Key': API_KEY } response = requests.get(url, params=params, headers=headers) if response.status_code == 200: data = response.json() event_roles = [event["role"] for event in data.get("events_page", [])] event_text = [event["message_text"] for event in data.get("events_page", [])] # Create the dictionary with the index as the key and the role:text as the value events_dict = { index: {event_roles[index]: event_text[index]} for index in range(len(event_roles)) } print(events_dict.items()) return events_dict else: print(f"Failed to fetch chats: {response.status}") return [] def fetch_transcripts(): # Fetch top 3 chats chat_ids = fetch_top_chats() ##ctx.send() transcripts = [] if chat_ids: # Process each chat ID iteratively for chat_id in chat_ids: print(f"Processing chat ID: {chat_id}") transcripts.append(process_chat_id(chat_id)) return transcripts @agent.on_interval(period=5) async def process_transcripts(ctx: Context): transcripts = fetch_transcripts() # Your API key OPENAI_API_KEY = "" # API endpoint url = "https://api.openai.com/v1/chat/completions" # System prompt system_prompt = """ You are a 911 AI agent bot, you will segregate 911 call transcripts into departments of wildlife, police, water, medical and fire. In each department, stack rank the calls based on severity. Give the output in JSON format, with each department containing a list of cases. Each case should include the following fields: - case number - location - dispatch - situation - open status (yes/no) - stack rank for each department. """ # User prompt made from event transcripts user_prompt = f"The following are the three 911 call transcripts which you need to segregate: {transcripts}" # JSON payload to be sent to the API payload = { "model": "gpt-4o-mini", "messages": [ { "role": "system", "content": system_prompt }, { "role": "user", "content": user_prompt } ] } # Headers for the API request headers = { "Content-Type": "application/json", "Authorization": f"Bearer {OPENAI_API_KEY}" } # Send the request to the OpenAI API response = requests.post(url, headers=headers, data=json.dumps(payload)) # Check if the response is successful if response.status_code == 200: # Parse the response data = response.json() # Extract the actual response content from the 'choices' array response_content = data['choices'][0]['message']['content'] else: print(f"Failed to get a response: {response.status_code}") print(response.text) # Print the response from the API output = data['choices'][0]['message']['content'] ctx.logger.info(output) if __name__ == "__main__": agent.run() ``` ### agents/transcript_agent_vapi/agent.py ```python """ This agent requests, fetches call transcripts of most recent 911 calls periodically and generates a report """ from uagents import Agent, Context from simple_protocol import simples class Request(Model): message: str agent = Agent() agent.include(simples) def fetch_transcripts(): # Define the API endpoint and parameters url = "https://api.vapi.ai/call" params = { 'assistantId': 'ID', 'phoneNumberId': 'PID', 'limit': 3 # Retrieve the top 3 elements } # Authorization header headers = { 'accept': 'application/json', 'Authorization': 'Bearer Token' } # Make the GET request to the API response = requests.get(url, params=params, headers=headers) # Check if the response is successful if response.status_code == 200: data = response.json() # Parse the JSON response # Prepare a list of dictionaries with 'id', 'transcript', and 'customer number' transcripts = [ { 'id': item.get('id'), 'transcript': item.get('transcript'), 'customer_number': item.get('customer', {}).get('number'), 'analysis' : item.get('analysis',{}).get('summary') } for item in data ] return transcripts else: return [] def send_report(report): # Define the API endpoint url = "https://a4ff-199-115-241-212.ngrok-free.app/webhook" # Define the JSON data to be sent in the POST request data = report # Define the headers headers = { 'Content-Type': 'application/json' } # Make the POST request to the API response = requests.post(url, headers=headers, data=json.dumps(data)) # Check if the response is successful if response.status_code == 200: print("Request was successful!") print("Response data:", response.json()) # Print the response data (if any) else: print(f"Failed to make the request: {response.status_code}") print(response.text) # Print the response message for debugging @agent.on_interval(period=60) async def process_transcripts(ctx: Context): transcripts = fetch_transcripts() # Your API key OPENAI_API_KEY = "" # API endpoint url = "https://api.openai.com/v1/chat/completions" # System prompt system_prompt = """ You are a 911 AI agent bot, you will segregate 911 call transcripts into departments of wildlife, police, water, medical and fire. If you feel an incident needs attention from multiple departments, you can add it to all. In each department, stack rank the calls based on severity. Give the output in JSON format, with each department containing a list of cases. Each case should include the following fields: - case number - location - dispatch - situation - open status (yes/no) - stack rank for each department. """ # User prompt made from event transcripts user_prompt = f"The following are the three 911 call transcripts which you need to segregate: {transcripts}" # JSON payload to be sent to the API payload = { "model": "gpt-4o-mini", "messages": [ { "role": "system", "content": system_prompt }, { "role": "user", "content": user_prompt } ] } # Headers for the API request headers = { "Content-Type": "application/json", "Authorization": f"Bearer {OPENAI_API_KEY}" } # Send the request to the OpenAI API response = requests.post(url, headers=headers, data=json.dumps(payload)) # Check if the response is successful if response.status_code == 200: # Parse the response data = response.json() # Extract the actual response content from the 'choices' array response_content = data['choices'][0]['message']['content'] else: print(f"Failed to get a response: {response.status_code}") print(response.text) # Print the response from the API output = data['choices'][0]['message']['content'] lines = output.splitlines() # Check if the string has more than two lines if len(lines) > 2: # Remove the first and last lines truncated_lines = lines[1:-1] else: # If the string has 2 lines or less, return an empty string return "" output = "\n".join(truncated_lines) send_report(output) ctx.logger.info("Done") if __name__ == "__main__": agent.run() ```