According to the CDC, in the US, only about 47% of emergency department calls and visits actually end up in hospital admission, creating stress for both patients and hospice staff. That's why we developed SensaAI, a voice agent embedded in our web application that calms down and allows patients to assess the severity of their medical situation classifies as a emergency, urgent care, or regular hospital appointment, before a panicked call to 911. 💡

What it does

SensaAI is an intelligent voice powered triage-assistant built to support patients during moments of medical uncertainty. We chose voice to help support patients more effectively and with more compassion during vulnerable times. By speaking naturally with our voice agent, users can describe their symptoms and receive an AI-powered triage classification: Emergency, Urgent, or Routine Care, posted on our dashboard. Our dashboard also offers helpful tools for hospitals as well, allowing for users to generate a full report of their symptoms and transcript with the voice assistant prior to even stepping inside the hospital, reducing visit time. Credibility wise, we've even integrated backed-up medical advice from the American Heart Association into the voice agent to ensure accurate and safe instructions. 🛠️

How we built it

We used Vapi.ai to power our real time voice assistant and integrated it into a Next.js application for the web. We used ClaudeAPI calls in order to summarize and display the summaries from the AI voice assistant chat in a professional and readable format. In order to classify the various triage levels, we used the ClinicalBERT NLP model from HuggingFace, and FastAPI in order to connect between the model and the voice agent. The frontend displays the transcript, extracted symptoms, confidence scores, and clinical notes in a clean, responsive dashboard. We also integrated features like PDF report generation and patient history tracking. 📚

Challenges we ran into

For us, one of the challenges we've had was to get the voice agent to act dynamically, however using datasets and medical instructions, we were able to overcome that. 🏆

Accomplishments we're proud of

We're proud of building a working end-to-end voice triage assistant, and an interactive, full-stack, and user-friendly dashboard to integrate results from the chat. We're also very proud to have integrated various kinds of AI technologies like LLMs, APIs, and a NLP algorithm. 🚀

What's next

We hope to deploy SensaAI soon on the web, and integrate our last goal- using patients' real-time location to give more accurate suggestions.