# Project export: SensaAI

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: UC Berkeley AI Hackathon 2025
- Tagline: Your voice-first health companion
- Devpost: https://devpost.com/software/sensaai
- GitHub: https://github.com/rfaraz/sensa.git
- Demo: https://sensa-theta.vercel.app/
- Video: https://www.youtube.com/embed/k67e5liylWU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 2 GitHub contributor(s) — Rida Faraz (21 commits), Kavi (4 commits)

## Devpost submission (written by the team)

### Overview

According to the CDC, in the US, only about 47% of emergency department calls and visits actually end up in hospital admission, creating stress for both patients and hospice staff. That's why we developed SensaAI, a voice agent embedded in our web application that calms down and allows patients to assess the severity of their medical situation classifies as a emergency, urgent care, or regular hospital appointment, before a panicked call to 911. 💡

### What it does

SensaAI is an intelligent voice powered triage-assistant built to support patients during moments of medical uncertainty. We chose voice to help support patients more effectively and with more compassion during vulnerable times. By speaking naturally with our voice agent, users can describe their symptoms and receive an AI-powered triage classification: Emergency, Urgent, or Routine Care, posted on our dashboard. Our dashboard also offers helpful tools for hospitals as well, allowing for users to generate a full report of their symptoms and transcript with the voice assistant prior to even stepping inside the hospital, reducing visit time. Credibility wise, we've even integrated backed-up medical advice from the American Heart Association into the voice agent to ensure accurate and safe instructions. 🛠️

### How we built it

We used Vapi.ai to power our real time voice assistant and integrated it into a Next.js application for the web. We used ClaudeAPI calls in order to summarize and display the summaries from the AI voice assistant chat in a professional and readable format. In order to classify the various triage levels, we used the ClinicalBERT NLP model from HuggingFace, and FastAPI in order to connect between the model and the voice agent. The frontend displays the transcript, extracted symptoms, confidence scores, and clinical notes in a clean, responsive dashboard. We also integrated features like PDF report generation and patient history tracking. 📚

### Challenges we ran into

For us, one of the challenges we've had was to get the voice agent to act dynamically, however using datasets and medical instructions, we were able to overcome that. 🏆

### Accomplishments we're proud of

We're proud of building a working end-to-end voice triage assistant, and an interactive, full-stack, and user-friendly dashboard to integrate results from the chat. We're also very proud to have integrated various kinds of AI technologies like LLMs, APIs, and a NLP algorithm. 🚀

### What's next

We hope to deploy SensaAI soon on the web, and integrate our last goal- using patients' real-time location to give more accurate suggestions.

## README (from the GitHub repository)

# SensaAI
### CalHacks AI Hackathon 2025

Full project page [https://devpost.com/software/sensaai]


## Detected evidence (automated analysis)

Indexed codebase: 21 recognized source files, 78 KB.
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- Hugging Face (technology) — detected in the code
- Next.js (technology) — detected in the code
- Python (language) — detected in the code
- PyTorch (technology) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code

## Codebase structure (from repository index)

### Files (37 of 37)

```
.gitignore
backend/app.py
backend/config.json
backend/Merged_Triage_Dataset.csv
backend/requirements.txt
backend/test_triage_classifier.py
backend/train_triage_classifier.py
backend/triage_bioclinicalbert/special_tokens.json
backend/triage_bioclinicalbert/tokenizer_config.json
backend/triage_bioclinicalbert/tokenizer.json
backend/triage_bioclinicalbert/vocab.txt
backend/triage_model/config.json
eslint.config.mjs
first_aid_manual_all_pages.txt
next.config.ts
package.json
postcss.config.mjs
README.md
src/app/api/claude/route.ts
src/app/api/context.ts
src/app/dashboard/page.tsx
src/app/globals.css
src/app/layout.tsx
src/app/page.tsx
src/components/InteractiveBlueBackground.tsx
src/components/PatientDashboard.tsx
src/components/SensaAI.tsx
src/components/ui/badge.tsx
src/components/ui/button.tsx
src/components/ui/card.tsx
src/components/VapiWidget.tsx
src/lib/utils.ts
tsconfig.json
vapi-context/aha_manual.txt
vapi-context/aha_output.txt
vapi-context/aha.py
vapi-context/firstaid.py
```

### Dependencies

- backend/requirements.txt: accelerate@==1.8.1, aiohappyeyeballs@==2.6.1, aiohttp@==3.12.13, aiosignal@==1.3.2, annotated-types@==0.7.0, anyio@==4.9.0, attrs@==25.3.0, certifi@==2025.6.15, charset-normalizer@==3.4.2, click@==8.2.1, datasets@==3.6.0, dill@==0.3.8, dnspython@==2.7.0, email_validator@==2.2.0, fastapi@==0.115.13, fastapi-cli@==0.0.7, filelock@==3.18.0, frozenlist@==1.7.0, fsspec@==2025.5.1, h11@==0.16.0, hf-xet@==1.1.5, httpcore@==1.0.9, httptools@==0.6.4, httpx@==0.28.1, huggingface-hub@==0.33.0, idna@==3.10, itsdangerous@==2.2.0, Jinja2@==3.1.6, joblib@==1.5.1, markdown-it-py@==3.0.0, MarkupSafe@==3.0.2, mdurl@==0.1.2, mpmath@==1.3.0, multidict@==6.5.0, multiprocess@==0.70.16, networkx@==3.5, numpy@==1.26.4, orjson@==3.10.18, packaging@==25.0, pandas@==2.3.0, propcache@==0.3.2, psutil@==7.0.0, pyarrow@==20.0.0, pydantic@==1.10.22, pydantic_core@==2.33.2, Pygments@==2.19.2, python-dateutil@==2.9.0.post0, python-dotenv@==1.1.0, python-multipart@==0.0.20, pytz@==2025.2, PyYAML@==6.0.2, regex@==2024.11.6, requests@==2.32.4, rich@==14.0.0, rich-toolkit@==0.14.7, safetensors@==0.5.3, scikit-learn@==1.7.0, scipy@==1.15.3, shellingham@==1.5.4, six@==1.17.0, sniffio@==1.3.1, starlette@==0.46.2, sympy@==1.14.0, threadpoolctl@==3.6.0, tokenizers@==0.21.1, torch@==2.2.2, tqdm@==4.67.1, transformers@==4.51.0, typer@==0.16.0, typing_extensions@==4.14.0, typing-inspection@==0.4.1, tzdata@==2025.2, ujson@==5.10.0, urllib3@==2.5.0, uv@==0.7.13, uvicorn@==0.34.3, uvloop@==0.21.0, watchfiles@==1.1.0, websockets@==15.0.1, xxhash@==3.5.0, yarl@==1.20.1
- package.json: @eslint/eslintrc@^3, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, @vapi-ai/web@^2.3.6, class-variance-authority@^0.7.1, eslint@^9, eslint-config-next@15.3.4, html2canvas@^1.4.1, html2pdf.js@^0.10.3, jspdf@^3.0.1, lucide-react@^0.522.0, next@15.3.4, radix-ui@^1.4.2, react@^19.0.0, react-dom@^19.0.0, style@^0.0.3, tailwind-merge@^3.3.1, tailwindcss@^4, typescript@^5

### Recent commits (newest first)

- Update README.md
- Update README.md
- Update README.md
- YEET
- BACKGROUND
- background fix
- clinical fixed
- actions fixed
- bruh
- plz plz plz plz
- plz plz plz
- correct patient dashboard
- yay
- no more military time!
- more changes
- triage_bioclinicalbert
- triage_bioclinicalbert
- more commits
- Add files via upload
- Add files via upload

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "sensa",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev --turbopack",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@vapi-ai/web": "^2.3.6",
    "class-variance-authority": "^0.7.1",
    "html2canvas": "^1.4.1",
    "html2pdf.js": "^0.10.3",
    "jspdf": "^3.0.1",
    "lucide-react": "^0.522.0",
    "next": "15.3.4",
    "radix-ui": "^1.4.2",
    "react": "^19.0.0",
    "react-dom": "^19.0.0",
    "style": "^0.0.3",
    "tailwind-merge": "^3.3.1"
  },
  "devDependencies": {
    "@eslint/eslintrc": "^3",
    "@tailwindcss/postcss": "^4",
    "@types/node": "^20",
    "@types/react": "^19",
    "@types/react-dom": "^19",
    "eslint": "^9",
    "eslint-config-next": "15.3.4",
    "tailwindcss": "^4",
    "typescript": "^5"
  }
}

```

### backend/requirements.txt

```
accelerate==1.8.1
aiohappyeyeballs==2.6.1
aiohttp==3.12.13
aiosignal==1.3.2
annotated-types==0.7.0
anyio==4.9.0
attrs==25.3.0
certifi==2025.6.15
charset-normalizer==3.4.2
click==8.2.1
datasets==3.6.0
dill==0.3.8
dnspython==2.7.0
email_validator==2.2.0
fastapi==0.115.13
fastapi-cli==0.0.7
filelock==3.18.0
frozenlist==1.7.0
fsspec==2025.5.1
h11==0.16.0
hf-xet==1.1.5
httpcore==1.0.9
httptools==0.6.4
httpx==0.28.1
huggingface-hub==0.33.0
idna==3.10
itsdangerous==2.2.0
Jinja2==3.1.6
joblib==1.5.1
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mdurl==0.1.2
mpmath==1.3.0
multidict==6.5.0
multiprocess==0.70.16
networkx==3.5
numpy==1.26.4
orjson==3.10.18
packaging==25.0
pandas==2.3.0
propcache==0.3.2
psutil==7.0.0
pyarrow==20.0.0
pydantic==1.10.22
pydantic_core==2.33.2
Pygments==2.19.2
python-dateutil==2.9.0.post0
python-dotenv==1.1.0
python-multipart==0.0.20
pytz==2025.2
PyYAML==6.0.2
regex==2024.11.6
requests==2.32.4
rich==14.0.0
rich-toolkit==0.14.7
safetensors==0.5.3
scikit-learn==1.7.0
scipy==1.15.3
shellingham==1.5.4
six==1.17.0
sniffio==1.3.1
starlette==0.46.2
sympy==1.14.0
threadpoolctl==3.6.0
tokenizers==0.21.1
torch==2.2.2
tqdm==4.67.1
transformers==4.51.0
typer==0.16.0
typing-inspection==0.4.1
typing_extensions==4.14.0
tzdata==2025.2
ujson==5.10.0
urllib3==2.5.0
uv==0.7.13
uvicorn==0.34.3
uvloop==0.21.0
watchfiles==1.1.0
websockets==15.0.1
xxhash==3.5.0
yarl==1.20.1

```

### backend/app.py

```python
from fastapi.middleware.cors import CORSMiddleware

from fastapi import FastAPI
from pydantic import BaseModel
from transformers import pipeline, AutoTokenizer, AutoModelForSequenceClassification
import torch

app = FastAPI()

app = FastAPI()

# Allow frontend at localhost:3000 to make requests to FastAPI
app.add_middleware(
    CORSMiddleware,
    allow_origins=["http://localhost:3000"],  # frontend origin
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

# Map numeric labels to human-readable triage levels
id2label = {
    0: "Emergency",
    1: "Urgent Care",
    2: "Regular Hospital Appointment"
}

# Load model + tokenizer from your fine-tuned directory
model_path = "./triage_bioclinicalbert"
model = AutoModelForSequenceClassification.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

classifier = pipeline("text-classification", model=model, tokenizer=tokenizer)

# Define input structure
class SymptomInput(BaseModel):
    symptoms: str

@app.post("/classify")
async def classify(symptoms: SymptomInput):
    # Run the model on input symptoms
    results = classifier(symptoms.symptoms, return_all_scores=True)[0]

    # Find the top label
    top = max(results, key=lambda x: x["score"])
    label_id = int(top["label"].replace("LABEL_", ""))
    top_label = id2label[label_id]

    # Return scores for all labels as well
    all_scores = {
        id2label[int(res["label"].replace("LABEL_", ""))]: res["score"]
        for res in results
    }

    return {
        "label": top_label,
        "confidence": round(top["score"], 4),
        "all_scores": all_scores
    }

```

### src/app/page.tsx

```typescript
'use client';

import VapiWidget from '../components/VapiWidget';
import SensaAI  from '../components/SensaAI';
import InteractiveBlueBackground from '@/components/InteractiveBlueBackground';

export default function Home() {
  return (
    <main>
      <InteractiveBlueBackground>
        <SensaAI></SensaAI>
        <VapiWidget 
          apiKey="916c8cdc-4a41-465c-8216-341cea5420a5" 
          assistantId="e8eb09ed-0197-456e-8f6e-39d87c702f23" 
        />
      </InteractiveBlueBackground>
    </main>
  );
}

```

### src/app/layout.tsx

```typescript
import type { Metadata } from "next";
import { Geist, Geist_Mono, Outfit } from "next/font/google";
import "./globals.css";

const geistSans = Geist({
  variable: "--font-geist-sans",
  subsets: ["latin"],
});

const geistMono = Geist_Mono({
  variable: "--font-geist-mono",
  subsets: ["latin"],
});

const outfit = Outfit({
  variable: "--font-Outfit",
  subsets: ["latin"],
});

export const metadata: Metadata = {
  title: "Create Next App",
  description: "Generated by create next app",
};

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en">
      <body
        className={`${geistSans.variable} ${geistMono.variable} ${outfit.variable} font-sans antialiased`}
      >
        {children}
      </body>
    </html>
  );
}
```

### src/app/dashboard/page.tsx

```typescript
"use client"
import { useEffect, useState } from "react"
import PatientDashboard from "@/components/PatientDashboard"

export default function DashboardPage() {
  // Define a type for your dashboard data
  type PatientDataType = {
    name: string
    age: number
    gender: string
    phone: string
    callDate: string
    duration: string
    urgency: string
  }

  type ExtractedSymptom = {
    symptom: string
    severity: string
    duration: string
    location: string
    description: string
  }

  type ClinicalNote = {
    assessment: string
    concerns: string
    plan: string
  }

  type DashboardData = {
    patientData: PatientDataType
    transcript: string
    extractedSymptoms: ExtractedSymptom[]
    actions?: string[]
    clinicalNotes?: ClinicalNote
  }

  const [dashboardData, setDashboardData] = useState<DashboardData | null>(null)

  useEffect(() => {
    const stored = localStorage.getItem("dashboardData")
    if (stored) {
      setDashboardData(JSON.parse(stored))
    }
  }, [])

  if (!dashboardData) {
    return <div className="p-8">Loading dashboard...</div>
  }

  return (
    <PatientDashboard 
      patientData={dashboardData.patientData}
      transcript={dashboardData.transcript}
      extractedSymptoms={dashboardData.extractedSymptoms}
      actions={dashboardData.actions ?? []}
      clinicalNotes={dashboardData.clinicalNotes ?? { assessment: "", concerns: "", plan: "" }}
    />
  )
}

```

### src/app/api/claude/route.ts

```typescript
// app/api/extract-claude/route.ts
import { NextRequest, NextResponse } from "next/server";

export async function POST(req: NextRequest) {
  const body = await req.json();
  const {transcript, elapsedTime } = body;

  const prompt = `
        You are a medical triage assistant. A transcript of a patient voice call is provided below.

        Your task is to analyze it and return a JSON object with the following structure:

        {
        "patientData": {
            "name": string,
            "age": number,
            "gender": string,
            "phone": string (or "Unknown"),
            "callDate": string (YYYY-MM-DD),
            "duration": ${elapsedTime} ,
            "urgency": "High" | "Medium" | "Low"
        },
        "extractedSymptoms": [
            {
            "symptom": string,
            "severity": string,
            "duration": string,
            "location": string,
            "description": string
            }
        ],
        "transcript": string, // cleaned up and readable full transcript
        "actions": [string] // list of actions taken or N/A if none
        "clinicalNotes": {
            "assessment": string,
            "concerns": string,
            "plan": string
        }
        }

        Please infer or approximate any missing fields (e.g., phone = "Unknown", callTime = "Unknown", etc.). Only return valid JSON with no extra commentary.

        Transcript:
        ${transcript}
    Fill in any missing values with "N/A"
    `;

  const response = await fetch("https://api.anthropic.com/v1/messages", {
    method: "POST",
    headers: {
      "x-api-key": process.env.ANTHROPIC_API_KEY!,
      "Content-Type": "application/json",
      "anthropic-version": "2023-06-01",
    },
    body: JSON.stringify({
      model: "claude-opus-4-20250514",
      max_tokens: 1024,
      temperature: 0.3,
      messages: [
        {
          role: "user",
          content: prompt,
        },
      ],
    }),
  });

  const data = await response.json();
//   console.log("Claude response data:", data); // Debugging output

  // Extract Claude's response text
  let message = data?.content?.[0]?.text?.trim() || "{}";

  // Strip markdown formatting
  message = message.trim();
  if (message.startsWith("```json")) {
  message = message.replace(/^```json/, "").replace(/```$/, "").trim();
  } else if (message.startsWith("```")) {
  message = message.replace(/^```/, "").replace(/```$/, "").trim();
  }
  console.log("Claude response:", message); // Print output

  try {
    const parsed = JSON.parse(message);
    return NextResponse.json({ result: parsed });
  } catch (error) {
    console.error("Error parsing Claude output:", error);
    return NextResponse.json({
      error: "Failed to parse Claude output",
      raw: message,
    }, { status: 500 });
  }
}

```

### next.config.ts

```typescript
import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  /* config options here */
};

export default nextConfig;

```

### vapi-context/firstaid.py

```python
from pypdf import PdfReader

reader = PdfReader('first_aid_manual.pdf')
with open('first_aid_manual_all_pages.txt', 'w', encoding='utf-8') as f:
    for i, page in enumerate(reader.pages[9:], start=10):
        # Skip pages 181 to 192 (inclusive)
        if 181 <= i <= 192:
            continue
        text = page.extract_text()
        f.write(f"--- Page {i} ---\n")
        f.write(text if text else "")
        f.write("\n\n")
```

### vapi-context/aha.py

```python

# Extract text from HTML
import re
import html as html_lib
import pprint

# Load the HTML file
with open("aha_manual.txt", "r", encoding="utf-8") as f:
    html = f.read()

# Helper to clean HTML text
def clean_html(text):
    text = re.sub(r"<[^>]+>", "", text)  # remove tags
    return html_lib.unescape(text.strip())  # decode HTML entities

# Find all caption-table pairs
tables = re.findall(r"<caption>(.*?)</caption>(.*?)</table>", html, re.DOTALL)

# Extract data into a dict
result = {}

for caption, table_content in tables:
    key = clean_html(caption)
    steps = re.findall(r"<td[^>]*>(.*?)</td>", table_content, re.DOTALL)
    cleaned_steps = [clean_html(step) for step in steps]
    result[key] = cleaned_steps

# Write results to output file
with open("aha_output.txt", "w", encoding="utf-8") as out_f:
    pprint.pprint(result, stream=out_f)

```

[12 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]