# Project export: Symbio This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Probabilistic intelligence.Deterministic safety. - Devpost: https://devpost.com/software/care-ai-4i3scy - GitHub: https://github.com/kwangel/care-ai - Video: https://www.youtube.com/embed/eIc3gpv9Hf0?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Gina Choi (7 commits), Katie Wang (6 commits) ## Devpost submission (written by the team) ### Inspiration You know the kind of pain that’s not "go to the ER" serious, but also not nothing? A weird rash that's probably fine. A headache that won't go away. A prescription you're not sure is safe to mix with your other meds. So you do what everyone else does: you ask ChatGPT or Claude. LLMs are incredible at generating medical-sounding explanations. But they hallucinate, justify confidently, and over-dramaticize... because that’s how medical literature is written. Clinical papers are trained to list out catastrophic edge cases. So the model does the same. And most importantly, they don’t take responsibility for action, they say, “consult your doctor”. That’s autocomplete, not care. We wanted to build the thing that should already exist. An AI that can look at your symptoms (described, photographed, or spoken aloud), reason about them the way a clinician would, and tell you what to actually do. Not a chatbot that hedges or hallucinates. An agent that builds a real plan: triage level, possible conditions, recommended actions, drug safety checks, then helps you execute it. What We Built Symbio is a multi-turn conversational healthcare agent. You talk to it like you'd talk to a nurse practitioner (describe what's wrong, upload a photo, or just speak) and it runs a full clinical pipeline behind the scenes to give you an actionable care plan. Under the hood, it's five systems working together: 1. Multimodal Intake. The agent accepts text, voice (via Whisper), and images. Images are classified first — is this a skin photo? A prescription bottle? An X-ray? An insurance card? — then routed to specialized analyzers. A photo of a rash gets feature extraction (color, texture, distribution, morphology). A prescription gets OCR and drug identification. Everything unifies into a structured patient event. 2. Mixture of Experts Planner. When generating a care plan, we fan out to four models in parallel: A judge model (Claude) then synthesizes the four opinions into a single plan. The synthesis isn't naive — it uses weighted consensus: [ \text{triage}_{\text{final}} = \text{consensus}(\text{structured experts}) \oplus \text{escalate_only_if}(\text{specialist explicitly recommends ER}) ] The large models set the triage baseline. The specialists contribute additional differentials and clinical nuances. But a small model merely mentioning stroke in a differential list doesn't override two large models saying "this is self-care." 3. Clinical Knowledge Graph. A SNOMED CT-inspired graph, combined with ontologies like RxNorm and LOINIC, encoding symptoms, conditions, medications, and their relationships — HAS_FINDING, RED_FLAG_FOR, CONTRAINDICATED_WITH, INTERACTS_WITH. This is the deterministic backbone. If you're on warfarin and ask about ibuprofen, the graph catches the bleed risk interaction. No LLM hallucination can bypass it. 4. Constraint Engine. A symbolic safety layer that validates every plan before it reaches the patient. It checks hard-coded emergency rules (chest pain + shortness of breath → call 911), blocks dangerous medication recommendations, and enforces scope-of-practice (the agent says "possible conditions," never "you have"). This runs after the AI and before the response — a deterministic safety net over a probabilistic system. 5. Conversational Agent. The agent orchestrates everything through Claude's tool-use API. It decides when to run intake, plan, validate, check drugs, or escalate — chaining up to six tool calls per message. Session state accumulates patient context across turns (conditions, medications, allergies, demographics), so the agent gets more informed as the conversation continues. How We Built It The backend is FastAPI (Python), chosen for async support — essential when calling four models simultaneously with asyncio.gather. The frontend is Next.js with Tailwind CSS. Voice input uses OpenAI Whisper for speech-to-text, and responses can be read aloud via OpenAI TTS. The biomedical models run on Modal for efficient GPU inference. Claude receives conversation history plus tool definitions, decides which tool to call, we execute it and feed the result back, and Claude decides the next step — looping until it's ready to respond to the patient. The MoE pipeline uses a two-phase wait strategy: API experts (Claude, GPT-4o) return in ~15 seconds, then we wait up to 7 minutes for Modal experts to handle potential cold starts. The backend pre-warms Modal containers on startup so they're ready by the first user message. Challenges The over-triage problem was the hardest thing we dealt with, and it taught us the most. Our first MoE implementation had a safety rule: "if ANY expert recommends emergency, the final triage MUST be emergency." Sounds responsible. In practice, it meant that BioMistral-7B listing "1. Stroke 2. Meningitis 3. Migraine" as a differential for a mild headache forced the entire system to tell the patient to call 911. Two well-calibrated models saying "self-care, risk score 0.2" were overridden by one small model doing what medical literature trained it to do — list the worst things first. The fix required rethinking what "safety" means. Telling someone with a tension headache to call 911 isn't safe — it erodes trust, wastes emergency resources, and desensitizes people to real warnings. We redesigned the judge to use weighted majority consensus: structured experts set the baseline, specialist models contribute insights proportionally, and escalation only happens when specialists explicitly recommend emergency action — not just when they mention a scary condition in passing. A differential diagnosis is a thinking tool, not an alarm. Modal cold starts were a constant UX challenge. The biomedical models need 3-5 minutes on first invocation (downloading weights, loading onto GPU). We couldn't block the user for that long, so we implemented two-phase waiting: return API expert results immediately, then incorporate Modal expert results when they arrive. On startup, the backend fires warm-up requests in the background so containers are hot by the time the user types their first message. The Future Just 2 years ago, this was impossible. But by combining symbolic research with agent capabilities we can build AI that reasons, validates, and acts. Imagine a world where your first line of care isn’t Google, or ChatGPT. Where AI can Schedule a telehealth visit, escalate appropriately, and guide you through home care steps. We believe this is the future of healthcare AI. Probabilistic intelligence. Deterministic safety. What We Learned Ensemble AI needs opinionated synthesis, not naive aggregation. "Take the most urgent assessment" and "include all red flags from all experts" sound like safe defaults. In practice, they produce plans that are simultaneously thorough and useless — treating every symptom like it could be fatal. A good judge model needs to understand confidence weighting, the difference between a differential and a recommendation, and when a minority opinion should be noted versus when it should set the triage level. Symbolic safety layers are non-negotiable for healthcare AI. The constraint engine and knowledge graph don't hallucinate. They don't have off days. If chest pain plus shortness of breath appears in the symptoms, triage goes to emergency — no prompt engineering can change that. LLMs handle the nuanced reasoning; deterministic systems handle the bright-line rules. About Us Undergrad & Master's students at UC Berkeley (EECS, CS, math, data science). ## README (from the GitHub repository) # CareGraph Multimodal healthcare agent that helps patients navigate their care. Upload images (symptoms, prescriptions, lab results, insurance cards), use voice input, and get AI-powered clinical plans with triage, possible conditions, and recommended actions. ## Architecture ``` frontend/ Next.js 16 React app (chat UI, voice recorder, image upload) backend/ FastAPI Python server app/ main.py API endpoints (sessions, messages, voice, TTS) agent.py Conversational agent loop intake.py Multimodal intake — image classification + analysis (Claude Vision), text extraction, voice transcription (Whisper) planner.py Clinical reasoning engine — generates triage + care plan moe.py Mixture of Experts — fans out to 4 planner models, judge synthesizes knowledge.py SNOMED CT clinical knowledge graph constraints.py Safety constraints + red flag detection executor.py Task execution (booking, prescriptions, referrals) sessions.py Session state management schemas.py Pydantic models for all data types config.py App configuration (env vars) modal_services/ GPU-backed biomedical LLMs on Modal planner_experts.py OpenBioLLM-8B + BioMistral-7B endpoints ``` ## Planner Mixture of Experts (MoE) When `MOE_ENABLED=true`, the planner runs 4 models in parallel and a judge synthesizes the best plan: | Expert | Source | What it does | |--------|--------|--------------| | Claude Sonnet | Anthropic API | General clinical planner (structured JSON) | | GPT-4o | OpenAI API | Different model family perspective (structured JSON) | | OpenBioLLM-8B | Modal (A10G GPU) | Llama 3 fine-tuned on PubMed + clinical trials (free-text reasoning) | | BioMistral-7B | Modal (A10G GPU) | Mistral fine-tuned on PubMed Central (free-text reasoning) | The judge (Claude) applies strict safety rules: triage is always at least as urgent as the most urgent expert, red flags from any expert are included, and specialist-only findings are never discarded. ## Prerequisites - Python 3.11+ - Node.js 18+ - API keys: Anthropic (required), OpenAI (optional — for voice, TTS, GPT-4o MoE expert) - Modal account (optional — for biomedical MoE experts) ## Setup ### 1. Backend ```bash # Create and activate a virtual environment python -m venv .venv source .venv/bin/activate # Install dependencies pip install -r backend/requirements.txt # Create your .env file cp backend/.env.example backend/.env # then fill in your API keys ``` Create `backend/.env` with: ``` ANTHROPIC_API_KEY=sk-ant-... OPENAI_API_KEY=sk-proj-... # optional — needed for voice + GPT-4o expert MOE_ENABLED=true # optional — enables multi-model planner ensemble MOE_OPENBIO_URL= # set after Modal deploy (see below) MOE_BIOMISTRAL_URL= # set after Modal deploy (see below) ``` ### 2. Frontend ```bash cd frontend npm install ``` ### 3. Modal (optional — for biomedical MoE experts) ```bash pip install modal modal setup # one-time auth # Deploy the biomedical planner experts modal deploy modal_services/planner_experts.py ``` This prints endpoint URLs. Add them to `backend/.env`: ``` MOE_OPENBIO_URL=https://YOUR_USERNAME--caregraph-planner-experts-openbiollmexpert-plan.modal.run MOE_BIOMISTRAL_URL=https://YOUR_USERNAME--caregraph-planner-experts-biomistralexpert-plan.modal.run ``` ## Running Start both servers (from the repo root): ```bash # Terminal 1 — Backend (port 8000) source .venv/bin/activate cd backend uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 # Terminal 2 — Frontend (port 3000) cd frontend npm run dev ``` Open **http://localhost:3000** in your browser. If MoE is enabled, the backend automatically warms up Modal containers on startup (takes 2-3 min on first boot, then stays warm for 5 minutes between requests). ## Features - **Text chat** — describe symptoms, ask health questions - **Image upload** — photos of symptoms, prescriptions, medications, lab results, insurance cards, X-rays - **Voice input/output** — record voice messages (Whisper STT), listen to responses (OpenAI TTS) - **Clinical knowledge graph** — SNOMED CT-based symptom-to-condition mapping, red flag detection, drug interaction checks - **Safety constraints** — automatic triage escalation, red flag detection, medication contraindication alerts - **Mixture of Experts** — 4-model ensemble for more thorough clinical reasoning ## API Docs With the backend running, visit **http://localhost:8000/docs** for the interactive Swagger UI. ---- # care-ai ## arch ``` ┌─────────────────────────────────────────────────────────────────┐ │ CAREGRAPH PIPELINE │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ 1. INTAKE │ │ ↓ Multimodal processing (text, images) │ │ PatientEvent │ │ │ │ 2. PLANNER │ │ ↓ Clinical reasoning + RAG │ │ ClinicalPlan (initial) │ │ │ │ 3. CONSTRAINT SERVICE (uses Knowledge Graph) │ │ ├─ Medical Safety Rules │ │ │ └─ queries KG for contraindications, red flags │ │ ├─ Drug Interaction Checker │ │ │ └─ queries KG for interactions │ │ ├─ Scope-of-Practice Validator (LLM) │ │ └─ queries KG for drug classes │ │ └─ Auto-Fix Engine │ │ ↓ │ │ ClinicalPlan (validated) + ConstraintViolations │ │ │ │ 4. EXECUTOR (TODO) │ │ ↓ Browser automation, booking, calling │ │ ExecutorTasks │ │ │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────┐ │ KNOWLEDGE GRAPH (Dual Mode) │ ├─────────────────────────────────┤ │ │ │ In-Memory (Default) │ │ • ~150 concepts │ │ • 300+ relationships │ │ • No setup required │ │ │ │ OR │ │ │ │ Neo4j (Production) │ │ • Unlimited concepts │ │ • Graph queries │ │ • SNOMED CT integration │ │ │ └─────────────────────────────────┘ ``` ## knowledge graph (`app/knowledge.py`) SNOMED CT-inspired clinical ontology: #### Concept Types - `Condition` (50+ conditions: MI, pneumonia, cellulitis, etc.) - `Symptom` (40+ symptoms: chest pain, fever, rash, etc.) - `Medication` (30+ drugs: ibuprofen, warfarin, metformin, etc.) - `DrugClass` (NSAID, ACE inhibitor, statin, etc.) #### Relationship Types - `IS_A` — Taxonomy (ibuprofen IS_A NSAID) - `HAS_FINDING` — Symptoms (MI HAS_FINDING chest pain) - `CONTRAINDICATED_WITH` — Safety (NSAID ↔ CKD) - `INTERACTS_WITH` — Drug interactions (warfarin ↔ [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 40 recognized source files, 344 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (48 of 48) ``` .gitignore backend/app/__init__.py backend/app/agent.py backend/app/config.py backend/app/constraints.py backend/app/executor.py backend/app/formatters.py backend/app/intake.py backend/app/knowledge.py backend/app/main.py backend/app/moe.py backend/app/neo4j_knowledge.py backend/app/planner.py backend/app/schemas.py backend/app/sessions.py backend/app/utils.py backend/migrate_to_neo4j.py backend/requirements.txt backend/tests/__init__.py backend/tests/clinical/__init__.py backend/tests/clinical/report.py backend/tests/clinical/run_eval.py backend/tests/clinical/runner.py backend/tests/clinical/scorer.py backend/tests/clinical/vignettes.json backend/tests/test_pipeline_e2e.py frontend/.gitignore frontend/app/globals.css frontend/app/layout.tsx frontend/app/page.tsx frontend/components/AlertCard.tsx frontend/components/AudioPlayer.tsx frontend/components/Chat.tsx frontend/components/ImageUpload.tsx frontend/components/MessageBubble.tsx frontend/components/PlanCard.tsx frontend/components/TypingIndicator.tsx frontend/components/VoiceRecorder.tsx frontend/eslint.config.mjs frontend/lib/api.ts frontend/lib/types.ts frontend/next.config.ts frontend/package.json frontend/postcss.config.mjs frontend/README.md frontend/tsconfig.json modal_services/planner_experts.py README.md ``` ### Dependencies - backend/requirements.txt: anthropic, fastapi, httpx, neo4j, openai, pydantic, pydantic-settings, pytest, pytest-asyncio, pytest-cov, python-dotenv, python-multipart, requests, uvicorn[standard] - frontend/package.json: @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@16.1.6, next@16.1.6, react@19.2.3, react-dom@19.2.3, react-markdown@^10.1.0, tailwindcss@^4, typescript@^5 ### Recent commits (newest first) - reverted changes - Enhance README with architecture and knowledge graph details - planning expert and readme - planning experts - readme and planner experts - Merge pull request #2 from kwangel/gina/kg - main conflicts - Merge remote-tracking branch 'origin/main' into gina/kg - 11/12 pass - mixture of experts - others - works - Add agent frontend, voice agent, and clinical validation - knowledge graph and constraint engine - planner and skeleton code - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### backend/requirements.txt ``` fastapi uvicorn[standard] pydantic pydantic-settings openai anthropic python-multipart python-dotenv neo4j requests # Testing pytest pytest-asyncio pytest-cov httpx ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "next": "16.1.6", "react": "19.2.3", "react-dom": "19.2.3", "react-markdown": "^10.1.0" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.1.6", "tailwindcss": "^4", "typescript": "^5" } } ``` ### frontend/app/page.tsx ```typescript import Chat from "@/components/Chat"; export default function Home() { return ; } ``` ### frontend/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "CareGraph — AI Clinical Assistant", description: "Multimodal healthcare agent. Describe symptoms, upload images, get clinical guidance.", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### backend/app/main.py ```python """ CareGraph API — session-based conversational healthcare agent. Endpoints: /sessions — create / list sessions /sessions/{id} — get / close a session /sessions/{id}/msg — send a message (runs the agent loop) /kg/* — standalone knowledge graph queries """ from __future__ import annotations import base64 import logging from typing import Optional from uuid import UUID from fastapi import FastAPI, File, Form, HTTPException, UploadFile from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import Response from app.agent import run_agent_turn from app.config import get_settings from app.intake import transcribe_audio from app.knowledge import get_knowledge_graph from app.schemas import ( MessageRequest, PatientContext, ) from app.sessions import get_session_store logger = logging.getLogger(__name__) app = FastAPI( title="CareGraph API", description="Multimodal healthcare agent — chat with an AI clinical assistant", version="0.3.0", ) app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # ── Startup: warm up Modal experts in the background ───────────────────────── @app.on_event("startup") async def _startup_warm_up(): """Fire lightweight requests to Modal endpoints so containers are warm by the time the user sends their first message.""" import asyncio settings = get_settings() if settings.moe_enabled: from app.moe import warm_up_modal_experts # Run in background — don't block server startup asyncio.create_task(warm_up_modal_experts()) # ── Session Endpoints ───────────────────────────────────────────────────────── class CreateSessionRequest(PatientContext): """Optionally provide patient context when creating a session.""" pass @app.post("/sessions", tags=["sessions"]) async def create_session(request: Optional[CreateSessionRequest] = None): """Create a new conversation session.""" store = get_session_store() ctx = PatientContext(**request.model_dump()) if request else None session = store.create(patient_context=ctx) return { "session_id": str(session.session_id), "created_at": session.created_at.isoformat(), "patient_context": session.patient_context.model_dump(), } @app.get("/sessions/{session_id}", tags=["sessions"]) async def get_session(session_id: UUID): """Get the full session state.""" store = get_session_store() session = store.get(session_id) if not session: raise HTTPException(status_code=404, detail="Session not found") return session.model_dump(mode="json") @app.delete("/sessions/{session_id}", tags=["sessions"]) async def close_session(session_id: UUID): """Close a session.""" store = get_session_store() session = store.get(session_id) if not session: raise HTTPException(status_code=404, detail="Session not found") store.close(session_id) return {"session_id": str(session_id), "status": "closed"} @app.get("/sessions", tags=["sessions"]) async def list_sessions(active_only: bool = True): """List all sessions.""" store = get_session_store() sessions = store.list_sessions(active_only=active_only) return { "count": len(sessions), "sessions": [ { "session_id": str(s.session_id), "created_at": s.created_at.isoformat(), "active": s.active, "escalated": s.escalated, "turns": len(s.conversation), "events": len(s.events), "plans": len(s.plans), } for s in sessions ], } # ── Chat Endpoint (the main one) ───────────────────────────────────────────── @app.post("/sessions/{session_id}/message", tags=["chat"]) async def send_message(session_id: UUID, request: MessageRequest): """ Send a message to the agent. This is the primary endpoint. The agent will: 1. Process your message (and images if provided) 2. Decide which tools to call (intake, planner, KG, executor) 3. Return a conversational response with structured results Accepts text, images, or both. """ store = get_session_store() session = store.get(session_id) if not session: raise HTTPException(status_code=404, detail="Session not found") if not session.active: raise HTTPException(status_code=410, detail="Session is closed") response = await run_agent_turn( session_id=session_id, user_text=request.text, image_urls=request.image_urls if request.image_urls else None, ) return response.model_dump(mode="json") # ── Voice Endpoints ─────────────────────────────────────────────────────────── async def _synthesize_speech(text: str) -> bytes: """Convert text → speech using OpenAI TTS. Returns MP3 bytes.""" settings = get_settings() if not settings.openai_api_key: raise RuntimeError("TTS requires OPENAI_API_KEY.") from openai import AsyncOpenAI client = AsyncOpenAI(api_key=settings.openai_api_key) # Trim to ~4000 chars (TTS limit is ~4096) trimmed = text[:4000] response = await client.audio.speech.create( model=settings.tts_model, voice=settings.tts_voice, input=trimmed, response_format="mp3", ) return response.content @app.post("/sessions/{session_id}/voice-message", tags=["voice"]) async def voice_message( session_id: UUID, audio: UploadFile = File(...), text: Optional[str] = Form(None), ): """ Send a voice message to the agent. 1. Transcribes the audio via Whisper 2. Sends the transcript (+ optional text) through the agent pipeline 3. Generates a TTS audio reply 4. Returns the full MessageResponse + transcript + base64 audio """ store = get_session_store() session = store.g [truncated — 4896 more characters] ``` ### frontend/next.config.ts ```typescript import type { NextConfig } from "next"; const nextConfig: NextConfig = { /* config options here */ }; export default nextConfig; ``` ### backend/migrate_to_neo4j.py ```python import argparse import sys import os from typing import Any from dotenv import load_dotenv load_dotenv() try: from neo4j import GraphDatabase except ImportError: print("❌ neo4j package not installed. Install with: pip install neo4j") sys.exit(1) from app.knowledge import get_knowledge_graph, ConceptType, RelationType from app.utils import normalize_id, normalize_name def clear_database(driver): with driver.session() as session: print("🗑️ Clearing existing database...") session.run("MATCH (n) DETACH DELETE n") print("✅ Database cleared") def create_indexes(driver): with driver.session() as session: print("📇 Creating indexes...") indexes = [ "CREATE INDEX concept_id IF NOT EXISTS FOR (c:Concept) ON (c.id)", "CREATE INDEX concept_name IF NOT EXISTS FOR (c:Concept) ON (c.name)", ] for index_query in indexes: session.run(index_query) print("✅ Indexes created") def migrate_concepts(driver, kg): with driver.session() as session: print(f"📦 Migrating {len(kg.concepts)} concepts...") for concept_id, concept in kg.concepts.items(): # 1. Normalize ID and Name clean_id = normalize_id(concept.id) clean_name = normalize_name(concept.name) # 2. Standardize label (drug_class -> DrugClass) label = ''.join(word.capitalize() for word in concept.type.value.replace('_', ' ').split()) # 3. Explicitly add :Concept label to EVERYTHING (including LabTests) query = f""" MERGE (c:Concept {{id: $id}}) SET c:Concept:{label}, c.name = $name, c.type = $type, c.snomed_code = $snomed_code, c.icd10_code = $icd10_code, c.rxnorm_code = $rxnorm_code, c.synonyms = $synonyms """ session.run( query, id=clean_id, name=clean_name, type=concept.type.value, snomed_code=concept.snomed_code, icd10_code=concept.icd10_code, rxnorm_code=concept.rxnorm_code, synonyms=concept.synonyms, ) print(f"✅ {len(kg.concepts)} concepts migrated") def migrate_relationships(driver, kg): with driver.session() as session: print(f"🔗 Migrating {len(kg.relationships)} relationships...") success_count = 0 fail_count = 0 for rel in kg.relationships: rel_type = rel.relation.value.upper() # 4. Use same normalization as concepts to ensure a match source_id = normalize_id(rel.source_id) target_id = normalize_id(rel.target_id) query = f""" MATCH (source:Concept {{id: $source_id}}) MATCH (target:Concept {{id: $target_id}}) MERGE (source)-[r:{rel_type}]->(target) SET r += $metadata RETURN count(r) as created """ result = session.run( query, source_id=source_id, target_id=target_id, metadata=rel.metadata, ).single() if result and result["created"] > 0: success_count += 1 else: print(f"⚠️ Failed to link: {source_id} -> {target_id} (Nodes not found)") fail_count += 1 print(f"✅ Relationships: {success_count} created, {fail_count} failed") def verify_migration(driver): with driver.session() as session: print("\n🔍 Verifying migration...") node_count = session.run("MATCH (n) RETURN count(n) as count").single()["count"] rel_count = session.run("MATCH ()-[r]->() RETURN count(r) as count").single()["count"] print(f" Total nodes: {node_count}") print(f" Total relationships: {rel_count}") # Check for LabTests missing the Concept label missing_label = session.run("MATCH (n:LabTest) WHERE NOT n:Concept RETURN count(n) as c").single()["c"] if missing_label > 0: print(f"❌ Alert: {missing_label} LabTest nodes are still missing the Concept label!") print("\n✅ Verification complete") def main(): parser = argparse.ArgumentParser(description="Migrate CareGraph to Neo4j") parser.add_argument("--uri", default=os.getenv("NEO4J_URI"), help="Neo4j URI") parser.add_argument("--user", default=os.getenv("NEO4J_USER", "neo4j"), help="Neo4j username") parser.add_argument("--password", default=os.getenv("NEO4J_PASSWORD"), help="Neo4j password") parser.add_argument("--clear", action="store_true", help="Clear database before migration") args = parser.parse_args() if not args.uri or not args.password: print("❌ Error: Neo4j URI and Password must be provided.") sys.exit(1) driver = GraphDatabase.driver(args.uri, auth=(args.user, args.password)) try: driver.verify_connectivity() kg = get_knowledge_graph() if args.clear: clear_database(driver) create_indexes(driver) migrate_concepts(driver, kg) migrate_relationships(driver, kg) verify_migration(driver) print("\n🚀 Success! CareGraph is now fully connected.") finally: driver.close() if __name__ == "__main__": main() ``` ### modal_services/planner_experts.py ```python """ CareGraph Planner MoE — Modal-hosted biomedical LLM experts. Deploys two specialist medical LLMs as GPU-backed endpoints on Modal: 1. OpenBioLLM-8B — Llama 3 fine-tuned on PubMed, clinical trials, medical textbooks 2. BioMistral-7B — Mistral fine-tuned on PubMed Central articles These models provide FREE-TEXT clinical reasoning (not JSON) which the judge model in the backend synthesizes into the final structured plan. This plays to their strengths: deep medical knowledge without requiring rigid output formats. Deploy: modal deploy modal_services/planner_experts.py Endpoints created: /openbio/plan — OpenBioLLM-8B clinical reasoning /biomistral/plan — BioMistral-7B clinical reasoning /health — health check """ import modal from pydantic import BaseModel app = modal.App("caregraph-planner-experts") # ── Request model (Pydantic) — required so FastAPI reads JSON body ─────────── class PlanRequest(BaseModel): patient_prompt: str kg_context: str = "" # ── Shared vLLM image ──────────────────────────────────────────────────────── vllm_image = ( modal.Image.debian_slim(python_version="3.11") .pip_install( "vllm>=0.4.0", "pydantic>=2.0.0", "fastapi[standard]>=0.110.0", ) ) # ── Biomedical expert system prompt (free-text, not JSON) ──────────────────── BIOMEDICAL_SYSTEM_PROMPT = """You are a biomedical clinical reasoning expert. Analyze the patient case and provide your clinical assessment. IMPORTANT: Your assessment must be PROPORTIONAL to the symptoms described. A mild headache gets a mild assessment. A severe headache with neurological symptoms gets an urgent assessment. Do NOT list extreme diagnoses (stroke, meningitis, cancer) unless the symptoms genuinely support them. Mentioning rare, serious conditions for common mild symptoms causes harm by triggering unnecessary emergency escalation. Cover these areas in your response: 1. DIFFERENTIAL DIAGNOSIS — List the 3-5 MOST LIKELY conditions given the actual symptoms. For each, explain which symptoms support or argue against it. Rate confidence (low/medium/high). Order by probability, not severity. Only include serious conditions if the symptoms actually warrant considering them. 2. RED FLAGS — ONLY flag emergency warning signs if the patient's symptoms actually match them. Do NOT list red flags that are merely theoretically possible for the symptom category. Example: a mild headache with no neurological symptoms does NOT warrant flagging stroke. A sudden severe "worst headache of my life" with neck stiffness DOES warrant flagging SAH. 3. RISK ASSESSMENT — Rate overall urgency based on the ACTUAL presentation: - Self-care (minor, manageable at home) - Telehealth (needs doctor but not urgent) - Primary care (schedule appointment within days) - Urgent care (same-day evaluation needed) - Emergency (go to ER / call 911 now) Most common symptoms with mild severity are self-care or telehealth. Reserve "emergency" for presentations with genuine emergency features. 4. RECOMMENDED ACTIONS — What should the patient do? Match urgency to the actual symptoms. 5. KEY QUESTIONS — What additional information would help narrow the diagnosis? 6. WHEN TO SEEK EMERGENCY CARE — List specific warning signs that would change this from a mild case to an emergency (e.g., "Seek ER if headache suddenly becomes the worst of your life, or you develop weakness, vision changes, or confusion"). Be thorough but PROPORTIONAL. Clinical wisdom means matching the response to the presentation. Do NOT use JSON. Write in clear, organized prose or bullet points.""" # ── OpenBioLLM-8B Expert ──────────────────────────────────────────────────── @app.cls( image=vllm_image, gpu="A10G", timeout=180, scaledown_window=300, ) class OpenBioLLMExpert: """OpenBioLLM-8B — Llama 3 fine-tuned on PubMed, clinical trials, and medical textbooks.""" @modal.enter() def load_model(self): from vllm import LLM, SamplingParams self.llm = LLM( model="aaditya/Llama3-OpenBioLLM-8B", dtype="float16", max_model_len=4096, trust_remote_code=True, ) self.sampling_params = SamplingParams( temperature=0.3, max_tokens=2000, top_p=0.9, ) @modal.fastapi_endpoint(method="POST") def plan(self, request: PlanRequest): """Generate clinical reasoning from a patient case.""" full_prompt = request.patient_prompt if request.kg_context: full_prompt += f"\n\n{request.kg_context}" # Llama 3 chat template messages = f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|> {BIOMEDICAL_SYSTEM_PROMPT}<|eot_id|><|start_header_id|>user<|end_header_id|> {full_prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|> """ outputs = self.llm.generate([messages], self.sampling_params) raw_text = outputs[0].outputs[0].text.strip() return { "expert": "openbio_8b", "model": "OpenBioLLM-8B (Llama 3, PubMed-trained)", "format": "free_text", "reasoning": raw_text, } # ── BioMistral-7B Expert ──────────────────────────────────────────────────── @app.cls( image=vllm_image, gpu="A10G", timeout=180, scaledown_window=300, ) class BioMistralExpert: """BioMistral-7B — Mistral 7B fine-tuned on PubMed Central articles.""" @modal.enter() def load_model(self): from vllm import LLM, SamplingParams self.llm = LLM( model="BioMistral/BioMistral-7B", dtype="float16", max_model_len=4096, trust_remote_code=True, ) self.sampling_params = SamplingParams( temperature=0.3, max_tokens=2000, top_p=0.9, ) @modal.fastapi_endpoint(method="POST") def p [truncated — 953 more characters] ``` ### frontend/components/TypingIndicator.tsx ```typescript "use client"; export default function TypingIndicator() { return (
CG
); } ``` ### backend/app/config.py ```python """App configuration. Required keys: - ANTHROPIC_API_KEY — planner, text extraction, and image analysis (Claude) - OPENAI_API_KEY — (optional) voice transcription (Whisper), TTS, and GPT-4o MoE expert MoE (Mixture of Experts) keys — all optional: - MOE_ENABLED — set to "true" to activate planner multi-model ensemble - MOE_OPENBIO_URL — Modal endpoint for OpenBioLLM-8B planner - MOE_BIOMISTRAL_URL — Modal endpoint for BioMistral-7B planner """ from pydantic_settings import BaseSettings from functools import lru_cache class Settings(BaseSettings): anthropic_api_key: str = "" openai_api_key: str = "" planner_model: str = "claude-sonnet-4-20250514" vision_model: str = "claude-sonnet-4-20250514" tts_model: str = "tts-1" tts_voice: str = "nova" # warm, professional voice for healthcare debug: bool = True neo4j_uri: str = "bolt://localhost:7687" neo4j_user: str = "neo4j" neo4j_password: str = "" # MoE (Mixture of Experts) settings — planner only moe_enabled: bool = False # feature flag — set to true to activate openai_planner_model: str = "gpt-4o" # GPT-4o model for planner MoE expert moe_openbio_url: str = "" # Modal OpenBioLLM-8B plan endpoint moe_biomistral_url: str = "" # Modal BioMistral-7B plan endpoint model_config = {"env_file": ".env", "env_file_encoding": "utf-8"} @lru_cache() def get_settings() -> Settings: return Settings() ``` [27 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]