Inspiration

We were inspired by the challenge faced by elderly citizens, like our grandparents, who live alone and for whom small changes in health vitals can go unnoticed — even when they are early warning signs of serious complications. Given that 65 million Americans wear a smartwatch that collects heart rate, sleep, and activity data, we wanted to create a personal assistant tool that can take actions based on health trends. Healthcare is slow, and by the time someone notices a problem or books an appointment, it’s often too late for effective intervention. This inspired us to create an autonomous way to facilitate doctor and patient interactions for elderly citizens. What It Does CareSync gives every patient an autonomous AI care agent, exposed as a secure API endpoint, that monitors their health and coordinates care on their behalf. For Patients, Your Agent Ingests real Apple Watch data via an iOS + WatchOS app connected to HealthKit (9 data types including steps, heart rate, sleep stages, workouts, and irregular rhythm events). Detects anomalies automatically using a 28-day rolling baseline (mean ± std), flagging events like resting heart rate spikes and sleep drops with a 0–100 severity score. Makes autonomous decisions through a Secretary Agent using Claude's Agent SDK that weighs severity, urgency, and medical context by querying PubMed and FDA sources and escalates to a doctor’s receptionist agent. Books appointments through agent-to-agent negotiation via a 3-turn propose/counter/confirm protocol, with Ed25519 cryptographic signatures on every message. Communicates through HeyGen streaming avatars in 12+ languages (Deepgram STT + ElevenLabs TTS), allowing a Mandarin-speaking grandmother to have face-to-face conversations with her AI doctor. For Doctors, Your Agent Receives patient escalations and triages them through a LangGraph state machine using Claude. Checks Google Calendar availability and proposes appointment slots. Proposes and books appointments with full clinical context. The Platform Provides A registry where every person has a canonical API endpoint (/u/:handle). Ed25519 cryptographic signatures on every agent-to-agent request with replay protection. Policy engine support for caller whitelists, scope-based permissions, and payment gating. Full execution traces showing exactly what agents did and why. CareSync is a multi-agent orchestration and healthcare platform where each person becomes an agent endpoint. We transform static wearable health data into actionable outcomes with structured summaries, risk modeling, and scheduling. This reduces missed appointments, preventable deterioration, administrative stress, and care delays. How We Built It Architecture Monorepo with three main layers: Next.js 14 frontend (deployed on Vercel) Python Doctor Agent microservice (FastAPI) Shared type system (@people/shared) that keeps both sides synchronized through Pydantic models and Zod schemas Web Platform (Next.js 14) React 18 TailwindCSS Clerk authentication Supabase via Prisma The centerpiece is a "Secretary Agent,” an LLM orchestrator using Claude Sonnet 4.5 or GPT-4o with function-calling to: Analyze anomalies Triage patients Schedule appointments Generate health summaries Doctor Agent (FastAPI + LangGraph) A state machine modeling triage as a multi-turn workflow: receive alert → classify severity → ask follow-ups → decide action → dispatch Claude handles clinical reasoning in a multi-tool loop where it: Checks for life-threatening conditions Classifies severity on a four-tier scale Determines appointment type Checks real Google Calendar availability HeyGen Streaming Interface Instead of notifications, patients can interact with real-time streaming agents (Deepgram STT + ElevenLabs TTS) that explain anomalies and walk through next steps in 12+ languages. Claude’s reasoning outputs feed into HeyGen’s API for personalized, conversational explanations. Agent-to-Agent Communication Ed25519 cryptographic signatures on every request Canonically serialized JSON (sorted keys) Signed and verified with nonce-based replay protection 5-minute timestamp windows Policy engine validates permissions before execution Pipeline Biometric data arrives → 28-day personal baseline computed (mean ± std). Patient agent calculates anomaly score (0–100) and flags concerns. Claude evaluates anomaly, decides escalation, and drafts clinic messages. Doctor agent runs multi-turn intake, checks availability, and books appointments. Full trace persisted: alert, decision, triage outcome, calendar event. Challenges We Ran Into Incorporating multi-agent orchestration and negotiation amongst agents. A singular LLM endpoint became a bottleneck during early development. Google Calendar configuration issues: the bot created events on an invisible calendar due to hallucinated state; we learned we must explicitly specify the correct Google Calendar each time. After booking confirmation, appointments sometimes did not appear or double-booked. Our first trace system only logged "triage: ok" instead of a full traceback, limiting interpretability. Time constraints prevented building full tracing infrastructure like Langfuse. The AI makes decisions inside its own internal loop, so we had to hook into every step to capture reasoning, tool calls, and responses. Context management was difficult due to frequent pivots in use case and workflow. Accomplishments We’re Proud Of We built CareSync because our grandparents deserve the same quality of monitoring that a hospital ICU provides — but from the comfort of their home, in their own language, through a face they can trust. We built an end-to-end pipeline that actually works: A simulated Apple Watch anomaly goes in → a real Google Calendar appointment comes out. The patient agent detects the problem. The doctor agent triages it across multiple turns. A real event with clinical context is proposed and booked on the doctor's calendar. We built platform infrastructure for a healthcare agent ecosystem. The “People API” concept includes a registry, policy engine, capability discovery, and permission model. Another developer could register an agent today, define capabilities, set access policies, and plug into the network. We bridged the gap between passive data and active intervention — turning watch data into a doctor’s appointment through anomaly detection, clinical reasoning, scheduling negotiation, and calendar integration in a multi-turn agent reasoning framework. What’s Next for People API Multi-doctor routing: match patients to the right specialist based on triage outcome, availability, insurance, and location. Patient consent controls: granular settings for what your agent can share, with whom, and under what conditions. HIPAA compliance: privacy and security for clinical deployment. Tech Stack Frontend Next.js 14 (App Router) React TailwindCSS Backend Next.js API Routes FastAPI (Python) LangGraph Database Supabase PostgreSQL via Prisma Auth Clerk LLMs Anthropic Claude Sonnet (primary) OpenAI GPT-4o-mini Agent Framework Claude Agent SDK with MCP tool server Voice & Streaming Deepgram (STT) ElevenLabs (TTS) HeyGen (streaming avatar) Mobile Swift / SwiftUI HealthKit WatchOS Crypto Ed25519 (tweetnacl) for agent-to-agent signing Clinical Data Sources PubMed (NCBI) openFDA AHA / CDC / WHO guidelines Google Calendar API Deployment & Infrastructure Vercel pnpm workspaces (monorepo) Zod schemas Pydantic v2 Canonical JSON serialization