# Project export: Hellocare
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2026
- Tagline: Left the doctor confused or worried you forgot something? Your grandparents probably have. Hellocare schedules, explains, and turns paperwork into clear next steps, so you focus on health, not stress.
- Devpost: https://devpost.com/software/hellocare
- GitHub: https://github.com/smallboar/HelloCare
- Demo: https://hello-care.vercel.app/
- Video: https://www.youtube.com/embed/45H1t7bpw9U?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner ([Google] Cloud AI Track)
- Team: 4 GitHub contributor(s) — William (36 commits), Andrew Zhou (29 commits), Yuchen Xin (8 commits), Cursor (2 commits)
## Devpost submission (written by the team)
### Inspiration
When my grandpa’s memory loss started to progress, doctor’s visits became overwhelming. He couldn’t remember what the doctor said, what medications changed, or what he was supposed to do next. My family tried to fill in the gaps, but between rushed appointments, medical jargon, and dense discharge paperwork, critical details slipped through the cracks. This isn’t unique to us. Around 82% of older adults say the healthcare system isn’t prepared to meet their needs (Source 1), and nearly 1 in 4 seniors report avoiding care because it’s confusing or hard to navigate (Source 2). For patients with cognitive decline, that kind of friction isn’t just frustrating, it’s also dangerous. Missed instructions can mean missed medications, missed follow ups, or in the worst case, worsening health. Hellocare was built because of that reality. We created a healthcare copilot that manages the entire doctor’s visit lifecycle: we schedule appointments, capture and explain visits, and turn complex paperwork into clear, structured next steps. That way, patients with memory challenges and their families don’t have to rely on memory recall alone. This isn’t just about efficiency. It’s about protecting dignity, reducing stress, and ensuring that no family navigating dementia has to guess what comes next. As a result, we built an app that would give people – especially the elderly or those with limited english proficiency (LEP) – a one-stop-shop to understand and act on their own health information. Our app aims to simplify the end-to-end process of planning for, scheduling, and understanding the results of a patient’s doctor’s appointment. To achieve this goal, we focused on simplifying this process by capturing data (patient/doctor conversations, day-to-day health notes, documents, etc), and parsing these into more readily understandable formats, especially to elders and those with LEP. Source 1: https://www.johnahartford.org/images/uploads/resources/The_Growing_Demand_for_Age-Friendly_Care_Report_FINAL.pdf Source 2: https://pubmed.ncbi.nlm.nih.gov/29249189
### What it does
Health Notes & Action Items Record a visit or health concern by voice. The app converts it into structured notes and automatically extracts follow-up tasks. Context-Aware AI Chat Assistant Answers questions strictly using the user’s own data (notes, appointments, documents, past sessions). If information isn’t available, it asks for further information and doesn’t try to guess. Respects the user’s preferred language. Document Scanning Upload or photograph labs, prescriptions, or summaries. A vision LLM generates concise, searchable summaries stored for future reference. Appointments & Visit Flow Schedule and manage appointments. After visits, record or paste conversations to generate structured summaries with key topics and action items. Voice Everywhere Real-time speech-to-text for notes and chat, plus outbound call flows directly from the app. Automatic Voice Agent Scheduling An AI voice agent can call clinics or hospitals to schedule, confirm, or reschedule appointments on the user’s behalf. Multilingual Support UI and AI responses available in multiple languages, including English, Spanish, Mandarin, Cantonese, Korean, Japanese, Vietnamese, Tagalog, Arabic, Portuguese, and more.
### How we built it
Frontend Next.js 16 (App Router), React 19, Tailwind CSS, and Motion for animations. The dashboard includes navigation to home, action items, health notes, appointments, sessions, visit flow, and documents. Backend & Data Next.js API routes with Firebase Auth and Firestore for user data, notes, tasks, sessions, appointments, and document summaries. Firestore security rules enforce per-user access control. AI Layer Vercel AI SDK with OpenAI models (GPT-4o-mini for chat, GPT-4o for vision). Structured outputs enforced with Zod schemas for visit summaries, action items, note extraction, and document parsing. System prompts are constructed from user-specific context with strict “use only this information” and anti-hallucination rules. Voice AssemblyAI streaming transcription for in-app speech input. Vapi SDK for outbound AI voice calls to clinics with customizable patient and assistant names. Internationalization (i18n) Language preferences propagate across UI and all LLM calls to ensure summaries and chat responses match the user’s selected language.
### Challenges we ran into
Keeping the Chat Grounded Ensuring the model only used provided context required strict prompt structure, explicit “do not invent” rules, and clear sectioning of user data. Turning Voice into Structured Data Separating clinical content from small talk and consistently outputting schema-validated summaries required careful prompt engineering. Real-Time Transcription UX Handling streaming transcripts, turn detection, token provisioning, and error states while maintaining responsiveness. **Document Pipeline Accuracy Balancing summary accuracy with token efficiency so vision outputs could reliably feed into future chat context.
### Accomplishments we're proud of
End-to-end voice and document pipelines that convert raw input into structured, queryable data End-to-end voice and document pipelines that convert raw input into structured, queryable data A grounded, context-aware chat assistant with explicit anti-hallucination safeguards A grounded, context-aware chat assistant with explicit anti-hallucination safeguards Structured LLM outputs across the app for reliability and frontend simplicity Structured LLM outputs across the app for reliability and frontend simplicity Multilingual support built into both UI and AI layers Multilingual support built into both UI and AI layers Vision-based document scanning integrated directly into conversational workflows Vision-based document scanning integrated directly into conversational workflows
### What we learned
Product & AI Design Learnings Filtering clinical content requires explicit exclusion rules and speaker attribution guidance. Filtering clinical content requires explicit exclusion rules and speaker attribution guidance. Clear prompt sectioning significantly reduces hallucination. Clear prompt sectioning significantly reduces hallucination. Structured outputs are more reliable than parsing free text in healthcare workflows. Structured outputs are more reliable than parsing free text in healthcare workflows. Technical Learnings Designing for elderly users increases complexity. Simple UI patterns like dropdowns or loading states can create confusion. Accessibility and cognitive load directly impact architecture decisions. We also learned that integrating AI into healthcare requires guardrails, determinism, and transparency. Trust matters more than model sophistication. Broader Learnings Healthcare challenges are often coordination failures rather than treatment failures. Many frustrations stem from document comprehension, not clinical effectiveness. Trust is earned through reliability and clarity. Care involves multiple stakeholders: patients, providers, and caregivers. Humanity remains central to healthcare. Even powerful AI must support, not replace, human relationships.
### What's next
HIPAA Compliance Expansion Firebase Auth and Firestore are HIPAA-compliant, and our scheduling voice assistant infrastructure supports compliance. The next step is integrating fully HIPAA-compliant LLM providers that sign BAAs to ensure end-to-end protection. Deeper System Integration Connect with hospital systems (e.g., FHIR, patient portals) for real-time syncing of appointments and results. Care Circle Allow users to securely share selected summaries or documents with family members and care coordinators. Proactive Reminders Send reminders and surface “questions to ask” based on upcoming appointments and recent notes. Richer Document Support Enable multi-page PDF uploads and structured extraction of medications, labs, and timelines. Accessibility & Transparency Expand voice-first workflows and add “show source” explanations for AI-generated answers.
## README (from the GitHub repository)
This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app).
## Getting Started
First, run the development server:
```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```
Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.
You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.
This project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel.
## Learn More
To learn more about Next.js, take a look at the following resources:
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome!
## Deploy on Vercel
The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.
Check out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details.
## Detected evidence (automated analysis)
Indexed codebase: 93 recognized source files, 418 KB.
- Anthropic (technology) — detected in the code
- CSS (language) — detected in the code
- Firebase (technology) — detected in the code
- Next.js (technology) — detected in the code
- OpenAI (technology) — detected in the code
- React (technology) — detected in the code
- Tailwind CSS (technology) — detected in the code
- TypeScript (language) — detected in the code
- Vercel AI SDK (technology) — detected in the code
- AI coding agent: Cursor — evidence: commit authorship or trailers
## Codebase structure (from repository index)
### Files (101 of 101)
```
.firebaserc
.gitignore
app/(dashboard)/action-items/page.tsx
app/(dashboard)/appointments/conversation/page.tsx
app/(dashboard)/appointments/conversation/types.ts
app/(dashboard)/appointments/conversation/utils.ts
app/(dashboard)/appointments/conversation/views/ConfirmedView.tsx
app/(dashboard)/appointments/conversation/views/IdleView.tsx
app/(dashboard)/appointments/conversation/views/index.tsx
app/(dashboard)/appointments/conversation/views/RecordingView.tsx
app/(dashboard)/appointments/conversation/views/RetryView.tsx
app/(dashboard)/appointments/conversation/views/SummaryView.tsx
app/(dashboard)/appointments/conversation/views/VisitSummaryView.tsx
app/(dashboard)/appointments/page.tsx
app/(dashboard)/appointments/schedule/page.tsx
app/(dashboard)/DashboardShell.tsx
app/(dashboard)/documents/page.tsx
app/(dashboard)/health-notes/page.tsx
app/(dashboard)/layout.tsx
app/(dashboard)/page.tsx
app/(dashboard)/past-sessions/page.tsx
app/(dashboard)/scan-documents/page.tsx
app/api/assemblyai-token/route.ts
app/api/chat/route.ts
app/api/confirmTimeslot/route.ts
app/api/conversation-summary-from-transcript/route.ts
app/api/document-summary-from-images/route.ts
app/api/health-note-from-transcript/route.ts
app/api/proposeTimeslots/route.ts
app/api/vapi/route.ts
app/api/visit-summary-from-transcript/route.ts
app/api/voice-command/route.ts
app/auth/page.tsx
app/components/Brand.tsx
app/components/ChatWidget.tsx
app/components/Drawer.tsx
app/components/HomeSummary.tsx
app/components/I18nProvider.tsx
app/components/index.ts
app/components/MarkdownContent.tsx
app/components/PillDropdown.tsx
app/components/RecordHealthNoteModal.tsx
app/components/RouteGuard.tsx
app/components/Spinner.tsx
app/components/StreamingText.tsx
app/components/Toast.tsx
app/components/VoiceCommandFAB.tsx
app/globals.css
app/hooks/useStreamingTranscription.ts
app/hooks/useToolExecutor.ts
app/layout.tsx
app/onboarding/components/Step0.tsx
app/onboarding/components/Step1.tsx
app/onboarding/components/Step2.tsx
app/onboarding/components/Step3.tsx
app/onboarding/components/Step4.tsx
app/onboarding/components/StepWrapper.tsx
app/onboarding/page.tsx
app/onboarding/types.ts
CHAT_ACTIONS_TOOLS.md
eslint.config.mjs
firebase.json
firestore.rules
lib/assemblyai/keyterms.ts
lib/assemblyai/turn-detection-config.ts
lib/assistant-tools.ts
lib/auth-context.tsx
lib/chat-actions.ts
lib/chat-system-prompt.ts
lib/firebase-admin.ts
lib/firebase.ts
lib/firestore/actionItems.ts
lib/firestore/api.ts
lib/firestore/collections.ts
lib/firestore/entryFormConfig.ts
lib/firestore/healthNotes.ts
lib/firestore/hooks.ts
lib/firestore/index.ts
lib/firestore/serialize.ts
lib/firestore/sessions.ts
lib/firestore/types.ts
lib/i18n/format.ts
lib/i18n/index.ts
lib/i18n/locales.ts
lib/i18n/messages.ts
lib/llm/client.ts
lib/llm/index.ts
lib/llm/provider.ts
lib/llm/queries/conversation-summary.ts
lib/llm/queries/document-from-images.ts
lib/llm/queries/health-note.ts
lib/llm/queries/visit-summary.ts
lib/llm/types.ts
lib/logger.ts
lib/timeslot-store.ts
lib/ui/form-styles.ts
next.config.ts
package.json
postcss.config.mjs
README.md
tsconfig.json
```
### Dependencies
- package.json: @ai-sdk/anthropic@^3.0.44, @ai-sdk/openai@^3.0.29, @ai-sdk/react@^3.0.88, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, @vapi-ai/server-sdk@^0.11.0, ai@^6.0.86, assemblyai@^4.23.0, eslint@^9, eslint-config-next@16.1.6, firebase@^12.9.0, firebase-admin@^13.6.1, motion@^12.34.0, next@16.1.6, react@19.2.3, react-dom@19.2.3, react-icons@^5.5.0, react-markdown@^10.1.0, tailwindcss@^4, typescript@^5, zod@^4.3.6
### Recent commits (newest first)
- Fix: better error logging
- Migrate to firestore admin for vapi flow
- Fix broken drawer routes
- Merge remote-tracking branch 'origin/main'
- Merge branch 'wfang/claude-attempt-speech-controlled'
- Add docuemnts to the context for chat LLM (#29)
- Improve voice input send flow and FAB transcript display
- Update: better logging for vapi flow
- Strengthen core mcp tooling
- Add documents and fix camera staying on bug (#28)
- Remove accidental MCP server files from main
- Fix: deployment, Add: proper appointment writing flow
- Add document scanner, summarizer, and uploader (#27)
- Update conversation card styling and add MCP server scaffold
- Claude temp
- Add scheduling confirmation timeout and zero timeslots handling
- Merge branch 'i18n'
- Add app-wide i18n and multilingual transcription support
- Give appointment context and change prompt for chat service (#24)
- Add icons to homescreen widget (#23)
## Key source files (fetched from GitHub, selected and truncated for size)
### CHAT_ACTIONS_TOOLS.md
```markdown
# Chat Actions And Tools
Use this as the canonical tool list when prompting the LLM for tool calling.
## Actions -> Tool Names
1. Go to a page -> `navigate`
2. Update an action item -> `update_action_item`
3. Delete an action item -> `delete_action_item`
4. Delete a health note -> `delete_health_note`
5. Update health-note type -> `update_health_note_type`
6. Delete an appointment -> `delete_appointment`
7. Delete a past session -> `delete_session`
8. Open voice recorder for health notes -> `open_health_note_recorder`
9. Create an action item -> `create_action_item`
10. Create a health note (text) -> `create_health_note`
11. Create an appointment -> `create_appointment`
12. Create a past session record -> `create_session`
## Tool Inputs
### `navigate`
- Required: `page`
- Optional: `highlightId`
- `page` enum: `home`, `action_items`, `health_notes`, `appointments`, `past_sessions`, `schedule_appointment`, `doctor_visit_conversation`
### `update_action_item`
- Required: `id`
- Optional: `status`, `priority`, `type`
- `status` enum: `pending`, `in_progress`, `done`, `skipped`
- `priority` enum: `low`, `medium`, `high`
- `type` enum: `Medication`, `Exercise`, `Appointment`, `Other`
### `delete_action_item`
- Required: `id`
### `delete_health_note`
- Required: `id`
### `update_health_note_type`
- Required: `id`, `type`
- `type` enum: `Injury`, `Recurring pain`, `Temporary pain`
### `delete_appointment`
- Required: `id`
### `delete_session`
- Required: `id`
### `open_health_note_recorder`
- Required: none
### `create_action_item`
- Required: `title`
- Optional: `description`, `type`, `priority`, `dueBy`
- `dueBy` should be ISO 8601
### `create_health_note`
- Required: `title`, `description`
- Optional: `type`
- `type` enum: `Injury`, `Recurring pain`, `Temporary pain`
### `create_appointment`
- Required: `appointmentTime` (ISO 8601)
### `create_session`
- Required: `title`
- Optional: `summary`, `date` (ISO 8601)
## Accuracy Rules For LLM
1. Never invent IDs. Use IDs from context only.
2. If multiple records match and user did not disambiguate, ask a clarifying question.
3. If exactly one record clearly matches, call the tool with that ID.
4. For "all" requests, call the tool once per matching record.
5. Use `open_health_note_recorder` only for voice-record requests, not text note creation.
6. Use `navigate` only when user asks to move pages.
7. If no tool is needed, answer directly from context.
```
### package.json
```
{
"name": "hellocare",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"lint": "eslint"
},
"dependencies": {
"@ai-sdk/anthropic": "^3.0.44",
"@ai-sdk/openai": "^3.0.29",
"@ai-sdk/react": "^3.0.88",
"@vapi-ai/server-sdk": "^0.11.0",
"ai": "^6.0.86",
"assemblyai": "^4.23.0",
"firebase": "^12.9.0",
"firebase-admin": "^13.6.1",
"motion": "^12.34.0",
"next": "16.1.6",
"react": "19.2.3",
"react-dom": "19.2.3",
"react-icons": "^5.5.0",
"react-markdown": "^10.1.0",
"zod": "^4.3.6"
},
"devDependencies": {
"@tailwindcss/postcss": "^4",
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"eslint": "^9",
"eslint-config-next": "16.1.6",
"tailwindcss": "^4",
"typescript": "^5"
}
}
```
### app/layout.tsx
```typescript
import { AuthProvider } from "@/lib/auth-context";
import { I18nProvider } from "@/app/components/I18nProvider";
import { RouteGuard } from "./components/RouteGuard";
import type { Metadata } from "next";
import { Geist, Geist_Mono, Manrope } from "next/font/google";
import "./globals.css";
const geistSans = Geist({
variable: "--font-geist-sans",
subsets: ["latin"],
});
const geistMono = Geist_Mono({
variable: "--font-geist-mono",
subsets: ["latin"],
});
const manrope = Manrope({
variable: "--font-manrope",
subsets: ["latin"],
});
export const metadata: Metadata = {
title: "HelloCare",
description: "HelloCare",
};
export default function RootLayout({
children,
}: Readonly<{
children: React.ReactNode;
}>) {
return (
{children}
);
}
```
### lib/i18n/index.ts
```typescript
export { LANGUAGE_OPTIONS, DEFAULT_LANGUAGE_TAG, resolveAppLocale, resolveLanguageTag } from "./locales";
export { EN_MESSAGES, ES_MESSAGES, MESSAGES, type MessageKey } from "./messages";
export { formatLocalizedDate, formatLocalizedDateUTC, formatRelativeFromNow } from "./format";
```
### lib/llm/index.ts
```typescript
// Provider
export { createModel } from "./provider";
export type { LLMProviderName } from "./provider";
// Client helpers
export { queryLLMStructured, queryLLMText } from "./client";
// Types
export type {
StructuredQueryOptions,
StructuredQueryResult,
TextQueryOptions,
TextQueryResult,
LanguageModelUsage,
} from "./types";
```
### app/components/index.ts
```typescript
/**
* App-level UI components.
* Re-export for convenient imports: import { Drawer } from "@/app/components";
*/
export { Drawer } from "./Drawer";
export { ChatWidget } from "./ChatWidget";
export { HomeSummary } from "./HomeSummary";
export { MarkdownContent } from "./MarkdownContent";
export { PillDropdown } from "./PillDropdown";
export { StreamingText } from "./StreamingText";
export { Toast } from "./Toast";
export { VoiceCommandFAB } from "./VoiceCommandFAB";
```
### app/auth/page.tsx
```typescript
'use client';
import { useI18n } from "@/app/components/I18nProvider";
import { useAuth } from "@/lib/auth-context";
import { useRouter } from "next/navigation";
import { useEffect } from "react";
import { Brand } from "../components/Brand";
import { FcGoogle } from "react-icons/fc";
export default function Login() {
const { t } = useI18n();
const { user, loading, signInWithGoogle } = useAuth();
const router = useRouter();
useEffect(() => {
if (!loading && user) {
router.replace("/");
}
}, [user, loading, router]);
if (loading || user) return null;
return