Project Info
Inspiration
https://www.reddit.com/r/ArtificialInteligence/comments/1bsf9mi/my_parents_are_getting_dementia_can_ai_help/ Dementia is a progressive neurological condition affecting over 55 million people worldwide. It impairs memory, language, reasoning, and the ability to perform daily activities. While there is currently no cure, treatments focus on slowing cognitive decline, maintaining function, and improving quality of life. One evidence-based approach to combating dementia is Cognitive Stimulation Therapy (CST): a structured program of themed activities designed to actively stimulate thinking, concentration, memory, and social interaction. CST sessions, typically conducted regularly, have been shown to be more effective in groups rather than individually (https://pubmed.ncbi.nlm.nih.gov/34942157/). The social component of CST encourages shared attention, conversation, and peer engagement, which amplifies its therapeutic impact. However, dementia patients in rural or underserved areas often lack access to local CST groups and may need to travel long distances to participate in sessions. We also wanted to raise awareness for the incredibly difficult work that many caregivers go through, drawing inspiration from the stories shared in communities like r/dementia, where family members openly describe the emotional exhaustion, financial strain, role reversal, and constant uncertainty that come with caring for a loved one with cognitive decline—often while balancing jobs, parenting, and their own mental health, with little formal support. We noticed positive support of AI assisted therapy for dementia patients by caregivers on the posted link and felt that a project like ours would be appropriate method for alleviating some of the stress that caregivers go through.
What it does
Patronum is an intelligent platform that enables dementia patients to participate in simulated group CST sessions with humanlike AI participants and a moderator. Designed as a supplement to in-person therapy, it allows patients to engage in more frequent sessions at their own convenience. It has 4 main features: Simulated Group Sessions: An AI moderator guides structured CST activities with up to 5 AI participants and the patient, preserving the social dynamics critical to CST's effectiveness. To put it simply, we emulate a group therapy session with AI agents where patients are asked mentally stimulating questions that relate to reminiscent and sensory experiences rather than factual recall. For example: What do you miss from your childhood? What brings you back to your favorite memories? Simulated Group Sessions: An AI moderator guides structured CST activities with up to 5 AI participants and the patient, preserving the social dynamics critical to CST's effectiveness. To put it simply, we emulate a group therapy session with AI agents where patients are asked mentally stimulating questions that relate to reminiscent and sensory experiences rather than factual recall. For example: What do you miss from your childhood? What brings you back to your favorite memories? Photo-Based Memory Stimulation: Sessions incorporate family-uploaded photos, prompting participants to describe, interpret, and react to visual cues to strengthen recall and communication skills. Photo-Based Memory Stimulation: Sessions incorporate family-uploaded photos, prompting participants to describe, interpret, and react to visual cues to strengthen recall and communication skills. Caregiver Dashboard: Caregivers can assign sessions, upload photos, and view an analytics dashboard with session summaries, engagement metrics, memory performance trends, and social interaction data. We provide research backed statistics that caregivers can refer to the information we provide to track the progress of their patients. Caregiver Dashboard: Caregivers can assign sessions, upload photos, and view an analytics dashboard with session summaries, engagement metrics, memory performance trends, and social interaction data. We provide research backed statistics that caregivers can refer to the information we provide to track the progress of their patients. Cognitive Games: After a session, the patient is presented with a cognitive game that is related to the session (i.e. questions like who asked certain questions). We employ Error-Less Learning in these games, ensuring that patients have immediate access to the correct answers to the questions after answering them. This is done to ensure that patients have a relatively guilt-free environment. Cognitive Games: After a session, the patient is presented with a cognitive game that is related to the session (i.e. questions like who asked certain questions). We employ Error-Less Learning in these games, ensuring that patients have immediate access to the correct answers to the questions after answering them. This is done to ensure that patients have a relatively guilt-free environment. Additional features Session summaries written in plain language so caregivers can quickly understand how the patient engaged, without reviewing full transcripts Automated alerts when engagement or performance drops significantly across sessions Adjustable session length and difficulty to better match disease stage and patient stamina
How we built it
Our tech stack combines modern tools for a robust, scalable solution: Frontend: Built with React for a responsive, intuitive interface. Complex state management handles real-time session data and large analytics datasets seamlessly. Backend: Handles session orchestration, user management, and data persistence with a focus on low-latency performance across concurrent AI participants. AI Processing: Powered by Claude, ElevenLabs, and HeyGen, multiple AI agents run in real time with optimized prompting to keep conversations natural and responsive. HeyGen allows for realistic appearing agents, while ElevenLabs powers the agents’ speech. Real-time Updates: WebSocket connections ensure live session updates and smooth caregiver dashboard experiences.
Challenges we ran into
Prompt Optimization; We struggled with keeping agents speaking in a natural way, making the conversation feel realistic. Safety: We needed quite a lot of testing to ensure that the AI wouldn’t go past the guardrails we set. Meaningful Metrics: We did lots of research into what insights would be actionable for the caretaker but also generatable by AI. Accomplishments we’re proud of Building a realistic group CST experience that preserves the social interaction central to its effectiveness Creating caregiver analytics that surface meaningful trends instead of raw data Demonstrating how generative AI can support evidence-based healthcare interventions rather than replace human care
What we learned
Advanced video processing techniques in the browser Real-time data handling with WebSocket connections to handle real-time updates effectively AI model optimization for edge cases Complex state management in React applications, especially when dealing with large datasets Integration of multiple third-party services The importance of user experience in security applications
What's next
Future enhancements we're planning: Multiple language support Integration with existing healthcare systems to facilitate session assignment by healthcare providers Wider variety of conversation topics and questions available Data encryption to protect patient privacy Implementation of multiple papers used to protect against AI hallucination and malfunctions Built With HeyGen, ElevenLabs, Anthropic SDK (Claude), React, Vite, TypeScript, Tailwind CSS, Node.js, TypeScript, Express, Postgres, Prisma, AWS S3, Docker
Patronum
Patronum is a web platform where dementia patients join simulated Cognitive Stimulation Therapy group sessions with an AI moderator and AI participants, while caregivers assign sessions and follow progress on an analytics dashboard. Claude drives the moderator and participants, ElevenLabs and HeyGen give them voices and video avatars, a Python service scores speech graphs from session transcripts, and the project was a TreeHacks 2026 entry.
flowchart LR
web["Web<br/>TypeScript, React"] --> server["Server<br/>TypeScript, Express"]
web --> elevenlabs["ElevenLabs"]
web --> heygen["HeyGen"]
server --> db[("PostgreSQL")]
server --> speech["Speech Graph Service<br/>Python, FastAPI"]
server --> claude["Anthropic Claude API"]
server --> elevenlabs
server --> heygen
server --> s3["AWS S3"]
Analysis
View
Metric
- 8
- 4
Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.
Technology
- AnthropicIn code
- CSSIn code
- ExpressIn code
- FastAPIIn code
- HTMLIn code
- JavaScriptIn code
- PostgreSQLIn code
- PythonIn code
- ReactIn code
- Tailwind CSSIn code
- TypeScriptIn code
- AWSClaimed
- DockerClaimed
- Node.jsClaimed
11 of 14 appear in the indexed code. 3 claimed on Devpost could not be matched to code, which may simply mean the tool leaves no trace in the repository.
AI coding agents
No AI coding agent signals were found in this repository.
Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.
Codebase size
Source size
837 KB
Source files
124
Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.
Repository
RayHCai/patronum
141 files · 1.2 MB · @ bf51204
Structure
Interface
53 files · 38%Screens, components and styles rendered to the user.
API & routing
39 files · 28%Request entry points: routes, handlers and controllers.
Application logic
28 files · 20%Domain rules, services and shared utilities.
Data & schema
2 files · 1%Schema definitions, migrations and data access.
Supporting
Layers are inferred from where files sit in the tree, not from reading the code. A project that names its directories unconventionally will read oddly here — open the file browser to check anything the diagram implies.
Languages
- TypeScript96%
- Python2%
- Markdown1%
- CSS1%
- JavaScript0%
- YAML0%
- Other (1)0%
Share of indexed source by file size. Binary and vendored files are excluded.
Dependencies
server/package.json
npm · 30- @anthropic-ai/sdk
- @paralleldrive/cuid2
- aws-sdk
- bcrypt
- cors
- dotenv
- express
- express-jwt
- ffmpeg-static
- fluent-ffmpeg
- jsonwebtoken
- multer
- pg
- resend
- ws
- +15 more
client/package.json
npm · 28- @heygen/streaming-avatar
- @hookform/resolvers
- axios
- clsx
- d3
- date-fns
- framer-motion
- lucide-react
- react
- react-dom
- react-hook-form
- react-hot-toast
- react-router-dom
- reactflow
- recharts
- tailwind-merge
- wavesurfer.js
- zod
- +10 more
speech-graph-service/requirements.txt
pypi · 4- fastapi
- networkx
- pydantic
- uvicorn[standard]
package.json
npm · 3- bcryptjs
- +2 more
Declared in the repository’s manifests at the indexed commit. A declared package is not proof it is used, and runtime dependencies are listed first.
This project’s features have not been analysed yet.
Export this project's context (description, README, evidence, key source files) to chat with an AI agent elsewhere.