# Project export: HeatherAI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Your Story, Always Remember - Devpost: https://devpost.com/software/heatherai - GitHub: http://github.com/wesleykieu/voice-ai - Video: https://www.youtube.com/embed/ayWX-6qOLHM?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Wesley Kieu (4 commits) ## Devpost submission (written by the team) ### Inspiration Imagine slowly forgetting the people and moments that made your life meaningful. More than 10 million people are diagnosed with dementia every year, and that number continues to grow. It's heartbreaking, especially for families experiencing dementia for the first time. With HeatherAI, I wanted to bridge the gap so families don't have to bear such a heavy burden alone, while ensuring patients still receive meaningful companionship and care. ### What it does HeatherAI is a Voice AI Agent designed to help patients in elder care, specifically those with dementia, reconnect with their memories through the simple act of speaking with their younger self. Before diagnosis: HeatherAI helps you plan and prepare by recording your life story through natural text-to-speech or speech-to-text conversations. It learns your history as you go and creates a voice clone that sounds just like you. After diagnosis: When dementia sets in, you can open up and converse with HeatherAI. You might or might not realize it's your younger self speaking—but either way, it provides comfort, familiarity, and connection to your past. Core Features: Voice Cloning - Preserves your voice before memory loss begins Memory Lane - AI-powered recall of personal history and cherished moments Smart Tools - Emergency calling, safety monitoring, and dynamic agentic tool creation for personalized needs ### How we built it HeatherAI is built on a robust stack of cutting-edge technologies: ChromaDB - Vector database for semantic memory storage and retrieval LiveKit Agents - Voice AI framework with advanced VAD and turn detection ElevenLabs - Voice cloning and natural text-to-speech Twilio - Emergency calling capabilities Mailjet - Email notifications for caregivers Groq - High-speed LLM inference for natural conversations ### Challenges we ran into Multi-voice switching: I wanted users to hear from multiple loved ones—not just their own voice, but also their children, spouse, or friends. While I successfully created two voice clones with ElevenLabs, I couldn't figure out how to dynamically switch between different voice models during runtime. Hallucinations and background noise: The agent initially struggled with accuracy and was sensitive to background sounds. I solved this by refining prompts to be clear and concise, providing specific examples, and adjusting the temperature parameter. I also attended LiveKit's workshop and learned about their multilingual semantic turn detector, which layers on top of voice activity detection to filter out background noise. Connectivity issues: WiFi challenges at the hackathon made testing difficult, but persistence paid off! ### Accomplishments we're proud of ✨ Getting it working end-to-end - Connecting everything to a LiveKit room and hearing the voice model respond for the first time was magical 🛠️ Dynamic tool calling - The agent can intelligently select from a variety of tools during conversation, making memory recall seamless with ChromaDB integration 🎭 Voice quality and empathy - Testing different TTS models and seeing the voice agent display genuine empathy and understanding for users 🤖 Agentic tool creation - The most ambitious feature: when a patient needs something not covered by existing tools (like checking weather or creating reminders), our LLM dynamically creates new tools on-the-fly. This means HeatherAI can adapt to each patient's unique needs without requiring code changes. ### What we learned I discovered a passion for voice AI and its incredible potential to make a real difference in people's lives. I learned about LiveKit's system design—how voice models work from VAD to STT and beyond. I gained hands-on experience with ChromaDB for building RAG pipelines with vector databases. Most importantly, I learned that technology can preserve what makes us human: our memories, our voices, and our stories. ### What's next Multi-voice support - Enable switching between different loved ones' voices (children, spouse, friends) Caregiver dashboard - Real-time insights and alerts for family members and medical staff Mobile app - Make HeatherAI accessible anywhere, anytime Advanced memory triggers - Use photos, music, and video clips to enhance memory recall Clinical trials - Partner with elder care facilities to validate HeatherAI's impact Multilingual support - Help dementia patients worldwide, regardless of language Integration with medical records - Seamless connection with healthcare providers for coordinated care ## README (from the GitHub repository) LiveKit logo # LiveKit Agents Starter - Python A complete starter project for building voice AI apps with [LiveKit Agents for Python](https://github.com/livekit/agents) and [LiveKit Cloud](https://cloud.livekit.io/). The starter project includes: - A simple voice AI assistant, ready for extension and customization - A voice AI pipeline with [models](https://docs.livekit.io/agents/models) from OpenAI, Cartesia, and AssemblyAI served through LiveKit Cloud - Easily integrate your preferred [LLM](https://docs.livekit.io/agents/models/llm/), [STT](https://docs.livekit.io/agents/models/stt/), and [TTS](https://docs.livekit.io/agents/models/tts/) instead, or swap to a realtime model like the [OpenAI Realtime API](https://docs.livekit.io/agents/models/realtime/openai) - Eval suite based on the LiveKit Agents [testing & evaluation framework](https://docs.livekit.io/agents/build/testing/) - [LiveKit Turn Detector](https://docs.livekit.io/agents/build/turns/turn-detector/) for contextually-aware speaker detection, with multilingual support - [Background voice cancellation](https://docs.livekit.io/home/cloud/noise-cancellation/) - Integrated [metrics and logging](https://docs.livekit.io/agents/build/metrics/) - A Dockerfile ready for [production deployment](https://docs.livekit.io/agents/ops/deployment/) This starter app is compatible with any [custom web/mobile frontend](https://docs.livekit.io/agents/start/frontend/) or [SIP-based telephony](https://docs.livekit.io/agents/start/telephony/). ## Coding agents and MCP This project is designed to work with coding agents like [Cursor](https://www.cursor.com/) and [Claude Code](https://www.anthropic.com/claude-code). To get the most out of these tools, install the [LiveKit Docs MCP server](https://docs.livekit.io/mcp). For Cursor, use this link: [![Install MCP Server](https://cursor.com/deeplink/mcp-install-light.svg)](https://cursor.com/en-US/install-mcp?name=livekit-docs&config=eyJ1cmwiOiJodHRwczovL2RvY3MubGl2ZWtpdC5pby9tY3AifQ%3D%3D) For Claude Code, run this command: ``` claude mcp add --transport http livekit-docs https://docs.livekit.io/mcp ``` For Codex CLI, use this command to install the server: ``` codex mcp add --url https://docs.livekit.io/mcp livekit-docs ``` For Gemini CLI, use this command to install the server: ``` gemini mcp add --transport http livekit-docs https://docs.livekit.io/mcp ``` The project includes a complete [AGENTS.md](AGENTS.md) file for these assistants. You can modify this file your needs. To learn more about this file, see [https://agents.md](https://agents.md). ## Dev Setup Clone the repository and install dependencies to a virtual environment: ```console cd agent-starter-python uv sync ``` Sign up for [LiveKit Cloud](https://cloud.livekit.io/) then set up the environment by copying `.env.example` to `.env.local` and filling in the required keys: - `LIVEKIT_URL` - `LIVEKIT_API_KEY` - `LIVEKIT_API_SECRET` You can load the LiveKit environment automatically using the [LiveKit CLI](https://docs.livekit.io/home/cli/cli-setup): ```bash lk cloud auth lk app env -w -d .env.local ``` ## Run the agent Before your first run, you must download certain models such as [Silero VAD](https://docs.livekit.io/agents/build/turns/vad/) and the [LiveKit turn detector](https://docs.livekit.io/agents/build/turns/turn-detector/): ```console uv run python src/agent.py download-files ``` Next, run this command to speak to your agent directly in your terminal: ```console uv run python src/agent.py console ``` To run the agent for use with a frontend or telephony, use the `dev` command: ```console uv run python src/agent.py dev ``` In production, use the `start` command: ```console uv run python src/agent.py start ``` ## Frontend & Telephony Get started quickly with our pre-built frontend starter apps, or add telephony support: | Platform | Link | Description | |----------|----------|-------------| | **Web** | [`livekit-examples/agent-starter-react`](https://github.com/livekit-examples/agent-starter-react) | Web voice AI assistant with React & Next.js | | **iOS/macOS** | [`livekit-examples/agent-starter-swift`](https://github.com/livekit-examples/agent-starter-swift) | Native iOS, macOS, and visionOS voice AI assistant | | **Flutter** | [`livekit-examples/agent-starter-flutter`](https://github.com/livekit-examples/agent-starter-flutter) | Cross-platform voice AI assistant app | | **React Native** | [`livekit-examples/voice-assistant-react-native`](https://github.com/livekit-examples/voice-assistant-react-native) | Native mobile app with React Native & Expo | | **Android** | [`livekit-examples/agent-starter-android`](https://github.com/livekit-examples/agent-starter-android) | Native Android app with Kotlin & Jetpack Compose | | **Web Embed** | [`livekit-examples/agent-starter-embed`](https://github.com/livekit-examples/agent-starter-embed) | Voice AI widget for any website | | **Telephony** | [📚 Documentation](https://docs.livekit.io/agents/start/telephony/) | Add inbound or outbound calling to your agent | For advanced customization, see the [complete frontend guide](https://docs.livekit.io/agents/start/frontend/). ## Tests and evals This project includes a complete suite of evals, based on the LiveKit Agents [testing & evaluation framework](https://docs.livekit.io/agents/build/testing/). To run them, use `pytest`. ```console uv run pytest ``` ## Using this template repo for your own project Once you've started your own project based on this repo, you should: 1. **Check in your `uv.lock`**: This file is currently untracked for the template, but you should commit it to your repository for reproducible builds and proper configuration management. (The same applies to `livekit.toml`, if you run your agents in LiveKit Cloud) 2. **Remove the git tracking test**: Delete the "Check files not tracked in git" step from `.github/workflows/tests.yml` since you'll now want this file to be tracked. These are just there for development purposes in the template repo itself. 3. **Add your own repository secrets**: You must [add secrets](https://docs.github.com/en/actions/how-tos/writing-workflows/choosing-what-your-workflow-does/using-secrets-in-github-actions) for `LIVEKIT_URL`, `LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET` so that the tests can run in CI. ## Deploying to production This project is production-ready and includes a working `Dockerfile`. To deploy it to LiveKit Cloud or another environment, see the [deploying to production](https://docs.livekit.io/agents/ops/deployment/) guide. ## Self-hosted LiveKit You can also self-host LiveKit instead of using LiveKit Cloud. See the [self-hosting](https://docs.livekit.io/home/self-hosting/) guide for more information. If you choose to self-host, you'll need to also use [model plugins](https://docs.livekit.io/agents/models/#plugins) instead of LiveKit Inference and will need to remove the [LiveKit Cloud noise cancellation](https://docs.livekit.io/home/cloud/noise-cancellation/) plugin. ## License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. ## Detected evidence (automated analysis) Indexed codebase: 22 recognized source files, 94 KB. - Python (language) — detected in the code - PyTorch (technology) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository - AI coding agent: Codex — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (31 of 31) ``` .dockerignore .github/workflows/ruff.yml .github/workflows/template-check.yml .github/workflows/tests.yml .gitignore AGENTS.md CLAUDE.md DEMENTIA_AGENT_README.md Dockerfile GEMINI.md LICENSE pyproject.toml README.md src/__init__.py src/agent.py src/data/memories.json src/data/reminders.json src/example_memories.py src/tools/consent_escalation_tool.py src/tools/emergency_tool.py src/tools/mailjet_tool.py src/tools/memory_tool.py src/tools/twilio_tool.py taskfile.yaml test_email_context.py test_memory_simple.py test_twilio.py test_wedding_search.py tests/test_agent.py user_memories/chromadb/chroma.sqlite3 uv.lock ``` ### Dependencies - pyproject.toml: chromadb@>=1.2.1, livekit-agents[elevenlabs,groq,silero,turn-detector]@~=1.2, livekit-plugins-noise-cancellation@~=0.2, mailjet-rest@>=1.3.4, python-dotenv, torch@>=2.8.0, twilio@>=8.0.0 ### Recent commits (newest first) - tools done - tools working - voice works - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown # CLAUDE.md This project uses `AGENTS.md` instead of a `CLAUDE.md` file. Please see @AGENTS.md in this same directory and treat its content as the primary reference for this project. ``` ### GEMINI.md ```markdown # GEMINI.md This project uses `AGENTS.md` instead of a `GEMINI.md` file. Please see @./AGENTS.md in this same directory and treat its content as the primary reference for this project. ``` ### pyproject.toml ``` [build-system] requires = ["setuptools>=61.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "agent-starter-python" version = "1.0.0" description = "Simple voice AI assistant built with LiveKit Agents for Python" requires-python = ">=3.10" dependencies = [ "chromadb>=1.2.1", "livekit-agents[elevenlabs,groq,silero,turn-detector]~=1.2", "livekit-plugins-noise-cancellation~=0.2", "python-dotenv", "torch>=2.8.0", "twilio>=8.0.0", "mailjet-rest>=1.3.4", ] [dependency-groups] dev = [ "pytest", "pytest-asyncio", "ruff", ] [tool.setuptools.packages.find] where = ["src"] [tool.setuptools.package-dir] "" = "src" [tool.pytest.ini_options] asyncio_mode = "auto" asyncio_default_fixture_loop_scope = "function" [tool.ruff] line-length = 88 target-version = "py39" [tool.ruff.lint] select = ["E", "F", "W", "I", "N", "B", "A", "C4", "UP", "SIM", "RUF"] ignore = ["E501"] # Line too long (handled by formatter) [tool.ruff.format] quote-style = "double" indent-style = "space" ``` ### Dockerfile ``` # syntax=docker/dockerfile:1 # Use the official UV Python base image with Python 3.13 on Debian Bookworm # UV is a fast Python package manager that provides better performance than pip # We use the slim variant to keep the image size smaller while still having essential tools ARG PYTHON_VERSION=3.13 FROM ghcr.io/astral-sh/uv:python${PYTHON_VERSION}-bookworm-slim AS base # Keeps Python from buffering stdout and stderr to avoid situations where # the application crashes without emitting any logs due to buffering. ENV PYTHONUNBUFFERED=1 # Create a non-privileged user that the app will run under. # See https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#user ARG UID=10001 RUN adduser \ --disabled-password \ --gecos "" \ --home "/app" \ --shell "/sbin/nologin" \ --uid "${UID}" \ appuser # Install build dependencies required for Python packages with native extensions # gcc: C compiler needed for building Python packages with C extensions # python3-dev: Python development headers needed for compilation # We clean up the apt cache after installation to keep the image size down RUN apt-get update && apt-get install -y \ gcc \ g++ \ python3-dev \ && rm -rf /var/lib/apt/lists/* # Create a new directory for our application code # And set it as the working directory WORKDIR /app # Copy just the dependency files first, for more efficient layer caching COPY pyproject.toml uv.lock ./ RUN mkdir -p src # Install Python dependencies using UV's lock file # --locked ensures we use exact versions from uv.lock for reproducible builds # This creates a virtual environment and installs all dependencies # Ensure your uv.lock file is checked in for consistency across environments RUN uv sync --locked # Copy all remaining application files into the container # This includes source code, configuration files, and dependency specifications # (Excludes files specified in .dockerignore) COPY . . # Change ownership of all app files to the non-privileged user # This ensures the application can read/write files as needed RUN chown -R appuser:appuser /app # Switch to the non-privileged user for all subsequent operations # This improves security by not running as root USER appuser # Pre-download any ML models or files the agent needs # This ensures the container is ready to run immediately without downloading # dependencies at runtime, which improves startup time and reliability RUN uv run src/agent.py download-files # Run the application using UV # UV will activate the virtual environment and run the agent. # The "start" command tells the worker to connect to LiveKit and begin waiting for jobs. CMD ["uv", "run", "src/agent.py", "start"] ``` ### test_memory_simple.py ```python #!/usr/bin/env python3 import asyncio import sys import os # Add the src directory to the path sys.path.insert(0, os.path.join(os.path.dirname(__file__), "src")) from tools.memory_tool import MemoryTools async def test_memory_search(): """Test the memory search functionality directly""" print("Testing memory search...") # Create a mock context class MockContext: def __init__(self): self.room = type("Room", (), {"name": "test_room"})() context = MockContext() # Initialize memory tools memory_tools = MemoryTools() # Test queries test_queries = [ "hobbies", "high school graduation", "where did I graduate", "teaching career", "family children", ] for query in test_queries: print(f"\n--- Testing query: '{query}' ---") try: result = await memory_tools.search_memories(context, query) print(f"Result: {result}") except Exception as e: print(f"Error: {e}") if __name__ == "__main__": asyncio.run(test_memory_search()) ``` ### test_wedding_search.py ```python #!/usr/bin/env python3 import asyncio import sys import os # Add the src directory to the path sys.path.insert(0, os.path.join(os.path.dirname(__file__), 'src')) from tools.memory_tool import MemoryTools from livekit.agents import RunContext class MockContext: def __init__(self): self.room = MockRoom() class MockRoom: def __init__(self): self.name = "default_user" async def test_wedding_search(): """Test searching for wedding information""" print("Testing wedding search functionality...") # Create memory tools memory_tools = MemoryTools() # Create mock context context = MockContext() # Test different wedding-related queries queries = [ "when did I get married", "wedding day", "marriage", "Robert and I married", "June 12, 1955", "wedding dress", "St. Mary's Church" ] for query in queries: print(f"\nSearching for: '{query}'") try: result = await memory_tools.search_memories(context, query) print(f"Result: {result}") except Exception as e: print(f"Error: {e}") # Test getting memory summary print(f"\nGetting memory summary...") try: summary = await memory_tools.get_memory_summary(context) print(f"Summary: {summary}") except Exception as e: print(f"Error: {e}") if __name__ == "__main__": asyncio.run(test_wedding_search()) ``` ### test_email_context.py ```python #!/usr/bin/env python3 """ Test script to verify email includes conversation context """ import asyncio import os import sys from dotenv import load_dotenv # Add src to path so we can import our tools sys.path.append('src') from tools.emergency_tool import EmergencyTool async def test_email_context(): """Test that email includes conversation context""" print("Testing Email with Conversation Context...") print("=" * 45) load_dotenv('.env.local') emergency_tool = EmergencyTool() # Simulate emergency call print("1. Emergency call made...") await emergency_tool.emergency_call(None, "general", "I need help") # Simulate conversation print("2. Storing conversation...") emergency_tool.store_emergency_conversation("Maggie", "I fell down and can't get up") emergency_tool.store_emergency_conversation("Heather", "Are you hurt anywhere?") emergency_tool.store_emergency_conversation("Maggie", "My leg hurts and I can't move") emergency_tool.store_emergency_conversation("Heather", "Where are you right now?") emergency_tool.store_emergency_conversation("Maggie", "I'm in the living room") print("3. Sending email with conversation context...") result = await emergency_tool.send_emergency_email_now(None) print(f"Result: {result}") print("\n4. Check your email at wesleykieu13@gmail.com") print(" The email should now include the full conversation log!") return True if __name__ == "__main__": asyncio.run(test_email_context()) ``` ### taskfile.yaml ```yaml version: "3" output: interleaved dotenv: [".env.local"] vars: INDENT: 4 REL_PATH: "{{ relPath .USER_WORKING_DIR .ROOT_DIR }}" VENV_DIR: ".venv" PYTHON_MAIN: '{{ joinPath "./src" "agent.py" }}' tasks: post_create: desc: "Runs after this template is instantiated as a Sandbox or Bootstrap" cmds: - echo 'To try the new agent directly in your terminal:' - echo '' - echo '{{ indent .INDENT "cd" }} {{ .REL_PATH }}' - echo '{{ indent .INDENT "uv sync" }}' - echo '{{ indent .INDENT "uv run" }} {{ .PYTHON_MAIN }} download-files' - echo '{{ indent .INDENT "uv run" }} {{ .PYTHON_MAIN }} console' - echo '' - echo 'To deploy your agent to LiveKit cloud:' - echo '' - echo '{{ indent .INDENT "lk agent create" }}' - echo '' - task: set_agent_name_if_present - task: help_open_sandbox_if_present set_agent_name_if_present: status: - test -z "$LIVEKIT_AGENT_NAME" cmds: - | old="WorkerOptions(" new="WorkerOptions(agent_name=\"{{ .LIVEKIT_AGENT_NAME }}\", " file="{{ .PYTHON_MAIN }}" tmp="$(mktemp)" while IFS= read -r line; do printf '%s\n' "${line//$old/$new}" >> "$tmp" done < "$file" mv "$tmp" "$file" help_open_sandbox_if_present: status: - test -z "$LIVEKIT_SANDBOX_ID" cmds: - echo 'To chat with your running agent, visit:' - echo '' - echo '{{ indent .INDENT "https://" }}{{ .LIVEKIT_SANDBOX_ID }}.sandbox.livekit.io' - echo '' install: desc: "Bootstrap application for local development" cmds: - "uv sync" dev: interactive: true cmds: - "uv run src/agent.py dev" ``` ### test_twilio.py ```python #!/usr/bin/env python3 """ Test script for Twilio functionality """ import os import asyncio from dotenv import load_dotenv from src.tools.twilio_tool import TwilioTool load_dotenv(".env.local") async def test_twilio(): """Test Twilio tool functionality""" print("🧪 Testing Twilio Tool...") # Initialize the tool twilio_tool = TwilioTool() # Check if Twilio is configured if not twilio_tool.client: print("❌ Twilio not configured. Please add these to .env.local:") print(" TWILIO_ACCOUNT_SID=your_account_sid") print(" TWILIO_AUTH_TOKEN=your_auth_token") print(" TWILIO_PHONE_NUMBER=+1234567890") return print("✅ Twilio client initialized") # Test listing contacts print("\n📞 Testing list_contacts...") contacts = await twilio_tool.list_contacts(None) print(f"Contacts: {contacts}") # Test adding a contact print("\n➕ Testing add_phone_contact...") result = await twilio_tool.add_phone_contact(None, "Test Contact", "+15551234567") print(f"Add contact result: {result}") # Test listing contacts again print("\n📞 Testing list_contacts after adding...") contacts = await twilio_tool.list_contacts(None) print(f"Contacts: {contacts}") # Test making a call (this will actually make a call!) print("\n📞 Testing make_phone_call...") print("⚠️ This will make a real phone call to Mary!") response = input("Do you want to make a test call? (y/N): ") if response.lower() == "y": result = await twilio_tool.make_phone_call( None, "mary", "This is a test call from Heather." ) print(f"Call result: {result}") else: print("Skipping actual phone call test.") print("\n✅ Twilio tool test completed!") if __name__ == "__main__": asyncio.run(test_twilio()) ``` ### src/__init__.py ```python # This file makes the src directory a Python package ``` [13 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]