# Project export: Vera This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Vera - the world’s first on-device, hyper-private, mental health companion that bridges the gap between a companion and a journal. - Devpost: https://devpost.com/software/vera-8bjf1g - GitHub: https://github.com/carlosmbe/CYT-TreeHacks-2026 - Video: https://www.youtube.com/embed/XruDpfVkt38?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — YashAgarwal06 (16 commits), Carlos Mbendera (14 commits), TomSmail (6 commits) ## Devpost submission (written by the team) ### Inspiration Journaling is a powerful tool for mental clarity, but it is often a one-way street. We wanted to transform the static, passive diary into an active, intelligent companion. Inspired by the need for Radical Privacy in healthcare, we built Vera: a "Digital Confidant" that doesn't just store your words - it hears the emotion behind them. We wanted to create a space where users can be their most vulnerable without the fear of their mental health data ever touching a cloud server. ### What it does Vera is a privacy-first, on-device mental health companion that bridges the gap between a private diary and a responsive therapist. Acoustic Emotion Analysis: Using Wave2Vec converted to CoreML for on-device audio processing, Vera detects the emotional tone and sentiment of a user's voice, understanding the "how" behind their words in real-time. Voice Pipeline: Vera listens using Apple's SFSpeechRecognizer for fully on-device speech-to-text transcription with automatic silence detection. For high-quality text-to-speech, we deployed Kokoro TTS natively on-device via the FluidAudio framework, as built-in system voices were less than desirable for a companion that needs to sound warm and human. Intelligent Memory (RAG): Vera uses a fully on-device Retrieval-Augmented Generation pipeline to ground responses in real health knowledge. User queries are embedded in real-time using a MiniLM-L6-v2 sentence-transformer model (exported to Core ML), then matched against a pre-computed vector index of health documents via Accelerate-powered cosine similarity search. The top-matching chunks are injected into each conversation turn with an adaptive token budget that tapers across turns to balance context richness with conversation history. Everything runs locally; no data leaves the device. On-Device LLM: Conversations are powered by NVIDIA Nemotron-Mini-4B-Instruct, a 4-billion parameter language model running entirely on-device via llama.cpp (Q4_K_M 4-bit quantization, ~2.7 GB). The model uses a persistent KV cache for multi-turn context, Nemotron chat formatting, and automatic context-overflow recovery, delivering real-time, private conversational AI without any server dependency. Apple Watch Health Integration: Vera syncs with HealthKit to pull real-time biometric data from Apple Watch — heart rate, HRV, sleep duration, step count, and respiratory rate. Vera provides insights about these metrics to the users. Picture-in-Picture: Vera supports Picture-in-Picture mode, so the conversation stays with you even when you switch to other apps. The floating overlay shows live conversation status, waveform bars, and mood color, so you never lose your connection with Vera while multitasking. Active Support: Instead of just recording text, Vera engages in a therapeutic dialogue, helping users process their feelings by reflecting on their unique history. Zero-Cloud Privacy: By processing all audio and data locally, Vera ensures that a user's most intimate thoughts remain strictly on their device - a "Mental Health Vault." ### How we built it We focused on a high-performance, native stack to ensure seamless local execution: Audio & Speech: We utilized SFSpeechRecognizer for high-accuracy local transcription, a CoreML-exported Wave2Vec model for emotion classification on vocal patterns, and Kokoro TTS via FluidAudio for natural-sounding on-device voice synthesis. Knowledge Engine: We implemented a local RAG pipeline using MiniLM-L6-v2 embeddings on Core ML with Accelerate-powered vector search. This allows the app to query a mental health guideline vault to provide responses grounded in robust advice and insight. Intelligence: The conversational layer is powered by NVIDIA Nemotron-Mini-4B-Instruct running via llama.cpp, ensuring the "therapist" logic never requires an internet connection. Biometrics: HealthKit integration pulls real-time Apple Watch data (HR, HRV, sleep, steps, respiratory rate) so Vera can reference actual numbers and provide insight to the user. Frontend: A clean, minimal SwiftUI interface designed to reduce cognitive load and keep the focus on the user's journey, with Picture-in-Picture support via AVKit so Vera stays present across apps. ### Challenges we ran into The primary technical hurdle was implementing the various ML models and RAG on-device. Managing weights and vector embeddings and efficient inference within the processing and memory constraints of a mobile device required a very lean architecture. We also had to port over weights, a fine-tuned version of Wave2Vec for our emotion detection to CoreML to ensure it could accurately distinguish between subtle vocal shifts without relying on massive, cloud-based GPU clusters. Running five concurrent ML workloads — speech recognition, emotion classification, RAG embeddings, a 4B parameter LLM, and TTS synthesis — on a single mobile device required careful orchestration of the CPU, GPU, and Neural Engine to avoid thermal throttling and memory pressure. ### Accomplishments we're proud of We are incredibly proud of achieving Zero-Leak Privacy. Demonstrating a fully functional "Therapist-Diary" with RAG that works entirely in Airplane Mode was our "Eureka" moment. ### What we learned We gained deep experience in optimizing CoreML models to run efficiently on the Apple Neural Engine. As well as how to build Cascade Models on edge devices in an efficient way and with a good user experience. ### What's next The future of Vera is about deeper integration into the user's life. Actionable Outputs: Integrating with services like Spotify to suggest mood-shifting music based on the detected vocal sentiment. Self-care plan at the end. Cross-Device Sync: Implementing encrypted, peer-to-peer syncing so users can access their "Vera Vault" across their devices without cloud intermediaries. ## README (from the GitHub repository) # Project CYT - Vera: Your AI Wellness Companion Project CYT (Carlos, Yash, Tom) is a voice-first mental wellness iOS app built at **TreeHacks 2026**. It features **Vera**, an AI conversational companion that listens, asks good questions, and helps you check in with yourself. Everything runs on-device for privacy — no cloud APIs, no data leaving your phone. ## What It Does - **Voice Conversations**: Talk to Vera like you would a thoughtful friend. She listens, picks up on your tone, and asks questions that actually dig deeper. - **Real-Time Emotion Detection**: A Core ML model analyzes your voice in real-time to understand how you're feeling — the background colors shift to match your mood. - **Health-Aware Context**: Vera pulls in HealthKit data (heart rate, HRV, sleep, steps) to give responses that are grounded in how your body is actually doing. - **RAG-Powered Knowledge**: On-device retrieval-augmented generation so Vera's advice is backed by real wellness knowledge, not generic platitudes. - **Care Packages**: At the end of a session, you get personalized self-care suggestions — breathing exercises, music, movement, connection prompts — based on what came up in your conversation. - **Live Activity + Dynamic Island**: See your conversation status, mood, and card suggestions right from your Lock Screen or Dynamic Island. - **Picture-in-Picture**: Keep Vera visible while you use other apps. ## Requirements - **Xcode** with iOS 26 SDK - **iOS 26.0+** deployment target - A device or simulator that supports Apple Intelligence (`FoundationModels` framework) ## Setup - Important! This project uses large ML models that are **not included in the Git repo**. You need to download them separately before building. ### Step 1: Download the Model Zip Files Download the following zip files from Google Drive: | File | Size | Contents | Download | |------|------|----------| ---------- | | `CYT-Nemotron-Model.zip` | ~2.5 GB | NVIDIA Nemotron-Mini-4B-Instruct (quantized GGUF) — the main LLM that powers Vera | [Here](https://drive.google.com/file/d/1EXctUuIKNnKn1UccJYkbMao6zcc-kYPL/view?usp=share_link) | | `MiniLMEmbedder.mlpackage.zip` | ~40 MB | MiniLM sentence embedder for RAG search | [Here](https://drive.google.com/file/d/1II_hYj9HjFEVll4hn6dfsfVjvCU-_axL/view?usp=share_link) | | Optional `Emotion Recognition Wav2Vec2-IEMOCAP ` | ~180 MB |CoreML Port of `speechbrain/emotion-recognition-wav2vec2-IEMOCAP`| [Here](https://drive.google.com/file/d/1elYZgcwZJ4l6vBbMKI2VuBxPQzG-k-GZ/view?usp=share_link) | ### Step 2: Extract the Models into the Project 1. **Unzip `CYT-Nemotron-Model.zip`** — this will produce `CYT/nemotron-mini-4b-instruct-q4_k_m.gguf`. Place it so the file lives at: ``` CYT-TreeHacks-2026/CYT/nemotron-mini-4b-instruct-q4_k_m.gguf ``` 2. **Unzip `MiniLMEmbedder.mlpackage.zip`** — this will produce the `CYT/MiniLMEmbedder.mlpackage/` folder. Place it so the folder lives at: ``` CYT-TreeHacks-2026/CYT/MiniLMEmbedder.mlpackage/ ``` Both zips are structured so that if you unzip them from the project root directory, the files will land in the right place automatically. ### Step 3: Build and Run Just open `CYT.xcodeproj` in Xcode, change the Developer Team and hit Run. ## Architecture The app follows **MVVM + Services**: ``` ConversationView (SwiftUI) └─ ConversationViewModel (orchestrates everything) ├─ SpeechRecognizer — AVAudioEngine + SFSpeechRecognizer, silence detection ├─ EmotionClassifierService — Core ML emotion classification on audio ├─ LLMService — Nemotron-Mini-4B via llama.cpp, on-device ├─ RAGService — MiniLM embedder + vector index for knowledge retrieval ├─ TextToSpeechService — PocketTTS (local package), sentence-level pipelining ├─ HealthDataProvider — HealthKit integration (HR, HRV, sleep, steps) └─ JournalStore — JSON persistence for conversation journals ``` Slide2 ### Conversation Flow 1. Tap the orb to start recording 2. 3 seconds of silence → auto-stop and process 3. In parallel: emotion classification on your audio + LLM generates a response with health context 4. Vera speaks back with sentence-level TTS pipelining 5. Recording auto-restarts — it's a natural conversation loop ## Built With - Swift + SwiftUI - llama.cpp (Nemotron-Mini-4B-Instruct, Q4_K_M quantization) - Core ML (MiniLM embedder, emotion classifier) - AVAudioEngine + Speech framework - HealthKit - ActivityKit (Live Activities + Dynamic Island) - PocketTTS (on-device text-to-speech) ## Beautiful Pictures Slide4 ## Beautiful Video Make sure to unmute https://github.com/user-attachments/assets/91bb3aa6-1241-4ef7-8c8d-c372960db38c ## Detected evidence (automated analysis) Indexed codebase: 41 recognized source files, 241 KB. - Python (language) — detected in the code - Swift (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (57 of 57) ``` .gitignore CLAUDE.md CYT.xcodeproj/project.pbxproj CYT.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved CYT/AnimatedGradientBackground.swift CYT/Assets.xcassets/AccentColor.colorset/Contents.json CYT/Assets.xcassets/AppIcon.appiconset/Contents.json CYT/Assets.xcassets/Contents.json CYT/CarePackageModels.swift CYT/CarePackageResultView.swift CYT/CarePackageView.swift CYT/ContentView.swift CYT/ConversationActivityAttributes.swift CYT/ConversationSummaryView.swift CYT/ConversationView.swift CYT/CYT.entitlements CYT/CYTApp.swift CYT/DJCardView.swift CYT/EmotionClassifierService.swift CYT/FloatingCardView.swift CYT/HealthDataProvider.swift CYT/HealthTestView.swift CYT/Info.plist CYT/JournalEntryView.swift CYT/JournalStore.swift CYT/LlamaContext.swift CYT/LLMService.swift CYT/minilm_tokenizer/minilm_tokenizer_config.json CYT/minilm_tokenizer/minilm_tokenizer.json CYT/MockSignalProvider.swift CYT/PiPContentView.swift CYT/PiPSampleBufferRenderer.swift CYT/PiPService.swift CYT/rag_index.json CYT/RAG/CoreMLEmbedder.swift CYT/RAG/Embedder.swift CYT/RAG/RAGService.swift CYT/RAG/RAGTypes.swift CYT/RAG/README.md CYT/RAG/VectorIndex.swift CYT/SpeechRecognizer.swift CYT/TextToSpeechService.swift README.md scripts/build_rag_index.py scripts/download_nemotron.sh scripts/download_tokenizer.py scripts/export_coreml_embedder.py VeraLiveActivity/AppIntent.swift VeraLiveActivity/Assets.xcassets/AccentColor.colorset/Contents.json VeraLiveActivity/Assets.xcassets/AppIcon.appiconset/Contents.json VeraLiveActivity/Assets.xcassets/Contents.json VeraLiveActivity/Assets.xcassets/WidgetBackground.colorset/Contents.json VeraLiveActivity/ConversationActivityAttributes.swift VeraLiveActivity/Info.plist VeraLiveActivity/VeraLiveActivity.swift VeraLiveActivity/VeraLiveActivityBundle.swift VeraLiveActivity/VeraLiveActivityLiveActivity.swift ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Add download links for model zip files - Update README with images and video links - ReadME Draft - App Icons - Merge pull request #10 from carlosmbe/BackGroundAttempts - Merge branch 'main' into BackGroundAttempts - Picture in Picture WORKS!!!!!! - Merge pull request #9 from carlosmbe/nvidia-nemo-model-edge - Switch to NVIDIA Nemotron-Mini-4B model for on-device edge inference - I tried so hard to make this work in the background but failed - mlpackage unnecessary files deleted - gitignore addition - re-add MiniLM Core ML embedder model files - Merge pull request #8 from carlosmbe/local-rag - Merge branch 'main' into local-rag - implement full RAG functionality - Let's start wiring things up - tokenizer name fix - Merge pull request #7 from carlosmbe/speech-to-text - Resolve merge conflict ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown # CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Build & Run This is an Xcode-based iOS project (no SPM root package). Build and run through Xcode or: ```bash xcodebuild -project CYT.xcodeproj -scheme CYT -destination 'platform=iOS Simulator,name=iPhone 16 Pro' build ``` **Deployment target: iOS 26.0** — requires Xcode with iOS 26 SDK. The app uses Apple Intelligence (`FoundationModels` framework) which is only available on iOS 26+. There are no tests or linter configured in this project. ## Architecture **CYT** is a voice-first mental wellness iOS app featuring **Vera**, an AI conversational companion. Everything runs on-device for privacy. ### Core Pattern: MVVM + Services ``` ConversationView (SwiftUI UI) └─ ConversationViewModel (@Observable, orchestrates everything) ├─ SpeechRecognizer — AVAudioEngine + SFSpeechRecognizer, silence detection, writes 16kHz mono WAV ├─ EmotionClassifierService — Core ML model on audio chunks, returns emotion label ├─ LLMService — Apple Intelligence FoundationModels (iOS 26), Vera persona ├─ TextToSpeechService — FluidAudio/PocketTTS (local package), sentence-level pipelining ├─ HealthDataProvider — HealthKit (HR, HRV, sleep, steps, respiratory), synthetic fallback └─ JournalStore — JSON file persistence in Documents/ ``` ### Conversation Flow 1. User taps orb → `startRecording()` → AVAudioEngine tap streams to SpeechRecognizer + writes temp WAV 2. 3-second silence detected → `handleAutoStop()` → `stopAndProcess()` 3. Parallel: emotion classification on WAV file + LLM generation with last 6 messages + health context 4. TTS plays response with sentence-level pipelining (synthesize next while playing current) 5. Auto-restarts recording after TTS finishes ### Conversation State Machine `idle` → `recording` → `processing` → `veraSpeaking` → `recording` (auto-continue loop) `recording` ↔ `paused` (manual pause/resume) ### Targets - **CYT** — Main app - **VeraLiveActivityExtension** — Dynamic Island + Lock Screen widget showing conversation status, waveform bars, mood color, and card suggestions via ActivityKit ### Key Data Types (CarePackageModels.swift) - `CarePackageCard` — contextual self-care suggestions (breathe, music, connect, move, express, rest, celebrate) - `CarePackage` — end-of-session summary with cards, DJ session, health signals - `CheckInSignals` — emotion label + confidence + health snapshot - `ConversationActivityAttributes` — shared between app and widget for Live Activity state ### Local Packages - `LocalPackages/FluidAudio/` — PocketTTS wrapper for on-device text-to-speech ### UI - **Voice mode** (default): Central orb morphs by state (mic → waveform → spinner → glow) - **Text mode**: Message bubbles with compact orb - **AnimatedGradientBackground**: 7 drifting blobs at 30fps, colors shift instantly with detected mood - **Floa [truncated — 503 more characters] ``` ### VeraLiveActivity/VeraLiveActivityBundle.swift ```swift // // VeraLiveActivityBundle.swift // VeraLiveActivity // // Created by Carlos Mbendera on 14/02/2026. // import WidgetKit import SwiftUI @main struct VeraLiveActivityBundle: WidgetBundle { var body: some Widget { VeraLiveActivity() VeraLiveActivityLiveActivity() } } ``` ### VeraLiveActivity/AppIntent.swift ```swift // // AppIntent.swift // VeraLiveActivity // // Created by Carlos Mbendera on 14/02/2026. // import WidgetKit import AppIntents struct ConfigurationAppIntent: WidgetConfigurationIntent { static var title: LocalizedStringResource { "Configuration" } static var description: IntentDescription { "This is an example widget." } // An example configurable parameter. @Parameter(title: "Favorite Emoji", default: "😃") var favoriteEmoji: String } ``` ### CYT/ConversationActivityAttributes.swift ```swift // // ConversationActivityAttributes.swift // CYT // // ActivityKit attributes for the conversation Live Activity. // Shared between the main app and the widget extension. // import ActivityKit import Foundation struct ConversationActivityAttributes: ActivityAttributes { public struct ContentState: Codable, Hashable { var isRecording: Bool var latestCardTitle: String? var latestCardIcon: String? var moodColor: String // "neutral", "sad", "angry", "happy" var audioLevel: Int // 0=silent, 1=low, 2=mid, 3=high } var sessionStart: Date } ``` ### VeraLiveActivity/ConversationActivityAttributes.swift ```swift // // ConversationActivityAttributes.swift // CYT // // ActivityKit attributes for the conversation Live Activity. // Shared between the main app and the widget extension. // import ActivityKit import Foundation struct ConversationActivityAttributes: ActivityAttributes { public struct ContentState: Codable, Hashable { var isRecording: Bool var latestCardTitle: String? var latestCardIcon: String? var moodColor: String // "neutral", "sad", "angry", "happy" var audioLevel: Int // 0=silent, 1=low, 2=mid, 3=high } var sessionStart: Date } ``` ### CYT/CYTApp.swift ```swift // // CYTApp.swift // CYT // // Created by Carlos Mbendera on 14/02/2026. // import SwiftUI @main struct CYTApp: App { var body: some Scene { WindowGroup { NavigationStack { if #available(iOS 26.0, *) { ConversationView() } else { ContentUnavailableView( "Requires iOS 26", systemImage: "mic.slash", description: Text("Voice conversation requires iOS 26 or later.") ) } } } } } ``` ### CYT/ContentView.swift ```swift // // ContentView.swift // CYT // // Minimal STT -> LLM -> TTS voice conversation. // import SwiftUI struct ContentView: View { var body: some View { NavigationStack { if #available(iOS 26.0, *) { ConversationView() } else { ContentUnavailableView( "Requires iOS 26", systemImage: "mic.slash", description: Text("Voice conversation requires iOS 26 or later.") ) } } #if os(iOS) .navigationBarTitleDisplayMode(.inline) #endif } } #Preview { NavigationStack { ContentView() } } ``` ### CYT/JournalStore.swift ```swift // // JournalStore.swift // CYT // // Simple JSON-file persistence for journal entries. // import Foundation @Observable final class JournalStore { private(set) var entries: [JournalEntry] = [] private static var fileURL: URL { FileManager.default .urls(for: .documentDirectory, in: .userDomainMask)[0] .appendingPathComponent("journal_entries.json") } init() { load() } func save(_ entry: JournalEntry) { entries.insert(entry, at: 0) persist() } func loadAll() -> [JournalEntry] { entries } // MARK: - Private private func load() { guard let data = try? Data(contentsOf: Self.fileURL) else { return } entries = (try? JSONDecoder().decode([JournalEntry].self, from: data)) ?? [] } private func persist() { guard let data = try? JSONEncoder().encode(entries) else { return } try? data.write(to: Self.fileURL, options: .atomic) } } ``` ### scripts/download_tokenizer.py ```python #!/usr/bin/env python3 """ Download MiniLM tokenizer files for bundling in the iOS app. ============================================================= Downloads the tokenizer config from HuggingFace so the app can tokenize queries fully offline (no network needed on-device). Files are renamed with a ``minilm_`` prefix to avoid collisions with other tokenizer files (e.g. FastVLM) in the Xcode bundle. Usage: pip install huggingface_hub python scripts/download_tokenizer.py Output: CYT/minilm_tokenizer/minilm_tokenizer.json CYT/minilm_tokenizer/minilm_tokenizer_config.json Add the two output files to your Xcode project as individual bundle resources. """ from huggingface_hub import hf_hub_download from pathlib import Path import shutil MODEL_NAME = "sentence-transformers/all-MiniLM-L6-v2" OUTPUT_DIR = Path("CYT/minilm_tokenizer") # HuggingFace filename -> renamed bundle filename FILES = { "tokenizer.json": "minilm_tokenizer.json", "tokenizer_config.json": "minilm_tokenizer_config.json", } def main(): OUTPUT_DIR.mkdir(parents=True, exist_ok=True) for hf_name, bundle_name in FILES.items(): print(f"Downloading {hf_name}...") path = hf_hub_download( repo_id=MODEL_NAME, filename=hf_name, local_dir=OUTPUT_DIR, ) dest = OUTPUT_DIR / bundle_name shutil.move(path, dest) print(f" Saved to {dest}") print(f"\nDone. Add the files in '{OUTPUT_DIR}/' to your Xcode project as bundle resources.") if __name__ == "__main__": main() ``` ### scripts/download_nemotron.sh ```shell #!/usr/bin/env bash # ───────────────────────────────────────────────────────── # Download NVIDIA Nemotron-Mini-4B-Instruct Q4_K_M (GGUF) # ───────────────────────────────────────────────────────── # Source: bartowski/Nemotron-Mini-4B-Instruct-GGUF on HuggingFace # Size : ~2.7 GB # Quant : Q4_K_M (4-bit, k-quant medium – best quality/size for mobile) # # Usage: # chmod +x scripts/download_nemotron.sh # ./scripts/download_nemotron.sh # # The model will be saved to CYT/nemotron-mini-4b-instruct-q4_k_m.gguf # Then add this file to your Xcode project as a bundle resource. # ───────────────────────────────────────────────────────── set -euo pipefail MODEL_URL="https://huggingface.co/bartowski/Nemotron-Mini-4B-Instruct-GGUF/resolve/main/Nemotron-Mini-4B-Instruct-Q4_K_M.gguf" OUTPUT_DIR="$(cd "$(dirname "$0")/.." && pwd)/CYT" OUTPUT_FILE="${OUTPUT_DIR}/nemotron-mini-4b-instruct-q4_k_m.gguf" echo "╔══════════════════════════════════════════════════════════╗" echo "║ NVIDIA Nemotron-Mini-4B-Instruct Q4_K_M Downloader ║" echo "╚══════════════════════════════════════════════════════════╝" echo "" if [ -f "$OUTPUT_FILE" ]; then echo "✓ Model already exists at:" echo " $OUTPUT_FILE" echo "" echo " Delete it and re-run if you want to re-download." exit 0 fi echo "Downloading (~2.7 GB) …" echo " From: $MODEL_URL" echo " To: $OUTPUT_FILE" echo "" curl -L --progress-bar -o "$OUTPUT_FILE" "$MODEL_URL" echo "" echo "✓ Download complete!" echo " File: $OUTPUT_FILE" echo " Size: $(du -h "$OUTPUT_FILE" | cut -f1)" echo "" echo "Next steps:" echo " 1. Open the Xcode project" echo " 2. Drag the .gguf file into the CYT target (ensure 'Copy items if needed' is OFF)" echo " 3. Verify it appears under Build Phases → Copy Bundle Resources" ``` [29 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]