Project Info

Vera

Inspiration

Journaling is a powerful tool for mental clarity, but it is often a one-way street. We wanted to transform the static, passive diary into an active, intelligent companion. Inspired by the need for Radical Privacy in healthcare, we built Vera: a "Digital Confidant" that doesn't just store your words - it hears the emotion behind them. We wanted to create a space where users can be their most vulnerable without the fear of their mental health data ever touching a cloud server.

What it does

Vera is a privacy-first, on-device mental health companion that bridges the gap between a private diary and a responsive therapist. Acoustic Emotion Analysis: Using Wave2Vec converted to CoreML for on-device audio processing, Vera detects the emotional tone and sentiment of a user's voice, understanding the "how" behind their words in real-time. Voice Pipeline: Vera listens using Apple's SFSpeechRecognizer for fully on-device speech-to-text transcription with automatic silence detection. For high-quality text-to-speech, we deployed Kokoro TTS natively on-device via the FluidAudio framework, as built-in system voices were less than desirable for a companion that needs to sound warm and human. Intelligent Memory (RAG): Vera uses a fully on-device Retrieval-Augmented Generation pipeline to ground responses in real health knowledge. User queries are embedded in real-time using a MiniLM-L6-v2 sentence-transformer model (exported to Core ML), then matched against a pre-computed vector index of health documents via Accelerate-powered cosine similarity search. The top-matching chunks are injected into each conversation turn with an adaptive token budget that tapers across turns to balance context richness with conversation history. Everything runs locally; no data leaves the device. On-Device LLM: Conversations are powered by NVIDIA Nemotron-Mini-4B-Instruct, a 4-billion parameter language model running entirely on-device via llama.cpp (Q4_K_M 4-bit quantization, ~2.7 GB). The model uses a persistent KV cache for multi-turn context, Nemotron chat formatting, and automatic context-overflow recovery, delivering real-time, private conversational AI without any server dependency. Apple Watch Health Integration: Vera syncs with HealthKit to pull real-time biometric data from Apple Watch — heart rate, HRV, sleep duration, step count, and respiratory rate. Vera provides insights about these metrics to the users. Picture-in-Picture: Vera supports Picture-in-Picture mode, so the conversation stays with you even when you switch to other apps. The floating overlay shows live conversation status, waveform bars, and mood color, so you never lose your connection with Vera while multitasking. Active Support: Instead of just recording text, Vera engages in a therapeutic dialogue, helping users process their feelings by reflecting on their unique history. Zero-Cloud Privacy: By processing all audio and data locally, Vera ensures that a user's most intimate thoughts remain strictly on their device - a "Mental Health Vault."

How we built it

We focused on a high-performance, native stack to ensure seamless local execution: Audio & Speech: We utilized SFSpeechRecognizer for high-accuracy local transcription, a CoreML-exported Wave2Vec model for emotion classification on vocal patterns, and Kokoro TTS via FluidAudio for natural-sounding on-device voice synthesis. Knowledge Engine: We implemented a local RAG pipeline using MiniLM-L6-v2 embeddings on Core ML with Accelerate-powered vector search. This allows the app to query a mental health guideline vault to provide responses grounded in robust advice and insight. Intelligence: The conversational layer is powered by NVIDIA Nemotron-Mini-4B-Instruct running via llama.cpp, ensuring the "therapist" logic never requires an internet connection. Biometrics: HealthKit integration pulls real-time Apple Watch data (HR, HRV, sleep, steps, respiratory rate) so Vera can reference actual numbers and provide insight to the user. Frontend: A clean, minimal SwiftUI interface designed to reduce cognitive load and keep the focus on the user's journey, with Picture-in-Picture support via AVKit so Vera stays present across apps.

Challenges we ran into

The primary technical hurdle was implementing the various ML models and RAG on-device. Managing weights and vector embeddings and efficient inference within the processing and memory constraints of a mobile device required a very lean architecture. We also had to port over weights, a fine-tuned version of Wave2Vec for our emotion detection to CoreML to ensure it could accurately distinguish between subtle vocal shifts without relying on massive, cloud-based GPU clusters. Running five concurrent ML workloads — speech recognition, emotion classification, RAG embeddings, a 4B parameter LLM, and TTS synthesis — on a single mobile device required careful orchestration of the CPU, GPU, and Neural Engine to avoid thermal throttling and memory pressure.

Accomplishments we're proud of

We are incredibly proud of achieving Zero-Leak Privacy. Demonstrating a fully functional "Therapist-Diary" with RAG that works entirely in Airplane Mode was our "Eureka" moment.

What we learned

We gained deep experience in optimizing CoreML models to run efficiently on the Apple Neural Engine. As well as how to build Cascade Models on edge devices in an efficient way and with a good user experience.

What's next

The future of Vera is about deeper integration into the user's life. Actionable Outputs: Integrating with services like Spotify to suggest mood-shifting music based on the detected vocal sentiment. Self-care plan at the end. Cross-Device Sync: Implementing encrypted, peer-to-peer syncing so users can access their "Vera Vault" across their devices without cloud intermediaries.

Project CYT - Vera: Your AI Wellness Companion

Project CYT (Carlos, Yash, Tom) is a voice-first mental wellness iOS app built at TreeHacks 2026. It features Vera, an AI conversational companion that listens, asks good questions, and helps you check in with yourself. Everything runs on-device for privacy — no cloud APIs, no data leaving your phone.

What It Does

Voice Conversations: Talk to Vera like you would a thoughtful friend. She listens, picks up on your tone, and asks questions that actually dig deeper.
Real-Time Emotion Detection: A Core ML model analyzes your voice in real-time to understand how you're feeling — the background colors shift to match your mood.
Health-Aware Context: Vera pulls in HealthKit data (heart rate, HRV, sleep, steps) to give responses that are grounded in how your body is actually doing.
RAG-Powered Knowledge: On-device retrieval-augmented generation so Vera's advice is backed by real wellness knowledge, not generic platitudes.
Care Packages: At the end of a session, you get personalized self-care suggestions — breathing exercises, music, movement, connection prompts — based on what came up in your conversation.
Live Activity + Dynamic Island: See your conversation status, mood, and card suggestions right from your Lock Screen or Dynamic Island.
Picture-in-Picture: Keep Vera visible while you use other apps.

Requirements

Xcode with iOS 26 SDK
iOS 26.0+ deployment target
A device or simulator that supports Apple Intelligence (FoundationModels framework)

Setup - Important!

This project uses large ML models that are not included in the Git repo. You need to download them separately before building.

Step 1: Download the Model Zip Files

Download the following zip files from Google Drive:

File	Size	Contents	Download
`CYT-Nemotron-Model.zip`	~2.5 GB	NVIDIA Nemotron-Mini-4B-Instruct (quantized GGUF) — the main LLM that powers Vera	Here
`MiniLMEmbedder.mlpackage.zip`	~40 MB	MiniLM sentence embedder for RAG search	Here
Optional `Emotion Recognition Wav2Vec2-IEMOCAP`	~180 MB	CoreML Port of `speechbrain/emotion-recognition-wav2vec2-IEMOCAP`	Here

Step 2: Extract the Models into the Project

Unzip CYT-Nemotron-Model.zip — this will produce CYT/nemotron-mini-4b-instruct-q4_k_m.gguf. Place it so the file lives at:
```
CYT-TreeHacks-2026/CYT/nemotron-mini-4b-instruct-q4_k_m.gguf
```
Unzip MiniLMEmbedder.mlpackage.zip — this will produce the CYT/MiniLMEmbedder.mlpackage/ folder. Place it so the folder lives at:
```
CYT-TreeHacks-2026/CYT/MiniLMEmbedder.mlpackage/
```

Both zips are structured so that if you unzip them from the project root directory, the files will land in the right place automatically.

Step 3: Build and Run

Just open CYT.xcodeproj in Xcode, change the Developer Team and hit Run.

Architecture

The app follows MVVM + Services:

ConversationView (SwiftUI)
  └─ ConversationViewModel (orchestrates everything)
      ├─ SpeechRecognizer       — AVAudioEngine + SFSpeechRecognizer, silence detection
      ├─ EmotionClassifierService — Core ML emotion classification on audio
      ├─ LLMService             — Nemotron-Mini-4B via llama.cpp, on-device
      ├─ RAGService             — MiniLM embedder + vector index for knowledge retrieval
      ├─ TextToSpeechService    — PocketTTS (local package), sentence-level pipelining
      ├─ HealthDataProvider     — HealthKit integration (HR, HRV, sleep, steps)
      └─ JournalStore           — JSON persistence for conversation journals

Conversation Flow

Tap the orb to start recording
3 seconds of silence → auto-stop and process
In parallel: emotion classification on your audio + LLM generates a response with health context
Vera speaks back with sentence-level TTS pipelining
Recording auto-restarts — it's a natural conversation loop

Built With

Swift + SwiftUI
llama.cpp (Nemotron-Mini-4B-Instruct, Q4_K_M quantization)
Core ML (MiniLM embedder, emotion classifier)
AVAudioEngine + Speech framework
HealthKit
ActivityKit (Live Activities + Dynamic Island)
PocketTTS (on-device text-to-speech)

Beautiful Pictures

Beautiful Video

Make sure to unmute

https://github.com/user-attachments/assets/91bb3aa6-1241-4ef7-8c8d-c372960db38c

Analysis

Compare with all teams

View

Metric

Figures cover GitHub contributors during the hackathon window. A co-authored commit counts in full for each author, so per-member totals add up to more than the whole-team figures.

Technology

Found in codeClaimed only

PythonIn code
SwiftIn code

2 of 2 appear in the indexed code.

AI coding agents

Claude CodeConfig · Commits

Detected from committed agent config files and commit authorship. Absence of a signal is not proof an agent was unused.

Codebase size

Source size

241 KB

Source files

Counts recognized source files only; vendored directories, binaries and lockfiles are excluded, so this is smaller than the repository on disk.