# Project export: Clove This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: The augmented reality healthcare assistant to help elderly adults recall memories, recognize loved ones, and stay safe through real-time support. - Devpost: https://devpost.com/software/clove-ga6v5p - GitHub: https://github.com/dylanytran/clove - Video: https://www.youtube.com/embed/E9CYBohndM8?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 4 GitHub contributor(s) — Kevin He (32 commits), Linkai Wu (10 commits), Dylan Tran (6 commits), William Wu (5 commits) ## Devpost submission (written by the team) ### Inspiration “Where did I put my keys?” “Who is this person?” “What was I doing a moment ago?” Over 55 million people worldwide live with dementia, and millions more elderly adults struggle with memory, safety, and independence. The rapidly growing development in wearable technology inspired us to think about how spatial computing can be applied to tackle the problems that the elderly face today. Our curiosity led us to build Clove, an AI-powered augmented reality application to help elders recall information, recognize loved ones, and stay safe through real-time support. ### What it does 🧠 Contextual memory recall: Clove continually captures and indexes daily experiences. Simply ask a question like “Where did I put my keys” and receive instant, context-aware answers paired with a brief video clip of the exact moment—turning foggy memories into crystal-clear recall. ex. “Where did I put my keys?”, “Did I turn the stove off?” 🧠 Contextual memory recall: Clove continually captures and indexes daily experiences. Simply ask a question like “Where did I put my keys” and receive instant, context-aware answers paired with a brief video clip of the exact moment—turning foggy memories into crystal-clear recall. ex. “Where did I put my keys?”, “Did I turn the stove off?” 👤 Identity recognition overlays: Puts names to important faces. Our AR nametags instantly identify loved ones, caregivers, and frequent visitors, displaying their name and relationship in your field of view. Add contacts with cherished photo memories, and never experience that uncomfortable moment of forgetting again. 👤 Identity recognition overlays: Puts names to important faces. Our AR nametags instantly identify loved ones, caregivers, and frequent visitors, displaying their name and relationship in your field of view. Add contacts with cherished photo memories, and never experience that uncomfortable moment of forgetting again. 📞 Two-way care calls: Through Zoom-powered video calls, family members, doctors, and caregivers can see exactly what you see and connect with you in real-time, from taking medication to cooking a meal. ex. “Call my daughter", “Start a Zoom call” 📞 Two-way care calls: Through Zoom-powered video calls, family members, doctors, and caregivers can see exactly what you see and connect with you in real-time, from taking medication to cooking a meal. ex. “Call my daughter", “Start a Zoom call” 📋 Task manager and reminders: Caregivers and family members can create personalized reminders for important tasks, such as taking the right medications on time. The system proactively notifies users at the right time with clear, step-by-step guidance, and confirms completion back to caregivers. ex. Check “take Wednesday meds” off my list 📋 Task manager and reminders: Caregivers and family members can create personalized reminders for important tasks, such as taking the right medications on time. The system proactively notifies users at the right time with clear, step-by-step guidance, and confirms completion back to caregivers. ex. Check “take Wednesday meds” off my list 🚨 Instant fall detection & alerts: Help arrives before you ask. Motion sensors detect hard falls immediately and automatically notifies emergency contacts with your precise location. Give both users and families the peace of mind they deserve. 🚨 Instant fall detection & alerts: Help arrives before you ask. Motion sensors detect hard falls immediately and automatically notifies emergency contacts with your precise location. Give both users and families the peace of mind they deserve. 📊 Weekly cognitive reports: After each Zoom call, the app uploads the meeting transcript to a PostgreSQL database on Render. A weekly cron job analyzes all conversations and generates a cognitive health report. The report emails to a caregiver with insights about the user's mental health. Analysis includes: cognitive scores, mood patterns, areas of concern and strength, and more. 📊 Weekly cognitive reports: After each Zoom call, the app uploads the meeting transcript to a PostgreSQL database on Render. A weekly cron job analyzes all conversations and generates a cognitive health report. The report emails to a caregiver with insights about the user's mental health. Analysis includes: cognitive scores, mood patterns, areas of concern and strength, and more. ### How we built it We built a native iOS app in Swift/SwiftUI leveraging a combination of on-device Apple frameworks and cloud APIs to power each core feature, with the goal of shipping in the future to devices like Meta Ray-Bans. Contextual memory recall: The app continuously records rolling 6-second video clips with AVFoundation and indexes them on-device using Vision (scene/text labels) and Natural Language (sentence embeddings) for semantic search. OpenAI GPT-4o-mini improves clip descriptions. Voice input is handled by the Speech framework; we embed the query, find the best clip, and use the OpenAI API to generate a one-sentence answer, then play the clip and speak the answer with TTS. Identity recognition overlays: Contacts are stored with face embeddings from Vision (landmarks). At runtime we detect faces in the AR feed, match them to stored embeddings, and show name and relationship above the person in the AR view. Two-way care calls: We integrated the Zoom Video SDK for real-time video calls with transcript capture, and the VAPI voice AI platform for outbound phone calls. Both are accessible hands-free through our voice assistant—users simply say "call my daughter" or "start a Zoom call," and the assistant resolves the contact and initiates the session via OpenAI function calling. Instant fall detection & alerts: We monitor the device accelerometer via CoreMotion and apply a fall detection algorithm that triggers a VAPI-powered emergency call to a pre-configured caregiver and pushes a critical local notification, all with a 10-second cooldown to prevent false re-triggers. Universal voice assistant:* We use OpenAI function calling so voice commands can trigger search_memories, Zoom (Zoom Video SDK), or call_contact (VAPI). A shared AppSpeechManager speaks confirmations and errors. Weekly cognitive reports: Each Zoom call transcript is automatically stored in a PostgreSQL database hosted on Render. A weekly cron job aggregates the past week's conversations and sends them to OpenAI for analysis, generating cognitive health scores (clarity, coherence, memory recall), mood patterns, and conversation statistics like total calls and duration. The resulting report is emailed to caregivers via Resend, flagging any concerning patterns that may need attention. ### Challenges we ran into Adapting AR glasses-inspired vision for mobile form factor: Translating AR glasses interactions to mobile devices required reimagining our approach to UX while conveying the original potential for real in-glasses deployment. AR glasses offer hands-free, persistent overlays in the user's natural field of view, while mobile devices demand active engagement and screen-based interfaces. We solved this by developing a hybrid interaction model that maintains spatial awareness on mobile through camera-based AR, while designing for future scalability to dedicated wearable hardware. Real-time facial recognition accuracy: Achieving reliable facial recognition for elderly users presented unique obstacles—varied lighting conditions, users with glasses or changing appearances, and the need to avoid false positives that could cause confusion. Efficient storage and semantic search: Continuously recording and storing video clips would quickly become storage-prohibitive and computationally expensive. Our solution combines intelligent scene detection to capture only meaningful moments and an embedding system that converts video content into searchable vector representations, enabling natural language queries to surface relevant clips in milliseconds rather than hours ### What we learned 🫂 Empathy through design. This experience pushed me to build for people whose daily challenges I don’t face myself. Designing for elderly users involves recognizing the real consequences when technology fails someone who depends on it. - Linkai ⚖️ Technical trade-offs. I learned a lot about the technical trade-offs between speed and quality while working on this project. More frequent calls to GPT-4o mini would lead to higher quality memory recalls but would slow down performance noticeably. It was really important to find the right balance between the two. - Dylan 💡 Narrowing down ideas. This hackathon taught me to be specific and always have the end-user in mind. What do they want? What do they need? It is common for developers to build features that receive little appreciation from the public, which further emphasizes the need to think critically about the product at every step. - Kevin ⚙️ Exposure to new tools/techniques. This hackathon exposed me to feature mapping and how to use it for facial recognition. Also received exposure on how to integrate tools like Render into our project to automate sending emails that contain a summary of the user's actions and store user data. - Will ### What's next While our current prototype runs on mobile devices, our vision from day one has been deployment on dedicated AR glasses. The mobile version proves the concept, but the real magic happens when this technology lives naturally in a user's field of view: hands-free, always accessible, and truly seamless. As devices like Meta's Ray-Ban smart glasses and Apple's Vision products mature and become more affordable, we'll be ready to transition our platform to true wearable form. ## README (from the GitHub repository) ## YouTube Demo [![Clove Demo](https://img.youtube.com/vi/E9CYBohndM8/0.jpg)](https://youtu.be/E9CYBohndM8) ## Inspiration “Where did I put my keys?” “Who is this person?” “What was I doing a moment ago?” Over 55 million people worldwide live with dementia, and millions more elderly adults struggle with memory, safety, and independence. The rapidly growing development in wearable technology inspired us to think about how spatial computing can be applied to tackle the problems that the elderly face today. Our curiosity led us to build Clove, an AI-powered augmented reality application to help elders recall information, recognize loved ones, and stay safe through real-time support. ## What it does 1. 🧠 **Contextual memory recall:** Clove continually captures and indexes daily experiences. Simply ask a question like “Where did I put my keys” and receive instant, context-aware answers paired with a brief video clip of the exact moment—turning foggy memories into crystal-clear recall. > ex. “Where did I put my keys?”, “Did I turn the stove off?” 2. 👤 **Identity recognition overlays:** Puts names to important faces. Our AR nametags instantly identify loved ones, caregivers, and frequent visitors, displaying their name and relationship in your field of view. Add contacts with cherished photo memories, and never experience that uncomfortable moment of forgetting again. 3. 📞 **Two-way care calls:** Through Zoom-powered video calls, family members, doctors, and caregivers can see exactly what you see and connect with you in real-time, from taking medication to cooking a meal. > ex. “Call my daughter", “Start a Zoom call” 4. 📋 **Task manager and reminders:** Caregivers and family members can create personalized reminders for important tasks, such as taking the right medications on time. The system proactively notifies users at the right time with clear, step-by-step guidance, and confirms completion back to caregivers. > ex. Check “take Wednesday meds” off my list 5. 🚨 **Instant fall detection & alerts:** Help arrives before you ask. Motion sensors detect hard falls immediately and automatically notifies emergency contacts with your precise location. Give both users and families the peace of mind they deserve. 6. 📊 **Weekly cognitive reports:** After each Zoom call, the app uploads the meeting transcript to a PostgreSQL database on Render. A weekly cron job analyzes all conversations and generates a cognitive health report. The report emails to a caregiver with insights about the user's mental health. Analysis includes: cognitive scores, mood patterns, areas of concern and strength, and more. ## How we built it We built a native iOS app in Swift/SwiftUI leveraging a combination of on-device Apple frameworks and cloud APIs to power each core feature, with the goal of shipping in the future to devices like Meta Ray-Bans. - **Contextual memory recall:** The app continuously records rolling 6-second video clips with AVFoundation and indexes them on-device using Vision (scene/text labels) and Natural Language (sentence embeddings) for semantic search. OpenAI GPT-4o-mini improves clip descriptions. Voice input is handled by the Speech framework; we embed the query, find the best clip, and use the OpenAI API to generate a one-sentence answer, then play the clip and speak the answer with TTS. - **Identity recognition overlays:** Contacts are stored with face embeddings from Vision (landmarks). At runtime we detect faces in the AR feed, match them to stored embeddings, and show name and relationship above the person in the AR view. - **Two-way care calls:** We integrated the Zoom Video SDK for real-time video calls with transcript capture, and the VAPI voice AI platform for outbound phone calls. Both are accessible hands-free through our voice assistant—users simply say "call my daughter" or "start a Zoom call," and the assistant resolves the contact and initiates the session via OpenAI function calling. - **Instant fall detection & alerts:** We monitor the device accelerometer via CoreMotion and apply a fall detection algorithm that triggers a VAPI-powered emergency call to a pre-configured caregiver and pushes a critical local notification, all with a 10-second cooldown to prevent false re-triggers. - *Universal voice assistant:** We use OpenAI function calling so voice commands can trigger search_memories, Zoom (Zoom Video SDK), or call_contact (VAPI). A shared AppSpeechManager speaks confirmations and errors. - **Weekly cognitive reports:** Each Zoom call transcript is automatically stored in a PostgreSQL database hosted on Render. A weekly cron job aggregates the past week's conversations and sends them to OpenAI for analysis, generating cognitive health scores (clarity, coherence, memory recall), mood patterns, and conversation statistics like total calls and duration. The resulting report is emailed to caregivers via Resend, flagging any concerning patterns that may need attention. ## Challenges we ran into - **Adapting AR glasses-inspired vision for mobile form factor:** Translating AR glasses interactions to mobile devices required reimagining our approach to UX while conveying the original potential for real in-glasses deployment. AR glasses offer hands-free, persistent overlays in the user's natural field of view, while mobile devices demand active engagement and screen-based interfaces. We solved this by developing a hybrid interaction model that maintains spatial awareness on mobile through camera-based AR, while designing for future scalability to dedicated wearable hardware. - **Real-time facial recognition accuracy:** Achieving reliable facial recognition for elderly users presented unique obstacles—varied lighting conditions, users with glasses or changing appearances, and the need to avoid false positives that could cause confusion. - **Efficient storage and semantic search:** Continuously recording and storing video clips would quickly become storage-prohibitive and computationally expensive. Our solution combines intelligent scene detection to capture only meaningful moments and an embedding system that converts video content into searchable vector representations, enabling natural language queries to surface relevant clips in milliseconds rather than hours ## What we learned - 🫂 **Empathy through design.** This experience pushed me to build for people whose daily challenges I don’t face myself. Designing for elderly users involves recognizing the real consequences when technology fails someone who depends on it. *- Linkai* - ⚖️ **Technical trade-offs.** I learned a lot about the technical trade-offs between speed and quality while working on this project. More frequent calls to GPT-4o mini would lead to higher quality memory recalls but would slow down performance noticeably. It was really important to find the right balance between the two. *- Dylan* - 💡 **Narrowing down ideas.** This hackathon taught me to be specific and always have the end-user in mind. What do they want? What do they need? It is common for developers to build features that receive little appreciation from the public, which further emphasizes the need to think critically about the product at every step. *- Kevin* - ⚙️ **Exposure to new tools/techniques.** This hackathon exposed me to feature mapping and how to use it for facial recognition. Also received exposure on how to integrate tools like Render into our project to automate sending emails that contain a summary of the user's actions and store user data. *- Will* ## What's next for Clove While our current prototype runs on mobile devices, our vision from day one has been deployment on dedicated AR glasses. The mobile version proves the concept, but the real magic happens when this technology lives naturally in a user's field of view: hands-free, always accessible, and truly seamless. As devices like Meta's Ray-Ban smart glasses and Apple's Vision products mature and become more affordable, we'll b [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 211 recognized source files, 1286 KB. - C (language) — detected in the code - Express (technology) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - PostgreSQL (technology) — detected in the code - Swift (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 375) ``` .gitignore backend/.env.example backend/.gitignore backend/package.json backend/README.md backend/render.yaml backend/src/cron/weeklyAnalysis.js backend/src/db.js backend/src/index.js backend/src/migrate.js backend/src/routes/analysis.js backend/src/routes/transcripts.js backend/src/services/analysisService.js cron-reminder/package.json cron-reminder/send-reminder.js Frameworks/.DS_Store Frameworks/CptShare.xcframework/_CodeSignature/CodeDirectory Frameworks/CptShare.xcframework/_CodeSignature/CodeRequirements Frameworks/CptShare.xcframework/_CodeSignature/CodeRequirements-1 Frameworks/CptShare.xcframework/_CodeSignature/CodeResources Frameworks/CptShare.xcframework/_CodeSignature/CodeSignature Frameworks/CptShare.xcframework/.DS_Store Frameworks/CptShare.xcframework/Info.plist Frameworks/CptShare.xcframework/ios-arm64-simulator/CptShare.framework/_CodeSignature/CodeResources Frameworks/CptShare.xcframework/ios-arm64-simulator/CptShare.framework/CptShare Frameworks/CptShare.xcframework/ios-arm64-simulator/CptShare.framework/Info.plist Frameworks/CptShare.xcframework/ios-arm64-simulator/CptShare.framework/PrivacyInfo.xcprivacy Frameworks/CptShare.xcframework/ios-arm64/CptShare.framework/CptShare Frameworks/CptShare.xcframework/ios-arm64/CptShare.framework/Info.plist Frameworks/CptShare.xcframework/ios-arm64/CptShare.framework/PrivacyInfo.xcprivacy Frameworks/CptShare.xcframework/xros-arm64-simulator/CptShare.framework/_CodeSignature/CodeResources Frameworks/CptShare.xcframework/xros-arm64-simulator/CptShare.framework/CptShare Frameworks/CptShare.xcframework/xros-arm64-simulator/CptShare.framework/Info.plist Frameworks/CptShare.xcframework/xros-arm64-simulator/CptShare.framework/PrivacyInfo.xcprivacy Frameworks/CptShare.xcframework/xros-arm64/CptShare.framework/_CodeSignature/CodeResources Frameworks/CptShare.xcframework/xros-arm64/CptShare.framework/CptShare Frameworks/CptShare.xcframework/xros-arm64/CptShare.framework/Info.plist Frameworks/CptShare.xcframework/xros-arm64/CptShare.framework/PrivacyInfo.xcprivacy Frameworks/zoomcml.xcframework/_CodeSignature/CodeDirectory Frameworks/zoomcml.xcframework/_CodeSignature/CodeRequirements Frameworks/zoomcml.xcframework/_CodeSignature/CodeRequirements-1 Frameworks/zoomcml.xcframework/_CodeSignature/CodeResources Frameworks/zoomcml.xcframework/_CodeSignature/CodeSignature Frameworks/zoomcml.xcframework/.DS_Store Frameworks/zoomcml.xcframework/Info.plist Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/_CodeSignature/CodeResources Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/afn_iOS_hori/afn_iOS_hori.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/afn_iOS_hori/afn_iOS_hori.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/afn_iOS_vert/afn_iOS_vert.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/afn_iOS_vert/afn_iOS_vert.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/bgr_iOS_hori/bgr_iOS_hori.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/bgr_iOS_hori/bgr_iOS_hori.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/bgr_iOS_vert/bgr_iOS_vert.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/bgr_iOS_vert/bgr_iOS_vert.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/Headers/zoomcml_interface.h Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/Info.plist Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/PrivacyInfo.xcprivacy Frameworks/zoomcml.xcframework/ios-arm64-simulator/zoomcml.framework/zoomcml Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/.DS_Store Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/afn_iOS_hori/afn_iOS_hori.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/afn_iOS_hori/afn_iOS_hori.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/afn_iOS_vert/afn_iOS_vert.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/afn_iOS_vert/afn_iOS_vert.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/bgr_iOS_hori/bgr_iOS_hori.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/bgr_iOS_hori/bgr_iOS_hori.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/bgr_iOS_vert/bgr_iOS_vert.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/bgr_iOS_vert/bgr_iOS_vert.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/Headers/zoomcml_interface.h Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/Info.plist Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/PrivacyInfo.xcprivacy Frameworks/zoomcml.xcframework/ios-arm64/zoomcml.framework/zoomcml Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/_CodeSignature/CodeResources Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/afn_iOS_hori/afn_iOS_hori.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/afn_iOS_hori/afn_iOS_hori.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/afn_iOS_vert/afn_iOS_vert.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/afn_iOS_vert/afn_iOS_vert.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/bgr_iOS_hori/bgr_iOS_hori.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/bgr_iOS_hori/bgr_iOS_hori.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/bgr_iOS_vert/bgr_iOS_vert.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/bgr_iOS_vert/bgr_iOS_vert.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/Headers/zoomcml_interface.h Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/Info.plist Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/PrivacyInfo.xcprivacy Frameworks/zoomcml.xcframework/xros-arm64-simulator/zoomcml.framework/zoomcml Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/_CodeSignature/CodeResources Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/afn_iOS_hori/afn_iOS_hori.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/afn_iOS_hori/afn_iOS_hori.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/afn_iOS_vert/afn_iOS_vert.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/afn_iOS_vert/afn_iOS_vert.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/bgr_iOS_hori/bgr_iOS_hori.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/bgr_iOS_hori/bgr_iOS_hori.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/bgr_iOS_vert/bgr_iOS_vert.mlmodelc/metadata.json Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/bgr_iOS_vert/bgr_iOS_vert.mlmodelc/model.espresso.weights Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/Headers/zoomcml_interface.h Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/Info.plist Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/PrivacyInfo.xcprivacy Frameworks/zoomcml.xcframework/xros-arm64/zoomcml.framework/zoomcml Frameworks/ZoomTask.xcframework/_CodeSignature/CodeDirectory Frameworks/ZoomTask.xcframework/_CodeSignature/CodeRequirements Frameworks/ZoomTask.xcframework/_CodeSignature/CodeRequirements-1 Frameworks/ZoomTask.xcframework/_CodeSignature/CodeResources Frameworks/ZoomTask.xcframework/_CodeSignature/CodeSignature Frameworks/ZoomTask.xcframework/.DS_Store Frameworks/ZoomTask.xcframework/Info.plist Frameworks/ZoomTask.xcframework/ios-arm64-simulator/ZoomTask.framework/_CodeSignature/CodeResources Frameworks/ZoomTask.xcframework/ios-arm64-simulator/ZoomTask.framework/Info.plist Frameworks/ZoomTask.xcframework/ios-arm64-simulator/ZoomTask.framework/PrivacyInfo.xcprivacy Frameworks/ZoomTask.xcframework/ios-arm64-simulator/ZoomTask.framework/ZoomTask Frameworks/ZoomTask.xcframework/ios-arm64/ZoomTask.framework/Info.plist Frameworks/ZoomTask.xcframework/ios-arm64/ZoomTask.framework/PrivacyInfo.xcprivacy Frameworks/ZoomTask.xcframework/ios-arm64/ZoomTask.framework/ZoomTask Frameworks/ZoomTask.xcframework/xros-arm64-simulator/ZoomTask.framework/_CodeSignature/CodeResources Frameworks/ZoomTask.xcframework/xros-arm64-simulator/ZoomTask.framework/Info.plist Frameworks/ZoomTask.xcframework/xros-arm64-simulator/ZoomTask.framework/PrivacyInfo.xcprivacy Frameworks/ZoomTask.xcframework/xros-arm64-simulator/ZoomTask.framework/ZoomTask Frameworks/ZoomTask.xcframework/xros-arm64/ZoomTask.framework/_CodeSignature/CodeResources Frameworks/ZoomTask.xcframework/xros-arm64/ZoomTask.framework/Info.plist Frameworks/ZoomTask.xcframework/xros-arm64/ZoomTask.framework/PrivacyInfo.xcprivacy Frameworks/ZoomTask.xcframework/xros-arm64/ZoomTask.framework/ZoomTask Frameworks/ZoomVideoSDK.xcframework/_CodeSignature/CodeDirectory [255 more files omitted for size] ``` ### Dependencies - backend/package.json: cors@^2.8.5, dotenv@^16.4.5, express@^4.18.2, nodemon@^3.0.3, openai@^4.28.0, pg@^8.11.3, uuid@^9.0.1 ### Recent commits (newest first) - Update demo video section to YouTube link - added youtube demo - Add README - fixed video orientation - removed useless section - email config - SSL fix - weekly analysis - weekly analysis - render and transcripts - render + transcripts - transcription - transcription - VAPI change - Merge branch 'master' of https://github.com/dylanytran/treehacks-0 - zoom mic - added basic text to speech - adjusted video clip rotation and sizing - Add minimizable tab bar to camera view - Zoom Transcription ## Key source files (fetched from GitHub, selected and truncated for size) ### cron-reminder/package.json ``` { "name": "treehacks-reminder-cron", "version": "1.0.0", "description": "Render Cron Job: send a daily reminder email (e.g. for person with dementia)", "type": "module", "main": "send-reminder.js", "engines": { "node": ">=18" } } ``` ### backend/package.json ``` { "name": "iris-backend", "version": "1.0.0", "description": "Backend API for Iris app - transcript storage and analysis", "main": "src/index.js", "scripts": { "start": "node src/index.js", "dev": "nodemon src/index.js", "migrate": "node src/migrate.js", "weekly-analysis": "node src/cron/weeklyAnalysis.js" }, "dependencies": { "cors": "^2.8.5", "dotenv": "^16.4.5", "express": "^4.18.2", "openai": "^4.28.0", "pg": "^8.11.3", "uuid": "^9.0.1" }, "devDependencies": { "nodemon": "^3.0.3" }, "engines": { "node": ">=18.0.0" } } ``` ### backend/src/index.js ```javascript require('dotenv').config(); const express = require('express'); const cors = require('cors'); const transcriptRoutes = require('./routes/transcripts'); const analysisRoutes = require('./routes/analysis'); const app = express(); const PORT = process.env.PORT || 3000; // Middleware app.use(cors()); app.use(express.json({ limit: '10mb' })); // Health check app.get('/health', (req, res) => { res.json({ status: 'ok', timestamp: new Date().toISOString() }); }); // Routes app.use('/api/transcripts', transcriptRoutes); app.use('/api/analysis', analysisRoutes); // Error handler app.use((err, req, res, next) => { console.error('Error:', err); res.status(500).json({ error: err.message || 'Internal server error' }); }); app.listen(PORT, () => { console.log(`🚀 Iris API running on port ${PORT}`); }); ``` ### treehacksTests/treehacksTests.swift ```swift // // treehacksTests.swift // treehacksTests // // Created by Dylan Tran on 2/13/26. // import Testing @testable import treehacks struct treehacksTests { @Test func example() async throws { // Write your test here and use APIs like `#expect(...)` to check expected conditions. } } ``` ### treehacksUITests/treehacksUITestsLaunchTests.swift ```swift // // treehacksUITestsLaunchTests.swift // treehacksUITests // // Created by Dylan Tran on 2/13/26. // import XCTest final class treehacksUITestsLaunchTests: XCTestCase { override class var runsForEachTargetApplicationUIConfiguration: Bool { true } override func setUpWithError() throws { continueAfterFailure = false } @MainActor func testLaunch() throws { let app = XCUIApplication() app.launch() // Insert steps here to perform after app launch but before taking a screenshot, // such as logging into a test account or navigating somewhere in the app let attachment = XCTAttachment(screenshot: app.screenshot()) attachment.name = "Launch Screen" attachment.lifetime = .keepAlways add(attachment) } } ``` ### backend/render.yaml ```yaml # Render Blueprint - deploy with one click # See: https://render.com/docs/blueprint-spec databases: - name: iris-db plan: free databaseName: iris user: iris services: - type: web name: iris-api runtime: node plan: free buildCommand: npm install && npm run migrate startCommand: npm start healthCheckPath: /health envVars: - key: DATABASE_URL fromDatabase: name: iris-db property: connectionString - key: OPENAI_API_KEY sync: false - key: NODE_ENV value: production - type: cron name: weekly-analysis runtime: node schedule: "0 0 * * 0" # Every Sunday at midnight UTC buildCommand: npm install startCommand: npm run weekly-analysis envVars: - key: DATABASE_URL fromDatabase: name: iris-db property: connectionString - key: OPENAI_API_KEY sync: false - key: NODE_ENV value: production ``` ### treehacksUITests/treehacksUITests.swift ```swift // // treehacksUITests.swift // treehacksUITests // // Created by Dylan Tran on 2/13/26. // import XCTest final class treehacksUITests: XCTestCase { override func setUpWithError() throws { // Put setup code here. This method is called before the invocation of each test method in the class. // In UI tests it is usually best to stop immediately when a failure occurs. continueAfterFailure = false // In UI tests it’s important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this. } override func tearDownWithError() throws { // Put teardown code here. This method is called after the invocation of each test method in the class. } @MainActor func testExample() throws { // UI tests must launch the application that they test. let app = XCUIApplication() app.launch() // Use XCTAssert and related functions to verify your tests produce the correct results. } @MainActor func testLaunchPerformance() throws { if #available(macOS 10.15, iOS 13.0, tvOS 13.0, watchOS 7.0, *) { // This measures how long it takes to launch your application. measure(metrics: [XCTApplicationLaunchMetric()]) { XCUIApplication().launch() } } } } ``` ### cron-reminder/send-reminder.js ```javascript /** * Send a single reminder email via Resend. * Run by Render Cron Job on a schedule (e.g. daily). */ const REMINDER_EMAIL = process.env.REMINDER_EMAIL; const RESEND_API_KEY = process.env.RESEND_API_KEY; const REMINDER_SUBJECT = process.env.REMINDER_SUBJECT || "Take your medication"; const REMINDER_MESSAGE = process.env.REMINDER_MESSAGE || "This is a reminder to take your medication and have a sip of water."; const FROM_EMAIL = process.env.FROM_EMAIL || "Treehacks Reminder "; async function main() { if (!REMINDER_EMAIL?.trim()) { console.error("REMINDER_EMAIL is not set. Set it in Render → Cron Job → Environment."); process.exit(1); } if (!RESEND_API_KEY?.trim()) { console.error("RESEND_API_KEY is not set. Get a key at https://resend.com/api-keys"); process.exit(1); } const res = await fetch("https://api.resend.com/emails", { method: "POST", headers: { "Content-Type": "application/json", Authorization: `Bearer ${RESEND_API_KEY}`, }, body: JSON.stringify({ from: FROM_EMAIL, to: [REMINDER_EMAIL.trim()], subject: REMINDER_SUBJECT, html: `

${REMINDER_MESSAGE.replace(/\n/g, "
")}

Sent by Treehacks reminder.

`, }), }); const data = await res.json().catch(() => ({})); if (!res.ok) { console.error("Resend API error:", res.status, data); process.exit(1); } console.log("Reminder email sent to", REMINDER_EMAIL, "id:", data.id); } main(); ``` ### treehacks/treehacksApp.swift ```swift // // treehacksApp.swift // treehacks // // Created by Dylan Tran on 2/13/26. // import SwiftUI import SwiftData import UserNotifications // MARK: - App Delegate for Notification Handling class AppDelegate: NSObject, UIApplicationDelegate, UNUserNotificationCenterDelegate { func application( _ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]? = nil ) -> Bool { UNUserNotificationCenter.current().delegate = self return true } // Handle notification when app is in foreground func userNotificationCenter( _ center: UNUserNotificationCenter, willPresent notification: UNNotification, withCompletionHandler completionHandler: @escaping (UNNotificationPresentationOptions) -> Void ) { // Show notification even when app is in foreground completionHandler([.banner, .sound, .badge]) } // Handle notification tap actions func userNotificationCenter( _ center: UNUserNotificationCenter, didReceive response: UNNotificationResponse, withCompletionHandler completionHandler: @escaping () -> Void ) { let actionIdentifier = response.actionIdentifier switch actionIdentifier { case "CHECK_IN": print("User confirmed they are OK after fall") case "EMERGENCY": print("User needs help after fall - trigger emergency action") // TODO: Implement emergency contact or call functionality default: break } completionHandler() } } @main struct treehacksApp: App { @UIApplicationDelegateAdaptor(AppDelegate.self) var appDelegate @StateObject private var deepLinkManager = DeepLinkManager.shared var sharedModelContainer: ModelContainer = { let schema = Schema([MeetingTranscript.self]) let modelConfiguration = ModelConfiguration( schema: schema, isStoredInMemoryOnly: false ) do { return try ModelContainer(for: schema, configurations: [modelConfiguration]) } catch { fatalError("Could not create ModelContainer: \(error)") } }() var body: some Scene { WindowGroup { ContentView() .environmentObject(deepLinkManager) .onOpenURL { url in handleIncomingURL(url) } } } private func handleIncomingURL(_ url: URL) { guard url.scheme == "treehacks", url.host == "join", let components = URLComponents(url: url, resolvingAgainstBaseURL: false), let sessionName = components.queryItems?.first(where: { $0.name == "session" })?.value else { return } print("Deep link received: joining session '\(sessionName)'") deepLinkManager.pendingSessionName = sessionName deepLinkManager.shouldShowZoomCall = true } } // MARK: - Deep Link Manager class DeepLinkManager: ObservableObject { static let shared = DeepLinkManager() @Published var shouldShowZoomCall = false @Published var pendingSessionName: String? private init() {} } ``` ### treehacks/ContentView.swift ```swift // // ContentView.swift // treehacks // // Created by Dylan Tran on 2/13/26. // import SwiftUI import ZoomVideoSDK /// Root view with tab-based navigation. /// Designed with large, clear icons and labels for accessibility. struct ContentView: View { @StateObject private var cameraManager = CameraManager() @StateObject private var clipManager = ClipManager() @StateObject private var fallDetectionService = FallDetectionService() @ObservedObject private var zoomService = ZoomService.shared @State private var recordingManager: RecordingManager? @State private var showFullCallView = false @State private var showMiniControls = true @State private var selectedTab = 0 @State private var cameraTabBarExpanded = false // Draggable floating call overlay state @State private var floatingCallOffset: CGSize = .zero @State private var floatingCallPosition: CGPoint = CGPoint(x: UIScreen.main.bounds.width / 2, y: UIScreen.main.bounds.height - 180) var body: some View { Group { if let recordingManager = recordingManager { TabView(selection: $selectedTab) { // Camera Tab MainCameraView( cameraManager: cameraManager, recordingManager: recordingManager, clipManager: clipManager ) .toolbar(.hidden, for: .tabBar) .tabItem { Image(systemName: "camera.fill") Text("Camera") } .tag(0) // Contacts tab ContactsView() .tabItem { Image(systemName: "person.3.fill") Text("Contacts") } .tag(1) // Tasks Tab TasksListView() .tabItem { Image(systemName: "checklist") Text("Tasks") } .tag(2) // Settings Tab SettingsView(fallDetectionService: fallDetectionService, clipManager: clipManager, onStartZoomCall: { showFullCallView = true }) .tabItem { Image(systemName: "gear") Text("Settings") } .tag(3) } .tint(.blue) .overlay { if selectedTab == 0 { VStack { Spacer() if cameraTabBarExpanded { expandedCameraTabBar } else { HStack { Spacer() collapsedCameraTabButton } .padding(.horizontal, 20) } } .padding(.bottom, 16) .animation(.spring(response: 0.35, dampingFraction: 0.8), value: cameraTabBarExpanded) } } .onChange(of: selectedTab) { _, newTab in if newTab != 0 { cameraTabBarExpanded = false } } .onAppear { fallDetectionService.requestNotificationPermission() } .overlay { // Global floating call overlay when in Zoom session but minimized if zoomService.isInSession && !showFullCallView { GeometryReader { geometry in floatingCallOverlay(in: geometry) } .transition(.opacity) } } .animation(.spring(response: 0.3, dampingFraction: 0.8), value: zoomService.isInSession) } else { ProgressView("Setting up...") .onAppear { recordingManager = RecordingManager(cameraManager: cameraManager) } } } .fullScreenCover(isPresented: $showFullCallView) { ZoomCallView() } } // MARK: - Custom Camera Tab Bar private var expandedCameraTabBar: some View { HStack(spacing: 2) { cameraTabItem(icon: "camera.fill", label: "Camera", tag: 0) cameraTabItem(icon: "person.3.fill", label: "Contacts", tag: 1) cameraTabItem(icon: "checklist", label: "Tasks", tag: 2) cameraTabItem(icon: "gear", label: "Settings", tag: 3) // Collapse button Button { withAnimation(.spring(response: 0.35, dampingFraction: 0.8)) { cameraTabBarExpanded = false } } label: { Image(systemName: "xmark") .font(.system(size: 11, weight: .bold)) .foregroundColor(.white.opacity(0.5)) .frame(width: 26, height: 26) .background(Color.white.opacity(0.12), in: Circle()) } .padding(.leading, 4) } .padding(.horizontal, 16) .padding(.vertical, 10) .background(.ultraThinMaterial, in: Capsule()) .overlay(Capsule().strokeBorder(Color.white.opacity(0.15), lineWidth: 0.5)) .shadow(color: .black.opacity(0.3), radius: 20, y: 10) .padding(.horizontal, 20) .transition(.scale(scale: 0.5, anchor: .bottomTrailing).combined(with: .opacity)) } private [truncated — 9588 more characters] ``` [201 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]