# Project export: RemembAR Me This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: We believe in the essentials—human connections, memories, and tech tools for the modern hunter-gatherers. Spectacles+RemembAR Me are your AI companion for putting memories to a face and converse. - Devpost: https://devpost.com/software/remembar-me - GitHub: https://github.com/muhammedhunaid/calHacksXSnapChat-spectacle - Video: https://www.youtube.com/embed/HxwPiU-MrCU?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Muhammed Hunaid Topiwala (2 commits) ## Devpost submission (written by the team) ### Inspiration In today's fast-paced world, we're constantly meeting new people—at parties, networking events, business meetings, and social gatherings. Yet, we often struggle to remember the small but meaningful details about the people we meet: their favorite restaurants, shared experiences, or even the context of our last conversation. We've all been in that awkward moment at a party, seeing a familiar face but drawing a blank on their name or how we know them.We were inspired by the fundamental truth that memories define human interactions. The modern world demands that we be hunter-gatherers of social connections, constantly networking and building relationships, but our brains can only hold so much. We envisioned a future where technology doesn't distance us from authentic human connection—instead, it enhances our ability to remember and cherish the people who matter most. With Snapchat's Spectacles, we saw the perfect opportunity to create an AR companion that helps us see beyond the face and into the shared experiences that truly connect us. The Problem: We Forget 150+ relationships, <20% detail retention Forgotten conversations damage trust Memory decay statistics and emotional impact One Lens, Every Context Business: Series A discussions, introductions, partnership follow-ups Hiring: Candidate backgrounds, publications, H1B questions Friends: Ohio trip, December engagement, ski plans ### What it does RemembAR Me is an AI companion that transforms how you interact with the people around you by recognizing faces and surfacing your shared memories right when you need them. Using Snapchat Spectacles, RemembAR Me: Recognizes people that matter to you in real-time through the Spectacles camera Displays cherished memories associated with each person—the last two meaningful interactions you've had together Provides conversation starters based on your shared experiences, giving you natural pointers to build new memories together Manages your social network through an intuitive mobile app where you can add friends, store details about them, and organize what you want to remember Whether you're alone at a party wondering if you know anyone, in a business meeting trying to recall key details about a client, or simply want to have more mindful interactions with loved ones, RemembAR Me serves as your AI wingman—helping you engage authentically and meaningfully with the world around you. The experience spans three use cases: Social: Remembering dining memories and friends' favorite spots Personal: Keeping special moments with loved ones close to heart Business: Not missing out when things happen quickly ### How we built it We built RemembAR Me as an integrated system combining AR wearables, mobile application, and AI processing: Hardware/AR Layer: Leveraged Snapchat Spectacles as the AR display platform to provide hands-free, real-time visual overlays Implemented facial recognition capabilities to identify people in the user's field of view Technical: Head Binding for facial recognition persistence RAG with on-premise vector database for encrypted memory storage Pipeline: Face detect → Head bind → RAG retrieve → LLM generate → AR display (<500ms) Privacy-first architecture User Experience: Designed a seamless flow where the Spectacles display shows memories when recognizing someone Ensured privacy and discretion in how information is presented through the AR overlay Built a philosophy around enhancing rather than replacing genuine human connection ### Challenges we ran into Building RemembAR Me presented several interesting topics to consider: Technical Integration: Connecting the Spectacles hardware with our mobile app and backend systems proved complex. Ensuring real-time facial recognition while maintaining smooth AR overlays required careful optimization and balancing of processing loads between devices. Privacy and Ethics: We had to carefully consider the ethical implications of facial recognition technology and memory storage. Designing a system that respects privacy while still being useful required thoughtful boundaries and user controls. User Experience Design: Finding the right balance of information to display was crucial. Too much text overwhelms the user, too little isn't helpful. We iterated extensively on what memories to show, how to present them, and when to surface conversation starters without being intrusive. Facial Recognition Accuracy: Achieving reliable face recognition in various lighting conditions, angles, and real-world scenarios was challenging. We had to account for people changing appearances over time while maintaining accuracy. Memory Relevance: Creating an AI system that could determine which memories are most relevant and meaningful to surface at any given moment required sophisticated algorithms and understanding of social context. ### Accomplishments we're proud of We're incredibly proud of what we achieved with RemembAR Me: Seamless AR Integration: We successfully created a smooth, non-intrusive AR experience that enhances real-world interactions rather than distracting from them. The information appears naturally in the user's field of view through Spectacles. Human-Centered Design: We built a system that genuinely prioritizes authentic human connection. RemembAR Me doesn't replace social skills—it enhances them by helping users be more present and mindful in their interactions. Complete End-to-End System: In a limited timeframe, we delivered a functional prototype spanning AR hardware, mobile application, and AI backend—demonstrating the full vision of how this technology could work in real life. Innovative Use Case for Spectacles: We pushed the boundaries of what's possible with Snapchat Spectacles, showing how AR wearables can be used for social enhancement rather than just content creation. Thoughtful Philosophy: Beyond the tech, we're proud of the philosophical foundation we built—the belief that technology should serve as a tool for modern hunter-gatherers, helping us maintain meaningful connections in an increasingly complex social world. ### What we learned This project taught us valuable lessons across multiple dimensions: AR Development: We gained deep insights into the constraints and possibilities of AR wearable technology, learning how to design interfaces that work in the real world and respect users' cognitive load. AI Ethics: Working with facial recognition and personal memory data taught us the importance of building ethical safeguards from the ground up, not as an afterthought. Social Dynamics: We learned that technology for social interaction must be invisible when it's working well—the best social tools enhance connection without making users aware of the technology itself. User-Centered Iteration: Through testing and feedback, we learned that small details matter enormously in social applications. The way information is presented, timed, and framed can make the difference between helpful and creepy. Cross-Platform Development: Integrating multiple platforms (AR hardware, mobile, cloud AI) taught us valuable lessons about system architecture, data synchronization, and seamless user experiences across devices. ### What's next The future of RemembAR Me is bright, and we have ambitious plans to expand its capabilities: Enhanced AI Understanding: We plan to implement more sophisticated natural language processing to automatically extract and categorize memories from conversations, making the system more effortless to use. Contextual Intelligence: Future versions will understand context better—recognizing not just who you're talking to but where you are, what you're doing, and what type of interaction is appropriate (casual, professional, intimate). Expanded Memory Types: Beyond conversation memories, we want to integrate photos, shared locations, mutual friends, and even emotional context to create richer, more meaningful memory associations. Privacy-First Sharing: We envision a feature where two RemembAR Me users can opt-in to share and synchronize their memories of each other, creating a collaborative memory space that benefits both parties. Accessibility Features: We want to make RemembAR Me useful for people with memory-related conditions, social anxiety, or neurodivergence, helping everyone have more confident and authentic social interactions. Platform Expansion: While we started with Spectacles, we plan to support other AR platforms and even smartwatches, making the technology accessible to more users. Community-Driven Memories: Building features for group memories and shared experiences, perfect for families, friend groups, and professional teams. RemembAR Me represents a new era of connections—where technology doesn't distance us from humanity but helps us remember what makes us human: our shared experiences and the people who matter most. We're excited to see where this journey takes us! ## README (from the GitHub repository) # calHacksXSnapChat-spectacle ## Detected evidence (automated analysis) Indexed codebase: 13 recognized source files, 62 KB. - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (13 of 13) ``` FaceMemoryLens/Assets/Scripts/FaceMemoryController.ts FaceMemoryLens/Assets/Scripts/MemoryDisplayReferences.ts FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay_CORRECT.ts FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay_Fixed.ts FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay_Improved.ts FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay_STATIC.ts FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay_v2.ts FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay.ts FaceMemoryLens/QUICKSTART.md FaceMemoryLens/README.md FaceMemoryLens/TROUBLESHOOTING.md FaceMemoryLens/UI_IMPROVEMENTS.md README.md ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Merge pull request #1 from muhammedhunaid/face-memory-lens - Add FaceMemoryLens: Spectacles memory display proof of concept - Initial commit ## Key source files (fetched from GitHub, selected and truncated for size) ### FaceMemoryLens/QUICKSTART.md ```markdown # 🚀 Face Memory Lens - Quick Start Guide ## TL;DR This is a **proof-of-concept Spectacles lens** that shows conversation memories above faces. It's ready to demo with hardcoded data! ## ⚡ 3-Minute Setup ### 1. Open Lens Studio ``` File → New Project → Spectacles ``` ### 2. Add Script ``` Assets Panel → Right Click → Add New → Script Name: SimpleMemoryDisplay Copy/Paste: Assets/Scripts/SimpleMemoryDisplay.ts ``` ### 3. Create Scene Objects ``` Objects Panel → + → Text → 3D Text (name: "Memory Display") Objects Panel → + → Empty → Scene Object (name: "Memory Container") Drag "Memory Display" into "Memory Container" ``` ### 4. Configure ``` Select "Memory Container" Inspector → Add Component → Script → SimpleMemoryDisplay Assign: - Memory Text: Drag "Memory Display" here - Distance: 50 - Height Offset: 10 - Show On Start: ✅ - Update Interval: 5 ``` ### 5. Style Text ``` Select "Memory Display" Inspector: - Size: 2-3 - Color: White - Alignment: Center ``` ### 6. Test ``` Click Preview (Cmd+P / Ctrl+P) ``` ## 📱 What You'll See Text bubble in front of you showing: ``` 👤 Alice 💬 Last Conversation: • Discussed working on AR glasses startup • Talked about moving to San Francisco ❓ Ask: How's the SF apartment search going? ``` Automatically cycles through 3 different people every 5 seconds. ## 🎯 What This Shows ### Current (POC): - ✅ Text positioning in 3D space - ✅ Billboard effect (faces user) - ✅ Hardcoded conversation memories - ✅ Three bullet points format - ✅ Smooth animations ### Next Steps for Cal Hacks: 1. Add speech recognition 2. Connect to AI API 3. Generate memories in real-time 4. Add face recognition 5. Store conversation history ## 🔧 Customization ### Change Memories Edit in `SimpleMemoryDisplay.ts` line 25: ```typescript private memories = [ { person: "Your Friend's Name", topics: [ "First conversation point", "Second conversation point" ], question: "Question to ask?" }, // Add more... ]; ``` ### Adjust Position In Inspector: - **Distance**: How far (30-100 cm recommended) - **Height Offset**: How high (-20 to +30 cm) ### Change Cycle Speed - **Update Interval**: - 5 = 5 seconds per person - 0 = Manual only (no auto-cycle) ## 💡 Pro Tips ### Make It Look Better: 1. Add a background panel (Image component) 2. Use rounded corners 3. Add drop shadow 4. Increase font size to 3cm for better readability ### For Face Tracking: Use `FaceMemoryController.ts` instead - Requires Head component setup - Shows memory above detected faces - See full README.md for instructions ## 🐛 Quick Troubleshooting | Problem | Solution | |---------|----------| | Text not showing | Check text is assigned in Inspector | | Position wrong | Adjust Distance/Height Offset | | Text too small | Increase Size to 3-4 | | Can't read text | Change color to bright/white | | Script error | Check Lens Studio Logger panel | ## 📚 Files ``` FaceMemoryLens/ [truncated — 613 more characters] ``` ### FaceMemoryLens/TROUBLESHOOTING.md ```markdown # 🔧 Troubleshooting Guide ## Error: "Failed to resolve dependency" ### Problem ``` Failed to resolve dependency ./SimpleMemoryDisplay.ts in asset Assets/SimpleMemoryDisplay.ts.ts InternalError: Cannot find type: ./SimpleMemoryDisplay.ts ``` ### Root Cause The file got double extension `.ts.ts` instead of just `.ts` ### Solution 1: Create Script Correctly in Lens Studio **DO THIS:** 1. Assets Panel → Right Click → **Add New** → **Script** 2. When naming, type JUST the name: `SimpleMemoryDisplay` - ✅ Type: `SimpleMemoryDisplay` - ❌ DON'T type: `SimpleMemoryDisplay.ts` 3. Lens Studio will automatically add `.ts` 4. THEN paste the code ### Solution 2: Fix Existing File If you already created the file: 1. **Delete the existing script:** - Select `SimpleMemoryDisplay.ts.ts` in Assets - Right click → Delete 2. **Create new script:** - Assets → Right Click → Add New → Script - Name it: `SimpleMemoryDisplay` (no extension!) - Paste the code 3. **Alternative - Rename:** - If you see file named `SimpleMemoryDisplay.ts.ts` - Right click → Rename - Change to: `SimpleMemoryDisplay` (remove the double .ts) ## Step-by-Step Fresh Start ### 1. Clean Slate ``` Delete any existing SimpleMemoryDisplay scripts Close and reopen Lens Studio project ``` ### 2. Create Script the RIGHT Way **In Lens Studio:** ``` 1. Click Assets panel (bottom) 2. Right click in empty space 3. Select "Add New" → "Script" 4. A dialog appears asking for name 5. Type EXACTLY: SimpleMemoryDisplay (DO NOT add .ts!) 6. Click OK ``` ### 3. Copy Code Open the newly created script: - Double click `SimpleMemoryDisplay` in Assets panel - This opens the script editor - Select ALL existing code (Cmd+A / Ctrl+A) - DELETE it - Paste the code from `SimpleMemoryDisplay_Fixed.ts` - Save (Cmd+S / Ctrl+S) ### 4. Set Up Scene **Create Text Object:** ``` Objects Panel → + → Text → 3D Text Name: "MemoryDisplay" ``` **Create Container:** ``` Objects Panel → + → Empty → Scene Object Name: "MemoryContainer" Drag "MemoryDisplay" to be child of "MemoryContainer" ``` **Add Script:** ``` Select "MemoryContainer" Inspector → Add Component → Script Choose: SimpleMemoryDisplay ``` **Configure:** ``` In Inspector, assign: - Memory Text: Drag "MemoryDisplay" text here - Distance: 50 - Height Offset: 10 - Update Interval: 5 ``` ## Common Lens Studio Issues ### Issue: Script Won't Compile **Check Logger:** ``` Window → Logger Panel Look for red errors ``` **Common Causes:** 1. Missing semicolons 2. Typo in class name 3. Wrong file extension 4. Code copied incorrectly **Fix:** - Use `SimpleMemoryDisplay_Fixed.ts` (cleaner version) - Copy entire file contents - Make sure no characters are missing ### Issue: "Component not found" **Symptoms:** - Can't find script in Add Component menu **Fix:** 1. Make sure script compiled (check Logger) 2. Script name must match class name 3. Must have `@component` decorator 4. Try restarting Lens Studio ### Issue: Text Not Showing [truncated — 2983 more characters] ``` ### FaceMemoryLens/Assets/Scripts/MemoryDisplayReferences.ts ```typescript /** * Memory Display References * Organizes all UI component references for the memory display * Pattern inspired by Think Out Loud HeadLabelReferences */ @component export class MemoryDisplayReferences extends BaseScriptComponent { @input @hint("Text component for person's name") textPersonName: Text; @input @hint("Text component for conversation topic 1") textTopic1: Text; @input @hint("Text component for conversation topic 2") textTopic2: Text; @input @hint("Text component for suggested question") textQuestion: Text; @input @hint("Background panel object (optional)") backgroundPanel: SceneObject; @input @hint("Icon or avatar object (optional)") personIcon: SceneObject; } ``` ### FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay_CORRECT.ts ```typescript /** * Simple Memory Display - Correct Spectacles Pattern * Based on Think Out Loud example */ import WorldCameraFinderProvider from "SpectaclesInteractionKit.lspkg/Providers/CameraProvider/WorldCameraFinderProvider"; @component export class SimpleMemoryDisplay extends BaseScriptComponent { @input memoryText: Text; @input distance: number = 50.0; @input heightOffset: number = 10.0; @input updateInterval: number = 5.0; private currentMemoryIndex: number = 0; private memories = [ { person: "Alice", topics: [ "Discussed working on AR glasses startup", "Talked about moving to San Francisco" ], question: "How's the SF apartment search going?" }, { person: "Bob", topics: [ "Planning Cal Hacks hackathon project", "Learning TypeScript for Spectacles" ], question: "Did you finish the API integration?" }, { person: "Carol", topics: [ "Showed me her new Spectacles lens", "Mentioned graduating in Spring" ], question: "What are your post-grad plans?" } ]; // Use WorldCameraFinderProvider - the correct Spectacles pattern! private cameraTransform: Transform = WorldCameraFinderProvider.getInstance().getTransform(); private transform: Transform; private timeElapsed: number = 0; onAwake() { this.createEvent("OnStartEvent").bind(() => this.onStart()); this.createEvent("UpdateEvent").bind(() => this.onUpdate()); } onStart() { print("🧠 SimpleMemoryDisplay: Starting..."); this.transform = this.sceneObject.getTransform(); this.updateDisplay(); print("✅ SimpleMemoryDisplay: Ready!"); } onUpdate() { this.positionInFrontOfUser(); if (this.updateInterval > 0) { this.timeElapsed += getDeltaTime(); if (this.timeElapsed >= this.updateInterval) { this.timeElapsed = 0; this.nextMemory(); } } } private positionInFrontOfUser() { const cameraPos = this.cameraTransform.getWorldPosition(); const cameraForward = this.cameraTransform.forward; const worldUp = new vec3(0, 1, 0); const targetPos = cameraPos .add(cameraForward.uniformScale(this.distance)) .add(worldUp.uniformScale(this.heightOffset)); const currentPos = this.transform.getWorldPosition(); const smoothPos = vec3.lerp(currentPos, targetPos, getDeltaTime() * 3); this.transform.setWorldPosition(smoothPos); // Face the user (billboard effect) const lookDir = cameraPos.sub(smoothPos).normalize(); this.transform.setWorldRotation(quat.lookAt(lookDir, vec3.up())); } private updateDisplay() { if (!this.memoryText) { print("⚠️ No text component"); return; } const memory = this.memories[this.currentMemoryIndex]; const displayText = `👤 ${memory.person}\n\n💬 Last Conversation:\n• ${memory.topics[0]}\n• ${memory.topics[1]}\n\n❓ Ask: ${memory.question}`; this.memoryText.text = displayText; print(`📝 Showing: ${memory.person}`); } public nextMemory() { this.currentMemoryIndex = (this.currentMemoryIndex + 1) % this.memories.length; this.updateDisplay(); } public previousMemory() { this.currentMemoryIndex = (this.currentMemoryIndex - 1 + this.memories.length) % this.memories.length; this.updateDisplay(); } public addMemory(person: string, topics: string[], question: string) { this.memories.push({ person: person, topics: topics, question: question }); print(`✅ Added memory for ${person}`); } } ``` ### FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay_Fixed.ts ```typescript /** * Simple Memory Display * Displays conversation memories in front of user */ @component export class SimpleMemoryDisplay extends BaseScriptComponent { @input memoryText: Text; @input distance: number = 50.0; @input heightOffset: number = 10.0; @input updateInterval: number = 5.0; private currentMemoryIndex: number = 0; private memories = [ { person: "Alice", topics: [ "Discussed working on AR glasses startup", "Talked about moving to San Francisco" ], question: "How's the SF apartment search going?" }, { person: "Bob", topics: [ "Planning Cal Hacks hackathon project", "Learning TypeScript for Spectacles" ], question: "Did you finish the API integration?" }, { person: "Carol", topics: [ "Showed me her new Spectacles lens", "Mentioned graduating in Spring" ], question: "What are your post-grad plans?" } ]; private transform: Transform; private worldCameraTransform: Transform; private timeElapsed: number = 0; onAwake() { this.createEvent("OnStartEvent").bind(() => this.onStart()); this.createEvent("UpdateEvent").bind(() => this.onUpdate()); } onStart() { print("🧠 SimpleMemoryDisplay: Starting..."); this.transform = this.sceneObject.getTransform(); // Get camera transform const cameraObj = this.sceneObject.getScene().getRoot().find("Camera"); if (cameraObj) { this.worldCameraTransform = cameraObj.getTransform(); } else { this.worldCameraTransform = this.transform; } this.updateDisplay(); print("✅ SimpleMemoryDisplay: Ready!"); } onUpdate() { this.positionInFrontOfUser(); if (this.updateInterval > 0) { this.timeElapsed += getDeltaTime(); if (this.timeElapsed >= this.updateInterval) { this.timeElapsed = 0; this.nextMemory(); } } } private positionInFrontOfUser() { if (!this.worldCameraTransform) return; const cameraPos = this.worldCameraTransform.getWorldPosition(); const cameraForward = this.worldCameraTransform.forward; const worldUp = new vec3(0, 1, 0); const targetPos = cameraPos .add(cameraForward.uniformScale(this.distance)) .add(worldUp.uniformScale(this.heightOffset)); const currentPos = this.transform.getWorldPosition(); const smoothPos = vec3.lerp(currentPos, targetPos, getDeltaTime() * 3); this.transform.setWorldPosition(smoothPos); const lookDir = cameraPos.sub(smoothPos).normalize(); this.transform.setWorldRotation(quat.lookAt(lookDir, vec3.up())); } private updateDisplay() { if (!this.memoryText) { print("⚠️ No text component"); return; } const memory = this.memories[this.currentMemoryIndex]; const displayText = `👤 ${memory.person}\n\n💬 Last Conversation:\n• ${memory.topics[0]}\n• ${memory.topics[1]}\n\n❓ Ask: ${memory.question}`; this.memoryText.text = displayText; print(`📝 Showing: ${memory.person}`); } public nextMemory() { this.currentMemoryIndex = (this.currentMemoryIndex + 1) % this.memories.length; this.updateDisplay(); } public previousMemory() { this.currentMemoryIndex = (this.currentMemoryIndex - 1 + this.memories.length) % this.memories.length; this.updateDisplay(); } public addMemory(person: string, topics: string[], question: string) { this.memories.push({ person: person, topics: topics, question: question }); print(`✅ Added memory for ${person}`); } } ``` ### FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay_STATIC.ts ```typescript /** * Simple Memory Display - STATIC VERSION * Text stays in one place, doesn't follow camera */ import WorldCameraFinderProvider from "SpectaclesInteractionKit.lspkg/Providers/CameraProvider/WorldCameraFinderProvider"; @component export class SimpleMemoryDisplay extends BaseScriptComponent { @input memoryText: Text; @input distance: number = 50.0; @input heightOffset: number = 10.0; @input updateInterval: number = 5.0; private currentMemoryIndex: number = 0; private memories = [ { person: "Alice", topics: [ "Discussed working on AR glasses startup", "Talked about moving to San Francisco" ], question: "How's the SF apartment search going?" }, { person: "Bob", topics: [ "Planning Cal Hacks hackathon project", "Learning TypeScript for Spectacles" ], question: "Did you finish the API integration?" }, { person: "Carol", topics: [ "Showed me her new Spectacles lens", "Mentioned graduating in Spring" ], question: "What are your post-grad plans?" } ]; private cameraTransform: Transform = WorldCameraFinderProvider.getInstance().getTransform(); private transform: Transform; private timeElapsed: number = 0; private isPositioned: boolean = false; onAwake() { this.createEvent("OnStartEvent").bind(() => this.onStart()); this.createEvent("UpdateEvent").bind(() => this.onUpdate()); } onStart() { print("🧠 SimpleMemoryDisplay: Starting..."); this.transform = this.sceneObject.getTransform(); // Set initial static position this.setStaticPosition(); this.updateDisplay(); print("✅ SimpleMemoryDisplay: Ready!"); } onUpdate() { // Only cycle through memories, don't update position if (this.updateInterval > 0) { this.timeElapsed += getDeltaTime(); if (this.timeElapsed >= this.updateInterval) { this.timeElapsed = 0; this.nextMemory(); } } } private setStaticPosition() { // Position text once at startup, then leave it there const cameraPos = this.cameraTransform.getWorldPosition(); const cameraForward = this.cameraTransform.forward; const worldUp = new vec3(0, 1, 0); const staticPos = cameraPos .add(cameraForward.uniformScale(this.distance)) .add(worldUp.uniformScale(this.heightOffset)); this.transform.setWorldPosition(staticPos); // Face the initial camera position const lookDir = cameraPos.sub(staticPos).normalize(); this.transform.setWorldRotation(quat.lookAt(lookDir, vec3.up())); this.isPositioned = true; print("📍 Text positioned at static location"); } private updateDisplay() { if (!this.memoryText) { print("⚠️ No text component"); return; } const memory = this.memories[this.currentMemoryIndex]; const displayText = `👤 ${memory.person}\n\n💬 Last Conversation:\n• ${memory.topics[0]}\n• ${memory.topics[1]}\n\n❓ Ask: ${memory.question}`; this.memoryText.text = displayText; print(`📝 Showing: ${memory.person}`); } public nextMemory() { this.currentMemoryIndex = (this.currentMemoryIndex + 1) % this.memories.length; this.updateDisplay(); } public previousMemory() { this.currentMemoryIndex = (this.currentMemoryIndex - 1 + this.memories.length) % this.memories.length; this.updateDisplay(); } public addMemory(person: string, topics: string[], question: string) { this.memories.push({ person: person, topics: topics, question: question }); print(`✅ Added memory for ${person}`); } } ``` ### FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay_v2.ts ```typescript /** * Simple Memory Display - Fixed Version * Displays conversation memories in front of user */ @component export class SimpleMemoryDisplay extends BaseScriptComponent { @input memoryText: Text; @input distance: number = 50.0; @input heightOffset: number = 10.0; @input updateInterval: number = 5.0; private currentMemoryIndex: number = 0; private memories = [ { person: "Alice", topics: [ "Discussed working on AR glasses startup", "Talked about moving to San Francisco" ], question: "How's the SF apartment search going?" }, { person: "Bob", topics: [ "Planning Cal Hacks hackathon project", "Learning TypeScript for Spectacles" ], question: "Did you finish the API integration?" }, { person: "Carol", topics: [ "Showed me her new Spectacles lens", "Mentioned graduating in Spring" ], question: "What are your post-grad plans?" } ]; private transform: Transform; private cameraTransform: Transform; private timeElapsed: number = 0; onAwake() { this.createEvent("OnStartEvent").bind(() => this.onStart()); this.createEvent("UpdateEvent").bind(() => this.onUpdate()); } onStart() { print("🧠 SimpleMemoryDisplay: Starting..."); this.transform = this.sceneObject.getTransform(); // Use global.scene to access the scene const scene = global.scene as Scene; const cameraObj = scene.getRoot().find("Camera"); if (cameraObj) { this.cameraTransform = cameraObj.getTransform(); print("✅ Found camera"); } else { // Fallback: use the script's own transform this.cameraTransform = this.transform; print("⚠️ No camera found, using fallback"); } this.updateDisplay(); print("✅ SimpleMemoryDisplay: Ready!"); } onUpdate() { this.positionInFrontOfUser(); if (this.updateInterval > 0) { this.timeElapsed += getDeltaTime(); if (this.timeElapsed >= this.updateInterval) { this.timeElapsed = 0; this.nextMemory(); } } } private positionInFrontOfUser() { if (!this.cameraTransform) return; const cameraPos = this.cameraTransform.getWorldPosition(); const cameraForward = this.cameraTransform.forward; const worldUp = new vec3(0, 1, 0); const targetPos = cameraPos .add(cameraForward.uniformScale(this.distance)) .add(worldUp.uniformScale(this.heightOffset)); const currentPos = this.transform.getWorldPosition(); const smoothPos = vec3.lerp(currentPos, targetPos, getDeltaTime() * 3); this.transform.setWorldPosition(smoothPos); const lookDir = cameraPos.sub(smoothPos).normalize(); this.transform.setWorldRotation(quat.lookAt(lookDir, vec3.up())); } private updateDisplay() { if (!this.memoryText) { print("⚠️ No text component"); return; } const memory = this.memories[this.currentMemoryIndex]; const displayText = `👤 ${memory.person}\n\n💬 Last Conversation:\n• ${memory.topics[0]}\n• ${memory.topics[1]}\n\n❓ Ask: ${memory.question}`; this.memoryText.text = displayText; print(`📝 Showing: ${memory.person}`); } public nextMemory() { this.currentMemoryIndex = (this.currentMemoryIndex + 1) % this.memories.length; this.updateDisplay(); } public previousMemory() { this.currentMemoryIndex = (this.currentMemoryIndex - 1 + this.memories.length) % this.memories.length; this.updateDisplay(); } public addMemory(person: string, topics: string[], question: string) { this.memories.push({ person: person, topics: topics, question: question }); print(`✅ Added memory for ${person}`); } } ``` ### FaceMemoryLens/Assets/Scripts/SimpleMemoryDisplay.ts ```typescript /** * Simple Memory Display (No Face Tracking Required) * * Displays conversation memory in front of the user * Easier to test on Spectacles without face tracking setup * * Shows: * - Last 2 things discussed * - 1 suggested follow-up question */ @component export class SimpleMemoryDisplay extends BaseScriptComponent { @input @hint("Text component for displaying memory") memoryText: Text; @input @hint("Distance in front of user (in cm)") distance: number = 50.0; @input @hint("Height offset (in cm)") heightOffset: number = 10.0; @input @hint("Show memory on startup") showOnStart: boolean = true; @input @hint("Update interval in seconds (0 = no auto-update)") updateInterval: number = 5.0; // Hardcoded conversation data for POC private currentMemoryIndex: number = 0; private memories = [ { person: "Alice", topics: [ "Discussed working on AR glasses startup", "Talked about moving to San Francisco" ], question: "How's the SF apartment search going?" }, { person: "Bob", topics: [ "Planning Cal Hacks hackathon project", "Learning TypeScript for Spectacles" ], question: "Did you finish the API integration?" }, { person: "Carol", topics: [ "Showed me her new Spectacles lens", "Mentioned graduating in Spring" ], question: "What are your post-grad plans?" } ]; private transform: Transform; private worldCameraTransform: Transform; private timeElapsed: number = 0; onAwake() { this.createEvent("OnStartEvent").bind(() => this.onStart()); this.createEvent("UpdateEvent").bind(() => this.onUpdate()); } onStart() { print("🧠 SimpleMemoryDisplay: Starting..."); // Get transforms this.transform = this.sceneObject.getTransform(); // Get world camera transform for positioning try { const camera = this.sceneObject.getScene().getRoot().find("Camera"); if (camera) { this.worldCameraTransform = camera.getTransform(); } else { // Fallback: use current scene object's transform this.worldCameraTransform = this.transform; } } catch (e) { print("⚠️ SimpleMemoryDisplay: Using fallback transform"); this.worldCameraTransform = this.transform; } // Initialize display if (this.showOnStart) { this.updateDisplay(); } print("✅ SimpleMemoryDisplay: Initialized"); } onUpdate() { // Position in front of user this.positionInFrontOfUser(); // Auto-rotate through memories if enabled if (this.updateInterval > 0) { this.timeElapsed += getDeltaTime(); if (this.timeElapsed >= this.updateInterval) { this.timeElapsed = 0; this.nextMemory(); } } } /** * Position text in front of user's view */ private positionInFrontOfUser() { if (!this.worldCameraTransform) return; const cameraPos = this.worldCameraTransform.getWorldPosition(); const cameraForward = this.worldCameraTransform.forward; const worldUp = new vec3(0, 1, 0); // Position in front and slightly above const targetPos = cameraPos .add(cameraForward.uniformScale(this.distance)) .add(worldUp.uniformScale(this.heightOffset)); // Smooth movement const currentPos = this.transform.getWorldPosition(); const smoothPos = vec3.lerp(currentPos, targetPos, getDeltaTime() * 3); this.transform.setWorldPosition(smoothPos); // Face the user const lookDir = cameraPos.sub(smoothPos).normalize(); this.transform.setWorldRotation(quat.lookAt(lookDir, vec3.up())); } /** * Update the displayed memory */ private updateDisplay() { if (!this.memoryText) { print("⚠️ SimpleMemoryDisplay: No text component assigned"); return; } const memory = this.memories[this.currentMemoryIndex]; // Format text with emojis and bullet points const displayText = `👤 ${memory.person}\n\n💬 Last Conversation:\n• ${memory.topics[0]}\n• ${memory.topics[1]}\n\n❓ Ask: ${memory.question}`; this.memoryText.text = displayText; print(`📝 Showing memory for ${memory.person}`); } /** * Show next memory (rotate through list) */ public nextMemory() { this.currentMemoryIndex = (this.currentMemoryIndex + 1) % this.memories.length; this.updateDisplay(); } /** * Show previous memory */ public previousMemory() { this.currentMemoryIndex = (this.currentMemoryIndex - 1 + this.memories.length) % this.memories.length; this.updateDisplay(); } /** * Add new memory data (for AI integration later) */ public addMemory(person: string, topics: string[], question: string) { this.memories.push({ person: person, topics: topics, question: question }); print(`✅ Added memory for ${person}`); } } ``` ### FaceMemoryLens/Assets/Scripts/FaceMemoryController.ts ```typescript /** * Face Memory Controller * * Displays conversation memory above detected faces * Shows: * - Last 2 things discussed * - 1 suggested follow-up question * * For POC, uses hardcoded data */ @component export class FaceMemoryController extends BaseScriptComponent { @input @hint("Text component for displaying bullet points") memoryText: Text; @input @hint("SceneObject that will be positioned above the face") memoryBubble: SceneObject; @input @hint("Head component for face tracking") headComponent: Head; @input @hint("Distance above face to position bubble (in cm)") verticalOffset: number = 25.0; @input @hint("Distance forward from face (in cm)") forwardOffset: number = 15.0; // Hardcoded conversation data for POC private conversationMemories = { "Person A": { lastTopics: [ "Discussed their new job at Snap", "Talked about hiking in Yosemite" ], suggestedQuestion: "How's the new role going?" }, "Person B": { lastTopics: [ "Planning a trip to Japan next month", "Working on AR glasses project" ], suggestedQuestion: "Ready for your Japan trip?" }, "Default": { lastTopics: [ "We talked about AR technology", "You mentioned Cal Hacks project" ], suggestedQuestion: "How's your project coming along?" } }; private cameraTransform: Transform; private bubbleTransform: Transform; private isTracking: boolean = false; onAwake() { this.createEvent("OnStartEvent").bind(() => this.onStart()); this.createEvent("UpdateEvent").bind(() => this.onUpdate()); } onStart() { print("🧠 FaceMemoryController: Starting..."); // Get transforms if (this.memoryBubble) { this.bubbleTransform = this.memoryBubble.getTransform(); } else { print("❌ FaceMemoryController: Memory bubble not assigned!"); return; } // Get camera transform for positioning this.cameraTransform = this.sceneObject.getTransform(); // Subscribe to face tracking events if (this.headComponent) { print("✅ FaceMemoryController: Head component found, setting up face tracking"); this.setupFaceTracking(); } else { print("⚠️ FaceMemoryController: No head component assigned"); } // Initialize with default memory this.updateMemoryDisplay("Default"); print("✅ FaceMemoryController: Initialized"); } onUpdate() { if (!this.isTracking || !this.headComponent || !this.bubbleTransform) { return; } // Position the memory bubble above the tracked face this.positionBubbleAboveFace(); } /** * Set up face tracking event handlers */ private setupFaceTracking() { // Face found event const faceFoundEvent = this.createEvent("FaceFoundEvent") as FaceFoundEvent; faceFoundEvent.bind(() => { print("👤 Face detected!"); this.isTracking = true; this.memoryBubble.enabled = true; this.updateMemoryDisplay("Default"); // In real version, identify person }); // Face lost event const faceLostEvent = this.createEvent("FaceLostEvent") as FaceLostEvent; faceLostEvent.bind(() => { print("👋 Face lost"); this.isTracking = false; this.memoryBubble.enabled = false; }); } /** * Position the memory bubble above the tracked face */ private positionBubbleAboveFace() { // Get face position from camera/head transform const facePos = this.cameraTransform.getWorldPosition(); const faceForward = this.cameraTransform.forward; const worldUp = new vec3(0, 1, 0); // Position bubble above and in front of face const targetPos = facePos .add(faceForward.uniformScale(this.forwardOffset)) .add(worldUp.uniformScale(this.verticalOffset)); // Smooth position update const currentPos = this.bubbleTransform.getWorldPosition(); const smoothPos = vec3.lerp(currentPos, targetPos, getDeltaTime() * 5); this.bubbleTransform.setWorldPosition(smoothPos); // Make bubble face the camera this.bubbleTransform.setWorldRotation( quat.lookAt(faceForward.uniformScale(-1), vec3.up()) ); } /** * Update the displayed memory text */ private updateMemoryDisplay(personId: string) { if (!this.memoryText) { print("⚠️ FaceMemoryController: No text component assigned"); return; } // Get memory data for this person (or default) const memory = this.conversationMemories[personId] || this.conversationMemories["Default"]; // Format as bullet points const displayText = `💬 Last Time We Talked:\n\n• ${memory.lastTopics[0]}\n\n• ${memory.lastTopics[1]}\n\n❓ Ask: ${memory.suggestedQuestion}`; this.memoryText.text = displayText; print(`📝 Updated memory display for ${personId}`); } /** * Public method to update memory for a specific person * (For future AI integration) */ public setMemoryData(personId: string, topics: string[], question: string) { this.conversationMemories[personId] = { lastTopics: topics, suggestedQuestion: question }; this.updateMemoryDisplay(personId); } } ``` [2 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]