# Project export: Voithos This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: Imagine having a knowledgeable friend who can see what you see, understand how you feel, and answer any question in real-time—all through AR glasses. - Devpost: https://devpost.com/software/voithos-k6opbm - GitHub: https://github.com/SujashB/Voithos-New.git - Video: https://www.youtube.com/embed/x7yQfZdzALk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — SujashB (7 commits), Krishna Gupta (4 commits) ## Devpost submission (written by the team) ### Inspiration We've all experienced that awkward moment in a conversation where we're not sure what to say next, or when someone asks a question and we wish we could instantly access the answer without pulling out our phones. What if your glasses could whisper suggestions to help you navigate conversations naturally while giving you real-time access to verified web information—all without breaking eye contact or disrupting the flow of interaction? Voithos was born from the vision of creating a truly intelligent, context-aware AR companion that understands not just what you're saying, but how you're feeling, and provides the information you need exactly when you need it—all displayed seamlessly in your field of view. ### What it does Voithos is an AI-powered conversation assistant for Snap Spectacles that combines three powerful capabilities: 🎭 Emotion-Aware Context: Using REKA AI's vision capabilities, Voithos continuously analyzes your camera feed to understand the emotional context of your environment, helping tailor suggestions to the social situation you're in. 💬 Intelligent Conversation Assistance: Through natural voice interaction, Claude AI listens to your conversations and provides contextually relevant suggestions and responses—displayed as text on your AR glasses to help guide your discussions naturally. 🌐 Real-Time Web Search: When factual questions arise, Tavily's web search instantly delivers verified, real-time information from the internet. We've optimized the display to show ultra-compact results (around 10 words) that are perfect for AR glasses—giving you the essence of web sources without overwhelming your view. All of this happens hands-free, with information displayed directly in your AR glasses, allowing you to stay present in conversations while having an intelligent assistant at your fingertips. ### How we built it Platform: Snap Spectacles with Lens Studio v5.7.2+ Core Technologies: REKA AI: Real-time emotion and context analysis from camera feed Claude AI (Anthropic): Natural language processing for conversation suggestions Tavily API: Real-time web search with verified sources Lens Studio Speech-to-Text: Voice transcription JavaScript: Custom integration scripts connecting all services ### Challenges we ran into Script Initialization Issues: Initially, TavilySuggestions wasn't executing searches. We discovered it was referencing the wrong component—trying to read from the ClaudeIntegration script object instead of the actual Text component where Claude writes its output. Fixed by properly linking to claudeSuggestionText as a Component.Text type. Irrelevant Search Queries: Tavily was initially searching based on Claude's conversational suggestions (e.g., "Ask if they've attended one") rather than the user's actual question (e.g., "What is hackathon?"). We solved this by: Modifying ClaudeIntegration.js to store and expose the user's raw transcript via script.api.getUserTranscript() Adding logic in TavilySuggestions.js to prioritize the user's actual question over Claude's suggestions Implementing smart query rewriting to convert fragments into proper search queries Display Limitations: The biggest challenge was fitting web search results on AR glasses. Initial results were thousands of characters—way too long for the limited screen space. We solved this through aggressive optimization: Removed headers and footers Truncated AI summaries to 60 characters (attempting to capture first sentence) Limited to top 3 sources only ### Accomplishments we're proud of Created a fully functional multi-AI system running entirely on AR glasses with no external server required ✅ Solved the "too much information" problem for AR displays—our ultra-compact format delivers useful web results in under 10 words per item ✅ Built intelligent query routing that distinguishes between conversational AI suggestions and factual web search needs ✅ Achieved real-time performance with emotion detection, voice transcription, AI processing, and web search all happening simultaneously ✅ Seamless integration of three separate AI services (REKA, Claude, Tavily) into a cohesive user experience ✅ Hands-free interaction that keeps users present in conversations while providing intelligent assistance ### What we learned Technical Insights: AR displays require extreme content optimization—every character counts Component referencing in Lens Studio can be tricky; always verify you're accessing the right object type Distinguishing between different types of AI outputs (conversational vs. factual) requires thoughtful architecture String manipulation and truncation strategies need to respect word boundaries for readability Design Insights: Less is more in AR—showing 3 compact results is better than 20 overwhelming ones Users need clear attribution (web sources vs. AI-generated) to trust information Context-aware computing requires coordination between multiple AI systems Development Process: Extensive logging was crucial for debugging multi-component systems Iterative refinement based on real-world testing (display length, query relevance) dramatically improved the user experience ### What's next Short-term enhancements: Conversation memory: Track conversation history to provide more contextually relevant suggestions Multi-language support: Extend to languages beyond English Customizable display preferences: Let users adjust text size, result count, and verbosity Offline mode: Cache frequently asked questions for basic functionality without internet Long-term vision: Proactive assistance: Anticipate needs before users ask (e.g., showing restaurant info when looking at a storefront) Social graph integration: Provide context about people you meet (if they opt-in) Learning mode: Dedicated functionality for students to get instant explanations of concepts Professional applications: Specialized modes for medical professionals, technicians, or customer service roles Privacy-first sharing: Allow users to selectively share their AR assistant's insights with others in real-time ## README (from the GitHub repository) # Décor Assistant: AR Interior Design for Spectacles [![Experimental API](https://img.shields.io/badge/Spectacles-Experimental_API-FFFC00?logo=snapchat&labelColor=000000)](https://spectacles.snap.dev) [![World Query Hit](https://img.shields.io/badge/World_Query_Hit-AR_Placement-6E6E6E)](https://docs.snap.com/lens-studio/references/guides/worldmesh/worldmesh-overview) [![Spawn On Surface](https://img.shields.io/badge/Spawn_On_Surface-Object_Placement-8E8E8E)](https://docs.snap.com/lens-studio/references/guides/worldmesh/spawning-objects) [![Surface Detection](https://img.shields.io/badge/Surface_Detection-WorldMesh-9E9E9E)](https://docs.snap.com/lens-studio/references/guides/worldmesh/worldmesh-basics) [![SIK TTS](https://img.shields.io/badge/SIK-Text_To_Speech-00F4FF)](https://docs.snap.com/lens-studio/references/guides/audio/sik-text-to-speech) [![AI Vision](https://img.shields.io/badge/AI_Vision-WorldQueryHit-8A2BE2)](https://docs.snap.com/lens-studio/references/guides/worldmesh/worldmesh-overview) 🔗 **Built on Snap's [AI Assistant Sample](https://github.com/Snapchat/Spectacles-Sample/tree/main/AI%20Assistant)** *(Original features: Text-to-Speech, Camera Access, AI Vision, Fetch API)* --- ## Project Description Décor Assistant is an augmented reality tool built for Snap Spectacles, enabling users to visualize and interact with interior design concepts within their own physical spaces. This project leverages Snap's AI Assistant Sample as a foundation, extending its capabilities for intelligent spatial design. ## Base Project Features - Voice commands (SIK Text-to-Speech) - Real-time camera feed processing - HTTP requests via Fetch API ## Key Features **Surface-Aware AR Placement**: Precisely anchors virtual décor objects to detected walls and floors using WorldQueryHit (integrated from Snap's AI Assistant Sample). **AI-Powered Style Analysis**: Employs custom integration with GPT-4o to analyze scanned room environments and suggest relevant interior design styles. **DALLE Visualization**: Generates photorealistic image previews of redesigned spaces based on AI style recommendations and user customizations. **Dual Interaction Modes**: - Toggle between AI suggestions & manual drag-and-drop - **AI Suggestions Mode**: Receives automated décor style recommendations and visualizations driven by AI. - **Manual Design Mode**: Allows for direct and intuitive placement of virtual objects using SpawnOnSurface for personalized design. ## 🛠️ Technical Stack | Component | Source | |-----------|--------| | AR Foundation | Snap Spectacles Experimental API | | Surface Detection | Custom WorldQueryHit extensions | | AI Integration | OpenAI GPT-4o + DALLE-3 | | 3D Models | CREDITS.md | ## 🛠️ Technical Implementation **Base Project**: Modified version of Snap's AI Assistant Sample (developed with Lens Studio v5.7.2.25). **Key Extensions**: - **Custom OpenAI Integration**: Seamlessly integrates GPT-4o for intelligent style analysis and DALLE for realistic image generation. ## Demo Video ## 🎬 Demo [![Décor Assistant Demo](https://img.youtube.com/vi/n2HYkKANrEA/0.jpg)](https://www.youtube.com/watch?v=n2HYkKANrEA "Décor Assistant Demo - Click to Watch!") ## ⚙️ Setup ### Prerequisites - Snap Spectacles (2021 or later) - Lens Studio (v5.7.2.25 or later) - OpenAI API key with access to both GPT-4o and DALLE models. ### Run the Project 1. Clone the repository: ```bash git clone [this-repository-url] ## Detected evidence (automated analysis) Indexed codebase: 1594 recognized source files, 5934 KB. - JavaScript (language) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (120 of 3862) ``` .gitattributes .gitignore AI_Decor_Assistant.esproj API_KEYS_SETUP.md Assets/3D Models/Boho_paint.glb Assets/3D Models/Boho_paint.glb.meta Assets/3D Models/boho_Painting.gltf Assets/3D Models/boho_Painting.gltf.meta Assets/3D Models/Boho_rug.glb Assets/3D Models/Boho_rug.glb.meta Assets/3D Models/BOHO_Vase_Pack.fbx Assets/3D Models/BOHO_Vase_Pack.fbx.meta Assets/3D Models/Scandi_Painting 2.gltf Assets/3D Models/Scandi_Painting 2.gltf.meta Assets/3D Models/Scandi_rug.gltf Assets/3D Models/Scandi_rug.gltf.meta Assets/3D Models/Scandinavian_Vase_Set.gltf Assets/3D Models/Scandinavian_Vase_Set.gltf.meta Assets/3D Models/Vintage_Painting 2.gltf Assets/3D Models/Vintage_Painting 2.gltf.meta Assets/3D Models/Vintage_Persian_Rug.gltf Assets/3D Models/Vintage_Persian_Rug.gltf.meta Assets/3D Models/vintage_vase 2.gltf Assets/3D Models/vintage_vase 2.gltf.meta Assets/Device Camera Texture.deviceCameraTexture Assets/Device Camera Texture.deviceCameraTexture.meta Assets/Echopark.hdr Assets/Echopark.hdr.meta Assets/flat.ss_graph Assets/flat.ss_graph.meta Assets/Image 2.mat Assets/Image 2.mat.meta Assets/Image 3.mat Assets/Image 3.mat.meta Assets/Image.mat Assets/Image.mat.meta Assets/Render Target.renderTarget Assets/Render Target.renderTarget.meta Assets/Scene.scene Assets/Scene.scene.meta Assets/Scripts/Audio Output.audioOutput Assets/Scripts/Audio Output.audioOutput.meta Assets/Scripts/JS/CameraAPI.js Assets/Scripts/JS/CameraAPI.js.meta Assets/Scripts/JS/ClaudeIntegration.js Assets/Scripts/JS/ClaudeIntegration.js.meta Assets/Scripts/JS/GroqSceneAnalyzer.js Assets/Scripts/JS/GroqSceneAnalyzer.js.meta Assets/Scripts/JS/RekaEmotionAnalyzer.js Assets/Scripts/JS/RekaEmotionAnalyzer.js.meta Assets/Scripts/JS/SnapchatSTT.js Assets/Scripts/JS/SnapchatSTT.js.meta Assets/Scripts/JS/SocialBadges.js Assets/Scripts/JS/SocialBadges.js.meta Assets/Scripts/JS/SpeechToText.js Assets/Scripts/JS/SpeechToText.js.meta Assets/Scripts/JS/TavilySuggestions.js Assets/Scripts/JS/TavilySuggestions.js.meta Assets/Scripts/JS/TextToSpeechOpenAI.js Assets/Scripts/JS/TextToSpeechOpenAI.js.meta Assets/Scripts/JS/VisionOpenAI_DALLE.js Assets/Scripts/JS/VisionOpenAI_DALLE.js.meta Assets/SikSpatialImageFrame/3D Loading Indicator/.hidden/ui_loading [EDIT_ME].ss_graph Assets/SikSpatialImageFrame/3D Loading Indicator/.hidden/ui_loading [EDIT_ME].ss_graph.meta Assets/SikSpatialImageFrame/3D Loading Indicator/3D Loading Indicator Resources/Materials/UI/ui_loading [EDIT_ME].mat Assets/SikSpatialImageFrame/3D Loading Indicator/3D Loading Indicator Resources/Materials/UI/ui_loading [EDIT_ME].mat.meta Assets/SikSpatialImageFrame/3D Loading Indicator/3D Loading Indicator Resources/Meshes/UI/ui_loading_indicator.mesh Assets/SikSpatialImageFrame/3D Loading Indicator/3D Loading Indicator Resources/Meshes/UI/ui_loading_indicator.mesh.meta Assets/SikSpatialImageFrame/3D Loading Indicator/3D Loading Indicator Resources/Textures/white.png.meta Assets/SikSpatialImageFrame/SikSpatialImageFrame.prefab Assets/SikSpatialImageFrame/SikSpatialImageFrame.prefab.meta Assets/SikSpatialImageFrame/Spatial Image Template/Components/LoadingIndicator.ts Assets/SikSpatialImageFrame/Spatial Image Template/Components/LoadingIndicator.ts.meta Assets/SikSpatialImageFrame/Spatial Image Template/Components/SpatialImageAngleValidator.ts Assets/SikSpatialImageFrame/Spatial Image Template/Components/SpatialImageAngleValidator.ts.meta Assets/SikSpatialImageFrame/Spatial Image Template/Components/SpatialImageDepthAnimator.ts Assets/SikSpatialImageFrame/Spatial Image Template/Components/SpatialImageDepthAnimator.ts.meta Assets/SikSpatialImageFrame/Spatial Image Template/Components/SpatialImageFrame.ts Assets/SikSpatialImageFrame/Spatial Image Template/Components/SpatialImageFrame.ts.meta Assets/SikSpatialImageFrame/Spatial Image Template/Components/SpatialImageSwapper.ts Assets/SikSpatialImageFrame/Spatial Image Template/Components/SpatialImageSwapper.ts.meta Assets/SikSpatialImageFrame/Spatial Image Template/UIMaterials/image_unlit.ss_graph Assets/SikSpatialImageFrame/Spatial Image Template/UIMaterials/image_unlit.ss_graph.meta Assets/SikSpatialImageFrame/Spatial Image Template/UIMaterials/Image.mat Assets/SikSpatialImageFrame/Spatial Image Template/UIMaterials/Image.mat.meta Assets/SikSpatialImageFrame/Spatial Image Template/UIMaterials/Image.png.meta Assets/SikSpatialImageFrame/Spatial Image.lsc Assets/SikSpatialImageFrame/Spatial Image.lsc.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Audio/HoverAudioTrack.wav.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Audio/TriggerEndAudioTrack.wav.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Audio/TriggerStartAudioTrack.wav.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Materials/NormalBasedTargetOutline.mat Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Materials/NormalBasedTargetOutline.mat.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Materials/PinchButtonDarkRadial.mat Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Materials/PinchButtonDarkRadial.mat.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Materials/TargetOutlineMaterial.mat Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Materials/TargetOutlineMaterial.mat.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Meshes/ButtonSphereMesh.mesh Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Meshes/ButtonSphereMesh.mesh.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Shaders/NormalBasedTargetOutline.ss_graph Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Shaders/NormalBasedTargetOutline.ss_graph.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Shaders/PinchDarkRadial.ss_graph Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Shaders/PinchDarkRadial.ss_graph.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Shaders/TargetOutline.ss_graph Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Assets/Shaders/TargetOutline.ss_graph.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Helpers/InteractableAudioFeedback.ts Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Helpers/InteractableAudioFeedback.ts.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Helpers/InteractableColorFeedback.ts Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Helpers/InteractableColorFeedback.ts.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Helpers/InteractableOutlineFeedback.ts Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Helpers/InteractableOutlineFeedback.ts.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Helpers/InteractableSquishFeedback.ts Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Helpers/InteractableSquishFeedback.ts.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Helpers/ScreenTransformAdapter.ts Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Helpers/ScreenTransformAdapter.ts.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Interaction/Billboard/Billboard.ts Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Interaction/Billboard/Billboard.ts.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Interaction/Billboard/BillboardController.ts Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Interaction/Billboard/BillboardController.ts.meta Assets/SikSpatialImageFrame/SpectaclesInteractionKit/Components/Interaction/Billboard/BillboardRotationCalculator.ts [3742 more files omitted for size] ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - added groq and context history - added groq and context history - Added tavily web search - Add Snapchat STT, improved Claude integration, and API key security - Update ClaudeIntegration with better prompts and fix RekaEmotionAnalyzer error handling - Pushed Stuff - Merge pull request #2 from SujashB/varun-branch - added the rank glass, and the suggestions glass - fixed scene objects and made ui pretty - Merge pull request #1 from SujashB/krishna - Add Snapchat Speech-to-Text (STT) feature using VoiceML - Emotion recognition works - Added EmotionAnalyzer - Added stuff - first commit ## Key source files (fetched from GitHub, selected and truncated for size) ### API_KEYS_SETUP.md ```markdown # API Keys Setup Guide ## 🔑 How to Set Up API Keys This project uses several AI services that require API keys. **DO NOT commit your API keys to GitHub!** --- ## 📋 Required API Keys ### 1. **Claude/Anthropic API Key** (for conversation suggestions) - **Get it from:** https://console.anthropic.com/ - **Used in:** ClaudeIntegration.js - **Format:** `sk-ant-api03-...` - **Cost:** Pay-as-you-go (add credits to your account) ### 2. **Reka API Key** (for emotion detection) - **Get it from:** https://www.reka.ai/ - **Used in:** RekaEmotionAnalyzer.js - **Format:** Custom Reka format - **Cost:** Check Reka pricing ### 3. **Tavily API Key** (for web search) - **Get it from:** https://tavily.com/ - **Used in:** TavilySuggestions.js - **Format:** `tvly-...` - **Cost:** Free tier available, check Tavily pricing ### 4. **Deepgram API Key** (optional - if using external STT) - **Get it from:** https://deepgram.com/ - **Used in:** DeepgramSTT.js (if enabled) - **Format:** Custom Deepgram format ### 5. **Groq API Key** (for visual scene analysis) - **Get it from:** https://console.groq.com/ - **Used in:** GroqSceneAnalyzer.js - **Format:** `gsk_...` - **Cost:** Free tier available, very fast inference ### 6. **OpenAI API Key** (optional - for TTS/Vision) - **Get it from:** https://platform.openai.com/ - **Used in:** TextToSpeechOpenAI.js, VisionOpenAI_DALLE.js - **Format:** `sk-...` --- ## ⚙️ How to Add API Keys in Lens Studio ### **Method 1: Direct in Inspector (Recommended)** 1. Open your project in **Lens Studio** 2. In the **Objects Panel**, select the script component: - For Claude: Select object with **ClaudeIntegration** script - For Reka: Select object with **RekaEmotionAnalyzer** script - For Groq: Select object with **GroqSceneAnalyzer** script - For Tavily: Select object with **TavilySuggestions** script 3. In the **Inspector Panel**, find the API Key field: - **Claude Api Key** field - **Reka Api Key** field - **Groq Api Key** field - **Tavily Api Key** field 4. **Paste your API key** directly into the field 5. **Save the project** ### **Method 2: Using Config File (Advanced)** 1. Copy `config.example.js` to `config.js` 2. Fill in your actual API keys in `config.js` 3. `config.js` is gitignored and won't be pushed to GitHub --- ## ⚠️ Security Best Practices 1. **Never commit API keys** to Git 2. **Scene.scene is gitignored** - it contains your API keys from the Inspector 3. **Keep `config.js` local** - don't push it 4. **Rotate keys** if accidentally exposed 5. **Use separate keys** for development and production --- ## 🚨 If You Accidentally Pushed API Keys 1. **Rotate the keys immediately** at the service's console 2. **Remove from Git history:** ```bash git filter-branch --force --index-filter \ 'git rm --cached --ignore-unmatch Assets/Scene.scene' \ --prune-empty --tag-name-filter cat -- --all ``` 3. **Force push** (if necessary) --- ## ✅ Current Setup in Project | Service | Script [truncated — 779 more characters] ``` ### GROQ_INTEGRATION_SUMMARY.md ```markdown # Groq Scene Analyzer Integration Summary ## ✅ What Was Added ### 1. **GroqSceneAnalyzer.js** (NEW) - **Location:** `Assets/Scripts/JS/GroqSceneAnalyzer.js` - **Purpose:** Analyzes the visual scene from the user's perspective in real-time - **How it works:** - Captures camera feed from the Spectacles - Sends frames to Groq API every 5 seconds (configurable) - Uses Meta's Llama 4 Scout model for fast inference - Returns concise scene descriptions ### 2. **Modified ClaudeIntegration.js** - **New Input:** `groqSceneAnalyzerScript` - Reference to the Groq Scene Analyzer script component - **Integration:** Claude now receives visual context along with speech and emotion - **Enhanced Prompt:** Claude's suggestions now consider what the user is seeing ## 🔧 How It Works ``` Camera Feed → GroqSceneAnalyzer → Visual Description ↓ User Speech + Emotion + Visual Context → Claude → Conversation Suggestion ``` ## ⚙️ Setup Instructions ### 1. Add the Script Component in Lens Studio 1. Open your project in **Lens Studio** 2. Create a new object or select an existing one 3. Add the **GroqSceneAnalyzer** script component 4. Configure the required inputs: - **Camera Texture**: Drag the Device Camera Texture - **Scene Text**: Optional text component to display analysis - **Analysis Interval**: How often to analyze (default: 5.0 seconds) - **Groq Api Key**: Your Groq API key from https://console.groq.com/ - **Enable Debug Mode**: Toggle debug logging ### 2. Connect to ClaudeIntegration 1. Select the object with **ClaudeIntegration** script 2. In the Inspector, find **Groq Scene Analyzer Script** field 3. Drag the object that has the **GroqSceneAnalyzer** script component into this field 4. Save the project ### 3. Configure API Keys **Groq API Key:** - Sign up at https://console.groq.com/ - Get your API key (starts with `gsk_`) - Paste it into the Groq Scene Analyzer script component **Claude API Key:** - Already configured in ClaudeIntegration - Make sure it's set up correctly ## 📊 What Claude Receives Now When generating conversation suggestions, Claude considers: 1. **User's Speech** - What they said 2. **Emotion** - From RekaEmotionAnalyzer 3. **Visual Context** - From GroqSceneAnalyzer (NEW!) - Objects visible in the scene - People present - Text detected - Scene description - Colors and notable details ## 🎯 Example Scenario **Before (without visual context):** - User says: "That's really nice" - Claude suggests: "Share what specifically caught your eye" **After (with visual context):** - User says: "That's really nice" - Groq sees: "A modern living room with a blue sofa, plants, and abstract art" - Claude suggests: "The blue sofa and plants create a calming vibe - point that out" ## 🔍 Debugging If scene analysis isn't working: 1. **Check API Key:** Make sure your Groq API key is valid 2. **Check Camera Texture:** Ensure Device Camera Texture is connected 3. **Chec [truncated — 1091 more characters] ``` ### Cache/c9a24731-bc37-4eee-b4d6-98c50d72248c/b6187372ae5d389442c659d4e568c87c/Data/Main.js ```javascript //@input Component.ScriptComponent texturePreviewController //@input Component.ScriptComponent materialPreviewController //@input Component.ScriptComponent meshPreviewController //@input Component.ScriptComponent vfxPreviewController //@input Component.ScriptComponent otherObjectsPreviewController //@input Component.ScriptComponent gridController //@input Component.ScriptComponent placementScript const updateEvent = script.createEvent('LateUpdateEvent'); updateEvent.bind(update); function update() { if (global.scene.getRootObjectsCount() <= 1) { return; } let newObject = null; for (let i = global.scene.getRootObjectsCount() - 1; i >= 0; --i) { const root = global.scene.getRootObject(i); if (root.name === '[ROOT]') { continue; } newObject = root; } script.gridController.show(); newObject.enabled = true; var hasChild = (newObject.getChildrenCount() != 0); if (!hasChild && (newObject.name == "Texture" || newObject.name == "Video Texture")){ var image = newObject.getComponent("Component.Image"); if (image) { var mainPass = image.mainPass; if (mainPass){ hide(); var texture = mainPass.baseTex; script.texturePreviewController.show(texture) newObject.destroy(); } return; } } if (!hasChild && newObject.name == 'Material') { var renderMeshVisual = newObject.getComponent("Component.RenderMeshVisual"); if (renderMeshVisual){ hide(); script.placementScript.reparentNew(); script.materialPreviewController.show(newObject); return; } } if (!hasChild && newObject.name == 'Mesh') { var renderMeshVisual = newObject.getComponent("Component.RenderMeshVisual"); if (renderMeshVisual){ hide(); script.placementScript.reparentNew(); script.meshPreviewController.show(newObject); return; } } if (newObject.name == 'VFX Graph') { hide(); script.vfxPreviewController.show(newObject); return; } if (newObject.name == '_ngsVfxManager') { script.vfxPreviewController.setVfxManager(newObject); return; } hide(); script.placementScript.reparentNew(); script.otherObjectsPreviewController.show(); } function hide() { script.texturePreviewController.hide(); script.materialPreviewController.hide(); script.meshPreviewController.hide(); script.vfxPreviewController.hide(); script.otherObjectsPreviewController.hide(); } ``` ### Cache/Asset Preview/Assets/Scripts/Main.js ```javascript //@input Component.ScriptComponent texturePreviewController //@input Component.ScriptComponent materialPreviewController //@input Component.ScriptComponent meshPreviewController //@input Component.ScriptComponent vfxPreviewController //@input Component.ScriptComponent otherObjectsPreviewController //@input Component.ScriptComponent gridController //@input Component.ScriptComponent placementScript const updateEvent = script.createEvent('LateUpdateEvent'); updateEvent.bind(update); function update() { if (global.scene.getRootObjectsCount() <= 1) { return; } let newObject = null; for (let i = global.scene.getRootObjectsCount() - 1; i >= 0; --i) { const root = global.scene.getRootObject(i); if (root.name === '[ROOT]') { continue; } newObject = root; } script.gridController.show(); newObject.enabled = true; const hasChild = (newObject.getChildrenCount() != 0); const image = newObject.getComponent('Component.Image'); if (!hasChild && image) { if (image) { const mainPass = image.mainPass; if (mainPass) { hide(); const texture = mainPass.baseTex; script.texturePreviewController.show(texture); newObject.destroy(); } return; } } var renderMeshVisual = newObject.getComponent('Component.RenderMeshVisual'); if (!hasChild && renderMeshVisual && (newObject.name.includes('Mesh using') || newObject.name.includes('Post Effect:'))) { if (renderMeshVisual) { hide(); script.placementScript.reparentNew(); script.materialPreviewController.show(newObject); return; } } if (!hasChild && renderMeshVisual) { var renderMeshVisual = newObject.getComponent('Component.RenderMeshVisual'); if (renderMeshVisual) { hide(); script.placementScript.reparentNew(); script.meshPreviewController.show(newObject); return; } } const vfxComponent = newObject.getComponent('Component.VFXComponent'); if (vfxComponent) { hide(); script.vfxPreviewController.show(newObject); return; } if (newObject.name == '_ngsVfxManager') { script.vfxPreviewController.setVfxManager(newObject); return; } hide(); script.placementScript.reparentNew(); script.otherObjectsPreviewController.show(); } function hide() { script.texturePreviewController.hide(); script.materialPreviewController.hide(); script.meshPreviewController.hide(); script.vfxPreviewController.hide(); script.otherObjectsPreviewController.hide(); } ``` ### config.example.js ```javascript // API Keys Configuration // INSTRUCTIONS: // 1. Copy this file and rename it to: config.js // 2. Fill in your actual API keys below // 3. config.js is gitignored and will not be pushed to GitHub const API_KEYS = { // Claude/Anthropic API Key // Get from: https://console.anthropic.com/ CLAUDE_API_KEY: "sk-ant-api03-your-key-here", // Reka API Key // Get from: https://www.reka.ai/ REKA_API_KEY: "your-reka-key-here", // Tavily API Key (for web search) // Get from: https://tavily.com/ TAVILY_API_KEY: "tvly-your-key-here", // Groq API Key (for visual scene analysis) // Get from: https://console.groq.com/ GROQ_API_KEY: "gsk-your-key-here", // Deepgram API Key (optional) DEEPGRAM_API_KEY: "your-deepgram-key-here", // OpenAI API Key (optional) OPENAI_API_KEY: "your-openai-key-here" }; // Export for use in Lens Studio scripts module.exports = API_KEYS; ``` ### Assets/SpectaclesInteractionKit/SIK.ts ```typescript import {InteractionManager as InteractionManagerProvider} from "./Core/InteractionManager/InteractionManager" import {CursorControllerProvider} from "./Providers/CursorControllerProvider/CursorControllerProvider" import {HandInputData as HandInputDataProvider} from "./Providers/HandInputData/HandInputData" import {InteractionConfigurationProvider} from "./Providers/InteractionConfigurationProvider/InteractionConfigurationProvider" import SIKLogLevelProvider from "./Providers/InteractionConfigurationProvider/SIKLogLevelProvider" import {MobileInputData as MobileInputDataProvider} from "./Providers/MobileInputData/MobileInputData" export interface SIKAPI { SIKLogLevelProvider: SIKLogLevelProvider InteractionConfiguration: InteractionConfigurationProvider HandInputData: HandInputDataProvider MobileInputData: MobileInputDataProvider InteractionManager: InteractionManagerProvider CursorController: CursorControllerProvider } export const SIK: SIKAPI = new Proxy( {}, { get: (target, prop: keyof SIKAPI) => { switch (prop) { case "SIKLogLevelProvider": return SIKLogLevelProvider.getInstance() case "InteractionConfiguration": return InteractionConfigurationProvider.getInstance() case "HandInputData": return HandInputDataProvider.getInstance() case "MobileInputData": return MobileInputDataProvider.getInstance() case "InteractionManager": return InteractionManagerProvider.getInstance() case "CursorController": return CursorControllerProvider.getInstance() default: throw new Error(`Property ${String(prop)} does not exist on SIK API`) } }, }, ) as SIKAPI ``` ### Assets/World Query Hit - Spawn On Surface/WorldQueryHitExample.ts ```typescript // Import required modules const WorldQueryModule = require("LensStudio:WorldQueryModule"); const SIK = require("SpectaclesInteractionKit/SIK").SIK; const InteractorTriggerType = require("SpectaclesInteractionKit/Core/Interactor/Interactor").InteractorTriggerType; const EPSILON = 0.01; @component export class SurfacePlacementScript extends BaseScriptComponent { private primaryInteractor; private hitTestSession: HitTestSession; private transform: Transform; @input hitIndicator: SceneObject; // Simple 3D mesh for indicating hit area (before hit) @input objectsToSpawn: SceneObject[]; // [Painting, Rug, Vase] @input objectPreviews: SceneObject[]; // Preview versions of objects (after hit, before tap) @input filterEnabled: boolean; private objectToSpawnIndex: number = -1; onAwake() { // Create hit session this.hitTestSession = this.createHitTestSession(this.filterEnabled); if (!this.hitIndicator) { print("Error: Missing Hit Indicator object."); return; } this.transform = this.hitIndicator.getTransform(); this.hitIndicator.enabled = false; // Hide initially // Hide all preview objects initially if (this.objectPreviews) { for (let i = 0; i < this.objectPreviews.length; i++) { if (this.objectPreviews[i]) { this.objectPreviews[i].enabled = false; } } } this.createEvent("UpdateEvent").bind(this.onUpdate.bind(this)); } createHitTestSession(filterEnabled) { var options = HitTestSessionOptions.create(); options.filter = filterEnabled; return WorldQueryModule.createHitTestSessionWithOptions(options); } onHitTestResult(results) { // Hide all preview objects initially if (this.objectPreviews) { for (let i = 0; i < this.objectPreviews.length; i++) { if (this.objectPreviews[i]) { this.objectPreviews[i].enabled = false; } } } if (results === null) { // No hit, hide indicator this.hitIndicator.enabled = false; this.objectToSpawnIndex = -1; } else { const hitPosition = results.position; const hitNormal = results.normal.normalize(); // First, always show the hit indicator const lookDirection = (1 - Math.abs(hitNormal.dot(vec3.up())) < EPSILON) ? vec3.forward() : hitNormal.cross(vec3.up()); const toRotation = quat.lookAt(lookDirection, hitNormal); // Position and show the hit indicator this.hitIndicator.enabled = true; this.hitIndicator.getTransform().setWorldPosition(hitPosition); this.hitIndicator.getTransform().setWorldRotation(toRotation); // Decide what object to spawn const UP_DOT = hitNormal.dot(vec3.up()); const threshold = 0.75; if (UP_DOT > threshold) { // Upward surface if (hitPosition.y > 0.5) { this.objectToSpawnIndex = 2; // Vase } else { this.objectToSpawnIndex = 1; // Rug } } else if (Math.abs(UP_DOT) < 0.3) { // Wall (mostly vertical) this.objectToSpawnIndex = 0; // Painting } else { // Ignore ceilings or bad surfaces this.objectToSpawnIndex = -1; return; } // Check if the object index is valid if (this.objectToSpawnIndex < 0 || this.objectToSpawnIndex >= this.objectsToSpawn.length || !this.objectPreviews || this.objectToSpawnIndex >= this.objectPreviews.length) { return; } // When the primary trigger is pressed, show the preview of the actual object if (this.primaryInteractor && this.primaryInteractor.isActive() && this.primaryInteractor.currentTrigger !== InteractorTriggerType.None) { // Hide the hit indicator this.hitIndicator.enabled = false; // Show the appropriate preview object if (this.objectPreviews[this.objectToSpawnIndex]) { const previewObj = this.objectPreviews[this.objectToSpawnIndex]; previewObj.enabled = true; previewObj.getTransform().setWorldPosition(hitPosition); previewObj.getTransform().setWorldRotation(toRotation); } } } } onUpdate() { this.primaryInteractor = SIK.InteractionManager.getTargetingInteractors().shift(); if (this.primaryInteractor && this.primaryInteractor.isActive() && this.primaryInteractor.isTargeting() ) { const rayStartOffset = new vec3(this.primaryInteractor.startPoint.x, this.primaryInteractor.startPoint.y, this.primaryInteractor.startPoint.z + 30); const rayStart = rayStartOffset; const rayEnd = this.primaryInteractor.endPoint; this.hitTestSession.hitTest(rayStart, rayEnd, this.onHitTestResult.bind(this)); // Handle tap (trigger end) if ( this.objectToSpawnIndex !== -1 && this.primaryInteractor.previousTrigger !== InteractorTriggerType.None && this.primaryInteractor.currentTrigger === InteractorTriggerType.None ) { let spawnPosition, spawnRotation; // Get the position and rotation from either the preview or the indicator if (this.objectPreviews && [truncated — 2239 more characters] ``` ### Assets/SpectaclesInteractionKit/Utils/aabb.ts ```typescript /** * Axis-Aligned Bounding-Box representation */ export type aabb = { min: vec3 max: vec3 } ``` ### Assets/SpectaclesInteractionKit/Utils/LogLevel.ts ```typescript /** * The severity of a log message. */ export enum LogLevel { Error = 3, Warning = 4, Info = 6, Debug = 7, Verbose = 8, } ``` ### Assets/SpectaclesInteractionKit/Utils/logger.ts ```typescript const printFn = print export const logWithTag = (tag: string) => (...args: any[]) => { let result = `${tag}:` for (const arg of args) { result += " " + arg } printFn(result) } ``` [1573 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]