# Project export: Moli This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: The world is your classroom - AR Language learning - Devpost: https://devpost.com/software/moli - GitHub: https://github.com/AnishKamatam/CalHacksProject - Team: 3 GitHub contributor(s) — JaCARYK (6 commits), Anirudh Sivakumar (2 commits), AnishKamatam (1 commits) ## Devpost submission (written by the team) ### Inspiration Between learning languages in school, or utilizing Duolingo, there is one big pain point - immersion. You learn a lot about the language, but much of it might not be practical for you. That’s why we created Moli, where the world around you becomes your classroom. ### What it does Moli allows you to immerse yourself within your own environment and learn the translations of daily objects and moments of your own life. ### How we built it We combined the snapchat AR spectacles SDK with Vision learning models finetuned on hugginface and marked objects with a label in english and their spanish translation. We also utilized the new snapchat supported supabase integration to save these objects into on our database for a unique learning experience for each and every user. ### Challenges we ran into Figuring out how to use the Snapchat-Spectacles SDK was quite difficult at first but as we kept messing around with the glasses we were able to get a hang of developing on the spectacles! ### Accomplishments we're proud of We are proud of accomplishing depth caching for our objects with an intuitive UX/UI display above each object detailing the translation of each object we see. ### What we learned We learned how to develop AR applications using Lens Studio and had a blast building with Snapchat! ### What's next We would like to iterate on the user experience and optimize the latency between sending an image request to our huggingface backend instance and getting an object label -> translation -> bounding box on the snap spectacles ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 1726 recognized source files, 10720 KB. - JavaScript (language) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - Hugging Face (technology) — claimed on Devpost, not found in the code - Supabase (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 3706) ``` .gitignore Depth Cache Project/Assets/image_unlit.ss_graph Depth Cache Project/Assets/image_unlit.ss_graph.meta Depth Cache Project/Assets/Image.mat Depth Cache Project/Assets/Image.mat.meta Depth Cache Project/Assets/ImageMaterial.mat Depth Cache Project/Assets/ImageMaterial.mat.meta Depth Cache Project/Assets/Images/loading.gif.meta Depth Cache Project/Assets/Images/MicIcon.png.meta Depth Cache Project/Assets/Materials/CaptureTexture.mat Depth Cache Project/Assets/Materials/CaptureTexture.mat.meta Depth Cache Project/Assets/Materials/CircleOutline.mat Depth Cache Project/Assets/Materials/CircleOutline.mat.meta Depth Cache Project/Assets/Materials/circleOutline.ss_graph Depth Cache Project/Assets/Materials/circleOutline.ss_graph.meta Depth Cache Project/Assets/Materials/ImageMaterial.mat Depth Cache Project/Assets/Materials/ImageMaterial.mat.meta Depth Cache Project/Assets/Materials/label.ss_graph Depth Cache Project/Assets/Materials/label.ss_graph.meta Depth Cache Project/Assets/Materials/loadingGif.mat Depth Cache Project/Assets/Materials/loadingGif.mat.meta Depth Cache Project/Assets/Materials/loadingGif.ss_graph Depth Cache Project/Assets/Materials/loadingGif.ss_graph.meta Depth Cache Project/Assets/Materials/micIcon.mat Depth Cache Project/Assets/Materials/micIcon.mat.meta Depth Cache Project/Assets/Materials/middler.png.meta Depth Cache Project/Assets/Materials/MyAi_Voice 2.mat Depth Cache Project/Assets/Materials/MyAi_Voice 2.mat.meta Depth Cache Project/Assets/Materials/MyAi_Voice.mat Depth Cache Project/Assets/Materials/MyAi_Voice.mat.meta Depth Cache Project/Assets/Materials/MyAi_Voice.ss_graph Depth Cache Project/Assets/Materials/MyAi_Voice.ss_graph.meta Depth Cache Project/Assets/Materials/PinchButton.mat Depth Cache Project/Assets/Materials/PinchButton.mat.meta Depth Cache Project/Assets/Materials/pointRedLabeled.mat Depth Cache Project/Assets/Materials/pointRedLabeled.mat.meta Depth Cache Project/Assets/Materials/SWIS_flat_color.ss_graph Depth Cache Project/Assets/Materials/SWIS_flat_color.ss_graph.meta Depth Cache Project/Assets/Materials/SWIS_tile.mat Depth Cache Project/Assets/Materials/SWIS_tile.mat.meta Depth Cache Project/Assets/Materials/unlitColor.ss_graph Depth Cache Project/Assets/Materials/unlitColor.ss_graph.meta Depth Cache Project/Assets/Materials/unlitTexture.ss_graph Depth Cache Project/Assets/Materials/unlitTexture.ss_graph.meta Depth Cache Project/Assets/Materials/unlitTextureColor.ss_graph Depth Cache Project/Assets/Materials/unlitTextureColor.ss_graph.meta Depth Cache Project/Assets/Materials/White.mat Depth Cache Project/Assets/Materials/White.mat.meta Depth Cache Project/Assets/Materials/WorldLabelFrame.mat Depth Cache Project/Assets/Materials/WorldLabelFrame.mat.meta Depth Cache Project/Assets/Mesh/arrow.mesh Depth Cache Project/Assets/Mesh/arrow.mesh.meta Depth Cache Project/Assets/Mesh/SWIS_tile_solid.mesh Depth Cache Project/Assets/Mesh/SWIS_tile_solid.mesh.meta Depth Cache Project/Assets/Mesh/Unit Box.mesh Depth Cache Project/Assets/Mesh/Unit Box.mesh.meta Depth Cache Project/Assets/Mesh/Unit Plane.mesh Depth Cache Project/Assets/Mesh/Unit Plane.mesh.meta Depth Cache Project/Assets/Mesh/Unit Sphere.mesh Depth Cache Project/Assets/Mesh/Unit Sphere.mesh.meta Depth Cache Project/Assets/middler.png.meta Depth Cache Project/Assets/Modules/Camera Module.cameraModule Depth Cache Project/Assets/Modules/Camera Module.cameraModule.meta Depth Cache Project/Assets/Prefabs/coloredPoint.prefab Depth Cache Project/Assets/Prefabs/coloredPoint.prefab.meta Depth Cache Project/Assets/Prefabs/WorldArrow.prefab Depth Cache Project/Assets/Prefabs/WorldArrow.prefab.meta Depth Cache Project/Assets/Prefabs/WorldLabel.prefab Depth Cache Project/Assets/Prefabs/WorldLabel.prefab.meta Depth Cache Project/Assets/Rendering/Device Camera Texture.deviceCameraTexture Depth Cache Project/Assets/Rendering/Device Camera Texture.deviceCameraTexture.meta Depth Cache Project/Assets/Rendering/Echopark.hdr Depth Cache Project/Assets/Rendering/Echopark.hdr.meta Depth Cache Project/Assets/Rendering/Render Target.renderTarget Depth Cache Project/Assets/Rendering/Render Target.renderTarget.meta Depth Cache Project/Assets/Scene.scene Depth Cache Project/Assets/Scene.scene.meta Depth Cache Project/Assets/Scripts/APIKeyHint.ts Depth Cache Project/Assets/Scripts/APIKeyHint.ts.meta Depth Cache Project/Assets/Scripts/ASRController.ts Depth Cache Project/Assets/Scripts/ASRController.ts.meta Depth Cache Project/Assets/Scripts/DebugVisualizer.ts Depth Cache Project/Assets/Scripts/DebugVisualizer.ts.meta Depth Cache Project/Assets/Scripts/DepthCache.ts Depth Cache Project/Assets/Scripts/DepthCache.ts.meta Depth Cache Project/Assets/Scripts/FaceCamera.ts Depth Cache Project/Assets/Scripts/FaceCamera.ts.meta Depth Cache Project/Assets/Scripts/GeminiAPI.ts Depth Cache Project/Assets/Scripts/GeminiAPI.ts.meta Depth Cache Project/Assets/Scripts/Labels.ts Depth Cache Project/Assets/Scripts/Labels.ts.meta Depth Cache Project/Assets/Scripts/Loading.ts Depth Cache Project/Assets/Scripts/Loading.ts.meta Depth Cache Project/Assets/Scripts/ResponseUI.ts Depth Cache Project/Assets/Scripts/ResponseUI.ts.meta Depth Cache Project/Assets/Scripts/SceneController.ts Depth Cache Project/Assets/Scripts/SceneController.ts.meta Depth Cache Project/Assets/Scripts/SpanishTranslator.ts Depth Cache Project/Assets/Scripts/SpanishTranslator.ts.meta Depth Cache Project/Assets/Scripts/SpeechUI.ts Depth Cache Project/Assets/Scripts/SpeechUI.ts.meta Depth Cache Project/Assets/Scripts/TouchDownButton.ts Depth Cache Project/Assets/Scripts/TouchDownButton.ts.meta Depth Cache Project/Assets/Scripts/TranslationManager.ts Depth Cache Project/Assets/Scripts/TranslationManager.ts.meta Depth Cache Project/Assets/Scripts/WorldLabel.ts Depth Cache Project/Assets/Scripts/WorldLabel.ts.meta Depth Cache Project/Assets/SpanishTranslator.ts Depth Cache Project/Assets/SpanishTranslator.ts.meta Depth Cache Project/Assets/Untitled TypeScript.ts Depth Cache Project/Assets/Untitled TypeScript.ts.meta Depth Cache Project/Cache/0043176b-94f3-4069-3d0e-541784e948df/6a84cdf12669f7a19c1ae349d7ddea05/Data/1b32560c33651b8238f996f1ac585562/CursorController.js Depth Cache Project/Cache/0043176b-94f3-4069-3d0e-541784e948df/6a84cdf12669f7a19c1ae349d7ddea05/Data/87fd6b237859649347d3d34c96989606/CursorController.js Depth Cache Project/Cache/0043176b-94f3-4069-3d0e-541784e948df/6a84cdf12669f7a19c1ae349d7ddea05/Data/936e2d4b20e3c5ffa82278ccf2be010c/CursorController_c.js Depth Cache Project/Cache/0043176b-94f3-4069-3d0e-541784e948df/6a84cdf12669f7a19c1ae349d7ddea05/Data/cc2ae24479196d49022da0d1686dc6ea/CursorController_c.js Depth Cache Project/Cache/0043176b-94f3-4069-3d0e-541784e948df/6a84cdf12669f7a19c1ae349d7ddea05/Data/CursorController.ts Depth Cache Project/Cache/0043176b-94f3-4069-3d0e-541784e948df/6a84cdf12669f7a19c1ae349d7ddea05/Data/f82d86462fe8fc11c55a552dc7c571d5/CursorController.js.map Depth Cache Project/Cache/010ed072-7103-30d9-face-f69bf7ad0db2/61b549f69a6f17281d38d95536ae9690/Assets.cache Depth Cache Project/Cache/01379ad4-f661-765f-feda-98bf306d096c/2c9be101ca1c4f34a401092bd131ba6a/Assets.cache Depth Cache Project/Cache/01622a44-38f1-47c9-84db-7219c19066c3/9288968dc4759807510cecb73ace195d/Data/0a0c9de6fd46e05a78aa7867a6e7d7d5/APIKeyHint.js.map [3586 more files omitted for size] ``` ### Dependencies No dependency index available. ### Recent commits (newest first) - Add Spanish translation files: TranslationManager.ts and SpanishTranslator.ts - Delete Remote Object Detection directory - Add complete Remote Object Detection project with translation system - Delete Remote Object Detection/Assets/ObjectDetection/Scripts/TS directory - Add complete depth cache project with Spanish translation - Add Spanish translation system for object detection - structure - Delete context directory - Add Spectacles-Sample code (imported, .git removed) ## Key source files (fetched from GitHub, selected and truncated for size) ### Depth Cache Project/Cache/Asset Preview/Assets/Scripts/Main.js ```javascript //@input Component.ScriptComponent texturePreviewController //@input Component.ScriptComponent materialPreviewController //@input Component.ScriptComponent meshPreviewController //@input Component.ScriptComponent vfxPreviewController //@input Component.ScriptComponent otherObjectsPreviewController //@input Component.ScriptComponent gridController //@input Component.ScriptComponent placementScript const updateEvent = script.createEvent('LateUpdateEvent'); updateEvent.bind(update); function update() { if (global.scene.getRootObjectsCount() <= 1) { return; } let newObject = null; for (let i = global.scene.getRootObjectsCount() - 1; i >= 0; --i) { const root = global.scene.getRootObject(i); if (root.name === '[ROOT]') { continue; } newObject = root; } script.gridController.show(); newObject.enabled = true; const hasChild = (newObject.getChildrenCount() != 0); const image = newObject.getComponent('Component.Image'); if (!hasChild && image) { if (image) { const mainPass = image.mainPass; if (mainPass) { hide(); const texture = mainPass.baseTex; script.texturePreviewController.show(texture); newObject.destroy(); } return; } } var renderMeshVisual = newObject.getComponent('Component.RenderMeshVisual'); if (!hasChild && renderMeshVisual && (newObject.name.includes('Mesh using') || newObject.name.includes('Post Effect:'))) { if (renderMeshVisual) { hide(); script.placementScript.reparentNew(); script.materialPreviewController.show(newObject); return; } } if (!hasChild && renderMeshVisual) { var renderMeshVisual = newObject.getComponent('Component.RenderMeshVisual'); if (renderMeshVisual) { hide(); script.placementScript.reparentNew(); script.meshPreviewController.show(newObject); return; } } const vfxComponent = newObject.getComponent('Component.VFXComponent'); if (vfxComponent) { hide(); script.vfxPreviewController.show(newObject); return; } if (newObject.name == '_ngsVfxManager') { script.vfxPreviewController.setVfxManager(newObject); return; } hide(); script.placementScript.reparentNew(); script.otherObjectsPreviewController.show(); } function hide() { script.texturePreviewController.hide(); script.materialPreviewController.hide(); script.meshPreviewController.hide(); script.vfxPreviewController.hide(); script.otherObjectsPreviewController.hide(); } ``` ### Depth Cache Project/Cache/c9a24731-bc37-4eee-b4d6-98c50d72248c/31451bb309ea0eeb2355a7fd0cccda50/Data/Main.js ```javascript //@input Component.ScriptComponent texturePreviewController //@input Component.ScriptComponent materialPreviewController //@input Component.ScriptComponent meshPreviewController //@input Component.ScriptComponent vfxPreviewController //@input Component.ScriptComponent otherObjectsPreviewController //@input Component.ScriptComponent gridController //@input Component.ScriptComponent placementScript const updateEvent = script.createEvent('LateUpdateEvent'); updateEvent.bind(update); function update() { if (global.scene.getRootObjectsCount() <= 1) { return; } let newObject = null; for (let i = global.scene.getRootObjectsCount() - 1; i >= 0; --i) { const root = global.scene.getRootObject(i); if (root.name === '[ROOT]') { continue; } newObject = root; } script.gridController.show(); newObject.enabled = true; const hasChild = (newObject.getChildrenCount() != 0); const image = newObject.getComponent('Component.Image'); if (!hasChild && image) { if (image) { const mainPass = image.mainPass; if (mainPass) { hide(); const texture = mainPass.baseTex; script.texturePreviewController.show(texture); newObject.destroy(); } return; } } var renderMeshVisual = newObject.getComponent('Component.RenderMeshVisual'); if (!hasChild && renderMeshVisual && (newObject.name.includes('Mesh using') || newObject.name.includes('Post Effect:'))) { if (renderMeshVisual) { hide(); script.placementScript.reparentNew(); script.materialPreviewController.show(newObject); return; } } if (!hasChild && renderMeshVisual) { var renderMeshVisual = newObject.getComponent('Component.RenderMeshVisual'); if (renderMeshVisual) { hide(); script.placementScript.reparentNew(); script.meshPreviewController.show(newObject); return; } } const vfxComponent = newObject.getComponent('Component.VFXComponent'); if (vfxComponent) { hide(); script.vfxPreviewController.show(newObject); return; } if (newObject.name == '_ngsVfxManager') { script.vfxPreviewController.setVfxManager(newObject); return; } hide(); script.placementScript.reparentNew(); script.otherObjectsPreviewController.show(); } function hide() { script.texturePreviewController.hide(); script.materialPreviewController.hide(); script.meshPreviewController.hide(); script.vfxPreviewController.hide(); script.otherObjectsPreviewController.hide(); } ``` ### Depth Cache Project/Workspaces/WorkspacesState.yaml ```yaml order: - Default current: Default ``` ### Depth Cache Project/PluginsUserPreferences/UserLogCollectorService.data.yaml ```yaml data: clearOnReset: false showAllCoreLogs: false ``` ### Depth Cache Project/PluginsUserPreferences/Snap_Plugin_Gui_InspectorPanel_780d9164-e681-4de1-b586-5213ed5523b9.yaml ```yaml PinnedEntities: type: uuids value: size: 0 options: 4 ``` ### Depth Cache Project/PluginsUserPreferences/Snap_Plugin_Gui_SceneEditor_a14d85d7-9a15-42a6-913e-3f24dea267c2.yaml ```yaml PinnedEntities: type: uuids value: size: 0 options: 4 ``` ### Depth Cache Project/Assets/Untitled TypeScript.ts ```typescript @component export class NewScript extends BaseScriptComponent { onAwake() { } } ``` ### Depth Cache Project/PluginsUserPreferences/ProjectLocalData.data.yaml ```yaml data: externalSources: - 031e38d4-742d-4c79-90f8-4e562d8d0bce: /Users/jacobkadari/Downloads/middler.png - 1226cb7b-474d-45ab-80ce-2b9c79d05d35: /Users/jacobkadari/Downloads/middler.png - 63882c1d-6c2c-48d4-9362-548ffc018098: /Users/jacobkadari/Desktop/SpanishTranslator.ts - 791f925c-d4a0-44da-9757-3764eacb1962: /Applications/Lens Studio.app/Contents/Plugins/Es_ShaderGraph_Model.bundle/InstantiationResources/image_unlit.ss_graph ``` ### Depth Cache Project/PluginsUserPreferences/Snap_Plugin_Gui_PreviewPanel_52fc2a95-41a3-427b-a138-1e00eb1c405e.yaml ```yaml backCameraMultimedia: type: string value: PreviewResources.bundle/Video/Idle/Idle.mp4 options: 4 cameraType: type: int value: 0 options: 4 deviceType: type: string value: Unknown options: 4 fpsLogging: type: bool value: false options: 4 frameRate: type: int value: 30 options: 4 frontCameraMultimedia: type: string value: PreviewResources.bundle/Video/Idle/Idle.mp4 options: 4 fullscreenSpectaclesSimulation: type: bool value: true options: 4 lastPosition: type: vec3 value: {x: 0.000000, y: 0.000000, z: 0.000000} options: 4 lastRotation: type: vec3 value: {x: 0.000000, y: 0.000000, z: 0.000000} options: 4 mirrorHorizontally: type: bool value: false options: 4 previewSource: type: string value: InteractiveResources.bundle/Scenes/Sunlit Room options: 4 previewTarget: type: int value: 0 options: 4 previewType: type: string value: Interactive options: 4 recordInput: type: bool value: false options: 4 recordTrackingData: type: bool value: false options: 4 resolutionScale: type: float value: 1.000000 options: 4 simulatedDevice: type: string value: Spectacles (2024) options: 4 stereoEye: type: string value: Center options: 4 tintSpectaclesSimulation: type: bool value: true options: 4 uiMode: type: int value: 0 options: 4 ``` ### Depth Cache Project/Assets/SpanishTranslator.ts ```typescript /** * Translates detected objects to Spanish * Works with the existing GeminiAPI and Labels system */ @component export class SpanishTranslator extends BaseScriptComponent { // Spanish translations private translations: { [key: string]: string } = { // Common objects "hat": "el sombrero", "bottle": "la botella", "water bottle": "la botella de agua", "phone": "el teléfono", "cup": "la taza", "mug": "la taza", "book": "el libro", "door": "la puerta", "window": "la ventana", "table": "la mesa", "chair": "la silla", "bed": "la cama", "lamp": "la lámpara", "light": "la luz", "computer": "la computadora", "laptop": "la computadora portátil", "keyboard": "el teclado", "mouse": "el ratón", "glasses": "las gafas", "pen": "el bolígrafo", "notebook": "el cuaderno", "backpack": "la mochila", "bag": "la bolsa", "shoes": "los zapatos", "shirt": "la camisa", "pants": "los pantalones", "jacket": "la chaqueta", "watch": "el reloj", "clock": "el reloj", "mirror": "el espejo", "picture": "el cuadro", "plant": "la planta", "pillow": "la almohada", "blanket": "la manta", "remote": "el control remoto", "remote control": "el control remoto", "tv": "el televisor", "television": "el televisor", "couch": "el sofá", "sofa": "el sofá", "refrigerator": "el refrigerador", "fridge": "el refrigerador", "microwave": "el microondas", "oven": "el horno", "stove": "la estufa", "sink": "el fregadero", "toilet": "el inodoro", "shower": "la ducha", "towel": "la toalla", "soap": "el jabón", "toothbrush": "el cepillo de dientes", "brush": "el cepillo", "comb": "el peine", "scissors": "las tijeras", "knife": "el cuchillo", "fork": "el tenedor", "spoon": "la cuchara", "plate": "el plato", "bowl": "el tazón", "glass": "el vaso", "napkin": "la servilleta" }; onAwake() { print("✅ SpanishTranslator: Ready with " + Object.keys(this.translations).length + " translations"); } /** * Translate English object name to Spanish */ public translate(englishWord: string): string { const normalized = englishWord.toLowerCase().trim(); // Direct match if (this.translations[normalized]) { return this.translations[normalized]; } // Try to find partial match for (const key in this.translations) { if (normalized.includes(key) || key.includes(normalized)) { return this.translations[key]; } } // No translation found, return original with note return englishWord + " (?)"; } /** * Translate an array of labels */ public translateLabels(labels: any[]): any[] { const translated = []; for (const label of labels) { const spanishLabel = { ...label, label: this.translate(label.label), originalLabel: label.label // Keep original for reference }; translated.push(spanishLabel); } return translated; } /** * Get all available translations */ public getTranslations(): { [key: string]: string } { return this.translations; } } ``` [1685 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]