# Project export: MemARy This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: MemARy: AR Glasses for Alzheimer's. See, hear, remember. - Devpost: https://devpost.com/software/memary - GitHub: https://github.com/AmaanBilwar/memARy - Video: https://www.youtube.com/embed/1LMXyNN5t1U?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Snap: Best Use of Snap Spectacles) - Team: 3 GitHub contributor(s) — arkankau (21 commits), Amaan Bilwar (1 commits), Vishesh (1 commits) ## Devpost submission (written by the team) ### Inspiration Over 7 million Americans over 65 live with Alzheimer’s today, a number set to double by 2060. 1 in 9 older adults already face fading memories, and by 85, that risk climbs past 1 in 3. But memory loss isn’t inevitable, it’s a gap in design. We’re building MemARy, an AI wearable that sees what you see, hears what you say, and remembers what matters. Since most old people already wear glasses, why not wear it like Tony Stark. And for every Tony Stark who never forgets, MemARy will be the Jarvis for billions. ### What it does Memary is an AI-powered wearable interface that transforms visual and auditory inputs into structured, retrievable memory. Every few seconds or for every trigger (button press or saying the word, the device captures a frame from the user’s perspective, processes it through Reka's vision language model to extract semantic information: objects, spatial relations, colors, and contextual cues, and encodes that data into vector embeddings stored in a ChromaDB-based memory system. Users can interact with Memary through natural language, either by voice or text, to recall or store information, such as where they left important items. The system retrieves semantically relevant entries using vector similarity search, merges them with timestamped metadata, and returns precise and context-aware responses. Memary’s pipeline combines computer vision, LLM-based scene summarization, semantic embedding, and memory indexing, creating a continuous cognitive layer that enables human-like recall through AI. ### How we built it We engineered Memary as a modular, microservice-based system combining real-time perception, semantic understanding, and long-term memory storage. Using SnapAR and Lens Studio, we integrated our AI pipeline directly with Snap Spectacles, enabling the glasses to capture image frames, visualize UI for memory recall, and stream them to our backend. Each frame is processed by Reka’s Llama Vision Model, which generates a structured natural language summary that identifies objects, positions, specific characteristics, and contextual cues. The summaries are then embedded and stored in ChromaDB, our vector database layer, which enables high-speed semantic retrieval using cosine similarity. A FastAPI service mediates all data ingestion and querying, ensuring clean abstraction and multi-tenant control. On the client side, a React frontend provides the user interface for memory review and session selection along with memory dashboards, while PokMmCP powers semantic queries and synchronization between the phone and glasses. We also integrated LiveKit for low-latency speech-to-text processing, allowing users to add or recall memories through natural voice interaction. Together, this stack forms a continuous AI memory loop that enables real-time recall, just like having a grandmaster-level memory.. ### Challenges we ran into Complex idea in a short timespan, the documentation of SnapAR being quite the challenge but rewarding at the end, rigorous execution websocket approach to video and audio input into the Spectacle device ### Accomplishments we're proud of Our idea, being broken down into smaller work functions, that when combined together as a group of microservices, produces an efficient and synergized product. ### What we learned Twinkling with a whole new development and product architecture in Snap Spectacles, the use of fabulous technologies by the sponsors that leveled up our production quality, and most importantly, thoughtful discussions makes complex missions possible. ### What's next for memARy Make it adaptable in any contextual use and eventually go beyond the market of old people with Alzheimer to everyone that does not want to forget anything, ever. ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 66 recognized source files, 367 KB. - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - React (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (120 of 153) ``` .claude/settings.local.json .gitignore .railwayignore API_QUICK_REFERENCE.md api-service/.gitignore api-service/HEROKU_DEPLOY.md api-service/main.py api-service/mcp_server.py api-service/Procfile api-service/PRODUCTION.txt api-service/README.txt api-service/requirements.txt api-service/run_background.sh api-service/run_production.sh api-service/runtime.txt api-service/stop.sh api-service/systemd-service.example api-service/test.sh CHROMADB_CLOUD_SETUP.md DEPLOY_QUICK_START.md DEPLOYMENT_READY.md example-client-service/query_service.py example-client-service/README.md example-client-service/test_integration.py frontend/.react-router/types/+future.ts frontend/.react-router/types/+routes.ts frontend/.react-router/types/+server-build.d.ts frontend/.react-router/types/app/+types/root.ts frontend/.react-router/types/app/routes/+types/home.ts frontend/.vite/deps/_metadata.json frontend/.vite/deps/package.json IMAGE_PIPELINE_GUIDE.md IMAGE_PIPELINE_README.md mcp_config_claude.json MCP_SETUP.md memARyLENS/.claude/settings.local.json memARyLENS/.gitattributes memARyLENS/.gitignore memARyLENS/Assets/Device Camera Texture.deviceCameraTexture memARyLENS/Assets/Device Camera Texture.deviceCameraTexture.meta memARyLENS/Assets/Echopark.hdr memARyLENS/Assets/Echopark.hdr.meta memARyLENS/Assets/Image 2.mat memARyLENS/Assets/Image 2.mat.meta memARyLENS/Assets/image_unlit.ss_graph memARyLENS/Assets/image_unlit.ss_graph.meta memARyLENS/Assets/Image.mat memARyLENS/Assets/Image.mat.meta memARyLENS/Assets/ImageMaterial.mat memARyLENS/Assets/ImageMaterial.mat.meta memARyLENS/Assets/Images/MicIcon.mat memARyLENS/Assets/Images/MicIcon.mat.meta memARyLENS/Assets/Images/Microphone.png.meta memARyLENS/Assets/Microphone.png.meta memARyLENS/Assets/Render Target.renderTarget memARyLENS/Assets/Render Target.renderTarget.meta memARyLENS/Assets/Scene.scene memARyLENS/Assets/Scene.scene.meta memARyLENS/Assets/Scripts/MicScript.js memARyLENS/Assets/Scripts/MicScript.js.meta memARyLENS/Assets/Scripts/MicScript.ts memARyLENS/Assets/Scripts/MicScript.ts.meta memARyLENS/Assets/Scripts/SpeechRecognition.ts memARyLENS/Assets/Scripts/SpeechRecognition.ts.meta memARyLENS/Assets/Scripts/TextGeneration.ts memARyLENS/Assets/Scripts/TextGeneration.ts.meta memARyLENS/Assets/text_3d.ss_graph memARyLENS/Assets/text_3d.ss_graph.meta memARyLENS/Assets/Text3D 2.mat memARyLENS/Assets/Text3D 2.mat.meta memARyLENS/Assets/Text3D.mat memARyLENS/Assets/Text3D.mat.meta memARyLENS/Assets/Untitled JavaScript 2.js memARyLENS/Assets/Untitled JavaScript 2.js.meta memARyLENS/Assets/Untitled JavaScript.js memARyLENS/Assets/Untitled JavaScript.js.meta memARyLENS/memARyLENS.esproj nixpacks.toml PIPELINE_UPDATE.md RAILWAY_DEPLOY.md railway.json railway.toml README_STARTUP.txt README.md SETUP_ENV.md start_image_pipeline.sh start_services.sh start_with_mcp.sh STARTUP_GUIDE.md STATISTICS_AND_FLASHCARDS.md stop_services.sh test_deploy_config.sh test_full_integration.py test_image_upload.sh test_mcp.py TIMELINE_FEATURE.md vector-store/.chroma/chroma.sqlite3 vector-store/.env.example vector-store/.gitignore vector-store/app.py vector-store/chroma_client.py vector-store/embeddings.py vector-store/QUICKSTART.md vector-store/README.md vector-store/requirements.txt vector-store/schema.py vector-store/test_api.py vector-store/test_chroma_cloud.py vision-processor/.env.example vision-processor/.gitignore vision-processor/camera_stream.py vision-processor/capture_scheduler.py vision-processor/INSTALL_OPENCV.md vision-processor/integration_reka.py vision-processor/LIVE_CAMERA_UPDATE.md vision-processor/README.md vision-processor/requirements.txt vision-processor/TEST_RESULTS.md vision-processor/test_search.py vision-processor/vision_reka.py [33 more files omitted for size] ``` ### Dependencies - api-service/requirements.txt: fastapi[standard], httpx, mcp@>=1.19.0, pillow@>=11.0.0, python-dotenv@>=1.0.0, reka-api@>=3.2.0, requests@>=2.31.0, uvicorn[standard] - vector-store/requirements.txt: chromadb@>=0.5.5, colorama@==0.4.*, fastapi@==0.115.*, pydantic@==2.*, python-dotenv@==1.*, sentence-transformers@==3.*, uvicorn[standard]@==0.30.* - vision-processor/requirements.txt: opencv-python@>=4.8.0, pillow@>=11.0.0, python-dotenv@>=1.0.0, reka-api@>=3.2.0, requests@>=2.32.0 ### Recent commits (newest first) - Merge branch 'snapcode' - Update Claude settings - Combine - reka picture - question query - item key - chromadb cloud - test delete - testing mcp - wallet test - tracking works, chromadb backbone done - flashcards and dashboard - day month - really working - image complete but change again - detection fixed - fastapi deploy - Remove unnecessary docs and API key references - fast api deployed - Add Reka vision processor: Image → Keywords + Vectors ## Key source files (fetched from GitHub, selected and truncated for size) Source contents could not be fetched from GitHub right now. [62 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]