# Project export: MemARy This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: Cal Hacks 12.0 - Tagline: MemARy: AR Glasses for Alzheimer's. See, hear, remember. - Devpost: https://devpost.com/software/memary - GitHub: https://github.com/AmaanBilwar/memARy - Video: https://www.youtube.com/embed/1LMXyNN5t1U?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Snap: Best Use of Snap Spectacles) - Team: 3 GitHub contributor(s) — arkankau (21 commits), Amaan Bilwar (1 commits), Vishesh (1 commits) ## Devpost submission (written by the team) ### Inspiration Over 7 million Americans over 65 live with Alzheimer’s today, a number set to double by 2060. 1 in 9 older adults already face fading memories, and by 85, that risk climbs past 1 in 3. But memory loss isn’t inevitable, it’s a gap in design. We’re building MemARy, an AI wearable that sees what you see, hears what you say, and remembers what matters. Since most old people already wear glasses, why not wear it like Tony Stark. And for every Tony Stark who never forgets, MemARy will be the Jarvis for billions. ### What it does Memary is an AI-powered wearable interface that transforms visual and auditory inputs into structured, retrievable memory. Every few seconds or for every trigger (button press or saying the word, the device captures a frame from the user’s perspective, processes it through Reka's vision language model to extract semantic information: objects, spatial relations, colors, and contextual cues, and encodes that data into vector embeddings stored in a ChromaDB-based memory system. Users can interact with Memary through natural language, either by voice or text, to recall or store information, such as where they left important items. The system retrieves semantically relevant entries using vector similarity search, merges them with timestamped metadata, and returns precise and context-aware responses. Memary’s pipeline combines computer vision, LLM-based scene summarization, semantic embedding, and memory indexing, creating a continuous cognitive layer that enables human-like recall through AI. ### How we built it We engineered Memary as a modular, microservice-based system combining real-time perception, semantic understanding, and long-term memory storage. Using SnapAR and Lens Studio, we integrated our AI pipeline directly with Snap Spectacles, enabling the glasses to capture image frames, visualize UI for memory recall, and stream them to our backend. Each frame is processed by Reka’s Llama Vision Model, which generates a structured natural language summary that identifies objects, positions, specific characteristics, and contextual cues. The summaries are then embedded and stored in ChromaDB, our vector database layer, which enables high-speed semantic retrieval using cosine similarity. A FastAPI service mediates all data ingestion and querying, ensuring clean abstraction and multi-tenant control. On the client side, a React frontend provides the user interface for memory review and session selection along with memory dashboards, while PokMmCP powers semantic queries and synchronization between the phone and glasses. We also integrated LiveKit for low-latency speech-to-text processing, allowing users to add or recall memories through natural voice interaction. Together, this stack forms a continuous AI memory loop that enables real-time recall, just like having a grandmaster-level memory.. ### Challenges we ran into Complex idea in a short timespan, the documentation of SnapAR being quite the challenge but rewarding at the end, rigorous execution websocket approach to video and audio input into the Spectacle device ### Accomplishments we're proud of Our idea, being broken down into smaller work functions, that when combined together as a group of microservices, produces an efficient and synergized product. ### What we learned Twinkling with a whole new development and product architecture in Snap Spectacles, the use of fabulous technologies by the sponsors that leveled up our production quality, and most importantly, thoughtful discussions makes complex missions possible. ### What's next for memARy Make it adaptable in any contextual use and eventually go beyond the market of old people with Alzheimer to everyone that does not want to forget anything, ever. ## README (from the GitHub repository) # 🧠 Remembar - AI Memory System **Remember anything. Search everything. Ask naturally.** Remembar is an AI-powered memory system that lets you store observations as natural language and search them conversationally. With Model Context Protocol (MCP) support, you can integrate Remembar directly with Claude Desktop and other AI assistants! ## ✨ Features ### 🎯 Core Capabilities - **Natural Language Storage**: Just describe what you see - "there's a yellow key on the table" - **Conversational Search**: Ask naturally - "where are my keys?" or "what color is the car?" - **AI-Powered Understanding**: Uses Reka AI to extract structured data from descriptions - **Cloud-Native Storage**: ChromaDB Cloud integration for true persistence across deployments - **Vector Search**: Semantic search powered by ChromaDB for intelligent matching ### 🔌 MCP Integration (NEW!) - **Direct AI Assistant Access**: Use with Claude Desktop, ChatGPT, or any MCP client - **Voice-to-Memory**: Speak to Claude, have it remember for you - **Seamless Queries**: Ask Claude about your memories conversationally - **No Extra Interface Needed**: Your AI assistant becomes your memory interface ### 📊 Advanced Features - **📅 Timeline View**: See memories grouped by date (Today, Yesterday, Last Week) - **📌 Item Tracking**: Track important items with customizable alerts - **🔗 Object Relationships**: Automatically learns which items are seen together - **📈 Statistics Dashboard**: Analytics on your memories and patterns - **🎴 Flashcards**: Memory reinforcement with intelligent Q&A generation - **☁️ Cloud Persistence**: ChromaDB Cloud integration for data that survives any deployment - **💾 Local Fallback**: Automatic fallback to local storage if cloud is unavailable ## 🚀 Quick Start ### Option 1: Cloud Deployment (Share with Others) 🌐 Deploy to Railway and let others access your Remembar instance: ```bash railway login --browserless railway init railway variables set REKA_API_KEY=your_key railway up railway domain # Get your public URL ``` **Full guide**: **[DEPLOY_QUICK_START.md](./DEPLOY_QUICK_START.md)** | **[RAILWAY_DEPLOY.md](./RAILWAY_DEPLOY.md)** ### Option 2: MCP Integration (Local) Use Remembar directly through Claude Desktop or other AI assistants: ```bash # Start Remembar with MCP support ./start_with_mcp.sh ``` Then follow the setup guide: **[MCP_SETUP.md](./MCP_SETUP.md)** ### Option 3: Web Interface Use the browser-based interface: ```bash # Start services ./start_services.sh # Open web interface open web-test/index.html ``` ### Option 4: REST API Direct API access: ```bash # Start API server cd api-service python3 main.py # Store a memory curl -X POST http://localhost:8000/store_text \ -H "Content-Type: application/json" \ -d '{"text": "yellow key on the table"}' # Search memories curl "http://localhost:8000/search?query=where+is+the+key" ``` ## 📋 Prerequisites - **Python 3.9+** - **Reka AI API Key** (get one at [reka.ai](https://reka.ai)) - **pip** or **pip3** ## 🛠️ Installation 1. **Clone the repository** ```bash git clone cd remembar ``` 2. **Set up environment** ```bash # Create .env file in api-service/ echo "REKA_API_KEY=your_key_here" > api-service/.env ``` 3. **Install dependencies** ```bash cd api-service pip3 install -r requirements.txt ``` 4. **Start services** ```bash ./start_with_mcp.sh ``` ## 📖 Documentation ### Setup & Deployment - **[Deploy Quick Start](./DEPLOY_QUICK_START.md)** - ⚡ Fast Railway deployment - **[Railway Deploy Guide](./RAILWAY_DEPLOY.md)** - Complete Railway deployment - **[ChromaDB Cloud Setup](./CHROMADB_CLOUD_SETUP.md)** - ☁️ Cloud storage configuration - **[MCP Setup Guide](./MCP_SETUP.md)** - Connect to Claude Desktop - **[Startup Guide](./STARTUP_GUIDE.md)** - Local startup instructions ### Technical Guides - **[Architecture](./ARCHITECTURE.md)** - System design and components - **[API Reference](./API_QUICK_REFERENCE.md)** - REST API endpoints - **[Pipeline Update](./PIPELINE_UPDATE.md)** - Text-to-JSON pipeline details ### Feature Guides - **[Timeline Feature](./TIMELINE_FEATURE.md)** - Timeline view documentation - **[Statistics & Flashcards](./STATISTICS_AND_FLASHCARDS.md)** - Analytics features ## 🎮 Usage Examples ### With Claude Desktop (MCP) Once configured, just talk to Claude naturally: ``` You: I saw a yellow key on the table in the kitchen Claude: ✓ I've stored that memory! You: What color is the key? Claude: The key was yellow. I saw it on the table in the kitchen, just now. You: Track my medication and alert me if I haven't seen it in 12 hours Claude: ✓ Now tracking 'medication'. I'll alert if not seen for 12 hours. ``` ### With Web Interface 1. Open `web-test/index.html` 2. Go to "📝 Text to JSON" tab 3. Enter: "there's a yellow key on the table" 4. Click "Convert to JSON" 5. Switch to "🔍 Search Memories" tab 6. Ask: "what color is the key?" ### With REST API ```python import requests # Store memory response = requests.post('http://localhost:8000/store_text', json={ 'text': 'yellow key on the table', 'session_id': 'my-session' }) # Search response = requests.get('http://localhost:8000/search', params={ 'query': 'where is the key' }) print(response.json()['answer']) ``` ## 🏗️ Architecture ``` ┌─────────────────┐ │ Claude Desktop │ ← MCP Client (NEW!) └────────┬────────┘ │ MCP Protocol ↓ ┌─────────────────┐ │ mcp_server.py │ ← MCP Server └────────┬────────┘ │ HTTP REST ↓ ┌─────────────────┐ │ main.py API │ ← FastAPI Service └────────┬────────┘ │ ├─→ Reka AI (Image/Text → JSON) │ └─→ ChromaDB (Vector Store) ``` ## 🔧 Technology Stack - **Backend**: FastAPI (Python) - **AI Processing**: Reka AI (Vision + Text Models) - **Vector Store**: ChromaDB Cloud + Local Fallback - **MCP Server**: Model Context Protocol SDK - **Frontend**: HTML/CSS/JavaScript (Vanilla) ## 📁 Project Structure ``` remembar/ ├── api-service/ # Main API server │ ├── main.py # FastAPI endpoints │ ├── mcp_server.py # MCP server (NEW!) │ └── requirements.txt ├── vector-store/ # ChromaDB vector storage │ └── app.py ├── vision-processor/ # Image processing (optional) │ └── vision_reka.py ├── web-test/ # Web interface │ └── index.html ├── start_with_mcp.sh # Startup with MCP ├── start_services.sh # Startup without MCP └── MCP_SETUP.md # MCP configuration guide ``` ## 🧪 Testing ### Test MCP Server ```bash python3 test_mcp.py ``` ### Test API ```bash cd api-service python3 test.sh ``` ### Test Integration ```bash cd example-client-service python3 test_integration.py ``` ## 🤝 Contributing Contributions welcome! Areas of interest: - Additional MCP client integrations - Mobile app development - Additional AI model support - Performance optimizations - Documentation improvements ## 📝 License [Your License Here] ## 🆘 Support - **Issues**: [GitHub Issues](your-repo-url/issues) - **Discussions**: [GitHub Discussions](your-repo-url/discussions) - **Email**: [Your Email] ## 🎯 Roadmap - [x] Basic memory storage and search - [x] Timeline view - [x] Item tracking with alerts - [x] Object relationships - [x] Statistics dashboard - [x] MCP integration - [ ] Mobile app - [ ] Multi-user support - [ ] Cloud deployment - [ ] Additional AI model support - [ ] Offline mode ## ⭐ Star History If you find Remembar useful, please consider giving it a star on GitHub! --- **Made with 🧠 by [Your Name]** *Remember everything. Search naturally. Live smarter.* ## Detected evidence (automated analysis) Indexed codebase: 66 recognized source files, 367 KB. - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - TypeScript (language) — detected in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - React (technology) — claimed on Devpost, not found in the code - Vercel (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (120 of 153) ``` .claude/settings.local.json .gitignore .railwayignore API_QUICK_REFERENCE.md api-service/.gitignore api-service/HEROKU_DEPLOY.md api-service/main.py api-service/mcp_server.py api-service/Procfile api-service/PRODUCTION.txt api-service/README.txt api-service/requirements.txt api-service/run_background.sh api-service/run_production.sh api-service/runtime.txt api-service/stop.sh api-service/systemd-service.example api-service/test.sh CHROMADB_CLOUD_SETUP.md DEPLOY_QUICK_START.md DEPLOYMENT_READY.md example-client-service/query_service.py example-client-service/README.md example-client-service/test_integration.py frontend/.react-router/types/+future.ts frontend/.react-router/types/+routes.ts frontend/.react-router/types/+server-build.d.ts frontend/.react-router/types/app/+types/root.ts frontend/.react-router/types/app/routes/+types/home.ts frontend/.vite/deps/_metadata.json frontend/.vite/deps/package.json IMAGE_PIPELINE_GUIDE.md IMAGE_PIPELINE_README.md mcp_config_claude.json MCP_SETUP.md memARyLENS/.claude/settings.local.json memARyLENS/.gitattributes memARyLENS/.gitignore memARyLENS/Assets/Device Camera Texture.deviceCameraTexture memARyLENS/Assets/Device Camera Texture.deviceCameraTexture.meta memARyLENS/Assets/Echopark.hdr memARyLENS/Assets/Echopark.hdr.meta memARyLENS/Assets/Image 2.mat memARyLENS/Assets/Image 2.mat.meta memARyLENS/Assets/image_unlit.ss_graph memARyLENS/Assets/image_unlit.ss_graph.meta memARyLENS/Assets/Image.mat memARyLENS/Assets/Image.mat.meta memARyLENS/Assets/ImageMaterial.mat memARyLENS/Assets/ImageMaterial.mat.meta memARyLENS/Assets/Images/MicIcon.mat memARyLENS/Assets/Images/MicIcon.mat.meta memARyLENS/Assets/Images/Microphone.png.meta memARyLENS/Assets/Microphone.png.meta memARyLENS/Assets/Render Target.renderTarget memARyLENS/Assets/Render Target.renderTarget.meta memARyLENS/Assets/Scene.scene memARyLENS/Assets/Scene.scene.meta memARyLENS/Assets/Scripts/MicScript.js memARyLENS/Assets/Scripts/MicScript.js.meta memARyLENS/Assets/Scripts/MicScript.ts memARyLENS/Assets/Scripts/MicScript.ts.meta memARyLENS/Assets/Scripts/SpeechRecognition.ts memARyLENS/Assets/Scripts/SpeechRecognition.ts.meta memARyLENS/Assets/Scripts/TextGeneration.ts memARyLENS/Assets/Scripts/TextGeneration.ts.meta memARyLENS/Assets/text_3d.ss_graph memARyLENS/Assets/text_3d.ss_graph.meta memARyLENS/Assets/Text3D 2.mat memARyLENS/Assets/Text3D 2.mat.meta memARyLENS/Assets/Text3D.mat memARyLENS/Assets/Text3D.mat.meta memARyLENS/Assets/Untitled JavaScript 2.js memARyLENS/Assets/Untitled JavaScript 2.js.meta memARyLENS/Assets/Untitled JavaScript.js memARyLENS/Assets/Untitled JavaScript.js.meta memARyLENS/memARyLENS.esproj nixpacks.toml PIPELINE_UPDATE.md RAILWAY_DEPLOY.md railway.json railway.toml README_STARTUP.txt README.md SETUP_ENV.md start_image_pipeline.sh start_services.sh start_with_mcp.sh STARTUP_GUIDE.md STATISTICS_AND_FLASHCARDS.md stop_services.sh test_deploy_config.sh test_full_integration.py test_image_upload.sh test_mcp.py TIMELINE_FEATURE.md vector-store/.chroma/chroma.sqlite3 vector-store/.env.example vector-store/.gitignore vector-store/app.py vector-store/chroma_client.py vector-store/embeddings.py vector-store/QUICKSTART.md vector-store/README.md vector-store/requirements.txt vector-store/schema.py vector-store/test_api.py vector-store/test_chroma_cloud.py vision-processor/.env.example vision-processor/.gitignore vision-processor/camera_stream.py vision-processor/capture_scheduler.py vision-processor/INSTALL_OPENCV.md vision-processor/integration_reka.py vision-processor/LIVE_CAMERA_UPDATE.md vision-processor/README.md vision-processor/requirements.txt vision-processor/TEST_RESULTS.md vision-processor/test_search.py vision-processor/vision_reka.py [33 more files omitted for size] ``` ### Dependencies - api-service/requirements.txt: fastapi[standard], httpx, mcp@>=1.19.0, pillow@>=11.0.0, python-dotenv@>=1.0.0, reka-api@>=3.2.0, requests@>=2.31.0, uvicorn[standard] - vector-store/requirements.txt: chromadb@>=0.5.5, colorama@==0.4.*, fastapi@==0.115.*, pydantic@==2.*, python-dotenv@==1.*, sentence-transformers@==3.*, uvicorn[standard]@==0.30.* - vision-processor/requirements.txt: opencv-python@>=4.8.0, pillow@>=11.0.0, python-dotenv@>=1.0.0, reka-api@>=3.2.0, requests@>=2.32.0 ### Recent commits (newest first) - Merge branch 'snapcode' - Update Claude settings - Combine - reka picture - question query - item key - chromadb cloud - test delete - testing mcp - wallet test - tracking works, chromadb backbone done - flashcards and dashboard - day month - really working - image complete but change again - detection fixed - fastapi deploy - Remove unnecessary docs and API key references - fast api deployed - Add Reka vision processor: Image → Keywords + Vectors ## Key source files (fetched from GitHub, selected and truncated for size) ### SETUP_ENV.md ```markdown # 🔑 Setting Up Your API Key ## Quick Setup (30 seconds) ### Option 1: Create .env file (Recommended) ```bash cd /Users/arkanfadhilkautsar/Downloads/memary # Create .env file with your Reka API key echo "REKA_API_KEY=your_actual_reka_key_here" > vision-processor/.env ``` ### Option 2: Export environment variable (Temporary) ```bash export REKA_API_KEY=your_actual_reka_key_here ``` ## Where to Get Your Reka API Key 1. Go to: https://www.reka.ai/ 2. Sign up/Login 3. Navigate to API Keys section 4. Copy your API key ## After Setting Up Restart the API service: ```bash # Kill the current service lsof -ti:8000 | xargs kill -9 # Start with environment loaded cd /Users/arkanfadhilkautsar/Downloads/memary/api-service export REKA_API_KEY=your_key_here # or source ../.env python3 main.py > /tmp/api-service.log 2>&1 & ``` Or use the startup script which will auto-load the .env: ```bash cd /Users/arkanfadhilkautsar/Downloads/memary ./start_image_pipeline.sh ``` ## Test It Works ```bash # Should show your key (first few characters) echo $REKA_API_KEY | cut -c1-10 # Test the endpoint curl -X POST http://localhost:8000/store_text \ -H "Content-Type: application/json" \ -d '{"text_summary": "test", "session_id": "test"}' ``` --- **Once you have the API key set up, the image upload will work perfectly!** ``` ### DEPLOY_QUICK_START.md ```markdown # ⚡ Quick Deploy Guide ## For You (The Host) Deploy Remembar to make it accessible to others: ```bash # 1. Login railway login --browserless # → Open the URL shown, authorize, return to terminal # 2. Initialize cd /Users/arkanfadhilkautsar/Downloads/remembar railway init # 3. Set your API keys railway variables set REKA_API_KEY=your_reka_key_here # 4. Configure ChromaDB Cloud (recommended for persistence) railway variables set USE_CHROMA_CLOUD=true railway variables set CHROMA_API_KEY=ck-7QBdriXEhMjhkLgbr5DBT8vPx1pgUQbfM96rQ8pe3sr3 railway variables set CHROMA_TENANT=053f90af-f7f0-48dd-bd77-1f67ee514158 railway variables set CHROMA_DATABASE=remembar # 5. Deploy railway up # 6. Get your URL railway domain # → You'll get: https://remembar-production-xyz.railway.app ``` **Done!** Share the URL with your friend. --- ## For Your Friend (The User) Use the deployed Remembar via MCP: ### Setup (One-Time) ```bash # 1. Install MCP pip3 install mcp # 2. Download the MCP server file # Get mcp_server.py from your friend's GitHub repo ``` ### Configure Claude Desktop Edit: `~/Library/Application Support/Claude/claude_desktop_config.json` ```json { "mcpServers": { "remembar": { "command": "python3", "args": [ "/full/path/to/mcp_server.py" ], "env": { "REMEMBAR_API_URL": "https://remembar-production-xyz.railway.app" } } } } ``` **Replace:** - `/full/path/to/mcp_server.py` → actual path where you saved the file - `https://remembar-production-xyz.railway.app` → the URL your friend gave you ### Restart Claude Desktop That's it! Now talk to Claude: - "I saw a yellow key on the table" - "Where did I leave my keys?" - "Show me my memories" --- ## What Gets Shared? ✅ **Your Friend Can:** - Use the same memory database as you - Add memories through Claude - Search all memories (theirs + yours) - Use all Remembar features ❌ **Your Friend Doesn't Need:** - To run their own API server - The full Remembar codebase - Their own Reka API key - To deploy anything --- ## Cost - **Railway Free Tier**: $5 credit/month (~500 hours) - **After free credit**: ~$0.01/hour - **For hobby use**: Usually stays within free tier --- ## Troubleshooting ### "Could not connect to Remembar API" - Check the URL in Claude config matches your deployed URL - Make sure your Railway app is running: `railway status` ### Friend can't use it 1. Verify they installed MCP: `pip3 install mcp` 2. Check their Claude config has the correct URL 3. Make sure they restarted Claude Desktop --- **That's it! You're live! 🚀** ``` ### vision-processor/requirements.txt ``` reka-api>=3.2.0 pillow>=11.0.0 requests>=2.32.0 python-dotenv>=1.0.0 opencv-python>=4.8.0 ``` ### api-service/requirements.txt ``` fastapi[standard] httpx uvicorn[standard] requests>=2.31.0 python-dotenv>=1.0.0 reka-api>=3.2.0 pillow>=11.0.0 mcp>=1.19.0 ``` ### vector-store/requirements.txt ``` chromadb>=0.5.5 fastapi==0.115.* uvicorn[standard]==0.30.* pydantic==2.* sentence-transformers==3.* python-dotenv==1.* colorama==0.4.* ``` ### frontend/.vite/deps/package.json ``` { "type": "module" } ``` ### vector-store/app.py ```python """ FastAPI service implementing 5-tier ChromaDB memory architecture with devil's-advocate safeguards for AR glasses workflow. """ import os import time import hashlib import json from dotenv import load_dotenv from fastapi import FastAPI, HTTPException from contextlib import asynccontextmanager from chroma_client import ( get_or_create_collection, init_collections, COLL_FRAMES, COLL_ENTITIES, COLL_LATEST, COLL_NOTES, COLL_LTM ) from embeddings import embed_texts, EMB_MODEL_VER, SCENE_MODEL_VER, DET_MODEL_VER from schema import ( FrameIngestRequest, NoteIngestRequest, SearchLastSeenRequest, SearchSemanticRequest, SearchTimeWindowRequest, CurateToLTMRequest, CompactRequest, CONFIDENCE_THRESHOLD, TRACKED_OBJECTS ) load_dotenv() @asynccontextmanager async def lifespan(app: FastAPI): """Initialize collections on startup""" init_collections() yield app = FastAPI( title="AR Glasses Vector Memory Store", version="1.0.0", description="5-tier ChromaDB architecture with safeguards", lifespan=lifespan ) # --- Helper Functions --- def _id_frame(tenant: str, session: str, ts: int) -> str: """Time-sortable frame ID""" return f"{tenant}:{session}:{ts}" def _id_entity(tenant: str, session: str, ts: int, label: str, idx: int) -> str: """Entity ID - using item name as primary key for easy querying""" # Format: tenant:label:session:timestamp # This allows querying by label easily return f"{tenant}:{label}:{session}:{ts}" def _id_latest(tenant: str, canonical_key: str) -> str: """Latest entity ID - using item name as key""" return f"{tenant}:{canonical_key}" def _id_note(tenant: str, note_id: str) -> str: """Note ID""" return f"{tenant}:note:{note_id}" def _id_ltm(tenant: str, origin_id: str) -> str: """LTM ID based on hash of origin""" hash_suffix = hashlib.md5(origin_id.encode()).hexdigest()[:8] return f"{tenant}:ltm:{hash_suffix}" def _assign_role(person_idx: int, hint: str = None) -> str: """Privacy safeguard: no biometric data, only ephemeral roles""" base = f"person_{person_idx}" return f"{base} ({hint})" if hint else base def _clean_metadata(meta: dict) -> dict: """ Clean metadata for ChromaDB - only str, int, float, bool allowed. Convert lists to JSON strings, remove None values. """ cleaned = {} for k, v in meta.items(): if v is None: continue # Skip None values elif isinstance(v, (list, dict)): cleaned[k] = json.dumps(v) # Serialize complex types elif isinstance(v, (str, int, float, bool)): cleaned[k] = v else: cleaned[k] = str(v) # Convert other types to string return cleaned # --- API Endpoints --- @app.post("/add_frame") async def add_frame(req: FrameIngestRequest): """ Ingest a frame captured every ~5 seconds. Stores in 3 places: 1. frames_ephemeral_v1 - full scene summary (TTL: 24-72h) 2. entities_stream_v1 - per-object mentions 3. latest_entities_v1 - upsert for tracked objects (if confidence ≥ threshold) Safeguards: - Only upsert to latest if object in TRACKED_OBJECTS and confidence ≥ CONFIDENCE_THRESHOLD - Record model versions for drift tracking - Privacy: no biometric data for people """ try: # --- 1) Add to frames_ephemeral_v1 --- frames = get_or_create_collection(COLL_FRAMES) frame_id = _id_frame(req.tenant_id, req.session_id, req.frame_ts) scene_text = req.scene_summary.strip() scene_vec = embed_texts([scene_text])[0] # Build objects metadata objs_meta = [] has_keys = False has_pills = False person_count = 0 for obj in req.objects: entry = { "label": obj.label, "confidence": obj.confidence, "bbox": obj.bbox, "color": obj.color, "rel_pos": obj.rel_pos, } if obj.is_person: person_count += 1 entry["role"] = _assign_role(person_count, obj.relationship_hint) if obj.label == "keys": has_keys = True if obj.label in {"pill_bottle", "pills", "medication"}: has_pills = True objs_meta.append(entry) frame_meta = { "tenant_id": req.tenant_id, "device_id": req.device_id, "session_id": req.session_id, "frame_ts": req.frame_ts, "tz": req.tz, "lat_lon_hash": req.lat_lon_hash, "objects": objs_meta, "has_keys": has_keys, "has_pills": has_pills, "model_scene_ver": SCENE_MODEL_VER, "emb_model_ver": EMB_MODEL_VER, } frames.add( ids=[frame_id], documents=[scene_text], embeddings=[scene_vec], metadatas=[_clean_metadata(frame_meta)] ) # --- 2) Add to entities_stream_v1 --- entities = get_or_create_collection(COLL_ENTITIES) ent_ids, ent_docs, ent_vecs, ent_metas = [], [], [], [] for idx, obj in enumerate(req.objects): ent_id = _id_entity(req.tenant_id, req.session_id, req.frame_ts, obj.label, idx) # Canonical object string parts = [obj.label] if obj.color: parts.append(obj.color) if obj.rel_pos: parts.append(obj.rel_pos) doc = " | ".join(parts) ent_ids.append(ent_id) ent_docs.append(doc) meta = { "tenant_id": req.tenant_id, "device_id": req.device_id, "session_id": req.session_id, "frame_ts": req.frame_ts, "obj_label": obj.label, "obj_attrs": {"color": [truncated — 12764 more characters] ``` ### api-service/main.py ```python """ Simple FastAPI service to store and fetch from vector database """ from typing import Union, Optional from fastapi import FastAPI, HTTPException from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel import httpx import time import os import base64 import tempfile from dotenv import load_dotenv # Load environment variables load_dotenv() app = FastAPI() # CORS - Allow requests from any domain app.add_middleware( CORSMiddleware, allow_origins=["*"], # In production, specify your domains allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) # Vector store URL - use env var for Heroku VECTOR_STORE_URL = os.getenv("VECTOR_STORE_URL", "http://localhost:8001") # Async HTTP client client = httpx.AsyncClient(timeout=30.0) # In-memory storage (fallback when vector store is down) memory_store = [] # Tracked items storage tracked_items = {} # {item_name: {alert_hours: 24, last_seen: timestamp, notes: ""}} # Object relationships storage object_relationships = {} # {obj1: {obj2: count, obj3: count}} # Request models class ImageUpload(BaseModel): image_base64: str session_id: Optional[str] = "default-session" class StoreMemory(BaseModel): scene: str objects: list session_id: Optional[str] = "default-session" class TextToJSON(BaseModel): text_summary: str session_id: Optional[str] = "default-session" class TrackItem(BaseModel): item_name: str alert_hours: Optional[int] = 24 notes: Optional[str] = "" @app.get("/") def read_root(): return {"service": "memory-api", "status": "running"} @app.get("/debug/memory") def get_memory_store(): """View in-memory storage (for debugging)""" return { "total_items": len(memory_store), "items": memory_store } @app.get("/memories") def get_all_memories(session_id: Optional[str] = None, limit: int = 100): """ Get all memories with entity IDs for each object. Now shows item-based IDs: tenant:ITEM_NAME:session:timestamp """ # Filter by session if specified if session_id: filtered = [m for m in memory_store if m.get("session_id") == session_id] else: filtered = memory_store # Sort by timestamp (newest first) sorted_memories = sorted(filtered, key=lambda x: x.get("timestamp", 0), reverse=True) # Limit results limited = sorted_memories[:limit] # Add entity IDs to each object enhanced_memories = [] for memory in limited: memory_copy = memory.copy() # Generate entity IDs for each object (item-name based format) if "objects" in memory_copy: enhanced_objects = [] for obj in memory_copy["objects"]: obj_copy = obj.copy() item_name = obj.get("label", "unknown") session = memory_copy.get("session_id", "default") timestamp = memory_copy.get("timestamp", int(time.time())) # New format: tenant:ITEM_NAME:session:timestamp obj_copy["entity_id"] = f"user_123:{item_name}:{session}:{timestamp}" enhanced_objects.append(obj_copy) memory_copy["objects"] = enhanced_objects # Add frame ID for reference memory_copy["frame_id"] = f"user_123:{memory_copy.get('session_id', 'default')}:{memory_copy.get('timestamp', 0)}" enhanced_memories.append(memory_copy) return { "ok": True, "total": len(enhanced_memories), "memories": enhanced_memories } @app.post("/clear_storage") async def clear_storage(): """Clear all stored memories (in-memory and vector store)""" try: # Clear in-memory storage initial_count = len(memory_store) memory_store.clear() # Try to clear vector store vector_cleared = False try: # Clear all collections in vector store collections = ["frames_ephemeral_v1", "entities_stream_v1", "latest_entities_v1"] for collection in collections: response = await client.post( f"{VECTOR_STORE_URL}/clear_collection", json={"collection_name": collection}, timeout=5.0 ) vector_cleared = True except Exception as e: print(f"[WARNING] Could not clear vector store: {e}") return { "ok": True, "message": "Storage cleared successfully", "cleared_memory_items": initial_count, "vector_store_cleared": vector_cleared } except Exception as e: raise HTTPException(status_code=500, detail=str(e)) @app.post("/store") async def store_memory(data: ImageUpload): """Upload and process an image, then store the results""" try: # Decode base64 image image_bytes = base64.b64decode(data.image_base64) # Save to temp file with tempfile.NamedTemporaryFile(delete=False, suffix='.jpg') as tmp_file: tmp_file.write(image_bytes) tmp_path = tmp_file.name try: # Import vision processor (make sure vision-processor is in path) import sys vision_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'vision-processor')) if vision_path not in sys.path: sys.path.insert(0, vision_path) print(f"[DEBUG] Importing vision_reka from: {vision_path}") print(f"[DEBUG] sys.path: {sys.path[:3]}") from vision_reka import analyze_image # Analyze image print(f"[DEBUG] Analyzing image: {tmp_path}") analysis_result = analyze_image(tmp_path) print(f"[DEBUG] Analysis result: {analysis_result}") # Check if result is dict with "ok" ke [truncated — 32211 more characters] ``` ### test_full_integration.py ```python ``` ### stop_services.sh ```shell #!/bin/bash # Stop all Remembar services echo "🛑 Stopping Remembar Services..." echo "" # Kill processes on ports echo "Stopping API Service (port 8000)..." lsof -ti:8000 | xargs kill -9 2>/dev/null && echo " ✓ API Service stopped" || echo " ℹ️ No process on port 8000" echo "Stopping Vector Store (port 8001)..." lsof -ti:8001 | xargs kill -9 2>/dev/null && echo " ✓ Vector Store stopped" || echo " ℹ️ No process on port 8001" echo "" echo "✅ All services stopped" ``` [52 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]