# Project export: Memory Anchors
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2026
- Tagline: Turning everyday objects into memory anchors with spatial AI.
- Devpost: https://devpost.com/software/memory-anchors
- GitHub: https://github.com/vaishvijs13/memory-anchors
- Team: 4 GitHub contributor(s) — Vaishvi Shah (8 commits), parivansjalia (3 commits), Aditya Makkar (1 commits), haley (1 commits)
## Devpost submission (written by the team)
### Inspiration
Alzheimer’s disease and cognitive decline affect not only memory recall but also how people interpret their physical environment. Objects that once carried personal meaning can lose context, increasing confusion and anxiety during everyday interactions. Research in reminiscence therapy shows that memory retrieval is strongly tied to physical context and familiar objects. Instead of relying on abstract recall, we explored whether computer vision and contextual interfaces could allow the environment itself to reinforce memory through gentle, situational cues. Memory Anchors was designed to transform everyday surroundings into cognitive support interfaces by associating personal memories with recognizable objects in real time. What we built Memory Anchors is a spatial memory interface that overlays personal memories onto real-world objects detected through a live camera feed. The system uses computer vision to recognize familiar household objects and attaches contextual memory cards to them in real time. When a user points their device at an object such as a chair, book, or television, the application detects the object and renders a memory card anchored to the detection region. These memory anchors can be created in advance by caregivers or family members and may include text, narration, or additional media context. The system combines real-time object detection, contextual memory retrieval, and voice narration to create a lightweight augmented interaction model. Instead of requiring users to actively recall information, Memory Anchors reinforces familiarity through situational cues tied to the physical environment. From a systems perspective, Memory Anchors integrates perception (object detection), retrieval (memory lookup), and assistive interaction (narration) into a single loop. The environment becomes a context-aware interface that supports memory through recognition rather than recall. This prototype demonstrates how spatial computing and assistive AI can transform everyday objects into cognitive anchors that help preserve identity and reduce confusion in early-stage memory decline. How it works The frontend is implemented in React and runs entirely in the browser. TensorFlow.js with the COCO-SSD object detection model performs real-time inference on the camera stream. Detection results are rendered on a canvas overlay synchronized with the video feed. When a supported object label is detected, the application queries a memory service and displays a memory card positioned relative to the detected bounding box. Detection stabilization logic reduces flicker by requiring consistent detections across frames. A FastAPI backend provides a memory anchor service indexed by object label. The service exposes endpoints for retrieving, creating, and updating memory anchors, enabling caregivers to pre-associate memories with specific household objects. A voice narration module built using the Web Speech API provides accessible playback of memory text directly within the interface. From a systems perspective, the application combines: in-browser computer vision inference contextual UI rendering on a live camera stream a memory retrieval service assistive voice interaction This creates a lightweight augmented interface that connects physical objects to personal memory context. Challenges Running object detection in real time inside the browser required careful handling of inference frequency and rendering synchronization to avoid unstable overlays and UI flickering. We implemented detection smoothing and fallback interaction modes to maintain reliability during demonstrations. Another challenge was designing an interaction model appropriate for cognitive support. Interfaces had to remain predictable, low-distraction, and accessible while still demonstrating the technical capabilities of real-time computer vision and contextual overlays.
### What we learned
We learned that assistive AI systems benefit more from reliability and contextual clarity than from model complexity. Integrating computer vision with memory retrieval demonstrated how physical environments can become meaningful interaction surfaces when AI is applied thoughtfully. The project also highlighted how lightweight camera-based AR interfaces can be implemented entirely in the browser without specialized hardware or native frameworks.
## README (from the GitHub repository)
# Memory Anchors
**Memories attached to the real world.**
Alzheimer's support: the home becomes a memory interface. Point your camera at household objects and see personal memories overlaid in real time...
Built at tree hacks 2026!
## Detected evidence (automated analysis)
Indexed codebase: 65 recognized source files, 186 KB.
- Anthropic (technology) — detected in the code
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- Supabase (technology) — detected in the code
- TensorFlow (technology) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (72 of 72)
```
.gitignore
backend/.env.example
backend/alembic.ini
backend/alembic/env.py
backend/alembic/script.py.mako
backend/alembic/versions/354efaaba29d_initial_schema.py
backend/app/__init__.py
backend/app/ai/__init__.py
backend/app/ai/anthropic_provider.py
backend/app/ai/base.py
backend/app/ai/elevenlabs_provider.py
backend/app/ai/openai_provider.py
backend/app/ai/prompts.py
backend/app/config.py
backend/app/database.py
backend/app/dependencies.py
backend/app/main.py
backend/app/models/__init__.py
backend/app/models/base.py
backend/app/models/memory.py
backend/app/models/object.py
backend/app/models/session.py
backend/app/models/user.py
backend/app/routers/__init__.py
backend/app/routers/auth.py
backend/app/routers/legacy.py
backend/app/routers/memories.py
backend/app/routers/objects.py
backend/app/routers/toolkit.py
backend/app/routers/upload.py
backend/app/routers/vision.py
backend/app/routers/voice.py
backend/app/schemas/__init__.py
backend/app/schemas/auth.py
backend/app/schemas/memory.py
backend/app/schemas/toolkit.py
backend/app/schemas/vision.py
backend/app/schemas/voice.py
backend/app/services/__init__.py
backend/app/services/auth_service.py
backend/app/services/memory_service.py
backend/app/services/storage_service.py
backend/app/services/toolkit_service.py
backend/app/services/vision_service.py
backend/app/services/voice_service.py
backend/app/utils/__init__.py
backend/app/utils/image.py
backend/requirements.txt
backend/seed.py
docker-compose.yml
eslint.config.js
index.html
main.py
package.json
README.md
requirements.txt
src/App.css
src/App.jsx
src/components/CameraView.jsx
src/components/Controls.jsx
src/components/DetectionOverlay.jsx
src/components/MemoryCard.jsx
src/components/MemoryUpload.jsx
src/components/SystemHUD.jsx
src/components/VoiceTestPanel.jsx
src/hooks/useVoiceNarration.js
src/index.css
src/main.jsx
src/services/detection.js
src/services/memoryApi.js
src/services/voiceService.js
vite.config.js
```
### Dependencies
- backend/requirements.txt: alembic@==1.13.1, anthropic@==0.18.0, asyncpg@==0.29.0, fastapi@==0.109.0, httpx@==0.27.0, openai@==1.12.0, passlib[bcrypt]@==1.7.4, pydantic@==2.5.3, pydantic-settings@==2.1.0, pypdf@>=3.17.0, python-docx@>=1.1.0, python-jose[cryptography]@==3.3.0, python-multipart@==0.0.6, sqlalchemy[asyncio]@==2.0.25, supabase@>=2.0.0, uvicorn[standard]@==0.27.0
- package.json: @eslint/js@^9.39.1, @supabase/supabase-js@^2.95.3, @tensorflow-models/coco-ssd@^2.2.3, @tensorflow/tfjs@^4.22.0, @types/react@^19.2.7, @types/react-dom@^19.2.3, @vitejs/plugin-react@^4.7.0, eslint@^9.39.1, eslint-plugin-react-hooks@^7.0.1, eslint-plugin-react-refresh@^0.4.24, globals@^16.5.0, react@^19.2.0, react-dom@^19.2.0, react-webcam@^7.2.0, vite@^5.4.21
- requirements.txt: fastapi@==0.109.0, pydantic@==2.5.3, uvicorn[standard]@==0.27.0
### Recent commits (newest first)
- new
- add file parsing and automatic llm generated memories
- update memories
- connect to supabase db
- vector db
- working partly:
- Mention event where project was developed
- update
- lol
- working llm
- project scope
- feat: fastapi backend
- frontend + AR
## Key source files (fetched from GitHub, selected and truncated for size)
### requirements.txt
```
fastapi==0.109.0
uvicorn[standard]==0.27.0
pydantic==2.5.3
```
### docker-compose.yml
```yaml
services:
db:
image: postgres:16-alpine
environment:
POSTGRES_USER: anchors
POSTGRES_PASSWORD: anchors_dev
POSTGRES_DB: memory_anchors
ports:
- "5432:5432"
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U anchors -d memory_anchors"]
interval: 5s
timeout: 3s
retries: 5
backend:
build: ./backend
ports:
- "8000:8000"
env_file:
- ./backend/.env
depends_on:
db:
condition: service_healthy
volumes:
- ./backend:/app
command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload
volumes:
pgdata:
```
### package.json
```
{
"name": "memory-anchors",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"lint": "eslint .",
"preview": "vite preview"
},
"dependencies": {
"@supabase/supabase-js": "^2.95.3",
"@tensorflow-models/coco-ssd": "^2.2.3",
"@tensorflow/tfjs": "^4.22.0",
"react": "^19.2.0",
"react-dom": "^19.2.0",
"react-webcam": "^7.2.0"
},
"devDependencies": {
"@eslint/js": "^9.39.1",
"@types/react": "^19.2.7",
"@types/react-dom": "^19.2.3",
"@vitejs/plugin-react": "^4.7.0",
"eslint": "^9.39.1",
"eslint-plugin-react-hooks": "^7.0.1",
"eslint-plugin-react-refresh": "^0.4.24",
"globals": "^16.5.0",
"vite": "^5.4.21"
}
}
```
### backend/requirements.txt
```
fastapi==0.109.0
uvicorn[standard]==0.27.0
pydantic==2.5.3
pydantic-settings==2.1.0
python-multipart==0.0.6
sqlalchemy[asyncio]==2.0.25
asyncpg==0.29.0
alembic==1.13.1
python-jose[cryptography]==3.3.0
passlib[bcrypt]==1.7.4
openai==1.12.0
anthropic==0.18.0
httpx==0.27.0
supabase>=2.0.0
pypdf>=3.17.0
python-docx>=1.1.0
```
### main.py
```python
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from pydantic import BaseModel
from typing import Optional
app = FastAPI(title="Memory Anchors API")
# Enable CORS for local React dev
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
class MemoryAnchor(BaseModel):
object_label: str
title: str
memory_text: str
audio_url: Optional[str] = None
# In-memory store with seed data
memory_store: dict[str, MemoryAnchor] = {
"chair": MemoryAnchor(
object_label="chair",
title="Grandpa's Chair",
memory_text="Grandpa always sat in this chair after dinner. He'd read to me while I sat on the floor next to him. I remember the way his voice got all soft during the sad parts. The leather's cracked now but it still smells like his pipe tobacco if you lean in close.",
),
"book": MemoryAnchor(
object_label="book",
title="Mom's Bedtime Book",
memory_text="Mom read this to me so many times the spine fell apart. She had to hold the pages together. I used to ask her to do the rabbit's voice over and over. She never got tired of it, or at least she never let on if she did.",
),
"tv": MemoryAnchor(
object_label="tv",
title="Sunday Movies",
memory_text="We'd all cram onto the couch on Sunday nights. Dad would make popcorn with too much butter, the way we liked it. Nobody was allowed to talk during the movie but we'd all look at each other during the funny parts. I miss that.",
),
"laptop": MemoryAnchor(
object_label="laptop",
title="Video Calls with Sarah",
memory_text="When Sarah moved away, we started doing video calls every week on this laptop. Sometimes we'd just sit there not saying much. She'd show me her apartment, I'd show her the dog. It helped, knowing she was right there on the screen even though she was so far away.",
),
}
@app.get("/health")
def health_check():
return {"ok": True}
@app.get("/memory", response_model=list[MemoryAnchor])
def list_memories():
return list(memory_store.values())
@app.get("/memory/{object_label}", response_model=MemoryAnchor)
def get_memory(object_label: str):
label = object_label.lower().strip()
if label not in memory_store:
raise HTTPException(status_code=404, detail=f"No memory found for '{object_label}'")
return memory_store[label]
@app.post("/memory", response_model=MemoryAnchor)
def create_or_update_memory(anchor: MemoryAnchor):
label = anchor.object_label.lower().strip()
anchor.object_label = label
memory_store[label] = anchor
return anchor
```
### src/main.jsx
```javascript
import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client'
import './index.css'
import App from './App.jsx'
import './services/voiceService.js'
createRoot(document.getElementById('root')).render(
,
)
```
### src/App.jsx
```javascript
import { useState, useRef, useCallback, useEffect } from "react";
import CameraView from "./components/CameraView";
import DetectionOverlay from "./components/DetectionOverlay";
import MemoryCard from "./components/MemoryCard";
import MemoryUpload from "./components/MemoryUpload";
import SystemHUD from "./components/SystemHUD";
import Controls from "./components/Controls";
import { loadModel, detect, SUPPORTED_OBJECTS } from "./services/detection";
import { fetchMemory } from "./services/memoryApi";
import "./App.css";
const STABLE_THRESHOLD_MS = 700;
const DETECT_INTERVAL_MS = 300;
export default function App() {
const webcamRef = useRef(null);
const stableTimerRef = useRef(null);
const lastStableLabelRef = useRef(null);
const runningRef = useRef(false);
const [cameraReady, setCameraReady] = useState(false);
const [modelReady, setModelReady] = useState(false);
const [demoMode, setDemoMode] = useState(false);
const [selectedLabel, setSelectedLabel] = useState(SUPPORTED_OBJECTS[0]);
const [detections, setDetections] = useState([]);
const [activeMemory, setActiveMemory] = useState(null);
const [videoSize, setVideoSize] = useState({ w: 0, h: 0 });
const [showUpload, setShowUpload] = useState(false);
// Load model on mount
useEffect(() => {
let cancelled = false;
loadModel()
.then(() => {
if (!cancelled) setModelReady(true);
})
.catch(() => {
console.warn("[SYS] Model failed to load — use Demo Mode");
});
return () => { cancelled = true; };
}, []);
// Detection loop
useEffect(() => {
if (demoMode || !modelReady || !cameraReady) return;
let animId;
runningRef.current = true;
const loop = async () => {
if (!runningRef.current) return;
const video = webcamRef.current?.video;
if (video && video.readyState >= 2) {
if (video.videoWidth && video.videoHeight) {
setVideoSize({ w: video.videoWidth, h: video.videoHeight });
}
try {
const results = await detect(video);
setDetections(results);
const best = results.length > 0 ? results[0] : null;
const bestLabel = best?.label ?? null;
if (bestLabel !== lastStableLabelRef.current) {
lastStableLabelRef.current = bestLabel;
clearTimeout(stableTimerRef.current);
if (bestLabel) {
stableTimerRef.current = setTimeout(() => {
triggerMemory(bestLabel, best.bbox);
}, STABLE_THRESHOLD_MS);
}
}
} catch {
// detection error
}
}
if (runningRef.current) {
animId = setTimeout(() => {
requestAnimationFrame(loop);
}, DETECT_INTERVAL_MS);
}
};
requestAnimationFrame(loop);
return () => {
runningRef.current = false;
clearTimeout(animId);
clearTimeout(stableTimerRef.current);
};
}, [demoMode, modelReady, cameraReady]);
const triggerMemory = useCallback(async (label, bbox) => {
const memory = await fetchMemory(label);
if (!memory) return;
let position = { top: 90, left: 24 };
if (bbox) {
const [x, y] = bbox;
const video = webcamRef.current?.video;
if (video) {
const scaleX = window.innerWidth / (video.videoWidth || 1);
const scaleY = window.innerHeight / (video.videoHeight || 1);
position = {
top: Math.min(Math.max(y * scaleY, 20), window.innerHeight - 260),
left: Math.min(Math.max(x * scaleX + 16, 16), window.innerWidth - 340),
};
}
}
setActiveMemory({ label, memory, position });
}, []);
const handleDemoTrigger = useCallback(() => {
triggerMemory(selectedLabel, null);
}, [selectedLabel, triggerMemory]);
const handleToggleDemo = useCallback(() => {
setDemoMode((prev) => !prev);
setDetections([]);
setActiveMemory(null);
lastStableLabelRef.current = null;
}, []);
const handleCloseMemory = useCallback(() => {
setActiveMemory(null);
lastStableLabelRef.current = null;
}, []);
return (