# Project export: Rem This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: We enable you to relive your memories by taking your photos and transforming them into 3D scenes you can navigate through. - Devpost: https://devpost.com/software/rem-xk6i7d - GitHub: https://github.com/haileyl6171/rem - Video: https://www.youtube.com/embed/ZMkQMt4md5s?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 5 GitHub contributor(s) โ€” Claude Opus 4.8 (28 commits), eduardo-hdez (26 commits), Yifan Luo (19 commits), = (10 commits), Hailey Lin (10 commits) ## Devpost submission (written by the team) ### Inspiration Memories as a whole are spatial, and one of the most powerful and nostalgic ways of capturing these memories is through photographs. They are the closest humans get to reliving their experiences, but we wondered if it was possible to get one step closer. ### What it does We came up with a way to relive your memories using Gaussian Splatting to reconstruct a 3D scene from natural language description of the memory, photos of the memory, or a video of the memory. Rem enables you to move inside the 3D space, which is customized by a creative agent to feel as closely as possible to how it felt in the moment. ### How we built it ๐ŸŽจ Frontend: Next.js 16 (App Router) + React 19 + Tailwind, with a three-screen flow (ingest โ†’ loading โ†’ 3D viewer) ๐ŸŽ™๏ธ Voice input: intuitive browser-native Web Speech API for live transcription โ€” no audio leaves the device ๐Ÿง  Agent-based personalization system: Gemini + Pika MCP power a lightweight agent pipeline that transforms raw inputs into a consistent memory representation and guided reconstruction. We use two main components: Structured memory agents (Gemini sub-agents) vision: extracts structured understanding from uploaded photos analyzer: combines past memories + current input into a recurring โ€œworld summaryโ€ extractor: isolates the relevant slice of that world for the new memory persona: builds a persistent, evolving visual identity across memories These help maintain consistency across reconstructions. Creative + tool-agent layer (Pika MCP) The system also includes a creative agent that generates the cinematic direction for each memory reconstruction. It takes the scene context and persona and translates them into a coherent visual style for the output. Supporting tools include: fix_look: re-grades video (lighting, palette, mood, clothing, accessories) while preserving geometry and identity music: selects or generates audio to maintain emotional continuity ๐ŸŒŽ 3D rendering: Three.js + gsplat for real-time Gaussian Splat rendering, with a key points extracted from SfM using COLMAP โ†’ gaussian initialization โ†’ rasterization and gradient descent โ†’ gaussian densification and pruning. ๐Ÿ”„ Pipeline: user input โ†’ personalization โ†’ 3D memory reconstruction ๐Ÿ’พ Storage: Redis-backed job/scene store with an in-memory fallback, so the whole app degrades gracefully without infra ๐Ÿ“ˆ Observability: OpenTelemetry tracing into Arize AX, plus an LLM-as-judge evaluator with a feedback loop for grading hotspot quality ### Challenges we ran into ๐ŸŽฅ Consistent scene generation for 3D Gaussian Splatting (3DGS): Our initial approach relied on fully generated videos from Midjourney. However, 3DGS depends on Structure-from-Motion, which requires smooth camera motion and consistent scene geometry across frames. Generated videos frequently introduced temporal inconsistencies that degraded reconstruction quality. To address this, we shifted toward real photos and videos while using Pika MCP to apply controlled personalization. This preserved the consistency required for reconstruction while still allowing creative modifications. ๐Ÿ’ธ Tool costs: We also experimented with Veo3-generated videos, which produced significantly better temporal consistency. Unfortunately, the cost of generating sufficient video data quickly exhausted our available credits. With greater resources, we believe a fully generative memory reconstruction pipeline could become feasible. โณ Training times: Training each splat took a significant amount of time (at least 30 minutes), and would sometimes hang for very long if the input had many photos or frames. The led us to spend a lot of effort trying different training inputs, from generated videos to generative mesh view points. In the end, we realized that sampling every other frame in a video could significantly speed up the process with minimal impact on visual quality. ### Accomplishments we're proud of ๐ŸŽฎ UI: We built a Three.js-powered viewer that successfully captures the feeling of stepping back into a memory rather than simply viewing media. โœจ 3DGS Quality: Despite having only a single day to develop and iterate, we achieved surprisingly strong reconstruction quality. We were especially excited to reconstruct a live human subject with limited distortion, since dynamic people are traditionally challenging for Gaussian Splatting yet are central to many memories. ### What we learned We learned how to design systems with long-latency AI pipelines involving video generation, scene reconstruction, and personalization. We also gained a much deeper understanding of 3D Gaussian Splatting, particularly the importance of input consistency and data quality. Most importantly, we explored how creative agents can personalize experiences rather than simply generate content. ### What's next We plan on expanding Rem to be able to traverse multiple memories as once by grouping them. For example, if someone went to Florida for vacation, they can upload their photos of the beach, the southernmost point of the continental US, and Disney World separately and then group them to be able to navigate from one scene to another. Another large area we could go into is increasing shareability of memories. We will likely make Rem a platform where users can share their memories and information hotspots with other users. These other users can add their own memories to make more hotspots, turning it into a multi-layered reconstruction of one scene. It's like Harry Potter's Pensieve, where multiple memories are being layered and pulled out of one's brain. Like Dumbledore said, "I sometimes find, and I am sure you know the feeling, that I simply have too many thoughts and memories crammed into my mind." Rem gives those memories a place to live. ## README (from the GitHub repository) # Rem โ€” walk through your memories in 3D Write down a moment (and drop a photo or video), and Rem turns it into a 3D Gaussian-splat scene you can walk through. Built at the Berkeley AI Hackathon. Two input modalities, same 3D output: - **Text / photo** โ†’ creative vision (Pika) โ†’ AI video (Veo 3) โ†’ - **Video** โ†’ re-graded to the memory's look (Pika fix-my-look; palette/lighting/mood changed, original geometry + camera motion preserved, so it stays COLMAP-friendly) โ†’ โ€ฆthen โ†’ frames โ†’ COLMAP โ†’ gaussian-splat training โ†’ a `scene.ply` you explore in the browser. --- ## Architecture โ€” 4 components, 2 deployments, 1 hosted DB | Component | Runs on | Does | Code | | ---------------- | ---------------------- | ------------------------------------------- | --------------------------------------------------- | ----------- | | **Frontend** | the user's **browser** | input UI, progress bar, 3D viewer | `src/app/**/page.tsx`, `src/components/` | | **Backend** | **Vercel** (Next API) | create memory, start pipeline, serve status | `src/app/api/`, `src/lib/` | | **DB + Storage** | **Supabase** | the memory row (status + splat URL) + files | `schema.sql`, accessed via `src/lib/` & `pipeline/` | | **GPU pipeline** | **Modal** | the heavy ML: (video | images)โ†’framesโ†’COLMAPโ†’gsplatโ†’`scene.ply` | `pipeline/` | Frontend + Backend are **one Next.js app** (one deploy). The pipeline is a **separate** Python deploy on Modal. Supabase is a hosted service. ``` ๐ŸŸฆ BROWSER โ”€โ”€HTTPโ”€โ”€โ–บ ๐ŸŸฉ VERCEL (Next API) โ”€โ”€triggerโ”€โ”€โ–บ ๐ŸŸฅ MODAL (GPU pipeline) โ–ฒ โ”‚ โ”‚ โ”‚ poll status โ”‚ create / read row โ”‚ write status + splat โ”‚ download scene.ply โ–ผ โ–ผ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€ ๐Ÿ“ฆ SUPABASE (Postgres + Storage) โ—„โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` The browser and the GPU never talk directly โ€” **the DB is the shared whiteboard.** --- ## End-to-end flow ``` 1. ๐ŸŸฆ Browser user submits a journal entry + photo(s) and/or a video 2. ๐ŸŸฉ Backend POST /api/memories โ†’ insert row (PENDING) โ†’ upload files โ†’ trigger Modal โ†’ return { id } 3. ๐ŸŸฅ GPU run_pipeline: agent layer โ†’ persona-coherent prompt / creative look (Gemini + Pika MCP) generate โ†’ Veo 3 video (photo/text) OR fix-my-look re-grade (video) score โ†’ scene-appropriate music (Pika MCP, optional) reconstruct โ†’ frames โ†’ COLMAP โ†’ gaussian-splat training โ†’ scene.ply (writes status GENERATINGโ†’RECONSTRUCTINGโ†’TRAINING to the DB as it goes) 4. ๐ŸŸฅ GPU upload scene.ply to Storage โ†’ update row (status=READY, splat_url=...) 5. ๐ŸŸฆ Browser polls GET /api/memories/:id every 2s โ†’ sees READY โ†’ loads splat_url into the viewer ``` `POSTโ†’Modal` is **async** (fire-and-forget). The pipeline runs its steps **in order (sync)**. The browser **polls** to learn when it's done. The big `scene.ply` is downloaded **directly from Storage** โ€” it never passes through the backend. Memories are also **embedded** (Voyage AI) and indexed in **Redis** for "find memories like this one" similarity search (`/api/memories/:id/similar`). --- ## The agent layer (how a memory becomes a coherent scene) The pipeline is a plain script, but the GENERATE half is driven by a small set of agents in `pipeline/agents/` so each new memory stays visually consistent with the person's past memories. Two patterns: **A. Prompted Gemini sub-agents** โ€” one shared multimodal client, each agent is a focused system prompt: - `vision` โ€” reads the uploaded photos **once** โ†’ a cached structured `PhotoAnalysis`. - `analyzer` โ€” past memories + this memory's vision โ†’ the recurring "world summary." - `extractor` โ€” pulls the slice of that world relevant to the new entry. - `persona` โ€” merges it into a persistent, evolving **persona spec** (the visual identity). **B. Pika MCP tool-agents** โ€” Gemini connected to the **Pika MCP server** as an MCP _client_ (OAuth via `agents/pika_auth.py`), using automatic tool-calling: - `creative` โ€” authors the **creative vision** for the shot from the scene + persona. - `fix_look` โ€” runs Pika's _fix-my-look_ skill to re-grade an input **video** to the memory's look (palette/lighting/mood) while preserving geometry, motion and identity. - `music` โ€” picks scene-appropriate music (`search_music` / `generate_music`) and mixes it under the clip (`edit_audio_mix`). `steps/compose_scene.py` orchestrates them: `vision โ†’ analyzer โ†’ extractor โ†’ persona โ†’ creative vision โ†’ final prompt`. Every Pika MCP agent is **gated** (`PIKA_MCP_ENABLED`) and **fail-safe** โ€” if disabled or erroring it returns nothing and the pipeline falls back (persona-only prompt, raw clip, no music), so reconstruction always runs. --- ## Repo structure ``` hack-berkeley/ โ”œโ”€โ”€ src/ โ”€โ”€ THE NEXT.JS APP (browser + backend) โ”€โ”€ โ”‚ โ”œโ”€โ”€ app/ โ”‚ โ”‚ โ”œโ”€โ”€ page.tsx ๐ŸŸฆ ingest screen โ”‚ โ”‚ โ”œโ”€โ”€ memories/[id]/page.tsx ๐ŸŸฆ progress โ†’ 3D viewer (polls status) โ”‚ โ”‚ โ””โ”€โ”€ api/ โ”‚ โ”‚ โ”œโ”€โ”€ memories/route.ts ๐ŸŸฉ POST create + start pipeline, GET list โ”‚ โ”‚ โ”œโ”€โ”€ memories/[id]/route.ts ๐ŸŸฉ GET status (the poll endpoint) โ”‚ โ”‚ โ”œโ”€โ”€ memories/[id]/similar/route.ts ๐ŸŸฉ semantic "similar memories" search โ”‚ โ”‚ โ””โ”€โ”€ redis-health/route.ts ๐ŸŸฉ Redis connectivity check โ”‚ โ”œโ”€โ”€ components/ ๐ŸŸฆ ingest-screen, loading-screen, memory-viewer โ”‚ โ”œโ”€โ”€ lib/ โ”‚ โ”‚ โ”œโ”€โ”€ supabase.ts ๐ŸŸฉ server-only Supabase client โ”‚ โ”‚ โ”œโ”€โ”€ db.ts ๐ŸŸฉ create/read the memories row โ”‚ โ”‚ โ”œโ”€โ”€ storage.ts ๐ŸŸฉ upload inputs / public URLs โ”‚ โ”‚ โ”œโ”€โ”€ modal.ts ๐ŸŸฉ trigger the GPU pipeline โ”‚ โ”‚ โ”œโ”€โ”€ embeddings.ts ๐ŸŸฉ text embeddings (Voyage AI) โ”‚ โ”‚ โ”œโ”€โ”€ memory-search.ts ๐ŸŸฉ Redis vector index + KNN search โ”‚ โ”‚ โ””โ”€โ”€ redis.ts ๐ŸŸฉ Redis client โ”‚ โ””โ”€โ”€ types/memory.ts ๐Ÿ“œ shared types = Contract A + C โ”‚ โ”œโ”€โ”€ pipeline/ โ”€โ”€ THE GPU SERVICE (Modal) โ”€โ”€ โ”‚ โ”œโ”€โ”€ app.py ๐ŸŸฅ Modal app + trigger endpoint โ”‚ โ”œโ”€โ”€ run_pipeline.py ๐ŸŸฅ the recipe (generate โ†’ reconstruct) โ”‚ โ”œโ”€โ”€ db.py / storage.py ๐ŸŸฅ Supabase status writes / file I/O โ”‚ โ”œโ”€โ”€ media.py ๐ŸŸฅ photo-vs-video detection + first-frame grab โ”‚ โ”œโ”€โ”€ veo.py ๐ŸŸฅ Veo 3 video generation (gated) โ”‚ โ”œโ”€โ”€ persona_store.py ๐ŸŸฅ the evolving persona spec (singleton) โ”‚ โ”œโ”€โ”€ smoke_test.py / full_test.py ๐ŸŸฅ local GENERATE tests (no GPU/Supabase) โ”‚ โ”œโ”€โ”€ agents/ โ”€โ”€ the coherence agent layer โ”€โ”€ โ”‚ โ”‚ โ”œโ”€โ”€ client.py ๐ŸŸฅ shared multimodal Gemini client โ”‚ โ”‚ โ”œโ”€โ”€ vision.py ๐ŸŸฅ one-time photo read (cached) โ”‚ โ”‚ โ”œโ”€โ”€ analyzer.py ๐ŸŸฅ past memories โ†’ world summary โ”‚ โ”‚ โ”œโ”€โ”€ extractor.py ๐ŸŸฅ relevant slice for this entry โ”‚ โ”‚ โ”œโ”€โ”€ persona.py ๐ŸŸฅ merge slice โ†’ persona spec โ”‚ โ”‚ โ”œโ”€โ”€ creative.py ๐ŸŸฅ creative vision (Pika MCP) โ”‚ โ”‚ โ”œโ”€โ”€ fix_look.py ๐ŸŸฅ video re-grade (Pika fix-my-look) โ”‚ โ”‚ โ”œโ”€โ”€ music.py ๐ŸŸฅ scene-aware music (Pika MCP) โ”‚ โ”‚ โ””โ”€โ”€ pika_auth.py ๐ŸŸฅ Pika MCP OAuth (authorize + refresh) โ”‚ โ””โ”€โ”€ steps/ โ”‚ โ”œโ”€โ”€ compose_scene.py ๐ŸŸฅ orchestrates the agents โ†’ prompt + analysis โ”‚ โ”œโ”€โ”€ generate_video.py ๐ŸŸฅ Veo 3: creative prompt โ†’ video โ”‚ โ”œโ”€โ”€ make_prompt.py ๐ŸŸฅ legacy one-shot prompt (superseded) โ”‚ โ”œโ”€โ”€ extract_frames.py ๐ŸŸฅ ffmpeg: video โ†’ frames โ”‚ โ”œโ”€โ”€ colmap.py ๐ŸŸฅ frames โ†’ camera poses โ”‚ โ”œโ”€โ”€ train_gsplat.py ๐ŸŸฅ po [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 66 recognized source files, 252 KB. - CSS (language) โ€” detected in the code - FastAPI (technology) โ€” detected in the code - Next.js (technology) โ€” detected in the code - Python (language) โ€” detected in the code - React (technology) โ€” detected in the code - Redis (technology) โ€” detected in the code - SQL (language) โ€” detected in the code - Supabase (technology) โ€” detected in the code - Tailwind CSS (technology) โ€” detected in the code - TypeScript (language) โ€” detected in the code - Anthropic (technology) โ€” claimed on Devpost, not found in the code - Google Gemini (technology) โ€” claimed on Devpost, not found in the code - JavaScript (language) โ€” claimed on Devpost, not found in the code - Node.js (technology) โ€” claimed on Devpost, not found in the code - AI coding agent: Claude Code โ€” evidence: config files committed to the repository; commit authorship or trailers - AI coding agent: Codex โ€” evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (78 of 78) ``` .env.local.example .gitignore .gitmodules AGENTS.md ARIZE.md CLAUDE.md eslint.config.mjs next.config.ts package.json pipeline/agents/__init__.py pipeline/agents/analyzer.py pipeline/agents/client.py pipeline/agents/creative.py pipeline/agents/extractor.py pipeline/agents/fix_look.py pipeline/agents/music.py pipeline/agents/persona.py pipeline/agents/pika_auth.py pipeline/agents/vision.py pipeline/app.py pipeline/db.py pipeline/full_test.py pipeline/fused_ssim_shim.py pipeline/media.py pipeline/persona_store.py pipeline/README_RECONSTRUCTION.md pipeline/reconstruct_local.py pipeline/requirements.txt pipeline/run_pipeline.py pipeline/setup_env.sh pipeline/smoke_test.py pipeline/steps/__init__.py pipeline/steps/colmap.py pipeline/steps/compose_scene.py pipeline/steps/export.py pipeline/steps/extract_frames.py pipeline/steps/generate_video.py pipeline/steps/make_prompt.py pipeline/steps/train_gsplat.py pipeline/storage.py pipeline/veo.py postcss.config.mjs public/bonsai.splat public/sample_memory.splat README.md schema.sql scripts/generate-sample-splat.mjs scripts/run-evaluation.mjs src/app/api/memories/[id]/evaluate/route.ts src/app/api/memories/[id]/route.ts src/app/api/memories/[id]/similar/route.ts src/app/api/memories/route.ts src/app/api/redis-health/route.ts src/app/api/reindex/route.ts src/app/globals.css src/app/layout.tsx src/app/memories/[id]/page.tsx src/app/page.tsx src/components/ingest-screen.tsx src/components/loading-screen.tsx src/components/memory-grid.tsx src/components/memory-viewer.tsx src/components/new-memory-form.tsx src/instrumentation.ts src/lib/db.ts src/lib/demo-data.ts src/lib/embeddings.ts src/lib/evaluator.ts src/lib/memory-search.ts src/lib/modal.ts src/lib/redis.ts src/lib/storage.ts src/lib/supabase.ts src/lib/tracing.ts src/types/gaussian-splats-3d.d.ts src/types/memory.ts src/types/speech-recognition.d.ts tsconfig.json ``` ### Dependencies - package.json: @arizeai/phoenix-otel@^1.0.2, @mkkellogg/gaussian-splats-3d@^0.4.7, @react-three/drei@^10.7.7, @react-three/fiber@^9.6.1, @react-three/postprocessing@^3.0.4, @supabase/supabase-js@^2.108.2, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, @types/three@^0.184.1, eslint@^9, eslint-config-next@16.2.9, next@16.2.9, react@19.2.4, react-dom@19.2.4, redis@^6.0.0, tailwindcss@^4, three@^0.184.0, typescript@^5 - pipeline/requirements.txt: fastapi, google-genai, gsplat, mcp, modal, numpy@<2.0.0, plyfile, requests, supabase ### Recent commits (newest first) - Update ARIZE.md by modifying evaluator section - [Feat/UI] restyle media selector to match blue glass theme - Merge pull request #5 from haileyl6171/deployment - update readme - [Merge] deployment into main - updates to walk - updated code - [Chore] gitignore .ply files - [Chore] gitignore .ply files - [Fix/UI] flip splat preview orientation and raise above tile - UI redesign - [Feat/UI] integrate .ply splat files into memory grid with camera persistence - Merge branch 'vid-gen' - feat(pipeline): Music Supervisor โ€” scene-aware music via Pika MCP - feat(pipeline): Music Supervisor โ€” scene-aware music via Pika MCP - feat(pipeline): Music Supervisor โ€” scene-aware music via Pika MCP - feat(pipeline): Music Supervisor โ€” scene-aware music via Pika MCP - [Feat/UI] restyle app to dark monochrome theme with minimalist fonts - Merge pull request #3 from haileyl6171/gsplat-pipeline - Merge remote-tracking branch 'origin/main' into gsplat-pipeline ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown @AGENTS.md ``` ### AGENTS.md ```markdown # This is NOT the Next.js you know This version has breaking changes โ€” APIs, conventions, and file structure may all differ from your training data. Read the relevant guide in `node_modules/next/dist/docs/` before writing any code. Heed deprecation notices. ``` ### package.json ``` { "name": "hack-berkeley", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint" }, "dependencies": { "@arizeai/phoenix-otel": "^1.0.2", "@mkkellogg/gaussian-splats-3d": "^0.4.7", "@react-three/drei": "^10.7.7", "@react-three/fiber": "^9.6.1", "@react-three/postprocessing": "^3.0.4", "@supabase/supabase-js": "^2.108.2", "@types/three": "^0.184.1", "next": "16.2.9", "react": "19.2.4", "react-dom": "19.2.4", "redis": "^6.0.0", "three": "^0.184.0" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.2.9", "tailwindcss": "^4", "typescript": "^5" } } ``` ### pipeline/requirements.txt ``` # Python deps for the pipeline. Pin versions before the demo once things work. # --- core: trigger endpoint + DB/storage + the one text call --- modal # orchestration + GPU hosting + the trigger endpoint supabase # DB writes (db.py) + Storage I/O (storage.py) google-genai # agents/ (Gemini text + photo vision) AND veo.py (Veo 3 video generation) mcp # agents/creative.py + agents/fix_look.py โ€” MCP client to the Pika MCP server requests # agents/fix_look.py โ€” download the restyled clip from Pika fastapi # required by @modal.fastapi_endpoint # --- reconstruction (P4): frames โ†’ COLMAP โ†’ gsplat โ†’ .ply/.splat --- # COLMAP is a SYSTEM binary, NOT pip: apt install colmap / brew install colmap # # Install torch FIRST, matching your CUDA (https://pytorch.org/get-started), e.g.: # pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121 gsplat # CUDA rasterizer (builds against your torch/CUDA) plyfile # export.py: read the trained 3DGS .ply numpy<2.0.0 # gsplat examples pin numpy<2 # gsplat's TRAINER (examples/simple_trainer.py) has its own dependency set โ€” get # them from the cloned repo, not listed here individually: # git clone https://github.com/nerfstudio-project/gsplat vendor/gsplat # pip install -r vendor/gsplat/examples/requirements.txt ``` ### pipeline/app.py ```python # ============================================================================ # Modal app + trigger endpoint (GPU side of CONTRACT B) # Owned by P3. # # Deploy: cd pipeline && modal deploy app.py โ†’ prints the URL for MODAL_URL # Dev: cd pipeline && modal serve app.py โ†’ live-reloading dev URL # (run from the pipeline/ dir so the image's relative paths resolve) # # This file does TWO things: # 1. `start` โ€” a tiny HTTPS endpoint the backend POSTs to. It verifies the # secret, spawns the heavy job, and returns immediately. # 2. `run` โ€” the heavy GPU function that actually runs the pipeline. # ============================================================================ import os import modal app = modal.App("rem-pipeline") # --------------------------------------------------------------------------- # Container image โ€” mirrors pipeline/setup_env.sh (the VERIFIED, no-compile # stack). Deploy/serve FROM the pipeline/ dir so relative paths resolve: # cd pipeline && modal deploy app.py # # Why NOT debian_slim + apt colmap + pip gsplat (the old, broken version): # โ€ข gsplat has no clean source build here โ€” we must use the PREBUILT wheel, # which exists ONLY for Python 3.10 + torch 2.4/cu121 (pip would otherwise # try to compile gsplat โ†’ the build tar pit). # โ€ข apt colmap is CPU-only; setup_env.sh uses the conda-forge CUDA build. # So: micromamba (py3.10) + conda-forge colmap/ffmpeg + pip torch + gsplat wheel. # --------------------------------------------------------------------------- image = ( modal.Image.micromamba(python_version="3.10") # The build node has no GPU, so conda's __cuda probe is empty and it would # resolve the CPU colmap. Declaring a driver CUDA lets it prefer the CUDA # build (and fall back to CPU if no compatible CUDA build exists โ€” which is # fine: high_quality extraction is CPU DSP-SIFT anyway, GPU only speeds # matching). Bump this if you want to force a newer CUDA colmap. .env({"CONDA_OVERRIDE_CUDA": "12.4"}) .micromamba_install("colmap", "ffmpeg", channels=["conda-forge"]) # torch from its own index, THEN the prebuilt gsplat wheel. Deps are # pre-installed first so gsplat's single-index install doesn't need to # resolve them. Nothing compiles. .pip_install("torch==2.4.1", "torchvision==0.19.1", index_url="https://download.pytorch.org/whl/cu121") .pip_install("ninja", "numpy<2.0.0", "jaxtyping", "rich") .pip_install("gsplat==1.5.3", index_url="https://docs.gsplat.studio/whl/pt24cu121") .pip_install_from_requirements("requirements.txt") # simple_trainer.py ships in the gsplat REPO (not the wheel) โ€” clone it pinned # to the wheel's version, and install its example deps MINUS everything that # compiles CUDA (fused_ssim is shimmed just below). .run_commands( "git clone --depth 1 --branch v1.5.3 " "https://github.com/nerfstudio-project/gsplat.git /opt/gsplat", "grep -vE 'fused-ssim|fused_ssim|fused-bilagrid|ppisp|nvidia-ncore|" "rahul-goel|harry7557558|nv-tlabs' /opt/gsplat/examples/requirements.txt " "> /tmp/ex.txt && pip install -r /tmp/ex.txt", ) # our pipeline source + the pure-Python fused_ssim shim (so the trainer's # `from fused_ssim import fused_ssim` resolves without compiling). .add_local_dir(".", "/root/pipeline", copy=True, ignore=["**/__pycache__", "**/*.pyc", "third_party/**"]) .run_commands("cp /root/pipeline/fused_ssim_shim.py /opt/gsplat/examples/fused_ssim.py") # train_gsplat reads GSPLAT_REPO; run() imports run_pipeline from here. .env({"GSPLAT_REPO": "/opt/gsplat", "PYTHONPATH": "/root/pipeline"}) .workdir("/root/pipeline") ) # Secrets (set once): # modal secret create rem-secrets SUPABASE_URL=... SUPABASE_SERVICE_ROLE_KEY=... # GEMINI_API_KEY=... MODAL_SECRET=... # (video generation, when enabled: add VEO_ENABLED=1 โ€” Veo 3 reuses GEMINI_API_KEY) secrets = [modal.Secret.from_name("rem-secrets")] @app.function(image=image, gpu="A10G", timeout=3600, secrets=secrets) def run(memory_id: str, input_keys: list[str], description: str) -> None: """The heavy job. Runs for minutes on a GPU, then the machine is torn down.""" # Imported here so the endpoint container doesn't need the heavy deps. from run_pipeline import run_pipeline run_pipeline(memory_id, input_keys, description) @app.function(image=image, secrets=secrets) @modal.fastapi_endpoint(method="POST") def start(body: dict): """ CONTRACT B endpoint. Backend POSTs { memoryId, inputKeys, description } with header X-Secret. We verify, spawn the job, and return right away. NOTE: header access depends on the Modal/FastAPI version. If you need the header, switch the signature to accept a fastapi.Request and read request.headers["x-secret"]. For a hackathon you may also pass the secret in the JSON body. Verify the secret either way โ€” don't leave it open. """ # TODO(P3): verify X-Secret == os.environ["MODAL_SECRET"]; 401 if mismatch. run.spawn( body["memoryId"], body.get("inputKeys", []), body.get("description", ""), ) return {"ok": True} ``` ### src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Playfair_Display } from "next/font/google"; import "./globals.css"; const geist = Geist({ variable: "--font-geist", subsets: ["latin"], }); const playfair = Playfair_Display({ variable: "--font-playfair", subsets: ["latin"], }); export const metadata: Metadata = { title: "rem", description: "transform moments into 3d gaussian splat memories", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### src/app/page.tsx ```typescript "use client"; import { useState, useCallback, useRef } from "react"; import IngestScreen from "@/components/ingest-screen"; import LoadingScreen from "@/components/loading-screen"; import MemoryViewer, { type CameraState } from "@/components/memory-viewer"; import { DEMO_MEMORIES, buildDemoMemory, type DemoMemory, } from "@/lib/demo-data"; type ViewState = "grid" | "loading" | "viewer"; // Length of the cream "wash" that covers the swap between screens. const WASH_COVER_MS = 360; const WASH_CLEAR_MS = 460; export default function Home() { const [view, setView] = useState("grid"); const [memories, setMemories] = useState(DEMO_MEMORIES); const [activeMemory, setActiveMemory] = useState(null); const [pending, setPending] = useState(null); const [washing, setWashing] = useState(false); // Per-memory camera pose, so re-entering a scene drops you where you left off. const [cameraStates, setCameraStates] = useState>({}); const createdCount = useRef(0); // Cinematic screen swap: fade a cream wash over the screen, swap the view // underneath it, then clear the wash so the new screen's entrance plays. const transitionTo = useCallback((next: ViewState, after?: () => void) => { setWashing(true); window.setTimeout(() => { after?.(); setView(next); window.setTimeout(() => setWashing(false), WASH_CLEAR_MS); }, WASH_COVER_MS); }, []); // Click an existing memory tile โ†’ open its scene. const handleMemoryClick = useCallback( (id: string) => { const memory = memories.find((m) => m.id === id); if (!memory) return; transitionTo("viewer", () => setActiveMemory(memory)); }, [memories, transitionTo], ); // Submit the form โ†’ run the scripted reconstruction, then reveal the scene. const handleGenerate = useCallback( (description: string, imageFiles: File[], videoFile: File | null) => { const memory = buildDemoMemory(description, createdCount.current++); transitionTo("loading", () => setPending(memory)); const form = new FormData(); form.append("description", description); imageFiles.forEach((file) => form.append("photos", file)); if (videoFile) form.append("video", videoFile); fetch("/api/memories", { method: "POST", body: form }) .then(async (res) => { const body = await res.json().catch(() => ({})); if (!res.ok) { console.warn("[create memory] non-OK response:", res.status, body); } else { console.info("[create memory] created:", body.id); } }) .catch((err) => console.error("[create memory] request failed:", err)); }, [transitionTo], ); // The loading sequence finished: file the new memory into the grid and open it. const handleLoadingComplete = useCallback(() => { if (!pending) return; transitionTo("viewer", () => { setMemories((prev) => [pending, ...prev].slice(0, 8)); setActiveMemory(pending); setPending(null); }); }, [pending, transitionTo]); const handleReturn = useCallback( (state?: CameraState) => { if (activeMemory && state) { setCameraStates((prev) => ({ ...prev, [activeMemory.id]: state })); } transitionTo("grid", () => setActiveMemory(null)); }, [activeMemory, transitionTo], ); // Inside the viewer, jump to a related memory's scene without leaving. const handleSelectRelated = useCallback( (id: string) => { const memory = memories.find((m) => m.id === id); if (memory) setActiveMemory(memory); }, [memories], ); const related = activeMemory ? memories.filter((m) => m.id !== activeMemory.id).slice(0, 5) : []; return (
{view === "grid" && (
)} {view === "loading" && (
)} {view === "viewer" && activeMemory && (
)} {/* Cinematic cross-dissolve overlay (cream wash with a soft vignette). */}
); } ``` ### src/app/api/redis-health/route.ts ```typescript import { NextResponse } from "next/server"; import { getConnectedRedisClient } from "@/lib/redis"; export async function GET() { try { const client = await getConnectedRedisClient(); const pong = await client.ping(); return NextResponse.json({ ok: true, pong }); } catch (err) { return NextResponse.json( { ok: false, error: err instanceof Error ? err.message : String(err) }, { status: 500 } ); } } ``` ### src/app/api/reindex/route.ts ```typescript // ============================================================================ // POST /api/reindex โ€” backfill every memory's embedding into the Redis vector // index. The create path (POST /api/memories) indexes new memories, but this // rebuilds the index for EXISTING rows โ€” needed after switching REDIS_URL // (e.g. to a local Redis Stack when Redis Cloud is network-blocked, see // ARIZE.md) or changing the embedding model. Embeds with 429 backoff. // ============================================================================ import { NextResponse } from "next/server"; import { listMemories } from "@/lib/db"; import { indexMemory } from "@/lib/memory-search"; export const runtime = "nodejs"; export async function POST() { const memories = await listMemories(); const done: string[] = []; const failed: { id: string; error: string }[] = []; for (const m of memories) { if (!m.description?.trim()) continue; try { await indexMemory({ id: m.id, description: m.description, status: m.status ?? "PENDING", splat_url: m.splat_url ?? null, created_at: m.created_at, }); done.push(m.id); } catch (err) { failed.push({ id: m.id, error: err instanceof Error ? err.message : String(err) }); } } return NextResponse.json({ indexed: done.length, failed }); } ``` ### src/app/api/memories/route.ts ```typescript // ============================================================================ // POST /api/memories โ€” create a memory + start the pipeline (CONTRACT A) // GET /api/memories โ€” list memories (optional gallery) // Owned by P2. // // Next 16 notes (VERIFY against node_modules/next/dist/docs after `npm i`): // โ€ข Route handlers export named async functions (GET, POST, ...). // โ€ข Default runtime is Node.js โ€” required here (Supabase service key + file // upload). We set it explicitly for clarity. // ============================================================================ import { NextResponse } from "next/server"; import { traceChain } from "@arizeai/phoenix-otel"; import { createMemory, listMemories } from "@/lib/db"; import { uploadInput } from "@/lib/storage"; import { triggerPipeline } from "@/lib/modal"; import { indexMemory } from "@/lib/memory-search"; import { withMemoryTrace } from "@/lib/tracing"; import type { CreateMemoryResponse } from "@/types/memory"; export const runtime = "nodejs"; const handleCreateMemory = traceChain( async (request: Request) => { const form = await request.formData(); const description = String(form.get("description") ?? "").trim(); const photo = form.get("photo"); if (!description && !photo) { return NextResponse.json( { error: "Provide a description and/or a photo." }, { status: 400 }, ); } const id = await createMemory({ description, inputKeys: [] }); return withMemoryTrace(id, async () => { const inputKeys: string[] = []; if (photo instanceof File && photo.size > 0) { inputKeys.push(await uploadInput(id, photo)); } if (description) { try { await indexMemory({ id, description, status: "PENDING" }); } catch (err) { console.error("[memory-search] index failed for", id, err); } } await triggerPipeline({ memoryId: id, inputKeys, description }); const body: CreateMemoryResponse = { id }; return NextResponse.json(body, { status: 201 }); }); }, { name: "POST /api/memories" }, ); export async function POST(request: Request) { return handleCreateMemory(request); } export async function GET() { const memories = await listMemories(); return NextResponse.json(memories); } ``` [57 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]