# Project export: Reframe
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: OpenAI Build Week
- Tagline: Voice-driven extended reality. GPT Realtime hears you, GPT-5.6 Sol plans agentic, and Reframe segments the object and rebuilds the background from monocular depth and TSDF fusion without LiDAR.
- Devpost: https://devpost.com/software/reroom
- GitHub: https://github.com/Julian-AT/openai-build-week
- Video: https://www.youtube.com/embed/0-C9RTRu8j4?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — Julian S. (603 commits)
## Devpost submission (written by the team)
### Inspiration
Every AR furniture app can put a virtual couch in your room. Not one of them removes the couch that's already there, and that's the whole problem. You can't evaluate a new armchair when the old one is still sitting in the shot. Compositing a posed mesh over the feed is nothing, we've all done it. The hard part is the inverse: reconstructing the floor, wall, and skirting behind an object that is physically present and occluding those exact pixels in every frame you receive. We wanted to hold up a phone, point it at a chair, say "replace this with something warmer and red," and have it resolve on a live feed we never touch.
### What it does
Reframe is a live diminished/augmented reality room editor for iPhone. Place a catalog asset on the floor, swap a real object for one whose projected silhouette covers it, pull a real object out and reveal reconstructed background, undo instantly, offline included. Retrieval sits on a Qdrant vector DB with 20k+ furniture items, content-addressed. When you say "warmer and red" we embed it and run the ANN query under a hard pre-filter on the target region's measured footprint and clearance, so nothing that can't physically fit ever comes back, and we re-rank whatever survives on fit residual. Voice (GPT Realtime 2.1 over WebRTC), reticle, and tap all lower to the same intent struct and hit one resolver, because maintaining three of anything is how you get three sets of bugs. Sessions serialize to a durable .rfcap file so you can replay and re-edit from a Next.js web twin. Readiness is per-capability and we don't lie about it: replace lights up the moment a coarse proxy exists, remove holds until reveal coverage is good enough, and there's a visible "Healing" state while inpainting is still converging.
### How we built it
The rule the whole thing hangs on: models are advisory, deterministic code owns identity, geometry, revision, commit, restore. GPT-5.6 Sol plans through five read-only tools and hands back a proposed edit. It cannot commit anything. A deterministic validator does that, and only that. It runs collision and clearance in the metric scene frame, checks the support relation against the detected supporting plane, and computes a rasterized cover score by projecting the candidate mesh under the current pose against the target mask, \( c = |S_{\text{new}} \cap S_{\text{target}}| / |S_{\text{target}}| \). The edit lands only if \( c \geq \tau \). Commit is one compare-and-swap on a monotonic revision counter behind an idempotency key, so a stale or replayed proposal is a no-op instead of a corruption. Nothing a model returns ever writes scene state. Geometry runs on two tracks, and the split wasn't a design choice so much as something the latency forced on us. First-edit time cannot wait on dense reconstruction. So the fast track fuses ARKit VIO poses and plane anchors with a SAM 3.1 video track, carving the mask sequence into a voxel occupancy volume that we deliberately dilate rather than erode. That gives you an editable proxy in a couple of seconds that is guaranteed to over-cover the real object, which is exactly the failure direction you want. The dense track runs DA3 monocular metric depth and solves a per-frame affine alignment \( \hat{d} = a\,d_{\text{pred}} + b \) against sparse VIO depth with robust least squares, throwing out any frame whose residual is too high, then fuses the accepted frames into an Open3D CUDA TSDF with truncation a few voxels wide. Occlusion masks and metric extents tighten asynchronously as the SDF firms up. Reveal texturing follows a strict precedence and we never break it: reprojected observed pixels from other views first, deterministic planar/homography fills second, LaMa only for texels no geometry can account for. A synthesized texel is never permitted to paint over an observed one. On device it's SwiftUI, ARKit, RealityKit, with a Metal compositing fallback for the diminished-reality passes RealityKit won't express without a fight. The gateway is Bun and TypeScript over SQLite in WAL mode, fronting an in-process GPU lane scheduler. Vision workers are Python 3.12 in isolated Docker containers behind that scheduler. One JSON Schema contract defines the binary FramePacket and every message on the wire, and we implemented it three times, Swift, TypeScript, Python, so the bytes mean the same thing everywhere. Sol was good at taking loose constraints and turning them into schema-ready field definitions and leaning on the contract boundaries until they held. Codex wrote a large chunk of the cross-runtime implementation and its tests. Four people don't get this far in a week without both.
### Challenges we ran into
Base iPhone 17, no LiDAR. Every metric millimetre is VIO plus learned depth, and learned depth is affine-ambiguous in scale, full stop. So the alignment stage will drop a frame before it fuses a plausible-but-wrong one, because a single bad scale estimate doesn't degrade the TSDF locally, it contaminates it globally. The thing that ate the most hours, embarrassingly, was coordinate conventions. ARKit is right-handed looking down \( -Z \), OpenCV looks down \( +Z \), and the flip \( \text{diag}(1,-1,-1) \) is its own inverse. Apply it an even number of times and everything looks almost right, which is the worst kind of wrong because it survives eyeballing. We now carry the handedness tag next to every matrix and gate it in CI with a known-ray reprojection test that fails on any parity mismatch. Keeping the first edit off the serial depth-to-TSDF-to-mesh chain is what forced the two-track design in the first place. And one GPU multiplexing segmentation, depth, fusion, reveal, and naming will absolutely starve itself if you let it, so there's an explicit priority-lane coordinator with preemption in front of it that keeps a reveal job from ever blocking the interactive segmentation lane. Reveal isn't instant and we refused to pretend otherwise, so we pinned it to real gates: \( p_{10} \) and median reveal coverage across eight sampled viewpoints.
### Accomplishments we're proud of
The deterministic core is tested code, not a slide. The binary FramePacket contract, the coordinate conventions, the CAS transaction and undo model with idempotency keys, the cover-score math with its exact thresholds, crash-safe .rfcap capture on device, HMAC room-scoped session auth, the GPU lane coordinator, the 20k+ item catalog pipeline. Unit-tested across all three runtimes. And the invariants actually held: the camera feed is never punched per pixel, a model never writes scene state, synthesized texels never overwrite observed ones, and the phone never claims a capability is ready before the artifact behind it is verified and live.
### What we learned
Put every model behind a typed provider boundary. You will change your mind about which model you're using. You will not get to change your mind about what identity means, so don't couple them. Readiness is per-capability, "editable" was never one bit, and people trust the system more when replace unlocks before remove and the UI just tells them why. A dilated mask is invisible, a leaked chair leg is not, so conservative geometry wins over clever geometry every single time. Contracts-first is the only reason four people and two AI collaborators could work across three languages without the implementations quietly drifting apart. And in a fusion pipeline a dropped frame costs you nothing, you get the next one, but a confidently wrong frame poisons everything downstream of it.
### What's next
Closing the gates we set. A full replace on a live SAM 3.1 track, end to end. LaMa and the CUDA TSDF running in production at a budgeted cadence. Live voice turns and Qdrant retrieval over the full 20k+ catalog under real session load. FPS, thermal, and visual-quality validation on actual hardware, not a bench. After that, catalog ingestion at IKEA scale, the Mode B1 photoreal Gaussian-splat twin in the web viewer, tabletop and shelf support relations, multi-object scenes, and eventually multi-room capture. None of it worth giving up an invariant to get.
## README (from the GitHub repository)
# Reframe
> A live spatial design system for understanding and reshaping real rooms.
[](LICENSE)
Reframe turns a live room capture into reversible spatial edits. The native
iPhone experience combines ARKit tracking, spatial understanding, prepared 3D
assets, and an OpenAI design assistant without giving cloud services control of
the render loop or canonical scene state.
Instead of treating AR as a model viewer, Reframe understands what already
occupies the room, finds an asset that physically fits, and previews the change
against the live camera. The four operations are **place**, **replace**,
**remove**, and **restore**.
Read the [Reframe technical paper](REFRAME_PAPER.pdf) for the complete system
architecture, contracts, and execution model.
## How it works
```mermaid
%%{init: {"theme": "neutral"}}%%
flowchart TB
subgraph DEVICE["ON DEVICE | 60 FPS AND OFFLINE SAFE"]
direction LR
ARKIT["ARKit session
poses, planes, frame quality"]
ADMISSION["Frame admission
quality, baseline, backpressure"]
REPLICA["Local scene replica
revisioned artifact cache"]
RENDER["RealityKit compositor
camera, occlusion, reveal, assets"]
ARKIT --> ADMISSION
ARKIT --> RENDER
REPLICA --> RENDER
end
subgraph PERCEPTION["SPATIAL UNDERSTANDING | ASYNCHRONOUS GPU PATHS"]
direction LR
INGEST["Typed frame ingest"]
FAST["Fast path
semantic track, silhouette volume"]
DENSE["Dense path
metric depth, alignment, TSDF"]
OBJECT["Object model
identity, bounds, support"]
GEOMETRY["Scene geometry
surfaces, dimensions, occluders"]
REVEAL["Reveal synthesis
observed atlas, bounded fill"]
ARTIFACTS["Versioned spatial artifacts"]
INGEST --> FAST --> OBJECT
INGEST --> DENSE --> GEOMETRY
OBJECT --> REVEAL
GEOMETRY --> REVEAL
OBJECT --> ARTIFACTS
GEOMETRY --> ARTIFACTS
REVEAL --> ARTIFACTS
end
subgraph CONTROL["DESIGN CONTROL | BOUNDED AI AND DETERMINISTIC TOOLS"]
direction LR
INTENT["Voice, tap, pointer context"]
CATALOG["Eligible asset catalog
dimensions, provenance, render profile"]
PLAN["Typed design proposal"]
VALIDATE{"Deterministic validation
target, fit, clearance, revision"}
PREVIEW["Preview transaction"]
INTENT --> PLAN
CATALOG --> PLAN
PLAN --> VALIDATE --> PREVIEW
end
subgraph AUTHORITY["SCENE AUTHORITY | PREVIEW BEFORE COMMIT"]
direction LR
ACTIVATE["Local preview activation"]
CONFIRM{"User confirmation"}
COMMIT["Compare and swap commit
undo, restore, replay"]
ACTIVATE --> CONFIRM --> COMMIT
end
ADMISSION --> INGEST
ARTIFACTS --> REPLICA
ARTIFACTS --> PLAN
PREVIEW --> ACTIVATE
PREVIEW --> REPLICA
COMMIT --> REPLICA
```
Reframe separates rendering, inference, planning, and state mutation. ARKit is
the metric pose authority. Accepted frames fan out into a fast semantic path
for target identity and a dense reconstruction path for surfaces, dimensions,
and occlusion. Both paths publish versioned artifacts without entering the 60
FPS render loop.
Voice and tap input carry explicit pointer and scene context. The agent may
interpret intent and retrieve eligible assets, but deterministic tools resolve
the target, validate physical fit, and reject stale revisions. The iPhone
renders the proposal from its local replica. Only user confirmation creates a
new scene revision, and every committed edit remains undoable and restorable.
## Workspace
| Area | Responsibility |
| --- | --- |
| [iOS](apps/ios/README.md) | Native capture, interaction, and AR rendering |
| [API](apps/api/README.md) | Trusted gateway, sessions, transactions, and service coordination |
| [Vision](apps/vision/README.md) | Private segmentation, depth, geometry, and reveal inference |
| [Web](apps/web/README.md) | Room model, capture handoff, and replay experience |
| [Agent](packages/agent/README.md) | Bounded Responses and Realtime adapters |
| [Catalog](packages/catalog/README.md) | Acquisition, preparation, retrieval, and delivery of 3D assets |
| [Protocol](packages/protocol/README.md) | Canonical schemas, coordinates, and transaction behavior |
## Development setup
Reframe is a multi-runtime spatial system, not a one-command application. The
complete experience spans a physical iPhone, a trusted gateway, Qdrant, private
GPU vision workers, the web client, prepared 3D assets, and optional OpenAI
voice and planning.
### Prerequisites
| Requirement | Used for |
| --- | --- |
| macOS, Xcode with the iOS 18 SDK, and a physical iPhone | ARKit capture and the live spatial editor |
| [Bun 1.3.11](https://bun.sh/) | Gateway, web client, and TypeScript packages |
| Python 3.12 and `uv >=0.9.26,<0.12` | Vision service and its verification toolchain |
| Docker with Compose | Local gateway and Qdrant topology |
| CUDA-capable GPU plus prepared DA3 and SAM sources/checkpoints | Live depth, geometry, and target tracking |
| OpenAI API credentials | Realtime voice and bounded design-agent turns |
Install the JavaScript workspace with `bun install --frozen-lockfile`, then
bring up the system capability by capability:
| Order | Component | Required configuration | Runbook |
| ---: | --- | --- | --- |
| 1 | Gateway + Qdrant | Gateway, room-signing, and Qdrant secrets; absolute data directory | [API setup](apps/api/README.md) |
| 2 | Vision workers | Private service token, profile, model sources, checkpoints, revisions, and hashes | [Vision setup](apps/vision/README.md) |
| 3 | Asset catalog | Authorized source frontier, external asset store, processor tools, embeddings, and Qdrant access | [Catalog setup](packages/catalog/README.md) |
| 4 | Web client | Gateway URL and server-side token for connected routes | [Web setup](apps/web/README.md) |
| 5 | iPhone app | Gateway URL, room ID, short-lived room credential, and signing configuration | [iOS setup](apps/ios/README.md) |
The services must agree on gateway and vision URLs, scoped tokens, room/session
identity, and artifact storage. Model repositories and checkpoints are prepared
explicitly; workers do not download them at startup. The landing-page point
cloud viewer can run without those services, but it is a **UI-only preview**,
not a full Reframe deployment.
Run the repository checks with:
```sh
bun run check
bun run test:swift
```
## Stack
| Layer | Technology |
| --- | --- |
| Native | Swift 6.1, SwiftUI, ARKit, RealityKit |
| Web | Next.js 16, React 19, Three.js |
| Gateway | Bun, TypeScript, Hono, SQLite |
| Vision | Python 3.12, FastAPI, provider-isolated GPU models |
| AI | OpenAI Responses API and Realtime API |
| Assets | GLB, USDZ, Qdrant semantic retrieval |
## Design principles
- The live renderer never waits for the network or an inference worker.
- ARKit remains the pose authority for healthy native sessions.
- AI tools are bounded, typed, and unable to commit canonical state.
- Prepared assets are activated only after dimensions, provenance, and hashes verify.
- Committed edits remain available locally when cloud services disconnect.
## License
Reframe is available under the [MIT License](LICENSE). Built for OpenAI Build
Week 2026.