# Project export: Worldly This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2026 - Tagline: Automatically transform footage into explorable VR worlds in Meta Quest - Devpost: https://devpost.com/software/worldly-0tm3bl - GitHub: https://github.com/vihaanm23/worldly - Demo: https://marble.worldlabs.ai/world/ecc3945a-4313-4078-a869-0a4919ec27e5 - Video: https://www.youtube.com/embed/4riPKyUa62E?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Ayush Paul (5 commits), Vihaan Mathur (3 commits) ## Devpost submission (written by the team) ### Overview Worldly takes in any footage and converts it into fully explorable 3D VR Worlds in a Meta Quest Headset. ### Inspiration All of us are super passionate about the recent advancements in World Models and Machine Learning Research. When World Labs released their world foundation model API (Prompt → 3D world, Released ~2 weeks ago: worldlabs.ai/blog/announcing-the-world-api), we immediately saw the potential for TreeHacks, but we wanted to take it further. What if we could make this tech accessible to anyone with a drone, a VR headset, or even just a phone? We thought back to all our favorite memories and wished we could revisit that place exactly how it was in the videos we took on our phones. Thinking about literally stepping into those moments inspired us to build Worldly. The capabilities are what pushed us to go further: training simulations in reconstructed real-world environments, preserving historical sites before they change, enabling remote collaboration in actual physical spaces, or simply going back your favorite vacation spot whenever you want. That is Worldly. ### What it does Worldly lets you revisit any location you've ever been to in fully immersive VR. Here's how it works: Capture with your drone → Using our custom iOS app, you fly your DJI drone anywhere and record footage directly from the controller's live feed. Our app captures your footage and optimizes it for our World Model. Automated Processing → The second you stop recording, your footage automatically uploads to our backend server and enters our processing pipeline. AI-powered 3D reconstruction → We feed your footage into the World Labs API, which uses their world foundation model to reconstruct the entire environment as a photorealistic 3D Gaussian splat. Instant VR access → The generated world automatically appears in your Worldly dashboard and Meta Quest library. You can click on any world, put on your headset, and you're standing inside that location and can explore it in all directions. It also works with phones for smaller indoor spaces. You can record with your camera, and the same pipeline transforms it into an explorable VR world. We've already captured four locations across Stanford: outside the NVIDIA building, the church, inside the TreeHacks venue, and Hoover Tower. ### How we built it DJI Drone Integration (iOS/Swift + ReplayKit) → We built a native iOS app that captures live drone footage directly from the DJI controller app using ReplayKit. The app crops the center viewport for the best angle, cleans it, and automatically uploads it to our server. Backend Server/Connections (Node.js, Flask): Our Node.js backend receives uploaded footage and orchestrates the entire 3D world pipeline. It manages video preprocessing, API calls, file conversion, and world deployment automatically. World Model API (Python): We integrated the newly released World Labs Marble world-model API, which uses AI to reconstruct real-world geometry from video. It analyzes camera movement, depth, and scene structure to generate a photorealistic 3D Gaussian splat in SPZ format. We input a custom prompt along with the drone footage to create a hyperrealistic world. WebXR Viewer in Meta Quest (three.js, Vite, Meta Immersive Kit, TypeScript): We built a custom web-based renderer for Meta Quest that loads PLY Gaussian splat files directly, renders them interactively in real-time, and enables full spatial navigation inside the Meta Quest World Dashboard/Web app (C#, HTML, JSON): We built a web dashboard that tracks all your captured worlds, shows processing status, and lets you launch any world directly into VR with one click. It's your personal library of places you've been. ### Challenges we ran into WorldLabs API → Since the API is so new, we faced a lot of errors with I/O. We had to be very specific with the types of footage we input into the API, as well as the output format. The file format output was not compatible with the headset, so we had to integrate conversion into our custom world map viewer. We also had to carefully orchestrate our backend to queue requests efficiently and handle processing delays. Some reconstructions took 10-15 minutes, so we built a status tracking system to keep us informed. DJI SDK → Only worked on Android, but we needed iOS compatibility. We pivoted to controlling the drone manually while using ReplayKit to automatically screen-record the footage from the DJI controller app. This workaround gave us clean video capture without requiring SDK access. VR Issues →The PLY formatted output was super niche and lacked proper WebXR support, so we had to build our own custom WebXR viewer from scratch. ### Accomplishments we're proud of We are super proud of being one of the first devs to build on top of the World Labs API. We see this technology going so far and are super excited to have worked with it. We are also proud that we are about to turn our drone footage into 3D worlds. We weren't sure it would work or generate anything viewable, and it was such a reassuring moment when we saw what we created. ### What we learned We learned a lot. For all of us, this was one of our first times working with Unity and AR/VR, so it was a really fun and exciting experience to build on top of Meta Quest. ### What's next The applications for Worldly are endless, as World Models improves, so will Worldly: Multi-user worlds →Let multiple people explore the same captured space together in VR Social features → Share your worlds publicly, explore places others have captured around the world More VR support →More intuitive Menu/Exploration, expand beyond Meta Quest to Apple Vision Pro, smartphone-based VR headsets Higher resolution captures → Use better models, optimize for 4K+ drone footage to increase splat quality Vision/Real World Use Cases: Real-time collaboration → Capture a space and have your remote team members "walk through" it instantly Families → Create "albums" of real-world memories for families everywhere Historical preservation → Partner with museums and historical sites to create permanent VR archives of important locations Industrials → Property tours, construction site monitoring, spatial planning Training → Military, emergency response, search/rescue Tourism → Preview destinations before booking, revisit your favorite vacation spots ## README (from the GitHub repository) No README available. ## Detected evidence (automated analysis) Indexed codebase: 31 recognized source files, 130 KB. - C# (language) — detected in the code - CSS (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code ## Codebase structure (from repository index) ### Files (56 of 56) ``` .gitignore footage_extractor/Assets.xcassets/AccentColor.colorset/Contents.json footage_extractor/Assets.xcassets/AppIcon.appiconset/Contents.json footage_extractor/Assets.xcassets/Contents.json footage_extractor/BroadcastPickerView.swift footage_extractor/ContentView.swift footage_extractor/ScreenRecorderViewModel.swift footage_extractor/treeHacks26.entitlements footage_extractor/treeHacks26App.swift old-teleporter/.gitignore old-teleporter/app/globals.css old-teleporter/app/layout.tsx old-teleporter/app/page.tsx old-teleporter/app/viewer/page.tsx old-teleporter/app/viewer/viewer-client.tsx old-teleporter/bun.lock old-teleporter/next.config.js old-teleporter/package.json old-teleporter/postcss.config.js old-teleporter/public/splats/ceramic-500k-demo.spz old-teleporter/public/splats/ceramic-demo.spz old-teleporter/public/splats/dust-demo.spz old-teleporter/public/vr-viewer.html old-teleporter/README.md old-teleporter/tailwind.config.ts old-teleporter/tsconfig.json old-teleporter/types/gaussian-splats-3d.d.ts portal/.DS_Store portal/docs/SETUP_STEP_BY_STEP.md portal/README.md portal/test/check-worlds.js portal/test/web-launcher.html portal/unity/Assets/Scripts/WorldEntry.cs portal/unity/Assets/Scripts/WorldLauncher.cs portal/unity/Assets/Scripts/WorldListConfig.cs portal/unity/Assets/StreamingAssets/Worlds.json portal/web-app/index.html portal/web-app/worlds.json teleporter/.gitignore teleporter/index.html teleporter/package.json teleporter/public/gltf/environmentDesk/environmentDesk.gltf teleporter/public/gltf/plantSansevieria/plantSansevieria.gltf teleporter/public/gltf/robot/robot.gltf teleporter/public/splats/demo.spz teleporter/README.md teleporter/src/index.ts teleporter/src/panel.ts teleporter/src/robot.ts teleporter/tsconfig.json teleporter/ui/welcome.uikitml teleporter/vite.config.ts world-models/.gitignore world-models/create_world.py world-models/requirements.txt world-models/world_server.py ``` ### Dependencies - old-teleporter/package.json: @mkkellogg/gaussian-splats-3d@0.4.7, @types/node@^20, @types/react@^18, @types/react-dom@^18, @types/three@0.182.0, autoprefixer@^10.0.1, next@14.0.4, postcss@^8, react@^18, react-dom@^18, tailwindcss@^3.3.0, three@0.182.0, typescript@^5 - teleporter/package.json: @iwsdk/core@0.2.2, @iwsdk/vite-plugin-gltf-optimizer@0.2.2, @iwsdk/vite-plugin-iwer@0.2.2, @iwsdk/vite-plugin-uikitml@0.2.2, @sparkjsdev/spark@^0.1.10, @types/three@^0.178.1, three@npm:super-three@0.177.0, typescript@^5.5.0, vite@^7.1.4, vite-plugin-mkcert@^1.17.0 - world-models/requirements.txt: flask, python-dotenv@>=1.0.0, requests@>=2.28.0 ### Recent commits (newest first) - fix - del - ignore - i got motion - cookery - Drone Footage Extractor - Adding the web portal thats viewed in the Meta Quest - World Labs API + Web Server ## Key source files (fetched from GitHub, selected and truncated for size) ### portal/docs/SETUP_STEP_BY_STEP.md ```markdown # Worldly – Step-by-step setup (from zero to App Lab) Follow these steps in order. If something doesn't match your screen (e.g. different Unity version), use the closest option and then we can adjust. --- ## Test first (no Unity required) You can confirm the flow and that links work before installing Unity: 1. **Web launcher (see what it looks like)** Open **`test/web-launcher.html`** in your browser (double-click the file or drag it into Chrome/Safari/Firefox). You'll see a simple list of worlds; click one and the Marble URL opens in a new tab. If that page loads and you can explore the world there, the same link will work from the Quest app (it will open in Oculus Browser). 2. **Check your world list** From the project folder run: ```bash node test/check-worlds.js ``` This checks that `unity/Assets/StreamingAssets/Worlds.json` is valid and that each Marble URL is reachable. Use a real Marble world URL in `Worlds.json` to test; the sample uses a placeholder. To open the first world in your browser after the check, run: ```bash node test/check-worlds.js --open ``` --- ## Part 1: Install Unity and create the project ### Step 1.1 – Install Unity Hub and Unity 1. Go to [https://unity.com/download](https://unity.com/download) and download **Unity Hub**. 2. Install and open Unity Hub. Sign in or create a Unity account. 3. In Hub, go to **Installs** → **Install Editor**. Pick a version like **2022.3 LTS** or **6000.x** (Unity 6). 4. In the installer, under **Platforms**, enable: - **Android Build Support** - If it asks, also install **Android SDK & NDK** and **OpenJDK**. - You don't need an Android phone—Quest runs on Android, so Unity is building an app for the headset, not for a phone. 5. Finish the install. ### Step 1.2 – Create a new project 1. In Unity Hub, click **New project**. 2. Choose a template: **Core** (empty 3D) or **XR** if you see it. 3. Set **Project name** to something like `Worldly`, pick a folder, then **Create project**. 4. Wait for Unity to open. --- ## Part 2: Put the Worldly scripts in your project ### Step 2.1 – Copy the scripts 1. On your computer, open the **worldly** folder (where this repo lives). 2. Go into `unity/Assets/Scripts/`. You should see: - `WorldEntry.cs` - `WorldListConfig.cs` - `WorldLauncher.cs` 3. In Unity, in the **Project** window, go to `Assets`. If there's no `Scripts` folder, right‑click in `Assets` → **Create** → **Folder** → name it `Scripts`. 4. Copy the **three .cs files** from `worldly/unity/Assets/Scripts/` into your project's `Assets/Scripts/` folder (drag and drop, or copy/paste in Finder/Explorer so they end up inside `Assets/Scripts/`). 5. Back in Unity, wait for it to compile (no errors at the bottom). If you see red errors, tell me the exact message. ### Step 2.2 – Copy the world list (optional but useful) 1. In the worldly repo, open `unity/Assets/StreamingAssets/`. 2. In your Unity project, under `Assets`, create a fold [truncated — 11521 more characters] ``` ### world-models/requirements.txt ``` requests>=2.28.0 flask python-dotenv>=1.0.0 ``` ### old-teleporter/package.json ``` { "name": "webviewer", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@mkkellogg/gaussian-splats-3d": "0.4.7", "@types/three": "0.182.0", "next": "14.0.4", "react": "^18", "react-dom": "^18", "three": "0.182.0" }, "devDependencies": { "typescript": "^5", "@types/node": "^20", "@types/react": "^18", "@types/react-dom": "^18", "autoprefixer": "^10.0.1", "postcss": "^8", "tailwindcss": "^3.3.0" } } ``` ### teleporter/package.json ``` { "name": "webviewer2", "version": "0.1.0", "private": true, "type": "module", "scripts": { "dev": "vite", "build": "vite build", "preview": "vite preview" }, "dependencies": { "@iwsdk/core": "0.2.2", "@sparkjsdev/spark": "^0.1.10", "three": "npm:super-three@0.177.0" }, "devDependencies": { "@iwsdk/vite-plugin-gltf-optimizer": "0.2.2", "@iwsdk/vite-plugin-iwer": "0.2.2", "@iwsdk/vite-plugin-uikitml": "0.2.2", "@types/three": "^0.178.1", "typescript": "^5.5.0", "vite": "^7.1.4", "vite-plugin-mkcert": "^1.17.0" }, "engines": { "node": ">=20.19.0" } } ``` ### old-teleporter/app/layout.tsx ```typescript import type { Metadata } from 'next' import './globals.css' export const metadata: Metadata = { title: 'SPZ WebXR Viewer', description: 'Quest-friendly WebXR viewer for SPZ gaussian splat files.', } export default function RootLayout({ children, }: { children: React.ReactNode }) { return ( {children} ) } ``` ### old-teleporter/app/page.tsx ```typescript const SAMPLE = '/splats/ceramic-500k-demo.spz' export default function Home() { return (

TreeHacks Webviewer

SPZ WebXR Viewer

Launch the Quest-friendly WebVR/WebXR viewer and load any `.spz` scene. The viewer page includes URL loading, local file loading, and immersive VR entry support.

Open Viewer Open Sample Scene
) } ``` ### teleporter/src/index.ts ```typescript import * as THREE from "three"; import { SparkControls, SplatFileType, SplatMesh, VRButton } from "@sparkjsdev/spark"; const DEFAULT_SPZ_URL = "./splats/demo.spz"; const TARGET_SCENE_MAX_DIM_METERS = 900; const HUMAN_EYE_HEIGHT_METERS = 1.7; const query = new URLSearchParams(window.location.search); const spzUrl = query.get("spz") || DEFAULT_SPZ_URL; const scene = new THREE.Scene(); scene.background = new THREE.Color("#0b0d11"); const camera = new THREE.PerspectiveCamera( 70, window.innerWidth / window.innerHeight, 0.01, 1000, ); const cameraRig = new THREE.Group(); cameraRig.position.set(0, 1.6, 2.5); cameraRig.add(camera); scene.add(cameraRig); const renderer = new THREE.WebGLRenderer({ antialias: true, powerPreference: "high-performance" }); renderer.setPixelRatio(Math.min(window.devicePixelRatio, 2)); renderer.setSize(window.innerWidth, window.innerHeight); renderer.xr.enabled = true; renderer.xr.setReferenceSpaceType("local-floor"); document.body.style.margin = "0"; document.body.style.overflow = "hidden"; document.body.appendChild(renderer.domElement); const controls = new SparkControls({ canvas: renderer.domElement }); controls.fpsMovement.xr = renderer.xr; controls.fpsMovement.moveSpeed = 1.8; controls.fpsMovement.shiftMultiplier = 2.5; const vrButton = VRButton.createButton(renderer, { requiredFeatures: ["local-floor"], optionalFeatures: ["bounded-floor", "hand-tracking", "layers"], }); if (vrButton) { document.body.appendChild(vrButton); } const help = document.createElement("div"); help.style.position = "fixed"; help.style.top = "16px"; help.style.left = "16px"; help.style.maxWidth = "min(560px, 85vw)"; help.style.padding = "10px 12px"; help.style.borderRadius = "10px"; help.style.background = "rgba(6, 8, 12, 0.75)"; help.style.backdropFilter = "blur(6px)"; help.style.color = "#e6edf3"; help.style.fontFamily = "ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, Liberation Mono, monospace"; help.style.fontSize = "12px"; help.style.lineHeight = "1.35"; help.style.zIndex = "20"; document.body.appendChild(help); const filePicker = document.createElement("input"); filePicker.type = "file"; filePicker.accept = ".spz,.ply,.splat,.ksplat"; filePicker.style.position = "fixed"; filePicker.style.top = "16px"; filePicker.style.right = "16px"; filePicker.style.zIndex = "20"; filePicker.style.maxWidth = "40vw"; document.body.appendChild(filePicker); const grid = new THREE.GridHelper(40, 40, 0x27405b, 0x13202e); grid.position.y = -0.01; scene.add(grid); let activeSplat: SplatMesh | null = null; let activeSplatRoot: THREE.Group | null = null; function inferFileType(fileName: string): SplatFileType | undefined { const lower = fileName.toLowerCase(); if (lower.endsWith(".spz")) return SplatFileType.SPZ; if (lower.endsWith(".ply")) return SplatFileType.PLY; if (lower.endsWith(".splat")) return SplatFileType.SPLAT; if (lower.endsWith(".ksplat")) return SplatFileType.KSPLAT; return undefined; } function updateHelp(message: string, source: string): void { const inXR = renderer.xr.isPresenting ? "yes" : "no"; help.innerHTML = [ `
SPZ WebVR Viewer (Meta Quest target)
`, `
Source: ${source}
`, `
Status: ${message}
`, `
Desktop: drag (rotate), right-drag/two-finger (slide), wheel (move), WASD.
`, `
XR: Enter VR button, then use thumbsticks to rotate + locomote.
`, `
XR active: ${inXR}
`, ].join(""); } function recenterView(mesh: SplatMesh, root: THREE.Group): void { const bounds = mesh.getBoundingBox(true); if (bounds.isEmpty()) { return; } const center = bounds.getCenter(new THREE.Vector3()); const size = bounds.getSize(new THREE.Vector3()); const maxDim = Math.max(size.x, size.y, size.z, 0.01); const fitScale = THREE.MathUtils.clamp( TARGET_SCENE_MAX_DIM_METERS / maxDim, 0.2, 300, ); // SPZ captures are commonly upside down in this scene orientation. root.rotation.set(Math.PI, 0, 0); root.scale.setScalar(fitScale); mesh.position.set(-center.x, -center.y, -center.z); const fittedMaxDim = maxDim * fitScale; const fittedHeight = size.y * fitScale; const distance = THREE.MathUtils.clamp(fittedMaxDim * 0.12, 1.2, 12); const floorY = -fittedHeight * 0.5; const eyeY = floorY + HUMAN_EYE_HEIGHT_METERS; cameraRig.position.set(0, eyeY, distance); cameraRig.lookAt(0, eyeY, 0); } async function loadSplatFromUrl(url: string): Promise { updateHelp("Loading splat...", url); if (activeSplat) { if (activeSplatRoot) { scene.remove(activeSplatRoot); activeSplatRoot = null; } else { scene.remove(activeSplat); } activeSplat.dispose(); activeSplat = null; } const mesh = new SplatMesh({ url, onLoad: () => { updateHelp("Loaded", url); }, }); const root = new THREE.Group(); root.add(mesh); scene.add(root); activeSplat = mesh; activeSplatRoot = root; await mesh.initialized; recenterView(mesh, root); updateHelp("Loaded and centered", url); } async function loadSplatFromFile(file: File): Promise { updateHelp("Reading local file...", file.name); if (activeSplat) { if (activeSplatRoot) { scene.remove(activeSplatRoot); activeSplatRoot = null; } else { scene.remove(activeSplat); } activeSplat.dispose(); activeSplat = null; } const fileBytes = new Uint8Array(await file.arrayBuffer()); const mesh = new SplatMesh({ fileName: file.name, fileType: inferFileType(file.name), fileBytes, onLoad: () => { updateHelp("Loaded", file.name); }, }); const root = new THREE.Group(); root.add(mesh); scene.add(root); activeSplat = mesh; activeSplatRoot = root; await mesh.initialized; recenterView(mesh, root); updateHelp("Loaded and centered", file.name); } filePicker.addEventListener("change", async () => { const file = filePicker.files?.[0]; if (!file) [truncated — 1073 more characters] ``` ### old-teleporter/app/viewer/page.tsx ```typescript import ViewerClient from './viewer-client' export default function ViewerPage() { return } ``` ### old-teleporter/postcss.config.js ```javascript module.exports = { plugins: { tailwindcss: {}, autoprefixer: {}, }, } ``` ### old-teleporter/next.config.js ```javascript /** @type {import('next').NextConfig} */ const nextConfig = {} module.exports = nextConfig ``` [21 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]