# Project export: Nab 3D
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: TreeHacks 2024
- Tagline: Nab any object straight out of the real world and into your browser. Video to 3D web component -- all in one line of code.
- Devpost: https://devpost.com/software/nab-3d
- GitHub: https://github.com/soapantelope/nab
- Video: https://www.youtube.com/embed/-RIqBdwPtAk?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Result: winner (Best Beginner Hack (4x Razer Keyboard + Mouse))
- Team: 3 GitHub contributor(s) — Sophia Zhang (15 commits), nat hill (3 commits), Amina Rakhimbergenova (1 commits)
## Devpost submission (written by the team)
### Inspiration
Nab streamlines the process of bringing real-world objects into the digital sphere by utilizing video transformations, photogrammetry, providing a custom API to host and display them. Sophia is well acquainted with photogrammetry, having used it to create models of locations around Seattle. But the process is nothing short of terrible: Take 30 - 250 images of your object. Manually find and run a good photogrammetry model (or spend 10-20 hours 3D modeling your object on Blender) Process and export .glb Edit model in Blender and re-export Import .glb to project and set up a scene with Three.js (which usually takes at least 20+ lines of complicated, buggy 3js code) And, you have to repeat the entire process all over again for each model you want to add! Functionality Nab 3D, however, introduces a web component that does all of this in one line of code— in a matter of seconds. It revolutionizes this intricate process: It's as easy as... 1. Input a video of the object. 2. Paste our web component URL in your website. With Nab 3D The user does not have to touch a single line of Three.js or deal with any aspect of the photogrammetry/modeling process! How did we build Nab 3D? Filled with excitement for the photogrammetry process, we dove headfirst into development. The idea of creating 3D models from videos was incredibly captivating. Yet, as we immersed ourselves in our project, we found ourselves unsure of exactly how to realize the idea. So, midway through the hackathon, we decided to pause and reassess our value proposition. With the growing importance of 3D modeling in e-commerce, our mission became clear: make 3D modeling accessible. We want Nab 3D to truly stand out as a user-friendly solution in the world of 3D modeling. We used Bun, React, Vite, and Three.js for frontend, Cloudflare for the database, Apple Object Capture API/Swift for backend, and FFmpeg for video parsing. Challenges faced We spent the first half of the hackathon building our app around Autodesk’s Reality Capture API and fighting authorization bug after authorization bug, only to realize that you can't use their API until you contact their associates — weekdays only! We struggled to find another photogrammetry API and ended up using Apple’s Object Capture API with Swift— a language none of us had developed in before. Although each and every part of our pipeline had its own set of roadblocks, almost to the point of pivoting, we think sticking with the idea paid off. What We Learned Besides learning classic soft skills like ideation and resilience, we also had an opportunity to learn completely new skills, frameworks, and even languages. In fact, two of us were beginners in any kind of web dev/JavaScript, one of us had never touched APIs and databases before, and all of us had never worked in Swift and Bun before (even though a large part of our app ended up being written in Swift!). Everyone on the team is coming out of this hackathon with not only an entirely new stack but also new friends and hilarious sleep-deprived memories. Our team members have spent countless hours ripping our hair out trying to bring 3D scans of objects into the web. For something this relevant to the future of e-commerce, advertising, and media (especially considering the rise of AR), it was surprising how slow and laborious the process was. We realized that the process did not need to be this painful – in fact, it could be simplified down to one line of code, served by an API with the help of photogrammetry.
### What's next
? 3D e-commerce, advertising, & media are coming – and Nab's goal is to make that world accessible to everyone. Furthermore, photogrammetry lends itself to the demand for an easy-to-use scan to 3D asset pipeline, which will grow as AR/VR becomes a dominant technology and focus for advertising. The global 3D e-commerce market size was USD 11.11 Billion in 2023, with a projected 40.0% compound annual growth rate in 2024-2030. We plan to further develop this idea, with anything as big as a foray into AR/VR or as specific as potential implementations in food marketplaces like Doordash, or small businesses on Etsy. Thanks for reading!
## README (from the GitHub repository)
# Nab 3D
A way to "nab"any object straight out of the real world and into your website, with just a short video and a single line of code:
A photogrammetry tool that allows web developers to create 3D models of any real world object and add them to their websites using a short video scan and a single line of HTML.
• Awarded ‘Best Beginner Hack’ at TreeHacks 2024, the largest collegiate hackathon in the United States.
• Built using React, JavaScript, Vite, Bun, and Swift.
## Detected evidence (automated analysis)
Indexed codebase: 11 recognized source files, 21 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- React (technology) — detected in the code
- Swift (language) — claimed on Devpost, not found in the code
## Codebase structure (from repository index)
### Files (26 of 26)
```
.eslintrc.cjs
.gitattributes
.gitignore
bun.lockb
index.html
package.json
README.md
src/App.css
src/App.jsx
src/discarded.txt
src/index.css
src/main.jsx
src/Nab.jsx
src/poly.glb
src/Uploader.jsx
tempindex.html
video-to-model-src/ffmpeg
video-to-model-src/HelloPhotogrammetry
video-to-model-src/outputdata/bag_vid_2.usdz
video-to-model-src/outputdata/bag_vid_4fps.usdz
video-to-model-src/outputdata/bag_vid.usdz
video-to-model-src/outputdata/bag.usdz
video-to-model-src/outputdata/sparkling.usdz
video-to-model-src/outputdata/tea_vid.usdz
video-to-model-src/README.md
vite.config.js
```
### Dependencies
- package.json: @react-three/drei@^9.97.6, @react-three/fiber@^8.15.16, @types/react@^18.2.55, @types/react-dom@^18.2.19, @types/three@^0.161.2, @vitejs/plugin-react@^4.2.1, eslint@^8.56.0, eslint-plugin-react@^7.33.2, eslint-plugin-react-hooks@^4.6.0, eslint-plugin-react-refresh@^0.4.5, ffmpeg@^0.0.4, react@^18.2.0, react-dom@^18.2.0, three@^0.161.0, vite@^5.1.0
### Recent commits (newest first)
- Update README.md
- Update README.md
- Update README.md
- Update README.md
- Autodesk disclaimer
- Merge pull request #1 from soapantelope/WASM-filesys
- WASM branch stuff
- wasm wip
- Update Nab
- videomodel upload
- Tiny changes
- Small changes
- Scrap Autodesk code, got 3js model displayed
- More merge conflicts
- Merge Conflicts
- All Autodesk API functions implemented
- render3d not working
- Merged frontend
- frontend
- Comments explaining bug
## Key source files (fetched from GitHub, selected and truncated for size)
### package.json
```
{
"name": "my-app",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "bunx --bun vite",
"build": "vite build",
"lint": "eslint . --ext js,jsx --report-unused-disable-directives --max-warnings 0",
"preview": "vite preview"
},
"dependencies": {
"@react-three/drei": "^9.97.6",
"@react-three/fiber": "^8.15.16",
"@types/three": "^0.161.2",
"ffmpeg": "^0.0.4",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"three": "^0.161.0"
},
"devDependencies": {
"@types/react": "^18.2.55",
"@types/react-dom": "^18.2.19",
"@vitejs/plugin-react": "^4.2.1",
"eslint": "^8.56.0",
"eslint-plugin-react": "^7.33.2",
"eslint-plugin-react-hooks": "^4.6.0",
"eslint-plugin-react-refresh": "^0.4.5",
"vite": "^5.1.0"
}
}
```
### src/main.jsx
```javascript
import React from "react";
import ReactDOM from "react-dom/client";
import App from "./App.jsx";
import "./index.css";
ReactDOM.createRoot(document.getElementById("root")).render();
```
### src/App.jsx
```javascript
import { useState, useEffect, useRef, Suspense } from "react";
import "./App.css";
import Nab from "./Nab.jsx";
import Intro from "./intro.mp4";
// import FileUploadComponent from "./Uploader.jsx";
//
function App() {
// THE FOLLOWING CODE IS COMMENTED OUT BECAUSE IT WAS OUR ORIGINAL ATTEMPT TO USE THE AUTODESK API.
// IT WAS NOT IMPLEMENTED DUE TO AUTODESK NOT VERIFYING OUR ACCOUNT/A BUG ON THEIR END :(
// https://aps.autodesk.com/en/docs/reality-capture/v1/tutorials/create-3d-mesh-from-photos/
// const isMounted = useRef(false);
// async function extractFramesFromVideo(videoUrl, fps = 2) {
// return new Promise(async (resolve) => {
// // fully download it first (no buffering):
// let videoBlob = await fetch(url).then((r) => r.blob());
// let videoObjectUrl = URL.createObjectURL(videoBlob);
// let video = document.createElement("video");
// let seekResolve;
// video.addEventListener("seeked", async function () {
// if (seekResolve) seekResolve();
// });
// video.src = videoObjectUrl;
// // workaround chromium metadata bug (https://stackoverflow.com/q/38062864/993683)
// while (
// (video.duration === Infinity || isNaN(video.duration)) &&
// video.readyState < 2
// ) {
// await new Promise((r) => setTimeout(r, 1000));
// video.currentTime = 10000000 * Math.random();
// }
// let duration = video.duration;
// let canvas = document.createElement("canvas");
// let context = canvas.getContext("2d");
// let [w, h] = [video.videoWidth, video.videoHeight];
// canvas.width = w;
// canvas.height = h;
// let frames = [];
// let interval = 1 / fps;
// let currentTime = 0;
// while (currentTime < duration) {
// video.currentTime = currentTime;
// await new Promise((r) => (seekResolve = r));
// context.drawImage(video, 0, 0, w, h);
// let base64ImageData = canvas.toDataURL();
// frames.push(base64ImageData);
// currentTime += interval;
// }
// resolve(frames);
// });
// }
// URL.revokeObjectURL(url); // revoke URL to prevent memory leak
// AUTODESK API COULD NOT BE IMPLEMENTED DUE TO AUTHENTICATION ISSUES
// useEffect(() => {
// const createAccessToken = async () => {
// try {
// const response = await fetch(
// "https://developer.api.autodesk.com/authentication/v2/token",
// {
// method: "POST",
// headers: {
// Authorization:
// "Basic " +
// btoa("8ts8GR4FytJuRuc8VzpWU2e9xJvV6yWr:W0EEVIkul2JZlIyg"),
// "Content-Type": "application/x-www-form-urlencoded",
// Accept: "application/json",
// },
// body: new URLSearchParams({
// grant_type: "client_credentials",
// scope: "data:create data:read data:write",
// }),
// }
// );
// if (!response.ok) {
// throw new Error("Failed to create access key");
// }
// const jsonData = await response.json();
// console.log(Date.now());
// return jsonData.access_token;
// } catch (error) {
// console.error("Error creating access key:", error);
// }
// };
// const createPhotoscene = async (token) => {
// try {
// console.log("Bearer " + token);
// const tokenData = JSON.parse(atob(token.split(".")[1])); // Decode token payload
// const tokenExpiration = tokenData.exp * 1000; // Convert expiration time to milliseconds
// console.log(
// tokenExpiration.toString() + ", " + (Date.now() >= tokenExpiration)
// ); // Check if current time is greater than or equal to expiration time???
// const response = await fetch(
// "https://developer.api.autodesk.com/photo-to-3d/v1/photoscene",
// {
// method: "POST",
// headers: {
// // THIS IS NOT WORKING FOR SOME REASON...
// // ERROR 401 (INVALID AUTH TOKEN)
// Authorization: "Bearer " + token,
// "Content-Type": "application/x-www-form-urlencoded",
// },
// body: new URLSearchParams({
// scenename: "testscene",
// format: "rcm,rcs,ortho",
// "metadata_name[0]": "targetcs",
// "metadata_value[0]": "UTM84-32N",
// }),
// }
// );
// if (!response.ok) {
// throw new Error("Failed to create photoscene" + response.statusText);
// }
// const jsonData = await response.json();
// console.log("Photoscene created:", jsonData);
// return jsonData.photosceneid;
// } catch (error) {
// console.error("Error creating photoscene:", error);
// }
// };
// const addImages = async (token, photosceneid, photos) => {
// try {
// const response = await fetch(
// "https://developer.api.autodesk.com/photo-to-3d/v1/file",
// {
// method: "POST",
// headers: {
// Authorization: "Bearer " + token,
// },
// bodyL: new URLSearchParams({
// photosceneid: photosceneid,
// type: "image",
// // loop through photos
// file: "test.jpg",
// }),
// }
// );
// if (!response.ok) {
// throw new Error("Failed to add images");
// }
// const jsonData = await response.json();
// console.log(
[truncated — 5696 more characters]
```
### vite.config.js
```javascript
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";
// https://vitejs.dev/config/
export default defineConfig({
plugins: [react()],
assetsInclude: ["**/*.glb"],
});
```
### index.html
```html
Vite + React
```
### tempindex.html
```html
Nab 3D
// );
// };
// export default FileUploadComponent;
```
### src/Nab.jsx
```javascript
// import { useState, useEffect, useRef } from "react";
// import "./App.css";
// import React from "react";
import { Suspense } from "react";
import { Canvas } from "@react-three/fiber";
// import { Environment } from "@react-three/drei";
import { OrbitControls } from "@react-three/drei";
// import { useGLTF } from "@react-three/drei";
// import Glb from "./poly.glb";
// import { useLoader } from "@react-three/fiber";
// import { GLTFLoader } from "three/examples/jsm/loaders/GLTFLoader";
// export function Model(url) {
// const { nodes, materials } = useGLTF("/poly.glb");
// const gltf = useLoader(GLTFLoader, workingURL);
// return (
// //
//
//
//
// );
// }
// // useGLTF.preload(url);
// function Nab(video) {
// const [url, setURL] = useState("");
// // Send a request to the server to get the url of the 3D model based on the video prop
// // The url will be stored in the url variable
// return (
//