# Project export: Snap!

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: TreeHacks 2024
- Tagline: Your iPhone Live Photos but 100x better (made 3D with computer vision)
- Devpost: https://devpost.com/software/snap-mhexb1
- GitHub: https://github.com/chloechiaw/Snap
- Video: https://www.youtube.com/embed/CV9OTSPPOEE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 0 GitHub contributor(s) — 

## Devpost submission (written by the team)

### Inspiration

Snap! Is like your iPhone live photos (where the picture shows movement before and after slightly) but 100x more memorable. If you’ve ever taken a photo of a fun event like a concert, kickback, or house party but felt like it couldn’t exactly capture the “vibe”, this is probably for you. By turning 2d photos into 3d experiences with pretty low latency (~10 seconds automated), it feels like you’re in the room again. Good times == good memories, so why not start preserving the vibes in between :)

### What it does

Snap! applies depth mapping with computer vision to your photos to artificially create a 3D immersive scene. Then, that photo is rendered with ThreeJS in a 3D environment, so you can drag around in 180 degree angles and see what’s going on as if you were reliving that moment. This prototype is completely functional.

### How we built it

1) Deciding on the photo input This prototype presented a ton of unexpected, interesting technical challenges around cameras, optics, depth estimation, and 3d software like ThreeJS. Simulating a 3D camera with computer vision (depth estimation) and handling a completely different data type than ThreeJS typically uses for displacement maps was more hacky and required more workarounds. I had to first figure out how to use an iPhone to mimic a 3D camera (such as Reto 3D that takes pictures at various angles and then combines those pictures into a “moving GIF”) as well as stereo cameras, which is known for mimicking how a human perceives the world (with 2 cameras instead of 2 eyes). Traditional methods for stereo cameras work by taking two or more images and estimating a 3D model of the scene. This is done by finding matching pixels in the images and converting their 2D positions into 3D depths. But this traditional method requires special lenses with expensive equipment. Using an iPhone camera lens to have the same 3D effect is challenging, because it’s purely a 2d image. However, the data format I decided on was a .mov GIF created using the “Bounce effect” for Live Photos. It captures ~1-2 seconds before and after a picture was taken. It’s also the most functional for most, since handling stereo and 3d cameras along with the development process after could be complex. Afterwards, I used a CLI tool, ffmpeg, to individually extract every frame (around 10-20 frames) for the .mov file. 2) Depth Estimation on the photos Depth estimation is the technical meat of this hack as it provides the immersive effect. It quantifies spatial relationships within pixels, enabling the reconstruction of images into detailed 3D models by interpreting and mapping depth cues. I ended up applying off-the-shelf model weights from the paper Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. Foundation Model for Monocular Depth Estimation. I decided on using this model due to it achieving high benchmark results, and being trained on 6M+ unlabelled and labeled images. PyTorch and OpenCV were used for applying the model’s weighs onto my images folder (pure inference, no fine-tuning). The output was a concatenated image of the raw image and its depth map transformed into an image data format. The depth map is a 2D array 564x564 that has values ranging from 0-255 indicating the closeness of the pixel to the camera. 3) Layering the Depth Map onto the 2D Image in ThreeJS This was the tricky part where I had to come up with a work-around solution since ThreeJS doesn’t typically accept depth maps as displacement maps, so it required a bit of learning about 3D in ThreeJS. In Scene.JS, I essentially loaded the 2D image (½ of the inputted concatenated image) as a Canvas Texture and the depth map as a Canvas Texture as well (2/2 of the inputted concatenated image). Then, I loaded each corresponding depth map as a displacement map using ThreejS’s MeshStandardMaterial. As for the plane that the whole 3D environment was living on, a 10 x 10 x 512 x 512 plane was created with the mesh including the plane and the material (MeshStandardMaterial). OrbitControls from ThreeJS were also loaded to support camera movement (that way, you can explore the image with your mouse as if you were in the scene). For aesthetics, I loaded ambient light settings. 4) Animating the frames By this time, I had successfully outputted a singular 3d photo in ThreeJS. However, I wanted to see if I could make a simple animation loop that would apply depth maps onto its corresponding photo and render it quickly enough so it would look like it was a moving scene. This was completed by creating a custom animation function that would calculate the elapsed time and then determine when to re-render the next frame. Eventually, I landed on specific frame settings that mimicked a GIF, as the photos were moving quickly enough to simulate movement.

### Challenges we ran into

Originally, I applied the depth map as a separate layer from the image canvas texture. This was a bit tricky to troubleshoot because there was essentially zero output from the rendered ThreejS environment. Closely reading the ThreeJS docs helped, and eventually I found the custom function THREE.MeshStandardMaterial that led to the displacement map displaying. Figuring out whether I would need some sort of calibration for the moving GIF portion was also a challenge. I originally thought of a more complex solution: individually aligning similar points of the same edges across multiple frames in order to account for images where there was more movement occurring. I was worried that the depth maps would be misaligned across various frames. However, aligning the points was not needed since the frames were taken granularly enough that there didn’t seem to be too much diversion from the pin point.

### Accomplishments we're proud of

Rendering the GIF onto ThreeJS! I originally thought it was only feasible to render an image in the environment, but through a workaround solution using a custom animation function, I was able to render 10-20 frames fast enough to achieve the “movement” effect. Lots of workarounds in this prototype (previous depth map as a displacement map mentioned earlier).

### What we learned

Cameras are difficult to simulate on software! ThreeJS geometry, meshes, and materials were a bit abstract to grasp at first, as it was hard to figure out what would be the equivalent formats for this purpose. Eventually, I decided to have the 2d Image be a texture and then a material, and then layering the depth map as a displacement map through that.

### What's next

NerFS instead of depth estimation for turning 2D photos into accurate 3D scenes! It’d be cool to compare this classical CV depth estimation technique with a more recent finding. However, NeRFs take a lot of compute as it needs to perform inference on every x,y,z point. It’d be great to have an NVIDIA RTX 4080 GPU to do that ;)

## README (from the GitHub repository)

# Getting Started with Create React App

This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app).

## Available Scripts

In the project directory, you can run:

### `npm start`

Runs the app in the development mode.\
Open [http://localhost:3000](http://localhost:3000) to view it in your browser.

The page will reload when you make changes.\
You may also see any lint errors in the console.

### `npm test`

Launches the test runner in the interactive watch mode.\
See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.

### `npm run build`

Builds the app for production to the `build` folder.\
It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.\
Your app is ready to be deployed!

See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for more information.

### `npm run eject`

**Note: this is a one-way operation. Once you `eject`, you can't go back!**

If you aren't satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project.

Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point you're on your own.

You don't have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldn't feel obligated to use this feature. However we understand that this tool wouldn't be useful if you couldn't customize it when you are ready for it.

## Learn More

You can learn more in the [Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started).

To learn React, check out the [React documentation](https://reactjs.org/).

### Code Splitting

This section has moved here: [https://facebook.github.io/create-react-app/docs/code-splitting](https://facebook.github.io/create-react-app/docs/code-splitting)

### Analyzing the Bundle Size

This section has moved here: [https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size](https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size)

### Making a Progressive Web App

This section has moved here: [https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app](https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app)

### Advanced Configuration

This section has moved here: [https://facebook.github.io/create-react-app/docs/advanced-configuration](https://facebook.github.io/create-react-app/docs/advanced-configuration)

### Deployment

This section has moved here: [https://facebook.github.io/create-react-app/docs/deployment](https://facebook.github.io/create-react-app/docs/deployment)

### `npm run build` fails to minify

This section has moved here: [https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify](https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify)
# 3dscene


## Detected evidence (automated analysis)

Indexed codebase: 10 recognized source files, 12 KB.
- CSS (language) — detected in the code
- HTML (language) — detected in the code
- JavaScript (language) — detected in the code
- React (technology) — detected in the code
- Python (language) — claimed on Devpost, not found in the code
- PyTorch (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (14 of 14)

```
.gitignore
package.json
public/index.html
public/manifest.json
public/robots.txt
README.md
src/App.css
src/App.js
src/App.test.js
src/index.css
src/index.js
src/reportWebVitals.js
src/Scene.js
src/setupTests.js
```

### Dependencies

- package.json: @testing-library/jest-dom@^5.17.0, @testing-library/react@^13.4.0, @testing-library/user-event@^13.5.0, react@^18.2.0, react-dom@^18.2.0, react-scripts@5.0.1, three@^0.161.0, web-vitals@^2.1.4

### Recent commits (newest first)

- cooked.
- immersive gif made
- lol i think it works
- base works
- first commit
- Initialize project using Create React App

## Key source files (fetched from GitHub, selected and truncated for size)

### package.json

```
{
  "name": "scene",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "@testing-library/jest-dom": "^5.17.0",
    "@testing-library/react": "^13.4.0",
    "@testing-library/user-event": "^13.5.0",
    "react": "^18.2.0",
    "react-dom": "^18.2.0",
    "react-scripts": "5.0.1",
    "three": "^0.161.0",
    "web-vitals": "^2.1.4"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": [
      "react-app",
      "react-app/jest"
    ]
  },
  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      "last 1 chrome version",
      "last 1 firefox version",
      "last 1 safari version"
    ]
  }
}

```

### src/App.js

```javascript
import "./App.css";
import Scene from "./Scene";
import image from "/Users/chloechia/sceneproject/scene/src/cal.jpeg";

function App() {
  return (
    <div>
      <Scene
      />
    </div>
  );
}

export default App;

```

### src/index.js

```javascript
import React from 'react';
import ReactDOM from 'react-dom/client';
import './index.css';
import App from './App';
import reportWebVitals from './reportWebVitals';

const root = ReactDOM.createRoot(document.getElementById('root'));
root.render(
  <React.StrictMode>
    <App />
  </React.StrictMode>
);

// If you want to start measuring performance in your app, pass a function
// to log results (for example: reportWebVitals(console.log))
// or send to an analytics endpoint. Learn more: https://bit.ly/CRA-vitals
reportWebVitals();

```

### src/setupTests.js

```javascript
// jest-dom adds custom jest matchers for asserting on DOM nodes.
// allows you to do things like:
// expect(element).toHaveTextContent(/react/i)
// learn more: https://github.com/testing-library/jest-dom
import '@testing-library/jest-dom';

```

### src/App.test.js

```javascript
import { render, screen } from '@testing-library/react';
import App from './App';

test('renders learn react link', () => {
  render(<App />);
  const linkElement = screen.getByText(/learn react/i);
  expect(linkElement).toBeInTheDocument();
});

```

### src/reportWebVitals.js

```javascript
const reportWebVitals = onPerfEntry => {
  if (onPerfEntry && onPerfEntry instanceof Function) {
    import('web-vitals').then(({ getCLS, getFID, getFCP, getLCP, getTTFB }) => {
      getCLS(onPerfEntry);
      getFID(onPerfEntry);
      getFCP(onPerfEntry);
      getLCP(onPerfEntry);
      getTTFB(onPerfEntry);
    });
  }
};

export default reportWebVitals;

```

### src/index.css

```css
body {
  margin: 0;
  font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen',
    'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',
    sans-serif;
  -webkit-font-smoothing: antialiased;
  -moz-osx-font-smoothing: grayscale;
}

code {
  font-family: source-code-pro, Menlo, Monaco, Consolas, 'Courier New',
    monospace;
}

```

### src/App.css

```css
.App {
  text-align: center;
}

.App-logo {
  height: 40vmin;
  pointer-events: none;
}

@media (prefers-reduced-motion: no-preference) {
  .App-logo {
    animation: App-logo-spin infinite 20s linear;
  }
}

.App-header {
  background-color: #282c34;
  min-height: 100vh;
  display: flex;
  flex-direction: column;
  align-items: center;
  justify-content: center;
  font-size: calc(10px + 2vmin);
  color: white;
}

.App-link {
  color: #61dafb;
}

@keyframes App-logo-spin {
  from {
    transform: rotate(0deg);
  }
  to {
    transform: rotate(360deg);
  }
}

```

### public/index.html

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="theme-color" content="#000000" />
    <meta
      name="description"
      content="Web site created using create-react-app"
    />
    <link rel="apple-touch-icon" href="%PUBLIC_URL%/logo192.png" />
    <!--
      manifest.json provides metadata used when your web app is installed on a
      user's mobile device or desktop. See https://developers.google.com/web/fundamentals/web-app-manifest/
    -->
    <link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
    <!--
      Notice the use of %PUBLIC_URL% in the tags above.
      It will be replaced with the URL of the `public` folder during the build.
      Only files inside the `public` folder can be referenced from the HTML.

      Unlike "/favicon.ico" or "favicon.ico", "%PUBLIC_URL%/favicon.ico" will
      work correctly both with client-side routing and a non-root public URL.
      Learn how to configure a non-root public URL by running `npm run build`.
    -->
    <title>React App</title>
  </head>
  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
    <!--
      This HTML file is a template.
      If you open it directly in the browser, you will see an empty page.

      You can add webfonts, meta tags, or analytics to this file.
      The build step will place the bundled scripts into the <body> tag.

      To begin the development, run `npm start` or `yarn start`.
      To create a production bundle, use `npm run build` or `yarn build`.
    -->
  </body>
</html>

```

### src/Scene.js

```javascript
import React, { useEffect, useRef } from "react";
import { OrbitControls } from "three/examples/jsm/controls/OrbitControls.js";
import { GUI } from "three/examples/jsm/libs/lil-gui.module.min.js";
import * as THREE from "three";

const Scene = () => {
  const renderer = new THREE.WebGLRenderer({ antialias: true });
  const scene = useRef(new THREE.Scene());
  const mesh = useRef(null);
  const camera = useRef(null);

  const startTime = useRef(null);
  const currentFrameIndex = useRef(-1); // Start with -1 to ensure the first frame is loaded at start

  useEffect(() => {
    camera.current = new THREE.PerspectiveCamera(
      50,
      window.innerWidth / window.innerHeight,
      1,
      950
    );
    camera.current.position.z = 2;
    renderer.setSize(window.innerWidth, window.innerHeight);
    document.body.appendChild(renderer.domElement);

    new OrbitControls(camera.current, renderer.domElement);
    scene.current.add(new THREE.AmbientLight(0xffffff, 0.5));

    renderer.setAnimationLoop(animation);

    window.addEventListener("resize", onResize);
    return () => {
      window.removeEventListener("resize", onResize);
      document.body.removeChild(renderer.domElement);
    };
  }, []);

  const onResize = () => {
    camera.current.aspect = window.innerWidth / window.innerHeight;
    camera.current.updateProjectionMatrix();
    renderer.setSize(window.innerWidth, window.innerHeight);
  };

  const generateFilePath = (number) => {
    const basePath = `${process.env.PUBLIC_URL}/depthmaps2`;
    const fileName = `output_${number
      .toString()
      .padStart(5, "0")}_img_depth.png`;
    // console.log(`${basePath}/${fileName}`);
    return `${basePath}/${fileName}`;
  };

  const loadNewImage = (imageNumber) => {
    const imagePath = generateFilePath(imageNumber);
    console.log(imagePath);
    const image = new Image();
    image.onload = () => {
      console.log("hit");
      createTexturesAndApply(image);
    };
    image.src = imagePath;
  };

  const createTexturesAndApply = (image) => {
    const visualCanvas = document.createElement("canvas");
    const visualCtx = visualCanvas.getContext("2d");
    visualCanvas.width = image.width / 2; // Use only half the image width
    visualCanvas.height = image.height;
    visualCtx.drawImage(
      image,
      0,
      0,
      image.width / 2,
      image.height,
      0,
      0,
      image.width / 2,
      image.height
    );
    const visualTexture = new THREE.CanvasTexture(visualCanvas);

    const depthCanvas = document.createElement("canvas");
    const depthCtx = depthCanvas.getContext("2d");
    depthCanvas.width = image.width / 2; // Use only half the image width for the depth map as well
    depthCanvas.height = image.height;
    depthCtx.drawImage(
      image,
      image.width / 2,
      0,
      image.width / 2,
      image.height,
      0,
      0,
      image.width / 2,
      image.height
    );
    const depthTexture = new THREE.CanvasTexture(depthCanvas);

    //   // Apply the textures to the material of the mesh
    if (mesh.current) {
      mesh.current.material.map = visualTexture;
      mesh.current.material.displacementMap = depthTexture;
      mesh.current.material.needsUpdate = true;
    } else {
      // If the mesh doesn't exist yet, create it
      const material = new THREE.MeshStandardMaterial({
        map: visualTexture,
        displacementMap: depthTexture,
        // Include other material properties as needed
      });
      const geometry = new THREE.PlaneGeometry(10, 10, 512, 512); // Adjust size as needed
      mesh.current = new THREE.Mesh(geometry, material);
      scene.current.add(mesh.current);
    }
  };

  const animation = (time) => {
    if (!startTime.current) startTime.current = time;
    const elapsedTime = time - startTime.current;
    const frameDuration = 1000 / 15; // Duration per frame in milliseconds
    const totalNumberOfImages = 9;

    const targetFrame =
      (Math.floor(elapsedTime / frameDuration) % totalNumberOfImages) + 1;

    if (targetFrame !== currentFrameIndex.current) {
      currentFrameIndex.current = targetFrame;
      loadNewImage(currentFrameIndex.current);
    }

    renderer.render(scene.current, camera.current);
  };

  //   return <div />;
};

export default Scene;

```