# Project export: Vertext
This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.
## Project metadata
- Hackathon: UC Berkeley AI Hackathon 2025
- Tagline: End to end text or voice to dense 3d reconstruction
- Devpost: https://devpost.com/software/vertext
- GitHub: https://github.com/vbx1404/Vertext
- Team: 2 GitHub contributor(s) — vbx1404 (6 commits), CharlesXu1124 (4 commits)
## Devpost submission (written by the team)
### Inspiration
We were fascinated by the idea of turning imagination into reality—how a simple line of text could evolve into a full 3D environment. Inspired by generative media tools and cinematic AI, we set out to explore how language could become a spatial canvas, combining storytelling with immersive design.
### What it does
Our project transforms natural language prompts into immersive 3D scenes by combining generative video and geometry reconstruction. First, it creates short cinematic videos from text or verbal descriptions using a state-of-the-art video generation model. Then, it converts those videos into detailed 3D meshes using a high-speed transformer-based reconstruction system. The result is a fast, intuitive way to turn ideas into spatial experiences—bridging language, vision, and 3D design Use Cases: Game Development: Import the mesh into Unity, Unreal, or Godot as environmental assets or characters. Artists can sketch in text, then refine the 3D form interactively, Education: Generate 3D visuals of historical or scientific concepts from text prompts (e.g., "a medieval village"). Use in VR/AR learning modules, Architecture Design: Quickly generate visual prototypes of landscapes, interiors, or thematic spaces from descriptive specs, Embodied AI: Train agents in custom environments derived from descriptive tasks. Generate synthetic 3D training data.
### How we built it
Stage 1: We used Veo 2, Google’s latest text-to-video diffusion model, to generate cinematic 8-second clips from natural language prompts in text or verbal form (Groq Whisper API). Veo 2 accepts rich descriptions and camera terms (e.g. “tracking shot at dawn”) and outputs high-fidelity 720p video. It leverages a diffusion model guided by a multimodal transformer, trained on large-scale video-text data. While Google hasn’t disclosed the full parameter count, Veo 2 is part of its Gemini-scale ecosystem, likely operating in the billions of parameters. Stage 2: We passed the generated videos through Fast3R, a high-speed transformer-based model for multi-view 3D reconstruction. Fast3R uses a ViT-L fusion transformer with 24 layers and 1 billion parameters, capable of processing 1,000+ views in a single forward pass. Unlike pairwise methods, Fast3R reconstructs geometry and camera poses jointly in one shot, making it ideal for dense video frame input. We extracted evenly spaced frames from the videos and used Fast3R to generate clean, textured 3D meshes—turning cinematic outputs into immersive environments. Result: A lightweight app that transforms text prompts into realistic 3D mesh geometry and GIF in minutes that can be exported and applied for downstream uses in game design, embodied AI, .
### Challenges we ran into
Syncing video frame output with 3D frame ingestion without quality loss or temporal misalignment. Managing GPU memory and batch size when scaling to hundreds of frames per video. Compressing and serving the final 3D outputs in a way that preserved fidelity while remaining lightweight and fast inference.
### What we learned
We learned how to integrate multi-modal transformer models in a single pipeline—generating video from text, and then reconstructing that video into 3D scenes. We gained experience with high-throughput transformer architectures, 3D mesh processing, and video-to-geometry challenges.
### What's next
This project shows how storytelling, design, and AI can come together to lower the barrier to creating immersive worlds. It’s a step toward democratizing 3D content creation for artists, educators, and developers everywhere. Our next steps are to refine text to video prompt. Inspiration We were fascinated by the idea of turning imagination into reality—how a simple line of text could evolve into a full 3D environment. Inspired by generative media tools and cinematic AI, we set out to explore how language could become a spatial canvas, combining storytelling with immersive design. What it does Our project transforms natural language prompts into immersive 3D scenes by combining generative video and geometry reconstruction. First, it creates short cinematic videos from text or verbal descriptions using a state-of-the-art video generation model. Then, it converts those videos into detailed 3D meshes using a high-speed transformer-based reconstruction system. The result is a fast, intuitive way to turn ideas into spatial experiences—bridging language, vision, and 3D design Use Cases: Game Development - Import the mesh into Unity, Unreal, or Godot as environmental assets or characters. Artists can sketch in text, then refine the 3D form interactively Education - Generate 3D visuals of historical or scientific concepts from text prompts (e.g., "a medieval village"). Use in VR/AR learning modules. Architecture Design - Quickly generate visual prototypes of landscapes, interiors, or thematic spaces from descriptive specs Embodied AI - Train agents in custom environments derived from descriptive tasks. Generate synthetic 3D training data. How we built it Stage 1: We used Veo 2, Google’s latest text-to-video diffusion model, to generate cinematic 8-second clips from natural language prompts in text or verbal form (Groq Whisper API). Veo 2 accepts rich descriptions and camera terms (e.g. “tracking shot at dawn”) and outputs high-fidelity 720p video. It leverages a diffusion model guided by a multimodal transformer, trained on large-scale video-text data. While Google hasn’t disclosed the full parameter count, Veo 2 is part of its Gemini-scale ecosystem, likely operating in the billions of parameters. Stage 2: We passed the generated videos through Fast3R, a high-speed transformer-based model for multi-view 3D reconstruction. Fast3R uses a ViT-L fusion transformer with 24 layers and 1 billion parameters, capable of processing 1,000+ views in a single forward pass. Unlike pairwise methods, Fast3R reconstructs geometry and camera poses jointly in one shot, making it ideal for dense video frame input. We extracted evenly spaced frames from the videos and used Fast3R to generate clean, textured 3D meshes—turning cinematic outputs into immersive environments. Result: A lightweight app that transforms text prompts into realistic 3D mesh geometry and GIF in minutes that can be exported and applied for downstream uses in game design, embodied AI, . Challenges we ran into Syncing video frame output with 3D frame ingestion without quality loss or temporal misalignment. Managing GPU memory and batch size when scaling to hundreds of frames per video. Compressing and serving the final 3D outputs in a way that preserved fidelity while remaining lightweight and fast inference. What we learned We learned how to integrate multi-modal transformer models in a single pipeline—generating video from text, and then reconstructing that video into 3D scenes. We gained experience with high-throughput transformer architectures, 3D mesh processing, and video-to-geometry challenges. What's next for Vertext This project shows how storytelling, design, and AI can come together to lower the barrier to creating immersive worlds. It’s a step toward democratizing 3D content creation for artists, educators, and developers everywhere. Our next steps are to refine text to video prompt engineering and continue building out the use cases with end-to-end integration. ngineering and continue building out the use cases with end-to-end integration.
## README (from the GitHub repository)
# Vertext
## Inspiration
We were fascinated by the idea of turning imagination into reality—how a simple line of text could evolve into a full 3D environment. Inspired by generative media tools and cinematic AI, we set out to explore how language could become a spatial canvas, combining storytelling with immersive design.
## What it does
Our project transforms natural language prompts into immersive 3D scenes by combining generative video and geometry reconstruction. First, it creates short cinematic videos from text or verbal descriptions using a state-of-the-art video generation model. Then, it converts those videos into detailed 3D meshes using a high-speed transformer-based reconstruction system. The result is a fast, intuitive way to turn ideas into spatial experiences—bridging language, vision, and 3D design
**Use Cases:**
Game Development - Import the mesh into Unity, Unreal, or Godot as environmental assets or characters. Artists can sketch in text, then refine the 3D form interactively
Education - Generate 3D visuals of historical or scientific concepts from text prompts (e.g., "a medieval village"). Use in VR/AR learning modules.
Architecture Design - Quickly generate visual prototypes of landscapes, interiors, or thematic spaces from descriptive specs
Embodied AI - Train agents in custom environments derived from descriptive tasks. Generate synthetic 3D training data.
## How we built it
Stage 1: We used **Veo 2**, Google’s latest text-to-video diffusion model, to generate cinematic 8-second clips from natural language prompts in text or verbal form (**Groq Whisper API**). Veo 2 accepts rich descriptions and camera terms (e.g. “tracking shot at dawn”) and outputs high-fidelity 720p video. It leverages a diffusion model guided by a multimodal transformer, trained on large-scale video-text data. While Google hasn’t disclosed the full parameter count, Veo 2 is part of its Gemini-scale ecosystem, likely operating in the billions of parameters.
Stage 2: We passed the generated videos through **Fast3R**, a high-speed transformer-based model for multi-view 3D reconstruction. Fast3R uses a _ViT-L fusion transformer with 24 layers and 1 billion parameters, capable of processing 1,000+ views in a single forward pass_. Unlike pairwise methods, Fast3R reconstructs geometry and camera poses jointly in one shot, making it ideal for dense video frame input. We extracted evenly spaced frames from the videos and used Fast3R to generate clean, textured 3D meshes—turning cinematic outputs into immersive environments.
Result: A lightweight app that transforms text prompts into realistic 3D mesh geometry and GIF in minutes that can be exported and applied for downstream uses in game design, embodied AI, .
## Challenges we ran into
Syncing video frame output with 3D frame ingestion without quality loss or temporal misalignment. Managing GPU memory and batch size when scaling to hundreds of frames per video. Compressing and serving the final 3D outputs in a way that preserved fidelity while remaining lightweight and fast inference.
## What we learned
We learned how to integrate multi-modal transformer models in a single pipeline—generating video from text, and then reconstructing that video into 3D scenes. We gained experience with high-throughput transformer architectures, 3D mesh processing, and video-to-geometry challenges.
## What's next for Vertext
This project shows how storytelling, design, and AI can come together to lower the barrier to creating immersive worlds. It’s a step toward democratizing 3D content creation for artists, educators, and developers everywhere. Our next steps are to refine text to video prompt engineering and continue building out the use cases with end-to-end integration.
## Detected evidence (automated analysis)
Indexed codebase: 3 recognized source files, 36 KB.
- HTML (language) — detected in the code
- Python (language) — detected in the code
## Codebase structure (from repository index)
### Files (3 of 3)
```
index.html
main.py
README.md
```
### Dependencies
No dependency index available.
### Recent commits (newest first)
- Update README.md
- Added voice functions
- Update main.py
- Update README.md
- Merged two API endpoints
- Added frontend
- Update README.md
- project description update
- fix cors
- revert ngrok change
- Merge pull request #1 from vbx1404/add-ngrok
- added ngrok for api host
- add main video generation script with rest api
- first commit
## Key source files (fetched from GitHub, selected and truncated for size)
### main.py
```python
import cv2
import torch
import os
import subprocess
import time
import copy
import shutil
import trimesh
import numpy as np
import open3d as o3d
import tempfile
import logging
import uvicorn
import uuid
from pydantic import BaseModel
from fastapi import FastAPI, UploadFile, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from starlette.responses import FileResponse, JSONResponse
from google import genai
from google.genai import types
from google.genai.types import HttpOptions
# Assuming fast3r library is installed and its components are available
# These imports are based on your provided script
# from fast3r.viz.video_utils import extract_frames_from_video
from fast3r.dust3r.viz import cat_meshes, pts3d_to_trimesh
from fast3r.dust3r.utils.image import load_images, rgb
from fast3r.dust3r.inference_multiview import inference
from fast3r.models.fast3r import Fast3R
from fast3r.models.multiview_dust3r_module import MultiViewDUSt3RLitModule
logger = logging.getLogger("uvicorn")
logging.basicConfig(level=logging.INFO)
# --- Initialize Google GenAI client (assumes GOOGLE_API_KEY is in env) ---
# It's good practice to handle potential missing keys gracefully.
try:
# client = genai.Client(vertexai=True, api_key="AIzaSyDidVPLLZO4pR0XqXdy3ZykfQXi7ErMtT8")
# client = genai.Client(http_options=HttpOptions(api_version="v1"))
client = genai.Client()
except Exception as e:
logger.error(f"Failed to initialize Google GenAI Client: {e}")
client = None
# --- FastAPI App Initialization ---
app = FastAPI(
title="Fast3R 3D Reconstruction API",
description="Generate a video from a prompt, then reconstruct it into a 3D mesh.",
version="1.1.0"
)
app.add_middleware(
CORSMiddleware,
allow_origins=["*"], # Allows all origins
allow_credentials=True,
allow_methods=["*"], # Allows all methods
allow_headers=["*"], # Allows all headers
)
# --- Global Variables for Model, Device, and Video Cache ---
model = None
lit_module = None
device = None
# Simple in-memory cache to store the path of the last generated video.
# For a production system, a more robust cache like Redis would be better.
VIDEO_CACHE = {"latest_video_path": None, "latest_video_uid": None}
# --- Model Loading on Startup ---
@app.on_event("startup")
async def startup_event():
"""
Loads the Fast3R model into memory when the API server starts.
"""
global model, lit_module, device
logger.info("--- Loading Model ---")
# Set the device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Load the model from Hugging Face Hub
try:
model_name = "jedyang97/Fast3R_ViT_Large_512"
model = Fast3R.from_pretrained(model_name)
model = model.to(device)
# Wrap the model in the lightning module for helper functions
lit_module = MultiViewDUSt3RLitModule.load_for_inference(model)
# Set to evaluation mode
model.eval()
lit_module.eval()
logger.info(f"--- Model Loaded Successfully on {device} ---")
except Exception as e:
logger.error(f"--- Failed to load model: {e} ---")
# You might want to prevent the app from starting if the model fails to load.
model = None
lit_module = None
# --- Helper Functions ---
def create_mesh_from_preds(
preds,
views,
export_ply_path,
min_conf_thr_percentile=30,
flip_axes=True
):
"""
Generates and exports a combined 3D mesh from model predictions.
"""
logger.info("--- Generating 3D Mesh ---")
meshes = []
for i, pred in enumerate(preds):
pts3d = pred['pts3d_in_other_view'].cpu().numpy().squeeze()
img_rgb = views[i]['img'].cpu().numpy().squeeze().transpose(1, 2, 0)
conf = pred['conf'].cpu().numpy().squeeze()
conf_thr = np.percentile(conf, min_conf_thr_percentile)
mask = conf > conf_thr
img_rgb_uint8 = ((img_rgb + 1) * 127.5).astype(np.uint8).clip(0, 255)
mesh_dict = pts3d_to_trimesh(img_rgb_uint8, pts3d, valid=mask)
if mesh_dict['vertices'].shape[0] > 0:
meshes.append(mesh_dict)
if not meshes:
raise ValueError("No valid mesh parts could be generated.")
combined_mesh = trimesh.Trimesh(**cat_meshes(meshes))
if flip_axes:
combined_mesh.vertices[:, [1, 2]] = combined_mesh.vertices[:, [2, 1]]
combined_mesh.vertices[:, 2] = -combined_mesh.vertices[:, 2]
combined_mesh.export(export_ply_path)
logger.info(f"--- Mesh successfully exported to {export_ply_path} ---")
def extract_frames_from_video(video_path, output_dir):
"""
Extracts frames from a video file.
"""
saved_frames = []
# Use FFmpeg which is generally more robust
try:
logger.info(f"Extracting frames using FFmpeg from: {video_path}")
frame_pattern = os.path.join(output_dir, "frame_%04d.jpg")
ffmpeg_cmd = ["ffmpeg", "-i", video_path, "-vf", "fps=1", "-q:v", "2", frame_pattern]
subprocess.run(ffmpeg_cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for file in sorted(os.listdir(output_dir)):
if file.startswith("frame_") and file.endswith(".jpg"):
saved_frames.append(os.path.join(output_dir, file))
if not saved_frames:
raise RuntimeError("FFmpeg ran but extracted no frames.")
logger.info(f"Successfully extracted {len(saved_frames)} frames using FFmpeg.")
except Exception as e:
logger.error(f"FFmpeg extraction failed: {e}. Falling back to OpenCV.")
# Fallback to OpenCV if FFmpeg fails
try:
cap = cv2.VideoCapture(video_path)
fps = cap.get(cv2.CAP_PROP_FPS)
frame_interval = max(1, int(fps / 1)) # Aim for ~1 FPS
frame_count = 0
saved_count = 0
while cap.isOpened():
ret, frame = cap.read()
[truncated — 6753 more characters]
```
### index.html
```html
AI GIF Generator & 3D Viewer with Voice Input