# Project export: prototyp3d This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: TreeHacks 2025 - Tagline: The first ever 3D/VR prototyping tool - Devpost: https://devpost.com/software/prototyp3d-vc96d5 - GitHub: https://github.com/miyaliu627/prototyp3d - Demo: https://bcad-129-105-61-146.ngrok-free.app/devtool - Video: https://www.youtube.com/embed/7HlNzgyHP4M?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Harry Guan (43 commits), miyaliu627 (22 commits), jason-zhxn (15 commits) ## Devpost submission (written by the team) ### Inspiration Prototyping has revolutionized digital design—Figma made web and mobile seamless, Canva simplified graphic creation—but 3D and VR remain stuck in the past. Existing tools are slow, overly technical, and inaccessible to most creators. We wanted to change that by building Prototyp3D, the first effortless, AI-powered 3D/VR prototyping tool that generates real, working code and an easily interactive environment solely from natural language descriptions. ### What it does Prototyp3D enables anyone—developers, designers, educators, and entrepreneurs—to bring their 3D and VR ideas to life instantly. Users simply describe their vision in plain text, and our AI Agents go through an optimized pipeline to generate functional 3D/VR code, while rendering the results step by step for the user to interact with. It’s like having a virtual engineering team that plans, builds, tests, and refines your prototype—all without requiring coding skills. ### How we built it AI-Powered Prototyping Agent: Uses natural language processing to interpret user descriptions and generate structured development tasks. It breaks large projects into organized Jira-style tickets using OpenAI's GPT 4o model. AI-Powered Prototyping Agent: Uses natural language processing to interpret user descriptions and generate structured development tasks. It breaks large projects into organized Jira-style tickets using OpenAI's GPT 4o model. Autonomous Code Generation Pipeline: Uses OpenAI's GPT o3-mini-high model to generate/improve code given specific context and acceptance criteria (specified in tickets). Autonomous Code Generation Pipeline: Uses OpenAI's GPT o3-mini-high model to generate/improve code given specific context and acceptance criteria (specified in tickets). Iterative Debugging & Testing System: Feeds and runs fresh project code in multiple virtual machines; launches Scrapybara AI agents to interact and play with the running 3D environment (or provide error information) in order to evaluate and generate feedback based on given ticket goals. The AI agent “sees,” “clicks,” and “drags” elements in a virtual environment to verify functionality before iterating a debugging process or moving on to the next ticket after achieving a certain similarity score. Iterative Debugging & Testing System: Feeds and runs fresh project code in multiple virtual machines; launches Scrapybara AI agents to interact and play with the running 3D environment (or provide error information) in order to evaluate and generate feedback based on given ticket goals. The AI agent “sees,” “clicks,” and “drags” elements in a virtual environment to verify functionality before iterating a debugging process or moving on to the next ticket after achieving a certain similarity score. Intuitive Front End: A visually clean and modern interface designed as a one-stop shop for all development needs. It seamlessly integrates essential features like autosaving, on-demand compilation, full-screen mode, syntax highlighting, easy code copying, and quick downloads. Every element is optimized for a smooth and efficient workflow. Intuitive Front End: A visually clean and modern interface designed as a one-stop shop for all development needs. It seamlessly integrates essential features like autosaving, on-demand compilation, full-screen mode, syntax highlighting, easy code copying, and quick downloads. Every element is optimized for a smooth and efficient workflow. 3D & VR Code Generation: Leverages frameworks like Three.js, Babylon.js, and WebXR to build real, interactive 3D experiences. 3D & VR Code Generation: Leverages frameworks like Three.js, Babylon.js, and WebXR to build real, interactive 3D experiences. ### Challenges we ran into Balancing AI autonomy with user control: Ensuring users could guide and tweak the generated prototypes while keeping the process seamless. Debugging in 3D/VR environments: Creating a system that understands and interacts with complex 3D spaces like a human tester. We ensured that AI agents only used visual and interactive observations in order to feed unbiased feedback into our code generating pipeline. ### Accomplishments we're proud of The scope and complexity of projects that Prototyp3d can create far outpaces anything current models can output. We are proud to have utilized AI agents to create a debugging/feedback pipeline that models the thought process of real seasoned engineers. Developed a working end-to-end pipeline where users can describe ideas and receive functional 3D/VR applications in minutes. We are proud to have created a customer oriented interface that allows users to not only see and edit the generated code but also interact with the rendered 3d environment itself. ### What we learned The amount of possibilities that AI agents unlock is incredible: the use of them to simulate human engineers unlocks the possibility of creating projects over 10 times as complex as a query to the latest models. Natural language AI for software engineering is incredibly powerful, but structuring its output into reliable, functional code takes careful design . Speed and quality are hard to balance: we focused a lot on outputting quality code, but that meant sacrificing run times and having to iterate through long pipelines. ### What's next Expanding VR Compatibility: Adding support for Unity, Unreal Engine, and other VR platforms. Interactive Queries: We want to flexibility to the next level, letting users specify wanted changes by interacting with the 3D environment itself. Collaboration Features: Enabling real-time, multiplayer prototyping so teams can build together. An even bigger future: we think our ai agent pipeline can be generalized to more types of generation. It can be used to generate projects outside of the 3d space. It can be used to write essays. It can be used to generate videos. We think our pipeline unlocks massive potential for generating quality output, and think that this technology can be generalized and used for so many different products. ## README (from the GitHub repository) # prototyp3d prototyp3d is an AI-powered 3D and VR prototyping tool that transforms natural language descriptions into functional, interactive 3D/VR applications. By leveraging AI agents and autonomous code generation, prototyp3d streamlines the prototyping process, allowing users to build, test, and refine their 3D/VR ideas effortlessly. ## Inspiration Prototyping has revolutionized digital design—Figma simplified UI/UX, Canva streamlined graphic design—but 3D and VR remain difficult to access. Traditional tools are slow, overly technical, and not user-friendly. **prototyp3d** changes that by enabling intuitive, AI-powered 3D/VR development that generates real, working code in an interactive environment. ## What It Does prototyp3d allows users—including developers, designers, educators, and entrepreneurs—to create 3D/VR experiences simply by describing them in plain text. Our AI pipeline interprets user descriptions, generates code, and renders interactive 3D/VR environments, providing users with: - Instant code generation based on natural language input. - Real-time interactive rendering of 3D/VR prototypes. - Debugging and testing using AI agents that simulate real engineers. - A user-friendly interface with autosaving, full-screen mode, syntax highlighting, on-demand compilation, and many more! ## How We Built It ### **AI-Powered Prototyping Pipeline** - **Natural Language Processing / Autonomous Code Generation**: Utilizes OpenAI's **o3-mini** to interpret user descriptions and create structured development tasks. - **Iterative Debugging & Testing**: The **Scrapybara** agent interacts with the environment and performs visual inspections which are later fed into the feedback debugging loop. - **Optimized Rendering Pipeline**: Supports **Three.js, Babylon.js, WebXR**, and other 3D/VR frameworks. - **Modern Frontend**: A sleek UI built with **React, HTML, and CSS, and Three.js**. ## Accomplishments - Built a full AI-powered 3D/VR prototyping pipeline capable of generating functional applications. - Created a debugging system where AI agents interact with the generated 3D environment, mimicking real engineers. - Developed an intuitive interface that allows users to both **edit code and interact with their rendered environments**. ## What's Next - **Interactive Queries**: Let users modify prototypes directly within the 3D environment, interacting with specific elements and isolating blocks of code for accurate debugging and improvements. - **VR Compatibility**: Expanding support to **Unity, Unreal Engine, and other VR platforms**. - **Multithreading / Optimizations**: Queries could always run faster -- addressing tickets in separate threads or even cores would be greatly beneficial for the overall runtime of each query. ## Built With - **AI**: OpenAI o3-mini, Scrapybara, Anthropic Claude - **Frontend**: React, JavaScript (Three.js), HTML, CSS - **Backend**: Python, Flask - **Infrastructure**: Ubuntu, Bash, ngrok ## License This project is licensed under the **MIT License**. ## Detected evidence (automated analysis) Indexed codebase: 28 recognized source files, 78 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - Flask (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code ## Codebase structure (from repository index) ### Files (38 of 38) ``` .gitignore backend/.gitignore backend/app.py backend/debugger.py backend/llm.py backend/prototyper.py backend/requirements.txt backend/ticket.py frontend/.gitignore frontend/app/api/load/route.js frontend/app/api/save/route.js frontend/app/DevConsole.jsx frontend/app/devtool/page.jsx frontend/app/globals.css frontend/app/index.jsx frontend/app/layout.js frontend/app/page.jsx frontend/components/Chat.jsx frontend/components/CodeEditor.jsx frontend/components/FileNavigation.jsx frontend/components/Header.jsx frontend/components/Preview.jsx frontend/eslint.config.mjs frontend/jsconfig.json frontend/next.config.mjs frontend/package.json frontend/postcss.config.mjs frontend/public/tesseract-scene.html frontend/README.md frontend/scene/TesseractScene.jsx frontend/static/product/index.html frontend/static/product/script.js frontend/static/product/styles.css frontend/static/template/index.html frontend/static/template/script.js frontend/static/template/styles.css frontend/tailwind.config.mjs README.md ``` ### Dependencies - backend/requirements.txt: annotated-types@==0.7.0, anthropic@==0.39.0, anyio@==4.8.0, build@==1.2.2.post1, CacheControl@==0.14.2, certifi@==2025.1.31, cffi@==1.17.1, charset-normalizer@==3.4.1, cleo@==2.1.0, crashtest@==0.4.1, distlib@==0.3.9, distro@==1.9.0, dulwich@==0.22.7, fastjsonschema@==2.21.1, filelock@==3.17.0, findpython@==0.6.2, flask@==3.1.0, flask_cors@==5.0.0, greenlet@==3.1.1, gunicorn@==23.0.0, h11@==0.14.0, httpcore@==1.0.7, httpx@==0.28.1, idna@==3.10, installer@==0.7.0, jaraco.classes@==3.4.0, jaraco.context@==6.0.1, jaraco.functools@==4.1.0, jiter@==0.8.2, keyring@==25.6.0, more-itertools@==10.6.0, msgpack@==1.1.0, openai@==1.63.0, packaging@==24.2, pbs-installer@==2025.2.12, pkginfo@==1.12.0, platformdirs@==4.3.6, playwright@==1.50.0, poetry@==2.1.0, poetry-core@==2.1.0, pycparser@==2.22, pydantic@==2.10.6, pydantic_core@==2.27.2, pyee@==12.1.1, pyproject_hooks@==1.2.0, RapidFuzz@==3.12.1, requests@==2.32.3, requests-toolbelt@==1.0.0, scrapybara@==2.2.5, shellingham@==1.5.4, sniffio@==1.3.1, tomlkit@==0.13.2, tqdm@==4.67.1, trove-classifiers@==2025.1.15.22, typing_extensions@==4.12.2, urllib3@==2.3.0, uvicorn@==0.34.0, virtualenv@==20.29.2, xattr@==1.1.4, zstandard@==0.23.0 - frontend/package.json: @eslint/eslintrc@^3, @heroicons/react@^2.2.0, dotenv@^16.4.7, eslint@^9, eslint-config-next@15.1.7, file-saver@^2.0.5, jszip@^3.10.1, lucide-react@^0.475.0, next@15.1.7, postcss@^8, prismjs@^1.29.0, react@^19.0.0, react-dom@^19.0.0, tailwindcss@^3.4.1, three@^0.173.0 ### Recent commits (newest first) - Update README.md - Merge pull request #14 from miyaliu627/deployment - Merge branch 'main' into deployment - gitignore added - Removed debug_loop.log to resolve conflict - Merge pull request #13 from miyaliu627/chat - bugfix - Update README.md - Update README.md - Merge pull request #12 from miyaliu627/extraneous - removed extraneous code/comments - revert chat to working - some deployment - updated landing - landing page - deployment initial - Update ticket.py - Update prototyper.py - Update debugger.py - Update app.py ## Key source files (fetched from GitHub, selected and truncated for size) ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev --turbopack", "build": "next build", "start": "next start", "lint": "next lint" }, "dependencies": { "@heroicons/react": "^2.2.0", "dotenv": "^16.4.7", "file-saver": "^2.0.5", "jszip": "^3.10.1", "lucide-react": "^0.475.0", "next": "15.1.7", "prismjs": "^1.29.0", "react": "^19.0.0", "react-dom": "^19.0.0", "three": "^0.173.0" }, "devDependencies": { "@eslint/eslintrc": "^3", "eslint": "^9", "eslint-config-next": "15.1.7", "postcss": "^8", "tailwindcss": "^3.4.1" } } ``` ### backend/requirements.txt ``` annotated-types==0.7.0 anthropic==0.39.0 anyio==4.8.0 build==1.2.2.post1 CacheControl==0.14.2 certifi==2025.1.31 cffi==1.17.1 charset-normalizer==3.4.1 cleo==2.1.0 crashtest==0.4.1 distlib==0.3.9 distro==1.9.0 dulwich==0.22.7 fastjsonschema==2.21.1 filelock==3.17.0 findpython==0.6.2 greenlet==3.1.1 h11==0.14.0 httpcore==1.0.7 httpx==0.28.1 idna==3.10 installer==0.7.0 jaraco.classes==3.4.0 jaraco.context==6.0.1 jaraco.functools==4.1.0 jiter==0.8.2 keyring==25.6.0 more-itertools==10.6.0 msgpack==1.1.0 openai==1.63.0 packaging==24.2 pbs-installer==2025.2.12 pkginfo==1.12.0 platformdirs==4.3.6 playwright==1.50.0 poetry==2.1.0 poetry-core==2.1.0 pycparser==2.22 pydantic==2.10.6 pydantic_core==2.27.2 pyee==12.1.1 pyproject_hooks==1.2.0 RapidFuzz==3.12.1 requests==2.32.3 requests-toolbelt==1.0.0 scrapybara==2.2.5 shellingham==1.5.4 sniffio==1.3.1 tomlkit==0.13.2 tqdm==4.67.1 trove-classifiers==2025.1.15.22 typing_extensions==4.12.2 urllib3==2.3.0 virtualenv==20.29.2 xattr==1.1.4 zstandard==0.23.0 gunicorn==23.0.0 uvicorn==0.34.0 flask==3.1.0 flask_cors==5.0.0 ``` ### backend/app.py ```python import json from flask import Flask, request, jsonify, Response, stream_with_context from flask_cors import CORS from dotenv import load_dotenv from enum import Enum import os try: from .prototyper import Prototyper from .debugger import debug_with_scrapybara except: from prototyper import Prototyper from debugger import debug_with_scrapybara import scrapybara load_dotenv(dotenv_path="../.env") BACKEND_URL = os.getenv("NGROK_BACKEND", "http://localhost:5001") # ✅ Use correct env var print(f"Using BACKEND_URL: {BACKEND_URL}") OPENAI_API_KEY = os.getenv("OPENAI_API_KEY") SCRAPYBARA_API_KEY = os.getenv("SCRAPYBARA_API_KEY") app = Flask(__name__) CORS(app, resources={r"/*": {"origins": "*"}}) prototyper = None progress_messages = [] class MessageTypes(Enum): SETTING_UP = "setting_up" NEW_TICKET = "new_ticket" TICKET_COMPLETED = "ticket_completed" ERROR = "error" DEBUG = "debug" ITERATE = "iterate" COMPLETED = "completed" @app.route('/prototype/create', methods=['POST']) def prototype(): global prototyper global progress_messages try: scrapybara_client = scrapybara.Scrapybara() scrapybara_instance = scrapybara_client.start_ubuntu(timeout_hours=0.2) data = request.get_json() user_prompt = data.get("user_prompt") project_name = data.get("project_name") if not user_prompt: return jsonify({"error": "Missing 'user_prompt' in request"}), 400 progress_messages.clear() prototyper = Prototyper(user_prompt, scrapybara_client, scrapybara_instance, name=project_name) progress_messages.append({ "type": MessageTypes.SETTING_UP.value, "message": "I am setting up the repository..." }) prototyper.setup_repo() progress_messages.append({ "type": MessageTypes.SETTING_UP.value, "message": "I have finished setting up the initial repository!" }) prototyper.repo_summary = "a 3D interactive scene using Three.js, featuring a large green ground plane, a sky-blue background, and a perspective camera positioned at human eye level. Users can navigate using WASD and arrow keys for movement and OrbitControls for mouse-based rotation. The scene includes ambient and directional lighting to enhance realism. " progress_messages.append({ "type": MessageTypes.SETTING_UP.value, "message": "I am creating the tickets for this project..." }) prototyper.create_tickets() progress_messages.append({ "type": MessageTypes.SETTING_UP.value, "message": f"I have created {len(prototyper.tickets)} tickets for this project." }) # Store all ticket data ticket_responses = [] for ticket in prototyper.tickets: # send initial data for the ticket to frontend progress_messages.append({ "type": MessageTypes.NEW_TICKET.value, "message": f"Working on ticket: {ticket.description}" }) response = ticket.complete(prototyper.repo_path, prototyper.repo_summary) failure = debug_with_scrapybara(prototyper.repo_path, ticket.description, prototyper.scrapybara_client, prototyper.scrapybara_instance) if failure: progress_messages.append({ "type": MessageTypes.DEBUG.value, "message": f"Unable to debug output for {ticket.description}... aborting process right now", }) raise Exception("Debugging failed") else: ticket_responses.append({ "ticket": f"Completed ticket: {ticket.description}", "internal_dialogue": response, }) progress_messages.append({ "type": MessageTypes.TICKET_COMPLETED.value, "message": f"Finished ticket: {ticket.description}", "details": response }) progress_messages.append({ "type": MessageTypes.COMPLETED.value, "message": "All tickets completed!" }) if os.path.exists(prototyper.repo_path): return jsonify({ "success": "Created repo", "repo_path": prototyper.repo_path, "ticket_responses": ticket_responses }), 200 else: return jsonify({"error": "Generated repository not found"}), 500 except Exception as e: return jsonify({"error": f"An error occurred: {str(e)}"}), 500 @app.route('/prototype/iterate', methods=['POST']) def iterate(): global prototyper try: data = request.get_json() user_prompt = data.get("user_prompt") if not user_prompt: return jsonify({"error": "Missing 'user_prompt' in request"}), 400 prototyper.summarize_repo() prototyper.create_tickets() # Store all ticket data ticket_responses = [] for ticket in prototyper.tickets: # send initial data for the ticket to frontend initial_data = { "type": MessageTypes.NEW_TICKET.value, "message": f"I am working on completing the following ticket: {ticket.description}", } response = ticket.complete(prototyper.repo_path, prototyper.repo_summary) final_data = { "type": MessageTypes.TICKET_COMPLETED.value, "message": response } ticket_responses.append({ "ticket": f"Completed ticket: {ticket.description}", "internal_dialogue": response, }) if os.path.exists(prototyper.repo_path): [truncated — 1028 more characters] ``` ### frontend/app/layout.js ```javascript import './globals.css' export const metadata = { title: 'AI 3D Prototyping', description: 'AI-powered 3D prototyping platform', } export default function RootLayout({ children }) { return ( {children} ) } ``` ### frontend/app/page.jsx ```javascript 'use client'; import { useState } from 'react'; import { useRouter } from 'next/navigation'; import TesseractScene from '@/scene/TesseractScene'; import DevConsole from '@/app/DevConsole'; export default function Home() { const [isDevMode, setIsDevMode] = useState(false); const router = useRouter(); return (
{isDevMode ? ( ) : ( setIsDevMode(true)} router={router} /> )}
); } ``` ### frontend/app/index.jsx ```javascript 'use client'; import { useEffect, useState } from 'react'; import { useRouter } from 'next/navigation'; import DevConsole from '@/app/DevConsole'; export default function Home() { const [isDevMode, setIsDevMode] = useState(false); const router = useRouter(); useEffect(() => { if (!isDevMode) { const iframe = document.createElement('iframe'); iframe.src = '/tesseract.html'; iframe.style.width = '100vw'; iframe.style.height = '100vh'; iframe.style.border = 'none'; document.body.innerHTML = ''; document.body.appendChild(iframe); window.addEventListener('message', (event) => { if (event.data === 'enterDevMode') { setIsDevMode(true); } }); } }, [isDevMode]); return (
{isDevMode ? : null}
); } ``` ### frontend/app/devtool/page.jsx ```javascript 'use client'; import DevConsole from '@/app/DevConsole'; export default function DevToolPage() { return ; } ``` ### frontend/app/api/save/route.js ```javascript // app/api/save/route.js import { writeFile } from 'fs/promises'; import { join } from 'path'; import { NextResponse } from 'next/server'; export async function POST(request) { try { const files = await request.json(); const templateDir = join(process.cwd(), 'static/product/'); console.log(templateDir); await Promise.all( Object.entries(files).map(async ([filename, content]) => { const filePath = join(templateDir, filename); await writeFile(filePath, content, 'utf8'); }) ); return NextResponse.json({ success: true }); } catch (error) { return NextResponse.json({ success: false, error: error.message }, { status: 500 }); } } ``` ### frontend/app/api/load/route.js ```javascript import { readFile } from 'fs/promises'; import { join } from 'path'; import { NextResponse } from 'next/server'; export async function GET() { try { const templateDir = join(process.cwd(), 'static/product/'); const fs = require('fs'); fs.accessSync(templateDir, fs.constants.F_OK); const files = { 'index.html': await readFile(join(templateDir, 'index.html'), 'utf8'), 'styles.css': await readFile(join(templateDir, 'styles.css'), 'utf8'), 'script.js': await readFile(join(templateDir, 'script.js'), 'utf8') }; return NextResponse.json(files); } catch (error) { console.error('Error loading files:', error); return NextResponse.json( { error: 'Failed to load files' }, { status: 500 } ); } } ``` ### backend/llm.py ```python import openai import json import re client = openai.Client() def chatcompletion(user_prompt, system_prompt=""): try: response = client.chat.completions.create( model="o3-mini", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}, ], ) message_content = response.choices[0].message.content json_match = re.search(r"\{.*\}", message_content, re.DOTALL) if not json_match: print("Error: No valid JSON found in response.") return {} message_content = json_match.group(0) if "'" in message_content and '"' not in message_content: message_content = message_content.replace("'", '"') return json.loads(message_content) except openai.RateLimitError as e: print(f"RateLimitError: {e}") except json.JSONDecodeError: return {} except Exception as e: print(f"Error: {e}") return {} return {} def chatcompletion_stream(user_prompt, system_prompt=""): try: response = client.chat.completions.create( model="o3-mini", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_prompt}, ], stream=True, ) accumulated_response = "" for chunk in response: if chunk.choices and chunk.choices[0].delta.content: accumulated_response += chunk.choices[0].delta.content return accumulated_response.strip() except openai.RateLimitError as e: print(f"RateLimitError: {e}") except Exception as e: print(f"Error: {e}") ``` [18 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]