# Project export: Backyard Intelligence This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: From backyard idea to verified 3D build plan and AR preview in minutes. - Devpost: https://devpost.com/software/backyard-intelligence-nrev73 - GitHub: https://github.com/mp1678/cal-ai-hackathon-26 - Video: https://www.youtube.com/embed/RPIf7KsVd9k?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 3 GitHub contributor(s) — Kevin Toren (29 commits), Claude Opus 4.8 (27 commits), Ktej (2 commits) ## Devpost submission (written by the team) ### Overview Backyard Intelligence Backyard Intelligence turns a natural-language backyard project idea into a structured, inspectable build concept. A user can ask for something like a pergola, shed, or raised garden bed, and the system converts that request into a typed construction spec, deterministic 3D component model, verification report, bill of materials, and safety checklist. An AR companion app allows users to see the structure in real-life size, filter by specific materials and see how they fit together, and precisely fabricate parts using computer vision cutting hints. ### Inspiration We were interested in a version of AI for physical-world projects that does more than produce a nice-looking answer. Backyard construction is full of details that matter: dimensions, stock lengths, support paths, openings, fasteners, roofing layers, and material quantities. A language model is great at understanding messy human intent, but it should not be trusted to hallucinate geometry or a shopping list. So we built a pipeline where AI interprets the request, but deterministic code owns the buildable representation. The combination of AI generation and the AR app provides the speed of vibe coding, the safety of a hired contractor, and the spatial intuition of building a lego set. ### What it does The core flow is: natural language → AI-interpreted design spec → deterministic component generation → verification → BOM + GLB + AR handoff Backyard Intelligence currently supports: Pergolas Sheds Raised garden beds For each generated design, the app produces: A typed construction specification A true-scale 3D model with named parts A verification report for dimensions, connections, stock fit, openings, support paths, and overlaps A bill of materials traced back to generated component IDs A safety/building-guideline checklist A downloadable GLB model An AR handoff path for viewing and fabrication guidance on iOS ### How we built it The web app is built with React, TypeScript, Vite, Three.js, React Three Fiber, and Drei. Gemini is the primary AI interpreter, with Claude available as a fallback. The AI route lives in Vite middleware so API keys stay server-side. The important architectural choice was separating interpretation from generation. The AI produces a structured DesignSpec, then deterministic TypeScript code validates the spec, generates every component, derives the BOM, verifies the assembly, and exports the GLB. The BOM is not guessed by the model; it is calculated from the generated parts. The companion iOS app was built using SwiftUI, ARKit, RealityKit, Metal, OpenCV/ArUco marker tracking, and GLTFKit2. It can fetch the latest exported GLB and render placement/cutting guidance in AR. Challenges The hardest part was making the output inspectable instead of just visually plausible. We had to model real construction details like rafters, studs, siding, roof layers, gable/lean-to geometry, deck blocks, joists, openings, trim, fasteners, and stock-length segmentation. Another challenge was verification. A model can look fine while still having disconnected parts, impossible stock lengths, invalid openings, floating geometry, or overlapping solids. We built checks for required part groups, finite dimensions, connection references, physical joint fit, support paths, roof bearing, BOM correspondence, and more. The AR side also had its own challenges: true-scale unit conversion, loading GLB files on device, marker-based placement, LiDAR occlusion, and making the guide readable over a real camera feed. ### What we learned We learned that AI becomes much more useful for high-stakes physical domains when it is paired with deterministic systems. The language model should help translate intent, ask clarifying questions, and repair invalid specs, but the final geometry, quantities, verification, and exports should come from code. We also learned how much value there is in traceability. When every visible part has a node ID, every BOM item traces back to components, and every verification check explains what passed or failed, the result becomes something a user can inspect instead of blindly trust. What is next Next, we would expand the supported structure library, improve permitting/local-code awareness, add more complete cost and store availability integrations, and tighten the AR workflow so users can move from generated concept to guided layout and cutting more smoothly. Backyard Intelligence is not a permit package or structural certification tool. It is a prototype for a safer AI-assisted design workflow where human intent, deterministic generation, verification, and AR visualization work together. ## README (from the GitHub repository) # Backyard Intelligence Backyard Intelligence is a Berkeley AI Hackathon prototype for turning a natural-language backyard project idea into a structured, inspectable build concept. The core architecture is: `natural language → AI-interpreted design spec → deterministic component generation → verification → BOM + GLB + AR handoff` The app currently supports: - Pergolas - Sheds - Raised garden beds It is not a permit package or structural certification tool. The goal is to prove a safer AI-assisted architecture where language models interpret intent, while deterministic code owns geometry, verification, bill of materials, and export. ## Why this uses AI The AI layer is used where language models are strongest: - Understanding messy user prompts like “make it a 10x8 gable shed with double doors and two left windows.” - Asking one approval-style follow-up when the prompt is ambiguous. - Converting conversation history and user revisions into a complete typed construction specification. - Using the provided shed PDF as design context when forming the spec. - Falling back across Gemini and Claude providers when one model is unavailable. - Optionally attempting repair prompts when deterministic validation or verification fails. The AI does not guess the final BOM, mesh geometry, or verifier output. Those are produced from deterministic code. ## System diagram ```mermaid flowchart TD A["User prompt or revision"] --> B["Structure router
Any / Pergola / Shed / Garden bed"] B --> C["AI interpreter
Gemini primary, Claude fallback"] C --> D{"Ambiguity left?"} D -- "Yes" --> E["AI asks one approval-style
default confirmation question"] E --> A D -- "No" --> F["Structured DesignSpec v2
intent, dimensions, stock IDs,
openings, roof/foundation rules"] F --> G["Deterministic spec validator"] G --> H{"Spec valid?"} H -- "No" --> I["AI repair or recommended fix
depending on env mode"] I --> F H -- "Yes" --> J["Deterministic component generator
named part nodes, dimensions,
connections, materials"] J --> K["Model verifier
required parts, stock fit,
joint fit, support path,
overlaps, openings"] K --> L{"Model valid?"} L -- "No" --> I L -- "Yes" --> M["BOM generator
derived from generated components"] M --> N["Safety / guideline checklist
shed + pergola advisory checks"] N --> O["GLB exporter
one named mesh per visible part"] O --> P["UI response
spec, model, verification,
BOM, safety checklist, GLB"] O --> Q["Optional S3 current.glb upload
for AR viewer"] ``` ## How the system works 1. The user describes a structure in chat. 2. The app routes the prompt to a family: pergola, shed, garden bed, or “any.” 3. Gemini or Claude converts the prompt into a complete structured design specification. 4. If key choices are ambiguous, the AI proposes sensible defaults from the current sliders and asks the user to approve or modify them. 5. The deterministic validator checks the spec before any geometry is generated. 6. The component generator creates the actual construction model from reusable part rules. 7. The verifier checks that the generated model is internally consistent. 8. The BOM is derived from generated component nodes, not from AI text. 9. The GLB exporter creates a true-scale model with one named node per visible part. 10. For sheds and pergolas, a safety/building-guideline checklist compares the design against conventional guidance and flags advisory issues. ## Generated artifacts Each successful design returns separate artifacts: - **Design spec** — what the user asked for, normalized into typed construction data. - **Component model** — what the deterministic generator actually built. - **Verification report** — whether the generated model is internally consistent. - **Safety checklist** — advisory shed/pergola guideline checks based on sources like AWC, IRC, and Simpson Strong-Tie. - **BOM** — materials and purchase quantities derived from component nodes. - **GLB** — one named mesh per visible component, with material colors for true-scale 3D/AR viewing. Keeping these separate is the main safety pattern: the language model describes intent, but deterministic code produces and checks the buildable representation. ## Supported structure families ### Pergola Generated assemblies include: - Concrete footings - Post bases - 6x6 posts - Beam assemblies - Rafters - Shade slats - Knee braces - Rafter ties / connector markers - Reference figure, compass, and dimension guides The generator segments over-length lumber when the selected stock is shorter than the requested run. ### Shed Generated assemblies include: - Prepared base - Deck blocks or concrete slab - Floor skids, joists, and floor deck when raised-floor mode is enabled - Wall plates, studs, headers, and opening framing - Door/window units - Siding, trim, flashing, sealant packages - Gable or lean-to roof framing - Roof sheathing, underlayment, shingles, drip edge, fascia, and rake trim - Fastener and connector point markers ### Raised garden bed Generated assemblies include: - Side boards - Corner and intermediate posts - Top caps - Optional bottom mesh / liner - Optional trellis - Soil and compost volume estimate ## AI providers and modes Server-side AI credentials stay in Vite middleware and are not shipped to the browser. ```bash GEMINI_API_KEY=... ANTHROPIC_API_KEY=... AI_PROVIDER_ORDER=gemini,claude ``` By default, the app uses Gemini first and can fall back to Claude if configured. Useful modes: ```bash # Local deterministic parser only; no AI interpretation or AI repair. VITE_DETERMINISTIC_ONLY=true # AI still parses intent, but failed deterministic review surfaces a recommended fix # instead of running automatic AI repair turns. VITE_AI_REPAIR=false ``` ## Run locally ```bash npm install cp .env.example .env # Add GEMINI_API_KEY and/or ANTHROPIC_API_KEY to .env npm run dev ``` ## AR current GLB upload Every successful generation exports the same GLB used by the download button and POSTs it to the local server at `/api/glb/current`. The server overwrites: `https://backyard-intelligence.s3.us-west-1.amazonaws.com/current.glb` Public read access lets the AR viewer fetch the file, but it does not automatically allow overwriting it. The upload endpoint supports both modes: - With AWS credentials, it performs a signed `PUT`. - Without AWS credentials, it tries a public write to the object URL. That succeeds only if the bucket policy explicitly allows anonymous `s3:PutObject` for `current.glb`. Optional server-side AWS credentials: ```bash S3_GLB_BUCKET=backyard-intelligence S3_GLB_REGION=us-west-1 S3_GLB_KEY=current.glb AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= ``` The UI shows whether `current.glb` uploaded and whether the public AR URL verified. ## Verification and safety checks The internal verifier checks generated geometry and data consistency: - Required component groups exist - Dimensions are finite and positive - Stock lengths fit selected Home Depot-oriented materials - Connections reference real generated components - Declared joints physically meet - Unconnected solid geometry does not overlap - Openings fit walls - Roof planes and rafters touch correctly - BOM traces back to generated components The safety checklist is advisory and currently applies to sheds and pergolas. It checks the generated spec/model against conventional guidance for: - Foundation and post/base support - Exterior-rated materials - Framing spacing - Bracing - Uplift connector locations - Roof/weathering layers - Siding and sheathing fastener schedules - Local permit, frost, wind, snow, seismic, and engineering review warnings Research sources used in the app: - [American Wood Council DCA6 deck guide](https://awc.org/wp-content/uploads/2022/02/AWC-DCA62015-DeckGuide-1804.pdf) - [Simpson Strong-Tie Wood Construction Connectors catalog]( [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 61 recognized source files, 625 KB. - C (language) — detected in the code - CSS (language) — detected in the code - HTML (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Swift (language) — detected in the code - TypeScript (language) — detected in the code - AWS (technology) — claimed on Devpost, not found in the code - Google Gemini (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: commit authorship or trailers ## Codebase structure (from repository index) ### Files (78 of 78) ``` .env.example .gitignore backyard-151-parts.glb backyard-204-parts.glb BoundaryGhost.xcodeproj/project.pbxproj BoundaryGhost.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved BoundaryGhost.xcodeproj/xcshareddata/xcschemes/BoundaryGhost.xcscheme BoundaryGhost/AppModel.swift BoundaryGhost/ArucoBridge.h BoundaryGhost/ArucoBridge.mm BoundaryGhost/ARViewContainer.swift BoundaryGhost/Assets.xcassets/AccentColor.colorset/Contents.json BoundaryGhost/Assets.xcassets/AppIcon.appiconset/Contents.json BoundaryGhost/Assets.xcassets/Contents.json BoundaryGhost/BoundaryGhost-Bridging-Header.h BoundaryGhost/BoundaryGhostApp.swift BoundaryGhost/ContentView.swift BoundaryGhost/CutHintBuilder.swift BoundaryGhost/CuttingMask.metal BoundaryGhost/CuttingMask.swift BoundaryGhost/DepthLift.swift BoundaryGhost/ghost-model.usdz BoundaryGhost/GhostBuilder.swift BoundaryGhost/GhostGeometry.swift BoundaryGhost/GLBPipeline.swift BoundaryGhost/HUDView.swift BoundaryGhost/MarkerTracker.swift BoundaryGhost/ModelGhostBuilder.swift BoundaryGhost/PlacementCoordinator.swift BoundaryGhost/PlacementOrderSolver.swift BoundaryGhost/ReferenceFigureBuilder.swift BoundaryGhost/SeatedDetector.swift BoundaryGhost/Units.swift BUILDING SOURCES.md convert.sh docs/CUT_GEOMETRY.md docs/design-basis.md docs/GENERATOR_AUDIT.md glb_to_usdz.py index.html package.json PLACEMENT_ORDER.md placement_order.py README.md requirements-placement.txt scripts/generator-sweep.ts scripts/provider-slab-regression.ts src/AiBuildLoader.tsx src/App.tsx src/data/materialCatalog.ts src/data/priceCatalog.ts src/domain/constructionSpec.ts src/domain/geminiSchema.ts src/fasteners.ts src/features.css src/generators/primitives.ts src/geometry3d.ts src/glb.ts src/main.tsx src/materials.ts src/ModelViewer.tsx src/modern.css src/pipeline.css src/pipeline/bom.ts src/pipeline/cost.ts src/pipeline/generator.ts src/pipeline/index.ts src/pipeline/interpreter.ts src/pipeline/safety.ts src/pipeline/verifier.ts src/shade.ts src/styles.css src/textLabels.ts src/types.ts src/viewer-controls.css tsconfig.app.json tsconfig.json vite.config.ts ``` ### Dependencies - package.json: @react-three/drei@^10.7.4, @react-three/fiber@^9.4.0, @types/react@^19.0.0, @types/react-dom@^19.0.0, @types/three@^0.180.0, @vitejs/plugin-react@^4.3.4, lucide-react@^0.468.0, react@^19.0.0, react-dom@^19.0.0, three@^0.180.0, typescript@~5.7.2, vite@^6.0.5 ### Recent commits (newest first) - Add Ios App - prompt to 3d model - Add live GLB pipeline: in-app fetch, ordering, range renderer, spin - Smoothing fix - Accurate floor depth estimation using LIDAR - Wireframe-only ghost; drop disconnected red islands in cut highlighter - Gate cut highlighter slab above floor noise; add model-ghost path - Stop ghost-drag pan from stealing HUD taps (mode toggle snap-back) - Defer ghost placement until lock (no jittery pre-lock preview) - Add ghost debug readout: distance, world size, on-screen, height - Draw detected ArUco corners on screen (debug overlay) - Require consecutive square reads to lock; threshold 0.97 - Tighten lock gate to 0.99 squareness (tuned on-device) - Fix squareness metric (per-edge-pair ratios) + live on-screen readout - Gate the lock on 2D-corner squareness, not the solvePnP normal - Revert station frame to world-up (levelStationFrame); drop marker-normal dependency - Fix anchoring regression: lock station on stable reads, not noisy marker normal - Fix floating ghost (level-snap cone) and opaque cut-mask overlay - Drag the ghost across the station floor plane - Arbitrary-geometry cut hints + depth-based feedback ## Key source files (fetched from GitHub, selected and truncated for size) ### PLACEMENT_ORDER.md ```markdown # Placement order (`placement_order.py`) > **Note (in-app pipeline).** This script is the **reference spec** for the > bottom-up ordering algorithm. The app now runs the *entire* pipeline **live on > device**: it downloads a GLB, loads it straight into RealityKit entities via > GLTFKit2 (no USDZ step), and computes this same order in Swift — > `BoundaryGhost/PlacementOrderSolver.swift` (PCA oriented-bounding-box variant of > the OBB below) driven by `BoundaryGhost/GLBPipeline.swift`. The > "Renderer integration" section further down describes the **older offline / > bundled-USDZ** flow and is kept for historical context; the contact-graph / > layering algorithm itself is unchanged. Given a GLB/GLTF assembly, compute the order in which to **place** each part, bottom-up: 1. First, every part that **touches the ground**. 2. Then every remaining part that **touches an already-placed part**. 3. Repeat until nothing new can be placed. This is a breadth-first layering of the parts' *contact graph*, seeded from the parts resting on the floor. Layer 0 is the ground; layer *n* rests on layer *n − 1* (or below). The result reads like a real build sequence — for the backyard shed model it goes gravel pads → deck blocks → skids → joists → decking → walls/studs → siding → top plates → rafters → door → roofing trim. ## Usage ```bash # one-time setup python3 -m venv .venv .venv/bin/pip install -r requirements-placement.txt # run .venv/bin/python placement_order.py backyard-204-parts.glb .venv/bin/python placement_order.py model.glb --tol 0.02 --json order.json ``` Flags: | flag | meaning | default | |------|---------|---------| | `--tol` | contact tolerance in meters (gap below this counts as "touching") | `0.02` (2 cm) | | `--ground-tol` | floor-contact tolerance | same as `--tol` | | `--json` | output path | `.placement.json` | | `--quiet` | suppress the text report (still writes JSON) | off | ## Output A layer-by-layer text report on stdout, plus a JSON document. Each part record references the original GLB **node** and **geometry** so callers can map the order straight back onto the asset: ```jsonc { "source": "backyard-204-parts.glb", "up_axis": "Y", "floor_y": 0.0, "tolerance": 0.02, "part_count": 196, "layer_count": 9, "ground_count": 17, "orphan_count": 0, "placement": [ { "order": 27, // 0-based position in the global sequence "layer": 2, // BFS depth from the ground (null = orphan) "node": "skid-1", // GLB scene-graph node name "geometry": "GLTF_6", // GLB mesh/geometry name "touches_ground": false, "bbox_min": [-5.0, 0.6, -4.0], "bbox_max": [ 5.0, 1.058, -3.708], "supports": ["deck-block-1-1", "deck-block-1-2", "deck-block-1-3"], "contacts": ["floor-joist-1", "...", "deck-block-1-1"] } ] } ``` - `supports` — the already-placed parts (strictly earlier layers) this part rests on. These are the depende [truncated — 3826 more characters] ``` ### BUILDING SOURCES.md ```markdown # Building Sources This file catalogs the external building references, product catalogs, PDFs, and implementation sources used to design Backyard Intelligence. Backyard Intelligence is a concept-design prototype. These sources informed defaults, component families, safety warnings, BOM categories, and verifier checks. They do not make generated designs permit-ready or structurally certified. ## Local PDF used as AI context ### How-To-Build-A-Shed-eBook.pdf - Local file: [`How-To-Build-A-Shed-eBook.pdf`](How-To-Build-A-Shed-eBook.pdf) - Used by: `vite.config.ts` AI middleware and `src/domain/geminiSchema.ts` - Purpose: - Included as inline PDF context for Gemini and Claude design turns. - Guides shed specification defaults: floor framing, wall framing, openings, roof framing, sheathing, siding, trim, and fastener schedules. - Helps the AI convert natural-language shed prompts into lower-level structured specs. - Important limitation: - Used as construction guidance, not code approval or engineering certification. ## High-level construction guides ### The Home Depot — How to Build a Pergola - URL: https://www.homedepot.com/c/ah/how-to-build-a-pergola/9ba683603be9fa5395fab9017c56b4eb - Used by: `docs/design-basis.md`, pergola defaults, pergola component families. - Informed: - Freestanding pergola assembly order. - Post bases, posts, beams, rafters, braces, and exterior fasteners. - Cut-end treatment and local anchorage/code warnings. ### The Home Depot — How to Build a Shed - URL: https://www.homedepot.com/c/ah/how-to-build-a-shed/9ba683603be9fa5395fab90434412ca - Used by: `docs/design-basis.md`, shed generator defaults. - Informed: - Deck-block or pier-style support. - Pressure-treated perimeter/floor framing. - Joist spacing. - 2×4 wall framing. - Bottom and double top plates. - Framed openings. - T1-11 siding. - Repeatable roof framing. ### The Home Depot Canada — How to Build a Shed Floor - URL: https://ampservices.homedepot.com/en/home/ideas-how-to/outdoors/outdoor-living/how-to-build-a-shed-floor.html - Used by: `docs/design-basis.md`, raised-floor shed logic. - Informed: - Pressure-treated 2×6 floor framing. - 16-inch-on-center joists. - Joist hangers. - 3/4-inch exterior plywood decking. - Staggered joints and exterior fasteners. ### The Home Depot — Pro Guide to Roof Framing - URL: https://www.homedepot.com/c/ah/pro-guide-to-roof-framing/9ba683603be9fa5395fab90188285eb8 - Used by: `docs/design-basis.md`, gable/lean-to roof generator. - Informed: - Rise/run/pitch terminology. - Gable geometry. - Rafter/truss-style repeatable roof member generation. ## Safety, code, and connector references ### American Wood Council — DCA 6 Prescriptive Residential Wood Deck Construction Guide - URL: https://awc.org/wp-content/uploads/2022/02/AWC-DCA62015-DeckGuide-1804.pdf - Earlier design-basis URL also referenced: https://awc.org/wp-content/uploads/2022/02/AWC-DCA62012-DeckGuide-1405.pdf - Us [truncated — 7523 more characters] ``` ### package.json ``` { "name": "backyard-intelligence", "private": true, "version": "0.1.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "preview": "vite preview", "test:generator": "esbuild scripts/generator-sweep.ts --bundle --platform=node --format=esm --outfile=/tmp/backyard-generator-sweep.mjs && node /tmp/backyard-generator-sweep.mjs" }, "dependencies": { "@react-three/drei": "^10.7.4", "@react-three/fiber": "^9.4.0", "lucide-react": "^0.468.0", "react": "^19.0.0", "react-dom": "^19.0.0", "three": "^0.180.0" }, "devDependencies": { "@types/react": "^19.0.0", "@types/react-dom": "^19.0.0", "@types/three": "^0.180.0", "@vitejs/plugin-react": "^4.3.4", "typescript": "~5.7.2", "vite": "^6.0.5" } } ``` ### src/main.tsx ```typescript import { StrictMode } from 'react' import { createRoot } from 'react-dom/client' import App from './App' createRoot(document.getElementById('root')!).render() ``` ### src/pipeline/index.ts ```typescript import type { ConversationTurn, DesignSpec, GeminiLogEntry, PipelineResult, StructureFamily } from '../types' import { interpretDeterministic, interpretPrompt, interpretWithGemini, validateSpec } from './interpreter' import { generateModel } from './generator' import { createBOM } from './bom' import { verify } from './verifier' import { reviewSafetyGuidelines } from './safety' export function runPipeline(prompt:string,family?:StructureFamily):PipelineResult{const spec=interpretPrompt(prompt,family);const errors=validateSpec(spec);if(errors.length)throw new Error(errors.join(' '));const model=generateModel(spec);const bom=createBOM(spec,model);const verification=verify(spec,model,bom),safety=reviewSafetyGuidelines(spec,model);return{spec,model,verification,safety,bom,repairAttempts:0}} export type PipelineStage='gemini'|'validate'|'generate'|'bom'|'verify'|'repair'|'complete' const viteEnv=()=>((import.meta as unknown as {env?:Record}).env||{}) const envFlag=(key:string,defaultValue=false)=>{const value=viteEnv()[key];if(value===undefined)return defaultValue;return ['1','true','yes','on'].includes(String(value).toLowerCase())} export const deterministicOnly=()=>envFlag('VITE_DETERMINISTIC_ONLY') export const aiRepairEnabled=()=>envFlag('VITE_AI_REPAIR',true) function runDeterministic(prompt:string,family:StructureFamily|undefined,currentSpec:DesignSpec|null,onStage?:(stage:PipelineStage)=>void):PipelineResult{onStage?.('validate');const spec=interpretDeterministic(prompt,family,currentSpec),errors=validateSpec(spec);if(errors.length)throw new Error(errors.join(' '));onStage?.('generate');const model=generateModel(spec);onStage?.('bom');const bom=createBOM(spec,model);onStage?.('verify');const verification=verify(spec,model,bom),safety=reviewSafetyGuidelines(spec,model);onStage?.('complete');return{spec,model,verification,safety,bom,repairAttempts:0}} export async function runPipelineWithGemini(prompt:string,family:StructureFamily|undefined,currentSpec:DesignSpec|null,history:ConversationTurn[],onStage?:(stage:PipelineStage)=>void,onLog?:(entry:GeminiLogEntry)=>void,defaultDesignControls?:unknown):Promise{ if(deterministicOnly())return runDeterministic(prompt,currentSpec?.family||family,currentSpec,onStage) const repairWithAi=aiRepairEnabled() onStage?.('gemini');let spec=await interpretWithGemini(prompt,family,currentSpec,history,[],onLog,defaultDesignControls) for(let attempt=0;attempt<=2;attempt++){ onStage?.('validate');const errors=validateSpec(spec) if(errors.length){ if(!repairWithAi)throw new Error(`Design needs deterministic review before generation: ${errors.join(' ')}`) if(attempt===2)throw new Error(errors.join(' '));onStage?.('repair');spec=await interpretWithGemini('Repair the construction specification without changing the requested design intent.',spec.family,spec,history,errors,onLog,defaultDesignControls);continue } onStage?.('generate');const model=generateModel(spec) onStage?.('bom');const bom=createBOM(spec,model) onStage?.('verify');const verification=verify(spec,model,bom) if(verification.valid){const safety=reviewSafetyGuidelines(spec,model);onStage?.('complete');return{spec,model,verification,safety,bom,repairAttempts:attempt}} if(!repairWithAi)throw new Error(`Design failed deterministic verification: ${verification.checks.filter(c=>c.status==='fail').map(c=>c.detail).join(' ')}`) if(attempt===2)throw new Error(`Design could not be repaired: ${verification.checks.filter(c=>c.status==='fail').map(c=>c.detail).join(' ')}`) onStage?.('repair');const issues=verification.checks.filter(c=>c.status==='fail').map(c=>`${c.id}: ${c.detail}`);spec=await interpretWithGemini('Repair the construction specification so the deterministic component model passes verification. Preserve the user intent.',spec.family,spec,history,issues,onLog,defaultDesignControls) } throw new Error('Design repair loop ended unexpectedly.') } ``` ### index.html ```html
``` ### convert.sh ```shell #!/usr/bin/env bash # Plug-and-play GLB -> USDZ converter for BoundaryGhost. # # Usage: ./convert.sh path/to/model.glb # ./convert.sh # auto-uses the single *.glb in this dir # # Output is ALWAYS written to BoundaryGhost/ghost-model.usdz — the one fixed name # the app loads (see ModelGhostBuilder.assetName). Because the Xcode group is # file-system-synchronized, the new asset is bundled on the next build with zero # project edits. Any older bundled model is removed first. set -euo pipefail here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" py="$here/.venv/bin/python" out="$here/BoundaryGhost/ghost-model.usdz" # Resolve the source GLB. src="${1:-}" if [[ -z "$src" ]]; then shopt -s nullglob globs=("$here"/*.glb) if [[ ${#globs[@]} -ne 1 ]]; then echo "error: pass a .glb path, or keep exactly one *.glb at the repo root (found ${#globs[@]})." >&2 exit 1 fi src="${globs[0]}" fi [[ -f "$src" ]] || { echo "error: no such file: $src" >&2; exit 1; } [[ -x "$py" ]] || { echo "error: venv python missing at $py — run: python3 -m venv .venv && .venv/bin/pip install trimesh usd-core numpy" >&2; exit 1; } # Clear any previously bundled model so only one ships. rm -f "$here"/BoundaryGhost/*.usdz "$here"/BoundaryGhost/*.usdc echo "Converting: $src -> $out" "$py" "$here/glb_to_usdz.py" "$src" "$out" echo "Done. Rebuild the app in Xcode to pick up the new model." ``` ### glb_to_usdz.py ```python #!/usr/bin/env python3 """Convert a GLB file to USDZ using trimesh (GLB import) + pxr/USD (USD authoring). Handles the scene graph (per-node world transforms), mesh geometry, vertex normals, UVs, and PBRMetallicRoughness -> UsdPreviewSurface materials. glTF and USD are both Y-up / right-handed in meters, so no axis conversion. """ import sys import numpy as np import trimesh from pxr import Usd, UsdGeom, UsdShade, Sdf, Gf, Vt def sanitize(name, used): safe = "".join(c if c.isalnum() else "_" for c in str(name)) if not safe or safe[0].isdigit(): safe = "_" + safe base, i = safe, 1 while safe in used: safe, i = f"{base}_{i}", i + 1 used.add(safe) return safe def main(src, dst): scene = trimesh.load(src, process=False) if isinstance(scene, trimesh.Trimesh): scene = trimesh.Scene(scene) stage = Usd.Stage.CreateInMemory() UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.y) UsdGeom.SetStageMetersPerUnit(stage, 1.0) root = UsdGeom.Xform.Define(stage, "/Root") stage.SetDefaultPrim(root.GetPrim()) mats_scope = "/Root/Materials" UsdGeom.Scope.Define(stage, mats_scope) mat_cache = {} used_mat_names = set() used_mesh_names = set() def norm_color(c, default): # trimesh returns glTF color factors as uint8 (0-255); USD wants 0-1 floats. if c is None: return tuple(default) a = np.asarray(c) a = a.astype(np.float64) if np.issubdtype(np.asarray(c).dtype, np.integer) or a.max(initial=0.0) > 1.0: a = a / 255.0 return tuple(a.tolist()) def get_material(m): bc = norm_color(getattr(m, "baseColorFactor", None), (1, 1, 1, 1)) metallic = float(getattr(m, "metallicFactor", 0.0) or 0.0) rough = float(getattr(m, "roughnessFactor", 1.0) or 1.0) emis = norm_color(getattr(m, "emissiveFactor", None), (0, 0, 0)) key = (bc, metallic, rough, emis) if key in mat_cache: return mat_cache[key] mname = sanitize(getattr(m, "name", None) or "mat", used_mat_names) mpath = f"{mats_scope}/{mname}" mat = UsdShade.Material.Define(stage, mpath) shader = UsdShade.Shader.Define(stage, f"{mpath}/Surface") shader.CreateIdAttr("UsdPreviewSurface") shader.CreateInput("diffuseColor", Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*bc[:3])) shader.CreateInput("opacity", Sdf.ValueTypeNames.Float).Set(float(bc[3]) if len(bc) > 3 else 1.0) shader.CreateInput("metallic", Sdf.ValueTypeNames.Float).Set(metallic) shader.CreateInput("roughness", Sdf.ValueTypeNames.Float).Set(rough) if any(emis): shader.CreateInput("emissiveColor", Sdf.ValueTypeNames.Color3f).Set(Gf.Vec3f(*emis[:3])) mat.CreateSurfaceOutput().ConnectToSource(shader.ConnectableAPI(), "surface") mat_cache[key] = mat return mat count = 0 for node in scene.graph.nodes_geometry: transform, geom_name = scene.graph.get(node) geom = scene.geometry.get(geom_name) if geom is None or not hasattr(geom, "vertices"): continue g = geom.copy() g.apply_transform(transform) # bake world transform into points mesh_name = sanitize(node, used_mesh_names) mesh = UsdGeom.Mesh.Define(stage, f"/Root/{mesh_name}") mesh.CreatePointsAttr(Vt.Vec3fArray.FromNumpy(g.vertices.astype(np.float32))) faces = g.faces.astype(np.int32) mesh.CreateFaceVertexIndicesAttr(Vt.IntArray.FromNumpy(faces.reshape(-1))) mesh.CreateFaceVertexCountsAttr(Vt.IntArray.FromNumpy(np.full(len(faces), 3, np.int32))) mesh.CreateSubdivisionSchemeAttr(UsdGeom.Tokens.none) try: normals = g.vertex_normals.astype(np.float32) if len(normals) == len(g.vertices): mesh.CreateNormalsAttr(Vt.Vec3fArray.FromNumpy(normals)) mesh.SetNormalsInterpolation(UsdGeom.Tokens.vertex) except Exception: pass mat = getattr(getattr(geom, "visual", None), "material", None) if mat is not None: binding = UsdShade.MaterialBindingAPI.Apply(mesh.GetPrim()) binding.Bind(get_material(mat)) count += 1 # Write the layer to a temp .usdc, then package as usdz. Keeping the # intermediate in a temp dir avoids polluting the app bundle with a stray .usdc. import os, tempfile from pxr import UsdUtils with tempfile.TemporaryDirectory() as tmpdir: tmp_usd = os.path.join(tmpdir, "stage.usdc") stage.GetRootLayer().Export(tmp_usd) if not UsdUtils.CreateNewUsdzPackage(tmp_usd, dst): raise RuntimeError("CreateNewUsdzPackage failed") print(f"OK: {count} meshes, {len(mat_cache)} materials -> {dst}") if __name__ == "__main__": main(sys.argv[1], sys.argv[2]) ``` ### placement_order.py ```python #!/usr/bin/env python3 """Compute a bottom-up assembly/placement order for the parts in a GLB file. The rule (from the spec): 1. Place every part that touches the ground first. 2. Then every remaining part that touches an already-placed part. 3. Repeat until nothing new can be placed. That is a breadth-first layering of the *contact graph* seeded from the parts sitting on the floor. Layer 0 = on the ground, layer 1 = resting on a layer-0 part, and so on. Within a layer parts are ordered low-to-high so the output is a sensible, reproducible build sequence. Contact model ------------- glTF parts here are essentially planks/boxes, so each part is approximated by its (world-space) oriented bounding box and two parts are "touching" when their OBBs are within ``--tol`` of each other. Detection is two-phase: * broadphase -- axis-aligned bounding boxes inflated by ``tol`` must overlap, * narrowphase -- separating-axis test (SAT) between the two oriented boxes. This is exact for box-like parts and needs only numpy (no fcl/rtree/scipy). Up axis is glTF-standard +Y; the floor is the global minimum Y over all parts. Usage ----- python placement_order.py model.glb [--tol 0.02] [--json out.json] Output: a human-readable layer-by-layer report on stdout, plus a machine readable JSON document (``--json`` path, default ``.placement.json``) that lists, in order, every part with its node reference, geometry reference, layer, bounding box, and the already-placed parts it rests on ("supports"). """ from __future__ import annotations import argparse import json import sys from collections import deque import numpy as np import trimesh # --------------------------------------------------------------------------- # # Geometry helpers # --------------------------------------------------------------------------- # def sanitize(name, used): """Sanitize a node name to a USD-prim-safe identifier. Kept byte-for-byte identical to glb_to_usdz.py's sanitizer (same rule, same de-dup) so the names emitted here match the prim names RealityKit sees in the converted USDZ — that is what lets the app map placement order onto entities. """ safe = "".join(c if c.isalnum() else "_" for c in str(name)) if not safe or safe[0].isdigit(): safe = "_" + safe base, i = safe, 1 while safe in used: safe, i = f"{base}_{i}", i + 1 used.add(safe) return safe class Part: """One placeable component: a geometry node with its baked world transform.""" __slots__ = ("index", "node", "geometry", "usd_name", "verts", "aabb_lo", "aabb_hi", "obb_center", "obb_axes", "obb_half") def __init__(self, index, node, geometry, usd_name, verts): self.index = index self.node = node self.geometry = geometry self.usd_name = usd_name # prim name in the converted USDZ self.verts = verts # (n, 3) world-space vertices self.aabb_lo = verts.min(axis=0) self.aabb_hi = verts.max(axis=0) # Oriented bounding box in world space. to_origin, extents = trimesh.bounds.oriented_bounds(verts) local_to_world = np.linalg.inv(to_origin) self.obb_center = local_to_world[:3, 3] self.obb_axes = local_to_world[:3, :3] # unit columns = box axes in world self.obb_half = np.asarray(extents) * 0.5 def load_parts(src): """Return the list of Parts from a GLB/GLTF file, with world transforms baked.""" scene = trimesh.load(src, process=False) if isinstance(scene, trimesh.Trimesh): scene = trimesh.Scene(scene) parts = [] used_names = set() # shared de-dup state, same order as the converter for node in scene.graph.nodes_geometry: transform, geom_name = scene.graph.get(node) geom = scene.geometry.get(geom_name) if geom is None or not hasattr(geom, "vertices") or len(geom.vertices) == 0: continue usd_name = sanitize(node, used_names) verts = trimesh.transformations.transform_points(geom.vertices, transform) parts.append(Part(len(parts), node, geom_name, usd_name, verts)) return parts def aabb_overlap(a, b, tol): """True if a and b's axis-aligned boxes overlap once inflated by tol.""" return bool(np.all(a.aabb_lo - tol <= b.aabb_hi) and np.all(b.aabb_lo - tol <= a.aabb_hi)) def obb_touch(a, b, tol): """Separating-axis test between two oriented boxes, with a tolerance gap. Returns True when no separating axis leaves a gap greater than ``tol`` -- i.e. the boxes are touching or within tol of each other. """ ca, cb = a.obb_center, b.obb_center A, B = a.obb_axes, b.obb_axes # columns are the box axes ea, eb = a.obb_half, b.obb_half d = cb - ca # Candidate separating axes: 3 of A, 3 of B, 9 cross products. axes = [] for i in range(3): axes.append(A[:, i]) for j in range(3): axes.append(B[:, j]) for i in range(3): for j in range(3): axes.append(np.cross(A[:, i], B[:, j])) for L in axes: n = np.linalg.norm(L) if n < 1e-9: # parallel axes -> degenerate cross continue L = L / n # Projected half-widths of each box onto L. ra = float(np.sum(np.abs(ea * (A.T @ L)))) rb = float(np.sum(np.abs(eb * (B.T @ L)))) dist = abs(float(d @ L)) if dist > ra + rb + tol: # a gap > tol on this axis -> separated return False return True # --------------------------------------------------------------------------- # # Contact graph + bottom-up layering # --------------------------------------------------------------------------- # def build_contacts(parts, tol): """Return adjacency (list of sets) of touching parts via AABB+OBB tests.""" n = len(parts) adj = [set() fo [truncated — 7489 more characters] ``` ### vite.config.ts ```typescript import { defineConfig, loadEnv } from 'vite' import react from '@vitejs/plugin-react' import { readFileSync } from 'node:fs' import { createHash, createHmac } from 'node:crypto' import { designResponseSchema, designSpecSchema, geminiSystemPrompt } from './src/domain/geminiSchema' type Attempt={provider:'gemini'|'claude';model:string;attempt:number;status:number;retryable:boolean;detail:string} class ProviderFailure extends Error{status:number;retryable:boolean;constructor(message:string,status=502,retryable=true){super(message);this.status=status;this.retryable=retryable}} const retryableStatus=(status:number)=>[429,500,502,503,504].includes(status) const wait=(ms:number)=>new Promise(resolve=>setTimeout(resolve,ms)) const claudeResponseSchema={type:'object',additionalProperties:false,properties:{kind:{type:'string',enum:['questions','spec']},message:{type:'string'},questions:{type:'array',items:{type:'object',additionalProperties:false,properties:{id:{type:'string'},question:{type:'string'},why:{type:'string'},required:{type:'boolean'}},required:['id','question','why','required']}},specJson:{type:'string'}},required:['kind','message','questions','specJson']} const schemaForPrompt=(value:any):any=>{if(Array.isArray(value))return value.map(schemaForPrompt);if(!value||typeof value!=='object')return value;const result:any={};for(const [key,item] of Object.entries(value))result[key]=key==='type'&&typeof item==='string'?({OBJECT:'object',ARRAY:'array',STRING:'string',NUMBER:'number',BOOLEAN:'boolean'}[item]||item.toLowerCase()):schemaForPrompt(item);return result} const claudeSpecBlueprint=JSON.stringify(schemaForPrompt(designSpecSchema)) const compatibleSpec=(spec:any)=>!!spec&&typeof spec==='object'&&['pergola','shed','garden-bed'].includes(spec.family)&&typeof spec.label==='string'&&spec.dimensions&&[spec.dimensions.width,spec.dimensions.depth,spec.dimensions.height].every((n:unknown)=>typeof n==='number'&&Number.isFinite(n))&&spec.construction&&typeof spec.construction==='object' const collectBody=(req:any)=>new Promise((resolve,reject)=>{const chunks:Buffer[]=[];req.on('data',(chunk:Buffer)=>chunks.push(Buffer.from(chunk)));req.on('error',reject);req.on('end',()=>resolve(Buffer.concat(chunks)))}) const hashHex=(value:Buffer|string)=>createHash('sha256').update(value).digest('hex') const hmac=(key:Buffer|string,value:string)=>createHmac('sha256',key).update(value).digest() const awsDateParts=()=>{const iso=new Date().toISOString().replace(/[:-]|\.\d{3}/g,'');return{amzDate:iso,dateStamp:iso.slice(0,8)}} const signS3Put=({region,bucket,key,body,accessKeyId,secretAccessKey,sessionToken,contentType}:{region:string;bucket:string;key:string;body:Buffer;accessKeyId:string;secretAccessKey:string;sessionToken?:string;contentType:string})=>{ const {amzDate,dateStamp}=awsDateParts(),host=`${bucket}.s3.${region}.amazonaws.com`,payloadHash=hashHex(body),encodedKey=key.split('/').map(encodeURIComponent).join('/'),headers:Record={'cache-control':'no-cache, max-age=0','content-type':contentType,host,'x-amz-content-sha256':payloadHash,'x-amz-date':amzDate} if(sessionToken)headers['x-amz-security-token']=sessionToken const signedHeaders=Object.keys(headers).sort().join(';'),canonicalHeaders=Object.keys(headers).sort().map(name=>`${name}:${headers[name]}\n`).join(''),canonicalRequest=['PUT',`/${encodedKey}`,'',canonicalHeaders,signedHeaders,payloadHash].join('\n'),credentialScope=`${dateStamp}/${region}/s3/aws4_request`,stringToSign=['AWS4-HMAC-SHA256',amzDate,credentialScope,hashHex(canonicalRequest)].join('\n'),signingKey=hmac(hmac(hmac(hmac(`AWS4${secretAccessKey}`,dateStamp),region),'s3'),'aws4_request'),signature=createHmac('sha256',signingKey).update(stringToSign).digest('hex') return{url:`https://${host}/${encodedKey}`,headers:{...headers,authorization:`AWS4-HMAC-SHA256 Credential=${accessKeyId}/${credentialScope}, SignedHeaders=${signedHeaders}, Signature=${signature}`}} } const s3UploadPlugin=()=>{ let bucket='backyard-intelligence',region='us-west-1',key='current.glb',accessKeyId='',secretAccessKey='',sessionToken='' const middleware=async(req:any,res:any)=>{ res.setHeader('content-type','application/json') const publicUrl=`https://${bucket}.s3.${region}.amazonaws.com/${key}` if(req.method==='GET')return res.end(JSON.stringify({bucket,region,key,url:publicUrl,configured:!!accessKeyId&&!!secretAccessKey,uploadMode:!!accessKeyId&&!!secretAccessKey?'signed':'unsigned-public-put'})) if(req.method!=='POST'){res.statusCode=405;return res.end(JSON.stringify({error:'Use POST with model/gltf-binary body.'}))} try{ const body=await collectBody(req) if(!body.length)throw new Error('No GLB bytes were sent.') const contentType='model/gltf-binary',hasCredentials=!!accessKeyId&&!!secretAccessKey,uploadMode=hasCredentials?'signed':'unsigned-public-put',signed=hasCredentials?signS3Put({region,bucket,key,body,accessKeyId,secretAccessKey,sessionToken,contentType}):null const put=hasCredentials&&signed ? await fetch(signed.url,{method:'PUT',headers:signed.headers,body}) : await fetch(publicUrl,{method:'PUT',headers:{'content-type':contentType,'cache-control':'no-cache, max-age=0'},body}) if(!put.ok){const detail=await put.text(),hint=hasCredentials?'Check the configured AWS credentials and bucket/key permissions.':`The bucket may be public-read, but S3 still rejected anonymous overwrite. To write without credentials, the bucket policy must allow public s3:PutObject for ${key}; otherwise add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.`;res.statusCode=put.status;return res.end(JSON.stringify({error:`S3 upload failed with ${put.status}: ${detail.slice(0,300)} ${hint}`,bucket,region,key,url:publicUrl,uploadMode}))} const verified=await fetch(`${publicUrl}?verify=${Date.now()}`,{method:'HEAD',cache:'no-store'}).then(r=>r.ok).catch(()=>false) if(!verified){res.statusCode=502;return res.end(JSON.stringify({er [truncated — 8404 more characters] ``` [49 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]