Backyard Intelligence

Backyard Intelligence is a Berkeley AI Hackathon prototype for turning a natural-language backyard project idea into a structured, inspectable build concept.

The core architecture is:

natural language → AI-interpreted design spec → deterministic component generation → verification → BOM + GLB + AR handoff

The app currently supports:

• Pergolas
• Sheds
• Raised garden beds

It is not a permit package or structural certification tool. The goal is to prove a safer AI-assisted architecture where language models interpret intent, while deterministic code owns geometry, verification, bill of materials, and export.

Why this uses AI

The AI layer is used where language models are strongest:

• Understanding messy user prompts like “make it a 10x8 gable shed with double doors and two left windows.”
• Asking one approval-style follow-up when the prompt is ambiguous.
• Converting conversation history and user revisions into a complete typed construction specification.
• Using the provided shed PDF as design context when forming the spec.
• Falling back across Gemini and Claude providers when one model is unavailable.
• Optionally attempting repair prompts when deterministic validation or verification fails.

The AI does not guess the final BOM, mesh geometry, or verifier output. Those are produced from deterministic code.

System diagram

flowchart TD
    A["User prompt or revision"] --> B["Structure router<br/>Any / Pergola / Shed / Garden bed"]
    B --> C["AI interpreter<br/>Gemini primary, Claude fallback"]
    C --> D{"Ambiguity left?"}
    D -- "Yes" --> E["AI asks one approval-style<br/>default confirmation question"]
    E --> A
    D -- "No" --> F["Structured DesignSpec v2<br/>intent, dimensions, stock IDs,<br/>openings, roof/foundation rules"]
    F --> G["Deterministic spec validator"]
    G --> H{"Spec valid?"}
    H -- "No" --> I["AI repair or recommended fix<br/>depending on env mode"]
    I --> F
    H -- "Yes" --> J["Deterministic component generator<br/>named part nodes, dimensions,<br/>connections, materials"]
    J --> K["Model verifier<br/>required parts, stock fit,<br/>joint fit, support path,<br/>overlaps, openings"]
    K --> L{"Model valid?"}
    L -- "No" --> I
    L -- "Yes" --> M["BOM generator<br/>derived from generated components"]
    M --> N["Safety / guideline checklist<br/>shed + pergola advisory checks"]
    N --> O["GLB exporter<br/>one named mesh per visible part"]
    O --> P["UI response<br/>spec, model, verification,<br/>BOM, safety checklist, GLB"]
    O --> Q["Optional S3 current.glb upload<br/>for AR viewer"]

How the system works

1. The user describes a structure in chat.
2. The app routes the prompt to a family: pergola, shed, garden bed, or “any.”
3. Gemini or Claude converts the prompt into a complete structured design specification.
4. If key choices are ambiguous, the AI proposes sensible defaults from the current sliders and asks the user to approve or modify them.
5. The deterministic validator checks the spec before any geometry is generated.
6. The component generator creates the actual construction model from reusable part rules.
7. The verifier checks that the generated model is internally consistent.
8. The BOM is derived from generated component nodes, not from AI text.
9. The GLB exporter creates a true-scale model with one named node per visible part.
10. For sheds and pergolas, a safety/building-guideline checklist compares the design against conventional guidance and flags advisory issues.

Generated artifacts

Each successful design returns separate artifacts:

• Design spec — what the user asked for, normalized into typed construction data.
• Component model — what the deterministic generator actually built.
• Verification report — whether the generated model is internally consistent.
• Safety checklist — advisory shed/pergola guideline checks based on sources like AWC, IRC, and Simpson Strong-Tie.
• BOM — materials and purchase quantities derived from component nodes.
• GLB — one named mesh per visible component, with material colors for true-scale 3D/AR viewing.

Keeping these separate is the main safety pattern: the language model describes intent, but deterministic code produces and checks the buildable representation.

Supported structure families

Pergola

Generated assemblies include:

• Concrete footings
• Post bases
• 6x6 posts
• Beam assemblies
• Rafters
• Shade slats
• Knee braces
• Rafter ties / connector markers
• Reference figure, compass, and dimension guides

The generator segments over-length lumber when the selected stock is shorter than the requested run.

Shed

Generated assemblies include:

• Prepared base
• Deck blocks or concrete slab
• Floor skids, joists, and floor deck when raised-floor mode is enabled
• Wall plates, studs, headers, and opening framing
• Door/window units
• Siding, trim, flashing, sealant packages
• Gable or lean-to roof framing
• Roof sheathing, underlayment, shingles, drip edge, fascia, and rake trim
• Fastener and connector point markers

Raised garden bed

Generated assemblies include:

• Side boards
• Corner and intermediate posts
• Top caps
• Optional bottom mesh / liner
• Optional trellis
• Soil and compost volume estimate

AI providers and modes

Server-side AI credentials stay in Vite middleware and are not shipped to the browser.

GEMINI_API_KEY=...
ANTHROPIC_API_KEY=...
AI_PROVIDER_ORDER=gemini,claude

By default, the app uses Gemini first and can fall back to Claude if configured.

Useful modes:

# Local deterministic parser only; no AI interpretation or AI repair.
VITE_DETERMINISTIC_ONLY=true

# AI still parses intent, but failed deterministic review surfaces a recommended fix
# instead of running automatic AI repair turns.
VITE_AI_REPAIR=false

Run locally

npm install
cp .env.example .env
# Add GEMINI_API_KEY and/or ANTHROPIC_API_KEY to .env
npm run dev

AR current GLB upload

Every successful generation exports the same GLB used by the download button and POSTs it to the local server at /api/glb/current. The server overwrites:

https://backyard-intelligence.s3.us-west-1.amazonaws.com/current.glb

Public read access lets the AR viewer fetch the file, but it does not automatically allow overwriting it. The upload endpoint supports both modes:

• With AWS credentials, it performs a signed PUT.
• Without AWS credentials, it tries a public write to the object URL. That succeeds only if the bucket policy explicitly allows anonymous s3:PutObject for current.glb.

Optional server-side AWS credentials:

S3_GLB_BUCKET=backyard-intelligence
S3_GLB_REGION=us-west-1
S3_GLB_KEY=current.glb
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

The UI shows whether current.glb uploaded and whether the public AR URL verified.

Verification and safety checks

The internal verifier checks generated geometry and data consistency:

• Required component groups exist
• Dimensions are finite and positive
• Stock lengths fit selected Home Depot-oriented materials
• Connections reference real generated components
• Declared joints physically meet
• Unconnected solid geometry does not overlap
• Openings fit walls
• Roof planes and rafters touch correctly
• BOM traces back to generated components

The safety checklist is advisory and currently applies to sheds and pergolas. It checks the generated spec/model against conventional guidance for:

• Foundation and post/base support
• Exterior-rated materials
• Framing spacing
• Bracing
• Uplift connector locations
• Roof/weathering layers
• Siding and sheathing fastener schedules
• Local permit, frost, wind, snow, seismic, and engineering review warnings

Research sources used in the app:

• American Wood Council DCA6 deck guide
• Simpson Strong-Tie Wood Construction Connectors catalog
• ICC 2021 IRC

Design basis

The researched assembly conventions and limitations are documented in docs/design-basis.md.

This is a concept-design prototype. Always verify local code, permits, loads, soil, frost depth, manufacturers’ instructions, and professional structural requirements before building.