# Project export: StudForge This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: StudForge turns natural language prompts into physics-validated, simulator-ready, robot-buildable swarm robotics block-based worlds with measurable training yield. - Devpost: https://devpost.com/software/studforge - GitHub: https://github.com/kokonut121/studsim - Video: https://www.youtube.com/embed/-LjeM4XZLQg?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Result: winner (Finalist) - Team: 4 GitHub contributor(s) — Patrick Feng (31 commits), Claude Opus 4.8 (22 commits), jerryjlwang (16 commits), Micah (14 commits) ## Devpost submission (written by the team) ### Inspiration Three of us grew up building with LEGO before we ever wrote a line of code, and it turns out stud-based bricks are a surprisingly good metaphor for how robotics simulation should work: small, validated, snap-together units that compose into something physically real. The problem is that building a multi-robot sim scene today (a warehouse, a search-and-rescue maze, a swarm-coordination test) usually means hand-placing geometry or trusting a generative model to hallucinate a "plausible" environment — with no guarantee the result is collision-free, connected, or even physically assemblable. ### What it does StudForge turns a plain-English prompt (both voice and text) — "a 4-robot search-and-rescue maze with two narrow corridors, victims, debris, and a recharge pad" — into a fully validated, exportable multi-robot training environment, end to end. The system compiles a user prompt into a typed SceneSpec using Claude with schema-constrained generation, optionally accepting voice prompts through Deepgram’s nova-3 live transcription. It renders an advisory visual prior with Midjourney via its MCP server so designers can preview the scene before committing geometry, but the image only softly nudges wall placement and can never bypass validation. From there, the platform assembles and compresses the scene into stud-based brick geometry using deterministic and learned compression planners, including a Qwen model path, while gating every proposal through the same validator suite as the deterministic baseline. It validates connectivity, collision-freedom, and coverage, repairing and rechecking until the scene is sound rather than merely plausible. The system then exports a canonical USD representation, with SDF, URDF, and MJCF as downstream exports, and runs a lightweight swarm simulation with a real multi-robot team plan. During each episode, three observer agents for coverage, coordination, and collision write findings to a Redis-backed agent-memory service, then recall only bounded, validator-gated bias into the next generation, letting the system improve its training environments over time without allowing memory agents to write geometry directly. Finally, it reports the full lineage of every decision, including compiler output, compression report, validation report, and swarm metrics, as both JSON and an HTML report. ### How we built it The core is a Python backend built around Pydantic v2 contracts that define the exact I/O of every pipeline stage. For the agent layer, we used Claude to generate per-robot configs and a runnable scripted policy from the compiled EpisodeManifest, with a learned-policy slot wired in. The three observer agents persist their findings through a shared memory store with strict Redis Cloud selection. For compression, we trained and benchmarked deterministic strategies (unit, strips, greedy, exact) as a validator-gated floor, and wired a learned/hybrid path that runs inference against a Qwen 2.5 model. ### Challenges we ran into Letting Claude/Midjourney/Deepgram advise without letting them author. The hardest design constraint was making three different generative integrations strictly advisory. We initially hoped to create scene reconstructions out of Midjourney generations but quickly realized the model was too probabilistic to paint reliably consistent scene graphs. It would have been much faster to let the visual prior or the compiler "just place the bricks," but that breaks the validation guarantee the whole project is built on — we spent real time engineering the boundary so creativity stays upstream of correctness. Voice latency vs. correctness. Wiring the Deepgram WebSocket relay so transcripts land cleanly in the same compiler path as typed prompts — including recording input modality and STT model in provenance — took more debugging than expected, especially keeping the typed-text fallback fully intact when no API key is set. Trainium Pivot Our original idea used a fine-tuned compression model to convert high-fidelity USD assets into more functional blocks, yet around 9pm the trainium instance we had been working with shut down, forcing us to completely restart and train a Qwen2.5-1.5B model on a smaller EC2 instance with LoRA. ### Accomplishments we're proud of We're proud that the entire pipeline runs cleanly and produces a schema-valid scene, a zero-coverage-error compression report, a connected single-component layout, and a full observer report from all three agents. Layered on top of that honest baseline, we got Claude-driven compilation, Midjourney visual priors, Deepgram live voice input, and Redis-backed agent memory all genuinely wired into the same validated path. Getting four sponsor integrations to coexist behind one settings module, with health reporting on all nine, is the part we're most proud of engineering-wise. ### What we learned We learned that validation is gold in compiler structure. Every time we added a new model in the loop — Claude, Midjourney, Deepgram, the Qwen-based compression path — the temptation was to let it shortcut the validator. Building the advisory boundary first, and writing tests that assert validators run regardless of which backend produced the input, is what let us add four different model integrations without the system's core guarantee ever weakening. We also learned that strict failure modes (Redis, in particular) are a feature for systems that claim to demonstrate agent memory, not a bug to be smoothed over. ### What's next Finish the scene playback endpoint so the frontend can animate generated robot trajectories live instead of falling back to a static view. Train the learned robot-agent policy now that the compressed scene landscape gives us a real distribution to train against. We can now scale beyond search-and-rescue, exploring warehouse workers, assembly lines, and more. Move from JSON USD stand-ins to real OpenUSD export, and connect the swarm sim to higher-fidelity backends (Isaac, MuJoCo, BenchMARL) for sim-to-real staging. ## README (from the GitHub repository) # StudForge Text-to-brick modular multi-agent swarm-robotics training platform (hackathon slice). Describe a swarm course in plain language and StudForge compiles it into a typed `SceneSpec`, advisory visual prior, compressed assembly, validation report, export artifacts, lightweight swarm run, and VGSY report. The project is a compiler-style stack, not an end-to-end text-to-3D generator: every stage exchanges typed contracts from `src/studforge/contracts.py`. ``` NL prompt → Compiler (Claude) → SceneSpec → Visual Prior (advisory) → Assembly → Compression → Canonical USD scene → Validators → exports → swarm sim → report (VGSY + compression metrics) ``` See `studforge-prd-v1_4-hackathon.md` for the full PRD and `CLAUDE.md` for the engineering conventions. ## Current Status Fully wired and tested offline: - Python package, Makefile, Typer CLI, FastAPI surface, `.env.example`, and pytest suite. - Typed contracts in `src/studforge/contracts.py`; frontend mirror in `frontend/src/lib/types.ts`. - Mock prompt compiler plus lazy Anthropic/Claude live adapter. - Mock visual prior, image API adapter, and Midjourney MCP adapter; all outputs remain advisory. - Assembly planner, deterministic compression strategies, validator suite, bounded repair loop, and repair memory. - In-process memory fallback plus strict Redis live adapter when `REDIS_URL` is set. - USD JSON fallback, parity-tagged SDF/URDF/MJCF stand-ins, `EpisodeManifest`, and report generation. - Lightweight scripted swarm sim plus a PettingZoo-compatible wrapper surface. - Compression benchmark and dry-run Trainium/Neuron training path with acceptance-gated artifacts. - FastAPI CORS middleware driven by `CORS_ALLOWED_ORIGINS`. - React/Vite/Three.js frontend with fixture mode, `/v1/generate` flow, 3D scene, validator/dashboard views, and health pills. Still needs work: - Backend Deepgram voice endpoints are not implemented yet. The frontend voice UI/client exists, but `/v1/healthz` does not report `deepgram`, and `/v1/voice/*` routes are absent. - `GET` or `WS /v1/scenes/{id}/playback` is not implemented; the frontend intentionally falls back to static robot poses. - Hardware execution is a stub until Ultimate Bots event SDK/arena details are available. - Real OpenUSD, full simulator fidelity, live Trainium fine-tuning, hosted deployment, and live third-party credentials remain optional/live-environment work. ## Quick Start ```bash make dev # create .venv and install backend dev deps make smoke # full offline pipeline on the example prompt make test # pytest make lint # ruff (non-blocking in Makefile) + naming lint make demo # generate ./out/report.html ``` Direct CLI examples: ```bash studforge generate --prompt "4-robot S&R maze with two corridors, victims, debris, recharge pad." \ --embodiment diffdrive_micro_v1 --team 4 --variants 5 --use-visual-prior --out ./out studforge healthz # which integrations are live vs mocked studforge compress-bench --seed 0 # compression harness -> out/compression_metrics.{csv,json} studforge world-model --variants 5 # world-model harness -> out/world_model_harness.{csv,json} studforge train-compression --dry-run # Trainium/Neuron dry-run training path python -m uvicorn studforge.api:app --reload # REST surface (§19.1), GET /v1/healthz ``` ## API Surface Implemented: - `GET /v1/healthz` - `POST /v1/compile` - `POST /v1/visual-prior` - `POST /v1/assemble` - `POST /v1/compress` - `POST /v1/validate` - `POST /v1/export` - `POST /v1/generate` - `GET /v1/scenes/{scene_id}/lineage` Not implemented yet: `/v1/voice/transcribe`, `WS /v1/voice/stream`, and scene playback endpoints. ## Offline-First Integrations With an empty `.env`, the backend pipeline runs without network, GPU, Redis, OpenUSD, or simulator installs. `config.py` currently resolves six backend/runtime integrations: | Integration | Live trigger | Offline behavior | |---|---|---| | Anthropic / Claude | `ANTHROPIC_API_KEY` | deterministic template compiler | | Redis | `REDIS_URL` | in-process lineage/cache/repair memory | | Arize / Phoenix | `OBSERVABILITY_ENABLED`, `PHOENIX_COLLECTOR_ENDPOINT` | no-op tracer | | Image provider | `IMAGE_PROVIDER=image_api` or `midjourney` | deterministic mock visual prior | | Trainium / Neuron | `COMPRESSION_BACKEND=learned` or `hybrid` | deterministic compression fallback | | Ultimate Bots | `HARDWARE_ENABLED=true` | in-repo lightweight sim | `.env.example` also contains planned Deepgram settings, but the current backend does not consume them yet. ## Frontend The browser UX lives in `frontend/`: ```bash cd frontend npm install npm run dev # http://localhost:5173, /v1 proxied to localhost:8000 npm run build ``` The app boots from fixture JSON when the backend is unreachable. When the backend is running, it calls `/v1/generate`, then best-effort `compile -> visual-prior -> assemble -> validate` to refresh the visible 3D scene. ## North-Star Metric **VGSY**: Validated Generated Scenario Yield, the fraction of generated worlds that pass validators or are auto-repaired and can run through the training/sim path without human repair. ## Naming Use StudForge / stud-based / brick-compatible / modular block in specs and exports. "LEGO" may appear only as a descriptive comparison with a non-affiliation note. This project is **not affiliated with the LEGO Group**. ## Detected evidence (automated analysis) Indexed codebase: 154 recognized source files, 963 KB. - Anthropic (technology) — detected in the code - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - Hugging Face (technology) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - Redis (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers - AI coding agent: Codex — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (120 of 175) ``` .env.example .gitignore AGENTS.md CLAUDE.md frontend/.env.example frontend/.gitignore frontend/CLAUDE.md frontend/index.html frontend/package.json frontend/postcss.config.js frontend/public/textures/generic/.gitkeep frontend/public/textures/README.md frontend/public/textures/school/.gitkeep frontend/public/textures/search_rescue/.gitkeep frontend/public/textures/warehouse/.gitkeep frontend/README.md frontend/src/App.tsx frontend/src/components/dashboard/CompressionBars.tsx frontend/src/components/dashboard/Dashboard.tsx frontend/src/components/dashboard/HealthPills.tsx frontend/src/components/dashboard/Inspector.tsx frontend/src/components/dashboard/LineageCard.tsx frontend/src/components/dashboard/ObserverFindingsCard.tsx frontend/src/components/dashboard/PolicyTrainingCard.tsx frontend/src/components/dashboard/SceneSummary.tsx frontend/src/components/dashboard/SimulationOutputsCard.tsx frontend/src/components/dashboard/SwarmRadar.tsx frontend/src/components/dashboard/SwarmStatusCard.tsx frontend/src/components/dashboard/ValidatorList.tsx frontend/src/components/dashboard/VGSYCard.tsx frontend/src/components/layout/LeftToolbar.tsx frontend/src/components/layout/Shell.tsx frontend/src/components/layout/StatusBar.tsx frontend/src/components/layout/TopBar.tsx frontend/src/components/prompt/PromptBar.tsx frontend/src/components/prompt/Transcript.tsx frontend/src/components/prompt/VoiceMic.tsx frontend/src/components/scene/agentPositionsRef.ts frontend/src/components/scene/BuildPhaseGate.tsx frontend/src/components/scene/buildSequenceRef.ts frontend/src/components/scene/BuildStageHUD.tsx frontend/src/components/scene/EnvCallout.tsx frontend/src/components/scene/FirstPersonController.tsx frontend/src/components/scene/MidjourneyRender.tsx frontend/src/components/scene/Objects.tsx frontend/src/components/scene/PipelineStages.tsx frontend/src/components/scene/PlaybackButton.tsx frontend/src/components/scene/Robots.tsx frontend/src/components/scene/Scene3D.tsx frontend/src/components/scene/sceneMath.ts frontend/src/components/scene/StudFloor.tsx frontend/src/components/scene/SwarmOverlay.tsx frontend/src/components/scene/SwarmTrails.tsx frontend/src/components/scene/Timeline.tsx frontend/src/components/scene/ValidatorOverlay.tsx frontend/src/components/scene/VisualPriorBackdrop.tsx frontend/src/components/scene/Walls.tsx frontend/src/components/ui/chrome.tsx frontend/src/components/ui/primitives.tsx frontend/src/fixtures/boot-report.json frontend/src/fixtures/sample-assembly-open.json frontend/src/fixtures/sample-assembly-tight.json frontend/src/fixtures/sample-assembly.json frontend/src/fixtures/sample-validation-open.json frontend/src/fixtures/sample-validation-tight.json frontend/src/fixtures/sample-validation.json frontend/src/hooks/useGenerate.ts frontend/src/hooks/useHealthz.ts frontend/src/hooks/useLineage.ts frontend/src/hooks/useObservers.ts frontend/src/hooks/usePlayback.ts frontend/src/hooks/useTextureSet.ts frontend/src/index.css frontend/src/lib/api.ts frontend/src/lib/constants.ts frontend/src/lib/types.ts frontend/src/lib/utils.ts frontend/src/lib/voice.ts frontend/src/main.tsx frontend/src/state/pipelineStore.ts frontend/src/state/renderStore.ts frontend/src/state/sceneStore.ts frontend/src/state/uiStore.ts frontend/src/state/voiceStore.ts frontend/src/vite-env.d.ts frontend/tailwind.config.ts frontend/tsconfig.json frontend/tsconfig.node.json frontend/vercel.json frontend/vite.config.ts Makefile PITCH.md pyproject.toml README.md scripts/naming_lint.py scripts/train_compression_neuron.py src/studforge/__init__.py src/studforge/agents/__init__.py src/studforge/agents/base.py src/studforge/agents/observers.py src/studforge/agents/policies.py src/studforge/agents/team.py src/studforge/agents/training.py src/studforge/api/__init__.py src/studforge/api/app.py src/studforge/assembly/__init__.py src/studforge/assembly/grid.py src/studforge/assembly/planner.py src/studforge/cli.py src/studforge/compiler/__init__.py src/studforge/compiler/anthropic_compiler.py src/studforge/compiler/base.py src/studforge/compiler/mock_compiler.py src/studforge/compiler/scene_spec_schema.py src/studforge/compression/__init__.py src/studforge/compression/harness.py src/studforge/compression/learned.py src/studforge/compression/semantics.py src/studforge/compression/strategies.py src/studforge/compression/training.py [55 more files omitted for size] ``` ### Dependencies - frontend/package.json: @react-three/drei@^9.114.0, @react-three/fiber@^8.17.10, @tanstack/react-query@^5.59.0, @types/node@^22.7.5, @types/react@^18.3.11, @types/react-dom@^18.3.0, @types/three@^0.169.0, @vitejs/plugin-react@^4.3.2, autoprefixer@^10.4.20, clsx@^2.1.1, framer-motion@^11.11.0, lucide-react@^0.453.0, postcss@^8.4.47, prettier@^3.3.3, react@^18.3.1, react-dom@^18.3.1, react-hot-toast@^2.4.1, recharts@^2.13.0, tailwind-merge@^2.5.4, tailwindcss@^3.4.13, three@^0.169.0, typescript@^5.6.3, vite@^5.4.9, zustand@^5.0.0 - pyproject.toml: anthropic@>=0.40, arize-phoenix@>=4.0, datasets@>=2.18, datasets@>=2.18, deepgram-sdk@>=3.7,<4, fastapi@>=0.110, fastmcp@>=2.3, fastmcp@>=2.3, httpx@>=0.27, networkx@>=3.2, neuronx-distributed@>=0.9, numpy@>=1.26, openinference-instrumentation-anthropic@>=0.1, opentelemetry-sdk@>=1.24, optimum-neuron@>=0.0.20, pandas@>=2.1, peft@>=0.11, peft@>=0.11, pettingzoo@>=1.24, pillow@>=10.0, py-key-value-aio[disk]@>=0.4, pydantic@>=2.6, pydantic-settings@>=2.1, pytest@>=8.0, redis@>=5.0, redisvl@>=0.20, redisvl@>=0.20, ruff@>=0.4, torch@>=2.2, torch@>=2.2, transformers@>=4.40, transformers@>=4.40, transformers@>=4.40, typer@>=0.12, usd-core@>=24.0, uvicorn@>=0.29, websockets@>=12.0 ### Recent commits (newest first) - Merge pull request #15 from kokonut121/fix/compression-prompt-format - Fix train/inference prompt-format mismatch in learned compression - Frontend overhaul: CAD chrome, pipeline narration, vibecode strip, team-count fix - Add learned swarm policy training - Merge branch 'main' of https://github.com/kokonut121/studsim - visual - Compile + visual prior once per generate, not once per variant - Make scene-render Midjourney prompt a photorealistic bird's-eye view - Give the dev proxy a 5-min ceiling for long generate requests - Revert CLAUDE.md/Makefile env-loading docs from "Persist Midjourney preview..." - Revert "midjourney new prompt" - Revert "Revert midjourney new prompt and Persist Midjourney preview..." - Revert "midjourney new prompt" and "Persist Midjourney preview in a corner window; clarify dev-server env loading" - midjourney new prompt - Persist Midjourney preview in a corner window; clarify dev-server env loading - Wire observer trace feedback into generation - Merge pull request #14 from kokonut121/codex/dynamic-scene-intent-fidelity - Make live Claude compiler output actually reach the scene graph - Merge pull request #13 from kokonut121/codex/dynamic-scene-intent-fidelity - Add dynamic scene intent fidelity ## Key source files (fetched from GitHub, selected and truncated for size) ### AGENTS.md ```markdown # AGENTS.md This file provides guidance to Codex (Codex.ai/code) when working with code in this repository. ## Current state This repo currently contains **only the PRD** (`studforge-prd-v1_4-hackathon.md`) — no code has been written yet. The PRD is the source of truth and is implementation-ready: §28 ("Implementation Guide for Codex") specifies the exact repo layout, typed contracts, adapters, orchestrator, build order, and Definition of Done. Read §28 before starting any implementation; read §9–§10 (architecture + interface contracts), §19 (API/CLI/SDK surface), and §23 (dependencies) for the big picture. The project (codename **Text-to-LEGO**, product name **StudForge**) is a hackathon slice of a text → physics-validated → simulator-ready → robot-buildable swarm-robotics world generator with a multi-agent training/eval loop. ## Architecture: a compiler, not a generator The system is a **compiler-style stack**, not an end-to-end text-to-3D model. Each stage consumes and returns a typed contract, so stages are independently testable and the wiring cannot drift: ``` NL prompt → Prompt Compiler (Codex) → SceneSpec → Visual Prior (advisory only) → VisualPrior → Assembly Planner → AssemblyGraph → Compression Planner → (AssemblyGraph, CompressionReport) → Validator Suite → ValidationReport (gates export; repair loop on reject) → USD export → artifacts + EpisodeManifest → lightweight swarm sim → reporting (VGSY + compression metrics) ``` Five principles govern every decision (PRD §1): **compiler over generator**; **USD is canonical, SDF/URDF/MJCF are exports**; **pre-trained at the edges, custom at the core**; **validation is the product**; **sim-to-real is staged, not zero-shot**. When metrics are weak, the fix is better validators / cleaner interfaces / narrower tasks — not a bigger model. `src/studforge/contracts.py` (Pydantic v2 models, mirroring PRD §10) is the **single source of truth for stage I/O**. `pipeline.py` is the wired orchestrator (`generate_one` / `generate`) that the API, CLI, and smoke test all call. See PRD §28.2 for the full monorepo layout and §28.6 for the orchestrator body. ## Non-negotiable conventions These are load-bearing for the design; violating them breaks the architecture, not just style: - **Never change a stage's signature without updating `contracts.py` and the orchestrator together.** No stage may read a downstream model. - **Every integration has a `live` and a `mock` backend, selected in `config.py` by env var.** Code imports the protocol from `/base.py`, never a concrete backend. - **The visual prior is advisory only.** It must never write geometry — only a soft, grid-snapped occupancy bias. The validator suite always runs regardless of the prior (PRD §10.7, FR-WM-04). - **Heavy deps are imported lazily inside adapters**, never at module load: `pxr` (OpenUSD), `mujoco`, `pettingzoo`, `benchmarl`/`torchrl`, `optimum-neuron`, `rclpy`. [truncated — 4820 more characters] ``` ### CLAUDE.md ```markdown # CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Current State The repository now contains a working backend spine, a React frontend, and tests. The PRD (`studforge-prd-v1_4-hackathon.md`, v1.5) remains the design source of truth, but this file describes what is actually wired today. Fully wired: - Backend package under `src/studforge/` with Pydantic v2 contracts, settings, stage protocols/adapters, orchestrator, CLI, API, reporting, and tests. - `pipeline.generate_one()` and `pipeline.generate()` connect compiler -> visual prior -> assembly -> compression -> validation/repair -> export -> swarm sim -> report. - Offline mock path works by design: template compiler, mock visual prior, deterministic compression, validators, JSON USD stand-in, scripted swarm sim, and HTML/JSON report. - FastAPI implements `/v1/healthz`, compile, visual-prior, assemble, compress, validate, export, generate, lineage, `agents/init`, scene `observers`, and voice (`/v1/voice/transcribe`, `WS /v1/voice/stream`) endpoints. - CLI implements `generate`, `validate`, `compress-bench`, `train-compression`, `agents-init`, and `healthz`; Make targets exist for dev/test/smoke/lint/demo/API/frontend. - Robot agents are config/initialized (`agents/`): per-robot `AgentConfig` + `RobotTeamPlan` built from the `EpisodeManifest`, a runnable scripted policy, and a `learned` policy stub that is configured but not trained (training deferred until the compressed landscape is known). Three observer agents (coverage/coordination/collision) record findings to the agent-memory service and recall a bounded, validator-gated soft bias into the next run. - Visual prior has mock, image API, and Midjourney MCP backends, with live failures falling back cleanly to the mock provider. - Memory has in-process fallback and strict Redis live selection; repair failures and lineage are recorded through the shared store. - Deepgram voice entry is wired: `voice/` package (SpeechToText protocol, live `nova-3` backend, offline mock), `POST /v1/voice/transcribe`, `WS /v1/voice/stream` relay, and `Settings.health()` now reports `deepgram` (`live` with a key, else `disabled`). The `/v1/compile` endpoint records `provenance.input_modality`/`stt_model`. UX-only: the empty-`.env` smoke path is unchanged and the mic button hides without a key. - Compression includes deterministic `unit`, `strips`, `greedy`, and `exact` strategies, plus learned-inference and dry-run Trainium training artifact paths. The learned/ `hybrid` backend also runs a base (un-finetuned) Qwen model directly from a hub id (`COMPRESSION_BASE_MODEL`) when no fine-tuned artifact is set, so the path is wired end-to-end ahead of training; every proposal stays validator-gated to greedy. - FastAPI CORS middleware is configured from `CORS_ALLOWED_ORIGINS`. - Frontend under `frontend/` renders fixture and generated scenes with React, Vite, Type [truncated — 7958 more characters] ``` ### pyproject.toml ``` [project] name = "studforge" version = "0.1.0" description = "Text-to-LEGO modular multi-agent swarm robotics training platform (hackathon slice)" readme = "README.md" requires-python = ">=3.11" dependencies = [ "pydantic>=2.6", "pydantic-settings>=2.1", "numpy>=1.26", "networkx>=3.2", "pandas>=2.1", "pillow>=10.0", "typer>=0.12", "fastapi>=0.110", "uvicorn>=0.29", "redis>=5.0", "redisvl>=0.20", ] [project.optional-dependencies] # Heavy / live-only lanes. Imported lazily inside adapters; absence triggers fallback. anthropic = ["anthropic>=0.40"] redis = ["redisvl>=0.20"] observability = [ "arize-phoenix>=4.0", "openinference-instrumentation-anthropic>=0.1", "opentelemetry-sdk>=1.24", ] sim = ["pettingzoo>=1.24"] usd = ["usd-core>=24.0"] # Local base-model compression inference (COMPRESSION_BASE_MODEL via plain # transformers/torch). No AWS-Neuron stack required; runs on CPU/GPU anywhere. compression-base = ["transformers>=4.40", "torch>=2.2"] trainium = [ "optimum-neuron>=0.0.20", "transformers>=4.40", "datasets>=2.18", "peft>=0.11", "neuronx-distributed>=0.9", ] # compute_backend="generic": same compression-model fine-tune, any CPU/GPU EC2 instance, # no Neuron SDK. compression-ec2 = [ "torch>=2.2", "transformers>=4.40", "datasets>=2.18", "peft>=0.11", ] midjourney = ["fastmcp>=2.3", "py-key-value-aio[disk]>=0.4"] # official Midjourney MCP client (visual prior) + persistent OAuth token cache # UX voice entry (FR-UX-06..08). websockets powers WS /v1/voice/stream and # the playback feed; deepgram-sdk relays browser audio to Nova STT server-side. # Capped below 4: deepgram-sdk 4.x+ is a generated rewrite with a different client # surface (no PrerecordedOptions/LiveOptions, no listen.*.v("1")). The voice adapter # targets the stable v3 API documented in PRD §28.5. voice = ["deepgram-sdk>=3.7,<4", "websockets>=12.0"] dev = ["pytest>=8.0", "ruff>=0.4", "httpx>=0.27", "fastmcp>=2.3"] [project.scripts] studforge = "studforge.cli:app" [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["src/studforge"] [tool.ruff] line-length = 100 target-version = "py311" [tool.pytest.ini_options] pythonpath = ["src"] testpaths = ["tests"] ``` ### frontend/package.json ``` { "name": "studforge-frontend", "private": true, "version": "0.1.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc -b && vite build", "preview": "vite preview", "lint": "tsc -p tsconfig.json --noEmit", "format": "prettier --write \"src/**/*.{ts,tsx,css,json}\"" }, "dependencies": { "@react-three/drei": "^9.114.0", "@react-three/fiber": "^8.17.10", "@tanstack/react-query": "^5.59.0", "clsx": "^2.1.1", "framer-motion": "^11.11.0", "lucide-react": "^0.453.0", "react": "^18.3.1", "react-dom": "^18.3.1", "react-hot-toast": "^2.4.1", "recharts": "^2.13.0", "tailwind-merge": "^2.5.4", "three": "^0.169.0", "zustand": "^5.0.0" }, "devDependencies": { "@types/node": "^22.7.5", "@types/react": "^18.3.11", "@types/react-dom": "^18.3.0", "@types/three": "^0.169.0", "@vitejs/plugin-react": "^4.3.2", "autoprefixer": "^10.4.20", "postcss": "^8.4.47", "prettier": "^3.3.3", "tailwindcss": "^3.4.13", "typescript": "^5.6.3", "vite": "^5.4.9" } } ``` ### frontend/src/main.tsx ```typescript import { StrictMode } from "react"; import { createRoot } from "react-dom/client"; import { QueryClient, QueryClientProvider } from "@tanstack/react-query"; import { Toaster } from "react-hot-toast"; import App from "./App"; import "./index.css"; const queryClient = new QueryClient({ defaultOptions: { queries: { retry: 1, staleTime: 30_000, refetchOnWindowFocus: false, }, }, }); createRoot(document.getElementById("root")!).render( , ); ``` ### frontend/src/App.tsx ```typescript import { useEffect } from "react"; import { useSceneStore } from "@/state/sceneStore"; import { useUiStore } from "@/state/uiStore"; import { useHealthz } from "@/hooks/useHealthz"; import Shell from "@/components/layout/Shell"; import bootReport from "@/fixtures/boot-report.json"; import type { AssemblyGraph, GenerateReport, SceneSpec, ValidationReport, } from "@/lib/types"; // Boot from a pre-baked /v1/generate report: a real generated search-and-rescue // school (4 robots fanning out to 4 rooms, victims/debris/recharge markers, a // full swarm trace, 6/6 valid variants). This mirrors a live generate response // so the canvas AND every dashboard card are populated, and playback auto-runs, // before the user touches anything (FR-FE-09: always renders offline). The // scene store keeps source="fixture" — this is honest offline sample data, not // a live backend call; source flips to "live" on a real generate. const BOOT = bootReport as unknown as GenerateReport; export default function App() { const setBootScene = useSceneStore((s) => s.setBootScene); const restartPlayback = useUiStore((s) => s.restartPlayback); useEffect(() => { const variant = BOOT.variants.find( (v) => v.verdict === "pass" || v.verdict === "repaired", ) ?? BOOT.variants[0]; if (variant) { setBootScene( variant.scene_spec as SceneSpec, variant.assembly_graph as AssemblyGraph, variant.validation as ValidationReport, BOOT, ); // Auto-play the swarm so judges see motion on load. Robots derives its // own playDuration once the assembly is set; playing stays true. restartPlayback(); } }, [setBootScene, restartPlayback]); // Poll /v1/healthz in the background; results power the integration pills. useHealthz(); return ; } ``` ### src/studforge/cli.py ```python """`studforge` CLI (§19.3). Thin wrapper over the orchestrator.""" from __future__ import annotations import json from pathlib import Path import typer app = typer.Typer(add_completion=False, help="StudForge — text-to-brick swarm world generator.") @app.command() def generate( prompt: str = typer.Option(None, help="Course description in plain language."), prompt_file: Path = typer.Option(None, help="Read the prompt from a file."), embodiment: str = typer.Option("diffdrive_micro_v1"), team: int = typer.Option(4), variants: int = typer.Option(5), use_visual_prior: bool = typer.Option(True, "--use-visual-prior/--no-visual-prior"), out: Path = typer.Option(Path("./out")), ): """Compile -> visual-prior -> assemble -> compress -> validate -> export -> sim -> report.""" from .pipeline import generate as run if prompt_file: prompt = prompt_file.read_text().strip() if not prompt: raise typer.BadParameter("Provide --prompt or --prompt-file.") report = run(prompt, embodiment, team, variants, use_visual_prior=use_visual_prior, out_dir=out) s = report["summary"] typer.echo(f"VGSY = {s['vgsy'] * 100:.0f}% ({s['valid_variants']}/{s['n_variants']} valid)") typer.echo(f"mean compression ratio = {s['mean_compression_ratio']}x best parts = {s['best_part_count']}") if s["swarm"]: typer.echo(f"swarm success = {s['swarm']['success_rate']}") if report.get("agents"): a = report["agents"] typer.echo(f"robot team = {a['team_count']}x {a['embodiment_id']} (policy: {a['policy_backend']})") if report.get("observers") and report["observers"].get("findings"): for f in report["observers"]["findings"]: typer.echo(f" observer[{f['observer_id']}] {f['signal']}: {f['recommendation']}") typer.echo(f"report -> {out / 'report.html'}") @app.command("agents-init") def agents_init( manifest: Path = typer.Argument(..., help="EpisodeManifest JSON (e.g. ./out/.manifest.json)."), embodiment: str = typer.Option("diffdrive_micro_v1"), ): """Initialize the robot-agent team from an EpisodeManifest.""" from .agents import build_team from .config import get_settings from .contracts import EpisodeManifest em = EpisodeManifest.model_validate(json.loads(manifest.read_text())) plan = build_team(em, embodiment, get_settings()) typer.echo(json.dumps(plan.model_dump(), indent=2)) @app.command("train-policy") def train_policy( scene: Path = typer.Argument(..., help="Canonical scene JSON (e.g. ./out/.canonical_scene.json)."), manifest: Path = typer.Argument(..., help="EpisodeManifest JSON (e.g. ./out/.manifest.json)."), out: Path = typer.Option(Path("./out/policies/swarm_policy.json")), episodes: int = typer.Option(320), seed: int = typer.Option(0), max_steps: int | None = typer.Option(None), ): """Train a local tabular Q-policy for the swarm on a canonical scene.""" from .agents import train_tabular_q_policy from .contracts import EpisodeManifest em = EpisodeManifest.model_validate(json.loads(manifest.read_text())) report = train_tabular_q_policy( scene, em, out, episodes=episodes, seed=seed, max_steps=max_steps, ) typer.echo(json.dumps(report.model_dump(), indent=2)) typer.echo("activate with:") typer.echo(" ROBOT_AGENT_BACKEND=learned") typer.echo(f" ROBOT_POLICY_PATH={report.artifact_uri}") @app.command() def validate(assembly: Path, embodiment: str = typer.Option("diffdrive_micro_v1")): """Validate an existing assembly graph JSON.""" from .contracts import AssemblyGraph from .usd_export.usd_scene import build_canonical_scene from .validators import suite graph = AssemblyGraph.model_validate(json.loads(assembly.read_text())) scene = build_canonical_scene(graph, assembly.parent) report = suite.run(scene, embodiment) typer.echo(json.dumps(report.model_dump(), indent=2)) @app.command() def compress_bench( prompt: str = typer.Option("4-robot maze with two narrow corridors and debris."), embodiment: str = typer.Option("diffdrive_micro_v1"), team: int = typer.Option(4), seed: int = typer.Option(0), out: Path = typer.Option(Path("./out")), ): """Run the compression harness (>=3 strategies) and emit JSON+CSV (AC-H2).""" from .compression.harness import run_harness result = run_harness(prompt, embodiment, team, seed, out) typer.echo(json.dumps(result["summary"], indent=2)) typer.echo(f"metrics -> {out / 'compression_metrics.csv'}") @app.command("world-model") def world_model( prompt: str = typer.Option("4-robot search-and-rescue maze with victims and debris."), embodiment: str = typer.Option("diffdrive_micro_v1"), team: int = typer.Option(4), variants: int = typer.Option(5), use_visual_prior: bool = typer.Option(True, "--use-visual-prior/--no-visual-prior"), out: Path = typer.Option(Path("./out")), ): """Run the limited world-model generation harness and emit JSON+CSV.""" from .world_model import run_harness report = run_harness( prompt, embodiment, team, variants, use_visual_prior=use_visual_prior, out_dir=out, ) typer.echo(json.dumps(report.summary, indent=2)) typer.echo(f"metrics -> {out / 'world_model_harness.csv'}") @app.command() def train_compression( output_dir: Path = typer.Option(Path("./out/compression_model")), dataset: Path = typer.Option(None, help="Existing train JSONL."), eval_dataset: Path = typer.Option(None, help="Existing eval JSONL."), base_model: str = typer.Option("google/flan-t5-small"), model_arch: str = typer.Option( "seq2seq", help="'seq2seq' (e.g. flan-t5) or 'causal' (e.g. Qwen2.5)." ), compute_backend: str = typer.Option( "neuron", help="'neuron' (optimum.neuron.NeuronTrainer on [truncated — 4297 more characters] ``` ### src/studforge/api/app.py ```python """FastAPI v1 surface (§19.1). Endpoints are thin and reuse the orchestrator stages, so the API can never drift from the CLI/smoke path. Run: `uvicorn studforge.api:app`. """ from __future__ import annotations import base64 import binascii import contextlib import os import tempfile from pathlib import Path from fastapi import FastAPI, HTTPException, WebSocket, WebSocketDisconnect from fastapi.middleware.cors import CORSMiddleware from pydantic import BaseModel from .. import compression from ..assembly import planner from ..compiler import get_compiler from ..config import get_settings from ..contracts import AssemblyGraph, CanonicalScene, SceneSpec from ..memory import get_store from ..pipeline import generate from ..usd_export import export from ..usd_export.usd_scene import build_canonical_scene from ..validators import suite from ..visual_prior import get_visual_prior app = FastAPI(title="StudForge", version="1.0.0") app.add_middleware( CORSMiddleware, allow_origins=get_settings().cors_origins(), allow_credentials=False, allow_methods=["*"], allow_headers=["*"], ) _OUT = Path(tempfile.gettempdir()) / "studforge_api_out" # --- request models ---------------------------------------------------------- class CompileReq(BaseModel): prompt: str embodiment_id: str = "diffdrive_micro_v1" team_count: int = 4 seed: int = 0 # Voice-entry provenance (FR-UX-08): the prompt is identical whether typed or # spoken; these only record *how* it arrived, never altering the compile. input_modality: str = "text" stt_model: str | None = None class VisualPriorReq(CompileReq): scene_spec: dict | None = None class VoiceTranscribeReq(BaseModel): audio_b64: str mime: str language: str | None = None model: str | None = None class GenerateReq(CompileReq): n_variants: int = 5 use_visual_prior: bool = True class AssembleReq(BaseModel): scene_spec: dict visual_prior: dict | None = None variant_seed: int = 0 class ValidateReq(BaseModel): assembly_graph: dict | None = None canonical_scene: dict | None = None embodiment_id: str = "diffdrive_micro_v1" class CompressReq(BaseModel): scene_spec: dict assembly_graph: dict strategy: str = "deterministic" class CanonicalizeReq(BaseModel): assembly_graph: dict class ExportReq(BaseModel): scene_spec: dict assembly_graph: dict | None = None canonical_scene: dict | None = None targets: list[str] = ["usd", "sdf", "urdf", "mjcf"] class AgentsInitReq(BaseModel): manifest: dict embodiment_id: str = "diffdrive_micro_v1" class TrainPolicyReq(BaseModel): scene_spec: dict | None = None assembly_graph: dict | None = None canonical_scene: dict | None = None scene_path: str | None = None manifest: dict | None = None episodes: int = 320 seed: int = 0 max_steps: int | None = None out_path: str | None = None activate: bool = True class RenderSceneReq(BaseModel): assembly_graph: dict prompt: str | None = None # --- endpoints --------------------------------------------------------------- @app.post("/v1/compile") def compile_spec(req: CompileReq): spec = get_compiler().compile(req.prompt, req.embodiment_id, req.team_count, req.seed) # Record input modality on the SceneSpec without re-running the compiler: a spoken # prompt and its typed twin produce identical geometry, only differing provenance. spec.provenance["input_modality"] = req.input_modality if req.input_modality == "voice" and req.stt_model: spec.provenance["stt_model"] = req.stt_model return spec.model_dump() @app.post("/v1/visual-prior") def visual_prior(req: VisualPriorReq): if req.scene_spec is not None: spec = SceneSpec.model_validate(req.scene_spec) else: spec = get_compiler().compile(req.prompt, req.embodiment_id, req.team_count, req.seed) return get_visual_prior().generate(spec).model_dump() @app.post("/v1/assemble") def assemble(req: AssembleReq): spec = SceneSpec.model_validate(req.scene_spec) from ..contracts import VisualPrior prior = VisualPrior.model_validate(req.visual_prior) if req.visual_prior else None g = planner.assemble(spec, visual_prior=prior, variant_seed=req.variant_seed) return g.model_dump() @app.post("/v1/compress") def compress_ep(req: CompressReq): spec = SceneSpec.model_validate(req.scene_spec) g = AssemblyGraph.model_validate(req.assembly_graph) g2, rep = compression.compress(spec, g, strategy=req.strategy) return {"compressed_graph": g2.model_dump(), "compression_report": rep.model_dump()} @app.post("/v1/canonicalize") def canonicalize_ep(req: CanonicalizeReq): g = AssemblyGraph.model_validate(req.assembly_graph) scene = build_canonical_scene(g, _OUT) return scene.model_dump(mode="json") @app.post("/v1/validate") def validate_ep(req: ValidateReq): if req.canonical_scene is not None: scene = CanonicalScene.model_validate(req.canonical_scene) elif req.assembly_graph is not None: scene = build_canonical_scene(AssemblyGraph.model_validate(req.assembly_graph), _OUT) else: raise HTTPException(status_code=422, detail="provide assembly_graph or canonical_scene") return suite.run(scene, req.embodiment_id).model_dump() @app.post("/v1/export") def export_ep(req: ExportReq): spec = SceneSpec.model_validate(req.scene_spec) source_graph = None if req.canonical_scene is not None: scene = CanonicalScene.model_validate(req.canonical_scene) elif req.assembly_graph is not None: source_graph = AssemblyGraph.model_validate(req.assembly_graph) scene = build_canonical_scene(source_graph, _OUT) else: raise HTTPException(status_code=422, detail="provide assembly_graph or canonical_scene") artifacts, manifest = export(scene, spec, req.targets, _OUT, source_graph=source_graph) return {"artifac [truncated — 7128 more characters] ``` ### frontend/postcss.config.js ```javascript export default { plugins: { tailwindcss: {}, autoprefixer: {}, }, }; ``` ### frontend/vite.config.ts ```typescript import { defineConfig } from "vite"; import react from "@vitejs/plugin-react"; import path from "node:path"; export default defineConfig({ plugins: [react()], resolve: { alias: { "@": path.resolve(__dirname, "./src"), }, }, server: { port: 5173, proxy: { // dev convenience: hit `/v1/*` directly without setting VITE_API_URL. "/v1": { target: "http://localhost:8000", changeOrigin: true, ws: true, // Generate runs the full live pipeline (Anthropic compile per variant + // visual prior + learned compression), which can take minutes. Give the // dev proxy a 5-min ceiling so long requests aren't aborted mid-flight. timeout: 300_000, proxyTimeout: 300_000, }, }, }, build: { target: "es2022", sourcemap: true, }, }); ``` [142 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]