# Project export: Paper Cuts This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Empowering the next generation of builders, thinkers, and dreamers by bringing back play - Devpost: https://devpost.com/software/paper-cuts - GitHub: https://github.com/Jeremyliu-621/paper-cuts - Video: https://player.vimeo.com/video/1203255539?byline=0&portrait=0&title=0#t=0 - Result: winner (GRAND PRIZE - Ddoski's Playground Track; Finalist; Best UI/UX) - Team: 3 GitHub contributor(s) — CaellumYHL (68 commits), Jeremy Liu (56 commits), ChloeHouvardas (24 commits) ## Devpost submission (written by the team) ### Inspiration Paper Cuts started from a simple belief: play is one of the best ways people learn to build. Today, most people grow up consuming games, videos, and apps, but far fewer get to create them. We wanted to make game creation feel less like opening a complex engine and more like drawing an idea on paper. We are inspired by the creativity of childhood sketchbooks, playground games, paper prototypes, and the moment when someone says, “What if this was a level?” Paper Cuts is our attempt to turn that moment into something playable. We do not want to constrain people to one game mode, one visual style, or one fixed idea of what a game should be. Our goal is to give people tools that make creation feel immediate, playful, and approachable. What It Does Paper Cuts lets people create playable game worlds by drawing, editing, and remixing game pieces. At its core, it is a creation-first game platform. Players can draw ideas on an iPad or browser canvas, turn those drawings into game objects, and then play inside the world they made. The end-to-end loop: Draw a shape on the iPad canvas (or use a phone as a projector pen to draw directly into the shared scene). Recognize: a multi-tier perception stack names the drawing: a custom CNN fast-path for the common vocabulary, an open-vocabulary vision-language model for anything else, and retrieval-augmented recognition against a Redis vector memory of every doodle ever drawn. Enhance: the rough doodle is re-synthesized into a clean, on-style raster game sprite (label-conditioned image-to-image) and matted to a transparent asset, while preserving the kid's original shape. Compose mechanics: a neuro-symbolic composer turns the meaning of the drawing into actual gameplay: a sword swings, a drawn flame becomes a fire projectile that beats a drawn vine, a fruit heals, a star grants invulnerability, a spring launches you. Play: drag the object onto a live mini-map of the arena and it drops into the running match. Phones become gamepads via QR codes; the whole thing is built to be projected. Finish: the first item a fighter picks up imprints a generative cinematic finisher: a stylized KO sequence that ends the round in spectacle. How We Built It We built Paper Cuts as a multi-part web stack. Web App Game engine: vanilla JS + HTML5 Canvas. We deliberately avoided a heavyweight engine so we'd have direct control of the render loop, camera, physics, hitboxes, particles, and the procedural "marker-and-paper" art style (every stroke is drawn, never templated), with DPR-aware sizing for sharp projector output. Drawing surface: React + Vite + tldraw. A first-class iPad/browser drawing experience: freehand strokes, shapes, labels, and touch, with platform reference overlays so you can draw onto the world. Backend: FastAPI. Live rooms, drawing capture, semantic candidate generation, the clarification loop, generative-finisher jobs, the Paper Trail vector service, and WebSocket broadcasts. Realtime layer: Node.js relay. Serves the game, mints QR codes, hosts phone-controller pages, runs the phone-as-gamepad WebSocket relay (lobbies, slots, input edges), and proxies backend routes. AI Implementation We run a small fleet of powerful, specialized models: Recognizer — a multi-tier perception stack. A custom-trained CNN fast-path classifies the common vocabulary on-device in milliseconds; an open-vocabulary vision-language model handles true draw-anything recognition; and retrieval-augmented recognition (RAR) does k-NN vector search over our Redis doodle memory to recognize. The three are fused behind a single confidence-gated recognize() call. Recognizer — a multi-tier perception stack. A custom-trained CNN fast-path classifies the common vocabulary on-device in milliseconds; an open-vocabulary vision-language model handles true draw-anything recognition; and retrieval-augmented recognition (RAR) does k-NN vector search over our Redis doodle memory to recognize. The three are fused behind a single confidence-gated recognize() call. Caecae — our drawing-to-asset visual model. A multi-stage image-to-image pipeline that takes a child's rough, shaky doodle and re-renders it as a clean, flat, bold-outline raster game sprite without discarding the original shape. The training stack was deliberately layered for reliability and fidelity: SD1.5 floor compile (guaranteed) — a Stable Diffusion 1.5 baseline that was certain to compile, as a floor we could always fall back to. SDXL primary compile in parallel — SDXL as the high-fidelity target, compiled concurrently so the floor never blocked the ceiling. Teacher dataset on Colab — we distilled a teacher dataset of doodle→sprite pairs on Google Colab. InstructPix2Pix fine-tune — an instruction-conditioned fine-tune so a label-based semantic hint from the recognizer steers the edit (it knows it's cleaning up a sword, not a snake). Fuse the best base → serve, then tune — model-merge the strongest base, ship it, and keep tuning online. Output is raster-sprite + rembg background stripping, yielding a transparent, drop-in asset. We trained Caecae on AWS Trainium (trn1) accelerators, but were unable to compile/serve the model on them. Caecae — our drawing-to-asset visual model. A multi-stage image-to-image pipeline that takes a child's rough, shaky doodle and re-renders it as a clean, flat, bold-outline raster game sprite without discarding the original shape. The training stack was deliberately layered for reliability and fidelity: SD1.5 floor compile (guaranteed) — a Stable Diffusion 1.5 baseline that was certain to compile, as a floor we could always fall back to. SDXL primary compile in parallel — SDXL as the high-fidelity target, compiled concurrently so the floor never blocked the ceiling. Teacher dataset on Colab — we distilled a teacher dataset of doodle→sprite pairs on Google Colab. InstructPix2Pix fine-tune — an instruction-conditioned fine-tune so a label-based semantic hint from the recognizer steers the edit (it knows it's cleaning up a sword, not a snake). Fuse the best base → serve, then tune — model-merge the strongest base, ship it, and keep tuning online. Output is raster-sprite + rembg background stripping, yielding a transparent, drop-in asset. We trained Caecae on AWS Trainium (trn1) accelerators, but were unable to compile/serve the model on them. Moose — a neuro-symbolic mechanic composer. Instead of letting a black-box model rewrite our game, Moose composes mechanics from a safe-by-construction operation graph (operations × triggers × element tags). A LoRA-tuned model proposes the intent of a drawing; the graph guarantees a valid, non-crashing, balanced mechanic. Interactions (fire melts ice, fire burns through vines, water douses fire) are generated on the fly with element-tag algebra. Moose — a neuro-symbolic mechanic composer. Instead of letting a black-box model rewrite our game, Moose composes mechanics from a safe-by-construction operation graph (operations × triggers × element tags). A LoRA-tuned model proposes the intent of a drawing; the graph guarantees a valid, non-crashing, balanced mechanic. Interactions (fire melts ice, fire burns through vines, water douses fire) are generated on the fly with element-tag algebra. Asset isolation pipeline. Generated sprites pass through saliency-based matting (rembg / BiRefNet) and a connected-component "keep-largest" pass that strips backgrounds and stray decoration blobs, yielding clean transparent assets that drop straight into the scene. Asset isolation pipeline. Generated sprites pass through saliency-based matting (rembg / BiRefNet) and a connected-component "keep-largest" pass that strips backgrounds and stray decoration blobs, yielding clean transparent assets that drop straight into the scene. Generative finishers. The first item a fighter grabs imprints a cinematic KO, generated with a generative-video model (Pika via fal.ai) styled to the characters and scene, then cached and pre-baked so the spectacle lands with zero in-match latency. Generative finishers. The first item a fighter grabs imprints a cinematic KO, generated with a generative-video model (Pika via fal.ai) styled to the characters and scene, then cached and pre-baked so the spectacle lands with zero in-match latency. Paper Trail — Redis as our vector brain (sponsor track) Every doodle anyone draws is embedded and written into a Redis Stack (RediSearch) HNSW vector index alongside its confirmed label, composed mechanic, and a thumbnail — a living, shared visual memory of every drawing. Today that memory powers Retrieval-Augmented Recognition (RAR): a new drawing is embedded and recognized by k-NN vector search over the community's collective memory, so recognition gets smarter and cheaper the more people play — and we lean on the expensive, high latency VLM less and less. Redis is our vector brain and agent memory, not a TTL cache: RediSearch HNSW does the similarity search, and the doodle index is the model's long-term memory. The same vector memory is built to power "Déjà Draw" remixing and cross-room mechanic consistency for Moose; see What's Next. Realtime, multi-device coordination The desktop game has its own camera, zoom, and world coordinates; the iPad has a completely separate canvas. We built a coordinate-reconciliation layer so a stroke drawn over a platform reference lands pixel-correct in the real game world, and a WebSocket fan-out so phones-as-controllers, phones-as-pens, and the host screen all stay in lockstep. Challenges We Ran Into Training Caecae on Trainium. Getting Caecae to compile and train on AWS Trainium was a real fight, the toolchain had many unforgiving nuances. We hedged with a layered compile strategy: an SD1.5 floor that was guaranteed to compile while the SDXL primary compiled in parallel, a teacher dataset on Colab, an InstructPix2Pix fine-tune fused onto the best base, then serve-then-tune. We ultimately ran out of Trainium access before serving the full SDXL weights at our target latency, so we swapped in hosted inference for the live demo while keeping the trained pipeline intact. Latency vs. quality. Turning a doodle into a beautiful asset and a fast asset pull in opposite directions. We chased sub-500ms enhancement, profiled diffusion paths (including on-device distilled variants), and learned exactly where the quality/latency cap is for flat 2D art on custom trained models. We were also forced to introduce clever latency masking features after running out of Trainium access. Keeping the doodle a doodle. Generative models love to add realistic shading, depth, and motion at the cost of time. Preserving the crisp, flat, hand-drawn aesthetic through image-to-image (and especially generative-video) took heavy prompt constraint, strength tuning, label-conditioning, and rembg post-processing. How much should the AI control? We refused to ship a black box that silently mutates the game, so we built a clarification loop — the system proposes candidates, the player confirms or corrects and made mechanics safe-by-construction so a quick wrong guess never causes a crash. Redis as memory, not cache. Designing the doodle embedding, the RediSearch HNSW schema, and the retrieval-augmented recognition vote (so the vector memory improves recognition without ever blocking the game) was a difficult part of integration. Coordinate systems across devices. Reconciling the iPad canvas, the projector world, and phone inputs into one coherent, drift-free scene was a surprising amount of math. Orchestrating a fleet of models. Recognizer, Caecae, Moose, the generative-video finisher, and the Paper Trail vector service (plus graceful degradation when any one is slow, rate-limited, or offline). Accomplishments That We’re Proud Of A real draw → recognize → enhance → compose → play loop that feels like magic. A multi-tier, open-vocabulary recognition stack — CNN + VLM + retrieval-augmented recognition over a Redis vector memory. Caecae's layered diffusion training stack (SD1.5 floor + SDXL primary, Colab teacher set, InstructPix2Pix fine-tune, model fusion) on Trainium. Moose's neuro-symbolic, safe-by-construction mechanic composition — AI creativity with deterministic guarantees. Paper Trail — a living Redis (RediSearch HNSW) vector memory of every doodle that powers retrieval-augmented recognition. A character system where custom sketches animate through the same rig as the built-in cast. Generative cinematic finishers that turn a KO into a moment. Custom maps, portals, hazards, cannons, bouncy platforms, and breakable objects. Phone-as-gamepad and phone-as-projector-pen multiplayer over QR codes. Most importantly, we are proud that the project feels playful and isn't boring. You can draw something, make a choice, and see it become part of a game world. What We Learned We learned that making creation feel simple requires a lot of structure underneath. We also learned that game creation tools should not start with complexity. Most engines ask people to think like developers before they can play. Paper Cuts tries to reverse that: start with play, drawing, and imagination, then gradually expose more power. Technically, we learned a lot about real-time sync, WebSockets, canvas rendering, semantic object detection, structured game patches, and multi-device workflows. What’s Next For Paper Cuts Next, we want to expand Paper Cuts from a single playable prototype into a broader creation platform. The next steps are: Support more game modes beyond the current platform-fighter demo Let creators choose or define different art styles Add richer character creation and prop refinement Let users build rules, win conditions, pickups, enemies, and hazards Add better world saving, sharing, and remixing "Déjà Draw" discovery & remix and cross-room mechanic memory, built on the Paper Trail vector index — surface and remix kindred community creations. Collaborative multi-author creation and richer world saving, sharing, and remixing — scaling the Paper Trail vector memory into a global, cross-session creation graph. Create a smoother “draw → clarify → playtest” loop Add collaborative creation, where multiple people can draw and edit together The long-term dream is that Paper Cuts becomes a bridge from imagination to interaction. Instead of only consuming games, people can sketch, remix, test, and play their own ideas. ## README (from the GitHub repository) # Doodle Smash A hand-drawn 2D platform fighter (Super Smash Bros–inspired) rendered in a charcoal "soft marker" doodle style. Vanilla HTML5 Canvas + JavaScript — **no build, no deps**. See [GOAL.md](GOAL.md) for the project's north star: a **live creation game** where players draw characters, weapons, and hazards on an iPad and an AI pipeline injects them — refined and functional — into a projected match in real time. Runtime design: [docs/13](docs/13-ai-pipeline.md). **Working on this?** Read [`docs/`](docs/) first — especially [`docs/02-aesthetic-rules.md`](docs/02-aesthetic-rules.md), the visual contract that keeps the whole game looking hand-drawn. It documents the architecture, mechanics, the character rig, the draw tool, how to extend things, and the dev workflow. ## Run it Just open `index.html` in a browser (double-click it, or drag it into Chrome). No server or install needed. ### MagicBoard drawing Local desktop and iPad testing can stay on HTTP: ```bash cd backend cp .env.example .env # fill OPENAI_API_KEY and MAGICBOARD_VLM_MODEL for VLM classification uv run uvicorn app.main:app --reload --host 0.0.0.0 --port 8000 cd ../draw-client npm install npm run dev -- --host 0.0.0.0 cd .. npm install npm start ``` Open the desktop game with `?backend=http://YOUR-LAN-IP:8000&drawClient=http://YOUR-LAN-IP:5173/`, then open the draw client on the iPad. The flow is doodle first: VLM classification can auto-confirm platform/spike/etc. candidates, and the iPad manual choice menu is the fallback when classification fails or is unavailable. Provider keys belong only in `backend/.env`; do not put OpenAI keys in `draw-client/.env`. ### Phone controllers (optional) To let people **join by scanning a QR code** and use their phone as a controller (a landscape Brawlhalla-style pad — D-pad + jump/attack buttons + a special-aim joystick, up to 6 per lobby), run the bundled server: ``` npm install # one time (ws + qrcode, server-only) npm start # → http://localhost:8080 ``` Open that URL, hit **≡ Menu → Players** for the QR, and point a phone at it. Deploy `server.js` to any Node host (with HTTPS) for play across the internet. Full details: [docs/11](docs/11-online-controllers.md). (The game itself still runs from `file://` with the keyboard — the server is only for phone controllers.) Optional URL hashes (handy for testing/demos; not the long-term product entry point): - `index.html#play` — jump straight into a match - `index.html#demo` — attract-mode: two AI fighters battle on their own - `index.html#editor` — open the editor (`#editor-stage`, `#editor-settings` for sub-tabs) ## Controls (2 players, one keyboard) | | Player 1 | Player 2 | |---|---|---| | Move | `A` / `D` | `←` / `→` | | Jump (×2) | `W` | `↑` | | Crouch / drop-through | `S` | `↓` | | Attack (melee) | `F` | `.` | | Special (ranged) | `G` | `/` | | Shield | `Left Shift` | `Right Shift` | `Enter` start / rematch · `P` pause · `?` (top-right) shows this in-app. Mechanics: run, double jump, fast-fall (hold down in air), drop through soft platforms (down on a pass-through platform), shield, a melee attack and a ranged **Special** (throws a projectile), all with frame data, percent-scaled knockback by weight, stocks (hearts), blast-zone KOs, respawn, match timer. ## Modes & maps Open the **≡ Menu** (top-right, also shown on load) to pick a **mode** and a **map**: - **Smash** — the classic; knock rivals off the stage, last one with stocks wins. - **King of the Hill** — stand alone on the high platform to bank time; first to 12s. Infinite respawns. - **Gem Grab** — slow-drifting gems float through the air; first to grab 5. - **K.O. Rush** — no stocks; every knockout scores, first to 5 K.O.s. Maps: **Meadow** (the editable Editor stage), **Twin Peaks**, **Sky Loft**, **Quarry**, **Ruins** — big themed arenas with background structures, plants, several material types, **swinging platforms** you can ride, and **breakable crates**. Modes and maps are small registries in `js/modes.js` — see [docs/10](docs/10-modes-and-maps.md) to add more. ## Editor Click the **Editor** tab. Everything is editable and saved to your browser (localStorage); use **Export/Import** to move setups between machines. - **Characters** — pick a character + action (idle/walk/jump/attack/…), then reshape its pose with the joint sliders (the big canvas preview updates live). Tune stats (speed, jumps, weight, size) and, for attack/special, the hitbox + frame data. - **Draw** — draw your own fighter over a faint "ghost" body. Each stroke is auto-sorted into the body part it lands on (head, body, both arms, both legs); lock a part with the buttons, or undo/clear. Because the drawing rigs onto the same skeleton, your character instantly animates through *every* move. Toggle "use drawing" off to fall back to the built-in stick figure. (Each part = vector strokes stored relative to its joint.) - **Stage** — drag platforms to move them, drag a platform's bottom-right corner to resize, drag the dotted circles to reposition spawns, add/remove platforms, toggle pass-through. - **Settings** — gravity, timer, stocks, knockback scale, hitstop. ## Code map | File | Role | |---|---| | `js/data.js` | Data model (characters/poses, stage, settings) + localStorage store. Single source of truth. | | `js/draw.js` | Rough "marker" Canvas2D renderer + offscreen pose-cache + paper texture. | | `js/character.js` | Parametric doodle fighter: pose (joint angles) → line-art (used until a character has a drawn skin). | | `js/skin.js` | User-drawn "skins": 6 hand-drawn parts rigged to the same joints; stroke→part auto-assignment; mannequin guide. | | `js/physics.js` | AABB platformer collision (solid + pass-through). | | `js/fighter.js` | Movement, jumps, attacks, hitboxes, knockback, KO, render. | | `js/stage.js` | Platforms + doodle decorations. | | `js/modes.js` | Game modes (Smash/KotH/Gems/K.O. Rush) + map presets, as data-driven registries. | | `js/effects.js` | Juice: particles, screen shake, hitstop, KO bursts. | | `js/game.js` | Match flow, active mode/map, HUD (timer/%/hearts/scores/portraits), overlays, attract AI. | | `js/editor.js` | The editor tab. | | `js/main.js` | Canvas/DPR sizing, tabs, frame loop. | ## Notes toward the AI creation pipeline - **Skin / stage / mechanic data is plain and serializable.** AI-generated content (vector strokes, `data.stage.platforms` rectangles, mechanic specs) flows through the same seams the editor uses, so the drawing pipeline, agents, and the editor all produce the data the game reads. See [docs/13](docs/13-ai-pipeline.md). - **Rendering is isolated** behind `draw.js`; `draw.getCached()` pose-caches to offscreen canvases so per-frame cost stays low when many drawn entities are on screen. - **(Optional, far-future)** a computer-vision module could also generate `data.stage.platforms` from detected real-world surfaces through the same seam — a nicety, not the goal. See [docs/08](docs/08-roadmap-and-cv-ar.md). ## Detected evidence (automated analysis) Indexed codebase: 110 recognized source files, 1583 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Node.js (technology) — claimed on Devpost, not found in the code - PyTorch (technology) — claimed on Devpost, not found in the code - Redis (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (120 of 131) ``` .gitignore backend/.env.example backend/app/__init__.py backend/app/agent_runtime.py backend/app/config.py backend/app/finishers.py backend/app/main.py backend/app/orchestrator.py backend/app/paper_trail.py backend/app/rooms.py backend/app/schemas.py backend/app/semantic.py backend/pyproject.toml backend/tests/test_api.py backend/uv.lock controller.html data/chloe_graph_pairs.jsonl data/chloe_pairs.jsonl data/gen_chloe_dataset_rules.py data/gen_chloe_dataset.py data/gen_chloe_graph_dataset.py data/gen_teacher.py data/get_quickdraw.py data/roughen.py docker-compose.redis.yml docs/00-vision.md docs/01-architecture.md docs/02-aesthetic-rules.md docs/03-gameplay-and-mechanics.md docs/04-character-rig-and-skins.md docs/05-editor.md docs/06-extending.md docs/07-rendering-and-coordinates.md docs/08-roadmap-and-cv-ar.md docs/09-conventions-and-dev-workflow.md docs/10-modes-and-maps.md docs/11-online-controllers.md docs/12-sound.md docs/13-ai-pipeline.md docs/13-ipad-drawing-capture-and-editor-seams.md docs/14-visual-creation-phases.md docs/15-creation-patch-contract.md docs/16-agent-clarification-loop.md docs/17-runtime-ai-pipeline-status.md docs/devpost.md docs/paper-trail.md docs/pitch-script.md docs/README.md draw-client/.env.example draw-client/index.html draw-client/package.json draw-client/src/App.jsx draw-client/src/main.jsx draw-client/src/styles.css draw-client/vite.config.js drawpad.html GOAL.md index.html ipadsetup.md js/ai.js js/audio.js js/campad.js js/character.js js/createOverlay.js js/data.js js/draw.js js/drawpad.js js/editor.js js/effects.js js/fighter.js js/finishers.js js/game.js js/graph.js js/input.js js/levelPreview.js js/magicBoardGame.js js/main.js js/mechanics.js js/modes.js js/net.js js/physics.js js/prop.js js/rng.js js/skin.js js/stage.js js/stageReferenceData.js js/ultimateRecorder.js js/worldLibrary.js NEW_TODO.md notebooks/caellum_colab.ipynb package.json PHASE1.md README.md render.yaml run-instructions.md RUNBOOK-chloe.md RUNBOOK-image.md RUNBOOK-recognizer.md RUNBOOK-trackb.md server.js services/caellum/compile.py services/caellum/config.py services/caellum/IMPLEMENTATION-SPEC.md services/caellum/README.md services/caellum/requirements-neuron.txt services/caellum/requirements-serve-local.txt services/caellum/requirements-serve.txt services/caellum/serve_local.py services/caellum/serve.py services/caellum/setup_neuron.sh services/chloe/config.py services/chloe/graph_config.py services/chloe/IMPLEMENTATION-SPEC.md services/chloe/README.md services/chloe/requirements-serve.txt services/chloe/serve.py services/recognizer/config.py services/recognizer/serve.py style.css tests/desktopSmoke.test.mjs [11 more files omitted for size] ``` ### Dependencies - backend/pyproject.toml: certifi@>=2025.11.12, fastapi@>=0.115.0, httpx@>=0.27.0, openai@>=2.0.0, pillow@>=12.2.0, pydantic@>=2.8.0, python-dotenv@>=1.0.1, uvicorn[standard]@>=0.30.0, websockets@>=14.0 - draw-client/package.json: @vitejs/plugin-react@6.0.2, react@19.2.7, react-dom@19.2.7, tldraw@5.1.1, vite@8.0.16 - package.json: @playwright/test@^1.61.0, qrcode@^1.5.3, ws@^8.18.0 ### Recent commits (newest first) - Merge pull request #1 from Jeremyliu-621/feat/ddoski-deterministic - Item finisher, Paper Trail (Redis RAG), Devpost + pitch - Item-based finisher: first pickup -> Pika KO video -> green aura -> 't' in range -> eliminate - Remove 'C' camera key (incoherent in iPad flow, like 'D'); ultimate recorder pose loader GPU->CPU (Safari) - iPad draw pad: category picker is a small dropdown (Make: …) defaulting to AI guess, instead of a chip row - server: CORS headers + OPTIONS preflight so the game can run on the LAN IP and still reach /fal-enhance, /vlm-recognize, /healthz - Heal/Star/Bouncy now work + palette gets the interaction elements - iPad draw pad polish + demo wiring - iPad draw pad: draw on iPad -> drag onto arena mini-map -> drops into the live match - docs: i2i default (reliable on real doodles) + open-vocab recognition fix + crude-doodle suite test - Default to image-to-image @ 0.72: reliably clean+isolated on real doodles (t2i invents scenes) - VLM recognition: open-vocab prompt (was over-steering to game vocab -> 3/9, now 8/9 on crude doodles) - docs: finisher tested + working (backend dep/key were the blockers); pre-warm strategy - docs: Bria two-stage cutout + suite-test findings - Sprite cutout: add Bria background removal (saliency) before client keep-largest - docs: update — teammate's AI finisher videos + AR pose ultimates are the wow factor - Merge remote-tracking branch 'origin/main' - docs: runtime AI pipeline status — decisions, setbacks, local-diffusion findings, finishers pointer - Make MagicBoard editor mobile first - Gate ultimate recording on skeleton alignment ## Key source files (fetched from GitHub, selected and truncated for size) ### ipadsetup.md ```markdown # iPad Drawing Client Setup Use your Mac's LAN IP on the iPad. Do not use `localhost` on the iPad; that points at the iPad itself. ## 1. Find Your Mac IP On the Mac: ```sh ipconfig getifaddr en0 ``` If that prints nothing, try: ```sh ipconfig getifaddr en1 ``` In the examples below, replace `MAC_IP` with that address. ## 2. Start the Backend From the repo root: ```sh cd backend .venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8001 ``` Check it from the Mac: ```text http://MAC_IP:8001/health ``` ## 3. Start the Desktop Game In a second terminal from the repo root: ```sh PORT=8081 npm start ``` Open this on the Mac: ```text http://MAC_IP:8081/?backend=http://MAC_IP:8001&drawClient=http://MAC_IP:5175/ ``` Go to Library, choose a level, then click `Edit Level`. That publishes the active room for the iPad. ## 4. Start the iPad Draw Client In a third terminal from the repo root: ```sh cd draw-client npm run build npm run preview -- --host 0.0.0.0 --port 5175 ``` Open this exact shape of URL on the iPad: ```text http://MAC_IP:5175/?backend=http://MAC_IP:8001 ``` The iPad should auto-join the level currently opened with `Edit Level` on the Mac. ## Direct Room Link If auto-join is not what you want, use a room link: ```text http://MAC_IP:5175/?room=ROOM_CODE&backend=http://MAC_IP:8001 ``` ## Common Fixes - Blank white screen: make sure the draw client is running with `--host 0.0.0.0`, then reload the iPad page. - Cannot connect: make sure the iPad and Mac are on the same Wi-Fi and use `MAC_IP`, not `localhost`. - Backend unreachable: open `http://MAC_IP:8001/health` on the iPad. If it does not load, check the backend terminal and macOS firewall. - Manual type buttons showing immediately: wait for the VLM pass. Manual choices should only appear if VLM fails, is unavailable, or needs correction. ``` ### run-instructions.md ```markdown # Run Instructions These commands run the current Phase 1 creation bridge: draw over a Doodle Smash reference in the draw client, and see that drawing as non-mutating, world-anchored scene annotation in the existing Doodle Smash game. Phase 1 target note: the draw client reference is a static, platform-only, fixed-camera view. It does not show a second live match with its own timer, fighters, or dynamic camera. Copy commands as full lines. If your terminal wraps a long command visually, do not press Enter in the middle of flags like `--host`. ## First-Time Setup From the repo root: ```sh cd backend python3 -m venv .venv .venv/bin/python -m pip install uv UV_CACHE_DIR=../.uv-cache .venv/bin/uv sync ``` Then install frontend/game packages: ```sh cd ../draw-client npm install cd .. npm install ``` ## Terminal 1: Backend From the repo root: ```sh cd backend UV_CACHE_DIR=../.uv-cache .venv/bin/uv run uvicorn app.main:app --host 0.0.0.0 --port 8000 ``` Health check: ```sh curl http://localhost:8000/health ``` Expected response: ```json {"ok":true,"version":"0.1.0"} ``` ## Terminal 2: Existing Game With Overlay From the repo root: ```sh npm start ``` Open this on your laptop: ```text http://localhost:8080/?overlay=demo&backend=http://localhost:8000#play ``` The `overlay=demo` query enables the creation overlay for room `demo`. Press `O` to toggle overlay visibility. The overlay does not mutate game data. ## Terminal 3: Draw Client From the repo root: ```sh cd draw-client npm run dev -- --host 0.0.0.0 ``` Open the Vite URL it prints. Usually: ```text http://localhost:5173/?room=demo ``` If Vite says it used another port, use that port instead. Draw inside the orange 1920 x 1080 game frame. Your drawing should appear over the laptop game canvas. The reference inside the drawing frame is the static platform-only level view so you can align your drawing with the actual platforms. In the laptop game, the drawing should stay attached to those scene coordinates as the camera pans or zooms. Inspect backend capture state: ```text http://localhost:8000/rooms/demo/capture ``` After drawing, `projection.strokes`, `projection.shapes`, or `projection.labels` should contain objects and `version` should increase. ## iPad Testing The iPad must use the laptop's LAN IP address, not `localhost`. Find your laptop LAN IP. On macOS, this often works: ```sh ipconfig getifaddr en0 ``` If it prints `10.31.151.244`, start the draw client like this: ```sh cd draw-client npm run dev -- --host 0.0.0.0 ``` When the iPad opens the draw client from `10.31.151.244`, the draw client automatically uses: - backend: `http://10.31.151.244:8000` - static platform reference from the current level data Open on the laptop: ```text http://localhost:8080/?overlay=demo&backend=http://localhost:8000#play ``` Open on the iPad: ```text http://10.31.151.244:5173/?room=demo ``` Replace `10.31.151.244` and `5173` with your actual IP and Vite port. Draw on the i [truncated — 208 more characters] ``` ### package.json ``` { "name": "doodle-smash-server", "version": "1.0.0", "private": true, "description": "Lobby + phone-controller relay for Doodle Smash. Serves the game, the mobile controller page, a QR endpoint, and a WebSocket relay so phones can join a lobby and drive fighters. The game itself stays zero-build and still runs from file:// for solo/keyboard play.", "scripts": { "start": "node server.js", "test": "node tests/magicBoardGame.test.mjs", "test:browser": "node tests/desktopSmoke.test.mjs" }, "engines": { "node": ">=18" }, "dependencies": { "qrcode": "^1.5.3", "ws": "^8.18.0" }, "devDependencies": { "@playwright/test": "^1.61.0" } } ``` ### draw-client/package.json ``` { "name": "magicboard-draw-client", "version": "0.1.0", "private": true, "type": "module", "scripts": { "dev": "vite", "build": "vite build", "preview": "vite preview" }, "dependencies": { "@vitejs/plugin-react": "6.0.2", "react": "19.2.7", "react-dom": "19.2.7", "tldraw": "5.1.1", "vite": "8.0.16" } } ``` ### backend/pyproject.toml ``` [project] name = "magicboard-backend" version = "0.1.0" description = "FastAPI backend for Magic Board iPad drawing snapshot persistence." readme = "../README.md" requires-python = ">=3.11" dependencies = [ "certifi>=2025.11.12", "fastapi>=0.115.0", "httpx>=0.27.0", "openai>=2.0.0", "pillow>=12.2.0", "pydantic>=2.8.0", "python-dotenv>=1.0.1", "uvicorn[standard]>=0.30.0", "websockets>=14.0", ] [dependency-groups] dev = [ "pytest>=8.2.0", ] [tool.pytest.ini_options] testpaths = ["tests"] pythonpath = ["."] ``` ### server.js ```javascript // Doodle Smash lobby/relay server. // // Serves three things from one origin so a deployed copy "just works" anywhere: // 1. the game itself (the existing static files — index.html, js/, style.css) // 2. /c the mobile controller page (a phone joystick + buttons) // 3. /qr?d=… a QR image (SVG) for a join URL // 4. /ws a WebSocket relay that ferries controller input to the host // // Phones never talk to the host directly; everything hops through here, so it works // across the internet once this is deployed to a public HTTPS host (Render/Fly/…). // The game still runs from file:// for solo/keyboard play — the relay is only needed // for phone controllers. 'use strict'; const http = require('http'); const https = require('https'); const fs = require('fs'); const path = require('path'); const { WebSocketServer } = require('ws'); const QRCode = require('qrcode'); const ROOT = __dirname; const PORT = process.env.PORT || 8080; const TLS_CERT = process.env.MAGICBOARD_TLS_CERT; const TLS_KEY = process.env.MAGICBOARD_TLS_KEY; const MAX_PLAYERS = 6; // ---- fal.ai enhance proxy config ---------------------------------------- // PIPELINE (default 'recraft'): the kid's rough drawing -> Recraft V3 image-to-image (a clean, // CONSISTENT illustration style that LEVELS UP the doodle into a real game sprite while still // following its shape) -> BiRefNet background removal (a real ML cutout, run SERVER-SIDE so it never // stalls the browser render loop) -> transparent PNG. Set FAL_PIPELINE=sdxl for the old canny path. const FAL_PIPELINE = (process.env.FAL_PIPELINE || 'recraft').toLowerCase(); // 'text' (default): generate a CLEAN game icon FROM the recognized label — ignore the doodle's shape // entirely (this is what gives a real game-asset look). 'image': image-to-image (traces the doodle — // only recolors it; kept for comparison via FAL_GEN_MODE=image). // 'image' (default): image-to-image augments the kid's ALREADY-ISOLATED doodle — reliably clean + // on-shape across any object (text-to-image invents little SCENES for everyday objects like house/sun/ // heart, which break the cutout). 'text': clean redraw from the label (loses the kid's shape; only // reliable for game-item words). Suite-tested 2026-06-21: i2i 9/9 clean vs t2i ~4/9 on crude doodles. const FAL_GEN_MODE = (process.env.FAL_GEN_MODE || 'image').toLowerCase(); const FAL_GEN_MODEL_T2I = 'fal-ai/recraft/v3/text-to-image'; const FAL_GEN_MODEL_I2I = 'fal-ai/recraft/v3/image-to-image'; const FAL_RMBG_MODEL = 'fal-ai/bria/background/remove'; // saliency cutout (handles patterned backgrounds) // Recraft style + how far it may stray from the kid's drawing (0 = identical .. 1 = ignore it). // digital_illustration/hand_drawn fits the doodle world; vector_illustration/bold_stroke is flatter. // Both are env-tunable so we can dial the look without code edits. const FAL_STYLE = process.env.FAL_STYLE || 'digital_illustration/hand_drawn'; const FAL_STRENGTH = Number(process.env.FAL_STRENGTH || 0.72); // i2i: high enough to clean up + color, low enough to keep the kid's shape // Stage-1 generator provider. 'openai' = GPT-image-1 edit with a native TRANSPARENT background: it // redraws the kid's doodle into a clean game-asset sticker with real alpha, so NO cutout step is needed // (this is what finally kills the background/edge artifacts). 'fal' = legacy Recraft i2i + Bria below. const GEN_PROVIDER = (process.env.GEN_PROVIDER || 'fal').toLowerCase(); const GPT_IMAGE_QUALITY = process.env.GPT_IMAGE_QUALITY || 'low'; // low|medium|high|auto ('low' ~22s, clean flat stickers; bump for complex drawings) const OPENAI_TIMEOUT_MS = Number(process.env.OPENAI_TIMEOUT_MS || 60000); // ---- VLM (open-vocab doodle recognition) ---- const VLM_MODEL = process.env.MAGICBOARD_VLM_MODEL || 'gpt-4.1-mini'; // the words the game can turn into mechanics — steer the VLM toward a USABLE label (js/mechanics.js). const VLM_VOCAB = 'sword, knife, axe, hammer, bat, gun, bow, slingshot, bomb, ball, dart, rock, ' + 'fire, water, ice, lightning, plant, poison, wind, metal, light, dark, ' + 'star, crown, gem, key, heart, shield, apple, banana, food, bread, cake, pizza, ' + 'cloud, mushroom, tree, anvil, skull, boomerang, umbrella'; const FAL_TIMEOUT_MS = 60000; // legacy fallback: fast-sdxl-controlnet-canny (FAL_PIPELINE=sdxl) const FAL_SDXL_MODEL = 'fal-ai/fast-sdxl-controlnet-canny'; const FAL_NEG_PROMPT = 'realistic, photo, photograph, 3d, render, detailed, shading, gradient, texture, ' + 'noise, busy background, scenery, shadow, reflection, blurry, watermark, text, signature'; // FAL_KEY (format "id:secret"): prefer the env var; otherwise parse a // FAL_KEY=... line from a .env in the repo root. Never hardcoded. function falKey() { if (process.env.FAL_KEY) return process.env.FAL_KEY.trim(); try { const txt = fs.readFileSync(path.join(ROOT, '.env'), 'utf8'); for (const line of txt.split(/\r?\n/)) { const m = line.match(/^\s*FAL_KEY\s*=\s*(.+?)\s*$/); if (m) return m[1].replace(/^['"]|['"]$/g, '').trim(); } } catch (e) { /* no .env => unset */ } return null; } // OPENAI_API_KEY for the VLM recognizer: env, then root .env, then backend/.env (where the // level-editor keeps it). Never hardcoded. function openaiKey() { if (process.env.OPENAI_API_KEY) return process.env.OPENAI_API_KEY.trim(); for (const rel of ['.env', path.join('backend', '.env')]) { try { const txt = fs.readFileSync(path.join(ROOT, rel), 'utf8'); for (const line of txt.split(/\r?\n/)) { const m = line.match(/^\s*OPENAI_API_KEY\s*=\s*(.+?)\s*$/); if (m) return m[1].replace(/^['"]|['"]$/g, '').trim(); } } catch (e) { /* skip */ } } return null; } // Recraft target prompt — the STYLE param carries the look, so the prompt just names the subject // and asks for a clean, single, game-ready sprite. function recraftPrompt(label) { return `a clean, well-drawn hand-drawn doodle of a ${label} — confident t [truncated — 19911 more characters] ``` ### draw-client/src/main.jsx ```javascript import { createRoot } from 'react-dom/client' import App from './App.jsx' import './styles.css' import 'tldraw/tldraw.css' createRoot(document.getElementById('root')).render() ``` ### backend/app/main.py ```python from __future__ import annotations import asyncio import json from typing import Any from fastapi import FastAPI, HTTPException, WebSocket, WebSocketDisconnect from fastapi.middleware.cors import CORSMiddleware from pydantic import ValidationError from .config import load_backend_env from .agent_runtime import agent_status, run_visual_observation from .finishers import create_finisher_job, get_finisher_job from .orchestrator import AgentOrchestrator from .paper_trail import router as paper_trail_router from .rooms import rooms, selection_payload from .schemas import ( AgentJobRequest, BACKEND_VERSION, CanvasCaptureMessage, ClarificationAnswerMessage, ErrorMessage, FinisherJobRequest, HelloMessage, RoomSelectionRequest, StageEditMessage, ) load_backend_env() app = FastAPI(title="Magic Board Backend", version=BACKEND_VERSION) orchestrator = AgentOrchestrator(rooms) app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=False, allow_methods=["*"], allow_headers=["*"], ) app.include_router(paper_trail_router) @app.get("/health") async def health() -> dict[str, bool | str]: return {"ok": True, "version": BACKEND_VERSION} @app.get("/agent/status") async def get_agent_status() -> dict[str, Any]: return agent_status().model_dump(mode="json", by_alias=True) @app.get("/rooms/{room_id}/capture") async def get_room_capture(room_id: str) -> dict[str, Any]: return rooms.capture_response(room_id).model_dump(mode="json", by_alias=True) @app.post("/finishers/jobs") async def create_finisher(request: FinisherJobRequest) -> dict[str, Any]: response = await create_finisher_job(request) return response.model_dump(mode="json", by_alias=True) @app.get("/finishers/jobs/{job_id}") async def get_finisher(job_id: str) -> dict[str, Any]: response = await get_finisher_job(job_id) if response is None: raise HTTPException(status_code=404, detail="finisher job not found") return response.model_dump(mode="json", by_alias=True) async def _run_visual_job( room_id: str, capture_version: int, job_id: str, world_id: str | None, projection: dict[str, Any], candidates: list[dict[str, Any]], ) -> None: observation = await run_visual_observation( room_id=room_id, world_id=world_id, capture_version=capture_version, job_id=job_id, projection=projection, candidates=candidates, ) update = rooms.store_visual_observation(room_id, observation) if update is not None: await rooms.broadcast_visual(room_id, update) def _schedule_visual_observation(room_id: str) -> None: room = rooms.get_room(room_id) observation = room.visual_observation if not observation or observation.status != "pending" or not room.projection: return candidates = [ candidate.model_dump(mode="json", by_alias=True) for candidate in (room.semantic_draft.candidates if room.semantic_draft else []) if candidate.status == "needs_answer" ] asyncio.create_task( _run_visual_job( room_id=room.room_id, capture_version=observation.capture_version, job_id=observation.job_id, world_id=room.world_id, projection=room.projection, candidates=candidates, ) ) @app.post("/rooms/{room_id}/capture") async def save_room_capture(room_id: str, capture: CanvasCaptureMessage) -> dict[str, Any]: update = rooms.store_capture(room_id, capture) await rooms.broadcast(room_id, update) _schedule_visual_observation(room_id) return rooms.capture_response(room_id).model_dump(mode="json", by_alias=True) @app.get("/rooms/{room_id}/semantic-draft") async def get_semantic_draft(room_id: str) -> dict[str, Any] | None: draft = rooms.semantic_draft(room_id) return None if draft is None else draft.model_dump(mode="json", by_alias=True) @app.get("/rooms/{room_id}/visual-observation") async def get_visual_observation(room_id: str) -> dict[str, Any] | None: observation = rooms.visual_observation(room_id) return None if observation is None else observation.model_dump(mode="json", by_alias=True) @app.post("/rooms/{room_id}/clarifications") async def answer_clarification(room_id: str, answer: ClarificationAnswerMessage) -> dict[str, Any]: try: update = rooms.store_answer(room_id, answer) except ValueError as error: raise HTTPException(status_code=409, detail=str(error)) from error await rooms.broadcast_semantic(room_id, update) return update.semantic_draft.model_dump(mode="json", by_alias=True) @app.post("/rooms/{room_id}/agent/jobs") async def enqueue_agent_job(room_id: str, request: AgentJobRequest) -> dict[str, Any]: return rooms.enqueue_agent_job(room_id, request).model_dump(mode="json", by_alias=True) @app.get("/selection/current") async def get_current_selection() -> dict[str, Any]: return selection_payload(rooms.current_selection()) @app.post("/selection/current") async def set_current_selection(selection: RoomSelectionRequest) -> dict[str, Any]: current = rooms.select_room( room_id=selection.room_id, world_id=selection.world_id, world_name=selection.world_name, stage_reference=selection.stage_reference, stage_reference_version=selection.stage_reference_version, ) await rooms.broadcast_selection() return selection_payload(current) @app.delete("/selection/current") async def clear_current_selection() -> dict[str, Any]: current = rooms.clear_selection() await rooms.broadcast_selection() return selection_payload(current) @app.websocket("/ws/selection") async def selection_socket(websocket: WebSocket) -> None: await websocket.accept() rooms.connect_selection(websocket) await websocket.send_json( { "type": "selection_hello", **selection_payload(ro [truncated — 3783 more characters] ``` ### docker-compose.redis.yml ```yaml # Paper Trail vector memory — Redis Stack (RediSearch HNSW + RedisInsight). # Start with: docker compose -f docker-compose.redis.yml up -d # RedisInsight UI: http://localhost:8001 · Redis: redis://localhost:6379 services: redis: image: redis/redis-stack:latest container_name: paper-trail-redis ports: - "6379:6379" # Redis (REDIS_URL=redis://localhost:6379) - "8001:8001" # RedisInsight web UI restart: unless-stopped ``` ### render.yaml ```yaml # Render blueprint — one-click deploy of the Doodle Smash server (game + phone-controller # relay). Render supports WebSockets on web services, so the /ws relay + QR controller work. # Deploy: render.com → New → Blueprint → connect this repo → it reads this file. services: - type: web name: doodle-smash runtime: node plan: free buildCommand: npm install startCommand: node server.js healthCheckPath: /healthz autoDeploy: true envVars: - key: NODE_VERSION value: "20" ``` [93 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]