# Project export: Explainer Kit This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: A Codex plugin for educational videos - Devpost: https://devpost.com/software/explainer-kit - GitHub: https://github.com/Gyana491/codex-explainer-video-plugin - Video: https://www.youtube.com/embed/tl_m-byUnSE?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 2 GitHub contributor(s) — Gyana Ranjan (31 commits), Reg421 (10 commits) ## Devpost submission (written by the team) ### Overview Watch Demo 1: Kubernetes Explained in 60 Seconds Watch Demo 2: The Ursidae Family Brought to Life ### Inspiration A picture is worth a thousand words, but it doesn't make something click the way a good video does. The problem is that making a good explainer video normally takes hours. The obvious shortcut, "just ask AI to generate the video," has its own problem: anyone who's tried it has seen what it looks like. Characters that shift between shots, objects that flicker or melt, a style that never quite holds together from one second to the next. Anddd it's expensive. We wanted the middle path: structured, illustrated slides that actually stay consistent and readable, brought to life with real, deterministic animation instead of AI-guessed motion. That's Explainer Kit. ### What it does Explainer Kit is a Codex plugin that turns a topic, script, article, or narration audio into a finished explainer video, end to end, inside a Codex workspace. Reads the source and extracts only what matters: a central idea, the essential supporting points, no invented facts. Then it builds a truthful causal story around it (an audience proxy, real stakes, a turning point, a payoff), capped at 3-6 scenes so nothing gets padded out just to hit a duration. Generates the entire storyboard in one single image-generation call, every scene together on one contact sheet, like a page of a comic book, plus (when a character or object recurs) a row of matching cutouts drawn once on a chroma-key background so it's never redrawn from scratch. Hands that sheet to deterministic code, not another AI call, to slice it into pixel-exact scenes. Image models can't reliably count pixels, so a local compositor rebuilds the exact geometry itself. Generates narration per scene, not as one giant voiceover, measuring each clip's real duration so the video's timeline is built from actual audio instead of a guess. Word-aligns the narration to drive karaoke-style captions and reveal timing. An arrow or label appears exactly when the narrator says the word it corresponds to. Renders the deterministic motion layer (arrows, highlights, transitions, reused cutouts) over the illustrated backgrounds with Remotion. Titles and labels are never baked into the AI-generated image, so they stay exact and editable. Runs a validator before it's allowed to call the job done: exact scene geometry, one voice clip per scene, audio and video synced to the millisecond. The result is a handful of animated, narrated scenes and roughly a minute of video explaining something that might otherwise take pages to understand. Watch Demo 1: Kubernetes Explained in 60 Seconds Watch Demo 2: The Ursidae Family Brought to Life ### How we built it We told Codex to build the plugin itself. Codex building a Codex plugin, editing its own future toolset. Sol handled planning: story-engine design, the skill contracts, the overall pipeline architecture. Terra and Luna picked up the smaller implementation tasks underneath that plan, split by difficulty, using the Superpowers plugin with brainstorming, plan and subagent-driven development to break the work into concrete, checkable steps instead of one giant undirected build. Watching Codex extend itself this way and later catch and fix its own bugs in a plugin it wrote was one of the most fun parts of this project! ...However human intervention was definitely needed (see the " ### Challenges we ran into " section). So we would go back and forth from Codex building big chunks to checking and adding fixes along the way after testing the plugin. Under the hood, Explainer Kit is a set of skills (structured markdown instructions Codex reads and follows) plus deterministic local tooling for the parts that shouldn't be left to AI judgment. Three cooperating skills: a storyboard-director skill (planning: essence, story engine, narration, slide plan, image prompt), a render-storyboard-video skill (rendering: split, voice, sync, captions, Remotion, finalize), and a create-explainer-video skill that orchestrates both end to end. Shared rules (visual style, story craft, the master image prompt, the overlay JSON contract) live in reference files the skills point to, instead of being duplicated three times. Deterministic scripts own anything that must be exact: a compositor that rebuilds pixel-perfect 4:3/16:9 geometry after generation, an FFprobe-based audio analyzer, a word-level aligner (faster-whisper) for captions and reveal timing, a chroma-key cutout extractor, and a final validator that hard-gates completion. A bundled Remotion project renders the deterministic overlay layer (shapes, text, animation cues, layered cutouts, camera moves) driven entirely by a JSON project file the skills generate, never by more AI generation. A small Cloudflare Worker (MCP server) wraps OpenAI's TTS and an upscaling model behind two tools (generate_voiceover, upscale_image) the skill calls mid-task, so the plugin needs no local API keys by default. Everything downstream of generation (cropping, mixing, syncing, final encoding) runs locally with FFmpeg and FFprobe. No cloud render farm. Challenges we ran into The gap between "the skill says to" and "Codex actually does." A real end-to-end run skipped the deterministic geometry step, generated one giant voiceover instead of per-scene clips (which broke sync), never consulted the style reference (drifting into an unrequested visual style), and produced twice as many scenes as intended... and still reported success, because nothing was actually checking. We fed that failed run straight back into Codex and had it fix its own plugin: turn every "should" into a script that exits non-zero and blocks completion if it's skipped. Windows on ARM64 broke rendering for two unrelated reasons. Remotion's native compositor has no ARM64 build at all, so the template needs to run under x64 Node. Separately, a native-dependency chain (word-alignment's backend) needed the same x64-vs-arm64 check. Every native binary in the stack had to be verified on the actual target architecture, nothing about it was assumable from the OS alone. Deciding what AI should own vs. what code should own. Image generation is great at "draw a believable scene" and bad at "produce an exact 4:3 canvas." The fix was to never ask the model for precision it can't deliver: let it draw, and let deterministic code measure, crop, and verify. Instruction cost. An earlier version of the skill files repeated the same visual-style and prompt-template rules across three separate files, which was expensive and time consuming on every single run. Splitting shared rules into reference files the skills point to (instead of duplicating them) cut the combined skill instructions from roughly 650 lines to about 135, same behavior, a fraction of the tokens. ### Accomplishments we're proud of The first iteration of the plugin was a complete disaster. The images were blurry, everything was out of sync and, cherry on top, it used a programable voice library which made the whole thing sound very creepy. Seeing it come together neatly with overlays, text and audio synced perfectly was very satisfying! ### What we learned Anything an agent can skip, it eventually will. The reliable fix for that is a script that fails loudly, not a stronger sentence in the instructions. The best output came from constraining the AI less on style guesses and more on geometry and timing. In other words: draw with AI, measure with code. The most confusing failures come from small platform details (native architecture mismatches, a missing optional dependency) that are invisible until the pipeline is actually run end to end on the real target machine, not something you catch by reading the code. ### What's next Self-hosting docs so the media service doesn't depend on a single shared endpoint. Vertical (9:16) and multi-aspect-ratio polish for short-form platforms, more catered to content creators. A wider house visual-style library so users pick a look instead of only getting the default. A fast, cheap smoke-test path that verifies the plugin works without requiring a full multi-minute generation. ## README (from the GitHub repository) # Codex Explainer Video Plugin Turns a topic, script, article, or narration audio into a story-driven explainer video: one pixel-verified storyboard of at most 6 scenes, locked OpenAI voiceover per scene, word-focused karaoke captions, deterministic Remotion overlays, and local FFmpeg delivery. ``` source → essence/story → storyboard sheet(s) → upscale → canonicalize → per-scene voiceover ∥ (parallel with the above) → measured timings → word captions → Remotion overlays → FFmpeg finalize ``` ## Prerequisites - Codex installed and available from your terminal. - FFmpeg installed and available as `ffmpeg` and `ffprobe`. - Node.js and npm for videos that use animated shapes or essential text overlays. - Python with `pydub` and `faster-whisper` for narration rhythm analysis and word alignment. Python 3.13+ also needs `audioop-lts`. - A Codex workspace where generated video files can be saved. Verify FFmpeg and install Python dependencies (run from the plugin root): ```bash ffmpeg -version ffprobe -version python3 -m pip install -r requirements-audio.txt ``` ## Install the plugin Run these commands in order: ```bash codex plugin marketplace remove codex-explainer-video-plugin codex plugin marketplace add Gyana491/codex-explainer-video-plugin codex plugin add codex-explainer-video-plugin@codex-explainer-video-plugin ``` The first command removes an older marketplace registration. If Codex reports that the marketplace is not installed, continue with the next command. ## Finish setup 1. Close and reopen Codex after the installation completes. 2. Start a new Codex task so the plugin's skills and media tools are loaded. 3. Open or create a writable workspace for the generated storyboard, audio, and video files. 4. Confirm that FFmpeg is available in the same environment where Codex is running. No local API keys or MCP server configuration are required for the published plugin — see [Media service](#media-service) below for the default endpoint and self-hosting. ## Quick start ``` codex plugin list ``` Confirm `codex-explainer-video-plugin` appears in the installed plugin list. In a new Codex task, try: ```text Create an explainer video about how solar panels work. Use the default style and voice. Save it in this workspace. ``` The skills carry the story, geometry, and production rules — the prompt only needs to state topic, duration, and any style preferences. ## Media service The plugin uses OpenAI voiceover through its bundled media service; it does not use ElevenLabs. By default `.mcp.json` points at the author's Cloudflare Worker (`explainer-video-media-mcp.gyan491.workers.dev`). This is third-party infrastructure — availability and quotas are not guaranteed. For production use, self-host: ```bash cd mcp-server cp .dev.vars.example .dev.vars # set OPENAI_API_KEY and REPLICATE_API_TOKEN npm install npx wrangler deploy ``` Then point `.mcp.json`'s `url` at your deployed worker's `/mcp` endpoint. See `mcp-server/README.md` for local dev, R2 configuration, and secret management. ## Overlays and layout QA Storyboard panels combine with editable Remotion overlays for diagrams, charts, equations, labels, counters, kinetic text, and transparent foreground cutouts. Overlays support explicit depth, anchored groups, and separate artwork/screen coordinate spaces so annotations follow camera motion while titles stay fixed. The default visual theme is an editorial paper-collage style (cream background, dark ink, warm and cool accents, Inter typography) — see `references/house-style.md` for the full direction and `references/overlay-storyboard.md` for the `theme` block. Run the layout analyzer before the final Remotion render to catch text or filled shapes that collide with dense illustration detail: ```bash node scripts/analyze-overlay-layout.mjs my-video/src/project.json --json my-video/output/layout-report.json ``` For smarter placement, add scene `objects`, anchored overlay `groups`, and element `intent` metadata to `project.json`, then run `npm run layout-fix` (moves colliding or auto-place text) and `npm run layout-stills` (renders a contact sheet at `output/qa/layout/layout-contact-sheet.png` for review before a full render). A successful render leaves only the finalized `output/explainer-video.mp4`; the Remotion intermediate is kept only when finalization fails, for diagnosis. ## Troubleshooting - **Windows ARM64:** run the bundled overlay template with x64 Node.js under Windows emulation — Remotion does not publish a native ARM64 compositor. The template preflight reports this before rendering. Set `REMOTION_BROWSER_EXECUTABLE` to override browser discovery for a custom Chrome or Edge path. - **Word alignment unavailable:** if `faster-whisper` is not installed, `scripts/align_words.py` falls back to proportional phrase-level timing and reports it in `timing_source` — captions stay phrase-accurate but are not word-verified. ## Reinstall or update ```bash codex plugin marketplace remove codex-explainer-video-plugin codex plugin marketplace add Gyana491/codex-explainer-video-plugin codex plugin add codex-explainer-video-plugin@codex-explainer-video-plugin ``` Then restart Codex and use a new task. ## Built with Codex We told Codex to build the plugin itself — Codex building a Codex plugin, editing its own future toolset. Sol handled planning: story-engine design, the skill contracts, the overall pipeline architecture. Terra and Luna picked up the smaller implementation tasks underneath that plan, split by difficulty, using the Superpowers plugin to break the work into concrete, checkable steps instead of one giant undirected build. At runtime the plugin keeps using Codex and GPT-5.6: Codex's built-in image generation draws every storyboard scene in a single call, and GPT-5.6 drives the planning, narration, and validation judgment calls the skills describe. A real end-to-end run once caught Codex skipping its own pipeline steps — one giant voiceover instead of per-scene clips, an unrequested visual style, twice the intended scene count — and reporting success anyway. We fed that failed run back into Codex and had it fix its own plugin: turn every "should" into a script that exits non-zero and blocks completion if it's skipped. Watching Codex extend, then debug, a plugin it wrote for itself was one of the most fun parts of building this. ## Detected evidence (automated analysis) Indexed codebase: 26 recognized source files, 652 KB. - OpenAI (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Codex — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (55 of 55) ``` .agents/plugins/marketplace.json .codex-plugin/plugin.json .gitignore .mcp.json assets/remotion-overlay-template/package.json assets/remotion-overlay-template/public/audio/.gitkeep assets/remotion-overlay-template/public/scenes/.gitkeep assets/remotion-overlay-template/remotion.config.ts assets/remotion-overlay-template/scripts/analyze-overlay-layout.mjs assets/remotion-overlay-template/scripts/finalize-render.mjs assets/remotion-overlay-template/scripts/preflight.mjs assets/remotion-overlay-template/scripts/render-layout-stills.mjs assets/remotion-overlay-template/src/animation.ts assets/remotion-overlay-template/src/ExplainerVideo.tsx assets/remotion-overlay-template/src/index.ts assets/remotion-overlay-template/src/layout.ts assets/remotion-overlay-template/src/OverlayRenderer.tsx assets/remotion-overlay-template/src/project.json assets/remotion-overlay-template/src/Root.tsx assets/remotion-overlay-template/src/types.ts assets/remotion-overlay-template/tsconfig.json LICENSE mcp-server/.dev.vars.example mcp-server/.gitignore mcp-server/.oxfmtrc.json mcp-server/.oxlintrc.json mcp-server/.vscode/settings.json mcp-server/AGENTS.md mcp-server/package.json mcp-server/README.md mcp-server/src/index.ts mcp-server/src/storage.ts mcp-server/src/tools/generate-voiceover.ts mcp-server/src/tools/upscale-image.ts mcp-server/tsconfig.json mcp-server/worker-configuration.d.ts mcp-server/wrangler.jsonc README.md references/house-style.md references/master-prompt-template.md references/overlay-storyboard.md references/story-rules.md requirements-audio.txt scripts/align_words.py scripts/analyze_audio.py scripts/analyze-overlay-layout.mjs scripts/canonicalize_storyboard.py scripts/extract-cutouts.mjs scripts/prepare-overlay-project.mjs scripts/setup-local-environment.ps1 scripts/validate-overlay-storyboard.mjs scripts/validate-project.mjs skills/create-explainer-video/SKILL.md skills/render-storyboard-video/SKILL.md skills/storyboard-director/SKILL.md ``` ### Dependencies - assets/remotion-overlay-template/package.json: @remotion/cli@^4.0.342, @types/react@^19.0.0, react@^19.0.0, react-dom@^19.0.0, remotion@^4.0.342, typescript@^5.7.2 - mcp-server/package.json: @modelcontextprotocol/sdk@^1.29.0, @types/node@^26.1.1, agents@^0.17.1, openai@^6.47.0, oxfmt@^0.56.0, oxlint@^1.71.0, replicate@^1.4.0, typescript@6.0.3, wrangler@^4.111.0, zod@^4.4.3 ### Recent commits (newest first) - updated readme with extra details about how codex built this - stricter criteria for "done", different visual style (no whiteboard), capped scenes at exactly 6 - readme edits - new reference files and skills cleanup - added whiteboard theme support - deduplicated audio code and speed optimizations - Merge pull request #3 from Gyana491/feature/assets-overlay - added cutout overlays - cleaned up overall additions - merge: integrate storyboard overhaul with Remotion overlays - feat: add deterministic Remotion asset overlays - refactor: update storyboard aspect ratio from 9:16 portrait to 4:3 landscape across all technical specifications and scripts. - feat: implement deterministic storyboard canonicalization and enforce strict 16:9 pixel geometry for all production scenes - refactor: update storyboard director to generate 9:16 portrait master grids with 16:9 panels - Merge pull request #2 from Gyana491/feat/improve-storyboard-generatation - refactor: update explainer video guidelines to use a structured story engine and standardized orange-accented whiteboard style - Merge pull request #1 from Gyana491/feat/improve-storyboard-generatation - refactor: update storyboard-director schema and rules to support automated captioning, word-level timing, and improved editorial design guidelines - feat: transition explainer video output to a story-driven, multi-beat editorial whiteboard presentation style - feat: add audio analysis script for rhythmic scene segmentation and update skill documentation ## Key source files (fetched from GitHub, selected and truncated for size) ### mcp-server/AGENTS.md ```markdown # Cloudflare Workers STOP. Your knowledge of Cloudflare Workers APIs and limits may be outdated. Always retrieve current documentation before any Workers, KV, R2, D1, Durable Objects, Queues, Vectorize, AI, or Agents SDK task. ## Docs - https://developers.cloudflare.com/workers/ - MCP: `https://docs.mcp.cloudflare.com/mcp` For all limits and quotas, retrieve from the product's `/platform/limits/` page. eg. `/workers/platform/limits` ## Commands | Command | Purpose | |---------|---------| | `npx wrangler dev` | Local development | | `npx wrangler deploy` | Deploy to Cloudflare | | `npx wrangler types` | Generate TypeScript types | Run `wrangler types` after changing bindings in wrangler.jsonc. ## Node.js Compatibility https://developers.cloudflare.com/workers/runtime-apis/nodejs/ ## Errors - **Error 1102** (CPU/Memory exceeded): Retrieve limits from `/workers/platform/limits/` - **All errors**: https://developers.cloudflare.com/workers/observability/errors/ ## Product Docs Retrieve API references and limits from: `/kv/` · `/r2/` · `/d1/` · `/durable-objects/` · `/queues/` · `/vectorize/` · `/workers-ai/` · `/agents/` ## Best Practices (conditional) If the application uses Durable Objects or Workflows, refer to the relevant best practices: - Durable Objects: https://developers.cloudflare.com/durable-objects/best-practices/rules-of-durable-objects/ - Workflows: https://developers.cloudflare.com/workflows/build/rules-of-workflows/ ``` ### references/house-style.md ```markdown # House visual style: editorial paper-collage Default visual style unless the user explicitly requests another. Applies to both planning (`storyboard-director`) and the image-generation prompt (`master-prompt-template.md`'s `{STYLE_AND_CONTINUITY_RULES}`). Derived from the two reference productions in `.local-videos/ursidae-mastercanvas-layered` and `.local-videos/viral-online-v2-layered`, which set the visual bar for this plugin. - Treat any supplied reference image as inspiration for visual language only. Never reproduce its exact layout, characters, objects, wording, typeface, palette, or decorative placements. - Warm editorial paper-collage illustration: hand-cut paper shapes with crisp dark ink contours, tactile layered depth, subtle print grain. Sophisticated and educational, not childish or corporate. - Cream or warm parchment background as the base plate. - **Palette formula:** cream/parchment base + one dark ink color (near-black, not pure black) + one warm accent + one cool accent, chosen per topic. Reference values that work well: cream `#FFF3D5`, ink `#172234`, warm accent burnt orange `#EF6C26`, cool accent cobalt `#184D9C`. For nature/field-guide topics, a fuller earth palette also works: parchment, deep forest green, rust brown, glacier blue, charcoal (see the ursidae project). Pick the two accents to fit the topic; keep the cream base and dark ink constant. - Recognizable, coherent-scale subjects with clean silhouettes; soft directional lighting; no photorealism, no 3D rendering, no glossy UI, no gradients, no saturated multicolor palettes, no stock-photo elements, no dense chaotic backgrounds, no heavy soft shadows. - Keep every named recurring character or object identical across slides: proportions, palette, contour weight, drawing language. - Use generous negative space, one clear visual path per slide, and a distinct composition per scene — never repeat the same arrangement of title, subject, and diagram in multiple panels. - Generated artwork provides environments, characters, and objects. Exact titles, labels, numbers, arrows, cards, and diagrams are added afterward as deterministic overlays — the artwork must leave the declared overlay zones quiet and empty, with no embedded lettering of any kind. ## Overlay typography Overlay text renders in a clean editorial sans (Inter or equivalent), not handwritten. Title, label, definition, value, and takeaway roles use weight and size to establish hierarchy; the warm accent color is reserved for emphasis (key phrases, outcomes, active caption word), never for body copy. See `overlay-storyboard.md` for the `theme` block — it defaults to this palette and font; only override it when the user requests a different visual direction. ## Cutout tray (optional, recommended when scenes share characters/objects) When the same character or object appears across multiple scenes, generate it once as a foreground cutout in the same image-generation call as the storyboard sheet, not redrawn p [truncated — 614 more characters] ``` ### mcp-server/package.json ``` { "name": "explainer-video-media-mcp", "version": "0.0.0", "private": true, "scripts": { "deploy": "wrangler deploy", "dev": "wrangler dev", "format": "oxfmt --write .", "lint:fix": "oxlint --fix", "start": "wrangler dev", "cf-typegen": "wrangler types", "type-check": "tsc --noEmit" }, "dependencies": { "@modelcontextprotocol/sdk": "^1.29.0", "agents": "^0.17.1", "openai": "^6.47.0", "replicate": "^1.4.0", "zod": "^4.4.3" }, "devDependencies": { "@types/node": "^26.1.1", "oxfmt": "^0.56.0", "oxlint": "^1.71.0", "typescript": "6.0.3", "wrangler": "^4.111.0" } } ``` ### assets/remotion-overlay-template/package.json ``` { "name": "explainer-overlay-render", "private": true, "version": "1.0.0", "scripts": { "preflight": "node scripts/preflight.mjs", "layout-check": "node scripts/analyze-overlay-layout.mjs src/project.json", "layout-fix": "node scripts/analyze-overlay-layout.mjs src/project.json --apply --json output/layout-report.json", "layout-stills": "node scripts/render-layout-stills.mjs", "prerender": "npm run preflight", "render": "npm run render:remotion && npm run finalize", "render:remotion": "remotion render src/index.ts ExplainerVideo output/explainer-video-remotion.mp4", "finalize": "node scripts/finalize-render.mjs", "type-check": "tsc --noEmit" }, "dependencies": { "@remotion/cli": "^4.0.342", "react": "^19.0.0", "react-dom": "^19.0.0", "remotion": "^4.0.342" }, "devDependencies": { "@types/react": "^19.0.0", "typescript": "^5.7.2" } } ``` ### mcp-server/src/index.ts ```typescript import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js"; import { createMcpHandler } from "agents/mcp"; import { registerGenerateVoiceover } from "./tools/generate-voiceover"; import { registerUpscaleImage } from "./tools/upscale-image"; function createServer(env: Env): McpServer { const server = new McpServer({ name: "explainer-video-media", version: "0.1.0", }); registerUpscaleImage(server, env); registerGenerateVoiceover(server, env); return server; } export default { async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise { const url = new URL(request.url); if (url.pathname === "/health") { return Response.json({ ok: true, service: "explainer-video-media", tools: ["upscale_image", "generate_voiceover"], }); } if (url.pathname === "/mcp") { return createMcpHandler(createServer(env))(request, env, ctx); } return new Response("Not found", { status: 404 }); }, }; ``` ### assets/remotion-overlay-template/src/index.ts ```typescript import {registerRoot} from "remotion"; import {Root} from "./Root"; registerRoot(Root); ``` ### scripts/align_words.py ```python #!/usr/bin/env python3 """Word-align per-scene narration clips and emit word timings plus karaoke captions. Reads output/scene-timings.json, transcribes each scene clip with faster-whisper (word timestamps on), snaps recognized words to the known narration text, offsets by cumulative scene start, and writes word-timings.json and captions.ass. Falls back to proportional phrase-level timing per scene when the model is unavailable, and says so in timing_source. """ from __future__ import annotations import argparse import json import re import sys from pathlib import Path ACCENT = "F97316" # warm orange, matches the plugin accent (BGR in ASS: 1673F9) def parse_args() -> argparse.Namespace: p = argparse.ArgumentParser(description=__doc__) p.add_argument("scene_timings", type=Path) p.add_argument("--words", type=Path, default=Path("output/word-timings.json")) p.add_argument("--captions", type=Path, default=Path("output/captions.ass")) p.add_argument("--model", default="base") p.add_argument("--language", default="en") p.add_argument("--max-phrase-words", type=int, default=7) p.add_argument("--min-phrase-words", type=int, default=3) return p.parse_args() def tokenize(text: str) -> list[str]: return [t for t in re.findall(r"\S+", text) if t] def norm(token: str) -> str: return re.sub(r"[^\w']", "", token).lower() def load_model(name: str): try: from faster_whisper import WhisperModel except ImportError: return None return WhisperModel(name, device="cpu", compute_type="int8") def align_scene(model, scene: dict, language: str) -> tuple[list[dict], str]: """Return (words, timing_source) for one scene, offsets applied.""" narration = tokenize(scene.get("narration_segment", "")) offset = float(scene["start_seconds"]) duration = float(scene["end_seconds"]) - offset if not narration: return [], "empty_narration" recognized: list[tuple[str, float, float]] = [] if model is not None: segments, _info = model.transcribe( scene["audio_file"], language=language, word_timestamps=True, initial_prompt=" ".join(narration)[:200], ) for seg in segments: for w in seg.words or []: recognized.append((w.word.strip(), w.start, w.end)) words: list[dict] = [] if recognized: # Greedy monotonic match: walk narration; consume the next recognized # word whose normalized form matches, else borrow neighbor timing. ri = 0 last_end = 0.0 for token in narration: start = end = None for look in range(ri, min(ri + 3, len(recognized))): if norm(recognized[look][0]) == norm(token): start, end = recognized[look][1], recognized[look][2] ri = look + 1 break if start is None: if ri < len(recognized): start, end = recognized[ri][1], recognized[ri][2] ri += 1 else: start, end = last_end, min(duration, last_end + 0.3) start = max(last_end, min(start, duration)) end = max(start + 0.01, min(end, duration)) last_end = end words.append({"word": token, "start_seconds": round(offset + start, 3), "end_seconds": round(offset + end, 3)}) source = "faster_whisper_alignment" else: # Proportional fallback: distribute by character weight across the clip. weights = [max(1, len(norm(t))) for t in narration] total = sum(weights) cursor = 0.0 for token, weight in zip(narration, weights): span = duration * weight / total words.append({"word": token, "start_seconds": round(offset + cursor, 3), "end_seconds": round(offset + cursor + span, 3)}) cursor += span source = "proportional_fallback" for w in words: w["scene_number"] = scene["scene_number"] w["audio_file"] = scene["audio_file"] w["timing_source"] = source return words, source def phrase_groups(words: list[dict], lo: int, hi: int) -> list[list[dict]]: groups: list[list[dict]] = [] current: list[dict] = [] for w in words: if current and (len(current) >= hi or w["scene_number"] != current[0]["scene_number"]): groups.append(current) current = [] current.append(w) if len(current) >= lo and re.search(r"[.!?,;:]$", w["word"]): groups.append(current) current = [] if current: groups.append(current) return groups ASS_HEADER = """[Script Info] ScriptType: v4.00+ PlayResX: 1920 PlayResY: 1080 [V4+ Styles] Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding Style: Karaoke,Arial,58,&H00252525,&H001673F9,&H00FFFFFF,&H64FFFFFF,-1,0,0,0,100,100,0,0,1,3,0,2,120,120,64,1 [Events] Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text """ def ass_time(seconds: float) -> str: cs = int(round(seconds * 100)) return f"{cs // 360000}:{(cs // 6000) % 60:02d}:{(cs // 100) % 60:02d}.{cs % 100:02d}" def write_ass(path: Path, groups: list[list[dict]]) -> None: lines = [ASS_HEADER] for group in groups: start, end = group[0]["start_seconds"], group[-1]["end_seconds"] parts = [] for w in group: k = max(1, int(round((w["end_seconds"] - w["start_seconds"]) * 100))) parts.append(f"{{\\k{k}}}{w['word']} ") text = "".join(parts).rstrip() lines.append(f"Dialogue: 0,{ass_time(start)},{ass_time(end)},Karaoke,,0,0,0,,{text}\n") path.parent.mkdir(parents= [truncated — 1090 more characters] ``` ### scripts/analyze_audio.py ```python #!/usr/bin/env python3 """Analyze narration rhythm and emit scene-ready timestamps. FFprobe supplies the container duration. pydub supplies loudness plus silence and speech ranges. When --scene-count is provided, scene cuts snap to nearby silence midpoints and fall back to evenly spaced cuts only when no usable pause exists. """ from __future__ import annotations import argparse import json import math import subprocess import sys from pathlib import Path from typing import Any def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("audio", type=Path, help="Narration audio file") parser.add_argument("--scene-count", type=int, help="Number of timed scenes to emit") parser.add_argument( "--output", type=Path, default=Path("output/audio-analysis.json"), help="JSON output path", ) parser.add_argument("--min-silence-ms", type=int, default=300) parser.add_argument( "--silence-thresh-dbfs", type=float, help="Explicit threshold; default is derived from average loudness", ) parser.add_argument("--seek-step-ms", type=int, default=10) parser.add_argument( "--minimum-scene-ms", type=int, default=1000, help="Minimum spacing between adjacent cuts", ) return parser.parse_args() def require_pydub() -> tuple[Any, Any]: try: from pydub import AudioSegment, silence except (ImportError, ModuleNotFoundError) as exc: raise SystemExit( "pydub could not be imported. Install `pydub`; on Python 3.13+ also " "install `audioop-lts` (or use Python 3.12 or earlier)." ) from exc return AudioSegment, silence def ffprobe_duration_seconds(audio_path: Path) -> float: command = [ "ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "default=noprint_wrappers=1:nokey=1", str(audio_path), ] try: result = subprocess.run(command, check=True, capture_output=True, text=True) return float(result.stdout.strip()) except FileNotFoundError as exc: raise SystemExit("ffprobe is required but was not found on PATH.") from exc except (subprocess.CalledProcessError, ValueError) as exc: details = getattr(exc, "stderr", "") or str(exc) raise SystemExit(f"ffprobe could not inspect {audio_path}: {details.strip()}") from exc def seconds_ranges(ranges_ms: list[list[int]]) -> list[dict[str, float]]: return [ { "start_seconds": round(start / 1000, 3), "end_seconds": round(end / 1000, 3), "duration_seconds": round((end - start) / 1000, 3), } for start, end in ranges_ms ] def choose_scene_boundaries( duration_ms: int, silent_ranges: list[list[int]], scene_count: int, minimum_scene_ms: int, ) -> tuple[list[int], list[dict[str, Any]]]: if scene_count < 1: raise SystemExit("--scene-count must be at least 1.") if scene_count == 1: return [0, duration_ms], [] if duration_ms < scene_count: raise SystemExit("The requested scene count exceeds the audio duration in milliseconds.") # Do not force an impossible minimum; preserve ordered, non-empty scenes. effective_minimum = min(minimum_scene_ms, max(1, duration_ms // scene_count)) pause_midpoints = sorted((start + end) // 2 for start, end in silent_ranges) boundaries = [0] decisions: list[dict[str, Any]] = [] for index in range(1, scene_count): ideal = round(duration_ms * index / scene_count) remaining_scenes = scene_count - index lower = boundaries[-1] + effective_minimum upper = duration_ms - (remaining_scenes * effective_minimum) if lower > upper: lower = boundaries[-1] + 1 upper = duration_ms - remaining_scenes eligible = [point for point in pause_midpoints if lower <= point <= upper] if eligible: chosen = min(eligible, key=lambda point: (abs(point - ideal), point)) source = "silence_midpoint" else: chosen = min(max(ideal, lower), upper) source = "uniform_fallback" boundaries.append(chosen) decisions.append( { "boundary_after_scene": index, "timestamp_seconds": round(chosen / 1000, 3), "source": source, "ideal_timestamp_seconds": round(ideal / 1000, 3), } ) boundaries.append(duration_ms) return boundaries, decisions def scene_ranges(boundaries: list[int]) -> list[dict[str, Any]]: scenes: list[dict[str, Any]] = [] for index, (start, end) in enumerate(zip(boundaries, boundaries[1:]), start=1): scenes.append( { "scene_number": index, "start_seconds": round(start / 1000, 3), "end_seconds": round(end / 1000, 3), "duration_seconds": round((end - start) / 1000, 3), } ) return scenes def main() -> int: args = parse_args() if not args.audio.is_file(): raise SystemExit(f"Audio file not found: {args.audio}") if args.min_silence_ms < 1 or args.seek_step_ms < 1 or args.minimum_scene_ms < 1: raise SystemExit("Silence and scene timing values must be positive integers.") AudioSegment, silence = require_pydub() audio = AudioSegment.from_file(args.audio) duration_ms = len(audio) ffprobe_duration = ffprobe_duration_seconds(args.audio) if math.isinf(audio.dBFS): derived_threshold = -45.0 else: derived_threshold = max(-50.0, min(-35.0, audio.dBFS - 14.0)) threshold = ( args.silence_thresh_dbfs if args.silence_thresh_dbfs is not None else derived_threshold ) silence_options = { "min_silence_len": arg [truncated — 1866 more characters] ``` ### scripts/canonicalize_storyboard.py ```python #!/usr/bin/env python3 """Build a pixel-exact 4:3 landscape storyboard master from scene images. Every populated and unused grid slot is exactly 16:9. Scene artwork is scaled to cover and center-cropped; it is never stretched. FFmpeg and FFprobe are the only external dependencies. """ from __future__ import annotations import argparse import json import math import re import shutil import subprocess import sys import tempfile from pathlib import Path SCENE_PATTERN = re.compile(r"^scene-(\d+)\.(?:png|jpe?g|webp)$", re.IGNORECASE) def fail(message: str) -> None: raise SystemExit(f"error: {message}") def run(command: list[str]) -> None: completed = subprocess.run(command, capture_output=True, text=True) if completed.returncode: detail = completed.stderr.strip() or completed.stdout.strip() fail(f"command failed: {' '.join(command)}\n{detail}") def image_size(path: Path) -> tuple[int, int]: completed = subprocess.run( [ "ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries", "stream=width,height", "-of", "json", str(path), ], capture_output=True, text=True, ) if completed.returncode: fail(f"ffprobe could not inspect {path}: {completed.stderr.strip()}") stream = json.loads(completed.stdout)["streams"][0] return int(stream["width"]), int(stream["height"]) def find_scenes(scene_dir: Path) -> list[Path]: numbered: list[tuple[int, Path]] = [] for path in scene_dir.iterdir(): match = SCENE_PATTERN.match(path.name) if match: numbered.append((int(match.group(1)), path)) numbered.sort() if not numbered: fail(f"no scene-NN images found in {scene_dir}") expected = list(range(1, len(numbered) + 1)) actual = [number for number, _ in numbered] if actual != expected: fail(f"scene numbers must be contiguous from 1; found {actual}") return [path for _, path in numbered] def choose_grid( scene_count: int, canvas_width: int, canvas_height: int, padding: int, gutter: int, ) -> tuple[int, int, int, int]: best: tuple[int, int, int, int, int, int] | None = None for columns in range(1, scene_count + 1): rows = math.ceil(scene_count / columns) available_width = canvas_width - 2 * padding - (columns - 1) * gutter available_height = canvas_height - 2 * padding - (rows - 1) * gutter if available_width <= 0 or available_height <= 0: continue unit = min(available_width // (16 * columns), available_height // (9 * rows)) if unit < 1: continue panel_width = 16 * unit panel_height = 9 * unit candidate = (unit, -columns * rows, columns, rows, panel_width, panel_height) if best is None or candidate[:2] > best[:2]: best = candidate if best is None: fail("canvas is too small for the requested scene count, padding, and gutter") _, _, columns, rows, panel_width, panel_height = best return columns, rows, panel_width, panel_height def main() -> None: parser = argparse.ArgumentParser() parser.add_argument("--scene-dir", type=Path, required=True) parser.add_argument("--output", type=Path, required=True) parser.add_argument("--manifest", type=Path) parser.add_argument("--canvas-width", type=int, default=1600) parser.add_argument("--canvas-height", type=int, default=1200) parser.add_argument("--padding", type=int, default=24) parser.add_argument("--gutter", type=int, default=12) parser.add_argument("--background", default="F7F3E8") parser.add_argument("--border", default="1F2937") parser.add_argument("--border-width", type=int, default=2) args = parser.parse_args() if shutil.which("ffmpeg") is None or shutil.which("ffprobe") is None: fail("ffmpeg and ffprobe must be installed") if args.canvas_width * 3 != args.canvas_height * 4: fail("master canvas must satisfy width * 3 = height * 4 (exact 4:3)") if min(args.padding, args.gutter, args.border_width) < 0: fail("padding, gutter, and border width cannot be negative") if not args.scene_dir.is_dir(): fail(f"scene directory does not exist: {args.scene_dir}") scenes = find_scenes(args.scene_dir) columns, rows, panel_width, panel_height = choose_grid( len(scenes), args.canvas_width, args.canvas_height, args.padding, args.gutter, ) if panel_width * 9 != panel_height * 16: fail("internal error: calculated panel is not exact 16:9") grid_width = columns * panel_width + (columns - 1) * args.gutter grid_height = rows * panel_height + (rows - 1) * args.gutter origin_x = (args.canvas_width - grid_width) // 2 origin_y = (args.canvas_height - grid_height) // 2 cell_count = columns * rows args.output.parent.mkdir(parents=True, exist_ok=True) manifest_path = args.manifest or args.output.with_suffix(".geometry.json") manifest_path.parent.mkdir(parents=True, exist_ok=True) slots = [] for index in range(cell_count): row, column = divmod(index, columns) x = origin_x + column * (panel_width + args.gutter) y = origin_y + row * (panel_height + args.gutter) slots.append( { "slot": index + 1, "scene_number": index + 1 if index < len(scenes) else None, "blank": index >= len(scenes), "x": x, "y": y, "width": panel_width, "height": panel_height, "aspect_ratio": "16:9", "ratio_verified": panel_width * 9 == panel_height * 16, } ) with tempfile.TemporaryDirectory(prefix="storyboard-canonical-") as temporary: temporary_path = Path(tempo [truncated — 3867 more characters] ``` ### assets/remotion-overlay-template/remotion.config.ts ```typescript import {existsSync} from "node:fs"; import {Config} from "@remotion/cli/config"; const configuredBrowser = process.env.REMOTION_BROWSER_EXECUTABLE; const windowsBrowsers = [ "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe", "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe", "C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe", "C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe", ]; if (configuredBrowser) { Config.setBrowserExecutable(configuredBrowser); } else if (process.platform === "win32" && process.arch === "arm64") { const installedBrowser = windowsBrowsers.find(existsSync); if (installedBrowser) Config.setBrowserExecutable(installedBrowser); } Config.setCodec("h264"); Config.setAudioCodec("aac"); Config.setPixelFormat("yuv420p"); Config.setVideoImageFormat("jpeg"); Config.setJpegQuality(90); Config.setCrf(18); ``` [15 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]