# Project export: Ariadne This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: A custom AI benchmarking and evaluation tool that runs on your code and grades on your criteria - Devpost: https://devpost.com/software/ariadne-gu1n8j - GitHub: https://github.com/ArkXero/Ariadne - Video: https://www.youtube.com/embed/afxfph59mCw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — Ronit Singh (17 commits) ## Devpost submission (written by the team) ### Overview MY MOM ANUPAM SINGH MADE A DEVPOST ACCOUNBT AND LET ME, RONIT SINGH, PARTICPATE ### Inspiration As I've gotten more caught up in the developer and AI world, I began to become very interested in the progress of Frontier-Level AI models from OpenAI, Anthropic, and more recently labs like DeepSeek and Moonshot for programming and general use tasks. A big part of this recent interest is monitoring model releases and their performance on AI benchmarks like SWEBenchPro, DeepSWE, and Artificial Analysis. But, I felt a big disconnect from these arbitrary scores I saw on these benchmarks and the performance I saw in my own work. So, I made Ariadne. ### What it does Ariadne is a AI testing and Benchmarking tool that is runs on your code and grades on what YOU value. ### How we built it Ariadne is a local-first Node.js and TypeScript CLI. I used Codex with GPT-5.6 throughout Build Week for the following: exploring the existing architecture and iterating on it, testing edge cases, reviewing changes, making and planning new features, and testing Ariadne against disposable repositories. Commander wires thin commands, Zod and YAML validate inputs, Execa manages processes, Git provides attributable change evidence, and Ink/React powers the TUI. There is no hosted backend or hidden database. ### Challenges we ran into The hardest part was executing the vision for Ariadne, I had a general Idea of what i wanted to do but I had to stumble along the way to really find the niche this product needed to target and what it was meant to function as for devs. Another thing was the TUI, codex is still not the best at designing TUIs, so balancing info and good design for the TUIs using my own taste that codex doesn't have took me a while. ### Accomplishments we're proud of I'm happy I grew this project from a simple runnable tool to a fully built CLI tool that has guardrails for safety and varying degrees of verbosity so anyone can use it how they want I am are especially proud of the safety details: ignored-file checks can still catch paths such as .env, read-only tasks are verified against repository mutations, promotion revalidates repository identity and cleanliness under a lock, and conflicts are preflighted and rolled back rather than left in the user's checkout. \ ### What we learned I learned what devs actually need in their tools, as Codex flagged a lot of things along the way that I wouldn't have thought about. I also learned how to more effectively use AI in my day-to-day dev work ## README (from the GitHub repository) # Ariadne Ariadne is a local-first CLI for evaluating coding-agent reliability and orchestrating file-based task workflows. It can run tasks in the shared checkout or detached Git worktrees, capture successful changes as durable local result commits, and promote reviewed result closures explicitly. Ariadne is an observability and policy tool. It is not an operating-system sandbox, secrets vault, hosted service, or proof that command-like output actually executed. ## Requirements and installation - Node.js 20 or newer - pnpm 10.34.1 for development - Git when changed-file/diff policies or worktree isolation are enabled ```sh pnpm install --frozen-lockfile pnpm build pnpm link ariadne --help ``` The repository is pinned to pnpm 10.34.1. With pnpm 11, use `pnpm add --global .` instead of the removed global-link behavior. A packed or published npm package needs Node and its production dependencies, not pnpm. Release validation installs the real tarball through npm locally and globally, npm exec, pnpm locally, and the direct package binary. ## Quick start Run these commands in a repository you want to evaluate: ```sh ariadne init ariadne doctor ariadne plan --all ariadne run --all ariadne list --batches ariadne report ariadne tui ``` Run Ariadne from the project root containing `ariadne.yml`. Nested invocation fails with project-root guidance instead of searching parent directories or writing state in the wrong folder. In an interactive terminal, `init` offers a repository-aware Default setup and a Custom setup. Default detects the project type, package manager, validation script, installed Codex/Claude Code executable, and Git worktree capability; it imports the strongest detected validation command as a task. Custom additionally configures task dependencies, isolation, concurrency, retries, sensitive-file protections, change limits, and timeouts, then provides YAML and file-diff review before writing. `ariadne init --yes` accepts detected defaults without prompts. Plain non-interactive `ariadne init` keeps the portable example-agent behavior used by automation. `--custom` requires a TTY. An existing `ariadne.yml` is never overwritten automatically. Interactive `init` defaults to validation and offers explicit Default/Custom replacement. Replacement always shows a diff, validates the proposal before touching the original, creates an ignored timestamped backup, and then performs atomic writes. Existing task files are never overwritten. ```yaml version: 5 agent: command: kind: exec file: codex args: [exec, --sandbox, workspace-write, "-"] timeout_ms: 600000 model_label: gpt-5.6-sol tasks: directory: .ariadne/tasks verification: commands: - kind: exec file: pnpm args: [test] timeout_ms: 300000 execution: termination_grace_ms: 2000 concurrency: 1 failure_mode: continue isolation: shared worktree: retention: on-failure preparation: commands: [] timeout_ms: 600000 checks: forbidden_files: [.env, ".env.*"] forbidden_commands: ["rm -rf"] max_changed_files: 20 max_diff_lines: 500 ``` Direct `exec` specifications preserve argument boundaries. Use `{ kind: shell, command: "pnpm typecheck && pnpm test" }` only when shell syntax is intentional. Versionless and v1–v4 configurations remain readable through compatibility adapters. Version 5 adds declared model provenance and optional professional benchmarking. ## Workflow tasks Tasks are strict YAML files loaded recursively from `tasks.directory`: ```yaml id: package name: Validate and package dependsOn: [integration-tests] workspaceMode: mutable retry: attempts: 3 delayMs: 1000 backoff: fixed verify: - kind: exec file: pnpm args: [build] metadata: description: Rebuild and verify the distributable package. group: release tags: [package, verification] issue: 42 prompt: Validate and package the project. ``` `dependsOn` is case-insensitive and includes transitive dependencies when a task is selected. `workspaceMode` defaults to `mutable`. In shared isolation, mutable tasks are exclusive and only read-only tasks may overlap. In worktree isolation, mutable tasks may overlap because each attempt has a detached checkout. Omitted `verify` inherits global verification; `verify: []` disables it. Retries in shared mode preserve the current tree. Worktree retries start from a fresh checkout of the recorded source plus successful dependency results; failed-attempt mutations are not inherited. V3 `parallelSafe: true` adapts to `workspaceMode: read-only` with a migration warning. IDs must match `[A-Za-z0-9][A-Za-z0-9._-]{0,63}`. Filename-derived IDs are supported when valid. Duplicate IDs and dependencies are compared case-insensitively. Missing, self, duplicate, and cyclic dependencies fail before execution. ## CLI Global flags are `--verbose`, `--quiet`, `--json`, and `--no-color`. Machine-readable modes reserve stdout for the payload; warnings and progress go to stderr. ```sh ariadne plan package --concurrency 2 ariadne run package ariadne run --task lint --task test ariadne run --all --failure-mode fail-fast ariadne run --all --isolation worktree ariadne benchmark typescript-cli-quality ariadne benchmark typescript-cli-quality --json ariadne plan --all --isolation worktree --allow-dirty-base ariadne resume --concurrency 2 ariadne rerun --failed ariadne rerun --task package ariadne list --tasks --format wide ariadne list --batches --format json ariadne list --batches --format csv --output exports/batches.csv ariadne report --run ariadne report --batch --output reports/workflow.html ariadne tui ariadne changes ariadne diff --output exports/result.patch ariadne diff --output exports/result.patch --force ariadne status ariadne apply ariadne discard ariadne worktree clean --dry-run ``` `run` with no selectors remains equivalent to `--all`. `plan` is read-only: it creates no run or batch record and launches no processes. `benchmark` accepts exactly one benchmark-enabled task and is intentionally outside basic onboarding; ordinary `run` never invokes a judge or incurs judge-model cost. `list` defaults to child task attempts. `report` follows `.ariadne/latest.json` by default. Existing list format flags remain aliases. See [Professional benchmarking](./docs/benchmarking.md) and [the CLI contract](./docs/cli-contract.md). `tui` opens a keyboard-first workflow control surface over the same planner, scheduler, review services, compatibility readers, and canonical report models as the CLI. Press `p` to plan and run work; use Tab on the dashboard to select attention categories and review results or retained workspaces. Result detail supports bounded per-file diffs, retry comparison, safe patch export, eligibility/preflight review, explicit apply/discard confirmation, and conflict diagnostics. Workspace detail supports pure cleanup previews followed by selected or bulk confirmed cleanup. Attached in-process workflows can still be cancelled, resumed, or rerun from history; persisted running/incomplete records from another or restarted process are labeled `no active runtime attached`. Redirected use exits 2 without ANSI output. `--verbose`, `--no-color`, `NO_COLOR`, ASCII fallback, responsive `100/60/40` layouts, contextual `?` help, and `r` reconciliation remain supported. Remote execution and mouse-first behavior remain out of scope. See [Ariadne TUI](./docs/tui.md). ## Records and reports ```text .ariadne/ ├── latest.json ├── batches/ │ ├── latest.json │ └── / │ ├── batch.json │ └── report.html ├── worktrees//workspace.json ├── promotions/.json ├── actions/.json ├── exports/-.patch └── runs/ ├── latest.json └── / ├── run.json ├── report.html └── artifacts//... [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 145 recognized source files, 1227 KB. - Python (language) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Codex — evidence: config files committed to the repository ## Codebase structure (from repository index) ### Files (120 of 161) ``` .codex/BASELINE.md .github/workflows/ci.yml .gitignore .npmignore AGENTS.md CHANGELOG.md docs/agent-adapters.md docs/architecture.md docs/benchmarking.md docs/cli-contract.md docs/decisions/0001-explicit-process-specs.md docs/decisions/0002-crash-safe-run-directories.md docs/decisions/0003-serial-same-working-tree.md docs/decisions/0004-attributed-repository-evidence.md docs/decisions/0005-versioned-read-adapters.md docs/decisions/0006-declaration-based-workflow-orchestration.md docs/decisions/0007-git-worktree-isolation-and-explicit-promotion.md docs/decisions/0008-ink-alternate-screen-tui.md docs/decisions/0009-operational-tui-runtime-events.md docs/decisions/0010-change-review-application-services.md docs/design-system.md docs/known-limitations.md docs/release-test-matrix.md docs/releasing.md docs/run-format.md docs/run-lifecycle.md docs/supported-environments.md docs/task-isolation.md docs/testing.md docs/troubleshooting.md docs/tui.md docs/workflows.md examples/sample-eval/.gitignore examples/sample-eval/agent.mjs examples/sample-eval/ariadne.yml examples/sample-eval/README.md examples/sample-eval/tasks/01-pass-notes.yml examples/sample-eval/tasks/02-fail-verification.yml examples/sample-eval/tasks/03-forbidden-file.yml examples/sample-eval/tasks/04-forbidden-command-log.yml examples/sample-eval/verify.mjs examples/self-hosting/README.md examples/self-hosting/run-scenarios.mjs LICENSE package.json README.md scripts/clean.mjs scripts/codex-loop.sh scripts/package-smoke.mjs scripts/release-check.mjs scripts/release-contract.mjs scripts/release-profile.mjs scripts/smoke-test.mjs scripts/tui-dogfood.mjs scripts/tui-pty-driver.py scripts/tui-pty-smoke.mjs src/cli.ts src/commands/benchmark.ts src/commands/changes.ts src/commands/doctor.ts src/commands/init.ts src/commands/list.ts src/commands/plan.ts src/commands/report.ts src/commands/rerun.ts src/commands/resume.ts src/commands/run.ts src/commands/tui.tsx src/commands/workflow-signals.ts src/commands/worktree.ts src/core/atomic.ts src/core/batch-persistence.ts src/core/batch-reader.ts src/core/batches.ts src/core/benchmark.ts src/core/bounded-map.ts src/core/change-application.ts src/core/change-capture.ts src/core/command-utils.ts src/core/config.ts src/core/doctor.ts src/core/errors.ts src/core/forbidden-commands.ts src/core/forbidden-files.ts src/core/git.ts src/core/init.ts src/core/management-actions.ts src/core/management-lock.ts src/core/onboarding-state.ts src/core/path-containment.ts src/core/path-match.ts src/core/persistence.ts src/core/process-runner.ts src/core/project-detector.ts src/core/promotion.ts src/core/report.ts src/core/run-reader.ts src/core/runner.ts src/core/runs.ts src/core/scorer.ts src/core/task-loader.ts src/core/terminal-sanitize.ts src/core/version.ts src/core/workflow-application.ts src/core/workflow-control.ts src/core/workflow-graph.ts src/core/workflow-planner.ts src/core/workflow-report.ts src/core/workflow-runner.ts src/core/workflow-runtime.ts src/core/workspace-application.ts src/core/workspace-manager.ts src/schema/batch-record.ts src/schema/benchmark.ts src/schema/management-action-record.ts src/schema/process-result.ts src/schema/promotion-record.ts src/schema/run-record.ts src/schema/task-result.ts src/schema/trace.ts [41 more files omitted for size] ``` ### Dependencies - package.json: @clack/prompts@^0.11.0, @types/fs-extra@^11.0.4, @types/node@^24.10.1, @types/react@^19.2.17, commander@^14.0.2, execa@^9.6.0, fs-extra@^11.3.2, ink@6.8.0, ink-testing-library@^4.0.0, minimatch@^10.1.1, react@^19.0.0, string-width@^7.2.0, strip-ansi@^7.1.0, typescript@^5.9.3, vitest@^4.1.8, yaml@^2.8.2, zod@^4.1.13 ### Recent commits (newest first) - Add professional benchmark evaluation workflow - Run npm release checks portably on Windows - Normalize Windows promotion line endings - Use a Windows-safe npm global prefix - Quote Windows shim paths in package smoke - Allow Windows retry integration test to finish - Run Windows command shims portably - Exercise TUI rendering correctly in CI - Keep TUI input subscribed across renders - Stabilize cross-platform integration tests - Fix cross-platform CI reliability - Add release-candidate verification and hardening gates - Expand reliability evaluation workflows - Add operational TUI workflow control surface - Refine task execution traces and reporting - Refine run tracing and reporting - Refactor Ariadne evaluation and reporting pipeline - Merge pull request #2 from ArkXero/feature/sample-eval-reporting - fixing merge conflict - Add sample eval docs and stronger run reporting ## Key source files (fetched from GitHub, selected and truncated for size) ### AGENTS.md ```markdown # AGENTS.md Guidance for AI coding agents and human contributors working on Ariadne. ## Project purpose Ariadne is a local-first developer tool for evaluating AI coding-agent reliability. It provides a Node.js CLI that runs eval tasks, captures traces, scores behavior, and generates reliability reports. Keep the MVP focused: - Local CLI first. - GitHub Action support can build on the CLI later. - No SaaS auth, hosted dashboards, teams, billing, or database until explicitly requested. - Prefer transparent trace JSON over hidden state. ## Tech stack - TypeScript - Node.js ESM - pnpm - commander for CLI commands - zod for config validation - yaml for YAML parsing - execa for shell commands - fs-extra for filesystem helpers - minimatch for file pattern checks ## Commands Use pnpm. Use pnpm `10.34.1`; exact developer-preview global linking depends on pnpm 10 behavior. ```sh pnpm install pnpm typecheck pnpm build pnpm dev --help pnpm ariadne --help pnpm ariadne -h ``` For developer-preview global installation: ```sh pnpm install pnpm build pnpm link ariadne --help ``` pnpm 11 removed global linking; when testing under pnpm 11, use `pnpm add --global .` instead. For local smoke tests after `pnpm build`: ```sh node dist/cli.js init node dist/cli.js doctor node dist/cli.js run node dist/cli.js report ``` Prefer running smoke tests in a temporary git repo so generated `.ariadne/` files do not pollute this repository. `pnpm smoke` verifies global binary installation from a disposable staged package with an isolated temporary `PNPM_HOME` and removes it afterward, leaving both the checkout and host global pnpm installation untouched. ## Repository map - `src/cli.ts`: CLI entrypoint and command wiring. - `src/commands/`: thin command handlers. - `src/core/config.ts`: `ariadne.yml` loading and validation. - `src/core/task-loader.ts`: task YAML discovery and validation. - `src/core/doctor.ts`: pre-run config, task, executable, and package script diagnostics. - `src/core/runner.ts`: task execution, verification, trace capture, run JSON writing. - `src/core/scorer.ts`: pass/fail scoring checks. - `src/core/git.ts`: git status and diff helpers. - `src/core/forbidden-files.ts`: forbidden file snapshot checks, including ignored files. - `src/core/report.ts`: terminal and HTML report generation. - `src/types/index.ts`: shared TypeScript interfaces. Keep commands thin. Put behavior in `src/core/`. ## Coding rules - Read relevant files before editing. - Make minimal, targeted changes. - Preserve public config and task shapes unless request requires changes. - Keep local-only behavior as default. - Do not add dependencies unless there is clear value. - Do not introduce frameworks, servers, databases, queues, auth, telemetry, or hosted services for MVP work. - Keep generated report output deterministic enough to test. - Prefer explicit typed data over ad hoc strings. - Keep CLI errors actionable. ## Runner behavior `ariadne run` should: - Read `a [truncated — 2388 more characters] ``` ### TESTING.md ```markdown # Testing Ariadne ## Prerequisites - Node.js 20 or newer - pnpm 10.34.1 - Git ```sh node --version pnpm --version git --version pnpm install --frozen-lockfile ``` ## Authoritative gate ```sh pnpm check ``` The gate runs, in order: 1. `pnpm lint` — TypeScript no-emit validation. 2. `pnpm build` — deletes `dist`, then compiles TypeScript. 3. `pnpm test` — unit, component, graph/planner, scheduler/retry, resume/rerun, worktree/v2-capture/review/promotion/cleanup, persistence/history/report, Git/policy, and black-box CLI suites. 4. `pnpm test:tui-pty` — uses the POSIX system pseudo-terminal facility when available for selection, planning, launch, live output, detach/reopen, cancellation, resize signaling, and teardown; unsupported platforms retain simulated-TTY coverage. 5. `pnpm smoke` — isolated global-binary installation plus passing and ignored-forbidden-file CLI flows. It uses no-argument `pnpm link` on pnpm 10 and pnpm 11's documented `pnpm add --global .` replacement, always from a disposable staged package. 6. `pnpm test:package` — packs the npm tarball, asserts clean contents and shebang/bin metadata, installs it outside the checkout, and exercises every help screen plus shared/worktree execution, changes/diff/export, clean apply, conflict rollback, idempotent discard, cleanup dry run/execution/history, failures, interruption, resume/rerun, corrupt history, renderer consistency, and latest pointers. The signal case is skipped on Windows because programmatic SIGINT delivery is not portable there. 7. `pnpm test:release` — checks package metadata/license/version consistency, the exact packed allowlist, documentation links, and YAML fence syntax without network access. The TUI-focused suites cover shared workflow/change/workspace application services, selection/options/confirmation, attention/result/manifest/diff navigation, retry comparison, elevated-risk acknowledgement, apply/discard/export/cleanup flows, runtime event ordering/overflow/failure, redacted UTF-8 streaming, retry/block reduction, fixed-height master/detail and log-dominant compact layouts, list windowing, footer packing, semantic color/ASCII fallbacks, live-output bounds, detach/headless continuation, signal finalization, PTY operation, terminal restoration, and CLI refusal contracts. Run focused stages while developing: ```sh pnpm typecheck pnpm build pnpm test pnpm test:tui-pty pnpm smoke pnpm test:package pnpm test:release pnpm release:profile pnpm release:check pnpm dogfood:tui ``` `pnpm release:profile` records point-in-time history, workspace, TUI, wide-workflow, and log scale evidence without enforcing brittle latency thresholds. The 20,000-line diff case remains a deterministic bounded-page test rather than a machine-speed assertion. `pnpm release:check` is the heavyweight candidate gate: it snapshots repository-owned files into a disposable directory, performs a frozen install, runs `pnpm check` twice, runs the profile and production dependency audit, inspect [truncated — 8653 more characters] ``` ### package.json ``` { "name": "@arkxronts/ariadne", "version": "0.1.0", "description": "Local CLI for coding-agent reliability and testing evals.", "type": "module", "bin": { "ariadne": "dist/cli.js" }, "files": [ "dist", "README.md", "LICENSE" ], "publishConfig": { "access": "public" }, "scripts": { "clean": "node scripts/clean.mjs", "build": "pnpm clean && tsc -p tsconfig.json", "check": "pnpm lint && pnpm build && pnpm test && pnpm test:tui-pty && pnpm smoke && pnpm test:package && pnpm test:release", "lint": "tsc -p tsconfig.json --noEmit", "prepack": "pnpm --silent build", "release:check": "node scripts/release-check.mjs", "release:profile": "node scripts/release-profile.mjs", "test": "vitest run --testTimeout=15000", "test:package": "node scripts/package-smoke.mjs", "test:release": "node scripts/release-contract.mjs", "test:tui-pty": "node scripts/tui-pty-smoke.mjs", "test:watch": "vitest", "dogfood:tui": "pnpm build && node scripts/tui-dogfood.mjs", "typecheck": "tsc -p tsconfig.json --noEmit", "dev": "pnpm build && node dist/cli.js", "ariadne": "node dist/cli.js", "smoke": "node scripts/smoke-test.mjs" }, "dependencies": { "@clack/prompts": "^0.11.0", "commander": "^14.0.2", "execa": "^9.6.0", "fs-extra": "^11.3.2", "ink": "6.8.0", "minimatch": "^10.1.1", "react": "^19.0.0", "string-width": "^7.2.0", "strip-ansi": "^7.1.0", "yaml": "^2.8.2", "zod": "^4.1.13" }, "devDependencies": { "@types/fs-extra": "^11.0.4", "@types/node": "^24.10.1", "@types/react": "^19.2.17", "ink-testing-library": "^4.0.0", "typescript": "^5.9.3", "vitest": "^4.1.8" }, "engines": { "node": ">=20" }, "packageManager": "pnpm@10.34.1", "repository": { "type": "git", "url": "git+https://github.com/ArkXero/Ariadne.git" }, "keywords": [ "ai", "cli", "coding-agents", "evaluation", "reliability" ], "license": "Apache-2.0", "bugs": { "url": "https://github.com/ArkXero/Ariadne/issues" }, "homepage": "https://github.com/ArkXero/Ariadne#readme" } ``` ### src/cli.ts ```typescript #!/usr/bin/env node import { Command, CommanderError, InvalidArgumentError } from "commander"; import { doctorCommand } from "./commands/doctor.js"; import { formatInitResult, initOnboardingCommand, initOutcomeJson } from "./commands/init.js"; import { listCommand, type ListFormat } from "./commands/list.js"; import { reportCommand } from "./commands/report.js"; import { exitCodeForBatch, runCommand } from "./commands/run.js"; import { planCommand } from "./commands/plan.js"; import { resumeCommand } from "./commands/resume.js"; import { rerunCommand } from "./commands/rerun.js"; import { applyCommand, changesCommand, diffCommand, discardCommand, statusCommand } from "./commands/changes.js"; import { worktreeCleanCommand, worktreeListCommand, worktreeRemoveCommand } from "./commands/worktree.js"; import { tuiCommand } from "./commands/tui.js"; import { benchmarkCommand, exitCodeForBenchmark } from "./commands/benchmark.js"; import { AriadneError, formatAriadneError } from "./core/errors.js"; import { getAriadneVersion } from "./core/version.js"; import type { FailureMode, IsolationStrategy } from "./types/index.js"; interface GlobalOptions { verbose?: boolean; quiet?: boolean; json?: boolean; color?: boolean; } function exitCodeForError(error: unknown): number { if (error instanceof InvalidArgumentError) return 2; if (!(error instanceof AriadneError)) return 70; if (error.category === "configuration" || error.category === "task_loading") return 2; if (error.category === "task_selection") return 3; if (error.category === "repository_validation") return 4; if (error.category === "workspace_preparation" || error.category === "workspace_management") return 14; if (error.category === "promotion_conflict") return 15; if (error.category === "benchmark_protocol") return 16; return 70; } function collect(value: string, previous: string[]): string[] { return [...previous, value]; } function parseFormat(value: string): ListFormat { if (["compact", "wide", "json", "csv", "markdown"].includes(value)) return value as ListFormat; throw new InvalidArgumentError("Format must be compact, wide, json, csv, or markdown."); } function parseConcurrency(value: string): number { const parsed = Number(value); if (!Number.isInteger(parsed) || parsed < 1 || parsed > 32) throw new InvalidArgumentError("Concurrency must be an integer from 1 through 32."); return parsed; } function parseFailureMode(value: string): FailureMode { if (value === "continue" || value === "fail-fast") return value; throw new InvalidArgumentError("Failure mode must be continue or fail-fast."); } function parseIsolation(value: string): IsolationStrategy { if (value === "shared" || value === "worktree") return value; throw new InvalidArgumentError("Isolation must be shared or worktree."); } function selectedIds(positional: string[], repeated: string[], all?: boolean): string[] | undefined { const values = [...positional, ...repeated]; if (all && values.length > 0) throw new InvalidArgumentError("--all cannot be combined with task IDs or --task."); const seen = new Set(); const result = values.filter((value) => { const key = value.toLowerCase(); if (seen.has(key)) return false; seen.add(key); return true; }); return result.length > 0 ? result : undefined; } const program = new Command(); const argv = process.argv[2] === "--" ? [...process.argv.slice(0, 2), ...process.argv.slice(3)] : process.argv; program.configureHelp({ showGlobalOptions: true }); program.exitOverride(); program .name("ariadne") .description("Run local coding-agent reliability evals and generate reports.") .version(await getAriadneVersion()) .option("--verbose", "Include stack traces and deeper diagnostics.") .option("--quiet", "Suppress progress and warning output.") .option("--json", "Write machine-readable JSON to stdout.") .option("--no-color", "Disable color and styled text output.") .hook("preAction", (command) => { const options = command.optsWithGlobals(); if (options.verbose && options.quiet) throw new InvalidArgumentError("--verbose and --quiet cannot be used together."); }); program.command("init").description("Configure Ariadne with repository-aware default or custom setup.") .option("-y, --yes", "Accept detected defaults without prompting.") .option("--custom", "Open Custom setup immediately (interactive terminals only).") .action(async (local: { yes?: boolean; custom?: boolean }, command: Command) => { const options = command.optsWithGlobals(); if (local.yes && local.custom) throw new InvalidArgumentError("--yes and --custom cannot be combined."); const interactive = Boolean(process.stdin.isTTY && process.stdout.isTTY && !options.json && !local.yes); if (local.custom && !interactive) throw new InvalidArgumentError("--custom requires an interactive terminal."); const outcome = await initOnboardingCommand(process.cwd(), { interactive, repositoryAware: interactive || Boolean(local.yes), custom: local.custom, json: options.json, quiet: options.quiet, color: options.color }); if (interactive) { if ("doctor" in outcome && !outcome.doctor.passed) process.exitCode = 2; return; } if (options.json) process.stdout.write(`${JSON.stringify(initOutcomeJson(outcome), null, 2)}\n`); else if (!options.quiet && "result" in outcome) process.stdout.write(`${formatInitResult(outcome.result)}\n`); if ("doctor" in outcome && !outcome.doctor.passed) process.exitCode = 2; }); program.command("doctor").description("Validate Ariadne configuration and commands before a run.") .option("-c, --config ", "Path to Ariadne config file.", "ariadne.yml") .action(async (local: { config: string }, command: Command) => { const options = command.optsWithGlobals(); const report = await doctorCommand(process.cwd(), local.confi [truncated — 12911 more characters] ``` ### src/types/index.ts ```typescript export const CURRENT_CONFIG_VERSION = 5 as const; export const CURRENT_RUN_SCHEMA_VERSION = 5 as const; export const CURRENT_BATCH_SCHEMA_VERSION = 3 as const; export const CURRENT_WORKSPACE_SCHEMA_VERSION = 1 as const; export const CURRENT_PROMOTION_SCHEMA_VERSION = 2 as const; export const CURRENT_CHANGE_ARTIFACT_SCHEMA_VERSION = 2 as const; export const CURRENT_MANAGEMENT_ACTION_SCHEMA_VERSION = 1 as const; export type LegacyConfigVersion = "versionless" | 1 | 2 | 3 | 4; export type IsolationStrategy = "shared" | "worktree"; export type WorktreeRetention = "always" | "on-failure" | "never"; export type WorkspaceMode = "mutable" | "read-only"; export type ProcessSpec = | { kind: "exec"; file: string; args: string[]; } | { kind: "shell"; command: string; }; export interface AriadneConfig { version: typeof CURRENT_CONFIG_VERSION; sourceVersion: LegacyConfigVersion | typeof CURRENT_CONFIG_VERSION; agent: { command: ProcessSpec; timeout_ms: number; model_label?: string; }; tasks: { directory: string; }; verification: { commands: ProcessSpec[]; timeout_ms: number; }; execution: { termination_grace_ms: number; concurrency: number; failure_mode: FailureMode; isolation: IsolationStrategy; worktree: { retention: WorktreeRetention; preparation: { commands: ProcessSpec[]; timeout_ms: number; }; }; }; checks: { forbidden_files: string[]; max_changed_files?: number; max_diff_lines?: number; forbidden_commands: string[]; }; benchmarking?: { judge: { command: ProcessSpec; model_label: string; timeout_ms: number; }; blind_candidate_identity: boolean; }; } export interface LoadedConfig { config: AriadneConfig; path: string; projectRoot: string; warnings: string[]; } export interface AriadneTask { id: string; name: string; file: string; prompt: string; metadata?: Record; dependsOn: string[]; workspaceMode: WorkspaceMode; retry: RetryPolicy; verify?: ProcessSpec[]; benchmark?: BenchmarkTaskContract; } export const BENCHMARK_ANCHORS = [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100] as const; export type BenchmarkAnchor = typeof BENCHMARK_ANCHORS[number]; export type BenchmarkFailureOutcome = "agent_failed" | "verification_failed" | "timeout" | "policy_failed"; export type BenchmarkFailureAction = "zero" | "keep" | "disqualify" | { cap: number }; export interface BenchmarkTaskContract { version: 1; id: string; rubric: Record; context_files: string[]; failure_policy: Record; } export type FailureMode = "continue" | "fail-fast"; export type RetryBackoff = "fixed" | "exponential"; export interface RetryPolicy { attempts: number; delayMs: number; backoff: RetryBackoff; } export type LifecycleStage = | "created" | "loading" | "validated" | "workspace_creating" | "workspace_ready" | "preparing" | "agent_running" | "agent_finished" | "verifying" | "collecting_trace" | "capturing_changes" | "workspace_cleanup" | "evaluating_policy" | "scoring" | "benchmark_packet" | "judging" | "benchmark_scoring" | "persisting" | "completed"; export type RunStatus = "running" | "completed" | "failed" | "interrupted" | "incomplete" | "abandoned"; export type TaskStatus = "running" | "passed" | "failed" | "interrupted" | "incomplete"; export type FailureCategory = | "configuration" | "task_loading" | "task_selection" | "repository_validation" | "workspace_preparation" | "workspace_management" | "promotion_conflict" | "agent_spawn" | "agent_nonzero" | "agent_timeout" | "verification_spawn" | "verification_nonzero" | "trace_collection" | "policy_violation" | "benchmark_protocol" | "persistence" | "user_interruption" | "internal"; export interface FailureRecord { category: FailureCategory; code: string; stage: LifecycleStage; message: string; source?: string; taskId?: string; details?: Record; } export interface LifecycleEvent { stage: LifecycleStage; at: string; taskId?: string; detail?: string; } export interface OutputPreview { head: string; tail: string; bytes: number; encoding: "utf8-replacement"; hadDecodingReplacement: boolean; } export interface ProcessCleanupResult { attempted: boolean; limitation?: string; gracefulSignal?: string; forceSignal?: string; gracefulSucceeded?: boolean; forceSucceeded?: boolean; error?: string; } export interface ProcessResult { kind: ProcessSpec["kind"]; executable: string; args: string[]; displayCommand: string; cwd: string; providedEnvironmentKeys: string[]; startedAt: string; completedAt: string; durationMs: number; exitCode: number | null; signal: string | null; timedOut: boolean; interrupted: boolean; spawnError?: string; stdoutArtifact: string; stderrArtifact: string; stdoutPreview: OutputPreview; stderrPreview: OutputPreview; cleanup: ProcessCleanupResult; redactionApplied?: boolean; } export type RepositoryChangeType = | "added" | "modified" | "deleted" | "renamed" | "copied" | "mode-changed" | "symlink-changed" | "untracked" | "ignored"; export interface RepositoryEntry { path: string; originalPath?: string; indexStatus: string; worktreeStatus: string; changeType: RepositoryChangeType; kind?: "file" | "symlink" | "other"; mode?: string; fingerprint?: string; } export interface RepositorySnapshot { available: boolean; unavailableReason?: string; head?: string; branch?: string; detached?: boolean; dirty: boolean; entries: RepositoryEntry[]; diffLineCount: number; } export interface ChangeEvidence { path: string; originalPath?: string; changeType: RepositoryChangeType; source: "preparation" | "agent" | "verification [truncated — 15161 more characters] ``` ### tests/release-hardening.test.ts ```typescript import { afterEach, describe, expect, it } from "vitest"; import path from "node:path"; import { mkdir, writeFile } from "node:fs/promises"; import { DEFAULT_IO_CONCURRENCY, mapWithConcurrency } from "../src/core/bounded-map.js"; import { loadRunHistory } from "../src/core/run-reader.js"; import { cleanupTempDirs, tempDir } from "./helpers.js"; afterEach(cleanupTempDirs); describe("release resource boundaries", () => { it("preserves input order while bounding asynchronous filesystem work", async () => { let active = 0; let maximum = 0; const values = Array.from({ length: 200 }, (_, index) => index); const result = await mapWithConcurrency(values, 7, async (value) => { active += 1; maximum = Math.max(maximum, active); await new Promise((resolve) => setImmediate(resolve)); active -= 1; return value * 2; }); expect(maximum).toBeLessThanOrEqual(7); expect(result).toEqual(values.map((value) => value * 2)); }); it("loads 1,000 valid and corrupt history records without unbounded fan-out", async () => { const cwd = await tempDir("ariadne-history-scale-"); const runs = path.join(cwd, ".ariadne", "runs"); await mkdir(runs, { recursive: true }); const indexes = Array.from({ length: 1_000 }, (_, index) => index); await mapWithConcurrency(indexes, DEFAULT_IO_CONCURRENCY, (index) => writeFile( path.join(runs, `run-${String(index).padStart(4, "0")}.json`), index % 100 === 0 ? "{broken" : `${JSON.stringify({ version: 1, startedAt: new Date(1_700_000_000_000 + index).toISOString(), results: [] })}\n` )); const history = await loadRunHistory(cwd); expect(history.records).toHaveLength(1_000); expect(history.records.filter((record) => record.ok)).toHaveLength(990); expect(history.warnings.filter((warning) => warning.includes("Could not parse"))).toHaveLength(10); }); }); ``` ### scripts/codex-loop.sh ```shell #!/usr/bin/env bash set -euo pipefail REPO="/Users/ronitsingh/programming/Ariadne" TASK_DIR="$REPO/.codex/tasks" LOG_DIR="$REPO/.codex/logs" SUMMARY_DIR="$REPO/.codex/summaries" BASELINE="$REPO/.codex/BASELINE.md" cd "$REPO" mkdir -p "$LOG_DIR" "$SUMMARY_DIR" if ! command -v codex >/dev/null 2>&1; then echo "codex command not found" exit 1 fi if [[ ! -f "$BASELINE" ]]; then echo "Missing $BASELINE" exit 1 fi for task_file in "$TASK_DIR"/*.md; do [[ -e "$task_file" ]] || { echo "No task files found in $TASK_DIR" exit 0 } task_name="$(basename "$task_file" .md)" timestamp="$(date +"%Y%m%d-%H%M%S")" log_file="$LOG_DIR/${timestamp}-${task_name}.jsonl" summary_file="$SUMMARY_DIR/${timestamp}-${task_name}.md" echo "" echo "============================================================" echo "Running Codex task: $task_name" echo "============================================================" if ! git diff --quiet || ! git diff --cached --quiet; then echo "Working tree has existing changes. Review/commit/stash before continuing." git status --short exit 1 fi prompt_file="$(mktemp)" cat > "$prompt_file" < { it("defines the supplied palette as the single semantic source", () => { expect(ARIADNE_PALETTE).toEqual({ coral: "#F6453C", warningOrange: "#F59E0B", successGreen: "#4ADE80", infoCyan: "#22D3EE", snow: "#FCF7F8", paleSlate: "#CED3DC", deepSlate: "#64748B" }); expect(ARIADNE_THEME).toMatchObject({ accent: ARIADNE_PALETTE.coral, foreground: ARIADNE_PALETTE.snow, muted: ARIADNE_PALETTE.paleSlate, border: ARIADNE_PALETTE.coral, focusedBorder: ARIADNE_PALETTE.coral, warning: ARIADNE_PALETTE.warningOrange, error: ARIADNE_PALETTE.coral, success: ARIADNE_PALETTE.successGreen, info: ARIADNE_PALETTE.infoCyan }); expect(ARIADNE_REPORT_INK).toBe("#9C2630"); for (const value of Object.values(ARIADNE_PALETTE)) expect(ARIADNE_CSS_VARIABLES).toContain(value); expect(ARIADNE_CSS_VARIABLES).toContain(ARIADNE_REPORT_INK); expect(ARIADNE_CSS_VARIABLES).toContain("--ariadne-border:var(--ariadne-coral)"); expect(ARIADNE_CSS_VARIABLES).toContain("--ariadne-focused-border:var(--ariadne-coral)"); }); it("maps Clack's legacy semantic colors onto semantic Ariadne roles", () => { const output = applyAriadneAnsiTheme("\u001B[36mactive\u001B[39m \u001B[90mrail\u001B[39m \u001B[33mwarning\u001B[39m \u001B[34msecondary\u001B[39m"); expect(output).toContain("\u001B[38;2;246;69;60mactive"); expect(output).toContain("\u001B[38;2;246;69;60mrail"); expect(output).toContain("\u001B[38;2;245;158;11mwarning"); expect(output).toContain("\u001B[38;2;206;211;220msecondary"); expect(output).toContain("\u001B[38;2;252;247;248m"); expect(output).not.toMatch(/\u001B\[(?:33|34|36|90)m/); expect(output.endsWith("\u001B[0m")).toBe(true); }); it("scopes the Init writer adapter and restores the original writer", async () => { const writes: string[] = []; const originalWrite = ((chunk: string | Uint8Array) => { writes.push(String(chunk)); return true; }) as NodeJS.WriteStream["write"]; const output = { write: originalWrite }; await withAriadneTerminalTheme(true, async () => { output.write("\u001B[33mWarning\u001B[39m"); }, output); expect(writes.join("")).toContain("\u001B[38;2;245;158;11mWarning"); expect(output.write).toBe(originalWrite); }); }); ``` ### tests/helpers.ts ```typescript import os from "node:os"; import path from "node:path"; import { mkdtemp, rm, writeFile, mkdir } from "node:fs/promises"; import { execa } from "execa"; import type { AriadneConfig, RepositorySnapshot, RepositoryTrace } from "../src/types/index.js"; const directories: string[] = []; export async function tempDir(prefix = "ariadne-test-"): Promise { const directory = await mkdtemp(path.join(os.tmpdir(), prefix)); directories.push(directory); return directory; } export async function cleanupTempDirs(): Promise { await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))); } export async function initGit(cwd: string, files: Record = { "README.md": "initial\n" }): Promise { await execa("git", ["init", "--quiet"], { cwd }); for (const [file, contents] of Object.entries(files)) { await mkdir(path.dirname(path.join(cwd, file)), { recursive: true }); await writeFile(path.join(cwd, file), contents); } await execa("git", ["add", "."], { cwd }); await execa("git", ["-c", "user.name=Ariadne Test", "-c", "user.email=test@example.test", "commit", "--quiet", "-m", "initial"], { cwd }); } export function config(overrides: Partial = {}): AriadneConfig { return { version: 4, sourceVersion: 4, agent: { command: { kind: "exec", file: "node", args: ["-e", "process.stdin.resume()"] }, timeout_ms: 1_000 }, tasks: { directory: ".ariadne/tasks" }, verification: { commands: [], timeout_ms: 1_000 }, execution: { termination_grace_ms: 100, concurrency: 1, failure_mode: "continue", isolation: "shared", worktree: { retention: "on-failure", preparation: { commands: [], timeout_ms: 600_000 } } }, checks: { forbidden_files: [], forbidden_commands: [], ...overrides } }; } export function snapshot(entries: RepositorySnapshot["entries"] = []): RepositorySnapshot { return { available: true, head: "abc", branch: "main", detached: false, dirty: entries.length > 0, entries, diffLineCount: 0 }; } export function trace(overrides: Partial = {}): RepositoryTrace { const clean = snapshot(); return { baseline: clean, postPreparation: clean, postAgent: clean, final: clean, preexistingChanges: [], preparationChanges: [], agentChanges: [], verificationChanges: [], taskChanges: [], forbiddenFileChanges: [], diffLineCount: 0, observedCommands: [], ...overrides }; } export async function writeProject(cwd: string, options: { agentArgs?: string[]; tasks?: Array<{ id: string; prompt?: string }>; checks?: string } = {}): Promise { await mkdir(path.join(cwd, ".ariadne", "tasks"), { recursive: true }); const args = options.agentArgs ?? ["-e", "process.stdin.resume()"]; await writeFile(path.join(cwd, "ariadne.yml"), [ "version: 4", "agent:", " command:", " kind: exec", " file: node", ` args: ${JSON.stringify(args)}`, " timeout_ms: 1000", "tasks:", " directory: .ariadne/tasks", "verification:", " commands: []", " timeout_ms: 1000", "execution:", " termination_grace_ms: 100", " concurrency: 1", " failure_mode: continue", " isolation: shared", " worktree:", " retention: on-failure", " preparation:", " commands: []", " timeout_ms: 600000", "checks:", options.checks ?? " forbidden_files: []\n forbidden_commands: []" ].join("\n")); for (const task of options.tasks ?? [{ id: "example" }]) { await writeFile(path.join(cwd, ".ariadne", "tasks", `${task.id}.yml`), `id: ${task.id}\nname: ${task.id}\nprompt: ${task.prompt ?? "Do work."}\n`); } } ``` ### tests/workflow-runtime.test.ts ```typescript import { describe, expect, it } from "vitest"; import { WorkflowRuntimeChannel, type WorkflowRuntimeEvent } from "../src/core/workflow-runtime.js"; const tick = () => new Promise((resolve) => setImmediate(resolve)); function output(channel: WorkflowRuntimeChannel, chunk: string) { channel.emit({ type: "process.output", taskId: "task", attempt: 1, runId: "run", phase: "agent", commandIndex: 0, stream: "stdout", chunk }); } describe("workflow runtime channel", () => { it("assigns ordered runtime and per-stream sequences and replays bounded early history", async () => { const channel = new WorkflowRuntimeChannel("batch"); channel.emit({ type: "batch.started", startedAt: "2026-07-17T00:00:00.000Z", planId: "plan" }); channel.emit({ type: "process.started", taskId: "task", attempt: 1, runId: "run", phase: "agent", commandIndex: 0, displayCommand: "agent" }); output(channel, "one\n"); output(channel, "two\n"); channel.emit({ type: "process.completed", taskId: "task", attempt: 1, runId: "run", phase: "agent", commandIndex: 0, status: "passed", exitCode: 0, timedOut: false }); channel.emit({ type: "batch.completed", status: "succeeded", outcome: "passed", manifest: ".ariadne/batches/batch/batch.json" }); const received: WorkflowRuntimeEvent[] = []; channel.subscribe((event) => { received.push(event); }); await tick(); expect(received.map((event) => event.sequence)).toEqual([...received.map((event) => event.sequence)].sort((a, b) => a - b)); expect(received.filter((event) => event.type === "process.output").map((event) => event.type === "process.output" ? event.streamSequence : 0)).toEqual([1, 2]); expect(received[0]).toMatchObject({ batchId: "batch", type: "batch.started", sequence: 1 }); expect(received.at(-1)).toMatchObject({ type: "batch.completed" }); expect(channel.latestSnapshot()).toMatchObject({ state: "completed", lastSequence: 6 }); expect(Object.isFrozen(received[0])).toBe(true); }); it("does not block emitters, drops old output for a slow subscriber, and preserves terminal events", async () => { const channel = new WorkflowRuntimeChannel("batch"); let release!: () => void; const gate = new Promise((resolve) => { release = resolve; }); const received: WorkflowRuntimeEvent[] = []; let first = true; channel.subscribe(async (event) => { received.push(event); if (first) { first = false; await gate; } }); channel.emit({ type: "batch.started", startedAt: "2026-07-17T00:00:00.000Z", planId: "plan" }); await tick(); for (let index = 0; index < 600; index += 1) output(channel, `${index}:${"x".repeat(4096)}\n`); channel.emit({ type: "batch.cancellation_requested" }); channel.emit({ type: "batch.cancellation_progress", stage: "batch-finalizing" }); channel.emit({ type: "batch.completed", status: "interrupted", outcome: "interrupted", manifest: "batch.json" }); await tick(); release(); for (let index = 0; index < 20 && !received.some((event) => event.type === "batch.completed"); index += 1) await tick(); expect(received.filter((event) => event.type === "process.output").length).toBeLessThan(600); expect(received.some((event) => event.type === "runtime.warning" && event.category === "subscriber-overflow")).toBe(true); expect(received.some((event) => event.type === "batch.cancellation_requested")).toBe(true); expect(received.at(-1)).toMatchObject({ type: "batch.completed" }); }); it("disconnects a failing subscriber without affecting other subscribers", async () => { const channel = new WorkflowRuntimeChannel("batch"); let failures = 0; const received: WorkflowRuntimeEvent[] = []; channel.subscribe(() => { failures += 1; throw new Error("subscriber failed"); }); channel.subscribe((event) => { received.push(event); }); channel.emit({ type: "task.ready", taskId: "task" }); await tick(); channel.emit({ type: "task.started", taskId: "task", attempt: 1 }); await tick(); expect(failures).toBe(1); expect(received.at(-1)).toMatchObject({ type: "task.started" }); }); }); ``` [130 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]