# Project export: AgentOps Studio This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: OpenAI Build Week - Tagline: Turn failed AI-agent traces into cited diagnoses, regression tests, and safe replays. - Devpost: https://devpost.com/software/agentops-studio-xzlhtn - GitHub: https://github.com/afuyo/AgentOpsStudio - Video: https://www.youtube.com/embed/mycAuFNnPYY?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 1 GitHub contributor(s) — afuyo (1 commits) ## Devpost submission (written by the team) ### Inspiration AI teams have strong tools for collecting traces, prompts, latency, and token usage, but a failed agent run still leaves a developer manually searching events and guessing at the cause. We wanted to move beyond “here is the trace” to “here is the failure, the evidence, the fix, and a way to verify it.” AgentOps Studio explores what debugging looks like when GPT-5.6 is the reasoning engine while deterministic software remains responsible for verification. ### What it does AgentOps Studio turns a failed AI-agent execution trace into a complete debugging workflow. In our synthetic Build Week scenario, a customer-support refund agent finds a customer ID, sends customer_name to a tool that requires customer_id, retries the unchanged invalid call, and fails. GPT-5.6 analyzes the normalized trace and returns a structured diagnosis with: a root cause and contributing factors; observed facts separated from inferences; evidence strength and missing telemetry; a recommended developer-controlled action; and citations to exact trace-span IDs. A developer can jump from each claim to its supporting evidence, generate and review a declarative regression evaluation, and run that evaluation deterministically against the failed run. The developer then explicitly selects a prepared agent version and replays the scenario in a mocked sandbox. The same reviewed evaluation passes, and the UI calculates the before-and-after changes in latency, cost, tokens, spans, and tool calls. When telemetry is incomplete, AgentOps Studio returns an explicit insufficient-evidence result rather than inventing a root cause. If live model access is unavailable, the demonstration uses an honestly labeled, schema-validated cached result previously generated by the same GPT-5.6 workflow. ### How we built it AgentOps Studio is a single Next.js 16 App Router application written in strict TypeScript and React 19. Zod defines bounded contracts for normalized traces, diagnoses, and declarative evaluations. Synthetic fixtures provide deterministic broken, retry, fixed, and insufficient-evidence runs. The OpenAI TypeScript SDK connects to GPT-5.6 through a LiteLLM OpenAI-compatible Responses API endpoint with strict structured output. The application validates every cited span and rejects evaluation predicates outside a closed vocabulary. GPT-5.6 interprets evidence and drafts tests; TypeScript code owns pass/fail and every comparison metric. Reviewed evaluations, normalized runs, replay lineage, and results are persisted in DuckDB. Replay is deliberately constrained to one synthetic refund-agent workflow, with mocked tools that can record intent but cannot contact refund, email, payment, or customer systems. Vitest covers schemas, citation integrity, deterministic evaluation, replay safety, sanitization, configuration, and persistence. Playwright exercises the complete credential-free golden path. A standalone Debian-based Docker image provides a reproducible deployment target. How ChatGPT and Codex accelerated the build Before implementation, I used ChatGPT as a product-thinking partner to compare several possible Build Week directions: a broad agent platform, observability dashboards, evaluation tooling, replay, governance, and knowledge graphs. That brainstorming exposed the stronger product wedge: do not merely display a trace; explain the failure from evidence and prove whether a fix works. This narrowed the idea into the evidence-to-proof workflow implemented in AgentOps Studio. I then used two Codex threads with distinct responsibilities: Core developer thread (019f6216-1495-7071-a43f-8c1753bd8fa6): planning, architecture, implementation, tests, persistence, safety, and release work. Reviewer thread (019f6c2c-cfdb-7242-a672-e64a4a5da5e1): independent review of the plans and source code, evidence and claim checking, risk discovery, and demo critique. The two threads communicated asynchronously through repository-local Markdown. The core thread wrote the brainstorm, product plan, backlog, ADRs, typed contracts, implementation notes, and verification evidence. The reviewer read those artifacts alongside the code and returned structured critiques in review.md, review_v2.md, and review-demo-story-line.md. The core thread then reconciled the findings into TASKS.md, the implementation, tests, and final demo. Markdown acted as a shared protocol and left an auditable path from idea, through code review, to revision. Across that loop, Codex helped: turn the initial product thesis into a bounded vertical slice and dependency-aware backlog; document eight architecture decisions before implementation; scaffold the strict Next.js, TypeScript, ESLint, Prettier, Vitest, and Playwright environment; implement the trace schema, GPT-5.6 diagnosis boundary, closed evaluation DSL, deterministic evaluator, sandboxed replay, DuckDB adapter, and product UI; catch unsafe partial LiteLLM configuration and replace it with atomic key-and-endpoint selection; add trace redaction, truncation, cached fallback categories, spend controls, reset protection, and container packaging; and run iterative lint, type, unit, production-build, live-model, browser, and container verification loops. The reviewer loop drove a key product boundary: GPT may interpret and propose, but deterministic application code must verify. The detailed collaboration record is preserved in the repository’s Codex build log. ### Challenges we ran into The hardest challenge was preventing a convincing explanation from being mistaken for a correct one. We separated observed facts from inferences, required facts to cite real span IDs, and reject model output containing unknown citations. Missing telemetry became a first-class insufficient-evidence state. A second challenge was deciding where AI should stop. GPT-5.6 is excellent at interpreting a trace and proposing a regression test, but it should not decide whether its own test passes. We created a small declarative evaluation language and a deterministic evaluator. Finally, safe replay can easily become a generic orchestration platform. For Build Week we deliberately constrained replay to one synthetic workflow with mocked side effects. That made the end-to-end result testable, reproducible, and honest. ### Accomplishments we're proud of Diagnosis claims focus the exact supporting spans in the trace. Unsupported citations and evaluation predicates are rejected at the application boundary. Missing telemetry produces insufficient_evidence instead of a guessed root cause. One reviewed evaluation fails the broken run and passes the corrected sandbox replay. Before-and-after metrics are calculated from stored run data rather than supplied by the model. The complete workflow is deterministic, credential-optional, and demonstrated in 2 minutes 10 seconds. The trace, cited diagnosis, reviewed evaluation, replay lineage, and verified result become reusable run-level debugging knowledge. ### What we learned AI-native developer tools need stronger boundaries, not fewer. Structured output is only the start: model claims must be checked against domain evidence, and deterministic code should retain authority over verification and metrics. We also learned that a narrow, polished workflow communicates more value than a broad observability dashboard. Treating fixtures as stable product contracts let us iterate quickly without losing trust in the demo. ### What's next Today, knowledge is scoped to persisted runs and their replay lineage. Next, we want to add a bounded OpenTelemetry ingestion adapter and a cross-run context graph connecting runs, agent versions, prompts, tools, evaluations, and verified fixes. That would let teams ask: Have we seen this failure before? Which change caused it? Which fix actually held? After that: instrumentation adapters, prompt and configuration diffs, saved-evaluation CI checks, and additional agent-specific replay vocabularies. Generic replay and production integrations will come only after their safety boundaries are proven. Repository and testing Source code, MIT license, setup instructions, synthetic sample data, supported-platform notes, and a credential-free judge path are available at: https://github.com/afuyo/AgentOpsStudio ## README (from the GitHub repository) # AgentOps Studio > From opaque agent failure to evidence-linked diagnosis and executable proof. AgentOps Studio turns a failed AI-agent trace into an evidence-linked diagnosis, a reviewed deterministic regression evaluation, and a safe replay comparison. The hackathon build focuses on one synthetic refund-agent workflow and uses GPT-5.6 through a LiteLLM endpoint only for interpretation and evaluation drafting. Application code owns validation, pass/fail, replay, and metrics. - **Hackathon category:** Developer Tools - **Repository:** https://github.com/afuyo/AgentOpsStudio - **License:** [MIT](LICENSE) ## Judge Quick Start — No Credentials Required The complete cached demo, deterministic evaluation, sandboxed replay, comparison, and browser test run without LiteLLM or OpenAI credentials. ```bash pnpm install cp .env.example .env pnpm demo:reset pnpm dev ``` Open `http://localhost:3000`. The example environment defaults to `DEMO_MODE=true`. Do not provide judges with private LiteLLM credentials. ## Architecture - `app/` — Next.js UI and server API routes. - `src/domain/` — bounded trace contract, evaluation engine, replay, and comparison. - `src/server/` — LiteLLM configuration, demo guard, and DuckDB adapter. - `fixtures/` — synthetic canonical traces and cached diagnoses. - `e2e/` — credential-free Playwright golden path. - `docs/adr/` — architectural decisions and constraints. DuckDB stores normalized runs/spans, reviewed evaluations, and replay lineage. A single application replica owns one database file; this design is intentionally not a multi-writer production architecture. ## Run Locally Requirements: Node.js 22+, pnpm 10.30+, and Linux/macOS with a supported DuckDB native binary. ```bash pnpm install cp .env.example .env pnpm demo:reset pnpm dev ``` For optional live analysis, set `DEMO_MODE=false` and choose one provider: - **LiteLLM (preferred):** configure `LITELLM_API_KEY`, `LITELLM_BASE_URL`, and `LITELLM_MODEL=gpt-5.6-sol`. - **Direct OpenAI:** leave every `LITELLM_*` value blank and configure: ```dotenv OPENAI_API_KEY=sk-your-key-here OPENAI_BASE_URL=https://api.openai.com/v1 OPENAI_MODEL=gpt-5.6-sol OPENAI_TIMEOUT_MS=60000 ``` Never commit credentials. Live credentials are required only for **Run live analysis**, `pnpm model:check`, and `pnpm quality:ai`. Next.js loads `.env` for the application; export the same values into your shell before running the standalone model-check and AI-quality scripts. ## Sample Data All fixtures are synthetic: | Run | Purpose | | ------------------------------- | -------------------------------------------------------- | | `run_refund_broken_v1` | Wrong identifier and unchanged retry | | `run_refund_retry_v1` | Repeated lookup without changed input | | `run_refund_fixed_v2` | Corrected expected behavior | | `run_refund_unknown_failure_v1` | Insufficient telemetry; diagnosis must decline certainty | Fixture IDs are stable test interfaces. ## Supported Platforms - Node.js 22+ with pnpm 10.30+ - Linux and macOS with a supported DuckDB native binary - Docker on Linux, macOS, or Windows - Chromium for the automated Playwright judge path Native Windows development has not been verified; use Docker or WSL. ## Verify ```bash pnpm lint && pnpm typecheck && pnpm test && pnpm build pnpm test:e2e:install # once pnpm test:e2e ``` `pnpm model:check` and `pnpm quality:ai` make paid/live calls and require LiteLLM credentials. Unit and browser tests are deterministic and credential-free. ## Three-Minute Judge Path 1. Open `run_refund_broken_v1`; inspect the cached evidence-linked diagnosis. 2. Generate the evaluation draft, review its closed JSON contract, and approve it. 3. Run the evaluation and observe the expected failure with cited spans. 4. Select `refund-agent-v2-resolve-customer-id` and run the sandboxed replay. 5. Confirm no side effects, a passing evaluation, and the calculated Fail → Pass comparison. Reset locally with `pnpm demo:reset` or `POST /api/demo/reset`. In production, the reset route requires `x-demo-reset-token` matching `DEMO_RESET_TOKEN`. `GET /api/health` reports storage and cached/live mode without exposing secrets. Stop any running Next.js development server before `pnpm test:e2e`; Playwright starts an isolated server on port 3100. ## Container Deployment ```bash docker build -t agentops-studio . docker volume create agentops-data docker run --rm -p 3000:3000 -v agentops-data:/data agentops-studio ``` Deploy one replica to a container host with a durable `/data` volume. Configure LiteLLM variables only if live analysis is enabled; cached mode survives gateway/model unavailability. ## Safety and Limitations Trace input is validated, bounded, and redacted before persistence, rendering, or prompting. Refund and email tools only record sandboxed intent. Data is synthetic. The MVP supports one fixture-backed workflow, one deterministic template-backed prepared-version replay, one process-local live-call quota, and a single DuckDB writer. It does not claim generic replay, autonomous repair, or production-scale ingestion. ## GPT-5.6 and Codex GPT-5.6 performs structured evidence interpretation and drafts a closed regression contract through LiteLLM. The application validates structured citations and owns evaluation results and metrics. The live three-case quality suite passed with cached fallback disabled, including the case where missing telemetry must produce `insufficient_evidence`. Codex converted the initial brainstorm into the plan, backlog, design contract, and architecture decisions; implemented the trace, diagnosis, evaluation, replay, persistence, safety, and release layers; and ran the unit, live-model, browser, and container verification loops. See [docs/codex-build-log.md](docs/codex-build-log.md) for concrete examples and decision history. The product direction began as a ChatGPT brainstorming session. A core-development Codex thread and a separate reviewer thread then communicated asynchronously through Markdown plans, ADRs, implementation summaries, and adversarial reviews. Their roles, Thread IDs, and artifact handoff are documented in [SUBMISSION.md](SUBMISSION.md#chatgpt-brainstorming-and-multi-thread-collaboration). ## Post-Hackathon Roadmap The next ingestion boundary is OpenTelemetry: ```text Agent or observability platform → OpenTelemetry → AgentOps validation → DuckDB ``` A provider-neutral adapter would validate, redact, and project OTel spans into the existing bounded trace contract. The first increment will accept a deterministic OTel JSON export; a live OTLP endpoint and collector integration will follow. ## Submission Ready-to-paste project copy, judge evidence, and the remaining external fields are collected in [SUBMISSION.md](SUBMISSION.md). See [docs/codex-build-log.md](docs/codex-build-log.md) for where Codex accelerated delivery and where GPT-5.6 is used. ## Detected evidence (automated analysis) Indexed codebase: 71 recognized source files, 530 KB. - CSS (language) — detected in the code - HTML (language) — detected in the code - Next.js (technology) — detected in the code - OpenAI (technology) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - Docker (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (87 of 87) ``` .dockerignore .env.example .gitignore .prettierignore .prettierrc.json app/api/demo/reset/route.ts app/api/evaluations/review/route.ts app/api/health/route.ts app/api/replays/route.ts app/api/runs/[runId]/diagnosis/route.ts app/globals.css app/layout.tsx app/page.tsx app/runs/[runId]/evaluation-replay-panel.tsx app/runs/[runId]/page.tsx app/runs/[runId]/run-workbench.tsx brainstorm.md codex_summary.md demo-story-line.md DESIGN.md Dockerfile docs/adr/0001-single-nextjs-application.md docs/adr/0002-bounded-normalized-trace-contract.md docs/adr/0003-evidence-grounded-ai-diagnosis.md docs/adr/0004-declarative-regression-evaluations.md docs/adr/0005-sandboxed-fixture-backed-replay.md docs/adr/0006-use-duckdb-after-product-gate.md docs/adr/0007-route-model-traffic-through-litellm.md docs/adr/0008-persist-reviewed-workflow-in-duckdb.md docs/adr/README.md docs/codex-build-log.md docs/demo-video-plan.md docs/devpost-submission.md docs/golden-path.md docs/litellm.md docs/README.md e2e/golden-path.spec.ts eslint.config.mjs fixtures/cached-diagnoses.json fixtures/expected-diagnoses.json fixtures/refund-broken.json fixtures/refund-fixed.json fixtures/refund-retry.json fixtures/refund-unknown.json hackathon.html LICENSE next-env.d.ts next.config.ts package.json planv2.md playwright.config.ts README.md review_v2.md review-demo-story-line.md review.md scripts/check-model.ts scripts/record-demo-video.ts scripts/render-demo-video.ps1 scripts/reset-demo.ts scripts/run-ai-quality.ts src/ai/diagnose-trace.ts src/ai/prompt.ts src/data/fixture-repository.test.ts src/data/fixture-repository.ts src/domain/diagnosis.test.ts src/domain/diagnosis.ts src/domain/evaluation.test.ts src/domain/evaluation.ts src/domain/refund-replay.test.ts src/domain/refund-replay.ts src/domain/run-comparison.ts src/domain/trace-view.test.ts src/domain/trace-view.ts src/domain/trace.test.ts src/domain/trace.ts src/security/sanitize-trace.test.ts src/security/sanitize-trace.ts src/server/demo-guard.test.ts src/server/demo-guard.ts src/server/duckdb-store.test.ts src/server/duckdb-store.ts src/server/litellm-config.test.ts src/server/litellm-config.ts SUBMISSION.md TASKS.md tsconfig.json vitest.config.ts ``` ### Dependencies - package.json: @duckdb/node-api@1.5.4-r.1, @playwright/test@^1.61.1, @types/node@^26.1.1, @types/react@^19.2.17, @types/react-dom@^19.2.3, @vitest/coverage-v8@^4.1.10, eslint@^9.39.5, eslint-config-next@^16.2.10, next@^16.2.10, openai@^6.47.0, prettier@^3.9.5, react@^19.2.7, react-dom@^19.2.7, tsx@^4.20.6, typescript@^6.0.3, vitest@^4.1.10, zod@^4.4.3 ### Recent commits (newest first) - hackathon submission ## Key source files (fetched from GitHub, selected and truncated for size) ### review-demo-story-line.md ```markdown # Review: AgentOps Studio Demo Story Line **Verdict:** Strong, implementation-aligned, and judge-friendly. Make the following edits before recording. ## 1. Resolve the cached/live contradiction The diagnosis voiceover implies that GPT-5.6 is analyzing the trace live, while the recording notes prescribe cached output for reliability. Use this voiceover: > For reliability, this recording uses a cached, schema-validated diagnosis previously generated by GPT-5.6 through LiteLLM. Each observed fact links to an exact trace span. ## 2. Make live-model proof mandatory Include a brief inset showing a successful live response or the live source badge. Otherwise, judges may interpret **Cached demo** as simulated AI rather than a reliable fallback for previously generated model output. ## 3. Narrow the evidence claim Replace “every factual claim links back to an actual span” with “each structured observed fact links to an exact trace span.” The diagnosis schema attaches citations to facts, inferences, and contributing factors. The root-cause and summary strings are not independently cited, so the narrower wording is more accurate. ## 4. Tighten the replay wording The replay exercises the refund mock but does not invoke the available email mock. Use this voiceover: > Side-effecting tools are sandboxed mocks that can only record intent; they cannot reach refund or email systems. ## 5. Leave timing buffer The current script ends at exactly three minutes, leaving no room for click latency, transitions, or narration variance. Target approximately 2:40: - **0:00–0:12 — Hook** - **0:12–0:42 — Diagnosis** - **0:42–1:18 — Evaluation** - **1:18–2:00 — Replay** - **2:00–2:23 — Comparison** - **2:23–2:40 — Close** ## 6. Simplify the closing overlay The five-item closing overlay is too dense for a twenty-second close. Keep the three points most relevant to judging: - GPT-5.6 through LiteLLM for evidence-grounded interpretation - Deterministic application-owned evaluation and metrics - Codex for planning, implementation, review, and verification Move DuckDB and Playwright details to the written submission or repository documentation. Make the Codex claim concrete in the voiceover: > Codex helped turn the initial brainstorm into architectural decisions, typed contracts, the deterministic replay harness, and end-to-end verification. ## 7. Clarify the E2E rehearsal command The named workflow matches the implementation and Playwright golden-path test. However, `pnpm test:e2e` starts its own Next.js server and will fail if another development server already holds the build lock. Add this recording note: > Stop any running Next.js development server before running `pnpm test:e2e`. ## Overall assessment The core evidence-to-proof loop is clear and credible: ```text Failed agent → cited GPT-5.6 diagnosis → reviewed regression contract → deterministic failure → sandboxed prepared-fix replay → deterministic pass and comparison ``` The Sarah [truncated — 242 more characters] ``` ### DESIGN.md ```markdown # Design ## Source of truth - Status: Active - Last refreshed: 2026-07-15 - Primary product surfaces: recent problems and the diagnosis, evaluation, and replay workbench - Evidence reviewed: `planv2.md`, `TASKS.md`, `docs/golden-path.md`, ADR-0002/0003, existing `app/` UI and fixtures ## Brand - Personality: precise, calm, technical, trustworthy - Trust signals: evidence citations, observed-versus-inferred labels, deterministic metrics, explicit AI source state - Avoid: generic analytics dashboards, neon overload, unexplained scores, fake terminal aesthetics, hidden AI fallbacks ## Product goals - Goals: make a failed run understandable in under one minute; connect each factual claim to a span; make uncertainty visible - Non-goals: broad monitoring, charts, team navigation, generic replay, or autonomous code repair - Success signals: the failed refund run is the obvious first action, every diagnosis fact reaches evidence in one interaction, and one reviewed contract proves the prepared replay fixes the regression ## Personas and jobs - Primary personas: AI application developers and platform engineers - User jobs: triage a failed run, verify the explanation, identify the next fix - Key contexts: focused desktop debugging and a three-minute judge demo; mobile remains readable ## Information architecture - Primary navigation: product identity → Recent Problems → Run Workbench - Core routes/screens: `/` and `/runs/[runId]` - Content hierarchy: diagnosis → reviewed regression contract → prepared replay and comparison → complete trace ## Design principles - Answers before telemetry: lead with the explanation, then expose its evidence. - Earn trust visibly: distinguish live/cached/error states and facts/inferences. - Progressive disclosure: show a scan-friendly trace; reveal payload details on selection. - Tradeoff: optimize the golden workflow rather than general dashboard density. ## Visual language - Color: deep navy surfaces, cool slate text, mint success, amber uncertainty, coral failure - Typography: system sans for interface; system monospace for IDs, metrics, and payloads - Spacing/layout rhythm: 4/8px scale, wide desktop canvas, deliberate section gaps - Shape/radius/elevation: restrained 8–16px radii, border-led hierarchy, minimal shadow - Motion: brief focus/selection transitions only; respect reduced motion - Imagery/iconography: no imagery; compact semantic symbols with text labels ## Components - Existing components to reuse: root layout and CSS variables - New/changed components: app header, problem card, diagnosis panel, evaluation editor, assertion result, replay selector, run comparison, evidence citation, trace row, evidence drawer - Variants and states: error/warning/success/unknown spans; live/cached/loading/error diagnosis; draft/reviewed/failed/passed/replayed workflow - Token/component ownership: `app/globals.css` owns tokens; components consume semantic class names ## Accessibility - Target standard: WCAG 2.2 [truncated — 2127 more characters] ``` ### Dockerfile ``` FROM node:22-bookworm-slim AS dependencies WORKDIR /app RUN corepack enable COPY package.json pnpm-lock.yaml ./ RUN pnpm install --frozen-lockfile FROM node:22-bookworm-slim AS builder WORKDIR /app RUN corepack enable COPY --from=dependencies /app/node_modules ./node_modules COPY . . RUN pnpm build \ && cp "$(find node_modules -name libduckdb.so -print -quit)" .next/standalone/libduckdb.so FROM node:22-bookworm-slim AS runner WORKDIR /app ENV NODE_ENV=production \ HOSTNAME=0.0.0.0 \ PORT=3000 \ DEMO_MODE=true \ DUCKDB_PATH=/data/agentops.duckdb \ LD_LIBRARY_PATH=/app RUN groupadd --system --gid 1001 nodejs \ && useradd --system --uid 1001 --gid nodejs nextjs \ && mkdir -p /data \ && chown nextjs:nodejs /data COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./ COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static USER nextjs EXPOSE 3000 VOLUME ["/data"] HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \ CMD ["node", "-e", "fetch('http://127.0.0.1:3000/api/health').then(r=>{if(!r.ok)process.exit(1)}).catch(()=>process.exit(1))"] CMD ["node", "server.js"] ``` ### package.json ``` { "name": "agentops-studio", "version": "0.1.0", "private": true, "packageManager": "pnpm@10.30.1", "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint .", "typecheck": "tsc --noEmit", "format": "prettier --write app src scripts fixtures e2e package.json tsconfig.json next.config.ts eslint.config.mjs vitest.config.ts playwright.config.ts", "format:check": "prettier --check app src scripts fixtures e2e package.json tsconfig.json next.config.ts eslint.config.mjs vitest.config.ts playwright.config.ts", "test": "vitest run", "test:watch": "vitest", "test:e2e": "playwright test", "test:e2e:install": "playwright install chromium", "demo:reset": "tsx scripts/reset-demo.ts", "model:check": "tsx scripts/check-model.ts", "quality:ai": "tsx scripts/run-ai-quality.ts" }, "dependencies": { "@duckdb/node-api": "1.5.4-r.1", "next": "^16.2.10", "openai": "^6.47.0", "react": "^19.2.7", "react-dom": "^19.2.7", "zod": "^4.4.3" }, "devDependencies": { "@playwright/test": "^1.61.1", "@types/node": "^26.1.1", "@types/react": "^19.2.17", "@types/react-dom": "^19.2.3", "@vitest/coverage-v8": "^4.1.10", "eslint": "^9.39.5", "eslint-config-next": "^16.2.10", "prettier": "^3.9.5", "tsx": "^4.20.6", "typescript": "^6.0.3", "vitest": "^4.1.10" }, "pnpm": { "onlyBuiltDependencies": [ "esbuild", "sharp", "unrs-resolver" ] } } ``` ### app/layout.tsx ```typescript import type { Metadata } from "next"; import Link from "next/link"; import type { ReactNode } from "react"; import "./globals.css"; export const metadata: Metadata = { title: "AgentOps Studio", description: "Evidence-grounded debugging for AI agent traces.", }; export default function RootLayout({ children, }: Readonly<{ children: ReactNode }>) { return (
AgentOps Studio Evidence-grounded agent debugging Resettable demo · M4
{children} ); } ``` ### app/page.tsx ```typescript import Link from "next/link"; import { listProblemRuns } from "@/src/data/fixture-repository"; import { formatCost, formatDuration } from "@/src/domain/trace-view"; export default function Home() { const problems = listProblemRuns(); return (

Recent problems

Start with what failed.

AgentOps turns execution traces into explanations you can verify. Open a run to move from root cause to the exact supporting span.

{problems.length} runs need attention

Failure queue

Runs requiring investigation

Synthetic demo data · newest first

{problems.map(({ trace, title, summary, metrics }, index) => (
Failed {trace.agent_id} {trace.agent_version}

{title}

{summary}

Duration
{formatDuration(metrics.durationMs)}
Cost
{formatCost(metrics.costUsd)}
Errors
{metrics.errorCount}
Open diagnosis ))}
); } ``` ### app/api/health/route.ts ```typescript import { NextResponse } from "next/server"; import { diagnosisMode } from "@/src/server/demo-guard"; import { getDuckDBStore } from "@/src/server/duckdb-store"; export const runtime = "nodejs"; export async function GET() { try { const store = await getDuckDBStore(); const counts = await store.counts(); return NextResponse.json({ status: "ok", demoMode: process.env.DEMO_MODE === "true", diagnosisMode: diagnosisMode(), storage: "duckdb", seededRuns: counts.runs, }); } catch { return NextResponse.json( { status: "unavailable", storage: "duckdb" }, { status: 503 }, ); } } ``` ### app/runs/[runId]/page.tsx ```typescript import Link from "next/link"; import { notFound } from "next/navigation"; import { RunWorkbench } from "@/app/runs/[runId]/run-workbench"; import { findCachedDiagnosis, findTraceByRunId, listRunIds, } from "@/src/data/fixture-repository"; type RunPageProps = { params: Promise<{ runId: string }>; }; export function generateStaticParams() { return listRunIds().map((runId) => ({ runId })); } export default async function RunPage({ params }: RunPageProps) { const { runId } = await params; const trace = findTraceByRunId(runId); if (!trace) { notFound(); } return (
); } ``` ### app/api/replays/route.ts ```typescript import { NextResponse } from "next/server"; import { z } from "zod"; import { PREPARED_REFUND_AGENT_VERSION, replayRefundAgent, } from "@/src/domain/refund-replay"; import { compareRuns } from "@/src/domain/run-comparison"; import { getDuckDBStore } from "@/src/server/duckdb-store"; export const runtime = "nodejs"; const requestSchema = z .object({ sourceRunId: z.string().min(1), evaluationId: z.string().min(1), selectedAgentVersion: z.literal(PREPARED_REFUND_AGENT_VERSION), }) .strict(); export async function POST(request: Request) { try { const input = requestSchema.parse(await request.json()); const store = await getDuckDBStore(); const sourceTrace = await store.findTrace(input.sourceRunId); if (!sourceTrace) { return NextResponse.json({ error: "Run not found" }, { status: 404 }); } const evaluation = await store.findReviewedEvaluation( sourceTrace, input.evaluationId, ); if (!evaluation) { return NextResponse.json( { error: "Review and approve the evaluation before replay" }, { status: 409 }, ); } const replay = replayRefundAgent({ sourceTrace, selectedAgentVersion: input.selectedAgentVersion, evaluation, }); await store.saveReplay(replay); return NextResponse.json({ replay, comparison: compareRuns(sourceTrace, replay.trace, evaluation), persisted: true, }); } catch (error) { if (!(error instanceof z.ZodError)) { console.warn("Replay request failed", { category: error instanceof Error ? error.name : "UnknownError", }); } return NextResponse.json( { error: error instanceof z.ZodError ? "Replay request is invalid" : "Replay could not be completed", }, { status: error instanceof z.ZodError ? 400 : 500 }, ); } } ``` ### app/api/demo/reset/route.ts ```typescript import { NextResponse } from "next/server"; import { resetDemoGuardState } from "@/src/server/demo-guard"; import { getDuckDBStore } from "@/src/server/duckdb-store"; export const runtime = "nodejs"; function canReset(request: Request): boolean { if (process.env.NODE_ENV !== "production") { return true; } const configured = process.env.DEMO_RESET_TOKEN?.trim(); const supplied = request.headers.get("x-demo-reset-token")?.trim(); return Boolean(configured && supplied === configured); } export async function POST(request: Request) { if (!canReset(request)) { return NextResponse.json( { error: "Demo reset is not authorized" }, { status: 403 }, ); } const store = await getDuckDBStore(); await store.reset(); resetDemoGuardState(); return NextResponse.json({ reset: true, counts: await store.counts() }); } ``` [59 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]