# Project export: OpsPilot — conversational AI copilot for on-call and SRE

This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small.

## Project metadata

- Hackathon: OpenAI Build Week
- Tagline: OpsPilot is an AI SRE copilot that connects Kubernetes logs, metrics, and deploys to find root cause, answer on-call questions, apply human-approved fixes, and draft the postmortem.
- Devpost: https://devpost.com/software/a-conversational-ai-copilot-for-on-call-and-sre
- GitHub: https://github.com/GuruduGanesh/sre-opspilot
- Video: https://www.youtube.com/embed/iAHV63k_yGw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent
- Team: 1 GitHub contributor(s) — GuruduGanesh (10 commits)

## Devpost submission (written by the team)

### Inspiration

As an SRE and DevOps Architect, I have seen on-call engineers lose valuable time during production incidents by switching between five or more tools to understand what went wrong. The evidence is usually available, but it is scattered across logs, metrics, deployment history, alerts, and previous incident reports. Bringing that information together manually slows down triage, increases alert fatigue, and often depends on knowledge held by a few experienced engineers. This experience inspired me to build a tool that brings the relevant signals together and helps teams investigate incidents more quickly and consistently.

### What it does

OpsPilot is a conversational incident-response copilot demonstrated in a controlled Kubernetes test environment. The environment runs sample services and reproduces two realistic incidents: a bad configuration rollout that causes a spike in checkout errors, and a memory leak that results in an OOMKill and crash loop. Although the incidents are intentionally triggered for the demonstration, the Kubernetes events, pod logs, deployment history, and Prometheus metrics are generated by the running environment and investigated in real time. When an alert is raised, OpsPilot begins an automated investigation. It gathers evidence from the cluster, compares the timing of errors with recent deployment activity, and presents a root-cause conclusion with cited evidence and a confidence level. It then recommends an action from an approved list, such as restoring a known-good configuration, restarting a workload, or scaling a workload. OpsPilot never changes the environment without the engineer's approval. Every proposed action must come from an allowlisted set, pass a Kubernetes server-side dry-run preview that shows the exact change, receive explicit human confirmation, and be recorded in an append-only audit trail. After the action is applied, an independent verifier — not the model — confirms that error rates and workload health have returned to their defined baseline. The same investigation tools are available through a conversational interface. An on-call engineer can ask questions such as, "What evidence supports the current root-cause hypothesis?" or "What changed before the incident?" OpsPilot answers using current evidence from the test environment, without requiring the engineer to write PromQL or switch between kubectl commands. When evidence is missing, it says so and states what it needs — it does not invent logs, traces, or impact. Once the incident is resolved, OpsPilot drafts a structured postmortem from the persisted audit trail: summary, timeline, root cause with its evidence, actions taken, verification result, and prevention recommendations — with explicit unknowns rather than invented details.

### How we built it

Codex was my implementation partner throughout the week: it built the FastAPI backend and React/TypeScript console with me, wrote the Kubernetes and Prometheus adapters, implemented the remediation safety contracts, and wrote most of the test suite (59 passing tests). The Kubernetes simulation — a kind cluster with a sample checkout service, Prometheus, and a load generator — was also scaffolded and debugged with Codex. GPT-5.6 is the investigation runtime, called through the OpenAI Responses API with bounded, read-only function tools. It receives typed evidence — workload status, events, log excerpts, metric queries, deployment history — decides what is relevant, and returns a structured, evidence-cited conclusion. A recorded live GPT-5.6 investigation of the checkout incident identified the injected failure mode and cited its supporting evidence. Cluster changes are always executed by application code behind the human approval gate, never by the model. Stack: Python, FastAPI, React, TypeScript, Kubernetes (kind), Prometheus, SQLite, OpenAI Responses API.

### Challenges we ran into

The most interesting challenge was making automation trustworthy. Early on I assumed restarting or scaling would clear the memory-leak scenario — testing proved it would not, so I added a dedicated allowlisted restoration action instead of letting a plausible-but-wrong fix ship. Recovery verification had a similar lesson: a rollout that is still becoming ready is not a failed fix, so the lifecycle had to distinguish "still monitoring" from "verification failed." And keeping every public claim tied to tested evidence required real discipline — the project maintains a claim-verification ledger, and anything not demonstrated is stated as not demonstrated.

### Accomplishments we're proud of

The complete loop works end to end in a local Kubernetes test cluster: alert → automated investigation → evidence-cited root cause → dry-run preview → human-approved fix → independently verified recovery → audit-derived postmortem. Every conclusion links to its evidence, and no change ever happens without explicit approval.

### What we learned

Agents work best with small, deterministic tools instead of unrestricted commands. GPT-5.6 is most effective for reasoning and explanation, while application code handles execution and safety. Sharing the same tools between the automated investigation and the chat interface made the system simpler, more consistent, and easier to trust.

### What's next

for OpsPilot Prometheus Alertmanager integration (the ingress already accepts its webhook format), a hosted demo, integrations with PagerDuty and Datadog, searchable learnings from past postmortems, and multi-cluster support.

## README (from the GitHub repository)

# OpsPilot

> Evidence-first incident response for Kubernetes, with human-approved remediation.

OpsPilot is a conversational SRE copilot for the OpenAI Build Week Developer Tools
track. It brings Kubernetes events, logs, metrics, and deployment history into one
incident investigation, helping on-call engineers understand what changed, assess
impact, and decide on a safe response.

**Demo:** [Watch the 2-minute 48-second walkthrough on YouTube](https://youtu.be/iAHV63k_yGw)

## Current build

The local demonstration includes:

- a FastAPI health endpoint and Alertmanager generic-webhook v4-compatible
  scenario ingress;
- delivery-retry, alert-update, resolved-signal, and reset/new-run idempotency
  behavior backed by SQLite;
- typed, bounded Prometheus and Kubernetes workload-status adapters;
- typed Kubernetes event, redacted log-excerpt, and deployment-history adapters;
- server-owned lifecycle transitions, persisted alert evidence, and an evidence timeline;
- a server-owned action plan with a Kubernetes dry-run preview, evidence binding,
  explicit approval, fingerprint and target-version staleness checks, and an
  append-only audit trail;
- an independent recovery verifier that checks workload readiness, a bounded
  checkout 5xx recovery indicator, and observed post-recovery 2xx traffic for P1;
- an evidence-backed incident command center with current Prometheus rates,
  workload health, Kubernetes events, deployment history, evidence timeline,
  follow-up investigation input, approval controls, and an audit-derived
  postmortem draft; unknown blast radius, SLO, and model confidence are shown
  explicitly rather than inferred;
- a five-second refresh of the controlled dashboard and a bounded evidence
  collection step before every investigation request; unchanged event and
  deployment records are deduplicated while fresh telemetry is retained;
- a real local kind environment with a checkout service, Prometheus, and a load
  generator;
- P1: a controlled checkout rollout that changes live traffic from HTTP 200 to
  HTTP 500, produces Prometheus 5xx telemetry, and recovers after reset;
- local rehearsal controls that create or reopen the matching P1/P2 incident
  from the console. They modify only the dedicated demo checkout workload and
  retain the same preview, approval, audit, recovery, and RCA gates as the
  command-line path; and
- an auto-refreshing local on-call queue backed by persisted incident records.
  The console starts with an explicit selection screen; a reviewer can choose a
  retained local incident, inject P1/P2, or open `?incident=<id>` as a reproducible
  deep link.

The GPT-5.6 investigation route is contract-tested and has been exercised once
against the direct OpenAI API in the controlled P1 scenario. That report cited
persisted alert, Kubernetes, deployment, and Prometheus evidence, and its
conclusion is retained with the incident record. This is evidence for that
controlled P1 run only; it is not a claim that OpsPilot has been validated on
production workloads or arbitrary incidents.

## How Codex and GPT-5.6 are used

Codex was used as the implementation partner for the FastAPI/React scaffold,
Kubernetes simulation controls, remediation safety contracts, test coverage, and
the console. The project records those factual implementation sessions in its
private development log rather than reconstructing them for submission.

GPT-5.6 is the configured user-facing investigation runtime. Its bounded,
read-only evidence/tool contract is implemented and tested. One direct
GPT-5.6 Terra investigation has been recorded for the controlled P1 scenario;
the report cites persisted evidence and is included in the audit-derived RCA.
The product makes no broader accuracy, automation, or production-readiness
claim from that single run.

## Why OpsPilot

During an incident, the important signals are usually available but split across
multiple tools. OpsPilot does not replace Kubernetes, Prometheus, or an on-call
engineer. It brings evidence together into one investigation and keeps every
change behind an explicit approval gate.

The product is designed around four principles:

- **Evidence before conclusions.** Every hypothesis links to the logs, metrics,
  events, or deployment change that supports it.
- **Constrained remediation.** The model cannot run arbitrary commands. It may
  recommend only allowlisted actions with typed inputs and preflight checks.
- **A human stays accountable.** Every allowlisted remediation operation requires
  explicit approval from the engineer.
- **Recovery must be demonstrated.** An action is not marked successful until
  health checks and relevant service indicators return to their defined baseline.

## Scenario design

OpsPilot is organized around two representative Kubernetes incident scenarios.
When a scenario runs, its logs, events, deployment history, and metrics are
generated by the local environment.

| Incident | Trigger | Expected investigation | Approved recovery |
| --- | --- | --- | --- |
| P1: checkout degradation | Controlled response-mode rollout | Link the 5xx increase to the recent deployment revision | Restore the controlled response mode and verify error-rate recovery |
| P2: workload instability | Controlled memory leak | Link restarts and OOMKill events to the affected workload | Restore the controlled memory mode; verify readiness and a stable restart count for 30 seconds |

## Product flow

```mermaid
flowchart LR
    A[Scenario alert payload or engineer question] --> B[Evidence collection]
    B --> C[Evidence graph and hypothesis ledger]
    C --> D[Root-cause explanation]
    D --> E{Approved action?}
    E -->|No| F[Continue investigation]
    E -->|Yes| G[Policy and preflight checks]
    G --> H[Execute allowlisted action]
    H --> I[Verify recovery]
    I --> J[Structured postmortem]
```

## Direct GPT-5.6 validation

- **Completed, controlled P1:** “What evidence supports the current root-cause
  hypothesis?” The persisted report identified the controlled `FAIL_MODE=true`
  setting on checkout revision 158, cited its available evidence, and explicitly
  requested logs or a trace for direct execution-path confirmation.

Broader service ranking and blast-radius answers remain architecture targets, not
current submitted-build claims.

## Supported platforms

- **Windows 11 / PowerShell 7:** verified with Docker Desktop, kind, kubectl,
  Python 3.12, Node 20, npm, and `uv`.
- **macOS and Linux:** expected to work with equivalent Docker, kind, kubectl,
  Python, Node, and PowerShell tooling, but not yet verified.
- The demo targets only the dedicated local `opspilot-dev` kind cluster; it does
  not support external or production Kubernetes clusters.

## Local prerequisites

- Docker Desktop with Kubernetes-compatible containers enabled
- `kind` and `kubectl`
- Python 3.12+ and Node.js 20+
- For the final live validation and demo recording: an OpenAI API key with
  billing and access to the configured GPT-5.6 model

Copy `.env.example` to `.env` and add local configuration there. Credentials are
never committed to the repository.

## Run locally

The commands below are verified on Windows PowerShell with Docker Desktop, kind,
kubectl, Python, Node, npm, and `uv`.

```powershell
uv sync --all-groups
.\scripts\verify.ps1

# Starts the dedicated local kind cluster, checkout service, Prometheus, and load generator.
.\scripts\scenario.ps1 create

# In a separate terminal, start the API and its local Prometheus connection.
.\scripts\run-console.ps1
```

In another PowerShell terminal, exercise P1 and create its controlled scenario
alert:

```powershell
.\scripts\scenario.ps1 inject-p1
.\scripts\send-p1-alert.ps1
.\scripts\scenario.ps1 reset-p1
.\scripts\scenario.ps1 status

# Runs the controlled P1 integration test against kind, then resets the scenario.
.\scripts\test-e2e-p1.ps1

# Runs the complete controlled P1 approval/recovery path against kind. It starts
# a temporary local Promet

[README truncated for size]

## Detected evidence (automated analysis)

Indexed codebase: 52 recognized source files, 396 KB.
- CSS (language) — detected in the code
- FastAPI (technology) — detected in the code
- HTML (language) — detected in the code
- OpenAI (technology) — detected in the code
- Python (language) — detected in the code
- React (technology) — detected in the code
- TypeScript (language) — detected in the code
- Docker (technology) — claimed on Devpost, not found in the code

## Codebase structure (from repository index)

### Files (76 of 76)

```
.env.example
.github/workflows/ci.yml
.gitignore
backend/opspilot/__init__.py
backend/opspilot/adapters/__init__.py
backend/opspilot/adapters/kubernetes.py
backend/opspilot/adapters/prometheus.py
backend/opspilot/api/__init__.py
backend/opspilot/api/main.py
backend/opspilot/dashboard.py
backend/opspilot/domain/__init__.py
backend/opspilot/domain/actions.py
backend/opspilot/domain/alerts.py
backend/opspilot/domain/evidence.py
backend/opspilot/domain/incidents.py
backend/opspilot/domain/investigation.py
backend/opspilot/domain/tools.py
backend/opspilot/evidence_collection.py
backend/opspilot/investigation.py
backend/opspilot/llm_provider.py
backend/opspilot/model_selection.py
backend/opspilot/postmortem.py
backend/opspilot/recovery.py
backend/opspilot/remediation.py
backend/opspilot/settings.py
backend/opspilot/simulation.py
backend/opspilot/storage/__init__.py
backend/opspilot/storage/incidents.py
demo/checkout/app.py
demo/checkout/Dockerfile
demo/checkout/requirements.txt
docs/ARCHITECTURE.md
docs/CLAIM_VERIFICATION.md
docs/DELIVERY_PLAN.md
docs/PROGRESS.md
docs/USER_GUIDE.md
frontend/index.html
frontend/package.json
frontend/src/main.tsx
frontend/src/styles.css
frontend/src/vite-env.d.ts
frontend/tsconfig.json
frontend/tsconfig.node.json
frontend/vite.config.ts
infra/k8s/checkout.yaml
infra/k8s/load-generator.yaml
infra/k8s/namespace.yaml
infra/k8s/prometheus.yaml
infra/kind/config.yaml
JUDGE_PATH.md
LICENSE
pyproject.toml
README.md
scripts/demo-proof.ps1
scripts/eval.ps1
scripts/model-smoke.ps1
scripts/package-judge.ps1
scripts/prepare-demo.ps1
scripts/run-console.ps1
scripts/run-judge.ps1
scripts/scenario.ps1
scripts/send-p1-alert.ps1
scripts/test-e2e-p1-remediation.ps1
scripts/test-e2e-p1.ps1
scripts/test-e2e-p2-remediation.ps1
scripts/test-e2e-p2.ps1
scripts/verify.ps1
tests/conftest.py
tests/e2e_p1_kind.py
tests/e2e_p1_remediation.py
tests/e2e_p2_remediation.py
tests/test_alert_ingress.py
tests/test_investigation.py
tests/test_policy_contracts.py
tests/test_simulation.py
uv.lock
```

### Dependencies

- demo/checkout/requirements.txt: fastapi@==0.139.0, prometheus-client@==0.24.1, uvicorn@==0.51.0
- frontend/package.json: @types/node@latest, @types/react@latest, @types/react-dom@latest, @vitejs/plugin-react@latest, react@latest, react-dom@latest, typescript@latest, vite@latest
- pyproject.toml: fastapi@>=0.128.0, httpx@>=0.28.1, kubernetes@>=36.0.3, openai@>=2.45.0, pydantic@>=2.12.0, pydantic-settings@>=2.10.0, uvicorn[standard]@>=0.40.0

### Recent commits (newest first)

- docs: record green CI validation
- fix: make CI independent of local kubeconfig
- docs: finalize OpsPilot public submission materials
- docs: record prebuilt judge-path validation
- docs: link public OpsPilot demo video
- fix: harden controlled recovery and live telemetry
- feat: finalize OpsPilot evidence-first incident console and release assets
- feat: finalize OpsPilot controlled incident command center
- feat: deliver OpsPilot controlled incident response workflow
- feat: establish OpsPilot foundation with controlled Kubernetes incident simulation

## Key source files (fetched from GitHub, selected and truncated for size)

### JUDGE_PATH.md

```markdown
# Judge path

OpsPilot's local review path uses a prebuilt checkout image so the controlled
Kubernetes scenarios do not need to be rebuilt from source. It does not require
an OpenAI API key: the two scripts verify the evidence, dry-run, approval, and
recovery gates. A GPT-5.6 key is required only to try the separate conversational
investigation endpoint.

## Scope and platform

This reviewer path demonstrates the controlled evidence and safety gates, not a
live GPT-5.6 investigation. A submitted hosted demo or recorded live demo must
show the separately configured investigation route after its API access has been
verified; judges must never be asked to provide their own OpenAI key.

The supplied scripts are verified only on Windows PowerShell 7+. No macOS or
Linux judge path is currently claimed. A hosted demo is the preferred way to make
the final reviewer experience platform-independent.

## Prerequisites

- Windows PowerShell 7+, Docker Desktop, `kind`, `kubectl`, Python 3.12, Node.js,
  and `uv`.
- The `opspilot-checkout-0.1.tar` artifact attached to release `v0.1.0`.

## Run the controlled review

```powershell
uv sync --all-groups
.\scripts\run-judge.ps1 -ImageArchive .\opspilot-checkout-0.1.tar
```

The command imports the supplied checkout image, creates only the dedicated
`opspilot-dev` kind cluster, and runs the P1 and P2 end-to-end safety flows. Each
flow intentionally triggers a failure, proves an unapproved action is denied,
applies an approved, dry-run-previewed allowlisted recovery, and independently
checks recovery. The scripts reset the controlled workload after the run.

## Package the review artifact

Maintainers create the ignored local package with:

```powershell
.\scripts\package-judge.ps1
```

Attach `artifacts/judge-package/opspilot-checkout-0.1.tar` to the GitHub release
used for review. Do not attach API keys, local databases, private plans, or
research archives.

```

### docs/CLAIM_VERIFICATION.md

```markdown
# Claim verification ledger

This ledger prevents a common hackathon failure: describing a planned or
partially working feature as if it were a demonstrated result. A public claim is
allowed only when its evidence is linked here and the corresponding item is marked
verified in [PROGRESS.md](PROGRESS.md).

## Rules

1. **No invented metrics.** Do not state accuracy, MTTR reduction, latency, token
   cost, availability, or any percentage without a saved measurement method, raw
   result, date, and scenario.
2. **No invented features.** Do not say a feature exists because it is in an
   architecture diagram, backlog, prompt, mockup, or screen recording plan.
3. **No simulated results presented as production results.** The project may use
   injected incidents in a local Kubernetes test environment. Describe those
   incidents as controlled demonstrations and identify what was actually measured.
4. **No implied autonomy.** Remediation remains human-approved. Do not describe it
   as autonomous remediation or self-healing.
5. **No unsupported comparison.** Do not claim to outperform tools, save a stated
   amount of time, or be production ready unless an evaluation supports it.
6. **Use uncertainty honestly.** If the evidence is incomplete or a tool fails,
   the product and video must show that limitation rather than present certainty.

## Current public-claim status

| Proposed claim | Status | Required evidence before use in Devpost/video |
| --- | --- | --- |
| The repository contains the project architecture and delivery plan. | Verified | Files and Git history |
| Local prerequisites are available: Docker, kind, kubectl, Python, Node, and Git. | Verified | Recorded version checks in `PROGRESS.md` |
| OpsPilot collects live controlled Kubernetes/Prometheus telemetry. | Verified (controlled) | P1/P2 end-to-end scripts and saved evaluation artifact |
| In one controlled P1 run, OpsPilot obtained an evidence-cited GPT-5.6 conclusion about the simulated checkout failure. | Verified (controlled P1 only) | Persisted direct GPT-5.6 report, ignored local model-selection artifact, and `docs/assets/screenshots/02-p1-incident-overview-wide.png` / `04-recovery-rca.png` |
| OpsPilot proposes a controlled recovery and blocks execution before human approval. | Verified (controlled) | P1/P2 remediation flows and action-policy tests |
| OpsPilot verifies recovery after an approved action. | Verified (controlled) | P1/P2 remediation flows with independent verifier output |
| OpsPilot supports the OOMKill/P2 scenario. | Verified (controlled) | Repeatable P2 remediation run with OOMKill and recovery evidence |
| OpsPilot creates an audit-derived postmortem draft. | Verified (controlled) | Persisted incident audit record and `docs/assets/screenshots/04-recovery-rca.png` |
| OpsPilot reduces MTTR, improves accuracy, or lowers cost. | Not claimable yet | Defined baseline, method, raw measurements, and repeatable result |
| OpsPilot is production ready or works on any c
[truncated — 1011 more characters]
```

### pyproject.toml

```
[project]
name = "opspilot"
version = "0.1.0"
description = "Evidence-first incident response for a controlled Kubernetes environment"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
  "fastapi>=0.128.0",
  "httpx>=0.28.1",
  "kubernetes>=36.0.3",
  "openai>=2.45.0",
  "pydantic>=2.12.0",
  "pydantic-settings>=2.10.0",
  "uvicorn[standard]>=0.40.0",
]

[dependency-groups]
dev = [
  "pytest>=9.0.0",
  "pytest-cov>=7.0.0",
  "ruff>=0.14.0",
  "ty>=0.0.59",
]

[tool.ruff]
line-length = 100
target-version = "py312"

[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B"]

[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["backend"]
addopts = "-ra"
markers = [
  "e2e: requires the dedicated local kind cluster and the local API",
]

```

### frontend/package.json

```
{
  "name": "opspilot-console",
  "private": true,
  "version": "0.1.0",
  "type": "module",
  "scripts": {
    "dev": "vite",
    "build": "tsc -b && vite build",
    "lint": "tsc -b --pretty false"
  },
  "dependencies": {
    "@vitejs/plugin-react": "latest",
    "vite": "latest",
    "react": "latest",
    "react-dom": "latest"
  },
  "devDependencies": {
    "@types/node": "latest",
    "@types/react": "latest",
    "@types/react-dom": "latest",
    "typescript": "latest"
  }
}

```

### demo/checkout/requirements.txt

```
fastapi==0.139.0
prometheus-client==0.24.1
uvicorn==0.51.0

```

### demo/checkout/Dockerfile

```
FROM python:3.13-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .

RUN useradd --create-home --uid 10001 appuser
USER 10001
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

```

### demo/checkout/app.py

```python
import os

from fastapi import FastAPI
from fastapi.responses import JSONResponse, Response
from prometheus_client import CONTENT_TYPE_LATEST, Counter, generate_latest

app = FastAPI(title="OpsPilot checkout demo")
requests = Counter(
    "http_requests_total",
    "HTTP requests served by the controlled checkout service",
    ["service", "method", "route", "status"],
)
_retained_memory: list[bytearray] = []


@app.get("/healthz")
def healthz() -> dict[str, str]:
    return {"status": "ok"}


@app.get("/checkout")
def checkout() -> JSONResponse:
    if os.environ.get("MEMORY_LEAK_MODE", "false").lower() == "true":
        # Controlled P2 trigger: the 128 MiB pod limit turns repeated traffic into an OOMKill.
        _retained_memory.append(bytearray(16 * 1024 * 1024))
    if os.environ.get("FAIL_MODE", "false").lower() == "true":
        requests.labels(service="checkout", method="GET", route="/checkout", status="500").inc()
        return JSONResponse(status_code=500, content={"error": "controlled checkout failure"})

    requests.labels(service="checkout", method="GET", route="/checkout", status="200").inc()
    return JSONResponse(status_code=200, content={"status": "accepted"})


@app.get("/metrics")
def metrics() -> Response:
    return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)

```

### backend/opspilot/api/main.py

```python
import asyncio
import json
import logging
from collections.abc import AsyncIterator
from contextlib import asynccontextmanager
from datetime import datetime

from fastapi import FastAPI, HTTPException, Request, status
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import StreamingResponse
from pydantic import BaseModel, ConfigDict, Field

from opspilot.adapters.kubernetes import KubernetesAdapter
from opspilot.adapters.prometheus import PrometheusAdapter
from opspilot.dashboard import DashboardService, DashboardSnapshot
from opspilot.domain.actions import ActionPlan, ActionType
from opspilot.domain.alerts import AlertmanagerWebhookV4
from opspilot.domain.evidence import EvidenceRecord
from opspilot.domain.incidents import LifecycleState
from opspilot.domain.investigation import InvestigationReport
from opspilot.investigation import InvestigationWorkflow
from opspilot.postmortem import PostmortemDraft, PostmortemService
from opspilot.recovery import RecoveryResult, RecoveryVerifier
from opspilot.remediation import KubernetesRemediationAdapter, RemediationCoordinator
from opspilot.settings import Settings
from opspilot.simulation import DemoScenarioService, DemoScenarioStart
from opspilot.storage.incidents import SQLiteIncidentStore

logger = logging.getLogger(__name__)

_PUBLIC_LIFECYCLE_TARGETS = {
    LifecycleState.CLASSIFIED,
    LifecycleState.ENRICHED,
    LifecycleState.TRIAGING,
}


class IngestResponse(BaseModel):
    incident_id: str | None
    disposition: str


class LifecycleTransitionRequest(BaseModel):
    """Internal server command; the future agent may suggest, never apply, this."""

    model_config = ConfigDict(extra="forbid")

    target: LifecycleState
    actor: str = Field(min_length=1, max_length=64)
    reason: str = Field(min_length=1, max_length=500)


class InvestigationRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")

    question: str = Field(min_length=1, max_length=1_000)


class ActionPreviewRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")

    action_type: ActionType
    evidence_ids: list[str] = Field(min_length=1, max_length=20)
    target_replicas: int | None = Field(default=None, ge=1, le=3)
    requested_by: str = Field(default="local-oncall", min_length=3, max_length=128)


class ActionApprovalRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")

    approved_by: str = Field(min_length=3, max_length=128)


class ActionRejectionRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")

    rejected_by: str = Field(min_length=3, max_length=128)
    reason: str | None = Field(default=None, max_length=500)


class VerificationResponse(BaseModel):
    plan: ActionPlan
    recovery: RecoveryResult


class IncidentQueueItem(BaseModel):
    """Minimal current incident metadata for the controlled on-call queue."""

    model_config = ConfigDict(extra="forbid")

    id: str
    lifecycle_state: LifecycleState
    created_at: str
    updated_at: str
    severity: str
    service: str
    alert_name: str


def create_app(settings: Settings | None = None) -> FastAPI:
    runtime_settings = settings or Settings()
    store = SQLiteIncidentStore(runtime_settings.db_path)

    @asynccontextmanager
    async def lifespan(_: FastAPI) -> AsyncIterator[None]:
        store.initialize()
        yield

    app = FastAPI(title="OpsPilot API", version="0.1.0", lifespan=lifespan)
    app.add_middleware(
        CORSMiddleware,
        allow_origins=["http://localhost:5173", "http://127.0.0.1:5173"],
        allow_credentials=False,
        allow_methods=["GET", "POST"],
        allow_headers=["Content-Type", "X-OpsPilot-Scenario-Secret"],
    )
    app.state.store = store

    def remediation() -> RemediationCoordinator:
        return RemediationCoordinator(
            store,
            KubernetesRemediationAdapter(
                allowed_namespace=runtime_settings.demo_namespace,
                allowed_workloads={"checkout"},
            ),
            namespace=runtime_settings.demo_namespace,
            workload="checkout",
            recovery_max_5xx_rate=runtime_settings.recovery_max_5xx_rate,
            recovery_min_2xx_rate=runtime_settings.recovery_min_2xx_rate,
        )

    def action_or_404(action_id: str) -> ActionPlan:
        plan = store.action_plan(action_id)
        if plan is None:
            raise HTTPException(
                status_code=status.HTTP_404_NOT_FOUND, detail="action plan not found"
            )
        remediation().list_for_incident(plan.proposal.incident_id)
        return store.action_plan(action_id) or plan

    def postmortem_or_404(incident_id: str) -> PostmortemDraft:
        incident = store.incident(incident_id)
        if incident is None:
            raise HTTPException(
                status_code=status.HTTP_404_NOT_FOUND, detail="incident not found"
            )
        return PostmortemService().create(
            incident,
            store.list_evidence(incident_id),
            store.list_action_plans(incident_id),
            store.timeline(incident_id),
            store.latest_investigation(incident_id),
        )

    def queue_item(incident: dict[str, str]) -> IncidentQueueItem:
        service, severity, alert_name = DashboardService._alert_context(  # noqa: SLF001
            store.list_evidence(incident["id"])
        )
        return IncidentQueueItem(
            id=incident["id"],
            lifecycle_state=LifecycleState(incident["lifecycle_state"]),
            created_at=incident["created_at"],
            updated_at=incident["updated_at"],
            severity=severity,
            service=service,
            alert_name=alert_name,
        )

    @app.get("/healthz")
    def healthz() -> dict[str, str]:
        return {
            "status": "ok",
            "investigation_mode": (
                "controlled_simulation"
                if runtime_settings.simulation_investigation_enabled
                else "l
[truncated — 15519 more characters]
```

### frontend/vite.config.ts

```typescript
import { defineConfig } from "vite";
import react from "@vitejs/plugin-react";

export default defineConfig({
  plugins: [react()],
  server: { port: 5173 },
});

```

### frontend/index.html

```html
<!doctype html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>OpsPilot</title>
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="/src/main.tsx"></script>
  </body>
</html>

```

[44 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]