Evidence-bound authority for AI incident agents.
Open the live lab Run the 2-minute demo Run locally
> **An AI agent must earn the right to act.** Faultfix lets an agent investigate, but trusted evidence, deterministic policy, and human review decide whether an action is `ALLOW`, `REVIEW`, or `BLOCK`. Faultfix is not another incident investigator. It is the authority layer beneath one: the component that decides what evidence can influence an agent, what action is in scope, and whether the action is permitted. --- ## The authority boundary | Signal | What it can do | What it cannot do | | --- | --- | --- | | **Model recommendation** | Suggest the next evidence check | Prove cause or authorize a production write | | **Trusted, replay-bounded evidence** | Inform the policy decision | Override the causal proof gate | | **Action Lease** | Permit one narrow, reversible containment action after human review | Create standing permission | | **Causal proof + reproduction** | Unlock a human-reviewed permanent patch candidate | Bypass human approval | ```mermaid flowchart LR E["Incident evidence arrives"] --> F{"Faultfix Evidence Firewall"} F -->|"trusted and before replay cutoff"| C["Sanitized evidence context"] F -->|"untrusted or too late"| Q["Quarantine\nno model context / no authority"] C --> M["Model advisory"] M --> P{"Deterministic authority policy"} P -->|"read-only"| A["ALLOW"] P -->|"reversible + scoped"| R["REVIEW"] P -->|"permanent or proof incomplete"| B["BLOCK"] ``` The graphic above and the diagram are the same contract: raw, untrusted content cannot reach the model; model output cannot become authority; permanent changes remain blocked until independently proved. --- ## See it in action | Demo moment | What judges see | Why it matters | | --- | --- | --- | | **Hostile ticket** | A global production command is `BLOCK`ed and **0** raw ticket bytes reach model context | Prompt injection is stopped before inference, not “handled” by asking the model nicely | | **Authority Simulator** | A judge can name a non-sensitive scenario, then change trust, replay time, action scope, and proof state to produce a fingerprinted `ALLOW` / `REVIEW` / `BLOCK` receipt | The scenario label is display-only; the policy boundary is inspectable without an API key, a provider call, or any Hugging Face credit | | **INC-042** | A reversible containment route before causal proof is complete | Containment is not a root-cause verdict | | **Four-pack challenge** | Capacity, DNS, identity rotation, and insufficient-evidence packs | The model advises across different cases; the authority policy stays the same | | **Public case library** | Google Cloud and Cloudflare postmortems become provenance-tagged evidence | The product can work with real public evidence without pretending it has live production telemetry |Start with the hostile-ticket block Follow the judge run-of-show
--- ## How a permanent fix is earned ```mermaid flowchart TB S["Direct symptom"] --> D["Deploy + configuration evidence"] D --> X["Alternative explanation rejected"] X --> R["Regression reproduces failure"] R --> P["Human-reviewed permanent patch candidate"] D -. "bounded containment only" .-> L["Action Lease"] L --> H["Human review"] H --> C["Temporary, scoped containment"] M["Model hypothesis"] -. "never sufficient" .-> P ``` The deterministic demo incident is `INC-042`: payments fail after release `r42` reduces `DATABASE_POOL_LIMIT` from 40 to 20. Faultfix records the following evidence sequence: 1. Connection acquisition is exhausted in AZ-A. 2. The payment path stalls at the data-service pool. 3. Release `r42` changed the pool configuration. 4. The limit changed from `40` to `20`. 5. The overlapping DNS event is rejected: it affected another zone and cannot explain the symptom. 6. The regression test reproduces failure at `20` and resolves it at `40`. Before causal proof is complete, Faultfix can offer a separate, simulated containment packet: pause `r42` promotion and drain AZ-A traffic from `r42` instances. It is review-gated, time-boxed, resource-scoped, and bound to one evidence fingerprint. It records **impact contained**, never **cause established**. --- ## What is real, what is simulated | Surface | Boundary | | --- | --- | | `INC-042`, causal graph, regression, containment packet, and prevention guardrail | Deterministic fixture bundled with the app; no production system is queried | | Evidence Firewall and Action Lease | Real policy mechanics shown through a deterministic security demo; the lease is simulated, scoped, time-bounded, and evidence-bound | | Authority Simulator | A deterministic, fixed-enum policy evaluator. It makes zero model or provider calls and emits only a receipt derived from the selected policy attributes | | Google Cloud GCE and Cloudflare incident packs | Structured from official public postmortems; they are read-only public evidence, not private raw telemetry or independent re-investigations | | Hosted Space ranking | Runs keylessly with a small model plus deterministic fallback | | Hosted live investigator | Optional. It requires a deployer-configured provider secret; its responses are validated and advisory only | No model can alter the evidence sequence, proof score, containment authority, incident receipt, or permanent-fix gate. The public case library links directly to the original [Google Cloud GCE postmortem](https://status.cloud.google.com/incident/compute/16007?post-mortem=) and [Cloudflare November 2025 postmortem](https://blog.cloudflare.com/18-november-2025-outage/). Faultfix displays bounded, paraphrased facts from those sources; it does not ingest their raw content into a model. --- ## Install the policy CLI Faultfix includes an installable, offline **policy preflight** for terminal and CI use. It evaluates only fixed structured attributes; it never sends raw logs, prompts, or secrets anywhere, and it can never execute a production action. **Supported platforms:** Windows, macOS, and Linux with Python 3.10+. ```bash pipx install "git+https://github.com/jacklachan/faultfix.git" faultfix check \ --trust trusted \ --replay within-cutoff \ --action permanent \ --proof reproduced \ --scenario "Checkout timeouts after deployment" \ --format json ``` `pipx` keeps this developer tool in an isolated environment. If it is not installed, run `python -m pip install --user pipx` first (on Windows, use `py` in place of `python`), then restart the terminal after `python -m pipx ensurepath`. The command returns a fingerprinted receipt from the same policy used by the Space. It intentionally exits with `0` for `ALLOW`, `20` for `REVIEW`, `30` for `BLOCK`, and `64` for malformed policy input, so a pipeline or agent wrapper cannot mistake human review for permission to proceed. For CI, pass only the bounded policy fields in JSON—never a raw incident log: ```json { "evidence_trust": "trusted", "replay_status": "within-cutoff", "requested_action": "permanent", "proof_state": "reproduced", "scenario_label": "Checkout timeouts after deployment" } ``` ```bash faultfix check --input faultfix-policy.json --format json ``` The scenario label is display-only. It is not model context, policy e [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 23 recognized source files, 259 KB. - CSS (language) — detected in the code - Hugging Face (technology) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - PyTorch (technology) — detected in the code - React (technology) — detected in the code - TypeScript (language) — detected in the code - AI coding agent: Claude Code — evidence: config files committed to the repository - AI coding agent: Codex — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (30 of 30) ``` .gitignore AGENTS.md CLAUDE.md docs/judge-demo.md eslint.config.mjs hosted-ranking-space/app.py hosted-ranking-space/faultfix_policy.py hosted-ranking-space/README.md hosted-ranking-space/requirements.txt package.json pyproject.toml README.md scripts/sync-space.ps1 scripts/verify-authority-simulator.py scripts/verify-faultfix-cli.py src/app/advisory-ranking.module.css src/app/globals.css src/app/layout.tsx src/app/page.module.css src/app/page.tsx src/app/phase2.module.css src/lib/agent-lab.test.ts src/lib/agent-lab.ts src/lib/evidence-firewall.test.ts src/lib/evidence-firewall.ts src/lib/hosted-ranking.test.ts src/lib/hosted-ranking.ts src/lib/investigation.test.ts src/lib/investigation.ts tsconfig.json ``` ### Dependencies - hosted-ranking-space/requirements.txt: gradio@>=5.49,<6, sentencepiece@>=0.2, spaces@>=0.51,<0.52, torch@>=2.4, transformers@>=4.46,<5 - package.json: @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@16.2.10, next@16.2.10, react@19.2.4, react-dom@19.2.4, typescript@^5, vitest@^4.1.10 ### Recent commits (newest first) - feat: add installable authority policy CLI - feat: add safe scenario sandbox - fix: improve Space legibility - docs: add submission tester guide - fix: require cited supported claims - fix: restore ZeroGPU startup hook - fix: align hosted authority policy - feat: add deterministic authority simulator - feat: add public incident case library - fix: harden ranking and runtime dependencies - fix: clarify public evidence boundary - fix: label simulated incident benchmarks - refactor: make live investigator Hugging Face only - fix: retain required ZeroGPU declaration - fix: disable experimental Space SSR - refactor: remove unreachable demo code - fix: restore Space CTA hierarchy - feat: elevate demo presentation - fix: fail over live model providers - fix Space callback compatibility ## Key source files (fetched from GitHub, selected and truncated for size) ### CLAUDE.md ```markdown @AGENTS.md ``` ### AGENTS.md ```markdown # This is NOT the Next.js you know This version has breaking changes — APIs, conventions, and file structure may all differ from your training data. Read the relevant guide in `node_modules/next/dist/docs/` before writing any code. Heed deprecation notices. ``` ### pyproject.toml ``` [build-system] requires = ["setuptools>=68"] build-backend = "setuptools.build_meta" [project] name = "faultfix" version = "0.1.0" description = "Offline evidence-bound authority preflight for AI incident agents" readme = "README.md" requires-python = ">=3.10" authors = [{ name = "Faultfix contributors" }] classifiers = [ "Programming Language :: Python :: 3", "Programming Language :: Python :: 3 :: Only", "Topic :: Security", ] [project.scripts] faultfix = "faultfix_policy:main" [tool.setuptools] package-dir = { "" = "hosted-ranking-space" } py-modules = ["faultfix_policy"] ``` ### package.json ``` { "name": "faultfix", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint", "test": "vitest run", "test:space": "py scripts/verify-authority-simulator.py", "test:cli": "py scripts/verify-faultfix-cli.py", "verify": "npm test && npm run test:space && npm run test:cli && npm run lint && npm run build" }, "dependencies": { "next": "16.2.10", "react": "19.2.4", "react-dom": "19.2.4" }, "devDependencies": { "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.2.10", "typescript": "^5", "vitest": "^4.1.10" }, "overrides": { "next": { "postcss": "8.5.19" } } } ``` ### hosted-ranking-space/requirements.txt ``` gradio>=5.49,<6 spaces>=0.51,<0.52 transformers>=4.46,<5 torch>=2.4 sentencepiece>=0.2 ``` ### src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "./globals.css"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "faultfix — agent authority layer", description: "The evidence-first authority layer that makes incident-response agents earn the right to act.", }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return ( {children} ); } ``` ### scripts/verify-faultfix-cli.py ```python """Exercise the installable Faultfix policy CLI without dependencies or network access.""" from __future__ import annotations import contextlib import io import json import sys import tempfile from pathlib import Path ROOT = Path(__file__).resolve().parents[1] SPACE_DIR = ROOT / "hosted-ranking-space" sys.path.insert(0, str(SPACE_DIR)) import faultfix_policy def run(*arguments: str) -> tuple[int, str]: output = io.StringIO() with contextlib.redirect_stdout(output): code = faultfix_policy.main(arguments) return code, output.getvalue() def main() -> None: code, output = run( "check", "--trust", "trusted", "--replay", "within-cutoff", "--action", "permanent", "--proof", "incomplete", "--scenario", "Checkout timeouts after deploy", "--format", "json", ) receipt = json.loads(output) assert code == 30 assert receipt["authority"] == "BLOCK" assert receipt["model_calls"] == 0 assert receipt["scenario_label"] == "Checkout timeouts after deploy" assert receipt["scenario_label_boundary"].startswith("display-only") code, output = run( "check", "--trust", "trusted", "--replay", "within-cutoff", "--action", "permanent", "--proof", "reproduced", "--format", "json", ) receipt = json.loads(output) assert code == 20 assert receipt["authority"] == "REVIEW" assert receipt["receipt_fingerprint"] == faultfix_policy.evaluate_authority_simulator( "trusted", "within-cutoff", "permanent", "reproduced" )["receipt_fingerprint"] with tempfile.TemporaryDirectory() as temporary_directory: fixture = Path(temporary_directory) / "policy.json" fixture.write_text( json.dumps( { "evidence_trust": "untrusted", "replay_status": "within-cutoff", "requested_action": "observe", "proof_state": "reproduced", } ), encoding="utf-8", ) code, output = run("check", "--input", str(fixture), "--format", "json") receipt = json.loads(output) assert code == 30 assert receipt["authority"] == "BLOCK" assert receipt["disposition"] == "quarantine" fixture.write_text(json.dumps({"raw_log": "do not execute this"}), encoding="utf-8") errors = io.StringIO() with contextlib.redirect_stderr(errors): code = faultfix_policy.main(("check", "--input", str(fixture))) assert code == 64 assert "Unsupported policy input field" in errors.getvalue() print("PASS: installable CLI shares the deterministic Space policy") print("PASS: REVIEW and BLOCK use non-zero CI exit codes") print("PASS: raw-log-shaped JSON fields are rejected") print("PASS: CLI makes zero model, provider, and network calls") if __name__ == "__main__": main() ``` ### scripts/verify-authority-simulator.py ```python """Verify the Space policy boundary without starting the app. This standard-library test selectively loads only the module definitions needed for the simulator. It poisons every model and provider helper so a future refactor cannot accidentally turn the no-cost control surface into an inference path. """ from __future__ import annotations import ast import json import re import sys import types from collections import Counter from itertools import product from pathlib import Path APP_PATH = Path(__file__).resolve().parents[1] / "hosted-ranking-space" / "app.py" SPACE_DIR = APP_PATH.parent TARGET = "render_authority_simulator" LOAD_UNTIL = "reset_authority_simulator" def load_simulator_namespace(): sys.path.insert(0, str(SPACE_DIR)) source = APP_PATH.read_text(encoding="utf-8") tree = ast.parse(source, filename=str(APP_PATH)) body = [] for node in tree.body: if isinstance( node, ( ast.Import, ast.ImportFrom, ast.Assign, ast.AnnAssign, ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, ), ): body.append(node) if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name == LOAD_UNTIL: break else: raise AssertionError(f"{LOAD_UNTIL} was not found in {APP_PATH}") gradio = types.ModuleType("gradio") gradio.Request = object sys.modules["gradio"] = gradio spaces = types.ModuleType("spaces") spaces.GPU = lambda fn=None, *args, **kwargs: ( fn if callable(fn) else lambda decorated: decorated ) sys.modules["spaces"] = spaces hub = types.ModuleType("huggingface_hub") hub.InferenceClient = object sys.modules["huggingface_hub"] = hub transformers = types.ModuleType("transformers") transformers.pipeline = lambda *args, **kwargs: None sys.modules["transformers"] = transformers namespace: dict[str, object] = {"__name__": "__authority_simulator_test__"} module = ast.Module(body=body, type_ignores=[]) exec(compile(ast.fix_missing_locations(module), str(APP_PATH), "exec"), namespace) return namespace def attribute(markup: str, name: str) -> str: match = re.search( rf"\b{re.escape(name)}=([\"'])(.*?)\1", markup, flags=re.IGNORECASE | re.DOTALL, ) if not match: raise AssertionError(f"Missing {name} in receipt:\n{markup}") return match.group(2) def expected_disposition(trust: str, replay: str) -> str: # Replay status is evaluated before trust, mirroring the Evidence Firewall. if replay == "post-cutoff": return "future" if trust == "untrusted": return "quarantine" return "admit" def expected_authority(disposition: str, action: str, proof: str) -> str: if disposition != "admit": return "BLOCK" if action == "observe": return "ALLOW" if action == "contain": return "REVIEW" return "REVIEW" if proof == "reproduced" else "BLOCK" def main() -> None: namespace = load_simulator_namespace() calls: list[str] = [] def poison(name: str): def blocked(*args, **kwargs): calls.append(name) raise AssertionError(f"{name} must never run in Authority Simulator") return blocked # The simulator must never reserve budget, instantiate a provider client, or # invoke either the local ranker or the hosted investigator. for name in ( "ranker", "pipeline", "warm_ranker", "invoke_live_model", "invoke_hf_completion", "invoke_hf_with_single_fallback", "reserve_live_model_budget", "configured_hf_model", "maximum_hf_attempts", ): namespace[name] = poison(name) namespace["InferenceClient"] = poison("InferenceClient") evaluate = namespace["evaluate_authority_simulator"] render = namespace[TARGET] assert callable(evaluate) assert callable(render) outcomes: Counter[str] = Counter() fingerprints: set[str] = set() for trust, replay, action, proof in product( ("trusted", "untrusted"), ("within-cutoff", "post-cutoff"), ("observe", "contain", "permanent"), ("incomplete", "reproduced"), ): result = evaluate(trust, replay, action, proof) markup = render(trust, replay, action, proof) disposition = expected_disposition(trust, replay) authority = expected_authority(disposition, action, proof) model_reach = "admitted" if disposition == "admit" else "none" assert result["disposition"] == disposition assert result["authority"] == authority assert result["model_reach"] == model_reach assert attribute(markup, "data-authority").upper() == authority assert attribute(markup, "data-evidence-disposition").lower() == disposition assert attribute(markup, "data-model-reach").lower() == model_reach assert attribute(markup, "data-model-calls") == "0" fingerprint = attribute(markup, "data-receipt-fingerprint").upper() assert re.fullmatch(r"[A-F0-9]{12}", fingerprint), fingerprint fingerprints.add(fingerprint) outcomes[authority] += 1 assert outcomes == Counter({"BLOCK": 19, "REVIEW": 3, "ALLOW": 2}), outcomes assert len(fingerprints) == 24, "Every simulator state must produce a distinct receipt" # Malformed public input fails closed; it cannot select a friendly outcome. malformed = evaluate("trusted