# Project export: Vigil This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Keep Vigil Over Your Agents. - Devpost: https://devpost.com/software/vigil-ai-in7wfv - GitHub: https://github.com/jadhav-kunal/vigil - Team: 1 GitHub contributor(s) — Kunal Jadhav (9 commits) ## Devpost submission (written by the team) ### Overview AI agents are getting more capable, but they're also becoming more expensive and unpredictable. While building and experimenting with agentic systems, we kept running into the same problem: when an agent gets stuck, it doesn't always fail obviously. Instead, it enters a loop of slightly different tool calls, retries, and reasoning paths while continuing to consume tokens and API credits. The frustrating part is that most observability tools only tell you what went wrong after the run is already over. By then, the cost has already been incurred and the failure has already happened. We wanted something that could actively watch an agent while it was running and step in before things spiraled out of control. That idea became Vigil. Vigil is a transparent LLM proxy that sits between an AI agent and its model provider. Integrating it requires only a single change: point your existing OpenAI or Anthropic client to Vigil's local endpoint. Once connected, Vigil: Captures every agent step in real time Detects semantic loops using embedding similarity rather than exact string matching Tracks trajectory health using entropy and behavioral signals Automatically triggers a graduated circuit breaker when an agent appears stuck Downgrades expensive models when appropriate Restricts write operations during recovery attempts Halts runaway executions before costs explode Provides live dashboards, replay, and forensic debugging tools Unlike traditional observability platforms, Vigil is designed to intervene, not just observe. Vigil consists of several major components: Real-Time Proxy Layer A FastAPI-based proxy supports OpenAI-compatible and Anthropic-compatible APIs. Existing agents connect through a single base URL change. Semantic Watchdog We compute embeddings for agent actions and compare them against recent history to detect semantic repetition. This allows Vigil to catch loops even when the wording changes between steps. Circuit Breaker Inspired by distributed systems reliability patterns, Vigil uses a graduated response model: Normal operation Recovery mode Restricted mode Full halt Instead of immediately killing an agent, Vigil attempts recovery first. Effort Governor Not every step needs a frontier model. Vigil routes simpler work to smaller models and reserves expensive models for harder tasks. Context Compression We reduce redundant context accumulation and eliminate repeated information that unnecessarily inflates token usage. Replay & Forensics Every trajectory can be replayed and inspected, making debugging significantly easier than digging through logs. Live Dashboard A React dashboard streams agent activity in real time, including costs, token usage, breaker state, similarity scores, and intervention events. One of the hardest challenges was distinguishing genuine progress from repetition. Many agents naturally revisit the same concepts while solving a problem, so naive duplicate detection produces too many false positives. We had to combine multiple signals—including semantic similarity, entropy-based behavioral analysis, and state-change awareness—to build a detector that is useful in practice. Another challenge was ensuring that analysis never slowed down the agent itself. Vigil performs monitoring asynchronously so that observability does not become a bottleneck. Finally, we wanted the system to be framework-agnostic. Supporting existing OpenAI and Anthropic workflows without requiring developers to rewrite their applications became a key design constraint. Building Vigil taught us that the biggest challenge in agent systems is no longer model quality alone—it's runtime reliability. We learned how quickly costs can grow when context accumulates, how difficult semantic loop detection is compared to simple duplicate detection, and how valuable intervention mechanisms are compared to passive monitoring. We also learned that developers strongly prefer solutions that integrate with existing workflows rather than requiring framework-specific rewrites. We want to expand Vigil into a complete agent control plane with: Distributed deployment support Team-wide observability Advanced anomaly detection Security-focused policy enforcement Multi-provider routing Enterprise integrations Our long-term vision is to make Vigil the reliability layer that sits between every AI agent and every model provider. ## README (from the GitHub repository) # Vigil **A transparent LLM proxy that catches your agent looping — and stops it — while it runs.** AI agents loop and burn money. Every shipping tool either tells you *after* the fact (Langfuse, Arize, LangSmith) or counts tokens but not *meaning* (budget caps a reworded loop sails straight past). Vigil sits in the empty quadrant: **in-flight + semantic**. It watches the trajectory as it happens, detects when the agent is going in circles, and intervenes — downgrading the model, stripping write tools, or halting outright — before the bill runs away. ## 30-second quickstart Change **one line** — the `base_url` of your existing OpenAI/Anthropic client. Nothing else. ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8765/v1", # <- the only change api_key="sk-...", # your real provider key, passed straight through ) ``` Start the proxy: ```bash uv sync uv run uvicorn vigil_proxy.app:app --host 0.0.0.0 --port 8765 # health check curl localhost:8765/health # -> {"status":"ok"} ``` Your agent behaves identically — except Vigil now watches every step, and when the trajectory degenerates it acts. ## Run it locally — step by step **Prerequisites:** Python 3.11+ and [`uv`](https://docs.astral.sh/uv/) for the proxy; Node 18+ for the dashboard (optional). No Redis, no cloud account, no extra keys. ### 1. Install the proxy ```bash cd vigil uv sync # creates .venv and installs the proxy + deps cp .env.example .env # optional — the defaults already target OpenAI ``` ### 2. Start the proxy ```bash uv run uvicorn vigil_proxy.app:app --host 0.0.0.0 --port 8765 curl localhost:8765/health # -> {"status":"ok"} ``` > First start downloads the embedding model `all-MiniLM-L6-v2` (~90 MB, one time). To skip it > for a quick spin (uses a deterministic hashing embedder instead), prefix the command with > `VIGIL_EMBED_HASHING=true`. By default the proxy forwards to OpenAI (`OPENAI_BASE_URL=https://api.openai.com/v1`). Point it anywhere OpenAI-compatible by editing `.env`. ### 3. Send traffic through it Either change your client's `base_url` (the quickstart above), or test directly with curl: ```bash curl localhost:8765/v1/chat/completions \ -H "authorization: Bearer $OPENAI_API_KEY" \ -H "x-vigil-session-id: demo" \ -H "content-type: application/json" \ -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hello"}]}' ``` The optional `x-vigil-session-id` header groups requests into one trajectory (any string; one is auto-generated if omitted). Your provider key is passed straight through in `Authorization` and is **never stored**. ### 4. Watch it live — the dashboard (optional) ```bash cd packages/dashboard npm install npm run dev # http://localhost:5173 ``` It connects to the proxy's WebSocket at `ws://localhost:8765/ws` and streams every step — cost meter, similarity/entropy charts, breaker state, per-step model and token counts. (If the proxy isn't on `localhost`, set `VITE_VIGIL_WS=ws://host:8765/ws` before `npm run dev`.) ### 5. Inspect and act on a session ```bash curl localhost:8765/metrics/session/demo # steps, tokens, tokens_saved, cost curl localhost:8765/sessions/demo/breaker # breaker state + post-mortem curl -X POST localhost:8765/sessions/demo/replay # cached-trace replay (zero upstream calls) curl -X POST localhost:8765/sessions/demo/fork \ -H "content-type: application/json" -H "authorization: Bearer $OPENAI_API_KEY" \ -d '{"step_index":0,"model":"gpt-4o-mini"}' # counterfactual model fork (one call) curl -X POST localhost:8765/sessions/demo/override # reset a tripped breaker to CLOSED ``` ### 6. See a loop get caught (no API key needed) The deterministic benchmark drives scripted looping / healthy / normal trajectories through the **real** watchdog, breaker, compressor and governor — proving the breaker trips on loops, stays quiet on healthy work, and quantifying the savings: ```bash uv pip install -e ".[eval]" # matplotlib + scipy (plots + stats) uv run python -m eval.benchmark --seeds 20 ls eval/out/ # savings_table.md, ablation.md, net_savings.md, *.png, ... ``` ### 7. Run the tests ```bash uv run pytest -q # full suite (proxy + eval) # full quality gate, as run before every commit: uv run ruff check packages/proxy eval && uv run black --check packages/proxy eval \ && uv run mypy && uv run pytest -q ``` ## Endpoints | Method & path | What it does | |---|---| | `POST /v1/chat/completions` | OpenAI-compatible proxy (streaming + non-streaming) | | `POST /v1/messages` | Anthropic-compatible proxy | | `GET /health` | Liveness check | | `GET /metrics/session/{id}` | Per-session steps, tokens, compression savings, cost | | `GET /metrics/aggregate` | Cross-session totals (counts only — never prompt content) | | `GET /sessions/{id}/breaker` | Breaker state, trip step, post-mortem | | `POST /sessions/{id}/override` | Reset a tripped breaker to CLOSED | | `POST /sessions/{id}/replay` | Cached-trace replay — rebuilds the trajectory, zero upstream calls | | `POST /sessions/{id}/fork` | Re-run one step with a swapped model; diffs reasoning vs tool output | | `WS /ws` | Live step/cost/breaker stream for the dashboard | ## Configuration Everything is configured through environment variables (see `.env.example` for the full, commented list). The defaults boot a fully working local proxy. The notable toggles: | Variable | Default | Effect | |---|---|---| | `OPENAI_BASE_URL` / `ANTHROPIC_BASE_URL` | OpenAI / Anthropic | Upstream the proxy forwards to (any OpenAI-compatible host) | | `VIGIL_ALLOW_UPSTREAM_HEADER` | `false` | Allow per-request routing via the `x-vigil-upstream` header (constrain with `VIGIL_UPSTREAM_ALLOWLIST`) | | `VIGIL_COMPRESS_ENABLED` | `true` | Layer-1 loop-aware context compression (free, structural) | | `VIGIL_GOVERNOR_ENABLED` | `false` | Per-step model routing to the cheapest adequate model | | `VIGIL_FORENSICS_ENABLED` | `true` | Cache exchanges for replay/fork | | `REDIS_LANGCACHE_*` | unset | Semantic cache (M4): serve repeats from cache, skip upstream | | `VIGIL_EMBED_HASHING` | `false` | Use the offline hashing embedder (skip the ML model download) | | `VIGIL_WINDOW` / `VIGIL_TRIP_STREAK` / `VIGIL_THETA_SIM` / `VIGIL_THETA_ENT` | `5 / 3 / 0.85 / 0.30` | Watchdog detection thresholds | | `VIGIL_JUDGE_*` | unset | Optional LLM goal-judge (degrades to cosine+entropy if absent) | **Where the provider URL comes from:** Vigil forwards to the upstream set by `OPENAI_BASE_URL` / `ANTHROPIC_BASE_URL` (the request only carries the *key*, in `Authorization`, passed through and never stored). Point those at any OpenAI-compatible host (Azure, OpenRouter, vLLM, …). For **per-request** routing, enable `VIGIL_ALLOW_UPSTREAM_HEADER=true` and send a header: ```bash curl http://localhost:8765/v1/chat/completions \ -H "authorization: Bearer $KEY" \ -H "x-vigil-upstream: https://openrouter.ai/api/v1" \ -H "content-type: application/json" \ -d '{"model":"...","messages":[...]}' ``` Constrain it with `VIGIL_UPSTREAM_ALLOWLIST` (comma-separated URL prefixes) — an unconstrained upstream is an SSRF risk. The `x-vigil-*` control headers are stripped before forwarding. ## CLI A dependency-free Node CLI for setup and a self-contained demo. From a clone use `node cli/vigil.js `; once published, `npx vigil `. ```bash npx vigil init # print the one-line base_url integration (OpenAI/Anthropic, Python/Node) npx vigil prompt # print a copy-paste prompt for an AI coding agent (below) npx vigil demo # scripted runaway loop -> watch the breaker trip and freeze the cost meter ``` `vigil demo` starts a looping mock upstream, drives a running proxy with one session, and shows the breaker move `CLOSED → HALF_OPEN → OPEN` (by ~step 7) with the projected cost it capped — no API key required. [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 76 recognized source files, 335 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - HTML (language) — detected in the code - JavaScript (language) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Redis (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Anthropic (technology) — claimed on Devpost, not found in the code - OpenAI (technology) — claimed on Devpost, not found in the code ## Codebase structure (from repository index) ### Files (83 of 83) ``` .env.example .gitignore .python-version cli/package.json cli/vigil.js demo.sh eval/__init__.py eval/benchmark.py eval/conditions.py eval/datasets.py eval/engine.py eval/phoenix_eval.py eval/plots.py eval/report.py eval/stats.py eval/tests/__init__.py eval/tests/test_benchmark.py eval/tests/test_phoenix_eval.py packages/dashboard/index.html packages/dashboard/package.json packages/dashboard/src/App.tsx packages/dashboard/src/components/AggregateStrip.tsx packages/dashboard/src/components/BreakerBadge.tsx packages/dashboard/src/components/CostSparkline.tsx packages/dashboard/src/components/DocsPanel.tsx packages/dashboard/src/components/SessionList.tsx packages/dashboard/src/components/StepDetail.tsx packages/dashboard/src/components/StepLog.tsx packages/dashboard/src/components/TopBar.tsx packages/dashboard/src/components/TrajectoryChart.tsx packages/dashboard/src/hooks/useVigilSocket.ts packages/dashboard/src/index.css packages/dashboard/src/lib/format.ts packages/dashboard/src/lib/models.ts packages/dashboard/src/main.tsx packages/dashboard/src/types.ts packages/dashboard/tsconfig.json packages/dashboard/vite.config.ts packages/proxy/tests/conftest.py packages/proxy/tests/test_analyzer.py packages/proxy/tests/test_app.py packages/proxy/tests/test_breaker_manager.py packages/proxy/tests/test_breaker.py packages/proxy/tests/test_cli.py packages/proxy/tests/test_compressor.py packages/proxy/tests/test_forensics.py packages/proxy/tests/test_governor.py packages/proxy/tests/test_integrations.py packages/proxy/tests/test_normalize.py packages/proxy/tests/test_pricing.py packages/proxy/tests/test_semantic_cache.py packages/proxy/tests/test_state_mutation.py packages/proxy/tests/test_store.py packages/proxy/tests/test_streaming.py packages/proxy/tests/test_upstream_routing.py packages/proxy/tests/test_watchdog.py packages/proxy/vigil_proxy/__init__.py packages/proxy/vigil_proxy/analyzer.py packages/proxy/vigil_proxy/app.py packages/proxy/vigil_proxy/breaker_manager.py packages/proxy/vigil_proxy/breaker.py packages/proxy/vigil_proxy/compressor.py packages/proxy/vigil_proxy/embedder.py packages/proxy/vigil_proxy/forensics.py packages/proxy/vigil_proxy/governor.py packages/proxy/vigil_proxy/hub.py packages/proxy/vigil_proxy/integrations/__init__.py packages/proxy/vigil_proxy/integrations/compression_l2.py packages/proxy/vigil_proxy/integrations/semantic_cache.py packages/proxy/vigil_proxy/integrations/sentry_sink.py packages/proxy/vigil_proxy/integrations/tracing.py packages/proxy/vigil_proxy/judge.py packages/proxy/vigil_proxy/logging_config.py packages/proxy/vigil_proxy/models.py packages/proxy/vigil_proxy/normalize.py packages/proxy/vigil_proxy/pricing.py packages/proxy/vigil_proxy/settings.py packages/proxy/vigil_proxy/state_mutation.py packages/proxy/vigil_proxy/store.py packages/proxy/vigil_proxy/streaming.py packages/proxy/vigil_proxy/watchdog.py pyproject.toml README.md ``` ### Dependencies - packages/dashboard/package.json: @tailwindcss/vite@^4.0.0, @types/react@^18.3.3, @types/react-dom@^18.3.0, @vitejs/plugin-react@^4.3.1, react@^18.3.1, react-dom@^18.3.1, tailwindcss@^4.0.0, typescript@^5.5.3, vite@^5.4.0 - pyproject.toml: aiosqlite@>=0.20, black@>=24.0, fastapi@>=0.110, httpx@>=0.27, matplotlib@>=3.8, mypy@>=1.10, numpy@>=1.26, openinference-instrumentation@>=0.1, opentelemetry-exporter-otlp@>=1.24, opentelemetry-sdk@>=1.24, pydantic@>=2.6, pydantic-settings@>=2.2, pytest@>=8.0, pytest-asyncio@>=0.23, redis@>=5.0, redisvl@>=0.3, ruff@>=0.4, scipy@>=1.11, sentence-transformers@>=2.6, sentry-sdk@>=2.0, uvicorn[standard]@>=0.27, websockets@>=12.0 ### Recent commits (newest first) - fix: flush Sentry on breaker OPEN so the event delivers - feat: per-request provider routing via x-vigil-upstream header - chore: one-command demo script - feat: redis semantic cache (M4) - feat: aggregate metrics + deploy - feat: sponsor integrations - docs: step-by-step local setup, endpoints, and configuration in README - feat: cached-trace replay + fork - feat: evaluation harness - feat: effort governor - feat: loop-aware compression - chore: gitignore local PROGRESS handoff doc - feat: circuit breaker - feat: semantic watchdog - feat: live step dashboard - feat: pass-through proxy + step capture ## Key source files (fetched from GitHub, selected and truncated for size) ### pyproject.toml ``` [project] name = "vigil-proxy" version = "0.1.0" description = "Transparent, framework-agnostic LLM proxy with in-flight semantic loop detection and intervention." readme = "README.md" requires-python = ">=3.11" dependencies = [ "fastapi>=0.110", "uvicorn[standard]>=0.27", "httpx>=0.27", "pydantic>=2.6", "pydantic-settings>=2.2", "aiosqlite>=0.20", "numpy>=1.26", "sentence-transformers>=2.6", "websockets>=12.0", ] [project.optional-dependencies] eval = ["matplotlib>=3.8", "scipy>=1.11"] redis = ["redis>=5.0", "redisvl>=0.3"] sentry = ["sentry-sdk>=2.0"] tracing = [ "openinference-instrumentation>=0.1", "opentelemetry-sdk>=1.24", "opentelemetry-exporter-otlp>=1.24", ] dev = [ "ruff>=0.4", "black>=24.0", "mypy>=1.10", "pytest>=8.0", "pytest-asyncio>=0.23", ] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["packages/proxy/vigil_proxy"] [tool.ruff] line-length = 100 target-version = "py311" src = ["packages/proxy", "eval"] [tool.ruff.lint] select = ["E", "F", "I", "B", "UP", "ASYNC"] ignore = ["E501"] [tool.black] line-length = 100 target-version = ["py311"] [tool.mypy] python_version = "3.11" warn_unused_ignores = true warn_redundant_casts = true ignore_missing_imports = true check_untyped_defs = true files = ["packages/proxy/vigil_proxy", "eval"] [tool.pytest.ini_options] asyncio_mode = "auto" testpaths = ["packages/proxy/tests", "eval/tests"] pythonpath = ["packages/proxy", "."] ``` ### cli/package.json ``` { "name": "vigil-cli", "version": "0.1.0", "description": "One-line setup and a self-contained loop demo for the Vigil LLM proxy.", "bin": { "vigil": "./vigil.js" }, "type": "commonjs", "license": "MIT" } ``` ### packages/dashboard/package.json ``` { "name": "vigil-dashboard", "private": true, "version": "0.1.0", "type": "module", "scripts": { "dev": "vite", "build": "tsc --noEmit && vite build", "preview": "vite preview" }, "dependencies": { "react": "^18.3.1", "react-dom": "^18.3.1" }, "devDependencies": { "@tailwindcss/vite": "^4.0.0", "@types/react": "^18.3.3", "@types/react-dom": "^18.3.0", "@vitejs/plugin-react": "^4.3.1", "tailwindcss": "^4.0.0", "typescript": "^5.5.3", "vite": "^5.4.0" } } ``` ### packages/dashboard/src/main.tsx ```typescript import React from "react"; import ReactDOM from "react-dom/client"; import App from "./App"; import "./index.css"; ReactDOM.createRoot(document.getElementById("root")!).render( , ); ``` ### packages/dashboard/src/App.tsx ```typescript import { useEffect, useMemo, useState } from "react"; import { useVigilSocket } from "./hooks/useVigilSocket"; import type { SessionAgg, Step } from "./types"; import { TopBar } from "./components/TopBar"; import { AggregateStrip } from "./components/AggregateStrip"; import { CostSparkline } from "./components/CostSparkline"; import { TrajectoryChart } from "./components/TrajectoryChart"; import { SessionList } from "./components/SessionList"; import { StepLog } from "./components/StepLog"; import { StepDetail } from "./components/StepDetail"; import { DocsPanel } from "./components/DocsPanel"; function aggregate(steps: Step[]): SessionAgg[] { const map = new Map(); for (const s of steps) { let agg = map.get(s.session_id); if (!agg) { agg = { id: s.session_id, steps: [], cost: 0, tokensBefore: 0, tokensAfter: 0, lastTs: s.timestamp, models: new Set(), }; map.set(s.session_id, agg); } agg.steps.push(s); agg.cost += s.cost_usd || 0; agg.tokensBefore += s.tokens_before_compression ?? 0; agg.tokensAfter += s.tokens_after_compression ?? 0; agg.models.add(s.model_used); if (s.timestamp > agg.lastTs) agg.lastTs = s.timestamp; } for (const agg of map.values()) agg.steps.sort((a, b) => a.step_index - b.step_index); return [...map.values()].sort((a, b) => (a.lastTs < b.lastTs ? 1 : -1)); } export default function App() { const { conn, steps, thresholds, breakers } = useVigilSocket(); const sessions = useMemo(() => aggregate(steps), [steps]); const [selectedSession, setSelectedSession] = useState(null); const [selectedStep, setSelectedStep] = useState(null); const [view, setView] = useState<"live" | "docs">("live"); // Auto-select the most recently active session once traffic appears. useEffect(() => { if (selectedSession === null && sessions.length > 0) { setSelectedSession(sessions[0].id); } }, [sessions, selectedSession]); const current = sessions.find((s) => s.id === selectedSession) ?? null; const currentSteps = current?.steps ?? []; // selectedStep is a stable step_index value (unique per session), not an array position, so // it stays correct as the steps array grows, reorders, or is windowed. const detailStep = selectedStep !== null ? (currentSteps.find((s) => s.step_index === selectedStep) ?? null) : null; const totals = useMemo(() => { let cost = 0; let tokens = 0; let before = 0; let after = 0; for (const s of steps) { cost += s.cost_usd || 0; tokens += s.prompt_tokens ?? 0; before += s.tokens_before_compression ?? 0; after += s.tokens_after_compression ?? 0; } return { cost, tokens, saved: Math.max(0, before - after), before }; }, [steps]); const currentBreaker = selectedSession ? (breakers[selectedSession] ?? null) : null; return (
{view === "docs" ? ( ) : ( <>
{ setSelectedSession(id); setSelectedStep(null); }} /> setSelectedStep(stepIndex)} /> setSelectedStep(null)} />
)}
); } ``` ### packages/proxy/vigil_proxy/app.py ```python """Vigil proxy — FastAPI app. REQUEST PATH (must stay non-blocking on analysis): intercept -> forward to the real upstream with the caller's key passed through unchanged -> stream/return the response to the agent UNMODIFIED. ANALYSIS PATH: reconstruct the Step, persist it, and broadcast it to dashboards in a background task that never blocks the response (Invariant I1). Later slices hook the watchdog/breaker/governor into the same two paths. """ from __future__ import annotations import asyncio import json import uuid from collections.abc import AsyncIterator from contextlib import asynccontextmanager from dataclasses import dataclass import httpx from fastapi import FastAPI, Request, Response, WebSocket from fastapi.responses import JSONResponse, StreamingResponse from starlette.websockets import WebSocketDisconnect from .analyzer import Analyzer, make_analyzer from .breaker import CLOSED, is_mitigating, is_open from .breaker_manager import BreakerManager, make_breaker from .compressor import compress_messages from .embedder import make_embedder from .forensics import Forensics from .governor import Governor, make_governor from .hub import Broadcaster, step_event from .integrations.compression_l2 import TokenCompressor, make_l2_compressor from .integrations.semantic_cache import LangCacheClient, make_semantic_cache from .integrations.tracing import Tracer, make_tracer from .judge import make_judge from .logging_config import get_logger, log_event, set_level from .normalize import ( build_step, estimate_messages_tokens, normalize_anthropic_request, normalize_openai_request, ) from .pricing import PriceTable, estimate_cost, load_price_table from .settings import Settings, get_settings from .state_mutation import caused_state_mutation from .store import Store, make_store from .streaming import AnthropicStreamAccumulator, OpenAIStreamAccumulator logger = get_logger("proxy") # Hop-by-hop / length headers we must not relay verbatim (httpx recomputes them; content has # already been decoded so a stale content-encoding/length would corrupt the response). _DROP_REQUEST_HEADERS = {"host", "content-length", "connection", "accept-encoding"} _DROP_RESPONSE_HEADERS = { "content-length", "content-encoding", "transfer-encoding", "connection", } # Most recent steps replayed to a dashboard when it first connects. _SNAPSHOT_LIMIT = 200 # Keep strong refs to in-flight background tasks so they are not garbage-collected. _bg_tasks: set[asyncio.Task] = set() @dataclass class CaptureCtx: """Everything the analysis path needs, bundled so signatures stay small.""" store: Store broadcaster: Broadcaster price_table: PriceTable analyzer: Analyzer breaker: BreakerManager model_used: str | None = None # Same-estimator measurement of the message array before/after Layer 1 compression. tokens_before: int | None = None tokens_after: int | None = None forensics: Forensics | None = None # The original (pre-compression/route) request body, cached for forensic replay/fork. original_request: dict | None = None l2: TokenCompressor | None = None tracer: Tracer | None = None cache: LangCacheClient | None = None # The canonical prompt key for the semantic cache, and whether this turn was a cache hit. cache_prompt: str | None = None served_from_cache: bool = False @asynccontextmanager async def lifespan(app: FastAPI) -> AsyncIterator[None]: settings = get_settings() set_level(settings.log_level) app.state.settings = settings app.state.store = await make_store(settings) app.state.broadcaster = Broadcaster() app.state.price_table = load_price_table(settings) app.state.analyzer = make_analyzer(settings, make_embedder(settings)) app.state.governor = make_governor(settings, app.state.price_table) app.state.forensics = Forensics(app.state.store, app.state.price_table) app.state.l2 = make_l2_compressor(settings) app.state.tracer = make_tracer(settings) app.state.cache = make_semantic_cache(settings) app.state.breaker = make_breaker( settings, app.state.store, app.state.broadcaster, app.state.price_table, make_judge(settings), app.state.analyzer, ) app.state.http = httpx.AsyncClient(timeout=settings.upstream_timeout_s) log_event(logger, 20, "proxy.start", port=settings.port, redis=settings.use_redis) try: yield finally: # Let any trailing capture tasks finish before tearing down the store. if _bg_tasks: await asyncio.gather(*list(_bg_tasks), return_exceptions=True) await app.state.http.aclose() await app.state.store.close() app = FastAPI(title="Vigil", version="0.1.0", lifespan=lifespan) @app.get("/health") async def health() -> dict: return {"status": "ok"} @app.post("/v1/chat/completions") async def openai_chat(request: Request) -> Response: return await _proxy(request, provider="openai") @app.post("/v1/messages") async def anthropic_messages(request: Request) -> Response: return await _proxy(request, provider="anthropic") @app.get("/metrics/session/{session_id}") async def session_metrics(session_id: str, request: Request) -> dict: store: Store = request.app.state.store table: PriceTable = request.app.state.price_table steps = await store.get_steps(session_id) cost = sum( estimate_cost(s.model_used, s.prompt_tokens, s.completion_tokens, table) for s in steps ) before = sum(s.tokens_before_compression or 0 for s in steps) after = sum(s.tokens_after_compression or 0 for s in steps) return { "session_id": session_id, "steps": len(steps), "models_used": sorted({s.model_used for s in steps if s.model_used}), "tokens_before_compression": before, "tokens_after_compression": after, "tokens_saved": max(0, before - after), "completion_token [truncated — 23952 more characters] ``` ### demo.sh ```shell #!/usr/bin/env bash # # Vigil 90-second demo — starts the proxy + a scripted looping upstream and trips the breaker, # so you can watch a runaway loop get caught and the cost meter freeze. No API key required. # # ./demo.sh # proxy on :8765 (default) # PORT=8766 ./demo.sh # if :8765 is busy (then dashboard: VITE_VIGIL_WS=ws://localhost:8766/ws) # STEPS=14 ./demo.sh # drive more steps # set -euo pipefail cd "$(dirname "$0")" PORT="${PORT:-8765}" MOCK_PORT="${MOCK_PORT:-8799}" STEPS="${STEPS:-10}" DB="$(mktemp -t vigil-demo).db" LOG="$(mktemp -t vigil-demo-proxy).log" cleanup() { [ -n "${PROXY_PID:-}" ] && kill "$PROXY_PID" 2>/dev/null || true; rm -f "$DB"*; } trap cleanup EXIT INT TERM if lsof -tiTCP:"$PORT" -sTCP:LISTEN >/dev/null 2>&1; then echo "✗ Port $PORT is already in use." echo " Run on another port: PORT=8766 ./demo.sh" echo " (and start the dashboard with VITE_VIGIL_WS=ws://localhost:8766/ws npm run dev)" exit 1 fi echo "▶ Starting Vigil on :$PORT, pointed at a looping mock upstream on :$MOCK_PORT ..." OPENAI_BASE_URL="http://127.0.0.1:${MOCK_PORT}/v1" VIGIL_EMBED_HASHING=true VIGIL_DB_PATH="$DB" \ uv run uvicorn vigil_proxy.app:app --port "$PORT" --log-level info >"$LOG" 2>&1 & PROXY_PID=$! for _ in $(seq 1 60); do curl -sf "localhost:$PORT/health" >/dev/null 2>&1 && break; sleep 1; done curl -sf "localhost:$PORT/health" >/dev/null 2>&1 || { echo "✗ proxy did not start — see $LOG"; tail -5 "$LOG"; exit 1; } echo "✓ proxy healthy at http://localhost:$PORT (proxy logs: $LOG)" echo echo "▶ Driving a scripted runaway loop (the agent keeps calling the same tool) ..." node cli/vigil.js demo --proxy "http://localhost:$PORT" --mock-port "$MOCK_PORT" --steps "$STEPS" echo echo "▶ Cross-session aggregate (counts only — never prompt content):" curl -s "localhost:$PORT/metrics/aggregate" | (python3 -m json.tool 2>/dev/null || cat) echo echo "▶ Cached-trace replay of the demo session (zero upstream calls):" REPLAY="$(curl -s -X POST "localhost:$PORT/sessions/vigil-demo/replay")" echo "$REPLAY" | python3 -c "import sys,json;d=json.load(sys.stdin);print(' replayed',len(d['steps']),'steps; upstream_calls =',d['upstream_calls'],'; trace_hash =',d['trace_hash'][:12]+'...')" 2>/dev/null \ || echo " $REPLAY" echo echo "Done. For the live dashboard view: in another terminal," echo " cd packages/dashboard && npm run dev # then re-run ./demo.sh and watch it live" ``` ### eval/__init__.py ```python """Vigil evaluation harness (built in Slice 7).""" ``` ### eval/conditions.py ```python """The ablation ladder (eval design §5). Each condition toggles exactly the mechanisms it isolates; the task stream and seeds are identical across conditions (a paired design), so the delta between adjacent rows attributes a saving to one mechanism.""" from __future__ import annotations from dataclasses import dataclass @dataclass(frozen=True) class Condition: id: str label: str breaker: bool # M2 — loop interruption governor: bool # M3 — per-step routing compressor: bool # M1 — context compression cache: bool # M4 — semantic cache CONDITIONS: list[Condition] = [ Condition("C0", "Control", False, False, False, False), Condition("C1", "Breaker", True, False, False, False), Condition("C2", "Governor", False, True, False, False), Condition("C3", "Compressor", False, False, True, False), Condition("C4", "Cache", False, False, False, True), Condition("C5", "Full", True, True, True, True), ] BY_ID = {c.id: c for c in CONDITIONS} ``` ### eval/stats.py ```python """Statistics for the benchmark (eval design §6): bootstrap CIs (robust to the right-skewed cost distribution), a paired Wilcoxon signed-rank test (C0 vs C5), and the matched-pairs rank-biserial effect size. scipy is used when available and falls back to a deterministic numpy implementation, so the harness still runs without the optional `eval` extra (Invariant I2).""" from __future__ import annotations from dataclasses import dataclass import numpy as np try: # scipy is in the optional `eval` extra; degrade gracefully if absent. from scipy import stats as _scipy_stats except Exception: # pragma: no cover - exercised only without the extra installed _scipy_stats = None @dataclass class CI: mean: float lo: float hi: float def fmt(self, dollars: bool = False) -> str: u = "$" if dollars else "" p = 4 if dollars else 1 return f"{u}{self.mean:.{p}f} [{u}{self.lo:.{p}f}, {u}{self.hi:.{p}f}]" def bootstrap_ci(values: list[float], *, n: int = 2000, seed: int = 0, alpha: float = 0.05) -> CI: arr = np.asarray(values, dtype=float) if arr.size == 0: return CI(0.0, 0.0, 0.0) if arr.size == 1: return CI(float(arr[0]), float(arr[0]), float(arr[0])) rng = np.random.default_rng(seed) idx = rng.integers(0, arr.size, size=(n, arr.size)) means = arr[idx].mean(axis=1) lo, hi = np.quantile(means, [alpha / 2, 1 - alpha / 2]) return CI(float(arr.mean()), float(lo), float(hi)) @dataclass class PairedTest: statistic: float p_value: float rank_biserial: float n: int def fmt(self) -> str: return f"W={self.statistic:.1f}, p={self.p_value:.4g}, rank-biserial={self.rank_biserial:+.3f} (n={self.n})" def wilcoxon_signed_rank(a: list[float], b: list[float]) -> PairedTest: """Paired non-parametric test on a vs b (e.g. cost under C0 vs C5). Zero-difference pairs are dropped (Wilcoxon convention). Returns the matched-pairs rank-biserial effect size too.""" x = np.asarray(a, dtype=float) y = np.asarray(b, dtype=float) diff = x - y nz = diff[diff != 0] n = int(nz.size) if n == 0: return PairedTest(0.0, 1.0, 0.0, 0) ranks = _rankdata(np.abs(nz)) r_plus = float(ranks[nz > 0].sum()) r_minus = float(ranks[nz < 0].sum()) total = r_plus + r_minus rank_biserial = (r_plus - r_minus) / total if total else 0.0 if _scipy_stats is not None: res = _scipy_stats.wilcoxon(x, y, zero_method="wilcox", correction=False, mode="auto") return PairedTest(float(res.statistic), float(res.pvalue), rank_biserial, n) # Normal approximation with tie correction (deterministic fallback). w = min(r_plus, r_minus) mean_w = n * (n + 1) / 4.0 _, counts = np.unique(np.abs(nz), return_counts=True) tie = float((counts**3 - counts).sum()) var_w = (n * (n + 1) * (2 * n + 1) - tie / 2.0) / 24.0 z = (w - mean_w) / np.sqrt(var_w) if var_w > 0 else 0.0 p = float(2.0 * (1.0 - _norm_cdf(abs(z)))) return PairedTest(float(w), min(1.0, p), rank_biserial, n) def _rankdata(a: np.ndarray) -> np.ndarray: """Average ranks (1-based), ties shared — matches scipy.stats.rankdata('average').""" order = np.argsort(a, kind="mergesort") ranks = np.empty(a.size, dtype=float) sa = a[order] i = 0 while i < a.size: j = i while j + 1 < a.size and sa[j + 1] == sa[i]: j += 1 avg = (i + j) / 2.0 + 1.0 ranks[order[i : j + 1]] = avg i = j + 1 return ranks def _norm_cdf(x: float) -> float: import math return 0.5 * (1.0 + math.erf(x / math.sqrt(2.0))) ``` [68 more indexed source files omitted to keep this export small. The full file list is in the Codebase structure section above.]