Gauntlet

Break your bot before someone else does.

Gauntlet attacks your chatbot like a real social engineer, shows you exactly how it got in, then patches the weakness and proves the fix works.

📖 The full project story (inspiration, what we learned, challenges) lives in ABOUT.md.

---

What it does

Point Gauntlet at a chatbot, give it a goal ("extract the secret access code"), and it runs an autonomous, escalating attack campaign:

1. Attack — five social-engineering tactics (authority impersonation, urgency pretext, format/encoding tricks, hypothetical fiction, prompt injection) across three escalation levels, from naive to multi-turn and obfuscated.
2. Judge — an LLM scores how much the bot actually leaked, semantically (catches paraphrased, partial, and encoded leaks — not just exact strings).
3. Reflect — reads why the bot held or slipped and decides to escalate, pivot, or stop.
4. Self-heal — on a breach, a patcher agent rewrites the system prompt to close the exact loophole, hot-swaps it live, re-fires the winning attack, and confirms the bot now refuses.
5. Remember — every successful exploit is saved to a vector store so the attacker starts smarter next time.

Runs against a Claude-backed bot directly, or against a real chat widget in a live cloud browser.

Architecture

A five-agent escalation loop orchestrated with LangGraph:

strategist → attacker → target → judge → reflector ─┐
    ▲                                                │
    └──────────── loop until done ───────────────────┘
                                                     │
                                          (done) → finalize → self-heal

Key design idea: deterministic control + LLM creativity. The loop's decisions (escalate / pivot / stop, which tactic·level to try next) are governed by a deterministic policy that guarantees forward progress and termination; the attack content — personas, pretexts, wording — is authored by Claude (Opus 4.8).

Stack

Layer	Tech
Agent orchestration	LangGraph, Claude Opus 4.8
Backend / streaming	FastAPI, Server-Sent Events
Exploit memory	Redis + RedisVL (vector search)
Live-browser mode	Browserbase + Playwright
Observability	Phoenix / OpenTelemetry
Dashboard	Next.js + React

Module	Role
gauntlet/graph.py	LangGraph escalation loop
gauntlet/strategist.py	picks tactic + escalation level
gauntlet/attacker.py	crafts the adversarial message(s)
gauntlet/target.py · gauntlet/browser_target.py	bot under test (API or live widget)
gauntlet/judge.py	semantic leak scoring
gauntlet/reflector.py	escalate / pivot / stop
gauntlet/patcher.py	hardens the prompt on a breach
gauntlet/memory.py	Redis vector exploit library
gauntlet/api.py	FastAPI + SSE + self-heal loop
gauntlet/eval_harness.py · gauntlet/panel.py	8-bot benchmark + metrics

Running it

Prerequisites: Python 3.14, Node 18+, and a env file (not committed) with at least:

ANTHROPIC_API_KEY=...
# optional — enables the exploit memory and live metrics mirror
REDIS_URL=redis://localhost:6379
# optional — enables live-website (browser) mode
BROWSERBASE_API_KEY=...
BROWSERBASE_PROJECT_ID=...
# optional — Phoenix tracing
PHOENIX_API_KEY=...
PHOENIX_COLLECTOR_ENDPOINT=...
PHOENIX_PROJECT=gauntlet

Backend (auto-reloads on edit):

pip install -r requirements.txt
uvicorn gauntlet.api:app --reload --port 8000

Frontend:

cd gauntlet-ui
npm install
npm run dev          # http://localhost:3000  (hot-reloads)
# or, for a production build:
npm run build && npm run start

⚠️ In production mode (next start), source edits require npm run build + a server restart before they appear — next dev hot-reloads instead.

Evaluation harness (runs the full 8-bot panel and writes the baseline metrics):

python -m gauntlet.eval_harness            # full panel
python -m gauntlet.eval_harness --bot acme_incident   # single bot

Redis is optional — without it, the pipeline runs fine, just without the exploit memory. For the vector memory you need redis-stack (RediSearch module), e.g. docker run -p 6379:6379 redis/redis-stack-server:latest.

Note on safety

This is a defensive tool for testing bots you own. The panel's "secrets" (e.g. ORCHID-7741-ZEBRA) are deliberately fake — they exist only so the judge has a concrete string to detect. The red-team pipeline never sees a target's system prompt; it only sees the replies.