# Project export: WatchTower AI This document was generated by HackStack to give an AI agent context about a hackathon project. Sections are labeled with their provenance; content marked as truncated was cut to keep this document small. ## Project metadata - Hackathon: UC Berkeley AI Hackathon 2026 - Tagline: Is the AI down, or is it you? Watchtower probes Claude, GPT & Gemini in real time, tells you whose fault it is, and catches outages before the official status page does. - Devpost: https://devpost.com/software/watchtower-ai-qgi0sa - GitHub: https://github.com/kaiiiichen/WatchTower-AI - Video: https://www.youtube.com/embed/IKbsyVAsRDw?enablejsapi=1&hl=en_US&rel=0&start=&version=3&wmode=transparent - Team: 6 GitHub contributor(s) — Kai Chen (41 commits), Devin AI (11 commits), Claude Opus 4.8 (1M context) (6 commits), Cursor (4 commits), sentry[bot] (2 commits), coderabbitai[bot] (2 commits) ## Devpost submission (written by the team) ### Inspiration It was past midnight, and a Claude API call had just failed. I did what every developer does at that hour — I assumed it was me. I checked my code. I checked my network. I regenerated my API key. I re-read my own request three times. Then I opened the official status page — green, "all systems operational" — while my requests kept failing. I'd lost twenty minutes to a problem that was never mine. The provider was degraded; the status page just hadn't admitted it yet. That gap — the stretch where a service is already down but nothing official says so, and you're left debugging perfectly fine code — is the entire reason Watchtower AI exists. I wanted a tool that collapses those twenty minutes into a few seconds and answers the only question that matters at midnight: is it down, or is it me — and if it's down, whose fault is it, really? This was my first hackathon, done solo, in 24 hours. ### What it does Watchtower AI is a local "flight radar" for AI services. It runs on your machine, with your keys, and answers two questions: Is the provider okay right now? It fires concurrent probes at Claude, GPT, and Gemini every 30 seconds — and it doesn't just ping. It sends a real QA prompt and grades the answer, measures tokens-per-second, and tracks flagship and mid tiers separately, auto-selecting valid models so it never 404s on a renamed endpoint. If something's wrong, whose problem is it? A four-way attribution engine sorts every failure into your environment / your account / the provider / all-clear, and recommends a concrete fallback (e.g. route to Claude mid). When a provider degrades, Watchtower corroborates with live community signal and official status feeds, then states — honestly — how confident it can be from where you sit. ### How we built it The probe engine is a FastAPI + asyncio backend that runs concurrent health checks with httpx. Results persist to SQLite, so the trend sparklines survive a restart instead of resetting to zero — a small detail, but it's the difference between a toy and a tool you'd actually leave running. The attribution layer turns raw probe results into a verdict. Local diagnostics (DNS, TCP :443, API-key check) rule out your side in seconds; if every model on your account is impaired, it's likely the provider — but Watchtower deliberately frames this as "provider-wide from your vantage point," not as global truth (more on that below). Community corroboration went through a complete redesign mid-hackathon. We pull live Hacker News chatter through the Algolia Search API (free, no auth, works behind campus firewalls) and parse each provider's official status-page JSON. The most interesting source is Downdetector: it has no public API and hides behind Cloudflare, so we drive a real browser session through Browserbase (Playwright over CDP), auto-clear the Cloudflare challenge, wait for content, and scrape the headline and user comments. Then Claude Haiku reads the actual comments and returns a structured {problems_found, summary, comments[]}, which feeds back into spike detection. The payoff: Downdetector's own headline said "no current problems," but our LLM read the comments and surfaced connection failures and safety-routing degradation — more honest than the crowd-sourced top line. The research backbone is a backtest against the VU Amsterdam dataset from the ICPE 2025 paper An Empirical Characterization of Outages and Incidents in Public Services for LLMs (Chu et al.). On 542 real incidents, 29.7% (161) were never marked "investigating" in real time — they only ever got a "resolved" post. Median time from investigating to resolved was 73 minutes (Anthropic: 55.5, N=381). That's the documented blind window Watchtower is built to fill. The observability layer is Sentry, three layers deep — structured events, fingerprinting that groups every probe cycle, and performance traces. Then we closed the loop: we let Sentry's Seer agent do root-cause analysis on a real TypeError in our own backend and open a pull request to fix it. A tool that monitors AI, monitored and repaired by AI. The build pipeline leaned on agents at every stage. The initial probe engine and the early large-scale rewrites were built with Claude Code, with Sentry wired in alongside development for live observability from day one. Cursor handled later code generation. Simular's Sai drove a hands-off UI/UX redesign pass, and Devin (Cognition) ran whole-project syntax and security/vulnerability sweeps with up to five cloud agents in parallel. Frontend is Next.js + Tailwind; everything ships via Docker / docker-compose, running locally on your keys. How we used each tool We didn't bolt these on for the prizes — each tool does real work in the product or the build, which is exactly why we leaned into the sponsor tracks. Here's the honest, tool-by-tool breakdown: Claude (Best Use of Claude) — In-product, Claude Haiku reads the raw user comments we scrape from Downdetector and returns structured {problems_found, summary, comments[]}, which feeds straight into spike detection. That's the engine behind our sharpest line: Downdetector's headline says "no current problems," but the comments say image generation is down. Under the hood, Claude Code built the initial probe engine and the early rewrites. And across the whole 24 hours, Claude was the architecture-and-judgment partner — including talking me out of over-claiming on the research narrative. Sentry (Best Use of Sentry API) — Three layers in production: structured events, fingerprinting that groups every probe cycle, and performance traces — wired in alongside development from day one. Then Seer closed the loop, root-causing a real TypeError in our backend and opening a PR to fix it. A monitoring tool for AI, itself monitored and repaired by AI. Browserbase (Best Use of Browserbase) — Downdetector has no public API and sits behind Cloudflare. We run a real remote browser (Playwright over CDP) that auto-clears the challenge and pulls live user reports — 100+ sessions over the build. Browserbase's exact promise (give agents the web that has no API), used for a genuine need rather than a staged demo. Simular (Sai) (Best Use of Simular) — We used Sai, Simular's autonomous computer-use agent, to drive a hands-off UI/UX redesign — clicking through every screen and reshaping layout and copy. An AI redesigning the interface of an AI-monitoring tool. Cognition (Devin) (Best Use of Cognition) — Devin ran whole-project syntax and security/vulnerability sweeps with up to five cloud agents in parallel, catching issues across the codebase far faster than one solo builder could under a 24-hour clock. Claude Code & Cursor — The initial probe engine and the early big rewrites were built with Claude Code; Cursor handled later code generation. Tracks we're aiming for Ddoski's Toolbox (Grand Prize) — Watchtower is a developer utility in the purest sense: a local tool that erases the 20 minutes of "is it me?" at midnight. Useful, usable, and actually running — not just an idea. Best UI/UX — A clean provider dashboard with live sparklines, a plain-language "whose problem?" verdict, and an honesty-first design voice ("not a global oracle"), refined through the Sai-driven redesign. Best Solo Hack & Best Beginner Hack — Built solo, in 24 hours, as my first hackathon — where the judgment to cut 16 of 41 directions mattered more than any single feature. ### Challenges we ran into Reddit collapsed on us. Community signal was originally Reddit-based. Mid-build we hit Reddit's 2025 Responsible Builder Policy, which now requires pre-approval for all app access and explicitly bans scraping — impossible on a hackathon timeline. We killed it and revived Hacker News as the live source instead. Conditions changed; the judgment changed with them. Cloudflare vs. Downdetector. A plain fetch can't run JS or pass an interactive challenge. That's exactly why Browserbase fit — a real remote browser session clears the challenge, and we wait on actual content rather than a fixed sleep. A git crisis at hour ~20. Parallel AI-assisted branches, cherry-pick failures, and a broken SQLite wiring chain left main in pieces. Untangling it cost real time and taught me to stop running parallel code-gen on overlapping files. The hardest discipline was honesty, not code. The premise of a status-monitoring tool is "we catch what they miss" — and it's tempting to dress that up. We didn't. We pulled a spurious +1,140-minute case out of the backtest the moment it looked too good. We labeled every estimated value as an estimate. When the temptation came to borrow the paper's claim that outages are periodic and predictable, I cut it — our early-warning is real-time latency-trend extrapolation, not historical-cycle prediction, and conflating the two would have been dishonest. We state the detection gap as "the window we can fill," never as a measured head-start we never instrumented. For the demo, the GPT outage is triggered by a DEMO_FORCE_DOWN flag (disclosed on screen) — but the Downdetector and Hacker News corroboration around it is real. ### What we learned The thing I'm proudest of isn't how much I built — it's how much I cut. I proposed roughly 41 directions and shipped about 22, deliberately killing 16 (SLA loss tracking, a crowdsource button, public signal service, ML classifier, periodicity prediction…) because they either fought the local-tool positioning, couldn't be done honestly in the time, or just weren't worth it. The fastest way to die in a solo hackathon is to want everything and finish nothing at half quality. Saying no, correctly and repeatedly, turned out to be the most valuable skill of the whole 24 hours. I also learned that honesty is a feature, not a constraint. A monitor that's candid about its own blind spots ("not a global oracle") is more trustworthy than one that pretends to omniscience — and that candor is the differentiator, not a liability. And I learned to build with a swarm of agents without letting them trip over each other — Claude Code and Cursor writing code, Devin's five cloud agents sweeping for bugs, Sai redesigning the UI, Seer fixing what broke. The hour-20 git crisis taught me the hard way that parallel agents on overlapping files collide; the fix was clearer boundaries, not fewer agents. ### What's next Multi-vantage probing (multiple regions/keys) to move from "your route" toward genuine breadth. A real classifier on the VU dataset to put calibrated confidence behind anomaly calls. Optional, privacy-preserving aggregation so users can opt in to a shared signal without giving up the local-first model. Built with Next.js · Tailwind · FastAPI · asyncio · httpx · SQLite · Playwright · Browserbase · Sentry · Sentry Seer · Claude (Haiku) · Hacker News Algolia API · VU Amsterdam / ICPE 2025 dataset · Docker · Claude Code · Cursor · Devin (Cognition) · Simular Sai ## README (from the GitHub repository) # WatchTower AI [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](LICENSE) [![Next.js](https://img.shields.io/badge/Next.js-16-black)](frontend/package.json) [![FastAPI](https://img.shields.io/badge/FastAPI-0.115+-009688)](backend/requirements.txt) Next.js · React · TypeScript · Tailwind · FastAPI · Python · SQLite · Docker · Sentry [English](#english) · [**中文**](#中文) | | | | --- | --- | | **Source** | https://github.com/kaiiiichen/WatchTower-AI | | **Hackathon** | [UC Berkeley AI Hackathon 2026](https://ai-hackathon-2026.devpost.com/) | | **Local demo** | Frontend `http://localhost:3000` · Backend `http://localhost:8000` | | **Docker demo** | `docker compose up --build` → [http://localhost:3000](http://localhost:3000) | --- ## YouTube Demo Video [![Watch the video](./assets/youtube_preview.jpg)](https://www.youtube.com/watch?v=IKbsyVAsRDw&t=3s) --- ## English → [中文](#中文) ### Contents 1. [The idea](#the-idea) 2. [UC Berkeley AI Hackathon 2026](#uc-berkeley-ai-hackathon-2026) 3. [What WatchTower AI does](#what-watchtower-ai-does) 4. [Reception](#reception) 5. [Getting started — pick your path](#getting-started--pick-your-path) 6. [Prerequisites](#prerequisites) 7. [Quick start (Docker)](#quick-start-docker) 8. [Quick start (local dev)](#quick-start-local-dev) 9. [Verify it works](#verify-it-works) 10. [Configuration](#configuration) 11. [Troubleshooting](#troubleshooting) 12. [API reference](#api-reference) 13. [Architecture](#architecture) 14. [Detection gap & academic backing](#detection-gap--academic-backing) 15. [Product philosophy](#product-philosophy) 16. [Project structure](#project-structure) 17. [Testing](#testing) 18. [Deployment notes](#deployment-notes) 19. [Documentation map](#documentation-map) 20. [License](#license) ### The idea **WatchTower AI** is **flight radar for AI services** — detect Claude / GPT / Gemini outages before the official status page, and answer the question that keeps you up at 2 AM: *is it the service, or is it me?* Official status pages are slow, incomplete, and never tell you whether **your** environment is fine. WatchTower AI closes that gap with continuous independent probing, QA checks beyond "HTTP 200", local environment diagnosis, and honest data about how far official pages lag behind real user impact. It is a **local developer tool**: you run it on your machine, your API keys stay yours, and probe history lives in a local SQLite file — nothing is uploaded to a shared cloud service. ### UC Berkeley AI Hackathon 2026 Built at **[UC Berkeley AI Hackathon 2026](https://ai-hackathon-2026.devpost.com/)** (June 20–21, 2026) by **Kai Chen** ([@kaiiiichen](https://github.com/kaiiiichen)) as a **solo project**. All implementation occurred during the hackathon window. #### Elevator pitch (for Devpost) > WatchTower AI is a local flight radar for Claude, GPT, and Gemini — it probes providers every 30 seconds with real QA checks, diagnoses whether an outage is on your side or theirs, and backs its "detection gap" claims with peer-reviewed outage research. When something breaks at 2 AM, you get an honest answer in seconds instead of refreshing a status page that may still say "operational." #### Devpost submission checklist | Requirement | Where | | --- | --- | | 2–3 sentence summary | Use the elevator pitch above | | Project image | Screenshot of the dashboard (`localhost:3000` or deployed URL) | | GitHub repository link | https://github.com/kaiiiichen/WatchTower-AI | | Team name & table number | Enter on [Devpost](https://ai-hackathon-2026.devpost.com/) | | Demo | Live dashboard + `GET /health` JSON; 5-minute table presentation | | Built during hackathon | Yes — ideation allowed beforehand; all code written June 20–21, 2026 | **Judging alignment** (Application · Functionality · Creativity · Technical complexity): - **Application** — Every LLM developer hits midnight outages; independent probing + local diagnostics is immediately usable. - **Functionality** — Full probe loop, four-way verdict, alerts, HN + Downdetector corroboration, official status pages, VU dataset backtest, optional Sentry — all implemented, not mocked. - **Creativity** — QA probe ("2+2=4"), precursor `degrading` trend, multi-source corroboration as additive signals, honest boundaries on what we can claim. - **Technical complexity** — Dynamic model discovery, asyncio concurrent probes, SQLite history, Playwright adapters (Browserbase CDP + optional local Chromium), three-layer Sentry integration, research backtest from bundled CSV. **Sponsor track note:** Sentry integration (events + fingerprinting + performance traces with API-key redaction) qualifies for the [Best Use of Sentry API](https://ai-hackathon-2026.devpost.com/) prize criteria. ### What WatchTower AI does WatchTower AI is organized in layers. Each layer is implemented and live. #### 1. Probe layer — real-time monitoring | Capability | Description | | --- | --- | | **Independent probe network** | Concurrently probes Anthropic, OpenAI, and Google every 30 seconds (`asyncio.gather`). | | **Dynamic model discovery** | At startup, queries each provider's list-models API and picks **flagship** and **mid** tiers by rule — no hard-coded model IDs that 404 when retired. | | **Multi-tier coverage** | Each provider gets two dashboard cards (e.g. `claude-opus-*` + `claude-sonnet-*`). | | **QA quality probe** | Asks `"What is 2+2? Answer with just the number."` and verifies the reply contains `"4"`. | | **Token generation rate** | Estimates output tokens per second from each probe response. | | **Health scoring** | Rule-based score 0–100 → `operational` (≥85), `degraded` (≥50), or `down`. | | **Precursor warning (`degrading`)** | Detects steadily climbing latency *before* status crosses into degraded/down. | | **Failure semantics** | Distinguishes service faults (`down`, `degraded`) from account faults (`rate_limited`, `misconfigured`). | | **Graceful degradation** | Missing API key → `unknown`; probe loop never crashes. | #### 2. Attribution layer — whose problem is it? | Capability | Description | | --- | --- | | **Four-way verdict** | Local diagnostics: **your-side**, **account-side**, **service-side**, or **all-clear**. | | **Local environment checks** | Per provider: DNS, TCP `:443`, minimal authenticated request. | | **Smart alerts** | Rule-based alerts compare tiers, recommend failover, never conflate 429 with "service down". | | **Community corroboration** | Hacker News complaint-rate spikes + optional Downdetector (Browserbase CDP) — additive only. | | **Official status pages** | Statuspage JSON (Claude, OpenAI) + Gemini AI Studio adapter; cites provider wording when available. | #### 3. Research layer — why this matters | Capability | Description | | --- | --- | | **VU Amsterdam dataset backtest** | Real numbers from bundled CSV (`backend/data/vu_dataset/`). | | **Coverage gap** | **29.7%** of incidents (161/542) never marked "investigating" in real time. | | **Official response latency** | Median **73 min** investigating → resolved (N=381). | | **Honest boundaries** | Does **not** claim measured head-start without historical probe data. | #### 4. Observability — Sentry integration | Layer | What it does | | --- | --- | | **Events** | Sentry events for each degraded/down provider. | | **Fingerprinting** | Groups repeated probe cycles into one issue. | | **Performance traces** | One transaction per probe cycle, one span per provider. | | **Redaction** | Scrubs API keys from URLs before anything leaves the process. | #### 5. Persistence & engineering | Capability | Description | | --- | --- | | **SQLite history** | `backend/data/watchtower.db`; 7-day retention (ephemeral in Docker unless you mount a volume). | | **Frontend proxy** | Next.js `/api/*` routes proxy FastAPI; dashboard shows a clear offline state when backend is unreachable. | --- ### Reception At the hackathon, five judges stopped by the booth — above the event avera [README truncated for size] ## Detected evidence (automated analysis) Indexed codebase: 80 recognized source files, 458 KB. - CSS (language) — detected in the code - FastAPI (technology) — detected in the code - Next.js (technology) — detected in the code - Python (language) — detected in the code - React (technology) — detected in the code - Tailwind CSS (technology) — detected in the code - TypeScript (language) — detected in the code - Docker (technology) — claimed on Devpost, not found in the code - AI coding agent: Claude Code — evidence: config files committed to the repository; commit authorship or trailers - AI coding agent: Codex — evidence: config files committed to the repository - AI coding agent: Cursor — evidence: config files committed to the repository; commit authorship or trailers ## Codebase structure (from repository index) ### Files (103 of 103) ``` .cursor/rules/semantic-colors.mdc .env.example .github/pull_request_template.md .gitignore backend/.dockerignore backend/.env.example backend/app/__init__.py backend/app/backtest.py backend/app/community_downdetector.py backend/app/community_hub.py backend/app/community_shared.py backend/app/community.py backend/app/config.py backend/app/diagnostics.py backend/app/gemini_status_browser.py backend/app/main.py backend/app/models.py backend/app/monitoring.py backend/app/official_status.py backend/app/probes.py backend/app/redaction.py backend/app/store.py backend/data/vu_dataset/incident/2024-08-31/incident_stages.csv backend/data/vu_dataset/outage/2024-08-31/outage_unexploded.csv backend/Dockerfile backend/README.md backend/requirements.txt backend/tests/fixtures/status_claude_operational.json backend/tests/fixtures/status_gemini_operational.json backend/tests/fixtures/status_gemini_partial.json backend/tests/fixtures/status_openai_major.json backend/tests/fixtures/status_openai_minor.json backend/tests/test_backtest.py backend/tests/test_community.py backend/tests/test_config.py backend/tests/test_diagnostics.py backend/tests/test_discovery.py backend/tests/test_downdetector.py backend/tests/test_gemini_status_browser.py backend/tests/test_main.py backend/tests/test_models.py backend/tests/test_monitoring.py backend/tests/test_official_status.py backend/tests/test_probes.py backend/tests/test_redaction.py backend/tests/test_store.py CODE_OF_CONDUCT.md CONTRIBUTING.md docker-compose.yml frontend/.dockerignore frontend/.gitignore frontend/AGENTS.md frontend/CLAUDE.md frontend/Dockerfile frontend/eslint.config.mjs frontend/next.config.ts frontend/package.json frontend/postcss.config.mjs frontend/README.md frontend/scripts/check-semantic-colors.mjs frontend/src/app/api/backtest/route.ts frontend/src/app/api/diagnose/route.ts frontend/src/app/api/health/route.ts frontend/src/app/globals.css frontend/src/app/layout.tsx frontend/src/app/page.tsx frontend/src/components/about-page.tsx frontend/src/components/AlertBanner.tsx frontend/src/components/app-shell.tsx frontend/src/components/CommunitySignals.tsx frontend/src/components/corroboration-badge.tsx frontend/src/components/dashboard-home.tsx frontend/src/components/dashboard-settings.tsx frontend/src/components/DetectionGap.tsx frontend/src/components/diagnostic-check-help.tsx frontend/src/components/hover-tip.tsx frontend/src/components/LatencyChart.tsx frontend/src/components/LocalDiagnostics.tsx frontend/src/components/mag-chip.tsx frontend/src/components/MeasurementScope.tsx frontend/src/components/OfficialStatus.tsx frontend/src/components/provider-detail-view.tsx frontend/src/components/provider-unified-card.tsx frontend/src/components/ProviderCard.tsx frontend/src/components/providers.tsx frontend/src/components/theme-provider.tsx frontend/src/components/theme-toggle.tsx frontend/src/components/user-menu.tsx frontend/src/components/watchtower-logo.tsx frontend/src/lib/backend.ts frontend/src/lib/dashboard-nav.ts frontend/src/lib/diagnostic-check-help.ts frontend/src/lib/provider-aggregate.ts frontend/src/lib/research-foundation.ts frontend/src/lib/semantic-colors.ts frontend/src/lib/style-maps.ts frontend/src/lib/types.ts frontend/src/lib/typography.ts frontend/tsconfig.json LICENSE README.md scripts/make_thumbnail.py SECURITY.md ``` ### Dependencies - backend/requirements.txt: fastapi@>=0.115,<1.0, httpx@>=0.27,<1.0, playwright@>=1.49,<2.0, python-dotenv@>=1.0,<2.0, sentry-sdk@>=2,<3, uvicorn[standard]@>=0.30,<1.0 - frontend/package.json: @fontsource/jetbrains-mono@^5.2.8, @fontsource/nunito@^5.2.7, @tailwindcss/postcss@^4, @types/node@^20, @types/react@^19, @types/react-dom@^19, eslint@^9, eslint-config-next@16.2.9, next@16.2.9, react@19.2.4, react-dom@19.2.4, tailwindcss@^4, typescript@^5 ### Recent commits (newest first) - Merge pull request #16 from kaiiiichen/docs/youtube-demo-preview - docs: add hackathon reception note to README - docs: move YouTube demo preview above English section - docs: add YouTube demo preview with play-button thumbnail - fix: retain Downdetector signals on empty polls and load backend/.env reliably - docs: improve README for developer onboarding and troubleshooting - docs: update README for Docker deploy and current architecture - feat: dashboard overhaul, corroboration layers, and Docker deployment - Merge pull request #14 from kaiiiichen/seer/fix/missing-hn-config - fix(config): add missing community signal constants for HN Algolia - Merge pull request #13 from kaiiiichen/seer/fix/official-status-impact-labels - fix(official_status): handle 'minor' impact and unknown impact labels - feat: integrate official status pages into probe attribution - docs: rewrite README and standardize WatchTower AI branding - fix: complete SQLite persistence end-to-end (record + load + restore) - Merge pull request #12 from kaiiiichen/feat/backtest-vu-detection-gap - Update frontend/src/components/DetectionGap.tsx - Update frontend/src/app/api/backtest/route.ts - feat: add VU detection-gap backtest visualization - Merge pull request #11 from kaiiiichen/feat/diagnostics-env-profile ## Key source files (fetched from GitHub, selected and truncated for size) ### CODE_OF_CONDUCT.md ```markdown # Contributor Covenant Code of Conduct ## Our pledge We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation. We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community. ## Our standards Examples of behavior that contributes to a positive environment for our community include: - Demonstrating empathy and kindness toward other people - Being respectful of differing opinions, viewpoints, and experiences - Giving and gracefully accepting constructive feedback - Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience - Focusing on what is best not just for us as individuals, but for the overall community Examples of unacceptable behavior include: - The use of sexualized language or imagery, and sexual attention or advances of any kind - Trolling, insulting or derogatory comments, and personal or political attacks - Public or private harassment - Publishing others' private information, such as a physical or email address, without their explicit permission - Other conduct which could reasonably be considered inappropriate in a professional setting ## Enforcement responsibilities Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful. Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate. ## Scope This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. ## Enforcement Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at the contact below. All complaints will be reviewed and investigated promptly and fairly. All community leaders are obligated to respect the privacy and security of the reporter of any incident. ### Contact For this repository, contact the maintainer: - **Email:** [kaichen0728@gmail.com](mailto:kaichen0728@gmail.com), or - **GitHub:** [@kaiiiichen](https://github.com/kaiiiichen) for [truncated — 678 more characters] ``` ### SECURITY.md ```markdown # Security policy This document describes how we handle **security-sensitive information** for [WatchTower AI](https://github.com/kaiiiichen/WatchTower-AI) and this repository. --- ## Supported scope We care about vulnerabilities that affect: - **The probe engine** (FastAPI backend) when misconfiguration or code flaws could leak API keys, expose local diagnostics to unintended callers, or allow unauthorized control of probe behavior. - **The dashboard** (Next.js frontend) when deployed with a public `BACKEND_URL`, including proxy routes that forward to the probe engine. - **Local data** in `backend/data/watchtower.db` when issues stem from this application's code or documented deployment practices. - **Observability** (Sentry) when events or traces could leak provider API keys or other secrets before redaction. We do **not** provide a formal bug bounty program. Reports are handled **best-effort**. --- ## How to report a vulnerability **Do not** open a public GitHub issue with exploit details, payloads, or step-by-step instructions that put other users at risk. Instead: 1. **Contact the maintainer privately**, using one of: - **Email:** [kaichen0728@gmail.com](mailto:kaichen0728@gmail.com), or - GitHub **private security advisories** for this repository (if enabled), or - A direct message to [@kaiiiichen](https://github.com/kaiiiichen) for **non-sensitive** coordination only (not for long exploit write-ups). 2. Include **what component** is affected (e.g. route path, env var, file), **impact**, and **minimal reproduction** steps where safe. 3. Allow reasonable time for triage before public disclosure. Coordinated disclosure is appreciated. --- ## What we will do - Acknowledge receipt when possible. - Investigate and patch or mitigate **critical** issues in production configuration or code when they fall within project control. - Credit reporters in release notes or advisories if they want attribution (optional). --- ## Secrets hygiene (for contributors and users) WatchTower AI is designed as a **local developer tool**: your provider API keys and probe history stay on your machine. - **Never commit** `backend/.env`, `frontend/.env.local`, API keys, or `backend/data/watchtower.db`. - If a secret was ever committed — even briefly — **rotate it** in the provider (Anthropic, OpenAI, Google, Sentry) and **purge** from git history if the repo was public. - The backend redacts API keys from Sentry payloads (including Gemini `?key=` query params). Report gaps in redaction through private channels. - When deploying the backend publicly, restrict network access and set `CORS_ORIGINS` deliberately. The probe engine is not intended as an open multi-tenant service. See also [README.md — Configuration](README.md#configuration) and [CONTRIBUTING.md](CONTRIBUTING.md). --- ## Out of scope (examples) Reports may be **declined** or redirected when they concern: - Third-party provider outages or AP [truncated — 852 more characters] ``` ### docker-compose.yml ```yaml services: backend: build: context: ./backend env_file: - .env environment: CORS_ORIGINS: http://localhost:3000,http://127.0.0.1:3000 expose: - "8000" frontend: build: context: ./frontend env_file: - .env environment: BACKEND_URL: http://backend:8000 ports: - "3000:3000" depends_on: - backend ``` ### backend/requirements.txt ``` fastapi>=0.115,<1.0 uvicorn[standard]>=0.30,<1.0 httpx>=0.27,<1.0 python-dotenv>=1.0,<2.0 sentry-sdk>=2,<3 playwright>=1.49,<2.0 ``` ### backend/Dockerfile ``` FROM python:3.12-slim WORKDIR /app ENV PYTHONDONTWRITEBYTECODE=1 \ PYTHONUNBUFFERED=1 COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app ./app EXPOSE 8000 CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] ``` ### frontend/Dockerfile ``` FROM node:20-alpine AS base FROM base AS deps WORKDIR /app COPY package.json package-lock.json ./ RUN npm ci FROM base AS builder WORKDIR /app COPY --from=deps /app/node_modules ./node_modules COPY . . RUN npm run build FROM base AS runner WORKDIR /app ENV NODE_ENV=production \ PORT=3000 \ HOSTNAME=0.0.0.0 RUN addgroup --system --gid 1001 nodejs \ && adduser --system --uid 1001 nextjs COPY --from=builder /app/public ./public COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./ COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static USER nextjs EXPOSE 3000 CMD ["node", "server.js"] ``` ### frontend/package.json ``` { "name": "frontend", "version": "0.1.0", "private": true, "scripts": { "dev": "next dev", "build": "next build", "start": "next start", "lint": "eslint && npm run lint:colors", "lint:colors": "node scripts/check-semantic-colors.mjs" }, "dependencies": { "@fontsource/jetbrains-mono": "^5.2.8", "@fontsource/nunito": "^5.2.7", "next": "16.2.9", "react": "19.2.4", "react-dom": "19.2.4" }, "devDependencies": { "@tailwindcss/postcss": "^4", "@types/node": "^20", "@types/react": "^19", "@types/react-dom": "^19", "eslint": "^9", "eslint-config-next": "16.2.9", "tailwindcss": "^4", "typescript": "^5" } } ``` ### backend/app/main.py ```python """FastAPI app: background probe loop + GET /health returning a snapshot whose JSON matches frontend/src/lib/types.ts exactly.""" import asyncio import contextlib import logging from contextlib import asynccontextmanager import httpx from fastapi import FastAPI from fastapi.exceptions import HTTPException from fastapi.middleware.cors import CORSMiddleware from . import backtest, config, diagnostics from .community_downdetector import enabled as dd_enabled from .community_hub import CommunityHubState from .models import BacktestReport, HealthSnapshot, LocalDiagnosis from .monitoring import init_sentry from .official_status import OfficialStatusState from .probes import ProbeState, build_targets, probe_all from .store import ProbeHistoryStore logging.basicConfig(level=logging.INFO) log = logging.getLogger("watchtower") # httpx logs every request URL at INFO ("HTTP Request: GET https://...?key=..."). # Gemini passes its API key as a query param, so that line leaks the secret — # raise httpx (and its transport, httpcore) to WARNING to suppress it entirely. logging.getLogger("httpx").setLevel(logging.WARNING) logging.getLogger("httpcore").setLevel(logging.WARNING) async def _probe_loop(app: FastAPI) -> None: client: httpx.AsyncClient = app.state.client state: ProbeState = app.state.probe_state loop = asyncio.get_running_loop() while True: cycle_start = loop.time() try: await probe_all(client, state, app.state.targets) log.info("probe cycle complete: %s", state.updated_at) except Exception: # never let the loop die log.exception("probe cycle failed") elapsed = loop.time() - cycle_start await asyncio.sleep(max(0, config.PROBE_INTERVAL - elapsed)) async def _community_loop(app: FastAPI) -> None: """Refresh Hacker News community signals on their own cadence, fully decoupled from probing. Corroboration only: any failure is swallowed here so it can never touch the probe loop or the /health snapshot.""" client: httpx.AsyncClient = app.state.client community: CommunityState = app.state.community loop = asyncio.get_running_loop() while True: cycle_start = loop.time() try: await community.poll(client) except Exception: # never let the loop die (CommunityState already guards) log.exception("community poll failed") elapsed = loop.time() - cycle_start await asyncio.sleep(max(0, config.COMMUNITY_INTERVAL - elapsed)) async def _official_loop(app: FastAPI) -> None: """Refresh official status-page signals on their own cadence.""" client: httpx.AsyncClient = app.state.client official: OfficialStatusState = app.state.official loop = asyncio.get_running_loop() while True: cycle_start = loop.time() try: await official.poll(client) except Exception: log.exception("official status loop failed") elapsed = loop.time() - cycle_start await asyncio.sleep(max(0, config.OFFICIAL_STATUS_INTERVAL - elapsed)) @asynccontextmanager async def lifespan(app: FastAPI): init_sentry() # Layer 1: no-op when SENTRY_DSN is unset. app.state.client = httpx.AsyncClient(timeout=config.PROBE_TIMEOUT) app.state.probe_state = None try: app.state.targets = await build_targets(app.state.client) except Exception: log.exception("failed to build probe targets at startup") app.state.targets = [] log.info("probe targets: %s", [(t["id"], t["model"]) for t in app.state.targets]) if config.DEMO_FORCE_DOWN: log.warning( "DEMO_FORCE_DOWN=%s — synthetic degraded probe status enabled (demo only)", config.DEMO_FORCE_DOWN, ) history_store = ProbeHistoryStore() history_store.init() history_store.cleanup_old() initial_history = ( history_store.load_history(app.state.targets) if app.state.targets else {} ) restored = sum(len(pts) for pts in initial_history.values()) if restored: log.info("restored probe history from sqlite: %s points", restored) app.state.probe_state = ProbeState( app.state.targets, history_store=history_store, initial_history=initial_history, ) # Community signals (Hacker News) — corroboration source, attached to probe_state # so the snapshot can surface it. Built from the probed provider names; never # blocks startup if HN search is unreachable. provider_names = list(dict.fromkeys(t["name"] for t in app.state.targets)) monitored_models: dict[str, list[str]] = {} for t in app.state.targets: monitored_models.setdefault(t["name"], []).append(t["model"]) app.state.community = CommunityHubState(provider_names) app.state.probe_state.community = app.state.community if config.DOWNDETECTOR_ENABLED: if dd_enabled(): log.info("Downdetector enabled — will scrape all providers each community poll") else: log.warning( "DOWNDETECTOR_ENABLED=1 but Browserbase key missing/invalid — " "check backend/.env (shell exports override .env)" ) app.state.official = OfficialStatusState(provider_names, monitored_models) app.state.probe_state.official = app.state.official if app.state.targets: try: await probe_all(app.state.client, app.state.probe_state, app.state.targets) except Exception: log.exception("initial probe cycle failed") try: await app.state.community.poll(app.state.client) # best-effort warm-up except Exception: log.exception("initial community poll failed") try: await app.state.official.poll(app.state.client) except Exception: log.exception("initial official status poll failed") task = asyncio.create_task(_probe_loop(app)) community_task = asyncio.create_task(_community_loop(ap [truncated — 2486 more characters] ``` ### frontend/src/app/layout.tsx ```typescript import type { Metadata } from "next"; import { Geist, Geist_Mono } from "next/font/google"; import "@fontsource/nunito/300.css"; import "@fontsource/nunito/400.css"; import "@fontsource/nunito/600.css"; import "@fontsource/jetbrains-mono/400.css"; import Script from "next/script"; import "./globals.css"; import Providers from "@/components/providers"; import ThemeToggle from "@/components/theme-toggle"; import WatchTowerLogo from "@/components/watchtower-logo"; import { typeMdSemibold } from "@/lib/style-maps"; const geistSans = Geist({ variable: "--font-geist-sans", subsets: ["latin"], }); const geistMono = Geist_Mono({ variable: "--font-geist-mono", subsets: ["latin"], }); export const metadata: Metadata = { title: "WatchTower AI", description: "Flight radar for AI services — detect Claude / GPT / Gemini outages before the official status page does.", icons: { icon: "/logo.svg", }, }; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { return (