FlowGate -- Unified LLM Gateway & Control Plane

A shared control layer for applications that call large language models (LLMs). Instead of every application separately handling provider access, caching, limits, budgets, and outages, they can call one OpenAI-compatible gateway.

This README is written from the project state, verification reports, and source files. It distinguishes recorded results from items that are still open.

Project Overview

The FLowGate AI Middleware Platform is a Python 3.11+ and FastAPI service that sits between client applications and configured LLM providers. Clients send a Chat Completions request to POST /v1/chat/completions; the gateway applies the platform controls before it calls a provider.

The platform includes:

• Team authentication with Authorization: Bearer <team_api_key>.
• A semantic cache backed by Qdrant Server in production.
• Redis-backed rate limiting and budget enforcement.
• Provider routing, retries, fallback, and a circuit breaker.
• Incremental response streaming using Server-Sent Events (SSE).
• Telemetry, metrics, dashboards, alerts, and admin APIs.

It is not a new LLM or a replacement for a provider. It is a control plane around existing provider calls.

Why This Exists

The source plan and PRD/TRD describe a common problem: teams can make repeated LLM calls, exceed a shared budget, and have little visibility when a provider slows down or fails. Rebuilding those controls separately in every application adds cost and operational risk.

This project centralizes those controls. A cacheable repeat can return without another provider call, a request can be stopped before forwarding when a rate limit or budget is exceeded, and telemetry records what happened.

How It Works

Request Workflow

The runtime order is fixed and verified as follows:

flowchart TD
    C[Client application] --> A[Authentication and team lookup]
    A --> SC[Semantic cache]
    SC -- Cache hit --> CH[Return cached response]
    CH --> T[Telemetry]
    SC -- Cache miss --> RL[Redis rate limiting]
    RL -- Blocked --> R429[429 response]
    RL -- Allowed --> B[Redis budget enforcement]
    B -- Blocked --> BE[Budget error response]
    B -- Allowed --> PR[Provider routing]
    PR --> RT[Retries]
    RT --> CB[Circuit breaker]
    CB --> FB[Fallback]
    FB --> P[Provider call]
    P --> S[Response streaming]
    S --> CS[Store complete successful response in semantic cache]
    CS --> T

In plain language:

1. The gateway checks the incoming team API key and loads that team's allowed models and providers.
2. It looks for a usable answer in the semantic cache.
3. On a cache hit, it returns the cached response. This bypasses rate limiting and budget enforcement, as required by the project specification.
4. On a cache miss, Redis checks whether the team is within its request/token limit and budget before the request reaches a provider.
5. The provider router selects a permitted provider for the requested model.
6. The resilience stages handle retryable failures, circuit-breaker state, and configured fallback paths.
7. The provider response is returned normally or streamed in chunks. A complete successful response can be saved for later cache hits.
8. Telemetry records the request path, cache outcome, timing, provider outcome, and related metrics.

Major Components

Component	What it does
FastAPI gateway	Exposes the OpenAI-compatible Chat Completions endpoint and health/metrics routes.
Authentication	Looks up a team from its inbound Bearer API key.
Semantic cache	Uses prompt similarity and cache policy to return a stored response when appropriate. Qdrant Server is the production cache store.
Redis rate limiting	Uses token buckets to limit requests and tokens for each team. A priority queue component is also verified separately.
Budget enforcement	Reserves the maximum configured cost before a provider call, then settles against provider usage or releases the reservation according to the recorded streaming rules.
Provider routing	Applies the YAML policy that maps a requested model to a provider, while team policy controls which models/providers a team may request.
Retries, fallback, and circuit breaker	Retries eligible failures, prevents repeated calls to an unhealthy route, and selects a configured fallback when available.
Streaming	Forwards OpenAI-compatible SSE chunks incrementally. Complete successful streams may be cached after they finish.
Telemetry	Produces request, cache, spend, rate-limit, fallback, and circuit-breaker metrics plus tracing spans.
Admin APIs	Provide rate-limit status, limits/budgets updates, spend, alert-threshold updates, cache invalidation, and cache threshold testing.
Prometheus	Scrapes the gateway /metrics endpoint and stores metrics for querying.
Grafana	Displays the provisioned Operations, Business, Performance, and Cache dashboards.
OpenTelemetry Collector	Receives deployment tracing over OTLP/HTTP.
Qdrant Server	Stores semantic-cache vectors and metadata in the production deployment.

Architecture

flowchart LR
    Client[Client applications] --> Gateway[Production gateway\nFastAPI + Gunicorn]
    Gateway --> Redis[Redis\nrate limiting and budget enforcement]
    Gateway --> Qdrant[Qdrant Server\nsemantic cache]
    Gateway --> Providers[Configured provider adapters]
    Gateway --> Collector[OpenTelemetry Collector]
    Prometheus[Prometheus] -->|scrapes /metrics| Gateway
    Grafana[Grafana] -->|queries| Prometheus
    Admin[Admin APIs] --> Gateway

    Demo[Demo gateway\ndemo profile] --> Mock[Mock provider\ndemo profile]
    Demo --> Redis
    Demo --> Qdrant
    Demo --> Collector

Redis and Qdrant Server have deliberately separate jobs: Redis is for rate limits and budgets; Qdrant Server is for the semantic cache. The platform does not merge those responsibilities.

Development Runtime, Demo Gateway, and Production Gateway

These are separate ways to run the project. They are not interchangeable.

Runtime	Purpose	Providers and secrets	Local services	Default address
Development runtime	Local development and component exploration.	Injected provider doubles, mock Slack transport, and safe placeholders in .env.local. No real provider call.	Memurai at 127.0.0.1:6379 and Qdrant Embedded Mode under .dev/qdrant.	http://127.0.0.1:8000
Demo gateway	Safe Compose demonstration of the assembled production factory.	Demo-only mock provider and demo values from deploy/env/demo.env.	Compose Redis, Qdrant Server, Prometheus, Grafana, and OpenTelemetry Collector.	http://127.0.0.1:8001
Production gateway	Deployment path for real configuration.	Deployment-supplied provider credentials, Slack configuration, and team keys. Placeholders in deploy/env/production.env.example are not real production values.	Compose Redis, Qdrant Server, Prometheus, Grafana, and OpenTelemetry Collector.	http://127.0.0.1:8000

The development runtime is separate from the production image and production factory. The demo gateway uses the production image and factory but only the explicit demo Compose profile with a mock provider. The mock provider is not included in the production profile.

Running Locally

Option A: Development runtime

Use this option to explore the gateway without real provider credentials. It requires a Redis-compatible server such as the documented local Memurai instance listening on 127.0.0.1:6379. Qdrant runs in Embedded Mode for this runtime, so no Qdrant container is needed.

1. Create and activate a virtual environment in PowerShell:

   python -m venv .venv
   .\.venv\Scripts\Activate.ps1
   python -m pip install -r requirements-dev.txt
   

2. Create or keep an ignored .env.local in the repository root. The current development runtime supports safe values such as:

   TEAM_ALPHA_KEY=team-alpha-dev
   TEAM_BETA_KEY=team-beta-dev
   DEV_HOST=127.0.0.1
   DEV_PORT=8000
   REDIS_HOST=127.0.0.1
   REDIS_PORT=6379
   

3. Start the documented local Redis-compatible service, then run:

   python dev_runtime\run.py
   

4. Open Swagger UI or check health:

   Invoke-RestMethod http://127.0.0.1:8000/health
   

The development runtime has injected provider and Slack doubles. It is a safe local exercise path, not the production deployment.

Option B: Demo gateway with Docker Compose

Use this option to run the isolated demo profile. It builds the gateway image, starts the demo gateway and mock provider, and starts Redis, Qdrant Server, Prometheus, Grafana, and the OpenTelemetry Collector.

docker compose --profile demo build
docker compose --profile demo up -d --wait --wait-timeout 180

Useful addresses after startup:

• Demo gateway Swagger UI: http://127.0.0.1:8001/docs
• Demo gateway health: http://127.0.0.1:8001/health
• Demo gateway metrics: http://127.0.0.1:8001/metrics
• Prometheus: http://127.0.0.1:9090
• Grafana: http://127.0.0.1:3000

For Swagger, use Authorize and enter demo-team-key as the Bearer token. Then try this request body:

{
  "model": "demo-model",
  "max_tokens": 16,
  "messages": [
    {"role": "user", "content": "Explain semantic caching in one sentence."}
  ],
  "stream": false
}

Repeat the exact request to exercise the verified cache-hit path. In the recorded deployment validation, the first unique request used the mock provider and an identical repeat returned an ID beginning with cache-.

To stop the demo profile:

docker compose --profile demo down

This command stops containers but does not remove the named persistent volumes.

Production gateway

The production profile is intentionally not a no-configuration demo. Before using it, a deployer must copy deploy/env/production.env.example to a deployment-only environment file and replace every placeholder with real, approved deployment values. This includes provider credentials, the inbound team key referenced by TEAM_REGISTRY_JSON, and operational policy values.

The repository does not provide real credentials. Do not use the example file as if it were production-ready configuration.

Testing

Run the recorded regression suites from PowerShell:

.\.venv\Scripts\python.exe -m unittest discover -s tests\phase0 -p "test_*.py" -v
.\.venv\Scripts\python.exe -m unittest discover -s tests\unit -p "test_*.py" -v
.\.venv\Scripts\python.exe -m unittest discover -s tests\integration -p "test_*.py" -v

The host-Qdrant integration test is intentionally environment-gated and is not part of the default integration run. The Phase 13 report records that it was verified separately against a live Qdrant Server.

For a guided walkthrough, see the demo guide. For a source-bound technical handoff, see AI_MIDDLEWARE_PLATFORM_HANDOFF.md.

Verification Summary

The project state and Phase 13 verification report record these results:

• Latest regression: 3 Phase 0 tests, 75 unit tests, and 33 integration tests passed. The live-Qdrant host test is separately verified and skipped by default.
• Mixed-load validation: 5,000 concurrent gateway tasks across five team keys and two models, with exactly 2,500 rate-limit blocks.
• Priority validation: 5,000 Redis priority-queue items verified with real-time precedence and FIFO behavior.
• Cache-mix validation: 2,000 requests, 990 cache hits, and 9,900 configured microdollars avoided.
• Full FastAPI/ASGI middleware processing overhead: 0.6180 ms mean, 0.8419 ms P95, and 1.3167 ms P99. Redis, Qdrant, cache, provider-double, and end-to-end latency were separately measured.
• Docker Compose verification: production and demo gateways, Redis, Qdrant Server, Prometheus, Grafana, the OpenTelemetry Collector, and the demo mock provider were recorded healthy.
• Multi-worker startup verification: all 33 production workers booted and completed application startup without the prior Qdrant collection-creation failure.
• Grafana dashboard validation: the provisioned panels matched direct Prometheus values after controlled demo traffic. Cache hits, spend metrics, and rate-limit block metrics were verified after the Phase 13 corrective extension.

These are recorded validation results, not promises about a different machine, network, provider account, or future deployment.

Documentation

• Project handoff — source-bound architecture, deployment, validation, and open-item summary.
• Demo guide — safe walkthrough using the development runtime or demo gateway.
• Portfolio narrative — concise, evidence-based project framing.
• Phase mapping — relationship between the approved project phases and the original plan.
• Architecture decision records — documented implementation decisions.