Open source · Go · MIT

Governor: a hard spending cap for LLM calls

A lightweight Go gateway between your app and the model provider. It reserves cost before a request goes out, meters every streamed chunk, and cancels the upstream request the moment the cap would be crossed.

Try it against the mock provider, no API keys needed:

go run ./cmd/mockprovider &

go run ./cmd/governor

curl -X POST localhost:8080 -d '{"model":"fake","messages":[{"role":"user","content":"hi"}]}'

The problem

LLM spend is invisible until the invoice arrives. Most tools report cost after the fact, and the ones that enforce limits are heavyweight proxies to run. Governor answers one question in one binary: can this request afford to run, and should this stream keep going?

The original hand sketch: App, Governor gateway with cost and cap, and the providers — Where it started: the first sketch, with the question “cap check without added latency?”

How it works

Pre-flight: an estimated cost is atomically reserved against the cap before the provider is called. If that alone would exceed it, the request gets a 429 and no tokens are bought.
Per-chunk metering: each streamed chunk's real cost is reserved against the same cap as soon as it's known.
Upstream cancellation: when a reservation is denied, Governor tears down the provider request itself, not just the stream to the client, so generation actually stops.
Reconciliation: when a stream ends, estimated and actual cost are compared and any unused reservation is refunded.
Storage: an in-memory store by default (a mutex-guarded counter, zero infrastructure); Redis is opt-in for multiple replicas or a cap that survives restarts.

Proof

Benchmark: reserving budget

Store	Time per reserve
In-memory (default)	≈ 25 ns/op
Redis (opt-in)	≈ 96 µs/op

An in-process mutex vs a network round trip to a local Redis, on an 11th Gen Intel i7-1165G7. Reproduce with go test ./internal/budget/... -bench=BenchmarkReserve -run=^$.

Concurrency: no double spend

200 goroutines race for a cap that fits exactly 20 reservations, under Go's race detector. Exactly 20 get through, on both the in-memory and the Redis store.

Terminal: go test with -race passing TestReserve_AllowsUnderCap, DeniesOverCap and ConcurrentNoDoubleSpend for memory and Redis, then the benchmark at 25.28 ns/op in memory and 95,577 ns/op on Redis — Tests under -race, then the benchmark.

Mid-stream cutoff

A mock provider streams 30 chunks with a $0.06 cap. Governor forwards 7, then sends budget_exceeded and tears down the upstream connection: the result is Completed: false, CanceledByClient: true, and no connection is left open to the provider.

Terminal: the mock provider streams chunk-0 to chunk-6, then Governor returns finish_reason budget_exceeded, and a check shows no connection left to the mock provider — Chunks 0–6 go through; the 8th would cross the cap.

Honest limits

A provider that only reports usage on its final chunk (exactly how OpenAI's stream_options.include_usage behaves) makes true mid-stream cutoff impossible in principle: cost isn't knowable until the stream is already over. There, the cap is enforced by the pre-flight reservation, not live.
Anthropic's message_delta events report output tokens incrementally, so live cutoff works as designed there.
Restarts: the default in-memory store keeps spend in the process, so a restart resets it to zero. Use the Redis store for a cap that survives restarts or is shared by several replicas.
No time windows yet: a cap is a running total, with no daily or monthly reset.
Budgets are per key (an X-Governor-Key header), but every key gets the same cap value from GOVERNOR_CAP_DOLLARS.