The LLM proxy

What it carries

The proxy relays three APIs: Anthropic Messages, OpenAI Chat Completions and OpenAI Responses. Streams pass through as they arrive, and a request body keeps its original bytes unless an approved control changes it: a sanitization rule, a model substitution or an input limit.

Each connection names its upstream, how it is hosted (vendor-hosted, private cloud or on premises) and whether it fails closed, the default, or open. A connection can also point at an existing Azure API Management or Kong gateway.

Routing and failover

A route lists between 2 and 16 approved destinations and one of four strategies: priority failover, weighted, least latency or lowest cost. Lowest cost chooses by each connection’s approved prices, not by measured spend, and refuses a request whose cost it cannot bound.

Health is judged from live traffic. Timeouts, rate limits, server errors and interrupted responses count against a destination until it rests; client errors do not. A retry always goes to a different destination, and nothing is retried once response headers have been sent. A model marked internal-only never falls back to an external one. The receipt names the strategy, the destination chosen and every destination tried or skipped, with the reason.

Substitution serves requests for one approved model with one cheaper approved model, for an organization, a team or an application. It needs a registered comparison of exactly that pair and two people’s approval, and the receipt keeps both the requested and the served model.

Keys

Agents and developers present virtual keys, each bound to one enrolled device or agent runner, set to expire, and stored only as a hash. The proxy removes the virtual key and adds the provider credential, which never leaves the gateway. The hub holds credentials only as references to a vault or the environment, resolved at the moment of use. A revoked key stops working on every replica within 60 seconds.

Tokens and dollars

Before dispatch, the proxy reserves the request’s worst case: its input and its declared output limit. After the response it settles at the provider’s reported usage, by class: fresh input, output, cache reads and cache writes. Missing usage is recorded as unknown, never as zero. Settlement can exceed the reservation; the overage is signed and the next reservation is refused. Admission is a test, not a ceiling.

Dollar limits need approved per-million-token rates on every connection. The proxy’s figures are gateway-measured; they appear beside provider-billed spend and are never added to it.

Data classes NEXT

Routing by data class, so that a request carrying a given class of data can reach only the models approved for it, is designed and not built. Today the proxy keeps internal-only models internal, labels each connection by how it is hosted, and blocks or redacts structured sensitive data on the request, as Sanitization and DLP for AI describes.

Compiling to the gateways you run

The same approved policy compiles into each supported gateway’s own configuration, is applied once after a second person approves it, and is read back on a schedule. A gateway that cannot express a strategy refuses it rather than approximating: Azure API Management takes priority and weighted pools, and advanced routing in Kong needs Kong Enterprise.

Latency

On loopback in our lab, with a stand-in upstream, the proxy added a median of 1.5 milliseconds across 60 sequential requests. That excludes the network and is not a production figure. The design budget is 15 milliseconds at the median, and a same-region measurement has not been run.

Where it stands

LIVE: in the released product and running on our hosted production deployment, with synthetic data. LAB: built or being built, and run only in our lab or against a vendor’s documented interface. NEXT: designed, not yet built. How we use these words.