Cost-efficient routing

Use lower-cost Flex capacity when it meets your latency needs.

Cost-efficient routing uses the Flex service tier to lower inference cost on eligible requests. It is on by default. You can turn it off from the Experiments page, or pass allow_flex_tier: false to disable it for a specific request.

{
  "model": "your-model-id",
  "input": "Summarize this document.",
  "allow_flex_tier": false
}

An explicit allow_flex_tier always wins over the account setting, so true opts a single request in while the toggle is off.

What Router chooses

Router keeps the same provider and model and only chooses between that model's standard and Flex service tiers. It sends a request to Flex when doing so is not expected to degrade it, based on how each tier has been performing recently. Flex costs less, but its latency is higher and less predictable.

If the selected tier fails or times out, Router can try the other tier.

Where Flex applies

Cost-efficient routing applies only when the request uses an eligible direct model, rather than a models list, and does not set service_tier.

Anything else runs unchanged. Sending allow_flex_tier explicitly where Flex is unavailable returns 400 rather than being ignored. See allow_flex_tier.

Set latency guardrails

Because Flex latency varies, bound it explicitly:

Request	Field	Bounds
Non-streaming	`provider_timeout`	Seconds the provider call may take
Streaming	`timeout_before_headers`	Seconds to wait for the provider to start producing output

See Timeouts for the full semantics. The request detail in Logs shows which service tier handled the request.