When a team sets up inference routing and we ask what latency budget they want to use, the most common answer is either "as fast as possible" or a round number like 500ms or 1000ms. Both of those answers create problems downstream. "As fast as possible" gives the routing layer nothing to optimize against. A round number chosen without reference to the underlying latency components usually turns out to be either too tight to be achievable or so loose that it provides no real constraint.
This post is about how to derive a routing latency budget that is actually grounded in user expectations and model characteristics, rather than one that sounds reasonable and then causes problems in production.
Starting From the User Expectation
The correct starting point for a latency budget is not the infrastructure layer. It is the user experience layer. What is the user doing when they wait for the inference result? Typing in a chat interface and waiting for a reply? Triggering a background summarization that they will check in a few seconds? Running a synchronous API call that blocks their application?
These have fundamentally different latency tolerance profiles. Synchronous user-facing chat has a cognitive threshold around 300-500ms for perceived responsiveness. Beyond that, users notice the delay and it starts to feel like the system is thinking rather than responding. For streaming responses that start rendering after the first tokens arrive, the time-to-first-token is the threshold that matters, and that is usually 200-400ms before users perceive lag.
Background or asynchronous inference can tolerate 1-3 seconds easily, sometimes much more. Batch document processing where the user has submitted a job and moved on has latency tolerance measured in tens of seconds.
Before you configure any latency budget in your routing layer, make sure you know which category your requests fall into. Treating a background summarization call with the same latency budget as a real-time chat call is wasteful. Treating a real-time chat call with background latency tolerance will surface as a product quality complaint within weeks.
The Latency Budget Decomposition
End-to-end latency for an inference request has four main components:
Network ingress: the time from the client sending the request to the routing layer receiving it. For co-located services this is under 5ms. For cross-region requests this can be 50-150ms.
Routing decision: the time the routing layer takes to score nodes and make a dispatch decision. In Proximarun this is typically under 12ms at median. We publish this because it is a real overhead and should be in your budget calculation.
Model inference: the time from the node receiving the request to generating the complete response. This is by far the most variable component. A 7B model on an uncontested A100 might take 80-120ms for a short prompt. A 70B model on a shared node at high load might take 800ms-1.5s for the same prompt. Context length, concurrent queue depth, and model size all contribute.
Network egress: response transmission back to the client. Usually 5-20ms for typical response sizes.
Your routing latency budget is not the end-to-end budget. It is the budget for what the routing layer controls: routing decision time plus the node's model inference time. Network ingress and egress are outside the routing layer's control.
Working Backward From the User SLA
Let's use a concrete example. You have a user-facing chat application. Your product target is that 99% of responses start rendering within 400ms. That is your user-facing P99 SLA.
Subtract network ingress (assume 20ms for a well-located deployment): 380ms remaining. Subtract routing decision overhead (12ms median, 20ms P99): 360ms remaining. Subtract network egress (15ms): 345ms remaining.
Your routing layer should be dispatching to nodes that can complete inference in 345ms at P99 for your request type distribution. That is the number you pass as latencyBudgetMs in the SDK call: not 400ms (the user SLA), but 345ms (the budget after subtracting the overhead you control and the overhead you do not).
If your models cannot hit 345ms at P99 under normal load, you have a capacity problem, not a routing problem. Routing can steer away from overloaded nodes, but it cannot make an undersized cluster faster. The budget calculation at least makes this visible.
The Model Latency Baseline Problem
Most teams do not have accurate per-request-type latency baselines when they start configuring routing. They know roughly how fast the model is in a low-load test environment. That number is often 2-3x better than what you see in production at median load, and 4-5x better than what you see at peak.
The test environment number is not a useful budget baseline. You want the latency at the load level you actually operate at. If you have production traffic, measure P50, P90, and P99 latency at median production concurrency. If you are pre-production, run a load test at the concurrency level you expect and measure from that. Do not budget from unloaded benchmarks.
The token count variability matters here too. Latency scales with output token count. A request that generates 50 tokens completes in roughly a third of the time as a request that generates 150 tokens on the same model. If your prompt structure allows variable-length outputs, your latency distribution will be wider than a benchmark with fixed-length outputs suggests. Budget for the actual distribution, not an average.
Per-Request Budget vs. Global Budget
A common mistake is setting a single global latency budget for all requests rather than per-request budgets that reflect the actual latency sensitivity of each request type.
Consider a team that has two request types on the same routing pool: real-time chat (needs 350ms budget) and background document summarization (tolerates 2000ms budget). If you configure a global 350ms budget, your routing layer will treat the summarization calls as just as latency-sensitive as the chat calls, routing them to the fastest (and often most expensive) nodes. You will pay for performance you do not need on 40% of your traffic.
Proximarun lets you pass per-request latency budgets in the SDK call. The routing engine uses that value when scoring nodes for that specific request. Setting appropriate per-request budgets rather than a blanket global budget is one of the easiest routing optimizations available, and it has meaningful cost implications when your request mix includes both latency-sensitive and latency-tolerant workloads.
Adjusting Budgets Over Time
A latency budget set once is not a static configuration. As your traffic grows, as your models change, and as your infrastructure evolves, the budget that was appropriate six months ago may be wrong today. The calculation we described above should be revisited whenever any of the underlying components change: new model, new node tier, new deployment region, significant change in request volume or concurrency.
We are not saying you need to recalculate this monthly. We are saying that a budget configured at launch and never revisited is likely to be misaligned with reality by the time your product is a year old. The teams we see with the tightest alignment between their latency targets and actual user experience are the ones that revisit this calculation when something changes rather than treating it as a one-time setup task.
The routing layer is only as useful as the targets it is optimizing against. Give it accurate targets backed by real user expectation data and real latency component measurements, and it will make sensible tradeoffs. Give it a number chosen because it sounded right, and it will faithfully optimize toward a target that may have nothing to do with what your users actually experience.