There is a particular kind of graph that engineering teams love to show stakeholders: a smooth average latency line hovering around 180ms, looking stable and healthy over a 30-day window. Underneath that line, at the P99 tail, requests are taking 2.4 seconds. Nobody in that meeting sees those users. They have already closed the tab.
We built Proximarun specifically around tail latency as the primary optimization target. Not because average latency is unimportant, but because the distribution of inference latency has properties that make averages actively misleading for the problems we care about.
Why the Distribution Is Worse Than You Think
Standard HTTP request latency is roughly log-normal. You get a compact cluster of fast requests and a modest tail. AI inference is different in a few ways that compound the tail problem.
First, inference latency is heavily sensitive to prompt length and output token count. A request that generates 50 tokens has a fundamentally different latency profile than one generating 800 tokens, and both arrive on the same endpoint. Your average latency number mixes these together and tells you very little about either population.
Second, cold-start events create discrete latency modes. When a node has been idle and needs to load a model shard back into VRAM, you go from sub-200ms to potentially 3-8 seconds, depending on model size and storage speed. That step function doesn't average out gracefully. It shows up as occasional spikes in your tail that no amount of tuning the "normal" path can eliminate.
Third, GPU memory pressure causes non-linear degradation. When a node is running at 85-90% memory utilization, requests start hitting queuing delays that scale faster than linearly with concurrency. The P50 barely moves. The P99 climbs quickly.
None of these factors are visible in an average. All three directly affect user experience.
What P99 Actually Measures
P99 latency is the answer to: what does one in every hundred requests experience? For a product serving moderate traffic, that is not a rare event. If you process 50,000 inference calls per day, 500 of them are hitting your P99 threshold. Those 500 users may never tell you directly, but session abandonment rates and retry loops show up in downstream metrics eventually.
The number we pay attention to is not just the P99 value in isolation. It is the ratio of P99 to P50. A P99/P50 ratio below 3x suggests relatively compact tails. A ratio above 6x tells you there is a distinct failure mode somewhere in the distribution that you have not characterized. We have seen ratios above 10x on systems that looked fine on average dashboards.
When we are designing routing policies, P99 is the constraint. The question is: given this set of available nodes, how do we route the next request to minimize the probability that it lands in the tail?
How Routing Decisions Affect the Tail
Consider a concrete example. You have three available nodes. Node A has a P50 of 140ms and a P99 of 620ms. Node B has a P50 of 190ms and a P99 of 310ms. Node C has a P50 of 160ms and a P99 of 890ms.
If you route by lowest average latency, you send most traffic to Node A. If you route by P50, you still end up at Node A. But if your SLA says P99 must stay below 400ms, Node A and Node C are both violations waiting to happen. Node B, which looks slower on average, is actually the only compliant option.
This is the core tradeoff we work with constantly. Optimizing for mean can be directly opposed to optimizing for tail. In production, your users do not care about the mean. They care about what happens to their specific request.
The routing decision gets more complex when you factor in real-time queue depth. A node that has a healthy P99 under normal load may degrade quickly when concurrent requests pile up. We track per-node queue depth as a live signal, not just historical latency percentiles, because a node's current state matters more than its past performance for the next routing decision.
Setting P99 Budgets Before You Write Code
The teams that handle this well do the budget math before they start tuning. Start from the user-facing SLA. Say your product target is that 99% of AI responses appear within 600ms from the user's perspective.
Work backward: network round-trip to your closest edge point (call it 15ms each way), routing overhead (under 2ms with Proximarun), queue wait time at peak load (budget 30ms at P99), and the actual model inference time. That leaves you roughly 538ms for inference. But you cannot budget 538ms for a P99. You need headroom for variance. We typically recommend budgeting to 70-80% of the total inference allowance and treating the rest as variance buffer.
So your inference P99 target becomes around 380-400ms. Now you have a concrete constraint to hand to whoever is selecting models and deciding whether you can afford a 13B parameter model or need to stay with a 7B for this latency class of requests.
The Monitoring Setup That Actually Helps
We are not saying you should throw away average latency metrics. Averages are useful for capacity planning and cost estimation. The point is that averages should not be your primary alert signal or your primary optimization target.
For production inference, the monitoring setup we recommend to teams using Proximarun looks like this: alert on P95 crossing threshold (as an early warning), alert on P99 crossing threshold (as the primary SLA signal), and track P99/P50 ratio as a health indicator. When that ratio starts climbing, something in the distribution is changing. Investigate before it becomes an incident.
Per-node P99 is more important than aggregate P99. Aggregate P99 hides which node is responsible for the tail. If one of your three nodes is contributing 80% of tail latency events, you want to know that so you can route around it, deprioritize it, or investigate what is different about that node's workload.
We built per-node latency percentile tracking into Proximarun's routing engine precisely because aggregate metrics were insufficient to make good routing decisions. A request that could go to any of three nodes should go to the one whose current P99 profile best matches the latency budget for that request class.
When P99 Is Not Enough
There is a reasonable counterargument here: for some workloads, P99 is still too generous. If your product involves long-running batch inference where a single workflow waits for 50 sequential calls to complete, the tail of your end-to-end latency distribution is effectively your P99 raised to a power. P99 at each step of a 50-step chain produces a dramatically worse aggregate experience than the individual P99 would suggest.
For those workloads, you need to think about P99.9 or even P99.99. We have had that conversation with a handful of teams building agent-style orchestration where individual inference calls chain together. The math changes significantly, and the routing strategy needs to be correspondingly more conservative: prefer nodes with tighter distributions even if the mean is slightly higher, because distributional variance compounds across calls.
The right tail percentile to optimize for depends on your request structure. The principle does not change: pick the tail percentile that corresponds to the unit of user experience you are measuring, and design your routing policy around that constraint, not around what looks good in a weekly status update.