Writing an SLA for a traditional API endpoint is straightforward. Response time is mostly a function of your server capacity and database query time. Those variables are well-understood and relatively stable under load. You measure P99 over 30 days, add some headroom, and publish a number.
Writing an SLA for an AI inference endpoint is harder. Latency depends on at least four variables that interact nonlinearly: prompt token count, output token count, model size, and concurrent load on the serving node. The same endpoint can return a response in 80ms for a two-sentence input or 4 seconds for a 2,000-token document summary request. Both are working correctly.
If you publish a single latency number without qualifying it, you are either setting a target you will miss for long requests, or setting a target that looks worse than necessary for short ones. Neither is honest, and customers who care about latency will figure out the mismatch quickly.
Why Single-Number SLAs Fail for Inference
The core issue is that inference latency is a function of work done, not just server resources. A database query with good indexes runs in roughly the same time regardless of the size of the table. An inference call scales roughly linearly with output token count (for autoregressive generation) and non-linearly with prompt length for attention-based models. The work done is variable, and the latency varies accordingly.
Single-number SLAs paper over this by either setting the target high enough to cover the worst realistic case (making you look slower than you are for typical requests), or setting it based on typical requests (leading to frequent violations for longer ones). Both are wrong, but teams do this because it is simple and because most monitoring tooling presents single numbers.
The honest version requires acknowledging that you are offering a family of SLA targets, not one. This is not unusual in the industry: database vendors routinely specify read latency at different data sizes, CDN providers specify cache hit vs. miss latency separately. Inference SLAs should follow the same pattern.
Request Classes as the Foundation
The framework we recommend starts by defining request classes based on prompt length and expected output length. A simple classification might look like: short (under 500 input tokens, under 200 output tokens), medium (500-2,000 input tokens, 200-800 output tokens), long (over 2,000 input tokens or over 800 output tokens).
For each class, you measure P99 latency separately and set the SLA target for that class independently. Short requests might have a P99 SLA of 300ms. Medium requests, 900ms. Long requests, 3 seconds. All three are achievable on the same infrastructure; they just reflect the actual work done.
This approach requires that your monitoring infrastructure can classify requests. If you are using Proximarun, request span data includes input and output token counts, so the classification happens automatically in your metrics pipeline. If you are aggregating by endpoint only, you will need to add token count fields to your spans before you can compute per-class latency distributions.
The Load-Dependent Floor
Even with per-class SLAs, you need to account for load-dependent degradation. At low concurrent load, a short request might complete in 90ms. At peak concurrent load with 8 requests in the node queue, that same short request might take 400ms because it is waiting behind 8 other in-flight requests. Your SLA target needs to account for this.
The honest way to handle load-dependent behavior is to specify your SLA at a defined load level, not unconditionally. Something like: "P99 latency for short requests is under 300ms at up to N concurrent requests per node." This is truthful because it acknowledges that above that concurrency level, the SLA may not hold. It is also actionable because it gives you a clear threshold for when to add capacity.
What you are not saying is that the SLA holds at unlimited load. No system has that guarantee. The important thing is that your SLA document does not imply it by omission.
Measuring What You Are Actually Promising
SLA monitoring for inference requires per-request measurements, not sampled measurements. If you sample 1% of requests for latency measurement, you will miss the distribution tails, including the cold-start events and the occasional long-context request that took much longer than expected. For SLA compliance, you need complete coverage.
The measurement window matters too. A 99.9% P99 SLA measured over a 30-day rolling window is a much weaker commitment than the same SLA measured over a 5-minute window. The 30-day window allows you to have dozens of bad days as long as the month averages out. Users who experienced those bad days do not care about your monthly average. Consider publishing SLA targets at both horizons: a monthly availability number and a rolling hourly P99 target that reflects what users actually experience in short windows.
Error Budget Thinking for Inference Teams
The error budget concept from site reliability engineering translates well to inference SLAs. If your P99 SLA is 300ms for short requests, measured over a 30-day window, then 1% of short requests are allowed to exceed 300ms before you are in violation. At 10,000 short requests per day, that is 100 per day or roughly 3,000 per month that can exceed the target.
Error budget tracking gives your team a concrete signal: are you spending budget faster than expected? If you are burning through your monthly latency budget in the first week, something needs to change before month-end creates an SLA credit situation. If you are under budget, you have room to take maintenance actions (node updates, model updates) that might temporarily increase tail latency.
The error budget framing also forces an honest conversation about what your SLA actually costs to maintain. Maintaining a 300ms P99 for short requests across all load levels might require a specific minimum node count to avoid queuing delays. That minimum node count has a cost. If you set the SLA target before doing that capacity math, you may be making a commitment your current infrastructure cannot honor. The math should come first.
What to Put in the Customer-Facing SLA Document
After going through this internally, what you publish to customers does not need to be exhaustive. A useful customer-facing inference SLA contains: two or three request class definitions with their latency targets, the load level at which those targets are measured, and a measurement methodology statement (rolling window duration, coverage percentage). That is enough for a sophisticated engineering customer to evaluate whether your SLA meets their requirements.
Avoid publishing a single number without qualification. Avoid publishing targets that are statistically impossible to validate because your monitoring samples too coarsely. And avoid promising a number you have not measured under realistic load conditions. An SLA that you cannot prove you are meeting is not an SLA. It is a marketing sentence.
The internal discipline of building honest SLAs also tends to improve your engineering decisions. When you have to back out your infrastructure requirements from a specific latency target at a specific load level, you start asking the right questions about capacity, node selection, and routing policy. The SLA stops being a number someone picked and becomes a design constraint the system is built around.