At some point in the lifecycle of a team running AI inference at meaningful scale, someone from finance walks into the room and asks what you are paying per token. If you do not have an answer, the conversation gets uncomfortable fast. If your answer is "it depends on which model," that is technically true but probably not going to satisfy the question.
Cost-per-token is not a natural metric for infrastructure teams. We think in GPU-hours, utilization percentages, P99 latency, request throughput. Tokens are a model-layer concept. But as inference becomes a budget line that executives actually review, cost-per-token is the unit that connects technical decisions to financial impact. So it is worth understanding how to calculate it properly and where your routing architecture can actually change the number.
What Cost Per Token Actually Measures
The formula is straightforward: total compute cost over a billing period divided by total tokens generated. The hard part is the numerator.
If you are using managed inference APIs, the cost is the API invoice. That is simple. If you are running your own nodes, you need to account for GPU reservation or spot costs, CPU and memory for the host, networking costs for large context windows, and any orchestration overhead. Teams that run hybrid setups with some managed API calls and some self-hosted inference often have cost-per-token that looks quite different depending on which path a request takes, and they frequently do not track those separately.
The denominator matters too. Are you counting input tokens, output tokens, or total? Output tokens are more expensive to generate (autoregressive decoding is sequential), so they should carry more weight in any cost model. If you are computing a single cost-per-token number, make sure you are consistent about which token count you are using, and document that definition. "Our cost is $X per token" means nothing without knowing whether that is output tokens, total tokens, or something else.
The Routing Levers That Move This Number
There are roughly three places where routing decisions affect cost-per-token at the infrastructure layer, independent of model selection or prompt engineering.
The first is idle compute. We have written about over-provisioning separately, but the impact on cost-per-token is direct: if your cluster is running at 35% average utilization, your effective cost-per-token is about 2.9x what it would be at full utilization, because you are paying for capacity that is not producing tokens. Dynamic routing that tightens utilization directly lowers this denominator-to-numerator ratio.
The second is node tier matching. Not all tokens cost the same to generate on all hardware. A 7B parameter model running on a mid-tier GPU produces tokens at lower cost than a 70B model on an H100, and for many request types the quality difference is negligible. If your routing sends every request to the same node pool regardless of request complexity, you are paying 70B prices for work that a 7B model could handle. Routing that dispatches based on request characteristics can capture that cost delta.
The third is thermal efficiency within a node. This is subtle but real: a GPU that has been idle is slower for the first few seconds of a new batch than one that is in a steady generation state. Cold starts waste compute per token because the GPU is not fully engaged during the warmup phase. Routing policies that maintain a minimal steady throughput on warm nodes rather than letting nodes go completely cold and then spiking reduce this waste per token.
What Routing Cannot Fix
We should be direct: routing is an infrastructure concern, not a model concern. Routing can improve utilization efficiency, match request types to appropriate hardware tiers, and reduce cold-start waste. It cannot fix a fundamentally expensive model choice, a context window that is 10x larger than the use case requires, or a prompt engineering pattern that generates excessive output tokens.
If your cost-per-token is high because you are using a 70B model for tasks that a well-prompted 13B model would handle, routing will not close that gap. The model selection and prompt design layers are upstream of routing. Teams sometimes arrive at Proximarun expecting routing to solve a model selection problem. It will not. Those are separate levers that need separate work.
That said, routing infrastructure does create the observability foundation that makes model selection decisions tractable. When every routing decision is logged with the latency achieved, the cost incurred, and the model used, you can start to answer questions like: "What percentage of our 70B traffic could have been served by the 13B model at the observed quality bar?" That analysis requires the telemetry that a routing layer provides. Without it, you are guessing.
Building the Metric Into Your Tracking
To actually track cost-per-token, you need three things in your telemetry: total tokens generated per request (input and output separately if possible), which node or endpoint handled the request, and the cost basis for that node type.
The node cost basis is where teams get stuck. For managed APIs, use the published per-token price. For self-hosted nodes, you need an amortized cost-per-token estimate based on your actual GPU cost and observed throughput at normal utilization. A reasonable approximation: take the hourly node cost, divide by the average tokens-per-hour you observe at typical load. Update that estimate monthly as your traffic patterns change.
With those three telemetry fields, you can compute a running cost-per-token in your observability stack, slice it by model, by node type, by request category, or by time of day. That view is what makes it possible to have a coherent conversation with a finance team about what you are spending and why.
A Practical Benchmark Range
For context on what is realistic: teams running self-hosted 7B models on mid-tier GPUs at good utilization typically land in the range of $0.0001 to $0.0003 per output token. Teams running 70B models on H100s at lower utilization can be 10-20x higher before any optimization work. Managed API pricing for frontier models tends to run $0.002 to $0.015 per output token depending on the provider and model size.
We are not saying any of those numbers are good or bad in isolation. Whether a given cost-per-token is acceptable depends entirely on what revenue or value each token is generating. The point is that having the number gives you the ability to make that judgment. Without it, the question "are we spending too much on inference?" has no answer.
Once you have a per-token cost tracked as a real metric, the conversation shifts from "inference is expensive" to "here is what we are paying, here are the two routing changes that would reduce it by 25%, and here is the quality tradeoff." That is a solvable engineering problem. The first version is just a complaint.