The first thing teams instrument when they deploy an inference endpoint is GPU utilization and request count. Both are fine metrics. Neither tells you whether your routing layer is working well or whether your nodes are about to start misbehaving. This post is about the signals that actually matter for inference observability, and why most dashboards miss them.
We spend a significant amount of time looking at telemetry data from inference layers, specifically to build routing logic that responds intelligently to changing node conditions. That work has taught us which metrics predict problems before they become visible to users, and which ones look important but are mostly noise.
The Metrics That Are Overrated
GPU utilization is the most commonly cited inference health metric, and it is genuinely one of the least informative ones for latency prediction. A node can be at 70% GPU utilization and have perfectly healthy P99 latency, or it can be at 70% with severe queuing problems causing tail latency to spike. The utilization number does not distinguish these cases.
The reason is that GPU utilization measures compute throughput, not request queue state. When requests pile up faster than the GPU can process them, the queue grows, and latency increases. But the GPU itself may still be technically "busy" at a high utilization rate. You are measuring the engine temperature, not the queue at the entrance.
Similarly, total request count per node is a coarse signal. High request count on a fast, lightly loaded node is fine. The same request count on a node that is marginally slower or has a heavier in-flight queue is a different situation entirely.
Queue Depth: The Signal You Probably Are Not Tracking
Queue depth is the number of requests waiting to be processed at a node at any given moment. It is distinct from in-flight requests (those currently being processed). A queue depth of 0 means every request that arrived was immediately picked up. A queue depth of 8 means there are 8 requests waiting, and the next request routed to that node will wait behind all of them.
Queue depth is the earliest predictor of latency degradation that we have found. It reacts before throughput drops, before latency percentiles climb, and before utilization metrics signal anything unusual. When queue depth starts trending upward on a node, that node is about to have a latency problem. If your routing layer is watching queue depth in real time, it can start deprioritizing that node before the problem materializes in P99.
In Proximarun's routing engine, queue depth is one of the primary inputs to node scoring. We weight it heavily because historical latency measurements have a lag problem: they tell you how a node performed, not how it is likely to perform in the next 50 milliseconds. Queue depth is a live state signal, not a lagged historical one.
Per-Node P95 Divergence
Aggregate latency percentiles hide something important: the distribution of latency across nodes can diverge significantly even when aggregate numbers look healthy. If you have four nodes and one of them has a P95 that is 3x higher than the other three, your aggregate P95 looks merely elevated. The problem is concentrated on one node, and you would not know unless you are tracking per-node distributions.
We look at P95 rather than P99 for divergence detection because P99 is noisy at any individual node's request volume. Unless you are processing thousands of requests per node per minute, per-node P99 has high variance. P95 gives you a stable enough signal to detect meaningful divergence between nodes.
A practical threshold: if any single node's P95 is more than 2x the cluster median P95, something is different about that node. Common causes include: the node recently loaded a new model shard and is still warming up, the node is experiencing memory pressure from a batch of long-context requests, or there is a hardware issue starting to surface. In any case, the right response is to reduce routing weight to that node until it returns to normal, not to wait until it causes visible user-facing issues.
Cold-Start Frequency as a Health Signal
A cold start happens when a request arrives at a node that does not have the required model loaded in VRAM and must load it from storage first. Cold-start latency ranges from roughly 2 seconds for small models on fast NVMe to 8+ seconds for large models on slower storage. These events are easy to identify in traces because they appear as a distinct latency mode: a cluster of requests in the 2-8 second range that do not appear in normal operation.
Cold-start frequency tells you how well your node pool is sized and how well your warmup strategy is working. If you are seeing more than a handful of cold starts per hour on a node that should be handling steady traffic, something is wrong. Either the node is being allowed to evict models it should be keeping warm, or traffic is spiky enough that idle periods are leading to VRAM reclamation.
The metric to track is cold-start rate per node per hour, not just individual cold-start events. Trends matter more than snapshots. A node that went from 0 cold starts per hour to 12 per hour in the last 6 hours is telling you something changed.
Time-to-First-Token vs. Total Generation Latency
For streaming inference, the experience of latency splits into two phases: the wait until the first token appears (time-to-first-token, or TTFT), and then the ongoing generation speed until the last token. Users perceive these differently. A long TTFT feels like the system is unresponsive. Slow generation speed after the first token feels more like a typing animation. Both matter, but they have different root causes and different optimization handles.
TTFT is primarily sensitive to: prompt processing time (which scales with input length), queue wait at the node, and any cold-start overhead. Generation speed is primarily sensitive to: model size and hardware throughput, and batching behavior (whether the inference server is grouping your request with others).
If you are only measuring end-to-end latency, you cannot distinguish a TTFT problem from a generation throughput problem. Instrumenting them separately is worth the overhead. Proximarun tracks both in its node health metrics because routing decisions for latency-sensitive streaming workloads should weight TTFT heavily, not just total completion time.
What Good Observability Infrastructure Looks Like
We are not suggesting you build a custom telemetry platform. The instrumentation primitives you need are well-supported in standard tooling. The data model you want is roughly: per-node, per-request span data with TTFT, completion time, queue wait time, whether a cold start occurred, prompt token count, and output token count. From that span data, you can derive every aggregate metric we have described.
The observability gap we see most often is not missing raw data. It is missing per-node views. Most teams aggregate to the service level too early in their dashboard pipeline. The routing-relevant signals (queue depth divergence, per-node P95, cold-start rate) only exist in the per-node slice of the data. If you are pre-aggregating before you write to your metrics store, you are discarding the information you need.
The alerts worth setting: queue depth sustained above 5 on any node for more than 2 minutes, per-node P95 exceeding 2x cluster median for more than 5 minutes, cold-start rate above your baseline threshold. These are early-warning signals, not incident signals. By the time they turn into user complaints, you are in a reactive mode. The goal is to catch the trend before the P99 spike.