A cold start in AI inference happens when a request arrives at a node that does not currently have the required model loaded in GPU memory. The inference runtime must fetch the model weights from storage (NVMe SSD, network storage, or object storage depending on your setup), load them into VRAM, and then process the request. The result is a latency spike that looks nothing like normal inference latency: instead of 150-400ms, the user waits 2-8 seconds. Sometimes more.
Cold starts are particularly damaging because they create a bimodal latency distribution. Your P50 looks fine. Your P99 looks catastrophic. Users who trigger cold starts at random have a dramatically worse experience than the majority of users, and there is no obvious signal in aggregate metrics that shows why.
At Proximarun, cold-start avoidance is baked into routing decisions. Before we explore how, it is worth understanding exactly why cold starts happen and what the realistic mitigation options are.
Why Cold Starts Happen
The root cause is GPU memory scarcity. VRAM is the limiting resource: a node with 80GB of VRAM can hold a 13B parameter model in fp16 (roughly 26GB) with room for KV cache and activations, or a smaller 7B model with significantly more headroom. But if you are running multiple model variants and need to hot-swap between them, or if the inference runtime evicts an idle model to free memory for a different request type, the evicted model must reload on next use.
Idle-based eviction is the most common cold start source in production deployments with variable traffic. If a model sees no requests for 10-15 minutes, some inference runtimes will proactively evict it from VRAM. A request that arrives 20 minutes after the previous one will trigger a cold start even though the node itself is healthy and available.
Scale-out cold starts are a second category. When traffic spikes and your orchestration layer adds a new node instance to handle the load, that node starts cold: the model needs to load before it can serve requests. If your autoscaling policy does not pre-warm new nodes before adding them to the routing pool, the first requests they receive will be cold-start requests.
Mitigation 1: Routing-Layer Cold-Start Awareness
The routing layer is the first and most effective place to address cold starts, because it controls which node receives each request. If Proximarun can detect that a node is cold before routing to it, it can route that request to a warm node instead.
Detecting cold state from the routing layer requires a node warmth signal. We implement this as a per-node warm state flag that is updated based on: recent successful inference completions (warm), elapsed time since last completion (decay toward cold), and an explicit warmth heartbeat from nodes that support it. A node that has not served a request in 12 minutes without sending a warmth heartbeat is assumed to be cold until proven otherwise.
Practically: when Proximarun scores candidate nodes for a request, cold-suspected nodes receive a reliability score penalty that effectively removes them from consideration unless all other nodes are also cold or unavailable. The request goes to a warm node. The cold node stays off-rotation until it either receives a probe warmth confirmation or serves a request that comes back with normal latency (indicating it was actually warm all along).
This approach does not eliminate cold starts entirely. It avoids routing user requests to cold nodes. Cold starts can still happen during node initialization, and our probe mechanism causes a controlled cold start on a node that has been idle, but the probe happens at low priority without a real user waiting.
Mitigation 2: Keepalive Probing
The second mitigation is active warmth maintenance via scheduled probe requests. When a node transitions to idle (no real traffic for N minutes), Proximarun sends lightweight probe inference requests to it at a low but steady rate, just enough to prevent the inference runtime from evicting the model from VRAM.
Probe requests are minimal: they use a very short prompt, generate one token, and are marked as probe traffic so they do not affect routing metrics or billing. Their only purpose is to keep the inference runtime's LRU eviction from removing the model. For most runtimes, a probe once every 5-8 minutes is sufficient to prevent eviction.
The cost of keepalive probing is small but nonzero. You are running GPU compute on a node that has no real traffic, solely to maintain warmth. Whether this cost is worth it depends on your traffic pattern: if your idle node comes back to serving real traffic within the hour, the cost of probing is almost certainly less than the cost of routing cold-start latency to real users. If the node stays idle for 12+ hours, you may want to evict it from the routing pool entirely and let it go truly cold.
Proximarun's keepalive configuration lets you set both the probe interval and the maximum idle warmup duration. After that threshold, the node transitions to cold-pool status, receives no probes, and is re-initialized with a controlled warmup sequence before being added back to the active routing pool.
Mitigation 3: Pre-Warm Pools for Scale-Out Events
The third mitigation addresses scale-out cold starts specifically. When your traffic learning predicts an upcoming load increase (based on historical diurnal patterns or real-time trend detection), Proximarun can trigger pre-warm requests on standby nodes before they enter the active routing pool.
Pre-warming a node means: sending it a sequence of probe inference requests until it demonstrates warm-state latency, then adding it to the routing pool with a low initial weight that ramps up as it continues to perform at expected latency. The node never serves a cold-start request to a real user because it was never in the routing pool while cold.
This works well when your load patterns are predictable. Diurnal patterns with consistent ramp shapes are an ideal case: Proximarun detects the characteristic pre-ramp signal in your request rate and triggers pre-warm 10-15 minutes ahead of the predicted load increase. By the time the load ramp arrives, the standby node is warm and ready.
It works less well for sudden, unpredicted traffic spikes. If your traffic triples in 30 seconds because of an external event, the pre-warm system did not have warning, and scale-out cold starts are likely. The honest framing: pre-warming can eliminate cold starts for predictable traffic patterns and reduce them significantly for semi-predictable ones. It cannot eliminate them entirely for truly unpredictable spikes. That is a constraint of the mechanism, not a limitation we have solved.
When Cold Starts Are Acceptable
We are not arguing that cold starts are always an emergency to be eliminated at any cost. For batch inference workloads where a job runs overnight and latency for individual requests is irrelevant, cold starts are a non-issue. For internal tooling where a small team uses inference occasionally throughout the day, one cold start per user session is probably fine.
The cases where cold starts are genuinely a problem are interactive user-facing products: chatbots, copilots, real-time content generation. Anything where a human is waiting for the response and will notice a 4-second delay as a malfunction, not as expected behavior. For those products, cold-start avoidance is worth the engineering investment and the small ongoing cost of keepalive probing.
The routing layer is the right place to apply cold-start mitigation because it has the widest visibility. Individual nodes cannot solve this: they do not know whether they should pre-warm themselves or whether a warm sibling node is available to absorb the traffic. The orchestration layer, which sees the full node pool, can make the tradeoff intelligently.