Every time Proximarun routes an inference request, the routing engine scores available nodes and picks the one most likely to meet the request's latency and cost targets. The scoring function is not opaque to us or to our users: it is a weighted combination of four signals, and the weights are configurable. This post explains what each signal captures, why we chose these four, and how the scoring interacts with real-world edge deployment scenarios.
Routing decisions happen at sub-millisecond speed because the evaluation happens in process, not over a network. But fast is not useful if the decision logic is wrong. Getting the signals right took considerably more time than making the scoring fast.
The Four Scoring Signals
1. Latency Estimate
The latency estimate for a node is a rolling window prediction, not a fixed value. It is computed from the exponentially weighted moving average of recent request completion times on that node, adjusted upward for current queue depth. The formula approximates: estimated_latency = ewma_completion_ms + (queue_depth * ewma_completion_ms * queue_pressure_factor).
The queue pressure factor is the part that required the most tuning. Simple models assume linear queuing delay, but inference workloads are not linear: a queue of 3 requests might add 40% latency overhead, while a queue of 10 might add 200% because you are also hitting memory pressure effects. We use an empirically fit curve rather than a linear coefficient. It differs by node type and model size, which is why the routing engine maintains separate pressure curves per node profile.
Latency estimate by default gets the highest weight in the score. If a request has a P99 latency budget configured, the scoring function converts that budget into a maximum acceptable latency estimate threshold and hard-excludes nodes above it before scoring begins.
2. Current Load Factor
Load factor measures how close a node is to its practical capacity ceiling. It is distinct from GPU utilization. We define load factor as the ratio of active concurrent requests to the node's estimated comfortable concurrency limit, which is determined during node registration and periodically validated by the routing engine's probe requests.
A node with a load factor of 0.4 has significant headroom. A node at 0.85 is approaching the zone where latency starts degrading nonlinearly. A node above 1.0 (which happens when the concurrency limit estimate was too conservative) is overloaded and should receive zero new routing weight until load factor normalizes.
Load factor is a fast-reacting signal. It updates with every request dispatch and completion, whereas the latency estimate is a lagged average. Together they cover different time horizons: latency estimate reflects the last several minutes of behavior, load factor reflects right now.
3. Cost Weight
Different nodes have different per-inference costs. Cloud nodes may charge by the GPU-second, while edge nodes might be on a flat monthly rate or have a different cost structure entirely. The cost weight is a normalized cost-per-request estimate derived from each node's cost configuration.
By default, cost weight is given lower priority than latency estimate. The routing engine is a latency-first system: it will accept a more expensive node to meet a latency budget. But when multiple nodes are within acceptable latency range, cost becomes the tiebreaker. Teams that have a cost-first SLA mode can invert this weighting, accepting slightly higher latency variation in exchange for consistently routing to cheaper nodes.
We are not saying cost-first routing is wrong. It is the right default for batch workloads that do not have strict latency requirements. For interactive inference, latency-first with cost as a tiebreaker is what most teams want.
4. Reliability Score
The reliability score is a decay function over recent failure and degradation events on each node. A node that returned errors or timeouts in the last 10 minutes has a reduced reliability score. A node that has been error-free for 30+ minutes returns to baseline. Severe events (consecutive timeouts, repeated 5xx responses) cause the reliability score to drop faster and recover more slowly.
Reliability scoring is the safety mechanism. Even if a flaky node has a temporarily excellent latency estimate and low load factor (which can happen right after errors clear the queue), the reliability score prevents the routing engine from immediately sending heavy traffic back to it. The recovery ramp is intentional.
Combining the Signals
The final node score is a weighted linear combination: score = w1 * (1 / latency_estimate) + w2 * (1 - load_factor) + w3 * (1 / cost_weight) + w4 * reliability_score. Inversion of latency and cost converts them from "lower is better" to "higher is better" so all four signals point in the same direction. The weights w1 through w4 sum to 1.0 and are user-configurable per routing policy.
Default weights: w1 = 0.45, w2 = 0.25, w3 = 0.15, w4 = 0.15. These reflect latency primacy while giving load factor a meaningful voice, since it is the fastest signal for catching nodes that are about to degrade. Cost and reliability are correction terms rather than primary drivers in the default configuration.
A Concrete Routing Scenario
Consider a team running three nodes: two cloud instances (Node A, Node B) and one edge node closer to their end users (Node C). Node A is their primary cloud node. Node B is a backup. Node C is a cheaper edge node with lower capacity but much better network proximity.
At low traffic, Node C scores best because its proximity advantage makes its latency estimate favorable. Cost weight also favors it. As traffic ramps, Node C's load factor climbs faster than the cloud nodes because it has a lower concurrency limit. Once load factor on Node C passes 0.7, the scoring function starts shifting traffic to Node A. Node B stays as a low-weight option unless A becomes congested.
If Node A experiences a transient error burst (three timeouts in two minutes), its reliability score drops from 1.0 to approximately 0.35. The routing engine pivots traffic to Node B and partially back to Node C. As Node A's reliability score recovers over the next 20-30 minutes, traffic shifts back gradually. Users who were routed to Node B during the incident saw slightly higher latency but no errors. That is the intended behavior.
What the Scoring Does Not Solve
Node scoring does not solve the fundamental problem of having too few nodes for your traffic volume. If all three nodes are above 0.85 load factor, the routing engine picks the least-bad option, but it cannot manufacture capacity. Scoring is a resource allocation mechanism, not a scaling mechanism.
It also does not solve geo-routing optimally if all your nodes are in the same region. The latency estimate reflects node-side performance; it does not model network path from the user to the node. If you have geographically distributed users and all your nodes are in us-east-1, edge node scoring will not rescue you. That is a topology problem, not a scoring problem.
The scoring algorithm is a decision function that optimizes within the constraints of your node pool. The right framing is: given these nodes, what is the best assignment for this request? Not: what should my node pool look like? The second question is yours to answer. The first is what we solve.