From request to result, precisely routed.
Proximarun intercepts each inference call, queries the live node registry, scores healthy candidates against your latency budget and cost ceiling, dispatches to the winner, and writes a telemetry event. Five steps. The entire routing decision runs in memory and adds under 12ms to your request path.
What happens inside every routing decision.
SDK call with SLA parameters
Your application calls client.infer() and passes latencyBudgetMs and costCeilingUsd inline with the prompt. These parameters travel with the request through the entire routing decision. No separate config lookup is needed per call.
Registry query: which nodes are healthy?
The routing engine consults the node registry, which is continuously updated by background health checks every 5 seconds. Nodes that failed their last health check, are reporting error rates above threshold, or have queue depths above their declared limit are excluded from consideration before scoring begins.
Scoring pass: latency, cost, and load
Each healthy node is scored on three dimensions: its rolling p95 latency versus your declared budget, its cost-per-token versus your ceiling, and its current load factor. Nodes that fail any hard constraint are excluded. Remaining nodes are ranked by weighted sum. This pass runs entirely in memory against pre-computed rolling statistics and adds under 1ms to the routing decision.
Dispatch to the winning node
The highest-scoring node receives the request. The Proximarun client proxies the call transparently: request headers are passed through, the response body is forwarded unchanged, and streaming responses are supported. From your application's perspective, the call behaves identically to a direct API call, with added routing intelligence.
Response returned, telemetry written
After the response is returned to your application, the routing engine writes a telemetry event: which node was selected, the scores it received, the actual end-to-end latency, the estimated cost incurred, and whether any spill logic fired. This event is what feeds the traffic pattern learning model for future routing decisions.
The routing engine gets smarter with each request.
Initial routing is based on declared node characteristics and real-time health. After 7 days of traffic, Proximarun starts adjusting routing weights based on observed request distribution.
The learning model uses a sliding window of the last 500 requests per node, with exponential decay on older observations. This means recent performance has higher influence than historical performance, and sudden traffic pattern shifts are reflected in routing decisions within hours, not days.
What the model learns
Which nodes consistently hit their latency targets at which times of day. Which nodes perform better on long-context versus short-context requests. Which cost-per-token ratios hold up under production load versus synthetic benchmarks.
What it does not do
The learning model does not override hard constraints. If a node's current p95 exceeds your declared budget, it is excluded regardless of how well it performed last week. Learning adjusts relative preference among the nodes that currently meet your constraints. It is not a black box that silently repriotizes your pool. You can disable it per-request with disableLearning: true or globally in your routing config to return to pure declarative scoring.
# node selection distribution
edge-us-west-01: 34%
cloud-gpu-a100: 33%
cloud-fallback: 33%
# round-robin-like, no preference# node selection distribution
edge-us-west-01: 61%
cloud-gpu-a100: 31%
cloud-fallback: 8%
# edge preferred, consistently
# beats p95 budget by 22ms avgWhen Proximarun prefers edge, and when it prefers cloud.
Edge nodes are physically closer to your users or your application servers, which reduces network round-trip time. They tend to have smaller GPU capacity, which makes them better for short-context, low-latency tasks and worse for long-context or batch workloads.
Prefer edge when:
Latency budget is tight (under 200ms), request context is short (under 4K tokens), and the request is time-sensitive. Edge nodes also perform better on streaming responses where first-token latency matters more than throughput.
Prefer cloud when:
Context length exceeds edge node capacity, batch throughput matters more than per-request latency, or edge nodes are at capacity and cloud nodes are available within budget.
Proximarun evaluates both options on every request using live health data. The cost ceiling and latency budget you declare do the disambiguation automatically without you needing to write routing rules.
See it work in your environment.
The quickstart takes 15 minutes. Connect your first node pool and run a routed inference call.