The routing engine for production AI inference.
Proximarun sits between your application and your inference node pool. On every request, it scores each healthy node against your declared latency budget, cost ceiling, and minimum reliability threshold, then dispatches to the winner. The scoring pass runs entirely in memory and adds under 12ms of overhead at p99. You get the response unchanged; we write the telemetry.
Get Early AccessScoring algorithm: how a winner is picked in microseconds.
For each incoming inference request, Proximarun runs a scoring pass across all healthy nodes in your pool. Each node receives three sub-scores:
Latency score
Computed from rolling p95 latency over the last 500 requests to that node. If a node's observed p95 exceeds your declared latencyBudgetMs, its score drops to zero and it is excluded from consideration entirely.
Cost score
Each node in your pool carries a costPerToken value from your config. Nodes cheaper than your costCeilingUsd per estimated token output receive a higher cost score. Nodes that would bust the ceiling are excluded.
Reliability score
Based on recent error rates and current queue depth reported by the node's health endpoint. A node under heavy load or with elevated error rates gets a lower reliability score even if its latency numbers look acceptable on paper.
Weighted sum and dispatch
The three scores are combined with configurable weights (defaults: latency 0.5, cost 0.3, reliability 0.2). The node with the highest weighted sum receives the request. The scoring pass runs in the routing engine's hot path and adds under 12ms of overhead in median cases.
Define your pool once. Proximarun handles the rest.
Your node pool is declared in a single YAML config. Each entry specifies the endpoint, backend type, cost-per-token, and latency class. Proximarun reads this at startup and polls health endpoints every 5 seconds.
Hot standbys are polled continuously and are immediately eligible for routing. Cold standbys are brought online only when primary nodes are unavailable, adding a warm-up cost you define per node.
# node pool definition
nodes:
- id: edge-us-west-01
endpoint: https://edge-01.internal/v1
backend: vllm
costPerToken: 0.0000012
latencyClass: fast
standby: hot
- id: cloud-gpu-a100
endpoint: https://cloud.provider/v1
backend: openai-compat
costPerToken: 0.0000048
latencyClass: standard
standby: hot
- id: cloud-fallback
endpoint: https://api.openai.com/v1
backend: openai-compat
costPerToken: 0.000015
latencyClass: standard
standby: cold
warmupMs: 0Per-request targets override global defaults. Always.
Your global policy sets the defaults for all traffic. Any individual SDK call can override these with tighter or looser targets. Spill logic kicks in when no node in the primary pool meets the combined constraints.
Every routing decision is a data point.
Most inference stacks can tell you how many requests they served and at what aggregate latency. Proximarun can tell you which node served each request, what that node scored against every other candidate, the actual latency and cost incurred, and whether spill logic fired. The full scoring context is logged on every dispatch, not just the outcome.
Prometheus metrics
Scrape /metrics to get per-node latency histograms (p50, p95, p99), request counts broken down by outcome (dispatched, spilled, failed), and running cost totals per node. These are the metrics you actually need to evaluate whether your routing policy is working as intended.
Webhook events
Configure a webhook endpoint to receive a JSON payload after each routing decision. The payload includes node scores, request parameters, and outcome fields. Use it to push data to your own data warehouse, drive custom cost dashboards, or trigger alerting when spill frequency rises above a threshold.
# HELP proxima_routing_latency_ms
# TYPE proxima_routing_latency_ms histogram
proxima_routing_latency_ms_bucket{node="edge-us-west-01",le="10"} 4812
proxima_routing_latency_ms_bucket{node="edge-us-west-01",le="50"} 9941
# HELP proxima_node_cost_usd_total
# TYPE proxima_node_cost_usd_total counter
proxima_node_cost_usd_total{node="edge-us-west-01"} 1.4821
proxima_node_cost_usd_total{node="cloud-gpu-a100"} 0.3912
# HELP proxima_routing_decisions_total
# TYPE proxima_routing_decisions_total counter
proxima_routing_decisions_total{outcome="dispatched"} 98241
proxima_routing_decisions_total{outcome="spilled"} 312
proxima_routing_decisions_total{outcome="failed"} 7Ready to route?
Get early access and connect your first node pool in under 15 minutes.