Concepts

Understanding these six concepts gives you the mental model to configure Proximarun correctly and diagnose unexpected routing behavior in production. If a node is being skipped you did not expect to be skipped, or spill is firing more than you want, the answer is in one of these six areas.

Node Pool

A node pool is the set of inference backends Proximarun routes across. Each node in the pool has an endpoint URL, a backend type, a declared cost-per-token, a latency class, and a standby mode (hot or cold).

Hot standbys are polled every 5 seconds and are always eligible for routing. Cold standbys are brought online on demand and have a configurable warm-up delay before they become eligible. You would use cold standbys for expensive resources you only want active during peak demand.

# node pool structure
nodes:
  - id: string           # unique identifier
    endpoint: url       # inference API URL
    backend: type       # vllm | tgi | triton | openai-compat
    costPerToken: float  # USD per output token
    latencyClass: fast|standard|slow
    standby: hot|cold

Routing Policy

The routing policy defines default behavior for all requests. Individual SDK calls can override any policy parameter. Policy evaluation is deterministic: given the same node health state and the same request parameters, the same node will always win.

Policy weights control how the three scoring dimensions are balanced. The defaults (latency 0.5, cost 0.3, reliability 0.2) are appropriate for most latency-sensitive applications. For batch workloads, you might increase the cost weight and decrease the latency weight.

routing:
  defaultLatencyBudgetMs: 500
  defaultCostCeilingUsd: 0.005
  defaultReliabilityMin: 0.95
  weights:
    latency: 0.5
    cost: 0.3
    reliability: 0.2

SLA Targets

SLA targets are declared per-request. There are three parameters:

  • latencyBudgetMs: The maximum acceptable end-to-end latency in milliseconds. Nodes whose rolling p95 latency exceeds this value are excluded from consideration for this request.
  • costCeilingUsd: The maximum acceptable cost per request in USD. Calculated as costPerToken * estimatedOutputTokens. Nodes that would exceed this ceiling are excluded.
  • reliabilityMin: The minimum acceptable reliability score (0.0 to 1.0). Calculated from recent error rate and queue depth.

When no eligible node meets all three constraints, spill logic activates. Spill first relaxes the cost ceiling by one tier, then the reliability minimum, and finally the latency budget. If no node is available after all spill passes, the request fails with a 503 No Eligible Node error.

Traffic Pattern Learning

After 7 days of traffic, Proximarun begins adjusting routing weights based on observed request distribution. The model uses a sliding window of the last 500 requests per node with exponential decay on older observations (half-life: 48 hours).

The learning model operates within the constraints set by your routing policy. It cannot route to a node that fails a hard constraint (latency budget exceeded, cost ceiling exceeded). What it does is adjust the relative preference among nodes that all meet the constraints, based on which ones have historically performed best for your specific traffic pattern.

Learning can be disabled per-request with disableLearning: true in the request body, or globally in your routing config with patternLearning: false.

Health Check Cycle

Proximarun polls each hot-standby node's health endpoint every 5 seconds. A node is considered unhealthy if:

  • The health check HTTP request fails or returns a non-2xx status code;
  • The health check response indicates queue depth above the node's declared maximum;
  • The node's error rate over the last 60 seconds exceeds the reliabilityMin threshold in your policy.

Unhealthy nodes are removed from the routing pool immediately. They are re-added automatically when two consecutive health checks pass. There is no manual intervention required.

Telemetry Schema

Every routing decision produces a telemetry event. The schema is consistent across all event types:

{
  "eventId": "tel_01HX...",
  "requestId": "req_01HX...",
  "timestamp": "2026-07-04T10:22:05Z",
  "nodeSelected": "edge-us-west-01",
  "nodeScores": {
    "edge-us-west-01": 0.847,
    "cloud-gpu-a100": 0.621
  },
  "latencyMs": 187,
  "routingDecisionMs": 8,
  "costUsd": 0.00082,
  "spilled": false,
  "outcome": "dispatched"
}