Engineering Notes
Technical writing on inference routing, node placement, and GPU cost engineering from the team building Proximarun. We write what we wished we could find when we were operating inference stacks ourselves.
Why P99 Latency Is the Only Number That Matters for Production Inference
Mean latency is the number that looks good in dashboards and tells you almost nothing about what your slowest users experience. Here is why we route against p99 targets, not averages, and what that means for how you configure latency budgets in Proximarun.
Instrumenting Your Inference Layer: What to Measure and Why
GPU utilization and call count are the metrics every dashboard shows. The signals that actually predict routing failures before they surface as user-visible latency are queue depth per node, error rate over the last 60 seconds, and p95 delta between nodes in the same pool.
Read more
How We Pick an Edge Node: The Scoring Algorithm Under the Hood
A walkthrough of the three-dimensional scoring function Proximarun uses to pick a node: rolling p95 latency, cost-per-token, and current reliability score. How the weights are configured, how ties break, and why we chose this model over simpler approaches like least-connections or random-with-retry.
Read more
Rules vs. Learned Policies: When to Use Each for Inference Routing
Static routing rules are predictable and debuggable. ML-based policies adapt but can surprise you.
Read more
The Cold Start Problem in AI Inference (And Three Ways to Mitigate It)
Cold starts add 2-8 seconds to inference latency. Here is what Proximarun does to avoid routing requests to cold nodes.
Read more
Defining SLA Guarantees for AI Inference Without Lying to Yourself
SLA commitments for inference workloads are tricky. Here is a framework for setting realistic targets.
Read more
Routing Across Multiple Models: When to Pick the Smaller One
Not every request needs your largest model. Routing by task complexity can cut costs by 60% with negligible quality loss.
Read more
The Over-Provisioning Trap: How Teams End Up Paying for Idle GPUs
Teams size their clusters for peak load and pay for that capacity 24/7. We break down the real cost of static allocation.
Read more
Cost Per Token: The Metric Your Finance Team Will Eventually Ask About
As inference scales, cost-per-token becomes a KPI. We walk through how to calculate it and where routing decisions move the needle.
Read more
Building Infra Products as a Bootstrapped Team: What We Learned in Year One
Infrastructure products have long sales cycles and high trust bars. Here is what worked for us in year one.
Read more
Teaching the Router: How Traffic Pattern Learning Works in Practice
On day one, Proximarun routes based on declared node characteristics. After 7 days, it starts adjusting weights from observed traffic. This piece walks through the sliding window model, the 48-hour exponential decay, and the specific situations where learned routing outperforms static declaration.
Read more
Edge Inference vs. Cloud Inference: A Decision Framework for 2025
Edge nodes cut network RTT but cap out at smaller context windows. Cloud nodes handle 32K+ token requests but add 40-80ms round-trip. This is the decision framework we use to configure latency classes in a mixed pool, with worked examples at different p99 budgets.
Read more
Routing Latency Budgets: Setting Targets That Actually Match User Expectations
A 400ms P99 budget sounds fine until you realize your model takes 350ms on a good day.
Read more