Most teams start with one model. They pick the largest one they can afford to run, accept the latency and cost profile, and build their product around it. This works fine early on. As inference volume scales and cost-per-request becomes something the finance team asks about in planning meetings, it becomes worth examining whether every request actually needed the largest model.
Multi-model routing is the practice of maintaining a fleet with multiple model sizes and routing each request to the smallest model that can handle it acceptably. Done well, it can cut inference costs significantly while keeping quality acceptable for the majority of requests. Done poorly, it can quietly degrade output quality in ways that are hard to detect without structured evaluation.
This is not a feature we recommend to everyone immediately. It adds routing complexity and requires a clear understanding of your request distribution. But for teams with volume and diverse request types, it is one of the highest-leverage optimizations available.
The Case for Model Size Diversity
The capability gap between a 7B and 70B model varies dramatically by task type. For tasks like extracting a structured field from a semi-structured document, summarizing a product description into one sentence, or classifying an input into one of five predefined categories, the 7B model is often indistinguishable from the 70B model in terms of output quality. The task is well-defined, short-form, and within the smaller model's capability range.
For tasks like multi-step reasoning, long-form generation with coherent argument structure, or complex code synthesis from ambiguous natural-language specifications, the capability gap between 7B and 70B is significant and user-facing. Routing these to a smaller model produces noticeably worse output.
If your application mixes these two request types, routing everything to the 70B model means you are paying 70B inference cost for tasks that a 7B model handles just as well. The cost differential between these two model sizes at the same hardware level is roughly 4-6x in throughput terms. A request that costs $0.02 to serve on a 70B model might cost $0.004 on a 7B model with equivalent output quality for that task type.
How to Classify Request Complexity
The classification step is where most multi-model routing implementations either work well or fail. You need a signal that predicts whether a given request needs the larger model, without running the request through the larger model to find out (which would defeat the purpose).
The approaches that work reasonably well in practice: prompt-based classification using a lightweight classifier model (which itself should be very fast and cheap), explicit application-layer routing where your code marks requests with a complexity hint before sending them to the inference API, and heuristic rules based on input length and task type keywords.
Proximarun supports all three. The most reliable method for production use is application-layer routing: your code knows whether a given call is a simple extraction or a complex generation, and it can tag the request accordingly. The routing layer then uses that tag to select the appropriate model tier. This puts the classification decision where the context lives.
Heuristic rules are a decent fallback when application-layer tagging is not feasible. A rule like "if prompt contains instruction for multi-step reasoning, route to large model; otherwise route to small model" is imprecise but captures a useful signal. The failure mode is over-routing to the large model (wasting cost) rather than under-routing to the small model (degrading quality), which is the safer direction to err.
Fallback Routing: What Happens When the Small Model Is Not Enough
One pattern that reduces the quality risk of multi-model routing is fallback routing: send requests to the small model by default, and if the small model's output does not meet a confidence or quality threshold, re-route to the large model and return that output instead.
This requires a scoring mechanism for the small model's output. For structured outputs (JSON extraction, classification), the quality signal is often inherent: did the model return valid JSON in the expected schema? Did it return one of the allowed classification labels? For free-form generation, you need a lightweight judge model or heuristic quality check.
Fallback routing has a cost: the total latency for a fallback request is the sum of the small model call and the large model call, which is worse than going directly to the large model. For latency-sensitive applications, the fallback path may be unacceptable. For asynchronous or batch workloads, the latency overhead of fallback is usually acceptable if it reduces cost on the majority of requests that do not fall back.
We recommend fallback routing only for workloads where some latency variance is tolerable. For interactive real-time applications, explicit classification at the application layer is preferable because you avoid the latency penalty on fallback paths entirely.
Managing Model Warm State Across Multiple Tiers
Multi-model routing introduces a complexity that single-model deployments do not have: you need multiple model sizes warm and ready simultaneously. If your traffic is 80% small-model requests and 20% large-model requests, you may end up in a state where your large model nodes go cold during extended low-traffic windows, triggering cold starts when the next large-model request arrives.
Proximarun handles this by applying keepalive probing separately to each model tier. The probe rate for the large model tier can be configured lower than for the small model tier (since the cost of probing a 70B model is higher), but it should not be zero if you need that tier consistently warm. The cold-start penalty for large models is also proportionally larger: loading a 70B model in fp16 from NVMe storage takes considerably longer than loading a 7B model.
The operational implication: if you add large model tiers to your routing pool, budget for their ongoing keepalive cost even during low-traffic periods. If the cost of maintaining warmth is higher than the value of having the tier available, consider a mixed strategy where small model nodes stay always-warm and large model nodes are allowed to go cold with a pre-warm sequence triggered by traffic prediction.
When Multi-Model Routing Is Not Worth It
We are not arguing that every team should implement multi-model routing. The complexity cost is real. Maintaining two model tiers means maintaining two serving configurations, monitoring two latency distributions, debugging two cold-start scenarios, and explaining to your team why a given request went to one model and not the other.
If your request distribution is homogeneous (all requests are similar complexity), the classification overhead may outweigh the savings. If you are at low inference volume where per-request cost is not a material concern, the effort is not justified yet. If your application requires consistent output quality across all requests and you do not have a reliable classification mechanism, the quality risk may not be worth the cost savings.
The right time to consider multi-model routing is when you can identify request categories that are reliably distinct in complexity, when you have volume where the savings are meaningful, and when you have the engineering bandwidth to implement and monitor the classification logic correctly. All three conditions should hold before you add this complexity to your stack.