Most of your model calls don't need a frontier model

There's a version of this industry where 'better model' only ever means bigger, newer, top of the benchmarks. But better at what, exactly?

Here's the case for small models in application layer products with matured use cases.

Reason one: small models are often more accurate for the job

This is the part that gets buried under the cost conversation. On a narrow, well-scoped task, a small model tuned for that one job can outperform a frontier model.

• Capacity split. A frontier model is trained to be reasonably good at poetry, code, legal reasoning, and your classification task, all at once. None of that capacity is dedicated to your specific problem. A small model fine-tuned only on your domain isn't competing with anything else for the same weights.
• Instruction-tuning baggage. Frontier models are tuned to be broadly helpful, which shows up as hedging, caveats, unnecessary elaboration. That doesn't show up the same way in a small model trained tightly on your specific task, so it has less to unlearn.
• Failure modes you control. If a frontier model misreads your domain's terms or edge cases, your only lever is a longer prompt. A small model you fine-tune yourself, you retrain directly on the errors your users actually hit.
• Format reliability. Fine-tuning a small model to always return one exact schema holds up better at scale than prompting a frontier model to do the same.
• Version stability. Providers update frontier models on their own schedule, and your prompt's behavior can shift without warning. A model you host, you control the version indefinitely.

On a narrow, well-scoped task, it's common for the small model to just be the better one.

Reason two: it's dramatically cheaper, and cheap changes what you can build

Frontier model pricing runs per million tokens in a way that adds up fast once a step runs on every request. A well-run small model, self-hosted or through a provider built for it, costs a fraction of that, often 10-30x less on equivalent tasks.⁴

The bigger effect is on what becomes worth building. A marketplace might want every new listing checked for policy violations before it goes live, or a data team might want every incoming record checked for anomalies as it arrives, instead of catching it later in a nightly batch job. At frontier pricing, both of those are a real line item. At small-model pricing, they're just something you turn on.

Most of the product upside here comes from the freedom to make calls you wouldn't have been able to afford at frontier prices. That gap gives you room to experiment more, and that's usually what makes the product better.

Reason three: speed changes the feature, not just the response time

A classification step returning in 200ms instead of 2 seconds isn't a faster version of the same feature. It's a different feature. At 200ms it runs inline, in the same request-response cycle the user is already waiting on. At 2 seconds it moves behind a spinner or into a background job, and now you're also building a queue, a status endpoint, and a polling loop just to hide the wait.

Slow inference doesn't just delay the answer. It decides the architecture around the answer. Live validation as someone types, a recommendation that updates as inputs change, a check that runs before an action instead of after it, these tend to get built once the round trip is fast enough to sit inline, not before.

Look at what your calls are doing

Pull up your codebase and list every model call. Next to each one, write what it's being asked to do.

When I did this on a product I worked on, the list came out roughly:

• classify an event into one of nine categories
• extract three fields from a text blob
• decide if two records are the same record
• trim a sentence to fit a character limit
• check whether a response contained a number
• decide which of four follow-up questions to ask
• write the recommendation copy itself
• plan a multi-step investigation

The last two needed reasoning. The other six were pattern work, and all eight were going to a frontier model.

Most application layer products look like this. One or two hard steps, wrapped in a lot of classification, extraction, and routing, and every one of those narrow steps is a candidate for the small-model argument above.

Pick the model per step, not per product

Teams benchmark "our use case" as one blob, find the small model does worse, and stop there. What that measures is whether one model can cover ten different tasks at once, which was never really the question.

Split the steps out and the answer changes per row:

The test for any step: can you write a rubric for what a correct output looks like? If yes, a small model can hold it, and often hold it better. If you can't write the rubric, keep the frontier model there.

What counts as small

Parameter count is a bad definition. Use deployability: a model you can serve on one GPU, on-prem, or on-device. In practice that's a few hundred million up to about 20B, and the ceiling moves every few months.

What's in that range right now: Phi, Gemma 3n with its mobile-optimised encoders, Ministral 3 at 3B, SmolLM3 with a 64K trained context that extends to 128K via YaRN, Qwen2, Llama 3.2 at 1B and 3B.

Distillation and better post-training closed most of the gap. DeepSeek did this publicly with R1, distilling it into 7B and 8B versions that get close to the full 671B model's scores on focused reasoning benchmarks.¹ Microsoft's Phi-4 does something similar, matching models several times its size on math and logic tasks.² A well-tuned 7B from this year can hold up against much larger models from two years ago on narrow tasks. On open-ended work the gap is still wide.

Serving depends on where you're running:

• vLLM for throughput on your own GPU, with continuous batching, if you're handling real volume
• Ollama for local development and small internal deployments
• llama.cpp for edge and on-device, quantized down to 4-bit
• Together, Fireworks, Groq for small-model pricing without owning infrastructure

Quantization matters more than most teams expect. A 7B at 4-bit fits comfortably on consumer hardware, and quality loss on classification and extraction is small. On reasoning and math tasks it isn't, degradation there is well documented and can be substantial at low bit-widths.³

Route, don't replace

For most teams, implementing this doesn't mean swapping out the frontier model. It looks like a cascade.

The small model handles the request. An escalation check runs on its output. Anything that fails goes to the frontier model.

The escalation trigger is where the design work is. Options, roughly in order of how well they hold up:

1. Schema validation. The output has to parse into your expected structure. Cheapest and most reliable trigger available.
2. Two-pass disagreement. Run the small model twice at temperature and compare. Disagreement escalates. Costs 2x small-model calls, still far below one frontier call.
3. Out-of-distribution check. Input length, language, or domain outside what the small model was tuned on.
4. Logprob confidence. Works, but calibration on small models is unreliable enough that I'd use it as a tiebreak rather than a primary trigger.

Log every escalation. After a month you have an escalation rate per step, which tells you where the small model holds up and where to spend fine-tuning effort.

You need evals first

To move a step from a frontier model to a small one, you have to know whether that step got worse, or in a lot of cases, whether it got better. This is where strong evals come in.

Minimum version, per step:

• 50 to 200 labelled inputs with expected outputs
• a scoring function: exact match for classification, field-level F1 for extraction, an LLM judge with a rubric for anything generative
• a baseline run on your current model, saved

That's about a day of work. After that you can swap the model and roll back if it gets worse, and just as often you'll find it didn't.

Keep the frontier model when

Your product's core value is reasoning over inputs you can't anticipate. Your output has to read like a specific person wrote it. You're pre-PMF and inference architecture isn't where the next hour should go.

Any of those is a good reason to leave it alone.