THE SAMVIT LABS JOURNAL
All articles

From LLMs to SLMs: When a Smaller Model Is the Better System

How to choose a compact language model for a bounded business task, design a sensible fallback route and calculate the cost of getting an accepted result.

On this page

ABOUT THE EVIDENCEModel releases and methods are attributed to their authors. Routing costs are invented normalized units used to explain the calculation, not SAMVIT Labs pricing or measured customer results.

A service team needs to place incoming messages into twelve queues. Most messages are short. The output is a queue identifier, a few extracted fields and, when needed, a request for a person to review the case.

Should this use the largest available language model? The useful question is whether a smaller system can meet the acceptance criteria at a lower total cost. A compact model that completes this bounded task reliably may be a better component than a more capable model used without a clear reason.

That judgment needs evidence. A smaller parameter count is a starting point for an experiment, not proof of lower latency, lower energy or sufficient quality.

Give the model a narrow job

“Understand our business” is difficult to evaluate. “Classify this message into one of twelve approved queues, or return review-required” is much more concrete. The second task has a defined output, known failure cases and a decision that can be checked against a labelled dataset.

Small language model, or SLM, has no universal parameter boundary. Here it means a model compact enough to be a practical candidate for the intended deployment. State the actual parameter count and runtime rather than relying on the label.

SmolLM3 is one useful example of the direction: Hugging Face released a three-billion-parameter model with an open training recipe in July 2025. Its release describes both standard instruction following and an optional reasoning mode. Those published capabilities make it a candidate to evaluate; they do not establish accuracy on the twelve queues in our example. [1]

Start by writing ten examples where a wrong result would be costly. Include missing information, overlapping categories and requests outside the service team’s authority. These examples often reveal whether the task is truly narrow before a larger dataset is assembled.

Separate size from specialisation

A model can be smaller because it was trained at a smaller scale, compressed after training, or distilled from another model. These are different interventions.

Distillation trains a student using information from a teacher. Hinton, Vinyals and Dean’s original paper explains why a teacher’s output distribution can carry useful information beyond a single correct label. For a language workflow, teacher-generated examples still need a quality process: a fluent teacher can produce a wrong label or an invented explanation. [2]

Fine-tuning can instead teach a compact base model a stable output convention or task behaviour. It does not automatically provide current company policies. A routing model may need retrieved policy text at inference time even after adaptation.

Keep the evaluation set independent of teacher generation and training. Split related customer conversations and document templates together so that near-duplicates do not make the model appear more general than it is. Evaluate a later time period separately to see whether new terminology changes performance.

Build an acceptance gate outside the model

A JSON object can be syntactically valid and still point to the wrong queue. Validation therefore needs several layers: allowed schema, valid identifiers, evidence for extracted fields and a policy for ambiguous cases.

A useful output includes the source span supporting each important field. If the model claims that a customer requested cancellation, the application should be able to show the sentence that supports that interpretation. A missing span is a reason for review rather than a reason to invent an explanation.

Research on selective classification studies the trade-off between coverage and error when a system can abstain. The important distinction is that accepting fewer cases can change observed error among accepted cases; it does not make a language model’s self-reported confidence calibrated. [3]

For the queue example, a gate might reject messages containing two conflicting requests or an unknown product identifier. A separate, evaluated classifier could also estimate whether a result should be accepted. Test that gate on its own data. Do not treat a sentence such as “I am 98% confident” as a measured probability.

A small model feeds external schema, policy and evidence checks; rejected results go to a larger model or a person and are checked again.
Figure 1. A proposed routing architecture. The fallback uses the same acceptance criteria as the first attempt. This diagram describes a design, not a deployed customer system.

Count the cost of both attempts

Suppose a small-model attempt costs two normalized units and a large-model attempt costs twelve. The units could represent a consistently measured cost, but these values are invented. Let the fraction of requests sent to the large model be rr.

If the large attempt occurs after the small attempt, expected model cost is:

Ccascade=Cs+rCl=2+12rC_{\mathrm{cascade}}=C_s+rC_l=2+12r

At a 20% fallback rate, expected cost is 4.4 units. Against an all-large baseline of twelve, the reduction is about 63.3%. That is a property of the stated assumptions, not a benchmark result.

The model-only break-even point occurs when 2+12r=122+12r=12, giving r=0.8333r=0.8333. Additional gate computation, retrieval, human review and operational work can move that boundary. If the gate itself costs one unit per request, use 3+12r3+12r instead.

Illustrative expected cascade cost rises from 2 to 14 units as fallback rises from zero to all requests, crossing the 12-unit all-large baseline near 83 percent fallback.
Figure 2. Analytical cost sensitivity under the invented two-unit and twelve-unit assumptions. Every fallback pays for both attempts. Gate and human-review costs are excluded. Download the calculation data.

The denominator matters too. Cost per request can improve while cost per accepted result gets worse. If a deployment produces many unusable answers, divide total operating cost by the number of outcomes that pass the business rubric. Include review time rather than counting it as free.

Measure latency on the fallback path

A cascade often has two latency populations. Some requests complete after the first attempt; others wait for the gate and a second model. An acceptable average can hide a poor tail.

Record the time from request arrival to accepted result, with the route attached to each observation. Compare median and p95 latency for accepted first-pass results, fallback results and human reviews. These groups answer different operational questions.

Use the same hardware, context lengths, concurrency and output limits when comparing candidates. A small model producing a long explanation can take longer than a large model returning one validated identifier. Output design is part of the system being measured.

Choose with a task-specific scorecard

For a queue-routing pilot, the acceptance record could look like this:

DimensionWhat to recordWhy it matters
Task qualityCorrect queue and required fieldsMeasures the actual decision
Accepted coverageShare passing the external gateShows how much work stays on the compact route
Accepted error rateWrong accepted cases divided by accepted casesReveals mistakes hidden by abstention
Review workloadCases and minutes sent to peopleCaptures displaced work
Resource usePeak memory, latency and energy under loadTests deployment suitability
RecoveryCorrection and escalation outcomeTests what happens after failure

Keep results by queue, language and message length. A high aggregate score can conceal one category that repeatedly receives the wrong work. Report sample counts alongside percentages so that twenty perfect examples do not look equivalent to twenty thousand.

A sensible first decision may be a compact model for three well-defined queues and review for everything else. Expansion should follow measured coverage and acceptable errors. The engineering achievement is a dependable system for the intended work; the parameter count explains one part of how that system fits its budget.

References

  1. 1
    SmolLM3: smol, multilingual, long-context reasoner

    Elie Bakouch and colleagues · Hugging Face · 2025

    Back to text
  2. 2
    Distilling the Knowledge in a Neural Network

    Geoffrey Hinton, Oriol Vinyals and Jeff Dean · version 1 · 2015

    Back to text
  3. 3
    Selective Classification for Deep Neural Networks

    Yonatan Geifman and Ran El-Yaniv · version 1 · 2017

    Back to text

SAMVIT Labs Research

Research & engineering

The SAMVIT Labs research and engineering journal. We examine model efficiency, practical agent workflows and the evidence needed to make informed deployment decisions.

Figure view