Most teams pick one model and send it everything. That is the expensive mistake. The question is not "which model should we use?" It is "which parts of this job actually need the big model?"
In most real pipelines, only a small share of the work needs frontier reasoning. The rest is chores: classify a ticket, extract fields from a document, draft a first summary, triage a request, check a format. A small model can do those chores well. Reserve the large model for the final answer, where judgment, synthesis, and edge cases live.
This is a cost decision, not a quality slogan. Route by task, and measure cost per completed task — not per request, and not per token in isolation.
Split the job before you pick the model
Take a typical support or document workflow and split it into three buckets:
- Chores (small model): classification, field extraction, retrieval, routine summarization, formatting, first-pass drafting.
- Checks (rules plus a small model): schema validation, format checks, citation presence, obvious contradictions. Deterministic checks cost almost nothing and catch the errors that are cheapest to catch.
- Final answer (large model): synthesis across sources, ambiguous cases, decisions that are hard to reverse, and anything where a wrong answer is expensive.
The point of the split is that the large model never sees the raw, unfiltered workload. It sees a cleaned-up task with the relevant context already extracted. That also makes its output easier to audit: you can see what the small model handed over, and what the large model decided.
The cost math, with prices checked today
These are public GPU listing prices I re-checked on November 3, 2026 — not a production benchmark, and not a promise about your throughput:
- RunPod Community Cloud: RTX 4090 at $0.34 per GPU-hour, A100 SXM (80 GB) at $1.39, H100 SXM at $2.69.
- Lambda on-demand instances: H100 (80 GB) at $3.99 per GPU-hour, A100 (80 GB) at $2.79.
Now do the arithmetic on GPU-hours, because that is the unit you actually pay for when you self-host. Suppose a day's pipeline needs 9 GPU-hours in total, and the split is 8 hours of chores plus 1 hour of final answers:
- All-large on a RunPod H100: 9 × $2.69 = $24.21
- Split — chores on a 4090, final answers on an H100: (8 × $0.34) + (1 × $2.69) = $2.72 + $2.69 = $5.41
That is the 8:1 case, and I am labeling it as such. If your final-answer share is larger, the saving shrinks, and you should know by how much. At a 50/50 split (4.5 hours each):
- Split: (4.5 × $0.34) + (4.5 × $2.69) = $1.53 + $12.11 = $13.64, against $24.21 all-large.
The lesson is not "you will save 78%." The lesson is that your saving is driven by one number: what fraction of GPU-hours genuinely needs the large model. Measure that fraction on your own workload before you commit to a routing design. If you cannot measure it, you are guessing, and GPU bills punish guesses.
A note on the 4090 side of that math, from my own earlier sizing work: a 7B-class model fits on a 24 GB 4090 for inference, but a long context window changes the story because the KV cache grows with context. In an earlier post I showed how quantizing the KV cache lets a 128K context fit on a 4090. The same logic applies here: a "small model" chore is only cheap if you have also sized its context. A chore with a huge context can quietly become a large-model-sized job.
The redo rate is where split designs die
The split only saves money if the small model's work usually stands. Every chore that has to be redone by the large model costs you twice: the small-model hour plus the large-model hour.
Use this formula before you celebrate a saving:
effective cost = small-model cost + (redo rate × large-model cost) + final-answer cost
If a third of your extracted fields come back wrong and need a large-model redo, your "cheap" chores are no longer cheap. Track three numbers per task type, not per request:
- Success rate: the share of tasks completed without a redo.
- Redo rate: the share that fall back to the large model.
- Cost per completed task: total GPU cost divided by tasks that finished successfully, redos included.
Cost per request hides redos. Cost per completed task exposes them. If a routing change lowers cost per request but raises cost per completed task, it made things worse.
Guardrails that keep the small model honest
A small model fails differently from a large one: it is more likely to be confidently wrong on an edge case, and less likely to flag its own uncertainty. Do not fix that by sending everything to the large model for verification — that just re-creates the all-large bill. Instead:
- Route on task type, not on vibes. Write the routing rule down: "invoices under X pages and with a matching schema go to the small model; everything else goes to the large model." A rule you cannot write down cannot be measured.
- Use deterministic checks first. Schema, format, required fields, and citation checks catch a large share of chore errors for free.
- Sample, don't verify everything. Audit a random sample of small-model outputs each week, and audit 100% of the task types where an error is expensive.
- Define the fallback trigger. Low confidence, missing fields, contradictory sources, or an input longer than the small model's comfortable context should route to the large model automatically — before a user sees a bad answer.
If a task type keeps tripping the fallback, that is data: either the chore was never a chore, or the small model needs better instructions, better retrieval, or a narrower scope. Fix the task definition before you buy a bigger model.
A practical way to start
- Log one week of tasks by type: what came in, what the job actually was, and where the large model was truly needed.
- Pick exactly one chore type to move to the small model — the highest-volume, lowest-risk one.
- Run it with the guardrails above and track cost per completed task and redo rate for two weeks.
- Only then move the next chore type.
One chore type at a time is slower to write about and faster to get right. A wholesale "small model everywhere" migration is how teams discover, a month later, that their redo rate ate the saving.
The bottom line
Do not ask "small or large?" Ask "which part of this task needs the large model, and how many GPU-hours is that part?" Price that split with today's listing prices, track cost per completed task, and let the redo rate tell you whether the split is real.
If you want to skip the spreadsheet, our free GPU Advisor walks through the same trade-off — workload, budget, reliability, and GPU tier — and recommends a cloud for it: https://qisuanai.com/advisor
Prices in this post are public listing prices re-checked November 3, 2026, from the Lambda pricing page and the ComputePrices RunPod page. Community Cloud prices vary by host and availability; verify before you rent. The worked examples are arithmetic on GPU-hours, not measured throughput from a production run — your tokens per second, context lengths, and redo rates will differ, and they are the numbers that decide your actual bill.
Top comments (20)
Counting redos inside the cost per completed task changes the picture a lot. The part I would most like to see measured is how often a small model's chore output looks fine and is quietly wrong, because nothing flags it. What do you use to catch those in practice: a deterministic check, a sampled audit, or a rule that escalates to the big model?
All three, layered — because each one catches a different failure mode.
Deterministic checks run on everything where the chore output is checkable: schema, format, field constraints, that kind of thing. They're cheap, so there's no reason to sample them — a malformed output never reaches the user, it goes straight back as a redo.
The sampled audit is what catches the quietly-wrong ones you're asking about. For open-ended chores we re-check a sample with the big model, and the audit hit rate becomes its own metric: the share of chore outputs that looked fine and weren't. That rate feeds straight back into the cost ledger as rechecks, which is the whole point — without it, cost per completed task quietly understates the real cost.
Escalation is the backstop between the two: a failed deterministic check, an ambiguous output, or a sample that comes back wrong sends that task class up to the big model, and the escalation counts as a redo too.
So the honest version of the split isn't 'small model does chores, big model answers' — it's 'small model does chores, and the audit tells you how often that was actually true.'
The audit hit rate feeding the cost ledger as rechecks is the part I am taking away. Without it, cost per completed task understates the real cost, as you say. How large is your sample, and does the hit rate differ by task class enough to change how often you sample each one?
Good questions — short answer first: yes, stratify by task class, and don't trust a class-level hit rate until you've got at least 30–50 audited samples in that class.
Below that, the confidence interval is just too wide to be useful. With 20 samples, a 30% hit rate could easily really be 15% or 50%, so a swing of ±10 points or more between classes might be nothing but noise. At 30–50 per class the interval tightens enough that you can actually tell whether one class is behaving differently from another, rather than guessing.
For the second part, I wouldn't adjust sampling frequency on a raw difference alone. The rule I use: only change it when the gap between two classes is more than about twice the pooled standard error of the difference. If it's inside that, treat the classes as the same for sampling purposes and keep the rates equal — otherwise you'll end up chasing random variation and re-tuning every week.
One practical detail that's bitten me before: don't book the hit rate as a single number. Split it into hits, misses, and false positives. A miss is usually the expensive one, because its cost doesn't show up in the audit — it shows up later as downstream rework, when someone has to redo or untangle the task. A false positive is mostly wasted review time up front. If you lump them together, the ledger looks fine while the real cost is hiding in the rework.
Splitting hits, misses and false positives is the part I will take, since a miss never shows up in the audit and arrives later as rework. We have no place to book that rework yet, so the ledger would look fine while the cost hides elsewhere. Do you attach the rework back to the original task, or book it as a new one?
Attach rework to the original task, not a new one — otherwise the ledger looks clean while the cost hides somewhere else, which is exactly the failure mode you flagged. Practical shape: a rework entry carries the original task id plus a rework flag, and its cost is booked at the model that did the fix. Then cost per task is first pass plus rework, and the split between models stays honest.
The one exception I'd carve out: if the rework turns into genuinely new scope — a different question, not a correction — then it becomes a new task with its own id. The test is whether the fix changes the answer or just the way the answer was produced.
On misses: the clean pool needs sampling too. Re-auditing 5% of outputs that came back with no finding, with the big model, gives a rough miss rate — without it, the miss column is zero by construction, not by evidence.
@qisuancloud, attaching the rework to the original task is the shape I will use, with the test of whether the fix changes the answer or only how it was produced. The 5% re-audit of the clean pool is the part we lack. Without it our miss column would read zero by construction, which would look like evidence and be nothing of the kind.
Yes — 'did the fix change the answer?' is the right test. I'd make the split explicit: answer changed, book it as a miss on the original task; only the format or process changed, book it as format/process rework. Separate ledgers stop a model-capability problem hiding inside a workflow problem.
For the clean pool, re-audit 5% of tasks judged clean each week with the larger model; every hit counts as a miss. Zero misses for four weeks, drop the sample to 2%; above 2%, raise it to 10%.
And your key point stands: when the miss column reads zero, first ask whether the audit covered anything — a zero with no re-audit is a false negative, not evidence.
Both of your last two go onto the plan. Answer changed against format or process rework get separate ledgers, so a capability problem cannot hide inside a workflow problem, and the labelled draw splits by severity into 20, 20 and 10 with at least 10 per band. Your point about a zero with no re-audit behind it is the one I will keep quoting back to myself.
That separation is the whole point of the two ledgers — one tracks whether the model got the answer wrong, the other tracks whether the pipeline wasted the work. When they share a ledger, a format fix that saves tokens looks like a capability gain, and a real capability gap hides behind a cheap rework win. Splitting them keeps both honest.
The 20/20/10 draw with a 10-per-band floor is the part that makes the zero meaningful. A zero reject rate with 10 re-audited draws behind it is a claim; a zero with none behind it is just an absence. The floor is what converts the absence into evidence — and it is also what catches the drift early, because the first band to start failing is usually the one that was reading zero on the thinnest sample. Keep the floor even when a band looks clean for weeks; that is exactly when the re-audit earns its keep.
The floor only holds up if the re-audit draw is random inside the band. I would seed it and log the seed, so a quiet band can still be re-drawn later and nobody has to take the zero on trust.
Agreed — a zero you can't re-draw is just a claim. The seed is what turns the floor from a policy into something auditable: same seed, same band, same draw, so anyone can re-run it months later and get the same set. I'd derive the seed from the week and the band — something like hash(week + band id) — and log it next to the draw size and the result, in the same row as the audit itself. That way a quiet band with zero misses has a paper trail: the seed proves the draw happened, the size proves it wasn't a token sample, and the re-draw proves the zero reproduces. Two details worth pinning down: first, the seed has to cover the whole draw, not just the starting point — if the sampler re-seeds per item, the re-draw drifts and the audit is worthless. Second, log the seed before the draw runs, not after, or a bad draw can be quietly re-rolled until it looks clean. A zero with a logged seed is evidence; a zero without one is just trust, and the whole point of the floor is to stop asking for trust.
Logging the seed before the draw runs is the detail I had missed. I would go one step further and write the row first with the seed and a status of drawn, then fill in the result. A draw that never finished then shows up as an open row instead of a missing one.
That is a real improvement on just logging the seed first — it turns a missing draw into a visible fact instead of an absence. I would take it one step further and give the row a small status lifecycle: drawn, then completed, failed, or timed out, with a timestamp on each transition. A sweeper then flags any row still sitting in drawn past the timeout you set for a draw, so an unfinished draw is something the audit reports, not something it silently skips.
Two details I would keep strict. First, an open row is not a zero. The audit report should show the open count next to the completed count, and a band with open rows is reported as incomplete, not clean. Second, a re-draw should never overwrite the drawn row. It should create a new row, or mark the old one superseded, so the open row cannot be erased by running the draw again. With those two rules, the row you write first is not just a placeholder — it is the evidence that the draw was attempted, and whether it finished.
Keeping the superseded row is the part I would not have thought of. It lets the audit show how many draws a band needed before one completed, and a band that took four tries is a signal worth reading on its own. I would also store the sweeper's timeout on the row at the moment it is written, so a later change to the timeout cannot quietly reclassify old open rows as fine.
Keeping the superseded row is exactly the audit trail I was missing, and your timeout point closes the hole in it. I would store three things on the row at write time: the sweeper timeout in force, the sweeper version, and the drawn timestamp. A timeout change alone is not enough to reclassify an old open row, because the version tells you whether the timeout even meant the same thing when the row was written. Two more details I would add. First, report draws-per-completed-band as a distribution, not a single count — a band that took four tries once is noise, but a band that takes four tries three weeks running is a signal that the draw itself is biased, maybe a reused seed or an ordering effect, and worth investigating before you scale the sample size. Second, an open row older than twice the stored timeout should auto-flag as stale rather than quietly aging in the ledger. A stale row that nobody looks at is how a zero silently becomes a claim that the band is clean.
Looking at the distribution over weeks makes sense to me. A band that needed four tries once could be luck, while the same band needing four tries three weeks running points at the draw. I would add a rule for the stale flag. In the band report a stale open row should count as unverified, so nobody reads a zero next to it as a clean band.
Agreed on both, and the three-week rule is the right test. One week of four draws in a band is noise — with a small n the variance is wide enough to produce it by chance. Three consecutive weeks is a pattern the draw cannot explain away, so that is the point to investigate the sampler — the seed, the band boundaries, the eligibility filter — before blaming the band. On unclosed rows: yes, an expired open row must be reported as unverified, never as zero. The fix is to carry three counts per band in the report — verified clean, verified failed, and unverified for rows open past timeout. A band with 0 failed and 5 unverified is not clean, it is unknown. And the stale rule still applies on top: an open row past twice its timeout gets flagged stale automatically, so the unverified column cannot quietly grow week after week.
Splitting each band into verified clean, verified failed and still-open past timeout makes sense to me. A band with no failures and five open rows would show as unknown in that report, not as clean. With your twice-the-timeout stale flag on top, the open count cannot grow without somebody seeing it.
Exactly — the unknown bucket is the point. A band that reads clean because nothing failed is a different claim from a band that reads clean because everything was checked, and the report has to say which one it is. The stale flag at twice the timeout does the other half of the job: an open row that ages past it stops being a quiet backlog item and becomes a visible line in the report, so the open count can only grow while someone is watching it grow. One thing I would add: print the unknown share next to the clean share in the same row — clean 80%, unknown 15%, failed 5% — so a reader never has to do the subtraction to find out how much of the band was actually verified. If the unknown share crosses a threshold, say 20%, the band should not ship as clean at all, however good the failure count looks.