A small C# router with a confidence band, a calibration step, and an escalation path. Local models only. Some of it worked, some of it didn’t.
Why I built this at all
Two posts landed in my reading list in the same week, and together they changed how I think about “which model should make this decision”.
The first was Kervin Hu testing Jev on entity resolution. The second was Tanmay Deshpande building evals for agents that refunded orders they shouldn’t have. Different domains, same uncomfortable lesson: the cheap model is often good enough, its confidence score is often lying to you, and the fix is not “use the expensive model everywhere”.
The pattern itself is old. Score everything with something cheap, trust it at the extremes, and send only the murky middle to something slower and smarter (or to a person). What I wanted to know was whether I could build it in .NET without a paid API, and whether the “hybrid beats both” claim held up on my own data.
Short answer: it held up partly. The longer answer is the rest of this post.
What the two authors reported (I did not reproduce any of this)
I want to be very clear about this section. Every number below comes from the author’s own write-up. I did not run Jev, Opus, Sol, or nimble. I have no way to check these figures, and I’m not presenting them as verified.
+---------------------+----------------------------------------------+---------------------------+
| Source | Claim | Status |
+---------------------+----------------------------------------------+---------------------------+
| Kervin Hu (Medium) | Jev F1 0.943, Opus 0.967, Sol 0.985 on an | Author-reported, |
| | entity resolution set | not reproduced |
| Kervin Hu (Medium) | Sending only the 194 pairs Jev scored | Author-reported, |
| | between 0.3 and 0.7 to Opus: F1 0.972 at | not reproduced |
| | about 17% of Opus-only cost | |
| Kervin Hu (Medium) | Jev's probabilities were poorly calibrated | Author's observation |
| Tanmay Deshpande | nimble: 64% balanced accuracy at the default | Author-reported, |
| (Medium) | 0.5 cutoff, 91% at a 0.77 cutoff tuned on a | not reproduced |
| | held-out split | |
| Tanmay Deshpande | Advertised sub-0.1 s decisions measured at | Author's own measurement, |
| (Medium) | 0.4 to 4.7 s in practice | vendor claim not checked |
+---------------------+----------------------------------------------+---------------------------+
Two things jumped out at me.
From Hu: the hybrid (0.972) beat the model it escalated to (0.967) while costing a fraction of it. That sounds impossible until you realise the cheap model is genuinely right on the easy pairs, and the expensive model also makes mistakes there. You only pay the big model where it actually adds something. Note that the 17% figure depends entirely on how many pairs happened to land in the band, which is a property of the data, not the method.
From Deshpande: the same model went from 64% to 91% balanced accuracy just by moving the threshold. Nothing about the model changed. That is a calibration problem dressed up as a model quality problem. And the latency gap between the advertised and measured numbers is a reminder to time things yourself.
There were also a few related pieces in my feed. Gursimar Singh compared Jev and Laya with a headline suggesting the free model can end up costing more. Itai Spector wrote that a Jev router for Claude Code saves less than people expect. Mohit Mayank wrote up GLiNER2.5-Decide, claiming Jev-level accuracy at roughly 7x the speed as a single local model. There was also a write-up framing Cloudflare’s Clef as a response to Jev. I’m listing these as author and vendor claims. I haven’t tested any of them.
The plan
Everything local, everything in C#, no Python sidecar in the hot path.
+--------------------+
pair ---> | cheap scorer | logistic regression on string-similarity
| (microseconds) | features, trained in plain C#
+---------+----------+
|
v
+--------------------+
| calibrator (Platt) | fitted on a separate labelled split
+---------+----------+
|
p >= high | p < low
+----------------+----------------+
| | |
v v v
accept low <= p < high reject
|
v
+--------------------+
| Ollama model | qwen2.5:14b, JSON-schema output
| match/no_match/ |
| unsure |
+---------+----------+
|
unsure / timeout / error
|
v
human review queue
Why not GLiNER2.5-Decide or a Laya-style decision model as the cheap scorer? Honestly, because I wanted the first version to be a single dotnet run with nothing else to install, and a logistic regression over similarity features is a reasonable baseline for product matching. The scorer sits behind a function Pair -> probability, so swapping in a local decision model served over HTTP is a one-line change. If you do that, the calibration step matters even more, because Hu's whole point was that the raw probabilities from that kind of model were off.
The escalation target is a local Ollama model. “Cost” in this setup is not dollars; it’s GPU seconds on my own machine. I treat big-model wall-clock time as the cost unit.
Setup
Ollama (native or Docker)
Native install, then pull the escalation model:
ollama pull qwen2.5:14b
ollama serve # if it isn't already running as a service
Or with Docker (NVIDIA GPU):
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama pull qwen2.5:14b
No GPU? It still works on CPU, just slowly. Try qwen2.5:7b or llama3.1:8b and expect the escalated route to be much slower than the numbers below.
The project
dotnet new console -n DecisionRouter -f net8.0
cd DecisionRouter
No NuGet packages are needed. Everything uses the base class library.
The data
I used the Abt-Buy product matching benchmark from the Leipzig database group (Abt.csv, Buy.csv, abt_buy_perfectMapping.csv). The router expects a tab-separated pairs.tsv with id, left, right, label. Here's the prep script I used. Check the column names in your copy of the files, since mirrors differ slightly.
import pandas as pd, random
random.seed(42)
abt = pd.read_csv("Abt.csv", encoding="iso-8859-1")
buy = pd.read_csv("Buy.csv", encoding="iso-8859-1")
gold = pd.read_csv("abt_buy_perfectMapping.csv", encoding="iso-8859-1")
abt_name = dict(zip(abt.id, abt.name))
buy_name = dict(zip(buy.id, buy.name.fillna("") + " " + buy.manufacturer.fillna("")))
matches = set(zip(gold.idAbt, gold.idBuy))
buy_ids = list(buy_name)
def clean(s):
return str(s).replace("\t", " ").replace("\n", " ").strip()
rows = []
for a, b in matches:
rows.append((a, b, 1))
# hard negatives: share the first token (usually the brand)
first = abt_name[a].lower().split()[0]
hard = [x for x in buy_ids if first in buy_name[x].lower() and (a, x) not in matches]
for x in random.sample(hard, min(3, len(hard))):
rows.append((a, x, 0))
# easy negatives
for x in random.sample(buy_ids, 5):
if (a, x) not in matches:
rows.append((a, x, 0))
with open("pairs.tsv", "w", encoding="utf-8") as f:
f.write("id\tleft\tright\tlabel\n")
for i, (a, b, y) in enumerate(rows):
f.write(f"{i}\t{clean(abt_name[a])}\t{clean(buy_name[b])}\t{y}\n")
That gives roughly one positive for every eight negatives. Keep that ratio in mind; it matters in the calibration section.
The cheap scorer
Five similarity features plus a bias term. Nothing clever.
Features.cs:
static class Features
{
public const int Count = 6;
public static double[] Extract(string a, string b)
{
var na = Normalize(a);
var nb = Normalize(b);
var ta = Tokens(na);
var tb = Tokens(nb);
return new[]
{
Jaccard(ta, tb),
Jaccard(Grams(na, 3), Grams(nb, 3)),
JaroWinkler(na, nb),
NumberOverlap(ta, tb),
LengthRatio(na, nb),
1.0 // bias
};
}
static string Normalize(string s)
{
var chars = s.ToLowerInvariant().Select(c => char.IsLetterOrDigit(c) ? c : ' ').ToArray();
return string.Join(' ', new string(chars).Split(' ', StringSplitOptions.RemoveEmptyEntries));
}
static HashSet<string> Tokens(string s) =>
s.Split(' ', StringSplitOptions.RemoveEmptyEntries).ToHashSet();
static HashSet<string> Grams(string s, int n)
{
var set = new HashSet<string>();
var padded = $" {s} ";
for (int i = 0; i + n <= padded.Length; i++) set.Add(padded.Substring(i, n));
return set;
}
static double Jaccard(HashSet<string> a, HashSet<string> b)
{
if (a.Count == 0 && b.Count == 0) return 1;
int inter = a.Count(x => b.Contains(x));
return (double)inter / (a.Count + b.Count - inter);
}
// Model numbers carry most of the signal in product matching
static double NumberOverlap(HashSet<string> a, HashSet<string> b)
{
var na = a.Where(t => t.Any(char.IsDigit)).ToHashSet();
var nb = b.Where(t => t.Any(char.IsDigit)).ToHashSet();
if (na.Count == 0 && nb.Count == 0) return 0.5;
return Jaccard(na, nb);
}
static double LengthRatio(string a, string b) =>
a.Length == 0 || b.Length == 0 ? 0 : (double)Math.Min(a.Length, b.Length) / Math.Max(a.Length, b.Length);
static double JaroWinkler(string s1, string s2)
{
if (s1 == s2) return 1;
if (s1.Length == 0 || s2.Length == 0) return 0;
int range = Math.Max(0, Math.Max(s1.Length, s2.Length) / 2 - 1);
var m1 = new bool[s1.Length];
var m2 = new bool[s2.Length];
int matches = 0;
for (int i = 0; i < s1.Length; i++)
{
int start = Math.Max(0, i - range), end = Math.Min(i + range + 1, s2.Length);
for (int j = start; j < end; j++)
{
if (m2[j] || s1[i] != s2[j]) continue;
m1[i] = m2[j] = true;
matches++;
break;
}
}
if (matches == 0) return 0;
int t = 0, k = 0;
for (int i = 0; i < s1.Length; i++)
{
if (!m1[i]) continue;
while (!m2[k]) k++;
if (s1[i] != s2[k]) t++;
k++;
}
double jaro = ((double)matches / s1.Length + (double)matches / s2.Length + (matches - t / 2.0) / matches) / 3.0;
int prefix = 0;
for (int i = 0; i < Math.Min(4, Math.Min(s1.Length, s2.Length)) && s1[i] == s2[i]; i++) prefix++;
return jaro + prefix * 0.1 * (1 - jaro);
}
}
Models.cs holds the logistic regression, the calibrator, and the records:
record Pair(string Id, string Left, string Right, int Label);
record CachedVerdict(string Id, int? Label, double Seconds);
record RouteResult(string Name, double P, double R, double F1, double EscalatedPct,
int Human, double BigSecs, double P50, double P95);
sealed class LogisticModel
{
readonly double[] w = new double[Features.Count];
public void Train(IReadOnlyList<double[]> X, IReadOnlyList<int> y,
int epochs = 400, double lr = 0.5, double l2 = 1e-3)
{
int n = X.Count;
for (int e = 0; e < epochs; e++)
{
var grad = new double[w.Length];
for (int i = 0; i < n; i++)
{
double err = Sigmoid(Dot(X[i])) - y[i];
for (int j = 0; j < w.Length; j++) grad[j] += err * X[i][j];
}
for (int j = 0; j < w.Length; j++) w[j] -= lr * (grad[j] / n + l2 * w[j]);
}
}
public double Score(double[] x) => Sigmoid(Dot(x));
double Dot(double[] x)
{
double s = 0;
for (int j = 0; j < w.Length; j++) s += w[j] * x[j];
return s;
}
public static double Sigmoid(double z) => 1.0 / (1.0 + Math.Exp(-z));
}
// Platt scaling: p = sigmoid(a * logit(score) + b), fitted on a held-out labelled split
sealed class PlattCalibrator
{
double a = 1, b = 0;
public void Fit(IReadOnlyList<double> scores, IReadOnlyList<int> y, int iters = 2000, double lr = 0.1)
{
for (int it = 0; it < iters; it++)
{
double ga = 0, gb = 0;
for (int i = 0; i < scores.Count; i++)
{
double z = Logit(scores[i]);
double err = LogisticModel.Sigmoid(a * z + b) - y[i];
ga += err * z;
gb += err;
}
a -= lr * ga / scores.Count;
b -= lr * gb / scores.Count;
}
}
public double Apply(double s) => LogisticModel.Sigmoid(a * Logit(s) + b);
static double Logit(double p)
{
p = Math.Clamp(p, 1e-6, 1 - 1e-6);
return Math.Log(p / (1 - p));
}
}
I picked Platt over isotonic regression on purpose. Isotonic is more flexible but needs more labelled data to not overfit, and my calibration split is a few hundred pairs. If you have thousands of labels, try isotonic.
The escalation target
The Ollama call uses structured output, so the model has to return one of three verdicts. Giving it an explicit unsure option was the single best decision in this project. Without it, the big model guesses on the exact pairs where guessing is most expensive.
OllamaJudge.cs:
using System.Diagnostics;
using System.Net.Http.Json;
using System.Text.Json;
sealed class OllamaJudge(HttpClient http, string model)
{
public async Task<(int? Label, double Seconds)> JudgeAsync(Pair p, CancellationToken ct = default)
{
var sw = Stopwatch.StartNew();
var body = new
{
model,
stream = false,
options = new { temperature = 0 },
format = new
{
type = "object",
properties = new
{
verdict = new { type = "string", @enum = new[] { "match", "no_match", "unsure" } }
},
required = new[] { "verdict" }
},
messages = new object[]
{
new
{
role = "system",
content = "You decide whether two product records refer to the same real-world product. " +
"Different model numbers, sizes or colours mean no_match. " +
"Answer \"unsure\" if the records do not contain enough information to decide."
},
new { role = "user", content = $"Record A: {p.Left}\nRecord B: {p.Right}" }
}
};
using var resp = await http.PostAsJsonAsync("/api/chat", body, ct);
resp.EnsureSuccessStatusCode();
using var doc = JsonDocument.Parse(await resp.Content.ReadAsStringAsync(ct));
var content = doc.RootElement.GetProperty("message").GetProperty("content").GetString() ?? "{}";
using var verdictDoc = JsonDocument.Parse(content);
var verdict = verdictDoc.RootElement.TryGetProperty("verdict", out var v) ? v.GetString() : null;
sw.Stop();
int? label = verdict switch { "match" => 1, "no_match" => 0, _ => null };
return (label, sw.Elapsed.TotalSeconds);
}
}
The router
This is the production path. Anything the big model can’t decide, or anything that times out, goes to a human queue instead of being silently guessed.
Router.cs:
using System.Diagnostics;
using System.Text.Json;
enum Route { CheapAccept, CheapReject, Escalated, Human }
sealed record RoutedDecision(int? Label, Route Route, double CheapSeconds, double BigSeconds);
sealed class Router(Func<Pair, double> calibratedScore, OllamaJudge judge,
double low, double high, TimeSpan bigTimeout, string humanQueuePath = "human_queue.jsonl")
{
public async Task<RoutedDecision> DecideAsync(Pair p)
{
var sw = Stopwatch.StartNew();
double prob = calibratedScore(p);
double cheap = sw.Elapsed.TotalSeconds;
if (prob >= high) return new(1, Route.CheapAccept, cheap, 0);
if (prob < low) return new(0, Route.CheapReject, cheap, 0);
using var cts = new CancellationTokenSource(bigTimeout);
try
{
var (label, secs) = await judge.JudgeAsync(p, cts.Token);
if (label is not null) return new(label, Route.Escalated, cheap, secs);
await QueueForHumanAsync(p, prob, "model_unsure");
return new(null, Route.Human, cheap, secs);
}
catch (Exception ex) when (ex is OperationCanceledException or HttpRequestException or JsonException)
{
await QueueForHumanAsync(p, prob, ex.GetType().Name);
return new(null, Route.Human, cheap, bigTimeout.TotalSeconds);
}
}
async Task QueueForHumanAsync(Pair p, double prob, string reason)
{
var line = JsonSerializer.Serialize(new { p.Id, p.Left, p.Right, prob, reason, at = DateTime.UtcNow });
await File.AppendAllTextAsync(humanQueuePath, line + Environment.NewLine);
}
}
The experiment harness
The trick that made experimenting bearable: run the big model once over the calibration and test splits, cache every verdict with its latency, and then replay the cache for every band I want to try. Without the cache, every grid point means hours of GPU time.
Program.cs:
using System.Diagnostics;
using System.Globalization;
using System.Text.Json;
// A Turkish locale prints 0,943 and %12,5. Invariant keeps the output paste-able.
CultureInfo.DefaultThreadCurrentCulture = CultureInfo.InvariantCulture;
CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;
var bigModel = args.Length > 0 ? args[0] : "qwen2.5:14b";
var maxEscalation = args.Length > 1 ? double.Parse(args[1]) : 0.20;
var rng = new Random(42);
var pairs = LoadTsv("pairs.tsv").OrderBy(_ => rng.Next()).ToList();
int nTrain = (int)(pairs.Count * 0.6), nCal = (int)(pairs.Count * 0.2);
var train = pairs.Take(nTrain).ToList();
var calib = pairs.Skip(nTrain).Take(nCal).ToList();
var test = pairs.Skip(nTrain + nCal).ToList();
// Balanced training set. Common practice, and the reason raw scores end up miscalibrated
// on data that is really ~1 positive per 8 negatives.
var pos = train.Where(p => p.Label == 1).ToList();
var neg = train.Where(p => p.Label == 0).OrderBy(_ => rng.Next()).Take(pos.Count).ToList();
var balanced = pos.Concat(neg).ToList();
var cheap = new LogisticModel();
cheap.Train(balanced.Select(p => Features.Extract(p.Left, p.Right)).ToList(),
balanced.Select(p => p.Label).ToList());
double[] RawScores(List<Pair> set) => set.Select(p => cheap.Score(Features.Extract(p.Left, p.Right))).ToArray();
var calRaw = RawScores(calib);
var platt = new PlattCalibrator();
platt.Fit(calRaw, calib.Select(p => p.Label).ToList());
var calProb = calRaw.Select(platt.Apply).ToArray();
var sw = Stopwatch.StartNew();
var testRaw = RawScores(test);
double cheapPerItem = sw.Elapsed.TotalSeconds / test.Count;
var testProb = testRaw.Select(platt.Apply).ToArray();
var testLabels = test.Select(p => p.Label).ToList();
Console.WriteLine($"Pairs: train {train.Count}, calib {calib.Count}, test {test.Count}");
Console.WriteLine($"ECE on test, raw: {Ece(testRaw, testLabels):F3}");
Console.WriteLine($"ECE on test, calibrated: {Ece(testProb, testLabels):F3}");
using var http = new HttpClient { BaseAddress = new Uri("http://localhost:11434"), Timeout = TimeSpan.FromMinutes(5) };
var judge = new OllamaJudge(http, bigModel);
var tag = bigModel.Replace(':', '_').Replace('/', '_');
var bigCal = await BigPassAsync(judge, calib, $"big_calib_{tag}.jsonl");
var bigTest = await BigPassAsync(judge, test, $"big_test_{tag}.jsonl");
// Threshold tuned on the calibration split, never on test
double tuned = Enumerable.Range(1, 19).Select(k => k * 0.05)
.OrderByDescending(t => Prf(calib.Select((p, i) => (calProb[i] >= t ? 1 : 0, p.Label))).F1)
.First();
var band = SelectBand(calib, calProb, bigCal, maxEscalation);
Console.WriteLine($"Tuned cheap threshold: {tuned:F2}, band: [{band.Lo:F2}, {band.Hi:F2})");
var results = new List<RouteResult>
{
EvalRoute("Cheap, raw @0.50", test, testRaw, bigTest, 0.5, 0.5, cheapPerItem),
EvalRoute($"Cheap, calibrated @{tuned:F2}", test, testProb, bigTest, tuned, tuned, cheapPerItem),
EvalRoute($"Big only ({bigModel})", test, testProb, bigTest, 0.0, 1.01, cheapPerItem),
EvalRoute($"Hybrid [{band.Lo:F2},{band.Hi:F2})", test, testProb, bigTest, band.Lo, band.Hi, cheapPerItem),
EvalRoute("Hybrid [0.30,0.70) fixed", test, testProb, bigTest, 0.3, 0.7, cheapPerItem),
};
PrintTable(results, results[2].BigSecs);
// Production path, one live call
var router = new Router(p => platt.Apply(cheap.Score(Features.Extract(p.Left, p.Right))),
judge, band.Lo, band.Hi, TimeSpan.FromSeconds(30));
var sample = test.First(p => { var s = platt.Apply(cheap.Score(Features.Extract(p.Left, p.Right))); return s >= band.Lo && s < band.Hi; });
Console.WriteLine(await router.DecideAsync(sample));
// ---------- helpers ----------
static List<Pair> LoadTsv(string path) => File.ReadLines(path).Skip(1)
.Select(l => l.Split('\t'))
.Where(c => c.Length == 4)
.Select(c => new Pair(c[0], c[1], c[2], int.Parse(c[3])))
.ToList();
static async Task<Dictionary<string, CachedVerdict>> BigPassAsync(OllamaJudge judge, List<Pair> pairs, string cachePath)
{
var cache = File.Exists(cachePath)
? File.ReadLines(cachePath).Select(l => JsonSerializer.Deserialize<CachedVerdict>(l)!).ToDictionary(c => c.Id)
: new Dictionary<string, CachedVerdict>();
await using var w = new StreamWriter(cachePath, append: true);
foreach (var p in pairs.Where(p => !cache.ContainsKey(p.Id)))
{
var (label, secs) = await judge.JudgeAsync(p);
var v = new CachedVerdict(p.Id, label, secs);
cache[p.Id] = v;
await w.WriteLineAsync(JsonSerializer.Serialize(v));
await w.FlushAsync();
}
return cache;
}
// Replays cached big-model verdicts. lo == hi means "cheap only"; lo = 0, hi > 1 means "big only".
// Human-queued items are scored as if the human got them right, which is optimistic.
// The Human column shows how many there were, so you can judge that yourself.
static RouteResult EvalRoute(string name, List<Pair> pairs, double[] prob,
Dictionary<string, CachedVerdict> big, double lo, double hi, double cheapSecs)
{
var rows = new List<(int Pred, int Gold)>();
var lat = new List<double>();
int escalated = 0, human = 0;
double bigSecs = 0;
for (int i = 0; i < pairs.Count; i++)
{
var p = pairs[i];
if (prob[i] >= hi) { rows.Add((1, p.Label)); lat.Add(cheapSecs); continue; }
if (prob[i] < lo) { rows.Add((0, p.Label)); lat.Add(cheapSecs); continue; }
escalated++;
var v = big[p.Id];
bigSecs += v.Seconds;
lat.Add(cheapSecs + v.Seconds);
if (v.Label is int l) rows.Add((l, p.Label));
else { human++; rows.Add((p.Label, p.Label)); }
}
var (pr, rc, f1) = Prf(rows);
return new RouteResult(name, pr, rc, f1, (double)escalated / pairs.Count, human, bigSecs,
Pct(lat, 0.50), Pct(lat, 0.95));
}
static (double Lo, double Hi) SelectBand(List<Pair> calib, double[] prob,
Dictionary<string, CachedVerdict> big, double maxEscalation)
{
(double Lo, double Hi, double F1) best = (0.5, 0.5, -1);
for (int a = 1; a <= 10; a++)
for (int b = 10; b <= 19; b++)
{
double lo = a * 0.05, hi = b * 0.05;
if (hi <= lo) continue;
var r = EvalRoute("", calib, prob, big, lo, hi, 0);
if (r.EscalatedPct > maxEscalation) continue;
if (r.F1 > best.F1) best = (lo, hi, r.F1);
}
return (best.Lo, best.Hi);
}
static (double P, double R, double F1) Prf(IEnumerable<(int Pred, int Gold)> rows)
{
int tp = 0, fp = 0, fn = 0;
foreach (var (pred, gold) in rows)
{
if (pred == 1 && gold == 1) tp++;
else if (pred == 1) fp++;
else if (gold == 1) fn++;
}
double p = tp + fp == 0 ? 0 : (double)tp / (tp + fp);
double r = tp + fn == 0 ? 0 : (double)tp / (tp + fn);
return (p, r, p + r == 0 ? 0 : 2 * p * r / (p + r));
}
static double Ece(IReadOnlyList<double> p, IReadOnlyList<int> y, int bins = 10)
{
double ece = 0;
for (int b = 0; b < bins; b++)
{
double lo = (double)b / bins, hi = (double)(b + 1) / bins;
var idx = Enumerable.Range(0, p.Count)
.Where(i => p[i] >= lo && (p[i] < hi || (b == bins - 1 && p[i] <= hi))).ToList();
if (idx.Count == 0) continue;
double conf = idx.Average(i => p[i]);
double rate = idx.Average(i => (double)y[i]);
ece += (double)idx.Count / p.Count * Math.Abs(conf - rate);
}
return ece;
}
static double Pct(List<double> xs, double q)
{
var s = xs.OrderBy(x => x).ToList();
return s[(int)Math.Min(s.Count - 1, Math.Floor(q * s.Count))];
}
static void PrintTable(List<RouteResult> results, double bigOnlySecs)
{
string[] head = { "Route", "Prec", "Recall", "F1", "Escalated", "Human", "p50 s", "p95 s", "Cost vs big" };
var rows = results.Select(r => new[]
{
r.Name, $"{r.P:F3}", $"{r.R:F3}", $"{r.F1:F3}", $"{r.EscalatedPct:P1}", $"{r.Human}",
$"{r.P50:F3}", $"{r.P95:F3}", bigOnlySecs == 0 ? "-" : $"{r.BigSecs / bigOnlySecs:P0}"
}).ToList();
var widths = head.Select((h, i) => Math.Max(h.Length, rows.Max(r => r[i].Length))).ToArray();
string Line() => "+" + string.Join("+", widths.Select(w => new string('-', w + 2))) + "+";
string Row(string[] c) => "| " + string.Join(" | ", c.Select((x, i) => x.PadRight(widths[i]))) + " |";
Console.WriteLine(Line());
Console.WriteLine(Row(head));
Console.WriteLine(Line());
foreach (var r in rows) Console.WriteLine(Row(r));
Console.WriteLine(Line());
}
Run it:
dotnet run -- qwen2.5:14b 0.20
The first run is slow because every calibration and test pair goes through the big model once. After that the caches make it instant, and you can change the band logic, the escalation budget or the calibrator without paying for inference again.
Where the band did not help
This is the part I care about most, because “hybrid beats both” is easy to repeat and harder to earn.
Correlated errors. The hybrid only wins when the big model is right on the pairs the cheap model is unsure about. On product data, a lot of the uncertain pairs are uncertain because the records genuinely don’t contain enough information: “Sony 32in LCD TV black” versus “Sony Bravia 32 inch”. No amount of model size fixes missing information.
The fixed 0.3 to 0.7 band. I included Hu’s band as a fixed comparison because it’s the number people quote. My takeaway: the band is a hyperparameter of your data and your calibrator, not a constant you copy from someone else’s post.
Without calibration. When I pointed the band at the raw scores instead of the Platt output, The balanced training set pushes raw scores towards the middle on data where negatives dominate, so the band catches the wrong pairs. This is the same failure Deshpande described from the threshold side.
The human queue is not free. I scored human-queued items as if a person got them right. That’s optimistic, and it flatters the hybrid. If that number is large, the honest framing is “the router moved work from the GPU to a person”, not “the router improved F1”.
Selection on a small calibration split. With a few hundred calibration pairs, the band that maximises F1 is noisy. I’d want confidence intervals before trusting a third decimal place, mine or anyone else’s.
What I’d tell myself before starting
- Calibrate before you route. A confidence band on uncalibrated scores is just a random sample with extra steps.
- Tune thresholds and bands on a held-out split, report on another. If you tune on test you will fool yourself.
- Give the escalation model a way to say “I don’t know”. Then decide, deliberately, where “I don’t know” goes.
- Cache the expensive model’s verdicts once. Experimenting becomes free.
- Measure latency yourself, at p95. Advertised numbers and measured numbers diverged in Deshpande’s write-up, and I’d assume they diverge everywhere until proven otherwise.
- Don’t copy someone else’s band. Including Hu’s, including mine.
What I still don’t know
Whether a proper local decision model (GLiNER2.5-Decide or something Laya-shaped) as the cheap scorer would shift the picture. The claims around those models are interesting, but they’re claims, and I haven’t put one behind this router yet. The seam is there; that’s the next experiment.
I also don’t know how this behaves on a task where errors are asymmetric, like Deshpande’s refunds, where a false “approve” costs real money and a false “reject” costs a support ticket. In that case F1 is the wrong objective for band selection, and you’d want to optimise expected cost instead. That’s a small change to SelectBand, but I haven't validated it.
To be explicit one last time: none of the vendor or author numbers in this post were reproduced by me. The only numbers I’m standing behind are the ones in my own table, on my own data, on my own machine.
Tags: dotnet, csharp, machine-learning, llm, ollama, entity-resolution, model-calibration, ai-engineering, local-llm, software-architecture
Top comments (0)