At Glean, we tested a model that does not write text. It only makes decisions.
In our tests, it caught 99 out of 100 prompt injections hidden in documents, in about a tenth of a second, at about half the cost.
Most AI steps are decisions
One AI answer can hide many small calls. What kind of request is this: a question, a summary, or a task? Which model should handle it? Is this prompt safe? Which follow-up should we show?
Each has an answer you can list in advance: a label, a yes or no, or a score. Yet we send each one to a full LLM, wait for it to write text, and parse that text back into a label. It's like asking for an essay when all you need is a checkbox.
What a System 1 model does
A System 1 model is built to answer these bounded questions directly. The model we tested is Jev, a "System One" model from TypeSafe. You send it state, like a prompt, chat history, or JSON, plus typed questions. It returns typed answers with probabilities: yes or no, one pick from a list, or a score. There is no text to parse. Input costs $0.042 per million tokens, and output is free.
By the numbers
- 8.1x faster routing to the right expert, with better precision on every route
- 7x faster follow-up suggestions, with timeouts down from 10.2% to 0.8%
- 90% of unsafe content caught by our first-pass guardrail, at about half the monthly cost
- 0 training runs to reach 90% of a fine-tuned model's agreement
What we found
Guardrails. A fast first-pass screen checks every prompt, tool output, and response, and an LLM reviews only what it flags. With Jev as the screen, it caught 90% of unsafe content in our test sets, with about 91% precision. It did best where risk is hardest to see:
- Prompt injections hidden in retrieved documents: 99% caught
- Malicious code in responses: 97% caught
Across 22.5 million calls on our internal deployment, median time was 137 milliseconds. It runs alongside the LLM, so users do not wait. Monthly cost fell from about $4,000 to $2,100.
Follow-up suggestions. After each answer, a ranker shows at most one next step, or none. When a turn showed nothing, 58% of the time it was not a choice: the LLM ranker had timed out or returned an answer we could not use.
We replaced it with one Jev question whose options were the candidates plus "none." Median ranker time fell from 4.67 to 0.67 seconds, invalid answers went from 10 to 0, and turns with a suggestion rose from 83% to 91%.
Routing. On 751 labeled requests, Jev picked the right expert more often on every route. On one expert, precision went from 47% to 87%. When a request needed an expert, the call was 8.1 times faster at the median.
Query classification. This one was mixed. With no training, Jev agreed with our production labels 66.8% of the time, against 74.5% for a fine-tuned open model. Fine-tuning wins when labels are stable; no training wins when they keep changing.
What made it work
Let the model say "not sure." Every answer has a probability, so you can set a cutoff. If the top pick is 0.49 and the cutoff is 0.50, we show nothing. Showing nothing often beats a weak suggestion.
Give every call a time budget and a fallback. Our chat path allows 2 seconds, no retries. On a timeout or bad answer, we keep today's behavior.
Conclusion: A simple rule
- Can the output be listed in advance? If not, keep a generative model.
- If it can, benchmark a decision model.
- If you also need text, use both: the decision model picks the tool, and the LLM writes the arguments.
The fastest LLM call is the one you never make. Generate when you must. Decide when you can.
Top comments (4)
The "ask for a checkbox, get an essay" framing is the clearest way I've seen to describe where agent spend actually leaks. The part that matches our experience hardest is routing — we burned a surprising amount of latency asking a full model "which of these N handlers should take this?" when the answer is a closed set with a confidence score attached.
The detail I'd push on is the confidence cutoff. In our pipelines the win wasn't just speed, it was being able to say "below 0.5, show nothing / escalate to a human" — a weak suggestion is often worse than no suggestion, and a generative model never gives you that threshold cleanly because you're parsing prose back into a decision.
Curious how you handle label drift over time though — your query-classification result (no-training decision model loses to a fine-tuned one when labels are stable) suggests these two approaches aren't rivals but a lifecycle: decision model early while your taxonomy is still moving, fine-tune once it freezes. Did you ever try routing the router — cheap model first, escalate only ambiguous cases to the fine-tuned one?
Separating decision calls from generation calls is the right split: routing, classification, safety gating, and scoring all have a finite answer space, so forcing them through a text-completion model just adds latency and parsing surface for no quality gain. Typed outputs with probabilities also make the cheap path easy to reason about — you can gate on confidence and fall back to the larger model only when the System 1 answer is uncertain, instead of calling the heavy model by default. This maps cleanly onto desktop/agent pipelines too: pre-filtering and intent triage belong on the fast, small model, reserving the general LLM for the few steps that actually need prose or reasoning.
The 99-of-100 prompt injections caught in about a tenth of a second is the number that reframes this — a decision model as a first-pass screen means the LLM only ever reviews what something already flagged. The 'let the model say not sure' rule is load-bearing too; a 0.49 top pick showing nothing beats a weak suggestion every time. Where does this break down — is there a class of decision where the typed options themselves hide the ambiguity, so the model confidently picks the wrong label?
the 99 of 100 injection catch is great but i would want to see what the 1 looks like. a first pass screen that runs beside the llm only protects you if the miss path is covered, since an injection that slips past a 91 percent precision screen still reaches a model with full tool access.
we pair a cheap screen like this with a hard rule on the tool side, so a missed injection still cannot trigger a write without approval. the screen lowers volume, the tool boundary caps the damage.
did the misses skew toward a specific document type, like long pdfs or tables?