Which of your LLM calls are really decisions?
What we measured with a System One model.

“System One” is the name used for models that make fast, structured decisions instead of writing text. It borrows from Kahneman’s fast and slow thinking: a chat model reasoning through a problem is the slow side, and a System One model is the fast, intuitive side.

You give it some text or JSON and a typed question. It returns a typed answer with a probability, not a paragraph. There are three question types: yes/no (how likely is this statement to be true), choice (pick one option from a set you define) and score (place the input on an ordered scale).

Jev is the first model of this kind we came across. Three facts shape how you would use it. It is hosted only, with no downloadable weights. It accepts text only. And it was in early access when we tested it; we ran version jev-1.13.0 in October 2026.

You will see the same claims repeated: answers in 70 to 500 ms, about $0.042 per million input tokens with free output, and a model that “cannot hallucinate”. Most coverage repeats the maker’s figures, and independent reproductions are scarce; DataCamp’s overview says as much. The “cannot hallucinate” claim means the output is always a valid value from your list. It does not mean the value is right. So we measured how often it was right.

We ran Jev and Claude Haiku 5.5 on the same examples from public datasets: Banking77 (77 customer-support intents, 500 examples), a 10-class subset of it (400 examples) and a content-safety dataset for a safe-or-unsafe check (500 examples). Datasets: Banking77 (PolyAI, CC-BY-4.0) and the Aegis 2.0 content-safety dataset (NVIDIA, CC-BY-4.0).

Haiku ran with thinking turned off and a 128-token cap. That is not its default setting; with thinking on, some calls used the whole budget before answering. Latency was measured from our client, network included, with four calls in flight. Cost is tokens used times each model’s list price on 22 October 2026, not an invoice. We did not tune prompts for either model.

Speed and cost. Jev’s median latency was 346 to 372 ms against 1,035 to 1,057 ms for Haiku, about 2.8 to 3 times faster. At list prices it cost about 2.1 to 2.5 times less. Its slow tail was modest, with p95 between 463 and 677 ms, though on the safety task that went past the 500 ms top of the range its maker quotes.

Accuracy. On the 77-class task Jev was ahead by 4.0 points (95% interval +1.4 to +7.0, exact McNemar test p = 0.011). There were 38 examples only Jev got right and 18 only Haiku got right. The result still holds after a simple correction for testing three tasks. On the 10-class task the gap was within noise (−0.3 points, p = 1.0), and on the safety task it was too (+2.8 points, interval −0.0 to +5.8, p = 0.08). Each task was one run on one date, so treat the 77-class gap as promising, not settled. On a repeat of identical inputs Jev’s accuracy moved slightly, so the last digit of any figure here is noise.

The summary: on these three tasks a decision model was much faster and cheaper than a small chat model, and at least as accurate. That says nothing about frontier models, which we did not test.

Reproduce it. The harness, the pinned model versions and every run record are open source in system-one-decision-bench (Apache-2.0). Rerun it with your own API keys and check our numbers.

A popular design is a cascade: let the fast model decide, and send the call to a bigger model only when its probability falls below a threshold. We tested it with Jev first and Haiku as the fallback, sweeping the threshold from 0.50 to 0.99.

It barely helped. At the thresholds we would actually pick (the cheapest that kept accuracy within a point of the best), only 5.4%, 0.2% and 0% of calls were escalated on the three tasks. The one gain was on the 77-class task: +1.4 points over Jev alone (interval +0.4 to +2.6) at about 1.14 times Jev’s cost. We picked that threshold on the same 500 examples we measured on, so the real gain is probably smaller.

Raising the threshold made things worse. On the 77-class task accuracy fell from 80.4% at a threshold of 0.50 to 75.8% at 0.99, close to Haiku’s 75.0%. The reason is simple: the fallback was no more accurate than Jev. A cascade only pays when the second model is stronger than the first, and we only tested a weaker one. So this result says nothing about a cascade with a stronger fallback.

List your LLM calls and run the four questions. Pick the highest-volume call that passes, then try it on a few hundred of your own labelled examples before you change anything. Measure accuracy, median and p95 latency, and cost per 1,000 decisions on your data, not ours.

Keep the model behind an interface so you can swap it, pin the exact model version in config, and look first at your labels: if your own team would disagree about the right answer, no model will fix that. For the code, see the integration guide. We keep a provider abstraction starter for the interface part, and the benchmark harness above is a working example of the same pattern.

If you would rather not run this yourself, that is what our decision-layer assessment is for. We audit your LLM calls with the four questions above, test the strongest candidates on a sample of your own data, and hand back accuracy, latency and cost per 1,000 decisions with a clear recommendation, including when the answer is “keep the chat model”. The scope and price are fixed before we start. Book a call to scope it.