Which of your LLM calls
are really decisions?
List the LLM calls in your pipeline and answer five yes/no questions about each. You get a ranked list of the calls worth testing on a faster, cheaper decision model, and the ones that should stay on a chat model.
The scoring runs in your browser. No account is required, and nothing is sent to Dwayo unless you choose to export the report or request a review.
| LLM call | Closed set of answers? | Text or JSON input? | Latency matters? | High call volume? | Needs an explanation? | Verdict |
|---|
Summary
Ranked calls
Many “LLM calls” are
really small decisions.
Between a user’s message and the final reply, an agent often asks a model to route a ticket, flag a message, score urgency or pull out a field. Each has a short list of valid answers, and each runs on nearly every request.
Teams usually send these to the same chat model that writes the reply, then parse free text back into a label. A model built to return a typed answer with a probability can be much faster and cheaper on exactly these calls. In our benchmark, one such model had about 3x lower median latency and about 2x lower list-price cost than a small chat model, with accuracy at least as good on the three tasks we tested. This audit finds which of your calls are worth that test.
Five questions
about every call.
Answer yes or no for each call in your pipeline. The first two are hard requirements.
Can you list every valid answer in advance: a set of labels, a scale, or yes/no? If the output must be free text, such as a drafted reply or a summary, it is not a decision. Required.
Is the input text or structured data? Decision models of this kind generally accept text only, so images, audio and video need a different approach. Required.
Is a user or another service waiting on the answer, so that saving most of a second is felt? Background batch jobs rarely care.
Does the call run on a large share of requests? A saving of a fraction of a cent per call only matters at volume.
Does a person, an auditor or a customer need the model to say why? These models return a value and a probability, not a reason, so a yes here lowers the verdict one tier.
How the
verdict is decided.
Three steps from five answers to one verdict per call.
keep chat model whatever else is true.4 is a strong candidate, 3 is worth testing, and 2 or fewer means keep the chat model, because there is little to gain.The verdict is a screening aid, not a result. Accuracy still has to be measured on your own labelled examples before any call moves.
Two calls in one pipeline,
two different verdicts.
Route a support ticket to the right team
- Closed set: yes, a handful of team names
- Text input: yes, the ticket body
- Latency: yes, the customer is waiting for the first response
- Volume: yes, every ticket
- Explanation: no
- Result: 4 of 4, a strong candidate. Test it first.
Approve or reject a refund, and explain why
- Closed set: yes, approve or reject
- Text input: yes
- Latency: no, it can take a minute
- Volume: no, a small share of tickets
- Explanation: yes, the customer is told the reason
- Result: 2 of 4 with an explanation needed. Keep the chat model.
What this audit
doesn’t tell you.
A fast screen, not a benchmark. It does not capture:
- Whether any model is accurate enough on your data
- How good your labels are: overlapping labels cap accuracy for every model
- Real latency and cost, which depend on your inputs and provider
- How a confidence threshold or fallback model would perform
- Compliance or audit requirements beyond the explanation question
- Integration effort and how the model fails in production
When to run
this audit.
- Your LLM bill or latency is driven by high-volume small calls.
- You are building an agent and deciding which steps need a chat model.
- You are weighing a decision model or classifier against a chat model.
- You want a defensible shortlist before spending time on a benchmark.
- You are reviewing a pipeline after a cost spike.
- You need a starting point for a decision-layer assessment.
Before you move
a call.
How is each call scored?
Two questions are hard requirements: the answer must come from a closed set you can list in advance, and the input must be text or JSON. If either is a no, the call stays on a chat model. For the rest, each yes on latency sensitivity and call volume adds a point to a score out of four. Four is a strong candidate, three is worth testing, and two or fewer should stay where they are. If the call needs the model to explain its answer, the verdict drops one tier.
Does a strong candidate mean a decision model will be accurate enough?
No. The audit only says a call has the right shape. Whether a model is accurate enough has to be measured on a few hundred of your own labelled examples, looking at accuracy, median and p95 latency, and cost per 1,000 decisions. The exported report includes a measurement plan for that.
Is this only for one particular model?
No. The same questions apply to any model or system that returns a typed answer from a fixed set: a System One decision model, a fine-tuned classifier, or a rules-plus-embeddings approach. We benchmarked one System One model against a small chat model in a separate write-up, linked below.
Why does needing an explanation lower the verdict?
Decision models return a value and a probability, not a reason. If a person, an auditor or a customer needs the model to say why, a decision model alone will not provide it, so the call usually needs a chat model or a separate way to produce the explanation.
How many calls should I list?
List every distinct place your pipeline asks a model something, usually 5 to 20. Include the small calls such as routing, flagging, scoring and extraction, since those are the ones most likely to be decisions.
Does Dwayo see the calls I enter?
The scoring runs in your browser. Your call names are only shared with Dwayo if you choose to export the report or request a review through the lead-capture flow, and then only a short summary is sent. See our privacy policy.
A shortlist is a starting point
for a test, not a decision.
A Dwayo engineer can run a decision-layer assessment: your calls audited, the best candidates tested on your data, and a clear recommendation, including when the answer is to keep the chat model.