A team had given an LLM agent a list of tools it could call to handle a recurring operational workflow — update a record, send a notification, escalate a case, close a ticket. In testing, it worked. In production, ambiguous requests started producing the wrong tool call: closing a ticket that should have escalated, updating the wrong field on a similar-looking record. The individual error rate looked small. The problem was that each wrong call executed immediately and silently — nobody found out until a person noticed the downstream effect.
The agent wasn't wrong often.
It was wrong automatically.
The agent wasn't wrong often. It was wrong automatically.
What it was built on.
A representative setup for this kind of workflow automation — illustrative of the pattern, not a specific project's exact tools:
The data the agent acted on came from the same internal systems the tools wrote to — ticket records, case notes, and notification queues. The gap wasn't the data; it was the absence of a check between the model's decision and the system actually executing it.
Where the gate goes.
Guardrails before more capability.
Rather than trying to make the model smarter at disambiguating intent, we constrained what it was allowed to do when it wasn't sure:
- Strict output schemas on every tool call — the agent's output had to validate against a defined structure before execution, rejecting malformed or out-of-bounds calls outright.
- Action whitelists per workflow — the agent could only call the specific tools relevant to the task it was given, not the full tool list, cutting the space for a wrong-but-valid call.
- Human-in-the-loop on irreversible actions — closing a ticket, issuing a refund, or anything hard to undo required a confirmation step instead of executing on the model's first decision.
- Confidence-aware escalation — ambiguous requests the agent wasn't confident about routed to a person instead of taking the best guess.
How this class of fix gets tested.
For this pattern, the right test set is adversarial: ambiguous and boundary-case requests deliberately designed to tempt the agent into the wrong tool call, plus a representative sample of genuine historical requests. Acceptance criteria are two-sided — the agent must still complete the clear-cut majority of requests without unnecessary escalation, while rejecting or escalating the ambiguous minority instead of guessing. Both sides are measured; optimizing only for "fewer escalations" reintroduces the original failure.
What changed once actions were gated.
This is the pattern of improvement we typically see once schema enforcement and gating go in — illustrative, not a specific client's measured figures.
Permissions scoped per workflow, not per agent.
The whitelist pattern extends naturally into access control: an agent deployed for one workflow gets exactly the tool permissions that workflow needs, not a shared credential covering everything the underlying API can do. That boundary is what keeps a single misconfigured workflow from becoming a system-wide blast radius.
What a client walks away with.
For this class of engagement, the deliverables are the schema definitions and whitelist configuration, the adversarial test set used to validate them, and documentation of which actions require human confirmation — owned by the client's team, not held by us.
Tell us what
your agent is allowed to do.
Book a 30-minute call — we'll tell you honestly where the gaps are.