Stopping an agent from taking
the wrong action, at scale.

A team had given an LLM agent a list of tools it could call to handle a recurring operational workflow — update a record, send a notification, escalate a case, close a ticket. In testing, it worked. In production, ambiguous requests started producing the wrong tool call: closing a ticket that should have escalated, updating the wrong field on a similar-looking record.

The individual error rate looked small. The problem was that nothing was checking the agent's output before it executed — there was no schema constraining what a valid action looked like, and no distinction between actions that were safe to let the agent take freely versus actions that needed a second check first. A low per-request error rate, multiplied across thousands of automated calls, was still a steady stream of wrong actions nobody was catching until a customer or colleague noticed.

Rather than trying to make the model smarter at disambiguating intent, we constrained what it was allowed to do when it wasn't sure:

• Strict output schemas on every tool call — the agent's output had to validate against a defined structure before execution, rejecting malformed or out-of-bounds calls outright.
• Action whitelists per workflow — the agent could only call the specific tools relevant to the task it was given, not the full tool list, cutting the space for a wrong call.
• Human-in-the-loop gates on write operations — anything that changed state (closing, escalating, updating a record) required a lightweight confirmation step rather than executing silently.
• An eval suite of ambiguous, adversarial prompts run before every deploy, specifically targeting the kind of unclear requests that had caused the original failures.

This is a pattern-level breakdown of what our AI agent development work is built to prevent. If your agent is taking occasional wrong actions in production, the fix usually isn't a smarter model — it's tighter constraints on what it's allowed to do.