Case Study — AI Agent Development

The agent wasn't wrong often.
It was wrong automatically.

Illustrative scenario. This is a pattern-level write-up of a recurring failure mode and its fix — not a write-up of one named client project. Stack, numbers, and timeline are representative of this class of engagement, not a specific client's exact figures.
Pattern: Internal operations automation Illustrative timeline: ~6 weeks Related: AI Agent Development, AI Agent Testing & Evaluation

The agent wasn't wrong often. It was wrong automatically.

A team had given an LLM agent a list of tools it could call to handle a recurring operational workflow — update a record, send a notification, escalate a case, close a ticket. In testing, it worked. In production, ambiguous requests started producing the wrong tool call: closing a ticket that should have escalated, updating the wrong field on a similar-looking record. The individual error rate looked small. The problem was that each wrong call executed immediately and silently — nobody found out until a person noticed the downstream effect.

What it was built on.

A representative setup for this kind of workflow automation — illustrative of the pattern, not a specific project's exact tools:

LLM with function/tool calling Internal REST APIs as tools No output schema validation No action-level permissions No human-in-the-loop checkpoint

The data the agent acted on came from the same internal systems the tools wrote to — ticket records, case notes, and notification queues. The gap wasn't the data; it was the absence of a check between the model's decision and the system actually executing it.

Where the gate goes.

Request
→
Agent reasoning
→
Schema & whitelist gate
→
Tool execution
→
Human review (irreversible actions)

Guardrails before more capability.

Rather than trying to make the model smarter at disambiguating intent, we constrained what it was allowed to do when it wasn't sure:

  • Strict output schemas on every tool call — the agent's output had to validate against a defined structure before execution, rejecting malformed or out-of-bounds calls outright.
  • Action whitelists per workflow — the agent could only call the specific tools relevant to the task it was given, not the full tool list, cutting the space for a wrong-but-valid call.
  • Human-in-the-loop on irreversible actions — closing a ticket, issuing a refund, or anything hard to undo required a confirmation step instead of executing on the model's first decision.
  • Confidence-aware escalation — ambiguous requests the agent wasn't confident about routed to a person instead of taking the best guess.

How this class of fix gets tested.

For this pattern, the right test set is adversarial: ambiguous and boundary-case requests deliberately designed to tempt the agent into the wrong tool call, plus a representative sample of genuine historical requests. Acceptance criteria are two-sided — the agent must still complete the clear-cut majority of requests without unnecessary escalation, while rejecting or escalating the ambiguous minority instead of guessing. Both sides are measured; optimizing only for "fewer escalations" reintroduces the original failure.

What changed once actions were gated.

This is the pattern of improvement we typically see once schema enforcement and gating go in — illustrative, not a specific client's measured figures.

Before
Wrong actions executed silently, caught only after the fact
After
Malformed or out-of-scope calls rejected before execution
Ongoing
Irreversible actions confirmed, not assumed

Permissions scoped per workflow, not per agent.

The whitelist pattern extends naturally into access control: an agent deployed for one workflow gets exactly the tool permissions that workflow needs, not a shared credential covering everything the underlying API can do. That boundary is what keeps a single misconfigured workflow from becoming a system-wide blast radius.

What a client walks away with.

For this class of engagement, the deliverables are the schema definitions and whitelist configuration, the adversarial test set used to validate them, and documentation of which actions require human confirmation — owned by the client's team, not held by us.

If this looks familiar.

This is a pattern-level breakdown of what our AI agent development work is built to prevent, and exactly the class of problem our AI agent testing & evaluation service exists to catch before launch. If your agent is taking occasional wrong actions in production, the fix usually isn't a smarter model — it's tighter constraints on what it's allowed to do. The same discipline runs through all of our AI development services.

Get started

Tell us what
your agent is allowed to do.

Book a 30-minute call — we'll tell you honestly where the gaps are.