Case Study — LLM Integration & MLOps

Cutting inference cost on an
underwriting pipeline that outgrew its budget.

Client: Veridian Capital (name changed) Industry: Fintech underwriting Timeline: ~5 weeks Related: LLM Integration, MLOps

A $0.002 call doesn't look like a budget line. Until it is one.

Veridian's underwriting team had an LLM reading loan application documents and extracting structured risk signals — a strong use case, and it worked well in staging. Each call cost a fraction of a cent, so nobody modeled what it would look like once every application, every amendment, and every re-submission ran through the same pipeline. It's a pattern we see across fintech lending generally: document volume scales with application volume, and inference cost quietly scales with both.

By the time it reached production volume, the pipeline was making hundreds of thousands of calls a day, almost all of them at the same model tier regardless of how straightforward the document was. The monthly inference bill had become a line item finance was asking about — and growing faster than application volume, because retries and long-context re-reads were quietly adding up underneath it.

What it was built on.

The pipeline as we found it:

Single top-tier LLM for every document Full document text on every call, no trimming No response caching Synchronous, uncapped retries on failure No per-call cost tracking

Inputs were scanned and digitally-submitted loan application documents, arriving through the client's existing intake pipeline. No new data source was introduced — the fix was entirely in how the existing documents were processed, not where they came from.

Routing by complexity, not by habit.

Document intake
→
Complexity classifier
→
Cheap model (majority)
→
Top-tier model (flagged minority)
→
Cost & cache tracking

Spend the expensive model only where it earns its cost.

None of this needed a different LLM provider — it needed the pipeline to stop treating every document identically:

  • Model routing by document complexity — a cheaper, faster model handled the high-volume, low-ambiguity majority of documents; the top-tier model was reserved for the minority flagged as genuinely hard to parse.
  • Context trimming — only the sections of a document relevant to the extraction task were sent on each call, instead of the full document text every time.
  • Response caching on repeat submissions — re-submitted or amended applications that hadn't materially changed reused prior extraction results instead of re-running the full call.
  • Capped, backed-off retries with cost visibility — failures stopped silently multiplying cost, and per-call cost became a tracked metric in the same MLOps dashboards monitoring latency and error rate.

Accuracy held as a gate, not an afterthought.

The constraint on every change was simple to state and hard to skip: extraction accuracy on the cheaper routing path had to match the top-tier model's accuracy on a held-out set of real documents before that path was allowed to carry production traffic. Routing decisions that traded accuracy for cost were rejected, not shipped with a caveat.

The bill stopped tracking volume 1:1.

Figures are approximate and rounded to protect the client's specific financials, per our NDA. Measured over the first month at full production routing.

Before
Per-document inference cost rising in lockstep with application volume
After
Roughly 70% lower inference cost per document, same extraction accuracy
Ongoing
Cost per call tracked and alerted on, not discovered at month-end

What stayed constant through the change.

Document handling and retention policies were untouched by the cost work — the routing and caching layer operates on the same access-controlled infrastructure the original pipeline used, and caching explicitly excludes documents flagged as containing updated or sensitive fields, so a cached result is never served for a submission that materially changed.

The routing logic and the cost dashboard.

At completion: the routing and caching service's source code, the complexity-classification logic, and the cost-tracking dashboard, documented so Veridian's own engineering team owns the ongoing tuning rather than depending on us for every adjustment.

Routing only works if the classifier is honest.

The entire saving depends on the complexity classifier correctly identifying which documents are actually simple. An overconfident classifier that routes a hard document to the cheap model doesn't save money — it produces a wrong extraction that costs more to catch downstream than the model tier would have. We treated the classifier itself as something to evaluate continuously, not a one-time decision.

If this looks familiar.

This is the same failure mode we write about in why LLM inference costs blow up at scale, the kind of problem our LLM integration, LLMOps & AgentOps, and MLOps work is built to catch before the invoice does, and relevant to document AI work in fintech specifically. It's the same production discipline behind all of our AI development services.

Get started

Tell us what
you're trying to build.

Book a 30-minute call — we'll tell you honestly whether your inference bill has room to shrink, and by how much.