Cutting inference cost on an
underwriting pipeline that outgrew its budget.

Veridian's underwriting team had an LLM reading loan application documents and extracting structured risk signals — a strong use case, and it worked well in staging. Each call cost a fraction of a cent, so nobody modeled what it would look like once every application, every amendment, and every re-submission ran through the same pipeline.

By the time it reached production volume, the pipeline was making hundreds of thousands of calls a day, almost all of them at the same model tier regardless of how straightforward the document was. The monthly inference bill had become a line item finance was asking about — and growing faster than application volume, because retries and long-context re-reads were quietly adding up underneath it.

None of this needed a different LLM provider — it needed the pipeline to stop treating every document identically:

• Model routing by document complexity — a cheaper, faster model handled the high-volume, low-ambiguity majority of documents; the top-tier model was reserved for the minority flagged as genuinely hard to parse.
• Context trimming — only the sections of a document relevant to the extraction task were sent on each call, instead of the full document text every time.
• Response caching on repeat submissions — re-submitted or amended applications that hadn't materially changed reused prior extraction results instead of re-running the full call.
• Capped, backed-off retries with cost visibility — failures stopped silently multiplying cost, and per-call cost became a tracked metric in the same MLOps dashboards monitoring latency and error rate.

This is the same failure mode we write about in why LLM inference costs blow up at scale, and the kind of problem our LLM integration and MLOps work is built to catch before the invoice does.