One interface,
any provider underneath.
LiteLLM gives every model provider the same interface, so routing a request to a cheaper model, a different vendor, or a fallback during an outage is a configuration change — not a rewrite. Paired with Redis for response caching, it's how we keep inference cost from scaling 1:1 with request volume. It's a gateway layer, not an orchestration framework — a distinction that matters once it's sitting underneath an agent system rather than a simple chatbot, and one we're careful not to blur.
Serving UK and EU clients: GDPR, EU AI Act, and data residency are covered on our Trust & Safety page.
The provider abstraction,
not hand-rolled.
LiteLLM is an open-source proxy and SDK that normalizes well over a hundred model providers and endpoints behind one OpenAI-compatible API format — call Anthropic, OpenAI, a self-hosted open-weight model, or a dozen others through the same request shape, and switch between them with a config change rather than a code change. It can run as a standalone gateway service sitting in front of your application, or as a lightweight library imported directly into it; either way, it's a routing and translation layer, not an orchestration framework — it doesn't decide what an agent does next or hold conversation state the way LangGraph or CrewAI do. It sits below that layer, so a routing decision an orchestration framework makes ("use a cheaper model for this step") gets executed as a LiteLLM request underneath it, and the orchestration framework never needs its own logic for which provider SDK to call.
It's open-source (self-hosted, run as your own gateway) with a separate paid Enterprise tier that adds SSO, audit logs, and dedicated support for teams running it at larger scale — most of the builds we do run the open-source core directly, since the audit and SSO features matter most to larger organizations with their own compliance requirements around who can change routing configuration. It's the concrete implementation of a pattern we recommend regardless of which specific gateway tool gets used: a thin internal interface between application code and whichever provider's SDK is doing the actual work, so a provider's breaking API change, a price shift, or an outage gets absorbed in one place instead of requiring a rewrite across every call site.
See our note on avoiding AI vendor lock-in for why this architectural choice matters more than which specific provider you start with — the gateway is what makes switching providers later a configuration change rather than a migration project, and that's true whether the trigger is a price increase, a reliability problem, or simply a better model becoming available elsewhere.
Worth being honest about the tradeoff: a gateway in the request path is a new piece of infrastructure to run, monitor, and keep available — if it goes down, every model call behind it goes down too, which is the opposite of the reliability problem it's meant to solve. We run it with its own health checks and a direct-to-provider fallback path for the rare case the gateway itself is the thing that's unavailable, rather than treating it as infallible because it's the piece meant to catch everyone else's failures. That extra layer of operational responsibility is a real cost, not a footnote, and we weigh it honestly against the routing and caching benefits before recommending the gateway pattern for a given project's actual scale.
Routing and caching,
not just abstraction.
The abstraction layer is also where the cost-control and reliability logic lives, if it's set up to carry that weight rather than used as a pass-through that only translates request formats and leaves everything else to chance, which is the version that saves nothing and just adds a hop.
Simple, high-volume requests go to a cheaper model tier; harder requests route to a more capable one — configured once in the gateway's routing rules, applied automatically to every call site without touching application code, and adjustable later without a deployment.
Repeat or near-duplicate requests reuse a cached response instead of paying for a fresh model call every time — a meaningful saving on workloads with real repetition, and close to none on workloads that are genuinely unique each time, which is why we measure cache hit rate on actual traffic rather than assume a cache is paying for itself.
A primary provider timing out or rate-limiting falls through to a configured backup automatically, at the gateway level — the application code calling it never needs to know a failover happened, and the fallback chain itself gets tested on a schedule, not just configured once and left untouched until the day it's actually needed.
Cost per call becomes a tracked metric, broken out by model, by caller, and by feature, alongside latency and error rate in the same MLOps dashboards — not something discovered at month-end on a provider invoice after the spend has already happened.
Real inference bills,
brought back under control.
This is the exact technique set behind a real client engagement, not a hypothetical, and the gateway layer underneath a meaningful share of our LLM integration work generally — infrastructure that's invisible when it's working, which is the point, and that only becomes visible on a dashboard rather than in an outage when it's built right.
Model routing and caching cut per-document inference cost roughly 70% for a fintech underwriting pipeline — a real, disclosed-placeholder engagement, not a hypothetical number.
The broader service this sits inside — connecting models to your existing systems without new infrastructure or a provider-specific rewrite.
The monitoring layer that tracks cost, latency, and error rate once routing and caching are in place, so the gateway's own health is visible too.
Starter code (TS/Python) for a provider-abstraction layer, so a swap is a config change.
Before you
book a call.
The questions we get asked most about LiteLLM and multi-provider routing — answered straight, no sales pitch.
What does LiteLLM actually do?
It's an open-source proxy and SDK that gives well over a hundred model providers a single, OpenAI-compatible interface, so routing a request to a different model or provider is a configuration change rather than a rewrite of application code calling that provider's specific SDK. It can run as a standalone gateway in front of an application, or as a library inside it.
Is LiteLLM an orchestration framework, like LangGraph or CrewAI?
No — it's a gateway and routing layer, a different category entirely, and the two aren't competing choices. It doesn't decide what an agent does next or hold conversation state; it sits underneath an orchestration framework and executes the actual model calls that framework decides to make, with routing, caching, and failover handled at that lower layer rather than in the agent's own logic.
How does Redis fit in?
As the cache layer behind the router. Repeat or near-duplicate requests — the same document re-submitted, the same question asked twice — reuse a cached response instead of paying for a fresh model call. We measure actual cache hit rate on real traffic rather than assuming the cache is earning its keep, since the saving depends entirely on how much genuine repetition exists in a given workload.
What's a common mistake teams make setting this up?
Treating the gateway as a pass-through and never configuring fallback chains, which means a single provider outage still takes the whole system down — the exact failure mode the gateway exists to prevent, left unconfigured by default. A close second is never load-testing the gateway itself, so it becomes a new single point of failure instead of removing one, discovered only when it actually falls over under real traffic.
What have you shipped with this stack?
Model routing by request complexity and response caching are two of the core techniques behind our real, disclosed-placeholder case study on cutting LLM inference cost for a fintech underwriting pipeline by roughly 70% without changing the underwriting logic itself.
Tell us what
your inference bill looks like.
Book a 30-minute call — we'll tell you honestly whether your pipeline has room to shrink its per-call cost, and by how much.