How to wire a System One model
into an agent pipeline.

Model identifiers differ between providers, a hosted model can change under you, and in our own runs identical inputs gave slightly different accuracy on a repeat. So the first rule is that your agent never calls a vendor SDK directly. It calls a Decider that returns a plain Decision, and the vendor sits behind it. This is the same idea as in avoiding AI vendor lock-in.

@dataclass(frozen=True)
class Choice:
    prompt: str
    options: tuple[str, ...]          # the closed set of valid answers

@dataclass
class Decision:
    label: str | bool | int | None
    probabilities: dict[str, float] | float | None
    latency_ms: float
    cost_usd: float
    model_id: str                      # the version that actually answered
    error_kind: str | None = None      # None, "invalid" or "api"

    @property
    def confidence(self) -> float | None:
        p = self.probabilities
        if p is None:
            return None
        if isinstance(p, float):       # yes/no: probability of yes
            return max(p, 1.0 - p)
        return max(p.values())         # choice/score: top outcome

class Decider(Protocol):
    def decide(self, question, input) -> Decision: ...

Three things matter here. The set of valid answers is part of the question, so a bad answer can be detected. Every decision carries its latency, cost and the model version that produced it, so you can audit it later. And failures are values, not exceptions that vanish: error_kind is "invalid" when the model answered something unusable and "api" when the call itself failed.

Retry transport errors and the status codes that mean “try again” (429 and the 5xx family), back off exponentially with jitter, and honour a Retry-After header if you get one. Do not retry a bad request. Record how many retries a call needed, and measure latency on the successful attempt only, so backoff time does not pollute your latency figures.

RETRY_STATUS = {429, 500, 502, 503, 504, 529}

def with_retries(fn, max_retries=4, base_delay=0.5):
    for attempt in range(max_retries + 1):
        try:
            return fn(), attempt           # result and how many retries it took
        except RetryableError as e:
            if attempt == max_retries:
                raise
            delay = e.retry_after if e.retry_after is not None else base_delay * 2**attempt
            time.sleep(delay + random.uniform(0, base_delay))

Keep concurrency modest. We ran four calls in flight against a hosted early-access model, which is a polite default until you know your limits.

Run a few hundred of your own labelled examples through the interface and record the same five numbers every time:

• Accuracy with a confidence interval, because a few hundred examples is a small sample.
• Median and p95 latency, measured from your client with the network included. A median hides the slow tail your users will notice.
• Cost per 1,000 decisions, from the token usage each response reports times the published price, labelled as list price and not as spend.
• Invalid and error rates, counted separately.
• Calibration, to see whether 90% confidence really means about 90% correct. In our run, Jev’s top confidence band was 98.4% stated and 93.8% actual, so do not treat probabilities as certainty.

Cache each decision keyed by model version, a hash of the question text and the example ID. Reruns become cheap, and editing a prompt cannot silently reuse stale answers. The harness does all of this for Banking77 and a safety dataset, and its report shows what the output looks like.

Connecting models to a product’s real systems is our LLM integration work. If you want help deciding which calls to move and proving it on your data, that is the decision-layer assessment described there. The benchmark post has the measured results, and the provider abstraction starter has a general version of the interface above.