Write the runbook before
the incident, not during it.
Select the AI failure modes relevant to your system — hallucinations, cost spikes, provider outages, prompt injection, retrieval failures, agent loops, data leakage, and latency — and generate a tailored runbook with detection signals, containment steps, rollback, and a postmortem checklist.
No account is required, and nothing is sent to Dwayo unless you choose to save, export, or request a review.
System details
AI incidents don't look like
typical outages.
A traditional incident runbook assumes a service is either up or down. AI systems fail in subtler ways: a model keeps responding, but the responses are wrong; a provider degrades instead of going fully offline; an agent keeps running and keeps spending; a retrieval system quietly returns the wrong context for a subset of users.
Without a runbook written for these specific failure modes, the first response to an AI incident is often improvised — which costs time, and often means the same mistake gets investigated the same slow way the next time it happens.
Eight AI-specific
failure modes.
The model produces a confident but wrong or fabricated answer.
Spend rises sharply from retries, routing, volume, or a runaway agent loop.
The model provider is down, rate-limiting, or returning degraded responses.
Untrusted content manipulates the model into ignoring instructions or leaking data.
RAG returns stale, irrelevant, or improperly permissioned context.
An agent repeats steps, retries excessively, or takes unintended actions.
Sensitive data appears somewhere it shouldn't — a response, a log, a third party.
Response times climb past what the product experience can tolerate.
How to use the
generated runbook.
- Select every incident type that's plausible for your system — err toward including more, not fewer.
- Replace generic detection signals with your actual dashboards, alert names, and query examples.
- Confirm the rollback steps actually work before an incident forces you to find out.
- Store the runbook somewhere the on-call engineer can reach in under a minute, not buried in a wiki.
- Run a tabletop exercise against at least one incident type before relying on it live.
- Update the runbook after every real incident with what actually happened and what was missing.
What this runbook
doesn't replace.
- Your organization's broader security incident response plan
- Legal or regulatory breach-notification procedures
- Customer communication and support escalation processes
- Infrastructure and application incident response for non-AI failures
- A staffed, tested on-call rotation
- System-specific dashboards, alerts, and rollback tooling — add these yourself
When to build
this runbook.
- Before an AI system reaches production traffic.
- After adding a new provider, agent capability, or data source.
- Following a near-miss or an incident that exposed a missing procedure.
- When onboarding a new on-call engineer to an AI system.
- During a security or operational readiness review.
- Before scaling an existing AI feature to materially more traffic.
Before this goes
on-call.
What incident types does this runbook cover?
Hallucinated or incorrect output, cost spikes, model provider outages, prompt injection, retrieval failures, agent action loops, data leakage or privacy exposure, and latency degradation. Select the ones relevant to your system to generate a tailored runbook.
Is this a substitute for a broader security incident response plan?
No. This runbook focuses specifically on AI-system failure modes. It should sit alongside, not replace, your organization's broader security incident response, legal, and customer-communication processes — especially for data breaches, which often carry legal notification requirements this tool does not cover.
Who should use this runbook during an incident?
The on-call engineer or incident commander responding to an AI-system issue. It's designed to be usable under time pressure — detection signals and immediate steps are deliberately short and specific.
Should this be written before or after an incident?
Before. A runbook written during a live incident is usually too late to help with that incident. Build it while things are calm, store it somewhere the on-call team can reach quickly, and update it after every real incident.
Does the generator know about my specific architecture?
No. It provides a structured starting point based on common AI failure modes. Add system-specific detection queries, dashboard links, rollback commands, and contacts before treating it as complete — see our LLMOps & AgentOps page.
Does Dwayo see what I enter?
The runbook is generated in your browser. Your answers are only shared with Dwayo if you choose to save a result, export it through the lead-capture flow, or request a review. See our privacy policy.
A runbook is only useful
if it's been tested.
Run a tabletop exercise against your highest-risk incident type. If rollback, monitoring, or escalation paths aren't in place yet, a Dwayo engineer can help close the gaps before they're needed live.