Skip to content

Resilience Engine

The Resilience Engine runs agentic failure-mode and effects analysis (FMEA) over your live environment: it reasons about how your infrastructure could fail before it does, and how far each failure would spread. Requires the Enterprise plan.

How it works

Each run is a hypothetical "what if" analysis, not an incident report:

  1. Seed hypotheses. Candidate failure modes are derived from your knowledge graph — single points of failure, toxic combinations, and the most depended-on services — each with a projected blast radius over real dependency edges.
  2. Investigate live. An agent gathers fresh, read-only evidence for each hypothesis: recent error logs, latency and saturation metrics, current resource configuration, capacity and quota headroom.
  3. Report. Findings are ranked with a severity, a confidence score, the specific failure mode, and the cascade story — the trigger, which dependent services break first, how it propagates, and the user-facing impact. Every finding cites the live commands that were actually run and what they showed.

Scope is reliability and availability only — capacity exhaustion, missing redundancy, scaling failures, dependency loss, data-durability gaps. Security findings are handled by Pulse's security domain, not here.

Read-only guarantee

The engine never changes infrastructure and never injects faults. Every command it can issue is checked at the verb level before execution: only observe operations (describe, list, get, query, read, logs) are allowed, and anything that would create, delete, modify, scale, or restart a resource is hard-blocked. The analysis is purely hypothetical reasoning backed by read-only evidence.

Using it

Open /resilience.

  • Run analysis. The run button starts an on-demand scan. Status is shown while it runs; findings appear when it completes. Runs are bounded — up to 12 hypotheses per run (default 5).
  • "What if… " picker. Type a resource name (two or more characters) to pick any mapped resource and ask "what if this fails?". You get an instant structural scenario projected from the dependency graph — affected downstream services and an estimated severity — without waiting for a run. From there you can trigger a deep investigation of that one resource, which runs the full evidence-gathering analysis seeded on it.
  • Review findings. Findings show severity, confidence, the what-if cascade, and the evidence trail. The detail view includes a failure tree you can expand node by node to walk the real dependents of the failure point. Filter by severity, status, or degraded-evidence only.

Where results appear

  • The Resilience page (/resilience) — KPIs, severity matrix, and the findings list.
  • Pulse — resilience findings are Pulse findings (SRE domain, resilience category) and follow the same status lifecycle (open, acknowledged, in progress, resolved, dismissed).
  • Mission Control — the Failure modes KPI and the Resilience panel.

Configuration

Setting Where Notes
Plan — Requires the Enterprise plan; other plans see an upgrade notice
Hypotheses per run Run request 1–12, default 5
Finding status Findings list Shared with the Pulse lifecycle

Troubleshooting

  • Run does not start (403) — resilience analysis is an Enterprise feature; check your plan.
  • Run finishes with no findings — the agent could not back any hypothesis with live evidence, or no real reliability risks were found. Findings are never fabricated to fill the list.
  • "What if" picker finds nothing — the resource is not in the knowledge graph yet; verify the graph is populated and the mapper has run.
  • Analysis seems shallow — a sparse or stale graph limits hypothesis quality; fix graph coverage first.