Make AI agent quarantine remediation replay-safe

Quarantine protects an intended action during uncertainty. Remediation restores its dependencies. Those are linked operations, not the same operation.

A common failure begins when an automation workflow receives a quarantine event, runs a CRM repair, and reports success. If the workflow crashes after repair but before acknowledgement, the platform retries. Without stable remediation identity and idempotent repair, it may create another CRM record, repeat a backfill, open another ticket, or emit another release request.

The risk grows when teams conflate process completion with action authority. A remediation workflow can finish while backfill remains partial, a cache is stale, customer intent has changed, approval expired, or the original follow-up was canceled. “Workflow succeeded” should trigger verification, not direct resume.

Start with four durable identities: the source incident, the proposed customer action, the remediation request, and each workflow run or attempt. Repeated delivery reuses the same remediation request. A new attempt may execute, but its side effects use stable external keys and converge on one logical repair.

Keep handoffs durable. When the gate creates quarantine, write an outbox event in the same transaction or use equivalent guarantees. When automation accepts a remediation request, persist external identity before starting side effects. When repair completes, publish evidence through an outbox rather than acknowledging first and hoping publication succeeds.

The gate owns release. It verifies destination identity, count, mapping, authority, event time, backfill, and the exact read surface the agent will use. It reevaluates the proposed action and approval. If still appropriate, it issues a short-lived receipt bound to one action.

Replay safety is therefore a system property across state stores, queues, APIs, workflows, and agent workers. This guide defines the identities, transitions, failure boundaries, and tests needed to make the whole loop converge.

Define terminal states for action and remediation separately. A canceled action remains canceled even if remediation succeeds. A completed remediation can later receive more evidence without becoming running again. Reopened source incidents create a new evaluation or remediation generation instead of mutating history ambiguously.

Protect transitions with database constraints or compare-and-set versions. Unique constraints can enforce one remediation per action, incident, dependency set, and generation; one active release per action; and one destination external key per logical repair. Application checks alone can race.

A lease is not an identity. Leases expire so another worker can continue the same request. Store attempt count, heartbeat, worker reference, and last error for operations, but never derive a new logical remediation merely because the worker changed.

Separate event time from processing time. A backfill can arrive now for an older customer event. Verification should reconstruct business order and determine whether the proposed follow-up is still appropriate. Current ingestion does not make old intent current.

Keep user explanation close to durable state. The person should see what action is blocked, which dependency is uncertain, what repair is running, whether human input is needed, and what evidence allowed release. Replay should update one explanation thread rather than create repeated alerts.

The proposed action record should retain customer scope, purpose, content or change version, required dependencies, approval reference, current state, state version, quarantine reasons, terminal result, and timestamps. The remediation record should retain stable request key, incident and action references, dependency generation, desired repair, lifecycle, lease owner and expiry, attempt count, external side-effect references, last evidence, and outbox state.

Verification should be its own durable result. Store which destination objects, associations, field authorities, mapping version, event window, backfill coverage, and read surfaces were checked. Include observed time and evidence digests. A release can then cite immutable verification results instead of relying on mutable current rows.

The release receipt should contain action identity and version, verified dependency references, policy and mapping versions, actor or system authority, issue and expiration time, exception scope if any, and consumption state. The commit worker uses a unique consumption key so repeated receipt delivery returns the same external action result.

Use an append-only transition or audit table even if current state is materialized for speed. During incident review, teams must reconstruct which observation led to quarantine, which attempt repaired state, which verification cleared each blocker, and why release remained valid at commitment time.

Set operational acceptance criteria before launch: no duplicate logical repairs under concurrent delivery, no duplicate customer action after any injected crash, bounded lease recovery, one active release, monotonic terminal state, and a user explanation that remains coherent across retries. Anything less leaves correctness dependent on timing luck.