Resume an agent after a partial tool failure

Freeze automatic retries

Stop the affected run, its retry timer, and any competing worker lease. Capture the run ID, current attempt ID, workflow version, tool name, normalized input fingerprint, timestamps, and trace context. If the system has already scheduled another attempt, cancel or quarantine it before investigation.

• Do not delete the failed attempt.
• Do not mutate its recorded input or output.
• Do not launch a fresh run under a different identity.

Restate the intended outcome

Write one sentence describing the user-visible result, independent of implementation. “The customer receives one appointment confirmation” is safer than “retry sendMessage.” The first statement lets the recovery process consider existing messages, alternate channels, or cancellation; the second presumes a specific action must repeat.

Confirm that the user has not changed or revoked the intent while the run was paused. A recovery after delayed approval, cancellation, or new context may need to stop rather than resume.

Build an evidence ledger

List every completed and attempted step in order. For each external effect, record its stable key, provider operation ID, request timestamp, response status, emitted events, observed postcondition, and current classification. Separate model-produced claims from system evidence.

The ledger should answer what is known without reading free-form logs. If two sources disagree, preserve both observations and mark the effect unknown until a stronger source resolves it.

Reconcile uncertain effects

Query the system that owns the effect. Search a payment provider by idempotency key, a messaging provider by message ID, a database by a unique intent column, or a deployment platform by commit and service. For browser actions, reopen the target and inspect the expected postcondition: the created item, changed status, confirmation page, or outbound event.

If the target offers no reliable lookup and the effect is costly or irreversible, route the run to a human. An honest unknown state is safer than an automatic duplicate.

Select the last safe checkpoint

A safe checkpoint is not merely the last serialized state. It is the newest boundary where all earlier effects are either confirmed complete or intentionally compensated, and all later work can be recomputed without violating the original intent.

• Reuse stable outputs that remain valid.
• Refresh observations whose freshness window expired.
• Do not rerun completed irreversible actions.
• Pin the workflow and prompt version or run an explicit migration.

Create a linked recovery attempt

Resume under a new attempt ID linked to the original run and failed attempt. Carry forward the evidence ledger, selected checkpoint, operator or policy that authorized recovery, and the reason each step will be reused, retried, skipped, or reviewed.

At every service boundary, reuse the original operation identity for the same intended effect. A new attempt identifier is useful for observability; a new side-effect key can accidentally turn the same intent into a second action.

Verify and close the loop

After resume, verify the user-visible outcome and every repaired postcondition. Emit a recovery receipt containing the original run, recovery attempt, failed tool, checkpoint, reconciled effects, final state, and links to traces. Notify the user only when the final state is accurate and actionable.

Finally, turn the incident into a reusable rule: add a missing idempotency key, improve a provider lookup, shorten a lease, capture a new event, or require approval for that ambiguous class. Recovery should make the next run safer.