How to Recover from Mid-Workflow Tool Failures in Slack AI Agents

When a tool call times out after dispatch, treat the step as ambiguous: look the effect up before you retry, and never rerun a successful write to fix a missing reply. Recover forward, not backward. Verify what happened, finish what is missing, and hand the choice to a person when the risk is real. Undoing a created ticket or a posted reply is itself a new write that can fail, and a customer may already have seen the first one.

Why a multi-tool run is not one transaction

A Slack agent can finish three of five tools and still leave the thread looking broken. A run searches Kipwise, drafts a support reply, creates a ticket and posts a thread update. Each call can succeed, fail, time out after dispatch, or return an error after the remote system already applied the change. A research paper on verified tool calls names these non-atomic failures: timeouts after dispatch, delayed visibility, and partial state updates. In a simulated tool environment, a wrapper that checks the postcondition before retrying cut duplicate actions while keeping task success comparable, which is why a plain retry loop is the wrong default.

Slack's agent design guidance asks builders to preserve completed work when something goes wrong mid-task, tell the user what succeeded, and offer clear next steps such as retry, modify, or escalate. Leaving the session stuck in a loading state, or discarding three successful steps because the fourth timed out, breaks that. AWS durable-execution guidance on idempotency and retries treats at-least-once retry as safe only when the side effect is idempotent or protected by an idempotency key. A Slack agent that retries create_ticket after an ambiguous timeout without checking whether the ticket exists will create duplicates under load.

This is a different layer from Events API intake retries. Slack may redeliver an app_mention when your endpoint misses the three-second ack window, and claim-store design at the Events API boundary handles that. Mid-workflow recovery starts after you accepted the mention and began a multi-tool run.

Keep a run ledger outside the model

Treat each agent run as a durable workflow with a run ID and ordered steps. Before the first mutating tool, persist:

1. Run identity: Slack team_id, channel, thread_ts, requesting user, and a stable run_id.
2. Step plan: the intended tools in order, with a step ID for each.
3. Step state: pending, succeeded, failed, ambiguous, or skipped, plus timestamps.
4. Side-effect evidence: external IDs returned by tools (ticket ID, PR URL, knowledge page ID, message ts).
5. Delivery state: whether the Slack reply for this run was accepted (pending, sent, failed).

Kipwise Agent is designed as a thread-native work agent: a teammate mentions it once and continues with ordinary replies; applicable actions use the requesting user's permissions; it cites sources it actually opened; it refuses to guess; consequential changes stay explicit. Those behaviors only hold if each tool step leaves recoverable evidence when the process dies mid-run.

Keep the ledger outside the model context window. The model may summarize incorrectly after a crash. Your recovery code must read structured step state, not a regenerated plan.

Verify external postconditions before retrying

A timeout is not proof the side effect failed. A connector that returns HTTP 504 after 30 seconds may have already created the ticket. The verified tool calls paper found that verify-before-retry wrappers reduce duplicate actions compared with retry-only baselines under injected non-atomic faults.

Do not ask the model whether the ticket got created as your only check. The model is not the system of record. The ticket system is.

Generate the idempotency key once inside the durable step and pass the same key on every attempt, as AWS durable-execution idempotency guidance recommends for side-effecting APIs. If the provider returns a duplicate-request success, treat it as success, not failure.

Scope that key to the intent, not to the run. A key built from run_id plus step protects one run against its own retries, and it does nothing when the requester sees a quiet thread, assumes the agent failed, and mentions it again. That starts a new run with a new run_id, and the second ticket goes through. We recommend deriving the key from what the step is meant to achieve, for example the thread, the tool, and a hash of the arguments that matter. When a matching ticket already exists, the agent should answer with its link and ask whether the person wants another.

Recover forward and let a person choose to undo

The textbook answer to a half-finished workflow is a compensating transaction, a step that undoes completed work. Microsoft's compensating transaction pattern is candid about the limits: undoing a step can be harder than restoring the original state, and a compensating step is itself an operation that can fail and must be safe to retry. For a Slack agent the limits are tighter. A reply a customer has read, a ticket a teammate was notified about, and an email already delivered cannot be unseen, and deleting them is one more visible write.

So the default is forward recovery: verify what happened, finish the missing steps once, and tell the thread what is still open. Keep undo for invisible internal writes. For anything else, a person decides. The same Microsoft page makes that call: when one step of a booking fails, offering the customer an alternative can beat cancelling the rest, and that decision belongs to the customer, not the system.

Keep the write separate from the reply

Separate mutation success from Slack reply delivery so recovery can retry a missing reply without replaying a write that already landed. A Builder.io pull request (agent-native #2212) describes the failure: a Slack correction successfully updated content but left the thread silent, and retrying the whole integration task would risk replaying the mutation. The fix persists the formatted reply before provider delivery, carries provider receipts and deterministic conversation message IDs across retries, and retries only the missing reply.

Copy that split into your runtime:

1. Mutation phase: execute tools that change knowledge, tickets, code, or external systems. Persist each result and external ID before moving on.
2. Compose phase: build the user-visible Slack answer from those results and citations.
3. Delivery phase: post or stream the reply, then store the Slack message ts as a delivery receipt.

If delivery fails after mutations succeeded, recovery must not re-enter the mutation phase. It should recompose only if needed, then retry delivery with the same conversation identity. Slack does not expose a generic exactly-once primitive for replies, so delivery remains at-least-once when acceptance is ambiguous before a receipt returns. The guarantee you can own is that a successful mutation is never rerun to fix a missing reply.

Your app also owns the session status. Developing an agent says to set the status to active when the run finishes, because posting a message does not clear the loading state, and the agent sessions docs say a session left in processing times out after an hour. A run that dies mid-flight therefore does not leave the thread stuck forever. It leaves a "Working..." indicator for up to an hour, then the indicator ends with no reply and no error. Do not rely on that timeout. Re-send processing during long runs, and run a sweeper that moves ledger rows still marked running, with no recent heartbeat, to suspended or failed and posts a message to the thread.

Suspend ambiguous runs for a person

Set the session to suspended and park a dead-letter record when replay may duplicate a non-idempotent side effect. Slack's agents.sessions.setStatus method accepts suspended when the agent needs the user to intervene before it can continue, and the agent sessions docs place it after processing in the lifecycle. AWS defines a dead-letter queue as a special queue that stores messages a system cannot process because of errors. For a chat agent that means holding the run, with its tool history and request IDs, for review instead of replaying it blind when a tool may have changed an external system.

Minimum DLQ recovery record for a Slack agent:

1. Message identity: event or request ID, received time, channel, thread, account.
2. Processing state: which steps succeeded, failed, or are ambiguous.
3. Failure class: timeout, permission denial, malformed input, unavailable destination, unknown.
4. Side-effect evidence: tool-call IDs, external action IDs, delivery attempts.
5. Recovery owner: the ops or platform person who can approve replay.

Default policy:

Worked example: support draft plus ticket create

A CS ops lead mentions the agent in a customer escalation thread. The intended plan is:

1. Search Kipwise for the refund policy (read-only).
2. Draft a customer-ready reply from help-center evidence.
3. Create an Intercom ticket with the draft and thread link.
4. Post the draft and ticket link back into Slack.

Step 3 times out after 45 seconds. The Intercom API may or may not have created the ticket. Step 4 never ran, and the Slack session is still processing.

Recovery sequence:

1. Mark step 3 ambiguous in the run ledger. Do not mark the run failed-and-discarded.
2. Verify: look up Intercom by the idempotency key for this thread and step.
3. If the ticket exists, store its ID, mark step 3 succeeded, compose the Slack reply, and enter delivery-only recovery.
4. If the ticket does not exist, create once with the same key, then deliver.
5. If verification itself is unavailable (Intercom outage), set the session to suspended, write a DLQ record with the side-effect evidence so far, and tell the thread what completed (policy search and draft) and what is waiting (ticket create). Offer retry, escalate, or cancel.
6. If the requester mentions the agent again meanwhile, the intent key finds the same ticket or pending record, and the agent answers with its status instead of creating a second one.

The same pattern applies when step 3 succeeded and step 4 failed: reuse the ticket ID and retry only Slack delivery.

Failure modes to design for

1. Process crash after mutation, before ledger write. Persist the intent to mutate (key + step) before the call, then the result after. On restart, verify by key before calling again.
2. Model regenerates a different plan after resume. Recovery must follow the ledger, not a newly sampled tool list.
3. User clicks stop during processing. Agent sessions require you to handle agent_session_stopped, stop in-progress work, and move status away from processing yourself. Preserve completed steps; do not auto-replay mutations.
4. Delivery succeeds twice under ambiguous acceptance. Prefer deterministic client-side message identity where the API allows, and treat a second identical reply as a known residual risk you monitor rather than a reason to replay mutations.
5. DLQ becomes an ignored archive. Assign an owner, alert on age, and require the same approval standard for consequential replay as for the original write.

What to do next

Pick one production Slack agent that creates tickets or writes knowledge today. Add a run ledger and verify-before-retry for its mutating tools before you add more connectors, then wire suspended plus a DLQ for ambiguous side effects, so mid-run timeouts become recoverable operator work instead of silent duplicates.

Two chaos tests show whether it works. Kill the worker after a successful ticket create and before the Slack reply, and confirm you end with one ticket and one eventual reply. Inject an ambiguous timeout with the ticket already created, and confirm verify-before-retry reuses it. Over a two-week pilot, count duplicate side effects, how long suspended runs wait to be cleared, and sessions left in processing.