A Slack AI agent that works in a demo can double-post in production. IT and digital-workplace admins see it when a teammate mentions the agent once, the endpoint takes too long to answer Slack, and a retry arrives while the first run is still opening knowledge, drafting a support reply, or creating a ticket. The thread gets two answers. The helpdesk gets two tickets. Security asks which run was authoritative and the logs show both.
If you own production Slack agent endpoints for knowledge retrieval, support drafting, or multi-tool ops work, use this as an at-least-once delivery control plan. It covers Slack's retry rules, dual claim keys, ack-first handling, mutating-tool idempotency, audit fields, failure modes, and a pilot checklist. Use it when one mention must equal one agent run.
Why Slack retries create duplicate agent work
Slack Developer Docs on the Events API are explicit about delivery. Your app should respond with an HTTP 2xx within three seconds. If it does not, Slack considers the delivery failed and retries up to three times with exponential backoff. Retry attempts include an x-slack-retry-num header with values 1, 2, or 3, and a reason such as http_timeout. Slack also documents that the Events API is a best-effort system and that delayed delivery options can extend retries further when enabled.
Retries keep Slack reliable when endpoints stall. They become dangerous once an agent creates tickets, posts replies, or edits knowledge. A handler that verifies the signature, then immediately searches Kipwise, calls Intercom, opens GitHub, and posts a long reply inside the request path will regularly miss the three-second window. Slack retries. If your code treats every POST as a fresh job, the model runs again.
Practitioner write-ups match that failure mode. QASkills' guide on testing Slack Events API retries describes a handler that posted the same incident alert twice. Both requests carried the same Slack event ID; the second also carried x-slack-retry-num: 1 and http_timeout. The duplicate delivery was expected platform behavior. The duplicate side effect was an application defect.
ResumeLens' Slack Events API deep dive states the same contract in engineering language: Slack guarantees at-least-once delivery, so write handlers need explicit event_id deduplication through a conditional insert or Redis SET NX. Read-only logging can survive duplicates. Creating tickets, posting messages, or updating knowledge cannot.
Treat those practitioner sources as author reports about specific designs, not as measured search demand. Retries are normal on the Events API. Duplicate tool calls are the defect you can prevent.
Separate delivery identity from agent work
Every interactive Slack agent run needs three identities recorded up front:
- Delivery envelope: Slack's outer
event_id, which is globally unique for that callback. - Logical user message: coordinates that stay stable across fan-out, typically
(team_id, channel, message ts). - Agent run: your internal run ID that owns tool spans, citations, approvals, and delivery status.
Slack's Events API docs show event_id on the outer callback and note that one request carries one event. Agents need more than that field alone. Open-source Slack agent code shows why. The kortix-ai/suna Slack dedup.ts module documents that event_id alone does not stop double-runs when one user message arrives as both app_mention and message with different event IDs. The same source recommends claiming (team, channel, ts) once so retries, fan-out, and multi-replica delivery cannot start a second turn.
Generic Slack bot tutorials often stop at "store event_id for five minutes." Admins still need the logical message claim when the agent subscribes to both mention and message events, or when two workers can receive the same retry.
Kipwise Agent fits this model as a thread-native work agent. A teammate mentions it once and continues with ordinary replies. Applicable actions use the requesting user's permissions. It cites opened sources, refuses to guess, and keeps consequential writes explicit. Those product behaviors only stay trustworthy if one Slack delivery path maps to one run.
Acknowledge fast, claim before tools, process asynchronously
Build the request path in this order:
- Verify Slack's signing secret on the raw body.
- Handle
url_verificationif present. - Return HTTP 2xx quickly after a durable claim decision.
- Enqueue or continue the agent run only when this process won the claim.
- Never start mutating tools until the claim is held.
Slack's Events API guidance recommends responding as soon as you can and avoiding full processing in the same process that answers Slack. Use a queue. Model and tool latency routinely exceeds three seconds, so this is required for AI agents rather than optional polish.
ResumeLens describes the production tradeoff. Ack-first means Slack will not retry even if later processing fails, so you need internal retries, dead-letter handling, and alerting. Process-first means you can ask Slack to retry on failure, but you must still be idempotent and usually still miss the three-second budget for multi-tool agents. Most production Slack AI agents should ack first, claim durably, and own recovery themselves.
QASkills adds a second rule operators often miss: an already-accepted duplicate should usually receive 2xx. Returning an error invites another retry and spends Slack's failure budget without creating new value. Your claim store is the source of truth, not the HTTP status on the duplicate.
Also log retry headers when present. Slack documents x-slack-retry-num and x-slack-retry-reason. Slack's Governance and trust guide asks builders to record who triggered the request, tools called, retry attempts, and outcome. Retry headers belong on the same audit row as the claim result.
Claim two keys, not one
Implement two atomic claims with the same store pattern (database INSERT ... ON CONFLICT DO NOTHING, Redis SET key value NX EX, or equivalent):
1. Envelope claim on event_id
Use this to absorb Slack's own redelivery of the identical callback. If the insert loses, return 2xx and do not enqueue.
2. Message claim on (team, channel, ts)
Use this to absorb fan-out where one human mention produces multiple envelopes. If the message claim loses, return 2xx even when the new event_id was never seen before.
The open-source dedup.ts comments explain the incident class this prevents: a redelivery that lands after a thread-to-session mapping exists gets treated as a brand-new follow-up and runs the agent a second time. That is exactly how duplicate support drafts and duplicate wiki writes appear in Slack threads.
Choose fail-open versus fail-closed deliberately. Many implementations fail open on claim-store outages so a database blip does not drop user messages, accepting rare duplicates during the outage. Spam-suppression notices often fail closed. Document the choice. Silent drops during an outage can look like "the agent ignored me," which is harder to detect than a rare double post.
Make mutating tools idempotent under the same run
Delivery dedup stops a second run. It does not stop a single run from double-writing when a tool succeeds and the response times out.
For every mutating tool call, derive a stable idempotency key from:
- agent run ID
- step or tool name
- a hash of the normalized arguments that matter for uniqueness
Before creating a ticket, posting a customer-visible reply, editing a knowledge page, or opening a GitHub change, check whether that key already has a stored result. If yes, reuse it. If no, execute once and store the result with the key.
This differs from rank 84's human approval pause and from rank 86's unattended schedule contract. Approval decides whether a write may proceed. Scheduling decides when an unattended job fires and how delivery is proven. Retry idempotency decides whether Slack's at-least-once transport can create two writes from one interactive mention. You usually need all three controls, but they solve different jobs.
Keep read-only tools cheaper to retry. Knowledge search and citation gathering can often be retried inside one run. Mutating tools cannot. Separate them in your tool policy so the model cannot "helpfully" recreate a ticket after a timeout.
Worked example: support mention that must not double-file
Imagine a support operations lead mentions the agent in #customer-escalations:
> Draft an answer from the help center for ticket 18422 and open an internal follow-up if the refund policy is unclear.
Expected path:
- Slack POSTs
app_mention(and possibly amessageevent). - Endpoint verifies signature, claims
event_id, claims(team, channel, ts), returns 200. - Worker loads requester permissions, searches approved help-center and Kipwise sources, cites what it opened, and refuses to invent policy.
- If a knowledge write or ticket create is needed, the write waits on your approval gate and uses a run/step idempotency key.
- A Slack retry with
x-slack-retry-num: 1finds the claims and exits without a second draft or second ticket. - Audit shows one run, retry headers observed, tools called once, delivery state for the Slack reply recorded separately from model generation.
Failure modes to rehearse in staging:
- Claim succeeds, enqueue fails: recover from an outbox or pending-claim state, or you will ack duplicates while the work never runs.
- Tool write succeeds, HTTP to the tool times out: without a step key, a worker retry creates a second ticket.
- Only
event_idis claimed:app_mentionandmessageboth run. - Duplicate returns 500: Slack keeps retrying and your failure ratio climbs toward disablement thresholds documented in the Events API.
Audit fields that make retries diagnosable
Slack's Governance and trust guide recommends logging who triggered the request, tools called, model, retries, tokens, and outcome (success, partial, or failure). For retry-safe agents, add:
event_id- message key
(team, channel, ts) - claim results for envelope and message
x-slack-retry-numandx-slack-retry-reasonwhen present- agent run ID
- mutating-tool idempotency keys and whether the result was reused
- Slack reply delivery state separate from generation state
Without those fields, observability can show two successful generations without explaining that Slack redelivered the same mention. Adjacent Kipwise guidance on monitoring AI agent workflows in Slack already covers reconstructing tool calls, approvals, and delivery. Add claim-key fields to those traces so retries are visible.
Pilot checklist for IT and Slack admins
- Confirm Event Subscriptions are on and the request URL is validated.
- Measure p95 handler ack time. If it is near or above three seconds, move work off the request path.
- Deploy dual claims:
event_idand(team, channel, ts). - Return 2xx for duplicates that already hold a claim.
- Add run/step idempotency keys to every mutating tool.
- Log retry headers on the same audit row as the claim.
- Replay signed duplicate deliveries in staging, including
http_timeoutretries, as QASkills recommends. - Test mention fan-out if you subscribe to both
app_mentionandmessage. - Alert on claim-store errors, duplicate claims observed, and mutating-tool result reuse.
- Pilot on one high-volume support or ops channel before workspace-wide rollout.
Controls most Slack agent tutorials skip
Claim both the envelope event_id and the logical message key (team, channel, ts), because one mention can arrive as app_mention and message with different event IDs. Envelope dedup alone will not stop that fan-out.
Acknowledge Slack within three seconds and process asynchronously; returning errors on duplicates invites more retries and spends failure budget. Your claim store decides whether work continues. The HTTP status on a duplicate should usually stay successful.
Give every mutating tool call a stable run and step idempotency key so a timeout after a successful write cannot create a second ticket or reply. Delivery dedup protects against a second run. Tool keys protect one run that retries a write after a partial success.
What success looks like
When the controls work, Slack stays quiet in the useful way: one mention produces one agent run, one set of citations, and at most one approved write. Retries show up in logs as claimed duplicates instead of second tickets, and owners spend less time deleting double posts.
If your Slack AI agent still processes Events API callbacks synchronously, or keeps recent event IDs only in memory on one replica, you do not yet have a production idempotency plan. Start with dual durable claims and mutating-tool keys. Keep approval gates, permission mapping, and schedule controls as separate layers for the jobs they own.
References
- Slack Developer Docs: The Events API documents at-least-once delivery, the three-second ack window, retry attempts, and retry headers.
- Slack Developer Docs: Governance and trust documents agent audit baselines including who triggered the request, tools, retries, and outcome.
- QASkills: Testing Slack Events API retries is a practitioner guide on duplicate side effects during retries and event_id-based idempotency tests.
- ResumeLens: Slack Events API deep dive covers signing, replay protection, and event_id deduplication for write handlers.
- kortix-ai/suna Slack dedup.ts shows an open-source dual claim on event_id and (team, channel, ts) across app_mention and message fan-out.
- Kipwise: AI agent observability for Slack workflows covers reconstructing tool calls, approvals, and delivery state for Slack agents.
- Kipwise Agent is the product context for thread-native work, permission-aware tools, citations, and explicit consequential actions.
