How to Build AI Agent Observability for Slack Workflows

Key every trace on the Slack event_id and thread_ts, acknowledge each event within three seconds and run the agent from a queue, log decisions and digests instead of prompts and documents, and record generation and Slack delivery as separate outcomes. With those four habits, an admin who starts from a Slack permalink can reconstruct a run, including the case where a polished answer exists and never reached the thread. We recommend adding the approval and configuration fields below before the agent gets any consequential write.

This is a runbook for IT and digital-workplace admins who run an AI agent in Slack and need to explain a failed run without copying sensitive prompts and business documents into an unrestricted logging system. A source may have been inaccessible, a tool may have timed out, an approval may have expired, or the final message may never have reached Slack. Finding out which across application logs, tool histories and message threads wastes operations time and can turn a recoverable error into an audit problem.

The questions a trace has to answer

Generic LLM monitoring starts with tokens, latency and model errors. The first question from an operations owner is different: what happened to this request in this Slack thread? A useful trace lets an authorized admin determine:

1. Which Slack event started or resumed the run?
2. Which user and permission state applied?
3. Which sources and tools were attempted?
4. What succeeded, failed, or returned incomplete data?
5. Which output was proposed for approval?
6. Who approved it, and was that approval still valid?
7. Was the result generated, posted to Slack, and acknowledged as delivered?
8. Can the failed step be replayed without repeating a completed action?

The OpenTelemetry generative AI semantic conventions give common names for model, agent, event and framework telemetry. Use them as the base and add the Slack and approval fields needed to answer the questions above. The agent and tool attributes are still marked Development in the spec, so expect renames and pin the convention version you instrument against.

Correlate every record to the Slack event and thread

Carry Slack's event and thread identifiers through the whole run. Preserve the initiating event_id, workspace, channel and thread_ts on the root trace, and have every model call, source read, tool call, approval record and delivery attempt inherit the root trace ID. The Slack Events API guide documents event envelopes, unique event identifiers, authorization context, acknowledgements and retries.

A practical trace envelope can look like this:

trace_id: tr_01K_SLACK_WORKslack:event_id: Ev_REQUESTworkspace_id: T_WORKSPACEchannel_id: C_OPERATIONSthread_ts: "1789355000.004200"message_ts: "1789355071.006100"request:requester_id: U_OPERATORpermission_snapshot_id: perm_20260914_41agent:name: kipwise-agentversion: "2026.09"workflow: investigate-and-draftoutcome:run_state: runningdelivery_state: not_attempted

This envelope is our recommendation, not an official Slack or OpenTelemetry schema. Its value is the join: an admin who starts with a Slack permalink can find one trace without searching model logs by text, and a trace viewer can link back to the exact thread without storing the thread body.

Do not use the human-readable request as the correlation key. Similar requests collide, edited messages drift, and copied customer text creates unnecessary exposure. Stable platform identifiers and an internal trace ID are safer and easier to test.

Acknowledge the event first, then deduplicate on event_id. Slack expects an HTTP 2xx within three seconds and retries a failed delivery up to three times, marking each retry with an x-slack-retry-num header, so run the agent from a queue and let the acknowledgement return before the model starts. Key on event_id and not on the retry header: an open-source agent framework that answered every retry-marked delivery with an empty 200 lost messages whenever the original never reached its handler, because the retry was the only copy. Keep each event_id in a store all workers share, for longer than Slack keeps retrying (about five minutes, or 24 hours if you enable delayed events), and let a repeated id resume the existing trace.

Log decisions and digests, not hidden reasoning

An operations trace needs observable inputs, decisions and outcomes. It does not need private chain-of-thought. For each step, record the action category, target system, authorization result, input and output digests, timing, terminal state and a safe error classification.

{"span": "tool.call","tool": "knowledge.search","target": "approved-support-space","authorization": "allowed","input_digest": "sha256:redacted-record","result_count": 4,"source_ids": ["kb_104", "kb_221"],"state": "succeeded","duration_ms": 438}

The full query and returned document text should stay in the governed work system when possible. The trace can retain source IDs, digests, counts, classifications and permission decisions. That is enough to prove what was attempted and to detect a changed input without turning the observability backend into a second knowledge repository.

Traces usually have broader operator access and longer retention than the original Slack thread or connected source, so log fields should follow an allowlist. Reject credentials, authorization headers, raw customer messages, document bodies and unrestricted tool results before export. If an incident needs deeper inspection, retrieve the original data through the investigating admin's current permissions rather than relying on a copied payload.

Kipwise Agent describes a relevant product model: work continues in a Slack thread, connected actions use the requesting user's permissions where applicable, opened sources are cited, and sensitive writes remain explicit. Observability should preserve those boundaries instead of bypassing them through logs.

Record generation and delivery as separate outcomes

Many systems mark a run successful when text exists. For a Slack workflow, success also depends on posting the intended message to the intended thread exactly once. Use separate states:

run_state: succeeded | partial | failed | cancelled | expireddelivery_state: not_attempted | pending | delivered | failed | unknownapproval_state: not_required | pending | approved | rejected | expired

A generated draft with delivery_state: failed is not a successful workflow. A posting timeout may leave delivery unknown until the system reconciles the message by idempotency key. This also sharpens alerts: page an operator when a consequential action succeeded but its confirmation is unknown, and send a lower-severity notification when generation failed before any write. Do not treat both as a generic model error.

Test the observability layer itself, too. A Langfuse bug report describes an events list whose tool-call column showed zero for traces that contained tool calls. It concerns one product and version, but it shows why a dashboard count should be checked against known trace detail before anyone treats it as ground truth.

Make approvals first-class spans

Approval is not a Boolean attached to the final message. Record what was approved, by whom, under which permissions and until when. An approval span should include:

• approving user and authorization decision;
• proposed action type and destination;
• payload digest, not necessarily the payload;
• source-set digest and agent version;
• approval time and expiry;
• execution result and external action ID;
• invalidation reason when inputs change.

If the proposed answer changes, the payload digest changes and the old approval no longer applies. If source evidence changes materially, require a new approval. If the authorized user loses access before execution, recheck live permissions and fail closed.

This makes post-run investigation faster. The admin can tell "approved but posting failed" from "posting blocked because approval expired" without reading a long thread or inferring intent from emoji reactions.

Record the configuration that produced the run

A trace cannot explain behavior if it records the model but omits the available tools, prompt version, policy version and workflow graph. OpenTelemetry took up this gap: a request to include an agent's tool list in the agent conventions was closed as completed once the generative AI conventions defined a gen_ai.tool.definitions attribute. The spec makes it opt-in, warns that it may be large and sensitive, and offers gen_ai.tool.definitions_ref for definitions held elsewhere. That points the same way as the advice here: record configuration by version or digest, and do not copy complete system prompts and tool schemas into every trace. Useful fields include:

• agent and workflow version;
• model provider and model name;
• tool registry version and enabled tool names;
• policy and redaction-rule version;
• source connector versions;
• deployment environment;
• sampling decision.

This lets an admin compare a successful run with a failed run and see whether a tool schema or policy changed. It also stops a replay from silently using a different configuration.

Alert on operational consequences

The Langfuse observability overview covers traces, sessions, metadata, latency, cost, dashboards and evaluation, which are useful foundations. For Slack operations, add alerts for user-visible and control failures:

• delivery remains pending or unknown beyond the expected window;
• a tool span never reaches a terminal state;
• an approval is executed after expiry;
• the same event ID creates more than one completed action;
• a run claims success with one or more required sources unavailable;
• redaction rejects a trace export;
• tool-call counts disagree between aggregate and detail views;
• permission snapshots are missing for a connected action;
• latency or cost changes sharply for one workflow version.

Sample routine successful traces if volume requires it, but retain error and consequential-action traces under the team's approved retention policy. Alert messages should link to the Slack thread and trace, while respecting the viewer's permissions.

Reading one failed run

Suppose a support operations lead asks the agent to inspect product evidence and draft a response. The agent retrieves two knowledge pages, a GitHub issue times out, and the model produces a cautious draft. Slack posting then times out.

The correct trace shows the knowledge reads as succeeded, GitHub as failed, generation as partial because required evidence was unavailable, approval as not requested and delivery as not attempted. If posting was attempted before the timeout became clear, delivery stays unknown until reconciliation finds or fails to find the idempotent message.

The thread should receive a terminal update only after reconciliation: the draft was not sent, one evidence source failed, and a retry will reuse the same trace without duplicating completed reads. The workflow must not claim that the customer answer was prepared successfully simply because the model returned text.

Twelve drills before the first consequential write

Run controlled failure drills in a non-production workspace:

1. Start one thread and confirm its event ID and thread timestamp appear on every child span.
2. Reply without mentioning the agent again and confirm the run remains in the same thread trace chain.
3. Deny one source permission and confirm the trace records denial without copying protected content.
4. Force a tool timeout and confirm its span closes as failed.
5. Change the tool registry version and confirm the trace identifies the new configuration.
6. Approve one payload, modify it, and confirm the prior approval becomes invalid.
7. Force Slack posting to fail and confirm generation and delivery show different outcomes.
8. Replay the same Slack event, once marked as a retry, and confirm no duplicate external action occurs. Then drop the first delivery before your handler sees it and confirm the retry still starts a run.
9. Seed one known tool call and compare detail with aggregate dashboard counts.
10. Insert a credential-shaped value and confirm redaction blocks its export.
11. Revoke the requester's access before replay and confirm permissions are checked again.
12. Delete or expire trace data and confirm the Slack message does not expose a dead unrestricted link.

Measure mean time to explain a failed run, missing-span rate, false-success incidents, duplicate deliveries, expired approvals, and sensitive fields blocked from traces. Use the same definitions before and during the pilot.

Choose one recurring Slack workflow with a named admin owner. Implement the correlation envelope, terminal states, and redaction allowlist, then run all twelve drills before enabling a consequential write.