How to Evaluate AI Agent Workflows Before Deploying Them in Slack

Scoring a Slack agent on its final answer misses the failures that hurt: the wrong tool, the wrong thread, a permission denial ignored, a write repeated after a retry, or an answer that never reached Slack. We recommend scoring contract compliance separately from answer quality. Treat permissions, forbidden tools, duplicate side effects and delivery as hard pass/fail gates, record generation and delivery as two separate states, and use a model grader only for wording and judgment. This is a pre-release plan for IT and digital-workplace admins who run multi-step Slack agents: fixtures, assertions, permission cases, failure injection and retry replay.
Define the workflow contract before choosing graders
Start with the operational job, not a generic prompt set. Write down what begins the workflow, which systems it may open, which actions it may take, and what must be visible in Slack when the run ends. A useful contract for each workflow names:
- the Slack event and thread that start the run;
- the requester and workspace identity;
- sources the requester may and may not access;
- required and forbidden tools;
- expected evidence links;
- actions that require approval;
- the allowed retry count and time budget;
- the expected external side effect, if any;
- the final Slack delivery state.
Kipwise Agent provides the product context for this test plan. A teammate mentions it once, continues through ordinary replies in the same Slack thread, and can ask it to search or write knowledge, inspect permitted GitHub context, pull Google data, prepare support work, process files, or run scheduled requests. Its product page also states that applicable actions use the requester's permissions and consequential actions remain explicit.
The contract should match one real job. "Answer accurately" is too vague. "Open the current support conversation and permitted help-center sources, cite the evidence, draft a response, and stop before a customer-visible send" can be tested.
Build versioned Slack thread fixtures
Represent each case as a small, versioned record. Keep the fixture independent of a live Slack workspace so it can run in continuous integration and in a controlled staging workspace.
case_id: support-refund-policy-allowed-v3
workflow: draft_support_answer
requester:
workspace_id: T_STAGING
slack_user_id: U_SUPPORT_LEAD
roles:
- support-lead
thread:
channel_id: C_SUPPORT_STAGING
root_ts: "1789462800.001000"
messages:
- author: U_SUPPORT_LEAD
text: "@agent draft an answer using the current refund policy"
- author: U_SUPPORT_LEAD
text: "The customer is on the annual plan"
sources:
allowed:
- kb://policies/refunds-v7
- support://conversation/CASE-204
denied:
- kb://finance/board-refunds
expected:
required_tools:
- support.get_conversation
- knowledge.search
forbidden_tools:
- support.send_reply
required_source_ids:
- policies/refunds-v7
- conversation/CASE-204
approval_required: true
delivery: thread_replyThis is an implementation recommendation, not a Kipwise or Slack schema. Store synthetic or scrubbed data in test fixtures, and do not copy live customer messages, credentials, private documents, or production tokens into an evaluation dataset. Version the case whenever its expected policy, permission, tool contract, or grader changes, and record the agent version, model, connector versions, and test-data revision with every run. A score without those inputs cannot explain a regression.
Test four layers, not one final response
The OpenAI guide to working with evals defines evaluations as tests of model output against criteria you specify and recommends writing them when you upgrade or try new models. That is one layer of a Slack agent test, not the whole suite. Evaluate these four layers separately:
- Semantic result: Did the answer address the request, cite the right evidence, preserve uncertainty, and avoid unsupported claims?
- Tool behavior: Did the agent call required tools with valid arguments, avoid forbidden tools, respect limits, and stop when evidence was unavailable?
- Control behavior: Did it retain the correct thread, requester, permissions, approval state, and logical request identity across steps and retries?
- Delivery behavior: Did the intended message reach the expected Slack thread exactly once, and does the receipt match the generated result?
The OpenAI Agents SDK testing guide shows the boundary-based approach: replace the model or provider interface with scripted responses and run the real orchestration against them, which covers tool execution, handoffs, guardrails, retries and sessions without calling a provider. Use that for orchestration, then keep a smaller set of live-model cases for semantic quality. LangSmith's evaluation concepts separate offline evaluation on curated examples before deployment from online evaluation of production traces. Use both, but do not let online monitoring substitute for a pre-release gate. Production users should not become the regression suite.
Score generation and delivery as separate outcomes
Most evaluation guides stop at response quality. An agent can generate the expected answer, time out while posting it, and still mark the run successful if the application collapses both states into one boolean. Use distinct states:
generation: not_started | succeeded | failed
delivery: not_attempted | posting | delivered | failed | unknownA case passes only when its contract is satisfied. For a workflow that promises a thread reply, generation=succeeded plus delivery=failed is a failure. For a workflow that intentionally produces a draft without posting, delivery=not_attempted may be correct.
Slack's Events API documentation covers event envelopes, acknowledgements, retries and thread replies. Capture the incoming event ID, channel, thread timestamp, outgoing message timestamp, and delivery result in the test trace. Inject a timeout after generation and confirm that the run reports an undelivered answer instead of success.
Also test a slow acknowledgement. Slack expects your endpoint to answer an event with an HTTP 2xx within three seconds. If it does not, Slack counts the attempt as failed and retries up to three times, the first almost at once and the others after one minute and after five, each marked with an x-slack-retry-num header. An agent that runs the model before it returns 200 therefore starts a second run for the same mention. In CI, post the same signed event twice, the second copy with the retry headers, while the first run is still going, and assert one run, one side effect and one reply. The handler design that passes this test is covered in Slack event retries and idempotency for AI agents.
Treat identity and permissions as test dimensions
A test suite often fixes one privileged user and then varies only prompts. That proves the happy path under maximum access. It does not prove that the same request behaves correctly for a support agent, an operations analyst, a contractor, or a user whose permission changed during a paused run. Make requester identity and current permissions test dimensions, including denial cases, rather than static fixture metadata. For each high-risk workflow, include at least these cases:
- requester can read every required source;
- requester can read one source but not another;
- requester cannot use the requested tool;
- requester loses access after the run starts;
- similarly named source exists in another workspace or tenant;
- service credentials have broader access than the requester;
- approval exists, but current execution permission does not.
A denied case should end in a specific refusal, a safe fallback, or a request for an authorized person, and not in a lower-quality answer. Assert that the denied source was not opened and that its content did not appear in the output or trace.
For an internal Slack agent this changes the release decision, because one correct answer under an admin identity says nothing about tenant and role boundaries. Run the same semantic case under several permission fixtures and compare both the answer and the opened-source ledger.
Assert tools and evidence without overfitting the trace
Do not require an exact token sequence or every harmless intermediate step. Those assertions break when a model changes wording or performs equivalent reasoning. Assert the operational invariants instead:
- required authoritative source was opened;
- forbidden or denied source was not opened;
- customer-visible send tool was not called;
- requested reporting window reached the analytics tool unchanged;
- citation target supports the attached claim;
- tool loop stayed below the configured limit;
- an unavailable required source produced a visible incomplete state;
- an approval boundary stopped the workflow before a write.
Two LangChain issues show what to catch. One reports an agent with two knowledge bases and an MCP looping on the same reasoning once thinking mode was enabled. Another reports a structured-response agent silently missing an expected tool call. They describe specific implementations, not universal behavior, and they still make useful regression cases: cap turns and tool calls, require terminal states, and fail when a required tool call never occurs.
Use semantic graders for claims that genuinely require judgment. Use deterministic assertions for IDs, permissions, tool names, arguments, state transitions, counts, and delivery receipts. A language model should not decide whether the application posted twice.
Replay side effects before you trust retries
Replay every side-effect case with the same logical request ID and reconcile the destination before allowing a retry. A CrewAI issue describes a tool that re-executes when its task is retried, which would repeat an external action such as a payment or an email. It is one framework's report, and it is a good reason to write the test.
Build one controlled write case, such as creating a draft knowledge page in a throwaway space. Let the tool succeed, then drop the response before the agent records completion. Retry the same logical request. The expected behavior is:
- The retry reuses the original logical request ID.
- The worker checks the idempotency record or destination.
- It finds the completed write.
- It returns the existing result instead of writing again.
- It posts or retries only the missing Slack receipt.
This test prevents a realistic operational mistake: a second correct draft is still a duplicate. Admins should require this case before enabling knowledge writes, pull-request creation, customer-visible sends, or any other action that cannot be safely repeated.
If the destination offers an idempotency key, test it. If it does not, define a stable external identity and reconciliation query. An uncertain write should move to unknown until reconciled, not directly to failed and automatic retry.
Run a support-answer acceptance example
Suppose a support operations lead asks the agent in Slack to prepare an answer about an annual-plan refund. The workflow may read the current support conversation and approved policy knowledge, but it must not send a reply. Create six cases:
- The current policy and conversation are available. Require both sources, a cited draft, and no send tool.
- The policy is available but the conversation read fails. Require an incomplete result that names the missing evidence.
- An obsolete refund page ranks higher in retrieval. Require the current policy version and reject the stale source.
- The requester cannot open the policy. Require a refusal or authorized handoff without leaking policy text.
- The model omits the conversation tool call. Fail deterministically even if the drafted text sounds plausible.
- Slack delivery times out after a valid draft. Mark generation successful and delivery unknown or failed, then retry only the receipt.
Add a seventh case only if the workflow can write or send: simulate a successful external action followed by a lost acknowledgement, then prove the retry reconciles instead of repeating it.
Score semantic quality separately from contract compliance. A release must pass every permission, prohibited-action, idempotency, and delivery case. Do not average a critical control failure away with several polished answers.
Set release thresholds by risk
Use hard gates for controls and measured thresholds for variable quality. Hard-fail the release when any case:
- crosses a tenant or permission boundary;
- calls a prohibited consequential tool;
- repeats a side effect;
- loses the Slack thread or requester identity;
- reports success without required evidence or delivery;
- exceeds the loop or cost limit;
- exposes fixture secrets or denied content.
For semantic graders, define a baseline and a minimum acceptable rate. Compare the changed agent with the currently deployed version on the same cases. Review disagreements and borderline examples rather than trusting one aggregate score.
Run the deterministic suite on every workflow change. Run live-model semantic cases when changing the model, prompt, retrieval policy, tool descriptions, or important source configuration. Sample production traces only after privacy, redaction, and access rules are defined.
Verify the evaluation harness itself
A test harness can also report false success. Before using it as a release gate, break one invariant at a time and confirm that the suite fails.
- Remove a required tool call.
- Swap the thread timestamp.
- Change the requester to a denied identity.
- Return a stale source version.
- Loop beyond the turn limit.
- Execute the write twice.
- Drop the Slack delivery receipt.
- Mark an unknown write as failed and request a retry.
- Put denied-source text in the output.
- Change the fixture after the run and confirm the recorded revision exposes the mismatch.
The run report should show case ID, fixture version, agent version, model, tool versions, requester role, pass or fail by layer, opened sources, side-effect identity, generation state, delivery state, and the failed assertion. Keep sensitive source bodies out of the report when IDs and redacted evidence are sufficient.
Choose one reversible Slack workflow, create an allowed case, a permission-denied case, a missing-tool case, and a lost-delivery case. Then add the duplicate-write replay test before enabling any consequential tool in production.



