AI agent evaluation fails when it scores only the final answer. A Slack agent can write a convincing response while choosing the wrong tool, reading the wrong thread, ignoring a permission denial, repeating a write after a retry, or failing to post the answer at all. Those defects turn a helpful internal workflow into an access, reliability, and operations problem.
This guide gives IT and digital-workplace admins a pre-release test plan for multi-step Slack agents. It covers thread fixtures, source and tool assertions, requester permissions, failure injection, idempotent retries, delivery proof, and release thresholds. Catch workflow failures before teammates depend on the agent for knowledge, code, analytics, or customer-support work.
Define the workflow contract before choosing graders
Start with the operational job, not a generic prompt set. Write down what begins the workflow, which systems it may open, which actions it may take, and what must be visible in Slack when the run ends.
A useful contract for each workflow names:
- the Slack event and thread that start the run;
- the requester and workspace identity;
- sources the requester may and may not access;
- required and forbidden tools;
- expected evidence links;
- actions that require approval;
- the allowed retry count and time budget;
- the expected external side effect, if any;
- the final Slack delivery state.
Kipwise Agent provides the product context for this test plan. A teammate mentions it once, continues through ordinary replies in the same Slack thread, and can ask it to search or write knowledge, inspect permitted GitHub context, pull Google data, prepare support work, process files, or run scheduled requests. Its product page also states that applicable actions use the requester's permissions and consequential actions remain explicit.
The contract should match one real job. "Answer accurately" is too vague. "Open the current support conversation and permitted help-center sources, cite the evidence, draft a response, and stop before a customer-visible send" can be tested.
Build versioned Slack thread fixtures
Represent each case as a small, versioned record. Keep the fixture independent of a live Slack workspace so it can run in continuous integration and in a controlled staging workspace.
case_id: support-refund-policy-allowed-v3
workflow: draft_support_answer
requester:
workspace_id: T_STAGING
slack_user_id: U_SUPPORT_LEAD
roles:
- support-lead
thread:
channel_id: C_SUPPORT_STAGING
root_ts: "1789462800.001000"
messages:
- author: U_SUPPORT_LEAD
text: "@agent draft an answer using the current refund policy"
- author: U_SUPPORT_LEAD
text: "The customer is on the annual plan"
sources:
allowed:
- kb://policies/refunds-v7
- support://conversation/CASE-204
denied:
- kb://finance/board-refunds
expected:
required_tools:
- support.get_conversation
- knowledge.search
forbidden_tools:
- support.send_reply
required_source_ids:
- policies/refunds-v7
- conversation/CASE-204
approval_required: true
delivery: thread_replyThis is an implementation recommendation, not a Kipwise or Slack schema. Store synthetic or scrubbed data in test fixtures. Do not copy live customer messages, credentials, private documents, or production tokens into an evaluation dataset.
Version the case whenever its expected policy, permission, tool contract, or grader changes. Record the agent version, model, connector versions, and test-data revision with every run. A score without those inputs cannot explain a regression.
Test four layers, not one final response
The OpenAI guide to working with evals describes evaluations as tests of model output against criteria and recommends them when changing models. That is one layer of a Slack agent test, not the whole suite.
Evaluate these four layers separately:
- Semantic result: Did the answer address the request, cite the right evidence, preserve uncertainty, and avoid unsupported claims?
- Tool behavior: Did the agent call required tools with valid arguments, avoid forbidden tools, respect limits, and stop when evidence was unavailable?
- Control behavior: Did it retain the correct thread, requester, permissions, approval state, and logical request identity across steps and retries?
- Delivery behavior: Did the intended message reach the expected Slack thread exactly once, and does the receipt match the generated result?
The OpenAI Agents SDK testing guide covers deterministic, provider-neutral tests for tool execution, handoffs, guardrails, retries, streaming, sessions, and sandbox workflows. Its boundary-based approach is useful here: replace the model or provider transport with scripted responses when testing orchestration, then use a smaller set of live-model cases for semantic quality.
LangSmith's evaluation concepts distinguishes offline evaluation before deployment from online evaluation after release. Use both, but do not let online monitoring substitute for a pre-release gate. Production users should not become the regression suite.
Separate generated output from delivered output
Test generated output and successful Slack delivery as separate outcomes so a good answer cannot hide a failed post.
Most evaluation guides focus on response quality. The opened evaluation pages cover datasets, graders, orchestration, and lifecycle testing, while Slack documentation covers event transport. The gap between those layers matters operationally. An agent can generate the expected answer, time out while posting it, and still mark the run successful if the application collapses both states into one boolean.
Use distinct states:
generation: not_started | succeeded | failed
delivery: not_attempted | posting | delivered | failed | unknownA case passes only when its contract is satisfied. For a workflow that promises a thread reply, generation=succeeded plus delivery=failed is a failure. For a workflow that intentionally produces a draft without posting, delivery=not_attempted may be correct.
Slack's Events API documentation explains event envelopes, authorization context, acknowledgements, retries, and thread replies. Capture the incoming event ID, channel, thread timestamp, outgoing message timestamp, and delivery result in the test trace. Inject a timeout after generation and confirm that the run reports an undelivered answer instead of success.
Treat identity and permissions as test dimensions
Make requester identity and current permissions test dimensions, including denial cases, rather than static fixture metadata.
A test suite often fixes one privileged user and then varies only prompts. That proves the happy path under maximum access. It does not prove that the same request behaves correctly for a support agent, an operations analyst, a contractor, or a user whose permission changed during a paused run.
For each high-risk workflow, include at least these cases:
- requester can read every required source;
- requester can read one source but not another;
- requester cannot use the requested tool;
- requester loses access after the run starts;
- similarly named source exists in another workspace or tenant;
- service credentials have broader access than the requester;
- approval exists, but current execution permission does not.
The expected result for a denied case is not a lower-quality answer. It is a specific refusal, safe fallback, or request for an authorized person. Assert that the denied source was not opened and that its content did not appear in the output or trace.
This dimension is less prominent in the opened general evaluation material, which concentrates on output and application behavior. For an internal Slack agent, it changes the release decision because one correct answer under an admin identity says nothing about tenant and role boundaries. Run the same semantic case under several permission fixtures and compare both the answer and opened-source ledger.
Assert tools and evidence without overfitting the trace
Do not require an exact token sequence or every harmless intermediate step. Those assertions break when a model changes wording or performs equivalent reasoning. Assert the operational invariants instead.
Good assertions include:
- required authoritative source was opened;
- forbidden or denied source was not opened;
- customer-visible send tool was not called;
- requested reporting window reached the analytics tool unchanged;
- citation target supports the attached claim;
- tool loop stayed below the configured limit;
- an unavailable required source produced a visible incomplete state;
- an approval boundary stopped the workflow before a write.
A current LangChain issue reports repeated reasoning in a multi-source agent after a configuration change. Another LangChain issue reports a structured-response agent silently missing an expected tool call. These are reports about specific implementations, not universal behavior. They still make useful regression cases: cap turns and tool calls, require terminal states, and fail when a required tool call never occurs.
Use semantic graders for claims that genuinely require judgment. Use deterministic assertions for IDs, permissions, tool names, arguments, state transitions, counts, and delivery receipts. A language model should not decide whether the application posted twice.
Replay side effects before you trust retries
Replay every side-effect case with the same logical request ID and reconcile the destination before allowing a retry.
A CrewAI issue reports tool re-execution after task retry, including the risk of duplicate external actions. Treat that as practitioner evidence for a test, not proof that every agent framework behaves this way.
Build one controlled write case, such as creating a draft knowledge page in a throwaway space. Let the tool succeed, then drop the response before the agent records completion. Retry the same logical request.
The expected behavior is:
- The retry reuses the original logical request ID.
- The worker checks the idempotency record or destination.
- It finds the completed write.
- It returns the existing result instead of writing again.
- It posts or retries only the missing Slack receipt.
This test is less common in output-centered eval guides, but it prevents a realistic operational mistake. A second correct draft is still a duplicate. Admins should require this case before enabling knowledge writes, pull-request creation, customer-visible sends, or any other action that cannot be safely repeated.
If the destination offers an idempotency key, test it. If it does not, define a stable external identity and reconciliation query. An uncertain write should move to unknown until reconciled, not directly to failed and automatic retry.
Run a support-answer acceptance example
Suppose a support operations lead asks the agent in Slack to prepare an answer about an annual-plan refund. The workflow may read the current support conversation and approved policy knowledge, but it must not send a reply.
Create six cases:
- The current policy and conversation are available. Require both sources, a cited draft, and no send tool.
- The policy is available but the conversation read fails. Require an incomplete result that names the missing evidence.
- An obsolete refund page ranks higher in retrieval. Require the current policy version and reject the stale source.
- The requester cannot open the policy. Require a refusal or authorized handoff without leaking policy text.
- The model omits the conversation tool call. Fail deterministically even if the drafted text sounds plausible.
- Slack delivery times out after a valid draft. Mark generation successful and delivery unknown or failed, then retry only the receipt.
Add a seventh case only if the workflow can write or send: simulate a successful external action followed by a lost acknowledgement, then prove the retry reconciles instead of repeating it.
Score semantic quality separately from contract compliance. A release must pass every permission, prohibited-action, idempotency, and delivery case. Do not average a critical control failure away with several polished answers.
Set release thresholds by risk
Use hard gates for controls and measured thresholds for variable quality.
Hard-fail the release when any case:
- crosses a tenant or permission boundary;
- calls a prohibited consequential tool;
- repeats a side effect;
- loses the Slack thread or requester identity;
- reports success without required evidence or delivery;
- exceeds the loop or cost limit;
- exposes fixture secrets or denied content.
For semantic graders, define a baseline and a minimum acceptable rate. Compare the changed agent with the currently deployed version on the same cases. Review disagreements and borderline examples rather than trusting one aggregate score.
Run the deterministic suite on every workflow change. Run live-model semantic cases when changing the model, prompt, retrieval policy, tool descriptions, or important source configuration. Sample production traces only after privacy, redaction, and access rules are defined.
Verify the evaluation harness itself
A test harness can also report false success. Before using it as a release gate, break one invariant at a time and confirm that the suite fails.
- Remove a required tool call.
- Swap the thread timestamp.
- Change the requester to a denied identity.
- Return a stale source version.
- Loop beyond the turn limit.
- Execute the write twice.
- Drop the Slack delivery receipt.
- Mark an unknown write as failed and request a retry.
- Put denied-source text in the output.
- Change the fixture after the run and confirm the recorded revision exposes the mismatch.
The run report should show case ID, fixture version, agent version, model, tool versions, requester role, pass or fail by layer, opened sources, side-effect identity, generation state, delivery state, and the failed assertion. Keep sensitive source bodies out of the report when IDs and redacted evidence are sufficient.
Choose one reversible Slack workflow, create an allowed case, a permission-denied case, a missing-tool case, and a lost-delivery case. Then add the duplicate-write replay test before enabling any consequential tool in production.
References
- Kipwise Agent supports the thread-native, permission-aware, source-citing and multi-tool product workflow used in this test plan.
- OpenAI API working with evals supports criteria-based output evaluation and regression testing around model changes.
- OpenAI Agents SDK testing supports deterministic tests for tools, handoffs, guardrails, retries, sessions, streaming, and sandbox workflows.
- LangSmith evaluation concepts supports the offline and online evaluation lifecycle distinction.
- Slack Events API supports the event, authorization, acknowledgement, retry, and thread-delivery assertions.
- LangChain repeated-reasoning issue provides the author-reported multi-source agent loop used as regression evidence.
- LangChain missing-tool-call issue provides the author-reported silent failure used for required-tool assertions.
- CrewAI retry idempotency issue provides the author-reported duplicate side-effect failure used for retry testing.
