AI onboarding testing should happen before a new employee treats a company brain as a source of truth. An answer can sound polished while citing the wrong policy, skipping an approval step, or revealing instructions meant for another role. Replace the broad chatbot demo with a small acceptance suite built from real onboarding tasks. Run it under realistic identities, then block release when critical answers, citations, permissions, or refusals fail. This guide explains how to define that suite, score the results, pilot the system, and decide whether it is safe enough for a new hire's first week.
Why a good demo is not a useful test
A demo usually starts with a friendly question whose answer is already known. The operator uses an administrator account, the source document is easy to retrieve, and someone familiar with the company interprets any ambiguity. New hires have none of those advantages. They use unfamiliar vocabulary, do not know which policy is current, and cannot spot an answer that silently combines two incompatible procedures.
Retrieval augmented generation helps by giving a model external material to use. The AWS explanation of retrieval augmented generation describes a flow that retrieves relevant organizational data and uses it to improve the response. That grounding is necessary, but it does not prove that the right passage was retrieved, that the user may access it, or that the final answer preserved its conditions.
Treat the company brain as an onboarding system, not a text generator. Its job is to help a particular employee complete a particular task from approved evidence. A test therefore needs to inspect the task outcome, the cited source, the user's access, and the behavior when evidence is missing.
Define the release boundary first
Do not test every question an employee could ask. Define the first release around a bounded set of onboarding jobs, such as requesting software access, finding the expense policy, preparing for a customer call, or locating the correct escalation path. The GitLab onboarding handbook is a useful public example because it separates general tasks, role responsibilities, access requests, and human support such as onboarding buddies.
For each job, record five things:
- The employee role and location.
- The approved source or sources.
- The action the employee should take.
- Conditions that change the answer.
- The owner who can resolve uncertainty.
This turns a vague quality goal into a release contract. For example, "Help a new sales employee submit a travel expense" is testable. "Answer questions about expenses" is not. The first statement identifies a user, task, expected source, and outcome. The second invites subjective grading.
Start with high consequence and high frequency tasks. A wrong lunch recommendation is annoying. A wrong security, payroll, benefits, customer data, or approval instruction can create real harm. Give those cases stricter release rules and require human review of every failure.
Build tests from real onboarding tasks
Create test cases in a table or structured file so they can be rerun after documents, retrieval settings, or models change. Avoid writing only ideal questions copied from page titles. Include the language a new employee might actually use.
id: expense-policy-remote-us
persona: new-us-employee
question: "Can I buy a monitor for my home office, and who approves it?"
required_sources:
- remote-equipment-policy
required_facts:
- spending limit
- manager approval step
- reimbursement route
forbidden_sources:
- contractor-equipment-policy
expected_behavior: answer-with-citation
severity: critical
owner: people-operationsBuild several variants for each important task:
- A direct question using the document's terminology.
- A natural question using shorthand or an old internal term.
- A question missing a key condition, such as country or employment type.
- A question that combines two related policies.
- A request from a user who lacks access to the source.
- A question for which no approved answer exists.
The missing-condition case matters because a fluent system may guess rather than ask. If a benefits answer depends on location, the correct response may be a clarifying question. If no approved relocation policy exists, the correct response is a refusal plus a named escalation route, not a confident synthesis.
Use actual onboarding questions when they are available, but remove personal data and secrets. People operations can contribute recurring questions. IT can provide common access failures. Hiring managers can identify role specific vocabulary. Knowledge owners should approve the expected answer rather than letting the model's current output become the answer key.
Score five behaviors separately
A single thumbs-up score hides the cause of failure. Score at least five behaviors for every case.
Answer correctness
Does the response preserve the required facts, conditions, sequence, and owner? A response can be broadly relevant and still fail because it omits the approval step. Mark critical facts explicitly so graders do not reward polished but incomplete prose.
Use a simple result such as pass, partial, or fail. Define partial before testing. For a critical case, partial should normally block release until the omission is fixed.
Citation support
Can the employee open the source, and does the cited passage support the nearby claim? A citation to the right document is not enough when the response relies on a different section. Check source identity, passage support, and document version.
The company brain should make uncertainty visible. When two sources conflict, the expected behavior is to identify the conflict and route it to an owner. It should not choose whichever passage produces the smoothest answer.
Permission enforcement
Run the same question under more than one identity. The Microsoft overview of document level access control explains query-time patterns that trim results according to user or group access. Your acceptance test should prove the complete path, including identity mapping, retrieval filtering, answer generation, links, and cached results.
Test both directions. An authorized employee should receive the needed answer. An unauthorized employee should not receive the protected fact, a revealing summary, a document title that leaks the subject, or a link that grants accidental access. Use synthetic sensitive content in the test environment so a failure does not expose real records.
Refusal and escalation
Ask questions outside the approved corpus and questions that request restricted material. A useful refusal states what the system cannot verify and points to the correct person, team, or request process. A generic "I cannot help" response is safe but not useful. An invented answer is useful-looking but unsafe.
Include prompt injection attempts inside both the user question and a retrieved test document. The desired result is not clever debate with the instruction. The system should follow its approved workflow, avoid disclosing protected content, and direct the employee to a valid next step.
Freshness and version choice
Publish an old and a current synthetic policy, then confirm that retrieval selects the current one. Test effective dates, archived pages, duplicate titles, and regional variants. If the system cannot determine which source governs, it should say so.
Freshness tests differ from general knowledge maintenance. The release question is narrower: given the current index today, does the onboarding answer use the governing version and expose its date or status clearly enough for a new hire to trust it?
Use a release gate instead of an average
An average can conceal a serious failure. Nine correct lunch and office answers should not cancel one leaked payroll instruction. Group tests by severity and define nonnegotiable gates.
A practical gate might require:
- Every critical permission test passes.
- Every critical answer cites an approved source.
- No restricted fact appears for an unauthorized persona.
- Unsupported policy questions refuse or escalate correctly.
- All critical failures have an assigned owner and rerun result.
- Lower severity partial results are documented and accepted by the relevant owner.
The exact threshold is a governance choice. The important point is to write it before reviewing results. The NIST AI Risk Management Framework organizes work around governing, mapping, measuring, and managing risk. That structure fits this release process: identify the onboarding context, measure behavior with cases, assign owners, and manage unresolved risk rather than treating evaluation as a one-time benchmark.
Keep the raw results. For each run, record the system version, corpus snapshot, identity, question, retrieved sources, answer, grade, grader, and failure category. This creates an audit trail and makes regression comparisons possible.
Pilot with people who do not know the answers
After the acceptance suite passes, run a small pilot. Include at least one recent hire or employee outside the policy owner's team. Experts unconsciously fill gaps because they know the process. A realistic user reveals whether the answer explains unfamiliar terms, gives an actionable sequence, and makes citations easy to inspect.
Ask pilot users to complete tasks, not rate chatbot personality. Observe whether they reach the correct document, take the correct action, and know when to contact a person. Record unanswered questions as candidates for the test suite or content backlog.
The Kipwise new hire onboarding workflow combines searchable knowledge with assigned reading. That distinction is useful in a pilot. Some material should be proactively assigned because every hire must acknowledge it. Other knowledge can remain available on demand. Testing should cover both paths rather than assuming chat replaces the onboarding plan.
Do not let the company brain impersonate a manager. Culture, feedback, prioritization, and sensitive personal decisions still need people. The system can explain an approved process and connect the employee to its owner. It should not invent discretionary decisions.
Diagnose failures by layer
When a case fails, label the layer before changing prompts.
Fix the source
The approved answer may be missing, contradictory, stale, or badly structured. Fix the knowledge source and its ownership.
Fix retrieval
When the source exists but the system selects the wrong passage or misses the right one, inspect indexing, metadata, filters, query rewriting, and ranking.
Fix permissions
Identity or group context may be missing, late, or inconsistent. Stop the release for affected content. Response instructions cannot repair a broken access check.
Fix generation
If the right evidence is present but the answer drops a condition or adds an unsupported claim, tighten answer rules, required fields, or post-generation checks.
Fix the user experience
An answer can be correct but unusable because the link is inaccessible, the next step is unclear, or internal jargon is unexplained. Rewrite the source or response format.
This classification prevents a common mistake: repeatedly tuning the model when the real defect is an obsolete policy or broken identity mapping.
Keep evaluation running after launch
Rerun the suite whenever a governing document, permission model, retrieval component, or generation model changes. Also rerun it on a schedule because indexes and source collections drift even when code does not.
Add production questions only after review and redaction. Convert repeated escalations, failed searches, and low confidence answers into candidate cases. Keep a stable core suite for regression testing and a changing set for newly discovered gaps.
Assign each case to a source owner. People operations may own benefits tests, IT may own access tests, and department leaders may own role procedures. The system owner coordinates the run but should not decide whether every business answer is correct.
Run the first test this week
Choose ten onboarding tasks that a new employee must complete in the first two weeks. For each one, name the approved source, required facts, user persona, expected refusal conditions, and owner. Add unauthorized and missing-evidence variants for the critical tasks. Run the suite under real test identities, fix failures by layer, and require explicit owner approval before release. That small, repeatable gate provides more protection than another polished demo.
References
- NIST AI Risk Management Framework supports the risk governance and measurement structure used for release decisions.
- AWS explanation of retrieval augmented generation supports the description of retrieval grounding and external knowledge sources.
- Microsoft document level access control overview supports query-time permission filtering patterns.
- GitLab onboarding handbook provides a public example of structured onboarding tasks, access work, and human support.
- Kipwise new hire onboarding supports the distinction between assigned onboarding reading and searchable knowledge.
