How to Build an AI Onboarding Assessment for New Hire Understanding

Do not measure onboarding understanding with completions, page views or multiple-choice questions. Give the new hire a short scenario with a real decision, ask which internal source governs it and when to bring in a person, and score "ask the owner" as a correct answer when that is what the source says. We recommend this because a hire can finish every assigned item and still apply the wrong expense rule, use an outdated support workflow or miss the point where a manager must approve, and a guessable quiz cannot tell the difference.
Completion is not comprehension
Most onboarding systems can record that someone opened a page, watched a lesson or checked a task. Kipwise's new hire onboarding workflow combines assigned reading with searchable company knowledge and progress tracking. That records delivery and completion, but it does not prove that the employee can use the material correctly when the situation changes. Even a well-run checklist merges two states: in GitLab's onboarding handbook, a ticked box means either that the hire completed the task or that the hire checked and the task does not apply to them, so a completion rate cannot say which.
The gap matters because onboarding knowledge is operational. A support agent must choose the right escalation route, a manager must know which leave policy applies, and an engineer must recognize when a production change needs review. A recall question such as "What is the travel limit?" tests a fact in isolation. A scenario such as "A customer asks for a refund outside the standard window; what do you do next and which source supports it?" tests application, source selection and uncertainty.
Treat the company brain as a retrieval and coaching layer, not the final judge. It retrieves approved material, builds or selects a bounded scenario, compares the response with explicit criteria, and sends uncertain or consequential results to a person. NIST's Generative AI Profile (NIST-AI-600-1), published under its AI Risk Management Framework, is written to help organizations identify risks unique to generative AI, and a model that grades people brings those risks into a decision about a person. Test the grader on cases whose answers you already know before it scores anyone.
Define what the employee must be able to do
Start with tasks, not topics. "Understands security" is too vague to assess. "Can identify a suspicious access request and route it to the security team without sharing credentials" is observable. For each role, choose five to ten decisions that matter in the first month, using three filters:
- The decision occurs often enough to practice.
- A wrong choice causes delay, rework, customer harm or compliance risk.
- An owned internal source supports the correct action.
Write one capability statement for each decision: given this situation, the employee can choose the correct next action, cite the governing source and state when to escalate. If the expected response is "ask the onboarding buddy" or "request access from the named owner", that is a correct outcome, and the system should not force an automated answer.
Then classify each capability by consequence. Low consequence covers navigation, terminology and reversible workflow choices. Moderate covers customer handling, internal approvals and actions that create rework. High covers pay, benefits, safety, privacy, security, legal obligations and production access. Let AI coach low-consequence items immediately, and require a person to review failed or ambiguous high-consequence items. This is a decision rule, not a claim that a model can grade every domain reliably.
Build scenarios from governed sources
Every scenario needs a source owner, source version, scope and review date. Do not ask a model to write quizzes from an unbounded search of the wiki, because it can mix old and current procedures or invent a question whose expected answer is not documented. For each scenario, select one approved source and one capability, extract the decision, prerequisites, exceptions and escalation path, draft a situation with only the context needed to decide, and record the acceptable actions and required evidence. A compact assessment object looks like this:
{
"capability_id": "support-refund-exception",
"role": "support-agent",
"source_id": "refund-policy",
"source_version": "2026-07-12",
"scenario": "A customer requests a refund after the normal window because the service was unavailable.",
"required_actions": [
"check the documented exception conditions",
"route approval to the named owner"
],
"required_evidence": ["refund-policy"],
"consequence": "moderate",
"human_review_on": ["unsupported source", "uncertain decision"]
}Keep the expected response separate from the generated scenario. The model may phrase the situation, but the source owner defines the acceptable actions. Store the source identifier and not a copy of the policy paragraph, retrieve the current approved version at assessment time, and block the item if the expected criteria no longer match it.
The owner's approval is not enough to release a scenario, because the owner knows what the page meant. We recommend a stronger gate: before assignment, ask two people who know the policy to answer the scenario independently, without looking. Release it only if they agree. If they differ, the source is ambiguous, and the fix belongs on the page and not in the question, because a hire cannot be expected to resolve what two experts cannot.
Score the reasoning, not the guess
Check five independent parts of the response: the decision (an allowed next action), the evidence (the governing source), the scope (role, location, date or approval conditions noticed), the escalation (uncertainty or a required human decision recognized) and the explanation (the choice in plain language). Do not collapse these into an unexplained model score. Return a structured result, supported, partially_supported, unsupported or needs_review, with a reason for each part. A partially supported answer might choose the right action and cite the wrong page, which is a retrieval habit to correct and not the same failure as choosing an unsafe action.
Let the employee inspect the source and retry, because the aim is to close a knowledge gap and not to catch someone out. A good correction is specific: "Your action matches the policy, but this case requires approval from the regional owner. Review the exception section and try again." A bad one is a bare "Incorrect, score 60%." After the retry, present a changed scenario to see whether the rule transfers, and let the system refuse to invent an answer when the source is absent or contradictory.
When to test, and what to record
We recommend testing just before the hire first needs the rule at work, with a case that differs from the one in the training material. A check right after reading only shows that the page is still on screen. A check the day before the first refund or the first access request shows whether the rule was kept, and a wrong answer at that point is still cheap. Microsoft's Viva Learning page describes assignments, recommendations and completions tracked in one view, and completion records synced from a connected learning management system. That is the completion layer, and the scenario check goes on top of it.
Keep the event record narrow: a learner identifier (pseudonymous where you can), the capability and scenario, the attempt time and outcome, the source and version, the rubric results, whether corrective material was shown, and whether a human review was opened and resolved. Do not store free-text answers or chat transcripts by default, because they can hold personal or sensitive context.
xAPI, an IEEE standard, records activity as noun-verb-object statements sent to a Learning Record Store, and its design assumes records can travel: one store can share statements with another. You do not have to adopt the standard to borrow the shape of who attempted which governed capability, what happened and which source version supported it. If you do adopt a store, keep sharing off for new-hire assessment records.
Use results to fix sources and support
Individual outcomes need privacy and care, while aggregate patterns improve the company brain. If many new hires miss the same prerequisite, inspect the source and the scenario first: the cause may be unclear documentation, poor retrieval or an ambiguous expected answer and not the employees. Route each recurring gap to one owner: the knowledge owner for unclear or missing content, the onboarding owner for bad sequencing, the system owner for retrieval or citation failures, and the hiring manager or buddy for role context and coaching. Set a minimum sample before treating a pattern as real, and never rank employees publicly or compare teams from tiny cohorts.
Failure modes to plan for
- The source changed after approval. Block the item when its recorded source version differs from the current approved version, and ask the owner to revalidate the scenario and rubric. Never grade against stale copied text.
- The model accepts persuasive nonsense. Require the decision and the source to satisfy deterministic criteria before using model feedback, and send responses outside the allowed actions to review.
- The employee guesses the expected answer. Vary the details that affect scope, prerequisites and escalation, so a second scenario requires transferring the rule to a new case.
- The documentation cannot support an answer. Mark the item
needs_review, stop automated grading and open a knowledge-gap task. Missing evidence is a system defect and not a failed employee. - A high-consequence answer is ambiguous. Do not average rubric points and pass it. Route the response, the cited source, the scenario and the specific ambiguity to the accountable person, and update the scenario only after the governing source is clear.
Check before rollout, then start small
Pilot with a role expert, a recent hire and the people who handle escalations, and watch where the system retrieves, grades and explains wrongly. Before release, confirm that every expected action appears in the current source, that exceptions and escalation routes are represented, that a source change blocks stale grading, that unsupported answers never pass, that high-consequence ambiguity always reaches a person, that employees can challenge a result and get a human explanation, and that event records and aggregate reports hold only the fields the program needs. After launch, track the share of items needing human review, the share of retries that become supported after corrective retrieval, and the recurring gaps assigned to a source owner. These show whether the loop is usable. They do not prove job performance.
Choose one role and one moderate-consequence workflow. Define three capabilities, build one scenario for each, and have two people who know the source answer each one. Run the pilot, inspect every result by hand, and fix the sources before adding more scenarios.



