How to Add Human Escalation to an AI Onboarding Company Brain

An onboarding assistant should hand off according to what a wrong answer would cost, not according to a confidence score, and every handoff should reach a named owner with the evidence already attached. A similarity score only says that a passage resembles the question. It does not say the passage is current, applies to this employee or is enough to act on. When the evidence is weak or conflicting, or the employee may not see the source, the assistant should explain the limit, keep the question and the evidence, and send the case to the right person.

Set the answer policy by consequence

"Ask your manager" is not a handoff. It can send payroll, security, legal or benefits questions to someone who cannot resolve them, and "I do not know" leaves the employee with no next step. Start instead with an answer policy for each onboarding topic: the assistant may answer, answer with a warning, or must hand off. A simple registry can hold the rules:

topic: production_access
answer_mode: handoff_required
source_owner: security_operations
allowed_sources:
 - access-control-policy
 - engineering-onboarding-checklist
required_evidence:
 minimum_sources: 1
 owner_approved: true
 not_expired: true
route: security-intake
service_target_hours: 4

Build the rules around consequences. Office location and glossary questions can tolerate a lower evidence bar. Payroll elections, security access, medical benefits, compliance reports and customer commitments need a named authority or a mandatory handoff. Confidence can feed these rules but cannot replace them: escalate when no approved source covers every required step, however sure the model sounds. NIST describes its AI Risk Management Framework as intended for voluntary use and links examples of how other organizations apply it from its Resource Center. The registry is where you make the equivalent choice for this assistant: which topics require people, and what a wrong answer costs there.

An empty result is not "no policy"

Do not expose restricted evidence just to prove that the assistant found something. In Azure AI Search, for example, security filters trim results by the caller's user or group identity, so a document the employee may not open never reaches the assistant at all. That has a consequence that is easy to miss: the assistant cannot tell "there is no such policy" from "there is a policy you may not see". We recommend that on any topic with restricted sources, an empty result produces "I can't confirm this from the sources available to you" and a handoff, never "there is no policy on that". The owner can see everything and can answer.

Record the trigger, not just "low confidence"

A trigger should be inspectable after the event. Use at least these classes:

• Missing evidence: Retrieval found no approved source for the core instruction.
• Conflicting evidence: Two approved sources disagree about a decision, deadline, owner or sequence.
• Stale evidence: The supporting page is past its review date or references an obsolete system. AWS's explanation of RAG answers stale source data with maintenance, asynchronous updates of the documents and their embeddings in real time or in batches, so store the review date with the page.
• Permission boundary: The answer depends on material the employee cannot access.
• Approval required: A person must authorize the action even when the instructions are clear.
• Sensitive intent: The question concerns harassment, health, legal claims, credentials, security incidents or another protected channel.
• Repeated failure: The employee has rephrased the same unresolved question or rejected an answer.
• User request: The employee explicitly asks for a person.

Escalate a restricted question before generation, not after the model has processed content it should never have retrieved. For the sensitive-intent class we recommend skipping the model entirely: show a fixed, reviewed message that names the confidential route, such as the HR contact or the hotline, and keep the log entry minimal. A generated reply to a disclosure can promise confidentiality the process cannot keep or misjudge the tone, and its transcript would sit in an assistant log that was never designed to hold such cases.

For every other class, give the employee a precise message: "I cannot support this access request from the onboarding sources available to you. I have sent your question to Security Operations with the source checks I performed. You will see the response in this thread. Target response time: four business hours." It states the boundary, the owner, the next event and the timing, and it does not ask the employee to start over elsewhere.

Route each case to the owner of the source

Routing by department alone creates a shared queue with no clear responsibility. Build a topic-to-owner directory with a primary owner, a backup, a channel, a service target and an escalation path, and make the owner someone who controls or approves the source behind the answer.

GitLab's onboarding handbook shows what separate routes look like. Access requests belong to the IT team, with questions going to #it-help in Slack. A buddy helps with understanding and completing tasks. And a #questions channel, which new team members are added to from day one, takes anything the handbook cannot answer. Keep that division in your directory: the assistant retrieves instructions, a buddy explains team practice, a system owner grants access and People Operations handles employment policy.

Do not route from the words of the question alone. "I cannot access the customer refund page" could be a permissions problem, a broken link or doubt about refund authority. Combine the detected topic with the employee's role, location, onboarding stage, requested action and trigger class, using only attributes the routing service is permitted to use. When the topic is unknown, send the case to an onboarding triage owner and not to a random manager, and track how often triage has to reroute. A high reroute rate means the taxonomy or the intake questions need work.

Send a compact handoff packet

The owner should not have to read the whole conversation. Send a structured packet with only what is needed to resolve the case:

{
 "case_id": "onboarding-1842",
 "employee_role": "support_specialist",
 "onboarding_day": 6,
 "question": "Can I approve a refund above the standard limit?",
 "trigger": "conflicting_evidence",
 "retrieved_source_ids": ["refund-policy", "support-playbook"],
 "conflict": "approval owner differs between sources",
 "route": "support-operations",
 "employee_visible_thread": "onboarding-help",
 "service_target_hours": 8
}

Leave out hidden source text, unrelated chat history and sensitive employee attributes. Pass stable source identifiers and the reason for escalation, and let the receiving workflow retrieve authorized details under the owner's identity.

Keep the employee in the same visible thread when possible. A Kipwise onboarding workflow combines searchable knowledge with assigned reading and progress context. The handoff should fit that experience: the question starts in the knowledge workflow, moves to an owner, and returns as a clear resolution without forcing the new hire to copy information into another tool.

Create cases idempotently. If the employee repeats the question while a handoff is open, attach the message to the existing case instead of opening a duplicate, show the current owner and status, and let the employee add context or cancel.

Close the loop without training on replies

A human reply is not automatically approved company knowledge. The owner may give a case-specific exception, use shorthand or share something that should stay private, so separate resolving the case from publishing knowledge. Require one of four dispositions when the owner replies:

1. Resolved from an existing approved source.
2. Resolved as a documented exception that must not become general guidance.
3. Resolved with a new or corrected source proposed for review.
4. Transferred because the route or the ownership map was wrong.

Return the answer and any employee-visible citation to the original thread. If the gap is reusable, open a separate knowledge task and have a source owner review the change before it enters retrieval, with personal details removed. Do not let a model summarize every human reply into the company brain: that turns an informal answer into policy and can spread access-controlled information.

When the escalation path itself fails

• Owner unavailable: Route to the backup when the service target expires and tell the employee that ownership changed.
• Queue integration down: Save the case in a durable outbox, show a pending state and retry with an idempotency key.
• No valid owner: Send the case to onboarding triage and raise an ownership-gap alert.
• Sensitive case in a public channel: Move to the approved private intake without repeating the sensitive text.
• Permission changes mid-case: Recheck access before showing sources or resolutions.
• Contradictory human replies: Keep the case open until the accountable policy owner resolves the conflict.

Make pending, transferred, overdue and resolved states visible, so a cheerful assistant message cannot hide a failed handoff, and keep enough event data to reconstruct what happened without storing more about the employee than needed.

Test the route and size the queue

Test the whole route, not only trigger classification. Build scenarios from real onboarding questions with their expected owners, and include safe questions that should get direct answers so that excessive escalation shows up. For each scenario, check that:

• the assistant either answers from approved evidence or fires the expected trigger;
• restricted titles and excerpts never reach an unauthorized employee;
• the case reaches the primary or backup owner with the question, the trigger and the permitted evidence;
• a repeated message attaches to the open case and does not create a duplicate;
• the resolution returns to the employee's thread, and an exception does not enter the general knowledge base.

Track answer rate, escalation rate by trigger, reroute rate, time to first human response, time to resolution, overdue rate and repeat-question rate. Do not optimize for the lowest escalation rate: a lower rate can mean the assistant is answering beyond its evidence, so review samples of direct answers and escalations together.

Escalation is also a capacity decision. Every trigger class sends work to a person, so multiply escalations per hire by minutes per case by the hires you plan to add, and compare the result with the hours the owner actually has. A security queue with a four-hour target and nobody to staff it turns every refusal into a silent delay, and a hire who waits two days for access will read that as the assistant failing. We recommend widening the cohort only after owners have met their service target during the pilot. If the queue is the constraint, move low-risk topics to direct answers or add owner capacity before adding hires.

Put one route into production

Pick one bounded topic, such as routine software access. List its approved sources, answer boundary, primary owner, backup owner and service target, and implement the missing-evidence, permission, approval and user-request triggers. Then run ten representative questions through the whole workflow, including the owner's reply and the decision about whether the source needs updating, and ask recent hires whether each message made the next step clear.