How to Recover When AI Onboarding Knowledge Sync Fails

A quiet knowledge pipeline looks exactly like a broken one, so test it with a known change. We recommend a canary page that a scheduled job edits with a timestamp, a probe that reads the timestamp back through the assistant, and a daily list-and-compare of mandatory documents as the backstop for the change feed. Then show the index time beside answers about policy, because the new hire is the last monitor in the chain and the only one who reads the stale answer. This runbook assumes you run your own connector and index.

A working chat window says nothing about freshness

A company brain has several stages between a source document and an answer. The source system records a change, a connector discovers it, an ingestion worker fetches the content and permissions, an indexer transforms and stores it, and retrieval selects indexed passages for the model. The chat interface keeps responding when any earlier stage stalls.

Application uptime proves that a user can submit a question. It does not prove that the latest approved handbook, equipment guide or access procedure reached the index. Retrieval augmented generation depends on external knowledge that can be updated, as the AWS explanation of retrieval augmented generation makes clear, so when the update path breaks the answer stays fluent while its evidence is old or incomplete.

Common causes are an expired connector credential, a revoked source permission, a damaged change cursor, an indexer quota, a parser error, or a document format the pipeline cannot process. A full crawl can also run so slowly that it misses the onboarding team's freshness target. Treat each stage as a separate service with its own checkpoint instead of calling the whole system up or down.

Set a freshness target per knowledge class

A payroll deadline or a mandatory security instruction may need to appear within minutes. A culture guide can tolerate a daily refresh. One broad target for every source hides the importance of the documents new hires lean on most, so write the target down per class in a source registry:

The GitLab onboarding handbook is a practitioner example of onboarding organized around concrete tasks, access requests, buddies and support paths. That structure lets you list the first-week actions that are affected when a source falls behind.

Freshness targets are for you. The new hire also needs a way to see them. We recommend showing the source version and the time it was indexed beside every answer about mandatory content such as pay, security and policy. A dashboard watches the pipeline, while a hire who was just told that a policy changed yesterday can see "indexed three days ago" next to the answer and report it.

Monitor from source to answer

One connector success metric is not enough. Record checkpoints across the whole path and compare them continuously.

Watch the change cursor

Many source APIs expose an incremental change mechanism. Microsoft Graph delta queries, for example, return state links that clients use to request later changes without reading the whole collection again, and the Microsoft Graph delta query overview says clients must preserve and reuse the returned state URL. Store the cursor, its update time and the number of changes processed in each run. Alert when the cursor age exceeds the freshness target, when several runs process no changes despite known source edits, or when a cursor resets unexpectedly. A successful HTTP response is not enough, because the checkpoint has to advance.

Reconcile the feed against the source

A change feed is an optimization, not a guarantee. The same Microsoft documentation says delta tokens are valid only for a limited period before the client must run a full synchronization again, and that a 410 Gone response means starting over. We recommend a daily list-and-compare for mandatory content: list every document identifier and version in the source, list the same in the index, and alert on any difference older than the freshness target. It is usually much cheaper than a crawl, because it reads identifiers and versions instead of content, and it catches the missed events that a healthy-looking cursor never reports.

Watch indexer outcomes

A connector may fetch a changed file while the indexing stage rejects it. Capture the last start and finish time, processed document count, warning count, failure count and a sample of failed document identifiers. The Azure AI Search indexer monitoring guide describes status, execution history, errors, warnings and document counts as separate signals, and the same model applies when the implementation uses another search service. Alert on any critical onboarding document that fails indexing, because an aggregate success rate can hide one policy failure among thousands of ordinary pages.

Probe with first-week questions and a canary

Run a small set of questions that represent real first-week work. Each probe expects a source identifier, a minimum version or update timestamp, and the correct access decision. Check that retrieval cites the intended source and that a restricted document stays hidden when the probe uses a lower-permission role, rather than looking only for selected words in the answer.

probe_id: security_setup_new_engineer
identity_profile: engineer_new_hire
question: Where do I enroll my laptop in device management?
expected_source: security/device-enrollment
minimum_source_version: 2026-07-29T14:00:00Z
expected_access: allowed
maximum_index_age_minutes: 15

Real probes only prove freshness when something has changed. A source nobody edited for a week produces runs that process no changes, which is also what a stalled cursor produces. A canary page in each critical source settles it: a scheduled job writes a fresh timestamp into the page, and a probe asks the assistant for that timestamp. If the answer lags the last write by more than the freshness target, the pipeline is stale even though no human edited anything. That turns "no changes" from an ambiguous state into a measured one.

Keep probes small enough to run after every synchronization cycle, add one when an incident reveals a gap, and remove those that no longer distinguish a real failure.

Triage a suspected sync incident

Confirm whether the problem is content, transport, indexing, retrieval or permissions. Ask the reporter for the question, answer, cited source, time and user role, and preserve those details before anyone triggers a full crawl that could overwrite useful evidence. Then work through this sequence:

1. Open the cited source and confirm its current approved version with the owner.
2. Compare the source version with the connector's last observed change.
3. Compare the connector checkpoint with the indexer's last successful document version.
4. Run retrieval directly against the expected document identifier.
5. Repeat the query under the affected identity profile to test access filtering.
6. Run two unaffected onboarding probes to see whether the incident is isolated or systemic.

Classify the incident as narrowly as possible. One unparseable PDF needs a document repair, an expired source credential needs connector recovery, a stale change cursor needs replay, incorrect access metadata needs containment and a permission rebuild, and a broad index outage needs service restoration before content diagnosis.

Contain stale answers without switching everything off

Mark the affected knowledge classes unavailable and show a specific fallback. If payroll documents are stale, the assistant should say that payroll guidance is temporarily unavailable, link to the approved source when that is safe, and route the question to the named owner. It should not improvise from neighboring documents. Pause task completion for steps that depend on the affected evidence: a hire should not be marked complete on security training or a policy acknowledgment when the source version cannot be verified. Low-risk orientation material can stay available if its probes pass.

The Kipwise employee onboarding page describes assigned reading and searchable company knowledge as parts of onboarding. During an incident, treat those functions separately. Search may remain safe for unaffected sources while assigned reading tied to a failed policy index must pause.

Record the containment start time, affected sources, user groups, blocked tasks, fallback owner and decision maker. That record supports recovery and spares People Operations from guessing later whether a hire needs remediation.

Repair the smallest failed layer

Broad rescans take longer, create load and may hide the original cause, so repair the smallest failed layer first. For a credential failure, restore the connector identity with the minimum source permissions and check that it reads both content and access metadata. For a cursor failure, resume from the last known valid state when the source API supports it, and start a bounded rescan of the affected collection when continuity cannot be proved. For a parser or transformation error, quarantine the failing document, keep its source identifier and version, and test the corrected transformation against a copy. The page stays in a failed state until it is indexed or deliberately excluded by its owner. For indexer errors, clear the service constraint, retry only the failed set and read the execution history, since a green job after a retry does not show that earlier missed changes were recovered.

A reliable replay loop is idempotent:

for each change after last_verified_checkpoint:
 fetch content and source permissions
 transform content with a versioned parser
 write document using source_id and source_version
 verify indexed version and access metadata
advance checkpoint only after every required write succeeds

A stable source identifier stops replay from creating duplicate records, and advancing the checkpoint only after the required writes succeed stops the connector from acknowledging changes that retrieval cannot use.

Reopen answers through a recovery gate

A successful sync job is not recovery. Compare the source registry with the index: every affected source needs the expected version, a content hash or equivalent change marker, and the access policy version. Rerun the probes under each relevant role and confirm the answer cites the intended page and that a restricted identity cannot retrieve it. Replay the original reported question, compare the repaired answer with the authoritative source, and have the source owner confirm any policy interpretation. Do not let the model grade itself on a high-impact procedure.

Then inspect tasks completed during the incident window. Identify hires who saw stale guidance, missed required material or completed a step without verifiable evidence, send them a targeted correction, and reopen only the affected tasks. A vague company-wide warning is the wrong tool when the impact set can be identified precisely. Reopen answers when all of these hold:

• The change checkpoint is current and advancing.
• The indexer has no unresolved failures for affected documents.
• Indexed source and permission versions match the source registry.
• All critical onboarding probes pass under allowed and denied identities, and the canary reads back.
• The original incident question retrieves the approved source.
• The source owner approves the corrected answer or procedure.
• Affected hires and tasks have a documented remediation plan.
• Monitoring stays stable through at least one normal synchronization cycle.

After the incident

Write the root cause as the failed control, not the person who handled the incident: a connector credential with no expiry alert, a cursor kept only in memory, an indexer warning left out of alerts, or a probe set that covered general search but not mandatory policies. Add one durable improvement for each cause. Rotate credentials through a managed process, persist cursors transactionally, route critical document failures to an owner, keep a tested full rescan for collections that cannot replay history, and review the source registry whenever People Operations adds an onboarding task.

A perfectly current policy can still be missing from the company brain because transport failed. Source review and synchronization monitoring are separate controls, and each needs an accountable owner.

Run the first drill

Pick one critical onboarding source and record its source version, connector checkpoint, indexed version, permission version and probe. Make a harmless test edit and confirm that every checkpoint advances within the freshness target. Then pause the connector in a controlled environment and confirm that the system alerts, contains the affected answers, replays the missed change and passes the recovery gate. Fix whatever the drill exposes before the first real incident.