Building an Internal AI Knowledge Base: A Guide for Operations Teams

Build the first version small and let escalation do the maintenance. Start with the pages behind the questions your team answers most often, rewrite each as a standalone answer with an owner and a review date, and connect only those. When the assistant cannot answer, send the question to the owner of the closest page, so every escalation is also a fix request for the page that failed. Test with last month's real Slack questions instead of invented ones, because real questions are short and vague, and that is where retrieval breaks.

This is for operations managers whose team answers the same procedural questions in Slack all week while new processes and new hires keep outrunning a scattered wiki.

Why a wiki fails as an answer system

AWS's explanation of retrieval augmented generation describes the pattern behind an internal AI knowledge base: the language model references an authoritative knowledge base outside its training data before it writes an answer. In practice the system cuts your documents into passages and retrieves the few that best match the question, so it reads passages, not pages. A page that reads well from top to bottom can still be a poor source.

• A step that points back with "as above" or "the usual approval" depends on text that may not be retrieved with it.
• A table loses its header row when it is cut into passages, and a row of numbers with no header means nothing.
• A page that holds three policies returns the wrong one when only one passage matches the question.
• Two pages that disagree about the same process get quoted with equal confidence, because the system cannot tell that one is old. Archive the old page instead of leaving it as a reference.

Most operations wikis were written for a person who already knows the context. Before you connect anything, take the pages behind your most repeated questions and rewrite each as a standalone answer: one question, the answer in the first lines, the exceptions after it, a named owner and the date it was last reviewed. Leave the rest of the wiki out of the index until someone owns it, because the system treats an outdated policy and a current one alike.

Decide who may see what before anything is indexed

Internal knowledge bases hold sensitive material, and the privacy guarantee belongs to the plan, not to the vendor. OpenAI's enterprise privacy page says it does not train on business data by default, and its help article on data use says content from the individual plans (Free, Plus and Pro) may be used for training unless the user opts out. Keep the pilot off personal accounts, and check the equivalent terms for any other provider you use.

Access control has to reach the retrieval step. When the system fetches passages it must filter them by the permissions of the person asking, so tag every indexed document with its access tier, and leave finance and executive material out until you have tested that an unauthorized user cannot surface it.

Put it where people already ask

A bot that staff can mention in a Slack or Microsoft Teams channel gets used more than a separate web portal, which often sees low engagement because it forces a context switch. You can build the stack yourself or buy a platform that bundles the vector database, the language model connection and the interface. Judge either one by how it connects to your wiki and chat tool, whether every answer shows its source page and owner, whether document permissions carry over, and what an escalation looks like. Have the bot say in its first reply that its answers are generated from internal documents, with the source linked, so staff know what they are reading.

For the risk side of that evaluation, the NIST AI Risk Management Framework gives a neutral way to govern, map, measure and manage AI risks.

Escalate to the owner of the closest page

The assistant has to know when to say it does not know. It should hand off when it finds nothing close enough, when its sources contradict each other, or when the question needs a decision (an exception, a spend, a person) and not a procedure. Instead of guessing, it returns the closest documents and a named human.

We recommend that the human is the owner of the closest page, not the whole operations channel. One person gets the question, can answer it, and can fix the page that failed to answer it. Log each escalation with the question, the page and the outcome, whether that was a new page, a rewrite or an archive. After a month the log is your content backlog, and it shows which pages have an owner who does not exist.

Test with the questions your team really asked

Export a month of procedural questions your team answered in Slack. The count gives you the baseline of how many such messages arrive per day, and the questions themselves are your test set. Real questions are short, vague and full of internal names, such as "where do I log the Acme refund", so they test retrieval harder than the complex multi-part questions a test group invents. Score each answer as correct and cited, correct without a source, wrong, or correctly escalated. When an answer is wrong, the page behind it usually needs clarifying before the system does.

New hires make the best testers after launch. They have no institutional knowledge, so their questions expose the unwritten rules your pages assume, and each one belongs in the test set and on a page. Re-run the same set whenever the model provider ships a new version, when you change how documents are split, and when you load a large batch of pages, because each of these can change what the system retrieves and how it words the answer.

After launch, track the drop in direct messages to the operations team and the share of questions answered without a human. A high resolution rate means little if people accept wrong answers, so read a sample of accepted answers each week. Add a thumbs-down that asks one question, what was wrong, and read the weekly pile with the knowledge manager. A sudden spike in questions about one policy, travel reimbursement for example, often means the policy is confusing and needs a rewrite. Multiply the questions answered without a human by the minutes a person takes to answer one, and you have a figure to show leadership.

Keep the index current

Assign a knowledge manager who owns the owners: updating policies, reviewing the escalation log and the thumbs-down pile, and checking that every page still has a review date. Keep the index synchronized with the source with webhooks or a scheduled sync. When a page changes in the wiki or cloud storage its passages must be re-indexed, and when a page is deleted its passages must go too, or the system keeps quoting a retired guideline. Without ongoing maintenance, trust in the system degrades quickly.

Add departments as collections with owners

Human resources, IT and sales enablement will ask for their own version once operations has one. Give each department its own collection with its own owner and review dates, behind the same assistant. Marketing documents then do not pollute the operations answers, each team sees its own gaps, and you keep one place to ask and one escalation log. Each collection carries its own access tiers, so HR material stays with HR. This is our recommendation: separate bots per department mean separate maintenance and separate quality.

Start with a documentation audit: export last month's questions, list the pages behind the most common ones, name an owner for each, and choose a pilot group.