Building an Enterprise Knowledge Base for AI Agents: A Methodology That Survives Production

Most AI agent projects that fail did not fail at the model. They failed at the knowledge base, the body of the company's own material the agent retrieves from, because it was built as a one-off index rather than a maintained system. This is the methodology we use to build one that is still correct in month six.

Step 1: Source inventory and triage

List every candidate source: help centre, policies, product documentation, contracts, resolved tickets, internal wikis, the systems of record. Then triage. Which documents are authoritative and which are drafts, duplicates or superseded? Who owns each source, and does it change? A knowledge base that treats a 2019 policy and its 2026 replacement as equally true will contradict itself, and the agent will pick one confidently. Most of the value of this step is what you leave out.

Step 2: Chunking that follows the document

Split documents by their own structure, sections, clauses, table rows, ticket threads, not by a fixed token count. A chunk that cuts a policy condition off from its exception produces retrievals that are technically relevant and materially wrong. Keep metadata with every chunk: source, section, date, owner, access level. You will need all of it later.

Step 3: Embeddings and a retrieval layer built for re-ranking

Embed the chunks and store them with their metadata. Then add a re-ranking step, an initial vector retrieval of, say, fifty candidates narrowed to the five best by a cross-encoder or a model pass, because first-pass vector similarity alone retrieves things that are about the right topic and not the right answer. Hybrid retrieval that combines keyword and vector search catches the exact-term queries that embeddings miss: part numbers, policy codes, names.

Step 4: Evaluate retrieval before you evaluate anything else

Build a test set of real questions from your history paired with the passages that should answer them. Score retrieval alone: did the right passages come back, and how far down the list? This is the step most projects skip, and it is the reason a demo that worked falls apart later: a strong model over weak retrieval produces fluent, confident, wrong answers, and answer-level evaluation cannot see why. Fix retrieval until the number is right, then evaluate generation.

Step 5: Access control at retrieval time

The agent must not retrieve what the user is not allowed to read. That means the access level carried on each chunk is enforced at query time against the user's identity, not by trusting the model to withhold it. In an enterprise knowledge base this is a hard requirement, and it is why the metadata from step 2 matters.

Step 6: Sync, ownership and drift

Content changes. A knowledge base indexed once drifts within a quarter. Define how each source is re-indexed, on change, on a schedule, or on a manual trigger, and who owns it. Log what was indexed when, so a wrong answer can be traced to a stale or missing document rather than debated.

Step 7: Observability

Every retrieval logged: query, candidates, what was selected, what the model produced. When the answer rate drops, this is how you find out whether the content changed, the retrieval broke, or the questions shifted.

A failure we were called in to fix

A distribution company in the Gulf had an internal assistant that answered staff questions about pricing, returns and supplier terms. It had worked well at launch. By month five, the operations team had stopped using it, because it kept quoting a supplier agreement that had been renegotiated in March. Nobody had indexed the new one. Worse, the old one had not been removed, so the assistant had two versions and picked confidently between them depending on how the question was phrased. When we looked at the setup, there was no source inventory, no owner per source, and no log of what had been indexed when. The fix was not a better model. It was a spreadsheet, honestly: every source, who owns it, how often it changes, how it gets re-indexed. Then a retrieval evaluation set built from the questions staff had actually asked over the previous quarter. Retrieval recall went from roughly 60 percent to the low 90s before we touched a single prompt.

Chunking rules we actually use

Fixed-size chunks are what most tutorials show and what most production systems should avoid. A policy document cut every 500 tokens will separate a condition from its exception, and the retrieval layer will serve the condition alone. Our rules are simple and document-shaped. Contracts and policies split at clause boundaries, with the heading path carried into every chunk so a clause knows which section it belongs to. Tables become one chunk per row plus the header row, never a paragraph of flattened cells. Support tickets keep the question and the accepted answer together. Product documentation splits at the level of one task per chunk. Every chunk carries the source, the section, the effective date, the owner and the access level. That metadata is what makes step five, access control at query time, possible. Without it you are trusting the model to withhold documents, and models do not withhold documents.

How to know when it is ready

There is a tempting moment, usually around week three, when the assistant answers the demo questions well and someone suggests launching. Resist it, and instead run the numbers. We ask for three before go-live. Retrieval recall on the evaluation set, meaning the right passage appears in the top results for at least nine questions in ten. Answer accuracy on the same set, scored by a person, not by another model, with any confident wrong answer counted as a failure rather than a partial. And a refusal rate: the share of questions where the assistant correctly says it does not know. A system that never refuses is guessing. When those three hold across a set that reflects real usage, including the awkward questions, it is ready. Then the work shifts to the sync schedule and the drift dashboard, because the day of launch is the day the content starts going stale.

Who should own it

The question that decides whether a knowledge base survives is not technical. It is who owns it after the engineers leave. In the deployments that keep working, one person on the business side owns the source inventory: they know which documents are authoritative, they are told when a policy changes, and they have a button or a scheduled job that re-indexes. Engineering owns the retrieval evaluation and the drift dashboard. Nobody else touches either. In the deployments that fail, ownership sits with a project team that was disbanded at launch, and six months later a director asks why the assistant is quoting last year's prices. Name the owner in the project plan, give them a fifteen-minute monthly routine, and put the drift numbers somewhere they will see them without asking. It costs almost nothing and it is the single biggest predictor of whether the system is still trusted a year on.

What this costs, and what it saves

A first knowledge base over a bounded corpus typically takes three to five weeks including evaluation; enterprise-wide systems with access control and continuous sync run two to four months, and the governance decisions take as long as the code. What it saves is the far more common alternative: an agent switched off in month three because nobody can tell why it started being wrong.

Our LLM integration practice builds this layer; our data engineering work handles the pipelines underneath it, and the agents that sit on top come from our AI agent development practice. For the tool and permission layer that sits beside the knowledge base, see our note on AI agent tooling standards.