How an AI assistant over your internal documents actually works

When a business says "train an AI on our company data", what they almost always need is a different and far cheaper mechanism: find the right passage and give it to the model to read before it answers. This is usually called RAG — retrieval-augmented generation. Understanding it takes no technical background, and once you do, you know immediately when it will break.

Four steps, no more

1. 1Chunk the documents. Every file — procedures, template contracts, technical guides — is cut into passages of a few hundred words.
2. 2Index by meaning. Each passage becomes a list of numbers representing what it means, stored in a database. Two passages saying the same thing land close together even in different words.
3. 3Retrieve relevant passages. When a question arrives, the system finds the closest few in meaning — like search, but by idea rather than keyword.
4. 4Answer with sources. The retrieved passages go to the model with the question and an instruction: answer only from this material, and name the document you took it from.

All of the "intelligence" sits in steps 3 and 4. The model never memorises your documents — it reads only what is handed to it at question time. That matters: update a document and the system answers from the new version immediately, with no retraining.

Why the assistant answers wrongly

Almost every failure is in step 3, not step 4. If the system retrieves the wrong passage, no model however capable can answer correctly from it. The common causes:

• Contradictory documents. Three versions of a procedure exist and none states an effective date. The system picks the old one.
• The answer lives in a table or a scan. Without separate handling, tables get shredded into fragments and lose their meaning.
• The question needs figures aggregated. "Total revenue last quarter" is in no paragraph anywhere — that class of question belongs to the database, not the document store.
• Missing context. A passage reads "this case does not apply" without restating what "this case" is, because that was in the paragraph before.

A good internal assistant is one that says "I could not find that in the documents" instead of guessing.

Preparing the documents decides the quality

Output quality depends on the document store far more than on the model. Three jobs before rollout:

1. 1Remove superseded versions, or mark clearly which one is in force. This is the most tedious job and the most valuable.
2. 2Convert scanned images to readable text, tables especially.
3. 3Put a title and effective date on every document, so the system can prefer the newer one when they conflict.

Controlling quality with a sample question set

A simple, effective measure: write 30–50 questions staff genuinely ask, with correct answers approved by someone who knows. After every change — new documents, different chunking, a different model — rerun the set and count the right answers.

Without such a set, every improvement is a hunch, and it is very easy to fix one thing while quietly breaking another.

The permissions problem

This is the part most often overlooked and it can do real damage. If the document store also holds payroll, employment contracts or cost prices, the assistant must filter by the permissions of the person asking, at the retrieval step — not at the answer.

When NOT to use this approach

• Questions needing exact figures from a database — revenue, stock, receivables. Query directly and return the number; do not have a model read it out of prose.
• A very small document store, under a few dozen pages — simply putting all of it in the context is simpler and more accurate.
• Content that changes by the minute — connect to the live source rather than indexing on a schedule.

The running cost of this kind of system depends on the volume of questions and the length of context sent each time; how to estimate it is in The running cost of an AI feature.