The request comes in identically from everyone: "we want a bot that answers from our documents." Almost always the next line is "train the neural network on our regulations." Training has nothing to do with it, and understanding why saves a month and a couple of million.
Why the model is not "trained" on documents
Fine-tuning changes the model's weights. It is expensive, needs thousands of examples, and solves a different problem: teaching the model a style or a format of answer. It fixes factual knowledge poorly: the model starts confidently confusing similar documents.
And the main thing: a regulation changes on Monday, while retraining takes days. A knowledge base you cannot update in a minute is useless.
How it works instead of training
RAG stands for retrieval-augmented generation, generation with search. A four-step scheme:
- Documents are cut into chunks of 200 to 800 words. A chunk has to be self-contained: a paragraph torn from the middle of a table will help answer nothing.
- Each chunk is turned into a vector, a set of numbers that reflects the meaning of the text. Chunks close in meaning get close vectors.
- The user's question is turned into a vector too, and the system finds the five to seven nearest chunks.
- The model receives the question together with the found chunks and writes an answer relying only on them.
The model remembers nothing. Each time it gets the needed fragment straight in the request. You changed a regulation, reindexed the document, and the next answer is already the new one.
What this approach gives you
A source under the answer. The system knows which document a fragment came from and can show the link. An employee checks it in two seconds.
Access control. Search can be limited to documents the user is entitled to. The model physically will not see what it should not.
Updates in a minute. You uploaded a new version, reindexed it.
Where such systems break
Chunking. The most underrated part. A table cut in half cannot be fixed by any model. Documents with a complex structure, contracts, price lists, instructions with nested clauses, need their own chunking logic. This is hours of work; the default setting will not do here.
Search over your industry's synonyms. Vector search understands common language. It does not know the company's internal jargon: "form 4-FSS" and "report to the social insurance fund" are different things to it. Cured by hybrid search: vector plus ordinary full-text over keywords.
Questions about quantity. "How many suppliers do we have in Siberia" will have search bring back five chunks from a reference table, and the model will count from them and confidently get it wrong. Such questions have to go as a query to the database, bypassing text search.
Outdated versions next to current ones. If the base holds a 2023 regulation and its new edition, search brings back both, and the model picks the one that is textually closer to the question. Document versioning is a mandatory part of the system.
No answer at all. When there is nothing in the base, the model by default makes up something plausible. You need a direct instruction to answer "this is not in the documents" and a check that the found chunks are relevant to the question in the first place.
What we need from you before the start
Documents in machine-readable form. Scans of contracts without recognition are a separate project. A PDF exported from Word reads well; a PDF assembled from photographs needs OCR and proofreading.
An understanding of who is entitled to see what. Access rights are designed before indexing.
Twenty to thirty real questions from future users. Without them there is no way to check whether the system works or merely produces coherent text.
How to tell the system works
Assemble a set of fifty questions with reference answers and run it after every change. Watch two things: did the right document turn up, and is the answer from it correct.
Separating them is mandatory. If the document was not found, the problem is in chunking or search. If it was found and the answer is wrong, the problem is in the model's instruction. A "percentage of happy users" metric shows neither.
What it costs
A pilot on one knowledge base starts at ₽300,000. Most of the hours go on chunking, tuning the search, and the test set. Connecting the model takes a day.
Running it is ₽5,000 to 50,000 a month at a thousand requests, depending on how much text goes into the model per answer.