Chunking decides everything
A passage cut in the middle of a table will never be retrieved correctly, whatever model you use afterwards.
A RAG system (retrieval-augmented generation) works in two stages: it first retrieves, from your documents, the passages most relevant to the question asked, then asks the model to write an answer based on those passages. If the first stage brings back the wrong excerpts, the second can only produce a wrong answer, or an invented one, delivered with great confidence.
Yet this search does not run on whole documents, but on pieces cut out in advance. The way you chunk therefore determines what can be found. Here are the mistakes we come across most often.
Mistake no. 1: cutting at a fixed size
The default method in many tools is to cut the text every N characters or every N words. It is simple, and it is almost always a bad idea. The cut falls in the middle of a sentence, a list or a procedure, and separates a question from its answer.
It is better to follow the structure of the document: headings, sections, paragraphs. A chunk should correspond to a complete idea. Its size then varies, and that is perfectly fine.
Mistake no. 2: mangling the tables
In business documents, the most valuable information is often found in tables: price lists, product specifications, rate scales, schedules. Extracted naively from a PDF, they become a string of values without headers, unreadable for a human and for a model alike.
A table should stay whole, or be split row by row, repeating the column headers on each row. "Product: paracetamol 500 mg; pack: box of 16; minimum stock: 200 boxes" is easy to find; an orphan row of numbers, never.
Mistake no. 3: losing the context
A paragraph taken from page 47 of a manual often begins with "In this case, you must…". On its own, it no longer means anything: which case? which document? which version? The system will struggle to find it, and the model will misinterpret it.
The solution is to enrich each chunk with its context before indexing it:
- the document title and the section path ("Quality manual › Receiving › Non-conformities");
- the date and version of the document;
- possibly, a short sentence summarizing the subject of the passage.
These few added words clearly improve the search, and allow the model to cite its source precisely. An answer that comes with its reference can be checked in one click; an answer without a source is either believed or not.
Mistake no. 4: chunks too small or too large
Chunks that are too small are precise but thin: the sentence retrieved contains the keyword, not the explanation. Chunks that are too large are complete but diluted: the useful passage is buried in three pages, and the search ranks it poorly.
There is no universal ideal size. A step-by-step procedure, a contract and a technical data sheet are not chunked the same way. The right size depends on your documents and your questions, and it can be measured. Which brings us to the costliest mistake.
Mistake no. 5: not measuring the search
Most teams evaluate their RAG system by reading a few final answers. That is not enough: when an answer is wrong, you do not know whether the fault lies with the search or with the writing.
We evaluate the two stages separately. For the search, a set of real questions, each paired with the passage that contains the right answer. We then measure how often that passage appears among the top results. Every change in chunking strategy is tested on this same set, and we keep what works.
This question set is built with the future users, not on their behalf. The questions an engineer imagines never look like the ones a receptionist or a pharmacist actually asks, with their own vocabulary and shortcuts.
Mistake no. 6: forgetting the documents themselves
Finally, no chunking strategy makes up for poor-quality documents: scanned PDFs without text recognition, contradictory versions of the same regulation, obsolete procedures that were never withdrawn. The system will faithfully retrieve the old version, and answer with the same confidence.
A RAG project therefore often starts with a sorting job: which documents are authoritative, which ones must be withdrawn, who is responsible for keeping them up to date. This work seems far removed from artificial intelligence; yet it is what determines how much users will trust the tool.
In a RAG project, the language model gets all the attention. Yet it is chunking, invisible and unglamorous, that makes the difference between a reliable assistant and a generator of well-written errors.
