It is easy to create an impressive demonstration: upload a few PDFs, ask a question and receive a fluent answer. The real test begins when that answer helps prepare a legal file, qualify a tender or locate a contractual obligation.
In those settings, “the answer sounds plausible” is not an adequate acceptance criterion. People need to retrieve the source, understand why it was selected and know what the system failed to find.
Useful enterprise RAG is a research and evidence product. The language model is only one component.
1. Give every document a stable identity
A file, its OCR text, pages, index chunks and summaries are different representations of the same document. The system has to preserve their lineage.
For every source, I recommend storing at least:
- a stable identifier independent of the filename;
- document version, date and status;
- access rights and business scope;
- the link from each chunk to its page and original region;
- a file fingerprint to detect duplicates and changes.
Without this foundation, re-indexing can break citations, an obsolete version may appear current and two copies of one document may artificially reinforce an answer.
2. Treat layout as data
Extracting text is not enough. A table, footnote, repeated header or column can change meaning. OCR introduces its own failures: confused digits, merged words and incorrect reading order.
The pipeline should therefore keep the original, visual rendering, raw text and reconstructed structure separate. For important documents, retaining the coordinates of each block is useful. The interface can then open the exact page and highlight the excerpt that was used.
This stage is less spectacular than choosing a model. Yet it sets the quality ceiling for everything that follows.
3. Build chunks around meaning and evidence
Splitting every text at 500 characters is easy but rarely satisfactory. A good chunk is small enough to target and complete enough to remain understandable.
The strategy depends on the corpus: paragraphs for commentary, articles for legislation, clauses for contracts, or table cells enriched with their row and column headings. Overlap can preserve context but also increases duplicate results.
Every chunk should carry metadata such as title, section, page, date, document type, relevant entity and access level. This metadata supports retrieval filters and makes a citation useful.
4. Combine retrieval modes
Semantic search is good at finding an idea expressed in different words. It is less dependable for a case number, exact reference, rare name or date. Lexical search has the opposite profile.
A robust architecture will generally combine:
- structured filters to narrow the authorised corpus;
- lexical search for exact terms and references;
- vector search for meaning;
- reranking candidates against the complete question;
- a diversity rule to avoid returning five adjacent chunks from one passage.
The weight of each signal should be observable. Otherwise, when a relevant source disappears, no one can explain at which stage it was lost.
5. Make citations a product contract
A citation is not a link appended after generation. The answer should be built from evidence units identified before the model call.
A simple contract can require every factual statement to contain the identifier of an authorised source. The server then validates those identifiers, reconstructs links and rejects any identifier that was not in the context. The interface displays the title, version, page and exact excerpt.
The right interaction is more than “open PDF”. A user should move from a claim to the precise region supporting it, then return to the answer without losing their work.
6. Design for no answer
A professional system must be able to say, “I do not have sufficient evidence.” That requires an explicit exit rule, not just a sentence in the prompt.
Several signals can contribute: low retrieval scores, contradictory sources, obsolete information, incomplete corpus or a question outside scope. The product can then request clarification, suggest the closest documents or transfer the case to a person.
Uncertainty should trigger a useful action. A “medium confidence” badge with no consequence does not help anyone.
7. Evaluate with real business questions
Testing ten questions invented by the technical team mostly measures that team’s familiarity with the system. Build an evaluation set from real searches: frequent questions, ambiguous wording, exact references, cases with no answer and contradictory documents.
For each question, retain:
- expected sources;
- points the answer must contain;
- claims that are forbidden without evidence;
- the expected behaviour when information is insufficient.
Evaluate retrieval and generation separately. If the correct passage never reaches the context, changing the prompt will not solve the problem. If it is present but used incorrectly, the work concerns the instruction, model or answer construction.
8. Operate it as a living product
The corpus changes in production. Documents expire, permissions evolve and users introduce new wording. Track unanswered questions, opened sources, human corrections, response time and cost per journey.
Security must apply before retrieval: a model should never receive a passage the user is not allowed to access. Logs should support analysis without needlessly copying sensitive content.
Finally, every change to the model, OCR, chunking or ranking should be replayed against the evaluation set. Without that discipline, an improvement visible on three examples can silently damage twenty real uses.
A credible first release
The first scope does not need to cover an entire company. A clearly owned corpus, twenty to fifty evaluation questions, hybrid retrieval, page-level citations and a user feedback loop already form a serious product.
This is the approach AI7 applies to document systems such as LegalAZ: begin with the research journey, make evidence navigable and measure what the system retrieves separately from what it writes. The outcome is not a universal chatbot. It is a bounded, controllable and improvable work tool.