What is RAG, and why it matters

Ask a general AI model a question about your business and it will answer confidently — sometimes correctly, sometimes not. It has no access to your documents, and no way to show its work. For enterprise use, that is a non-starter. Retrieval-augmented generation (RAG) is the pattern that fixes it, and it is the architecture behind most AI systems that survive contact with real users.

The retrieval pipeline, step by step

Instead of relying on what a model memorized in training, a RAG system retrieves the relevant passages from your approved sources at question time, and asks the model to answer grounded in those passages, with citations back to the source. In practice that means four stages, and each one has its own failure modes: ingest documents and split them into chunks small enough to retrieve precisely but large enough to preserve context; convert each chunk into an embedding and store it alongside metadata (source, date, owner, permission level); at question time, retrieve and rank the passages most relevant to the query; and generate an answer that the model is instructed to base only on what was retrieved, with a citation trail back to each source. The model becomes a reasoning layer over your knowledge, not a stand-in for it.

Why it matters for the enterprise

• Grounding. Answers come from your documents, not the model’s guesswork, dramatically reducing hallucination.
• Citations. Every answer can point to its source, so a human can verify it.
• Permission-aware retrieval. The system respects who is allowed to see what, which is critical for regulated and sensitive data.
• Freshness. Update the documents, and the answers update; no retraining is required.

Chunking and citations are where quality is won or lost

Two decisions determine whether a RAG system feels trustworthy or feels like a slot machine. First, chunking: cut documents along their natural structure (sections, clauses, and table boundaries) rather than a fixed character count, or retrieval returns fragments that read correctly but mean something different out of context. Second, citation granularity: pointing to “the policy manual” is not a citation a person can verify; pointing to the specific section and paragraph is. Tag every chunk with the metadata that governs whether it should be retrieved at all: which source it came from, when it was last updated, and who is permitted to see it. That metadata makes permission-aware retrieval and freshness possible later, rather than as an afterthought bolted on.

Evaluating a RAG system honestly

“It seemed to work in the demo” is not an evaluation. A real eval set scores two things separately: faithfulness: whether the generated answer actually matches what the retrieved passages say, without the model adding unsupported claims; and groundedness, or retrieval quality itself: whether the system found the right passages in the first place. A system can fail at either layer independently: perfect retrieval with a model that ignores it, or a faithful model reasoning correctly over the wrong passage. Build a standing set of representative questions with known correct answers and sources, score both layers against it before every meaningful change, and rerun it whenever source documents, chunking, or the underlying model changes. Treat it as a regression suite, not a one-time acceptance test.

Failure modes to design against

• Stale sources. If the index is not refreshed on a defined schedule, the system will confidently cite a policy that was superseded last quarter.
• Permission leaks. Retrieval that ignores document-level access control can surface a passage from a file the asking user is not allowed to see, even if the final answer only paraphrases it.
• Retrieval misses disguised as model errors. A wrong answer is often blamed on the model when the real fault is that the right passage was never retrieved; the two require different fixes.
• Over-retrieval. Returning too many marginally relevant chunks buries the right one in noise and degrades the final answer.

When RAG beats fine-tuning

The two are solving different problems, and conflating them wastes budget. Fine-tuning changes how a model behaves, including its tone, output format, and handling of a narrow task pattern, but it is a poor way to teach a model new, changing facts, since every update requires retraining and the result still cannot cite its source. RAG is the right tool when the underlying knowledge changes over time, when answers must be traceable to a specific document, and when access to that knowledge needs to vary by user. The two are not mutually exclusive: a fine-tuned model can sit on top of a retrieval layer when a workflow needs both a specific behavior and grounded facts, but for most enterprise knowledge problems, retrieval solves the actual problem and fine-tuning does not.

Production ops: RAG is a system, not a demo

A RAG pipeline is not done at launch; it needs the same operational discipline as any production system. That means a defined re-indexing cadence tied to how often source documents actually change, monitoring for retrieval quality drift as the document set grows, and re-running the evaluation set whenever anything upstream changes. This is also, per MIT’s 2025 NANDA research, where most AI initiatives quietly stall: the gap between a working pilot and a system that stays accurate in production is an operations problem more than a modeling one. BCG’s research on AI value creation puts a similar split on it directly: the technology is a fraction of what determines results, and the rest is the process, ownership, and maintenance built around it.

RAG is necessary, not sufficient

Good retrieval is where AI quality is won or lost, but it still needs evaluation, access control, and a workflow that fits how people actually work. That is the architecture-first approach behind every Celadon build.

Start with an AI Decision Sprint, or read AI without breaking confidentiality.