"We want AI over our documents" is one of the most common requests in the AI market today. RAG (Retrieval-Augmented Generation) is the usual answer, but far from always the right one. Here is the decision tree we use before building anything.

What RAG is, and what it doesn't solve

RAG means: the user's query is used to retrieve relevant passages from your data (typically via vector embeddings), the retrieved passages are injected into the prompt, and the LLM composes an answer from them. The model doesn't need to "know" your data; it receives it in context.

What RAG doesn't solve: the quality of your data (outdated documents → outdated answers), access permissions (who may see what), structured queries ("how much did we invoice in Q2?"), or hallucinations. It merely gives them better source material.

The decision tree

  1. Is the data structured? Numbers, tables, DB records → you don't want RAG, you want an SQL/API layer, possibly an LLM that generates queries over the schema.
  2. How much is there? If the whole corpus fits into the context window (hundreds of pages these days), you don't need retrieval, just send the documents straight into context. No index, no infrastructure.
  3. Are people looking for answers, or for documents? If users need to find the right document (a contract, a policy), a proper full-text search with good filters is often enough. Search is an order of magnitude cheaper and its results are auditable.
  4. Is it about style, or about facts? Fine-tuning teaches a model style, format and domain jargon, not facts. For facts that change, fine-tuning is the worst possible tool: every change means retraining.
  5. Does the data change continuously, and is there a lot of it? Only here does RAG start making real sense.

When RAG makes sense

  • A large, living corpus (thousands of documents, continuous changes) that won't fit into context.
  • Answers must cite sources: RAG can return which passages an answer is based on, which is key for trust and auditability.
  • Queries come in natural language and the answer is a synthesis from multiple places.

When another path is better

  • Better search. When users know the domain and need to find a document fast. Try improving search first; it's often 20% of the cost for 80% of the value.
  • The whole corpus in context. Small knowledge bases (FAQ, price list, a few policies). Simpler, cheaper, with no index that can go stale.
  • A structured layer. Reports and numbers through SQL/API, not through embeddings.
  • Fine-tuning. Consistent tone, output format, domain language. In combination with RAG, not instead of it.

What to watch out for when you do build RAG

  • Access permissions. Retrieval must respect the user's document-level permissions. Otherwise an employee can "search up" the management salary table. This is the most common security hole in RAG deployments.
  • Chunking. How you slice documents determines answer quality more than model choice does. Slice along document structure, not fixed lengths.
  • Index freshness. A document changed, the index doesn't know, the AI answers from the old version. You need a re-indexing pipeline, not a one-off import.
  • Evaluation. A set of test questions with known answers, measured on every change (model, chunking, prompt). Without it you are tuning blind.
  • Hallucination with a citation. The model can "complete" an answer beyond the retrieved passages. Instruct it to answer only from context and admit "I don't know," and measure that in evaluation.

Summary

  • RAG is a tool for large, living, unstructured corpora that need citations.
  • For structured data, small bases and "find me the document," cheaper and more reliable paths exist.
  • The biggest deployment risks: access permissions, a stale index, and missing evaluation.

The best first step: take ten real questions from your users and try answering them manually over your data. Where you looked and how long it took will tell you more about the right architecture than any benchmark.