Production RAG: Building an Enterprise Knowledge Q&A System

Retrieval, chunking, reranking, and evaluation—the engineering work required to move RAG from demo to production.

A RAG demo is easy: split documents, create embeddings, run vector retrieval, and put the results into a model. Production is hard for a different reason—you need to know why an answer is correct, and when the system should refuse to answer at all.

Retrieval quality comes before generation quality

If the retrieval stage never finds the right evidence, a stronger generation model can only make the wrong context sound more fluent. Production systems should evaluate retrieval metrics and answer metrics separately instead of judging only whether the final response “looks good.”

query
  → rewrite / normalize
  → hybrid retrieval
  → metadata filtering
  → reranking
  → evidence packing
  → generation
  → citation / verification

Chunking is not a fixed parameter

Code, FAQs, product documentation, and long research reports have very different structure. Chunk size should follow the semantics of the content and the retrieval objective, then be validated with offline evaluation—not copied from a tutorial that happens to use 500 tokens.

What production systems should measure at minimum

Recall@K: did the system retrieve the relevant evidence?
MRR / NDCG: did the best evidence rank near the top?
Faithfulness: is the answer supported by the supplied evidence?
Citation accuracy: does each citation actually support the claim attached to it?
Abstention quality: can the system refuse correctly when evidence is missing?

RAG is not simply “search added to an LLM.” It is an evidence-engineering system, and the model is only one component.