Every RAG demo follows the same script: load a few documents, ask a question the team already knows the answer to, watch the model cite the right paragraph. It works because the corpus is small, the questions are friendly and nobody measures anything.
Production is where the assumptions meet real data. Here is what breaks first.
Retrieval drifts before generation does
When answers go wrong, teams instinctively blame the model. In our experience the retriever is the culprit far more often. Documents get added with different formatting, chunking rules that worked for the pilot corpus split tables in half, and embeddings of new content cluster away from old queries. The model then summarizes the wrong context perfectly.
Freshness is a pipeline problem
The second failure is silent staleness. The knowledge base updates, the index does not, and the system confidently answers from a policy that was replaced in March. Re-indexing needs to be an automated pipeline with alerts, not a script someone remembers to run.
Evaluate before users do
The fix for both is the same: a golden set of real questions with reviewed answers, run on every index update and every prompt change. Fifty good questions catch the majority of regressions before a customer sees them. Without that harness, your users are the eval, and they do not file tickets, they just stop trusting the answers.