We built a golden set of 300 real support questions with expected answers, then measured each change:

We built a golden set of 300 real support questions with expected answers, then measured each change: • Better chunking (by heading, ~400 tokens, 15% overlap): +6 • Hybrid search (BM25 + embeddings): +8 • Cross-encoder reranking of top 30 → top 5: +7 • Prompting the model to cite chunk IDs and refuse if none apply: +2 accuracy, huge drop in hallucinations • Bigger model: +1 (not worth 3x the cost for us) The eval set was the most valuable thing we built. Without it every change is a vibe check.