You cannot improve a RAG system you are not measuring, and "I asked it five questions and it seemed fine" is not measuring. Every serious change I made to RAG.NextUpgrad — hybrid search, reranking, the confidence gate — was justified by a number moving on a fixed evaluation set. Here is the setup, sized for a student or a small team.
Build the question set first
Collect 50 to 100 real questions. For each, record the passage (and page) that answers it, and the expected answer in one sentence. Include 10 to 20 questions the documents cannot answer — those test whether the system refuses correctly. This set is your most valuable asset; version it alongside the code.
Four metrics, in order of importance
- Retrieval recall@k — is the correct passage among the top k retrieved? If this is low, nothing downstream can save you. Fix chunking and retrieval before anything else.
- Faithfulness — does every claim in the answer appear in the retrieved context? Grade it with a rubric, by hand for the first hundred answers and with an LLM judge afterwards, spot-checking the judge.
- Refusal accuracy — on unanswerable questions, does the system say it does not know? On answerable ones, does it avoid refusing? A confidence gate is tuned entirely on this metric.
- Answer correctness — does the answer match the expected one? Useful, but it hides whether a correct answer came from the documents or from the model's memory.
A minimal evaluation loop
Run the set on every change — a script that indexes, queries, and writes a CSV is enough. Track recall@5, faithfulness rate, false-refusal rate and false-answer rate over time. When a change improves one metric and hurts another, you have a real decision to make instead of an argument.
Traps I fell into
- Writing questions by reading the documents, which produces questions phrased exactly like the text. Real users paraphrase; add paraphrased versions.
- Grading with the same model that generated the answer, without checking the judge. Judges are lenient on fluent prose.
- Evaluating only happy-path questions. The refusal set is where a production system earns trust.
A confidence gate without an evaluation set is a guess with a threshold. With one, it becomes the most defensible feature in the product — I explain the gate itself in why my RAG platform says "I don't know".
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
