Recall@k and precision@k need labelled query-document relevance pairs, which almost no student project has on day one. Waiting to build that dataset before evaluating anything means shipping blind for weeks.
What works before you have labels
- Write ten to twenty questions you know the answer to, by hand, and check whether the right chunk shows up in the top results — a small manual eval set beats no eval set.
- Use an LLM to judge whether retrieved context is relevant to a query, as a rough automated proxy while you build a real labelled set over time.
- Log real user queries and periodically spot-check a sample manually — this is where your eventual labelled set should come from.
Building toward real evaluation
Every manual check you do can become a labelled example if you save it. Within a few weeks of normal use, a genuine evaluation set accumulates almost for free.
See RAG evaluation metrics: faithfulness and recall.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
