It's tempting to stuff the top twenty retrieved chunks into the prompt just because the context window allows it. Past a certain point, more retrieved text doesn't improve answer quality — it dilutes the model's attention and costs more tokens for no gain.
A practical budget
- Start with the top 3-5 chunks by relevance score, not an arbitrary large number.
- Measure answer quality on your eval set as you vary the count — most tasks plateau surprisingly early.
- Reserve a fixed token budget for retrieved context, separate from the budget for instructions and conversation history, so one doesn't crowd out the other unpredictably.
The 'needle in a haystack' problem
Models can miss relevant facts buried in the middle of a long context even when they're technically 'in the window' — this is a documented weakness, not a rare edge case. Fewer, better-chosen chunks often beat more, weaker ones.
See where a RAG request's latency actually goes.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
