Every RAG tutorial picks a chunk size and moves on. I ran the same document set through 128, 256, 512 and 1024-token chunks, same embedding model, same twenty-question eval set, and measured recall@5 and answer faithfulness for each.
What moved and what didn't
Recall@5 rose from 128 to 512 tokens, then flattened. Past 512, chunks started mixing two unrelated ideas, and faithfulness dropped even though recall looked fine — the right chunk was retrieved, but it also carried a wrong neighbouring fact the model quoted.
The one variable that mattered more than size
Overlap mattered more than raw size. A 256-token chunk with 50-token overlap beat a 512-token chunk with no overlap, because fewer answers got cut across a chunk boundary.
- Start at 256–400 tokens with 15–20% overlap; it is a better default than 512 with none.
- Chunk by structure first (headings, paragraphs), then cap by token count — never split mid-sentence.
- Re-run your eval set after any chunking change; the effect is rarely intuitive.
See RAG systems, a complete beginner's guide for the pipeline this experiment ran on, and what a confidence gate is for what to do when a chunk sits at the edge of relevance.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
