Exact duplicate rows in a training set are easy to catch with a simple hash check. Near-duplicates — the same underlying example paraphrased slightly, or scraped from two sources — are far more common and much harder to catch, and they bias training just as much.
Why near-duplicates matter
If forty of your five hundred examples are all near-identical, the model effectively sees that pattern eight percent of the time treated as forty percent, distorting what it learns to prioritise.
Detecting them
- Compute embeddings for each training example and flag pairs above a high similarity threshold for manual review.
- Check for examples sourced from the same origin document or conversation, which often produce near-duplicates even after individual cleaning.
- Cap how many near-duplicate examples of any one pattern survive into the final set, rather than removing all but keeping the count arbitrary.
See dataset validation before training.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
