Vector search compares a query embedding to passage embeddings that were computed separately, so it never actually reads the query and the passage together. That is why it can rank a chunk about "refund policy for cancelled orders" above the one about "refund timelines" when a user asks how long refunds take. A reranker fixes exactly this gap.
What a cross-encoder does differently
A cross-encoder takes the query and one candidate passage as a single input and outputs a relevance score. Because it attends across both texts at once, it catches negations, specific entities and conditions that a bi-encoder blurs. The cost is that it must run once per candidate, so you never rerank the whole index — only the top 20 to 50 results from the first-stage retriever.
The two-stage pipeline
- Retrieve broadly: hybrid dense + BM25 search returns the top 30 candidates (see hybrid retrieval explained).
- Rerank precisely: score each candidate with the cross-encoder, keep the top 5.
- Gate: if the best reranked score is below a threshold, refuse instead of answering (see what a confidence gate is).
- Generate with citations from the surviving chunks only.
What it changed in RAG.NextUpgrad
Adding a reranker was the step that made the confidence gate trustworthy. Before it, first-stage scores were noisy enough that a threshold either blocked good answers or let weak ones through. Reranked scores separate relevant from irrelevant far more sharply, so the gate has something honest to measure. The platform still streams the answer token by token; the reranking pass adds a short pause before the first token, which users accept in exchange for fewer wrong answers.
Cost and latency, honestly
A small cross-encoder on CPU scores 30 passages in a few hundred milliseconds. On a free-tier server that is noticeable but fine; on GPU it is negligible. The memory footprint matters more than speed on small hosts — I covered that trade-off in the free-tier memory-budget playbook.
When to skip it
- Your corpus is tiny and questions are simple — first-stage retrieval already puts the right chunk on top.
- You have hard latency limits under 300 ms and no GPU.
- Your evaluation set shows recall@5 above 95% without reranking. Measure before adding machinery.
If your RAG app gives confident wrong answers, add a reranker before you touch the prompt. In my experience it is the highest-leverage component after chunking.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
