Students assume the LLM call is the slow part of a RAG request. I profiled a real endpoint end to end and it usually isn't.
The breakdown
- Query embedding: 20–80ms, cheap unless you're calling a hosted API per request.
- Vector search: usually under 50ms for a well-indexed store, but grows fast on an unoptimised flat index.
- Reranking: often the biggest slice — a cross-encoder over 20-50 candidates can take 200–500ms.
- Generation: streaming hides most of this from the user, even though it is the largest raw time.
The one fix that helps most requests
Stream the generation token by token so perceived latency drops even when total time doesn't, and rerank fewer, better-filtered candidates rather than reranking everything the first-pass search returns.
See streaming LLM responses with FastAPI and SSE and reranking with cross-encoders.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
