The embedding model decides what "similar" means in your retriever. Swap it and the same query returns different chunks. Yet most projects pick one from a tutorial and never revisit it. Here is how I choose, and how I keep the option to change later.
What actually differs between models
- Dimension — 384 versus 1024 dimensions changes index size and search cost more than it changes quality on small corpora.
- Context length — a model that truncates at 512 tokens silently ignores the end of long chunks. Match chunk size to the model.
- Domain — general web models handle policy PDFs well; code and legal text benefit from models trained on them.
- Language — Hindi and Hinglish queries against English documents need a multilingual model, or you will watch retrieval fail on your own users.
- Hosting — an API model is simplest; a local model keeps documents on your server, which some clients require.
Test before you commit
Take the evaluation set from how to evaluate a RAG system and index the corpus with two or three candidate models. Compare recall@5 and the rank of the correct passage. Differences are often smaller than tutorials suggest — and when they are small, pick the cheaper, faster or more private model.
Design for switching
Store the model name and dimension alongside each index, keep the embedding call behind one function, and make re-indexing a command rather than a migration. RAG.NextUpgrad's multi-provider fallback exists for generation, but the same discipline applies to embeddings: never let a single vendor's model become load-bearing for your architecture.
When to switch
Switch when recall on your own questions is the bottleneck and hybrid search plus reranking have already been added — not because a leaderboard changed. Leaderboards measure someone else's queries.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
