Key takeaways
- BM25 is a keyword ranking function that scores documents by how often query terms appear, how rare they are, and document length.
- It builds on TF-IDF ideas but saturates term frequency and normalises for length, which ranks better in practice.
- Elasticsearch made BM25 its default scoring from version 5 because it outperforms classic TF-IDF on most collections.
- In RAG, BM25 catches exact terms such as codes and names that embeddings miss; combine it with vector search.
- In Python, rank_bm25 gives you BM25 in a few lines; most search engines and vector databases offer it built in.
BM25, short for Best Match 25, is a ranking function used by search engines to score how relevant a document is to a keyword query. It rewards documents that contain the query terms, especially rare terms, while preventing very frequent repetitions and very long documents from dominating. It is the default scoring in Elasticsearch, OpenSearch and Lucene-based search, and the keyword half of hybrid retrieval in RAG systems such as RAG.NextUpgrad. This page explains how it works, how it relates to TF-IDF, why Elasticsearch adopted it, and how to use it in Python and RAG.
What is the BM25 algorithm?
BM25 is a probabilistic ranking function that scores each document for a query by summing, over the query terms, a weight based on how rare the term is across documents and how often it appears in the document, adjusted for document length.
Also asked as: what is bm25 algorithm
It came out of the probabilistic relevance framework developed by Stephen Robertson and colleagues, with the "Okapi" BM25 version becoming standard in the 1990s. Despite its age, it remains a strong baseline that neural methods are compared against, and it is fast, needs no training and works in any language with a tokenizer. In modern AI systems, it is used alongside embeddings because it handles exact words reliably.
How does the BM25 algorithm work?
For each query term, BM25 multiplies the term's inverse document frequency (IDF), which is high for rare terms, by a saturating function of how many times the term appears in the document, normalised by document length. The document's score is the sum over query terms.
Also asked as: how does bm25 algorithm work
In formula form, for each term:
score += IDF(term) * tf * (k1 + 1) / (tf + k1 * (1 - b + b * doc_len / avg_doc_len))
Here tf is the term's count in the document, k1 controls how quickly extra repetitions stop adding score, typically between 1.2 and 2.0, and b controls length normalisation, typically 0.75. A document mentioning "pgvector" twice scores well; mentioning it fifty times scores only a little more.
What is a BM25 retriever?
A BM25 retriever is a retrieval component that returns the top-ranked documents or chunks for a query using BM25 scoring. In RAG frameworks such as LangChain and LlamaIndex, it is the keyword-based counterpart to a vector store retriever.
Also asked as: what is bm25 retriever
It indexes your chunks by their terms, then for each query returns the highest-scoring chunks. It needs no embedding model or GPU and is fully explainable: you can see which words matched. Its weakness is vocabulary mismatch: it will not match "car" to "automobile" or a paraphrased question to its answer. That is why it is usually paired with a dense retriever in an ensemble or hybrid retriever.
Why use BM25?
Use BM25 because it is fast, cheap, transparent and excellent at exact matching: product codes, error messages, names, acronyms and technical terms that embeddings often blur. It is also a strong baseline to compare any fancier retrieval against.
Specific advantages:
- No training or GPU: works immediately on any text.
- Exact terms: finds "ERR_CONNECTION_REFUSED" or "BCA301" reliably.
- Explainable: scores come from visible word matches.
- Robust across domains: no embedding model mismatch.
- Cheap at scale: inverted indexes are mature and fast.
In hybrid RAG, BM25 often rescues questions where vector search returns topically similar but wrong passages.
Is BM25 the same as TF-IDF?
No, but they are related. Both weight terms by frequency in a document and rarity across documents. BM25 adds term-frequency saturation and document-length normalisation, and comes from a probabilistic model rather than a heuristic.
Also asked as: is bm25 the same as tf idf
With classic TF-IDF, a term appearing twenty times counts roughly twenty times as much as appearing once, and long documents gain an advantage because they contain more words. BM25 limits the benefit of repetition through the k1 parameter and corrects for length with b, producing rankings that match human relevance judgements better in most collections. You can think of BM25 as a carefully tuned evolution of the TF-IDF idea.
Does BM25 use TF-IDF?
BM25 uses the same two ingredients, term frequency and inverse document frequency, but combines them differently from the classic TF-IDF formula. So it uses TF and IDF concepts, not the TF-IDF score itself.
Also asked as: does bm25 use tf idf
The IDF part in BM25 is typically a smoothed log ratio of documents without the term to documents with it, which gives rare terms more weight. The TF part is saturating and length-normalised. In practice this means BM25 behaves like a smarter TF-IDF: rare query words still dominate, but spamming a word or writing a very long document does not unfairly boost the score.
Why is BM25 better than TF-IDF?
BM25 usually ranks better because it models diminishing returns from repeated terms and normalises document length, both of which match how relevance works in real text. Its two parameters can also be tuned for a collection.
Also asked as: why is bm25 better than tf idf
In evaluations on standard information retrieval datasets, BM25 consistently outperforms basic TF-IDF ranking, which is why it became the default in modern search engines. TF-IDF vectors still have uses, for example as features for classifiers or for quick document similarity, but for ranking documents against queries, BM25 is the standard choice. If you need semantic matching, add embeddings rather than tweaking TF-IDF.
What is BM25, and why did Elasticsearch choose it from version 5?
BM25 is a keyword relevance scoring function, and Elasticsearch, through Apache Lucene, made it the default similarity from version 5 because it gives better relevance than the older TF-IDF-based scoring on most data, with sensible parameters.
Also asked as: what is bm25 and why elasticsearch chose this algorithm for scoring in version 5 and above
The older Lucene scoring tended to over-reward term repetition and handled document length less well. BM25's saturation and length normalisation fix those issues, and its k1 and b parameters can be tuned per field if needed. For users, the change meant better default search quality without configuration. OpenSearch inherits the same default, and many vector databases now implement BM25 too, so hybrid search is easier than ever.
How do I use BM25 in RAG?
Index your chunks with BM25 alongside your vector index, retrieve top candidates from both for each question, merge the lists with reciprocal-rank fusion, rerank the merged candidates, and give the best passages to the LLM.
Also asked as: how to use bm25 in rag
Practical tips:
- Tokenise consistently: lowercase, handle punctuation, keep codes and numbers intact.
- Retrieve more candidates than you need, such as 20 from each, before fusion.
- Use reciprocal-rank fusion, which merges by rank rather than incompatible scores.
- Add a cross-encoder reranker on the fused list.
- Measure recall at 5 with and without BM25 on your test questions.
This exact pipeline, dense plus BM25 with fusion and re-ranking, is how RAG.NextUpgrad retrieves before generating cited answers.
How do I use BM25 in Python?
The rank_bm25 library provides BM25 in a few lines: tokenise your documents, build the index, and score a tokenised query.
Also asked as: how to use bm25 in python
# pip install rank_bm25
from rank_bm25 import BM25Okapi
docs = ["pgvector adds vector search to PostgreSQL",
"BM25 ranks documents by keyword relevance",
"FAISS is a library for similarity search"]
tokenized = [d.lower().split() for d in docs]
bm25 = BM25Okapi(tokenized)
query = "keyword ranking bm25".lower().split()
print(bm25.get_top_n(query, docs, n=2))
For larger collections, use a search engine such as Elasticsearch or OpenSearch, PostgreSQL full-text search, or a vector database with built-in BM25. LangChain also offers a BM25 retriever for quick experiments.
Sources
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.
