BM25 explained: Best Matching 25 meaning, origin, formula, TF-IDF, BM25F, SPLADE, BM42

BM25, short for Best Matching 25, is the ranking function behind most keyword search engines, including Lucene, Elasticsearch and OpenSearch. It scores documents by query term frequency, term rarity and document length. I use BM25 alongside dense vectors in RAG.NextUpgrad, merging both with reciprocal rank fusion before reranking, so this page covers what BM25 means, its name and origin, how it is calculated, and how it compares with TF-IDF, BM25F, SPLADE and BM42, plus determinism and probabilities.

What does BM25, Best Matching 25, mean?

BM25 stands for Best Matching 25, a ranking function that estimates how relevant a document is to a keyword query. It rewards documents containing rare query terms, with diminishing returns for repeated terms and adjustments for document length.

Also asked as: bm25 best matching 25

It is often called Okapi BM25, after the Okapi information retrieval system at City University London where it was developed. BM25 needs no training: it works from term statistics in your index, which makes it fast, explainable and strong on exact names, codes and rare words. It remains a standard baseline in search research and a key component of hybrid retrieval, where it complements dense embeddings that capture meaning but miss exact matches.

Why is BM25 called BM25?

BM stands for Best Matching, and 25 refers to the version number: it was the 25th in a series of weighting functions the Okapi team developed and tested, after earlier variants such as BM11 and BM15.

BM11 and BM15 handled document length differently, and BM25 combined their ideas with a tunable parameter, b, that controls length normalisation. It performed best in experiments at the Text REtrieval Conference, TREC, and became the standard. Later variants, such as BM25F for documents with fields and BM25+ for very long documents, kept the name. The number has no other meaning, which surprises many people who expect it to describe the formula.

When was BM25 created?

BM25 was developed by Stephen Robertson, Karen Spärck Jones and colleagues at City University London, and presented in the Okapi team's experiments at TREC-3 in 1994. It built on probabilistic retrieval research from the 1970s.

Spärck Jones introduced inverse document frequency in 1972, and Robertson and Spärck Jones formalised probabilistic relevance weighting in 1976. BM25 added practical handling of term frequency and document length on top of that foundation. Despite its age, it remains hard to beat on many keyword search tasks, and Lucene adopted it as the default similarity in version 6, which made it the default in Elasticsearch and Solr as well.

How is BM25 calculated?

For each query term, BM25 multiplies the term's inverse document frequency by a saturated term frequency, then sums over all query terms. Saturation uses the parameter k1, and document length normalisation uses the parameter b.

In words: the score equals the sum, over query terms, of IDF times term frequency times k1 plus 1, divided by term frequency plus k1 times one minus b plus b times document length over average document length. Typical values are k1 between 1.2 and 2.0 and b equal to 0.75. For example, a rare term appearing three times in a short document scores higher than a common term appearing ten times in a long one. Libraries such as rank_bm25 in Python implement this in a few lines.

How is BM25 different from TF-IDF?

BM25 refines TF-IDF in two ways: term frequency saturates, so repeating a word many times adds less and less, and document length is normalised, so long documents do not win simply by containing more words.

Also asked as: how is bm25 different from tfidf

Classic TF-IDF grows linearly with term frequency, which rewards keyword stuffing and long documents. BM25's k1 parameter controls how quickly frequency saturates, and its b parameter controls how strongly length is penalised. BM25 also uses a probabilistic form of inverse document frequency. In practice, BM25 ranks noticeably better than plain TF-IDF on most search tasks, which is why it replaced TF-IDF as the default in Lucene-based search engines. Both remain useful as features in machine learning models.

What is the difference between BM25 and BM25F?

BM25F extends BM25 to documents with multiple fields, such as title, headings and body. It weights each field differently, combining term frequencies across fields before applying saturation, so a match in the title can count more than one in the body.

Also asked as: bm25 vs bm25f

Simply summing separate BM25 scores per field overcounts terms that appear in several fields, which BM25F avoids. It suits web pages, product catalogues and structured documents. Elasticsearch approximates this with the combined_fields query, which treats multiple fields as one with per-field boosts. Tune field weights using real queries and relevance judgements. For RAG over documents with titles and sections, field-aware scoring often improves retrieval of the right chunk.

BM25 vs SPLADE: which is better?

SPLADE usually retrieves better than BM25 on benchmarks because it learns term weights and adds related terms through expansion, capturing some meaning. BM25 needs no model, runs faster, works on any language with tokenisation and is easier to debug.

SPLADE requires running a transformer model over every document at indexing time and over queries at search time, adding infrastructure and latency. It may also perform worse on domains or languages its model was not trained on, where BM25 stays robust. A practical approach starts with BM25 plus dense vectors in a hybrid setup, then tests SPLADE if retrieval quality still falls short. Evaluate both on your own queries, measuring recall at the depth your reranker uses.

BM25 vs BM42: what is the difference?

BM42 is a sparse retrieval method proposed by Qdrant in 2024. It keeps BM25's inverse document frequency but replaces term frequency with attention weights from a small transformer model, aiming to work better for the short text chunks common in RAG.

The idea is that in short chunks, term frequency is usually one, so BM25's frequency component adds little, while attention weights indicate how important each word is to the chunk's meaning. After community review, Qdrant revised its initial benchmark claims, and results suggest BM42 is not a clear winner over BM25 or SPLADE in general. Treat it as an experimental option: test it on your data alongside BM25 before switching production retrieval.

Is BM25 deterministic?

Yes. Given the same index, tokenisation, parameters and query, BM25 always produces the same scores and ranking. Apparent differences usually come from index changes, tie-breaking or distributed search statistics.

In Elasticsearch and OpenSearch, each shard computes inverse document frequency from its own documents by default, so scores can differ slightly between shards or after updates; the dfs_query_then_fetch search type uses global statistics instead. Deleted documents still affect statistics until segments merge. Equal scores may return in different orders unless you add a tie-breaker, such as document ID. These properties make BM25 easy to test and debug compared with neural retrieval.

Can a BM25 score be a probability?

No. A BM25 score is an unbounded relevance score that depends on the query and collection, not a probability between 0 and 1. It comes from a probabilistic model, but the final score only ranks documents within a query.

Also asked as: can bm25 be a probability

Scores are not comparable across queries: a long query with rare terms produces higher scores than a short common one. To combine BM25 with other signals, normalise scores with min-max scaling, calibrate them into probabilities with logistic regression on labelled data, or use rank-based fusion. Reciprocal rank fusion ignores raw scores entirely and merges ranks, which is why RAG.NextUpgrad uses it to combine BM25 with dense vector results before reranking.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail