How does BM25 work? Meaning, who developed it, uses in AI, Python, Qdrant and pgvector

BM25 works by scoring each document for a query: for every query word, it rewards documents where the word appears often, gives more weight to rare words, and adjusts for document length so long documents do not win just by being long. It is the default ranking in many search engines and the keyword half of hybrid retrieval in RAG. RAG.NextUpgrad fuses BM25 with dense vector search, then reranks, so this page explains how BM25 works, its name and origins, its role in AI, implementing it, and support in Qdrant and pgvector.

How does BM25 work?

For each query term, BM25 multiplies an inverse document frequency weight, higher for rare terms, by a term frequency component that grows with repetitions but saturates, normalised by document length relative to the average. The document's score is the sum over query terms.

Two parameters shape it: k1, typically 1.2 to 2.0, controls saturation, so the tenth occurrence adds much less than the first; and b, typically 0.75, controls length normalisation. An inverted index lists which documents contain each term, so only matching documents are scored, making BM25 fast at scale. Preprocessing such as lowercasing, stemming and tokenisation strongly affects results, especially for Indian languages and mixed-language text.

What does BM25 stand for?

BM25 stands for "Best Matching 25". "Best Matching" refers to a family of ranking functions developed for the Okapi information retrieval system, and 25 identifies the particular variant that became standard.

It is often called Okapi BM25 after that system. Variants exist, such as BM25F, which weights fields like titles and bodies differently, and BM25+, which adjusts for very long documents. In everyday engineering conversations, "BM25" simply means keyword relevance ranking as implemented in tools such as Lucene, Elasticsearch, OpenSearch and many libraries.

Who developed BM25?

BM25 was developed by Stephen Robertson, Steve Walker and colleagues working on the Okapi system at City University London, building on probabilistic retrieval theory by Robertson and Karen Spärck Jones. It was presented through the TREC evaluations in the mid-1990s.

Spärck Jones had earlier introduced inverse document frequency, the idea that rare terms matter more, in 1972, a foundation of BM25. Robertson and Hugo Zaragoza later wrote the standard theoretical review of BM25 in 2009. Their work remains one of the most influential contributions to search, still used in production systems decades later. Its longevity shows how well-designed simple methods can outlast many more complex ideas.

What is BM25 in AI?

In AI applications, BM25 is used as a lexical retriever: it finds documents containing the query's exact words, complementing embedding-based semantic search. In RAG systems, BM25 catches names, codes, acronyms and rare terms that vector search can miss.

It also serves as a baseline in information retrieval research, since neural retrievers must beat BM25 to be considered useful, and it often remains competitive on new domains. Learned sparse models, such as SPLADE, extend BM25's idea by learning which terms matter. In practice, combining BM25 with dense retrieval through reciprocal rank fusion gives more robust retrieval than either alone.

Where is BM25 used?

BM25 powers keyword search in Elasticsearch, OpenSearch, Apache Solr and Lucene-based systems, MongoDB Atlas Search, many website and documentation search tools, and libraries for Python and other languages. It is used in RAG pipelines as the keyword retriever in hybrid search.

You meet BM25 whenever you search documentation, e-commerce catalogues, support articles, logs or code with keyword-based search engines. Researchers use it as a standard baseline. In RAG.NextUpgrad, BM25 retrieves candidates alongside vector search, and reciprocal rank fusion merges both lists before reranking.

Is BM25 a reranker?

Usually no. BM25 is typically a first-stage retriever: it scores the whole indexed collection and returns top candidates quickly. Rerankers, such as cross-encoder models, then reorder a small set of candidates with more expensive, more accurate scoring.

BM25 can technically rerank a candidate set produced by another method, for instance rescoring vector search results by keyword overlap, but neural rerankers generally do this better. The common pipeline is: BM25 and vector retrieval in parallel, fusion, then a cross-encoder reranker on the top 20 to 50 results, then the language model. Each stage trades speed for accuracy. Measuring recall after each stage shows where your pipeline loses good candidates.

How do I implement BM25?

Tokenise documents, build an inverted index of term frequencies, compute each term's inverse document frequency and the average document length, then score each query by summing BM25 contributions for its terms over matching documents.

Also asked as: how to implement bm25

Steps:

• Tokenise and normalise text: lowercase, remove punctuation, optionally stem.
• For each term, count how many documents contain it.
• Compute IDF, commonly log of N minus df plus 0.5, over df plus 0.5, plus 1.
• Store term frequencies per document and document lengths.
• For a query, sum IDF times saturated, length-normalised term frequency.
• Sort documents by score.

For production, use a search engine or library rather than writing your own. Writing it once yourself, though, is an excellent way to understand ranking and to debug search results later.

How do I implement BM25 in Python?

The quickest way is the rank_bm25 library: tokenise your documents, build a BM25Okapi index, then score queries. For larger collections, use a search engine such as Elasticsearch, OpenSearch or a Postgres extension.

Also asked as: how to implement bm25 in python

from rank_bm25 import BM25Okapi
docs = ["refunds are processed within 7 days", "hostel fees are due in july", "refund policy for courses"]
bm25 = BM25Okapi([d.split() for d in docs])
print(bm25.get_top_n("refund policy".split(), docs, n=2))

Improve tokenisation with lowercasing, stemming and stop-word handling. LangChain's BM25 retriever wraps this library for quick hybrid RAG experiments.

Does Qdrant support BM25?

Yes, through sparse vectors. Qdrant supports sparse vectors with an IDF modifier, so you can store BM25-style term weights and have Qdrant apply inverse document frequency at query time. Libraries such as FastEmbed generate BM25 sparse vectors for Qdrant.

This lets you run keyword and semantic search in one database: store a dense vector and a sparse BM25 vector per chunk, then use Qdrant's hybrid query to fuse results with reciprocal rank fusion. Check the documentation for the current recommended setup, since Qdrant's hybrid search features have evolved across versions.

Does pgvector support BM25?

No. pgvector handles dense vector similarity only. PostgreSQL's built-in full-text search provides keyword ranking with functions such as ts_rank, which is not BM25, though it works well for many applications.

For true BM25 in Postgres, use extensions such as ParadeDB's pg_search, which adds BM25 indexes, or other community BM25 extensions. A common hybrid setup combines pgvector for semantic search with full-text or BM25 search in the same database, fusing results in SQL with reciprocal rank fusion. That keeps the whole hybrid pipeline inside PostgreSQL. Check an extension's maintenance status and managed-hosting support before relying on it in production.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail