What is BM25 search? How it works, how to cite it, hybrid search, and key release dates

BM25 is a keyword ranking function: it scores how well a document matches a query by counting how often the query's words appear in it, giving rare words more weight and adjusting for document length. It has been the default ranking in search engines such as Lucene, Elasticsearch and OpenSearch for years. RAG.NextUpgrad combines BM25 with dense vector search through reciprocal rank fusion and reranking, so this page explains BM25, citing it, its history, hybrid search, its future, MongoDB, and when FAISS and Llama arrived.

What is BM25 search?

BM25 search ranks documents for a keyword query using the BM25 formula, short for Best Matching 25, from the Okapi information retrieval system. Documents containing more of the query's rarer words, in shorter or typical-length text, rank higher.

It needs no training data or model, is fast with inverted indexes, and works well for many search tasks, which is why it remains a strong baseline even against neural methods. It powers search in tools from website search bars to log analysis. In RAG, BM25 catches exact matches, such as product codes, section numbers and names, that embedding search can miss.

How does BM25 search work?

For each query term, BM25 multiplies the term's inverse document frequency, which rewards rare terms, by a saturating function of how often the term appears in the document, normalised by document length. It sums these scores across query terms.

Two parameters tune it: k1, usually between 1.2 and 2.0, controls how quickly repeated occurrences stop adding score, so a word appearing fifty times is not fifty times better; b, usually 0.75, controls how strongly long documents are penalised. An inverted index maps each word to the documents containing it, so only relevant documents are scored. Text processing, such as lowercasing, stemming and stop-word removal, strongly affects results.

What is BM25 lexical search?

Lexical search matches the actual words in a query to words in documents, as opposed to semantic search, which matches meaning. BM25 is the most widely used lexical ranking method, which is why "BM25 search", "keyword search" and "lexical search" are often used interchangeably.

Lexical search is precise and explainable: you can see exactly which words matched. Its weakness is vocabulary mismatch: "car" does not match "automobile" unless you add synonyms. Semantic search fixes that but can blur exact identifiers. That complementary strength is why hybrid search, combining lexical and semantic retrieval, has become standard in production RAG systems.

How do I cite BM25?

Cite the original Okapi work and, for a full treatment, the later review. The commonly cited original is: Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulieu, M. M., and Gatford, M. (1995). "Okapi at TREC-3". In Proceedings of the Third Text REtrieval Conference (TREC-3).

Also asked as: how to cite bm25

For the theory and variants, cite: Robertson, S., and Zaragoza, H. (2009). "The Probabilistic Relevance Framework: BM25 and Beyond". Foundations and Trends in Information Retrieval, 3(4). Many papers cite both: the 1995 paper for origin and the 2009 review for explanation. If you used a specific implementation, such as Lucene or rank_bm25 in Python, mention it and its parameter settings in your methods section.

When did BM25 come out?

BM25 emerged from the Okapi retrieval system developed at City University London in the 1980s and 1990s, and was presented through the TREC evaluations in 1994, with the proceedings published in 1995. The name refers to the 25th in a series of "best match" weighting functions.

It built on earlier probabilistic retrieval models by Stephen Robertson and Karen Spärck Jones from the 1970s. Variants followed, such as BM25F for documents with multiple fields like title and body. Decades later, it remains a default in major search engines and a standard baseline in research, a rare longevity in computing.

How do I combine BM25 and vector search?

Run BM25 and vector search in parallel for each query, merge the two ranked lists with reciprocal rank fusion, which scores each document by the sum of one over its rank plus a constant, then rerank the top merged results with a cross-encoder.

Also asked as: how to combine bm25 and vector search

Steps:

• Index each chunk for both BM25 and embeddings.
• Retrieve the top 20 to 50 from each method.
• Fuse with reciprocal rank fusion, using the common constant of 60.
• Rerank the top 20 to 30 with a reranker model.
• Send the best few chunks to the language model with citations.

This is the RAG.NextUpgrad pipeline. Fusion by rank avoids comparing BM25 scores and cosine similarities, which are on different scales.

Will BM25 and TF-IDF keyword search be replaced by neural search?

Not entirely. Neural search improves meaning-based retrieval, but BM25 remains competitive, cheap, explainable and robust across domains, and often beats neural models on unfamiliar data. The trend is hybrid search, not replacement.

Also asked as: will keyword (bm25, td-idf) be replaced for search by neural search

Benchmarks such as BEIR showed that BM25 is a hard baseline for zero-shot retrieval across diverse domains, and that combining it with dense retrieval gives the best results on many tasks. Learned sparse models, such as SPLADE, blend both ideas. In practice, keyword search will remain essential for exact terms, filters and transparency, while neural methods handle paraphrase and meaning.

How do I run top-k BM25 queries with MongoDB search indexes?

In MongoDB Atlas, create an Atlas Search index on your collection, then use the $search aggregation stage with a text operator, followed by $limit for your top k. Atlas Search is built on Apache Lucene and scores results with BM25.

Also asked as: top-k queries with mongodb search indexes (bm25

db.docs.aggregate([
  { $search: { index: "default", text: { query: "refund policy", path: "body" } } },
  { $limit: 5 },
  { $project: { title: 1, score: { $meta: "searchScore" } } }
])

For hybrid search, combine $search with Atlas Vector Search's $vectorSearch, using MongoDB's documented hybrid patterns or rank fusion. Check the current documentation for supported stages in your version.

When did FAISS come out?

FAISS, Facebook AI Similarity Search, was released as open source by Facebook AI Research in 2017. It provided fast similarity search and clustering of dense vectors, with GPU acceleration, and became a foundation for many vector search systems.

Its accompanying paper, "Billion-scale similarity search with GPUs", appeared the same year. For context on the modern retrieval stack: HNSW was published in 2016, pgvector arrived in 2021, and dedicated vector databases grew from around 2019. FAISS remains widely used for research, offline indexing and as a component inside other systems.

When did Llama come out?

Meta released the first LLaMA models in February 2023, initially to researchers. Llama 2 followed in July 2023 with a licence allowing commercial use, and Llama 3 arrived in April 2024, with further versions since.

Llama's open weights helped create the local AI ecosystem: Ollama, which launched in 2023, made running models such as Llama on a laptop a single command, and frameworks such as LangChain, released in late 2022, connected these models to retrieval and tools. DocuLens AI uses Llama models hosted on Groq for fast document analysis, a typical pattern for open models in production.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail