What is vector search in Databricks? Indexes, GCP, Redis, SQLite, cost, and is it dead

Vector search in Databricks, called Mosaic AI Vector Search, is a managed service that indexes embeddings from your Delta tables and serves fast similarity search for RAG and recommendation applications, governed through Unity Catalog. It keeps indexes in sync with source data automatically. I build RAG systems such as RAG.NextUpgrad with hybrid retrieval, so this page covers Databricks Vector Search and its indexes, vector search on Google Cloud, Redis and SQLite, cost, whether vector search is dead, how Google uses it, and vector search engines.

What is vector search in Databricks?

Mosaic AI Vector Search is Databricks' built-in vector database. It creates vector indexes from Delta tables, serves similarity queries through endpoints, and integrates with Unity Catalog for access control and lineage, so retrieval follows the same governance as your data.

It supports embeddings computed by Databricks-managed embedding models or your own, metadata filtering, and hybrid search combining keyword and vector similarity. Typical uses are RAG over enterprise documents stored in the lakehouse, product recommendations and semantic search. Its main advantage is integration: if your data and pipelines already live in Databricks, you avoid copying data to a separate vector database and managing sync yourself.

How do I use vector search in Databricks?

Create a vector search endpoint, then create an index from a Delta table with a primary key and either a text column for managed embeddings or an embedding column. Query it with the Python SDK, REST API or SQL.

Also asked as: how to use vector search in databricks

from databricks.vector_search.client import VectorSearchClient
index = VectorSearchClient().get_index(endpoint_name="docs_ep", index_name="main.rag.docs_idx")
hits = index.similarity_search(query_text="refund policy", columns=["id", "text"], num_results=5)

Enable Change Data Feed on the source table for syncing. In RAG apps, pass retrieved text to a model serving endpoint and log retrievals. Check the documentation for current SDK names and options, which evolve.

What is a vector search index in Databricks?

A vector search index is the searchable structure built from your data's embeddings. Databricks offers Delta Sync indexes, which automatically stay in sync with a source Delta table, and Direct Vector Access indexes, where you write and update vectors yourself through the API.

Also asked as: what is vector search index in databricks

Delta Sync indexes can compute embeddings for you from a text column using a model endpoint, or use embeddings you computed in the table. Sync can be triggered or continuous. Direct Vector Access suits applications that produce vectors outside the lakehouse or need real-time writes. Both support filters on metadata columns. Choose Delta Sync when your source of truth is a Delta table, which is the common case.

What is vector search in GCP?

On Google Cloud, Vertex AI Vector Search, formerly Matching Engine, is the managed vector search service built on Google's ScaNN technology for low-latency search at large scale. Google Cloud also offers vector search in AlloyDB and Cloud SQL with pgvector, BigQuery and Firestore.

Choose Vertex AI Vector Search for very large, high-throughput workloads; AlloyDB or Cloud SQL for applications that want vectors next to relational data; BigQuery for analytics-style similarity over warehouse data; and Firestore for app developers already using it. Vertex AI also offers RAG-focused services that manage chunking, embedding and retrieval for you.

Can Redis do vector search?

Yes. Redis supports vector similarity search through its query engine, with vector fields indexed using exact FLAT or approximate HNSW indexes, plus filtering on other fields and hybrid queries. Newer Redis versions also add a dedicated vector data structure.

Redis keeps data in memory, so vector search is very fast, which suits real-time recommendations, semantic caching of LLM responses and session-level retrieval. The trade-off is memory cost for large collections. Many teams use Redis as a semantic cache in front of a model: if a new question's embedding is very close to a previous one, return the cached answer, saving cost and latency.

Can SQLite do vector search?

Yes, through extensions. sqlite-vec adds vector storage and similarity search to SQLite with no external dependencies, and runs anywhere SQLite runs, including browsers via WebAssembly and mobile devices. Some SQLite-based databases, such as libSQL, add native vector support.

SQLite vector search suits local-first apps, desktop tools, mobile apps, prototypes and small to medium collections, where running a separate server would be overkill. Brute-force search over tens of thousands of vectors is fast enough for many uses. For millions of vectors with heavy concurrent traffic, a server database or dedicated vector store is more appropriate.

Is vector search expensive?

It can be cheap or expensive depending on scale. Costs come from computing embeddings, storing vectors, keeping indexes in memory and serving queries. Small projects can run on an existing PostgreSQL database at almost no extra cost; billions of vectors need serious infrastructure.

Ways to control cost:

• Use an existing database with vector support before adding a new service.
• Choose smaller embedding dimensions where quality allows.
• Quantise vectors to cut memory, rescoring top results.
• Embed once and cache; re-embed only changed content.
• Filter by metadata to reduce search space.

Measure cost per thousand queries and storage per million vectors for your setup before committing.

Is vector search dead?

No. Claims that it is dead usually come from long-context models that read whole documents and from coding agents that search with tools like grep instead of embeddings. Both are useful, but neither replaces retrieval over large knowledge bases.

What is changing is that vector search is no longer used alone. Strong systems combine keyword search, vector search, metadata filters and reranking, and agents can choose among these tools. Long context helps after retrieval, letting models read more of the retrieved material. For millions of documents, fast similarity search remains essential, so expect vector search to be a standard component rather than a hype topic.

Does Google use vector search?

Yes. Google has described using neural matching and language models such as BERT in Search to understand the meaning of queries and pages beyond keywords, and it open-sourced ScaNN, its vector similarity search library, in 2020. Recommendations across its products rely on embeddings too.

Google's public explanations say AI systems help connect concepts in queries to relevant pages, alongside many other ranking signals. Its cloud products, such as Vertex AI Vector Search, are built on the same research. For website owners, this means writing clearly about topics in natural language matters more than exact keyword repetition, because semantic matching recognises related wording.

What is a vector search engine?

A vector search engine is software that stores vectors and returns the most similar ones to a query vector, using approximate nearest-neighbour indexes. It powers semantic search, recommendations and RAG retrieval.

Also asked as: what is vector search engine

Options include libraries such as FAISS, ScaNN and hnswlib for in-process search; vector databases such as Qdrant, Milvus, Weaviate and Pinecone; search engines with vector support such as Elasticsearch, OpenSearch and Vespa; and database extensions such as pgvector. Choose by scale, filtering needs, hybrid search support, operations capacity and where your data already lives.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail