Key takeaways
- Text embeddings are fixed-length vectors that let applications compare texts by meaning.
- Embedding models use transformer encoders trained contrastively so matching texts score high in cosine similarity.
- OpenAI's text-embedding-3 models beat ada-002; small defaults to 1,536 dimensions and large to 3,072.
- Google's text-embedding-004 uses task types and 768 dimensions; gemini-embedding-001 is its successor.
- Changing embedding models means re-embedding everything and re-evaluating retrieval.
A text embedding turns a piece of text into a vector of numbers that captures its meaning, so software can compare texts by similarity. LLM applications use embeddings for semantic search, RAG, clustering and classification, while LLMs themselves use embedding layers internally. RAG.NextUpgrad combines dense embedding retrieval with BM25, so this page explains how text embeddings work, the difference from LLM embeddings, OpenAI's ada-002 and text-embedding-3 models, how to use them, and Google's text-embedding-004.
What is a text embedding in LLM applications?
A text embedding is a fixed-length vector representing the meaning of a sentence, paragraph or document, produced by an embedding model. LLM applications use these vectors to find relevant text, for example retrieving passages to feed an LLM in RAG.
Also asked as: what is text embedding in llm
Embedding models are usually separate from the chat model: you call an embeddings API or run a local model such as a Sentence Transformers checkpoint, store vectors in a vector database, and compare them with cosine similarity. Texts about the same topic produce nearby vectors even when the wording differs, such as how to reset my password and I forgot my login. This semantic matching is what makes modern search and retrieval far better than keyword matching alone.
How do text embedding models work?
A transformer encoder reads the tokenized text and produces a vector for each token; the model then pools these, often by averaging or using a special token, into one vector. Training makes similar texts produce similar vectors.
Training typically uses contrastive learning: the model sees pairs that should match, such as a question and its answer passage, and pairs that should not, and it adjusts weights so matching pairs score higher in cosine similarity. Large datasets of paired text, such as search queries with clicked results, teach general semantic similarity. Some modern embedding models are built from decoder LLMs fine-tuned for this task. Benchmarks like MTEB compare models across retrieval, clustering and classification.
What is an embedding inside an LLM?
Inside an LLM, the embedding layer is a learned table that converts each token ID into a vector, the first step of processing. Positional information is added, and every transformer layer then transforms these vectors into richer contextual representations.
Also asked as: what is embedding in llm
These internal embeddings differ from text embeddings used for search. They represent individual tokens, change at every layer, and are not designed for comparing whole texts. You could average an LLM's hidden states to get a sentence vector, but dedicated embedding models usually perform much better for retrieval because they are trained specifically for similarity. The size of the embedding layer equals the vocabulary size times the hidden dimension, often hundreds of millions of parameters.
What is text-embedding-ada-002?
text-embedding-ada-002 is an OpenAI embedding model released in December 2022. It produces 1,536-dimensional vectors, accepts up to 8,191 input tokens, and at the time replaced several older OpenAI embedding models with one cheaper, stronger model.
Also asked as: what is text embedding ada 002
It became the default embedding model for early RAG tutorials and applications. In January 2024, OpenAI released text-embedding-3-small and text-embedding-3-large, which outperform ada-002 on benchmarks such as MTEB and the multilingual MIRACL retrieval benchmark. Existing systems built on ada-002 still work, but new projects should use the newer models. If you migrate, re-embed all documents, since vectors from different models are not comparable.
What is text-embedding-3-small?
text-embedding-3-small is OpenAI's efficient embedding model, released in January 2024. It outputs 1,536-dimensional vectors by default, supports shortening them with a dimensions parameter, and beats ada-002 on retrieval benchmarks at much lower cost.
Also asked as: what is text embedding 3 small
OpenAI reported an MTEB average of about 62.3 for text-embedding-3-small versus 61.0 for ada-002, and a large jump on the multilingual MIRACL benchmark, from about 31.4 to 44.0. It is a strong default for RAG, semantic search and classification when you want a hosted model with good quality and low latency. Like other OpenAI embeddings, its vectors are normalised, so cosine similarity and dot product give the same ranking.
What is text-embedding-3-large?
text-embedding-3-large is OpenAI's most capable embedding model from the January 2024 release. It produces 3,072-dimensional vectors by default and scores higher than the small model on retrieval benchmarks, especially multilingual ones.
Also asked as: what is text embedding 3 large
OpenAI reported an MTEB average of about 64.6 and a MIRACL score of about 54.9. Its vectors can be shortened with the dimensions parameter, for example to 256 or 1,024, trading a little accuracy for much less storage, and OpenAI noted that even shortened large embeddings can outperform full ada-002 vectors. Choose it when retrieval quality matters more than cost, such as multilingual or technical corpora, and test it against the small model on your own queries.
How do you use text-embedding-3-small?
Call the OpenAI embeddings endpoint with the model name and a list of texts, then store the returned vectors. In Python, client.embeddings.create(model="text-embedding-3-small", input=texts) returns one embedding per input.
Also asked as: how to use text embedding 3 small
from openai import OpenAI
client = OpenAI()
texts = ["How do I reset my password?", "Steps to recover a login"]
resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
vectors = [item.embedding for item in resp.data]
Batch many texts per request, respect the token limit per input, and add dimensions=512 if you want smaller vectors. Store vectors with their text and the model name, embed queries with the same model, and rank results by cosine similarity.
What is text-embedding-004?
text-embedding-004 is a Google text embedding model released in 2024 for the Gemini API and Vertex AI. It produces 768-dimensional vectors and supports task types, such as retrieval query and retrieval document, to optimise embeddings for their use.
Also asked as: what is text embedding 004
Task types matter: embedding documents with the retrieval document type and queries with the retrieval query type improves search quality. The model also supports reducing output dimensionality. It performed strongly on MTEB at release among models of its size. Google has since released gemini-embedding-001, a larger, multilingual model with 3,072-dimensional output that can be shortened, which Google positions as the successor for new projects.
Is text-embedding-004 deprecated?
Google announced the deprecation of text-embedding-004 in the Gemini API after gemini-embedding-001 became generally available in 2025, and directed users to migrate. Check Google's model deprecation pages for the exact dates that apply to the Gemini API and Vertex AI.
Also asked as: is text embedding 004 deprecated
If you still use it, plan a migration: choose the replacement model, re-embed your entire corpus, since old and new vectors are incompatible, update your vector index dimensions if they change, and run a retrieval evaluation comparing old and new results before switching traffic. Keep the old index until the new one is verified. This applies to any embedding model change, which is why storing the model name alongside every vector is a good habit.
How do you use text-embedding-ada-002?
Use the same OpenAI embeddings endpoint with model="text-embedding-ada-002", passing a string or list of strings, and read vectors from the response. Each vector has 1,536 numbers; compare them with cosine similarity.
Also asked as: how to use text embedding ada 002
Unlike the newer models, ada-002 does not support the dimensions parameter, so vectors are always 1,536 long. If you maintain an ada-002 system, keep queries and documents on the same model, since mixing models breaks similarity. For new work, text-embedding-3-small offers better quality at lower cost with the same default dimension, which makes migration simple: re-embed documents, rebuild the index and compare retrieval results on a test set before switching.
Sources
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.
