Embeddings deep dive: meaning, sentences, sum vs mean, history, OpenSearch, training

Embeddings are vectors that represent meaning, and they power semantic search, RAG and recommendations. Beyond the basics, practical questions arise about providers, pooling, history, search engines and training. RAG.NextUpgrad uses dense embeddings alongside BM25 with reranking, and FaceVision compares face embeddings, so this page covers OpenRouter, what LLM embeddings represent, sentence embeddings, sum versus mean pooling, the history of embeddings, OpenSearch, symmetric and asymmetric search, training and computing embeddings.

Does OpenRouter support embeddings?

OpenRouter is primarily a gateway for chat and completion models across many providers. Its support for embedding models has been evolving, so check its current documentation; many teams call embedding providers directly instead.

Embeddings have different requirements from chat: you need one consistent model for both documents and queries, stable versions, and high-throughput batch requests. Switching embedding models later means re-embedding your entire collection, so choose carefully and record the model name with every stored vector. Direct options include OpenAI, Cohere, Voyage AI, Google and Mistral embedding APIs, or open models such as BGE-M3 hosted yourself. Using OpenRouter for generation and a dedicated provider for embeddings is a common, workable split.

What do embeddings in LLMs represent?

Inside an LLM, embeddings represent tokens as vectors. The input embedding layer gives each token a learned starting vector, and each transformer layer updates it with context, so final representations capture meaning in that specific sentence.

Also asked as: what do embeddings in llm represent

Directions in embedding space often correspond to concepts, such as plurality, tense or sentiment, which interpretability research studies to understand model behaviour. Models pack many features into the same dimensions, a phenomenon called superposition. Embedding models used for search are trained differently, producing one vector per text that captures overall meaning for similarity comparison. Both kinds of embeddings are learned, not hand-designed, and are only meaningful relative to the model that produced them.

What are sentence embeddings?

Sentence embeddings are single vectors representing the meaning of a whole sentence or passage, so similar sentences have similar vectors. They power semantic search, clustering, duplicate detection and RAG retrieval.

Early approaches averaged word vectors, which lost word order and context. Sentence-BERT, introduced in 2019, fine-tuned BERT with siamese networks to produce sentence embeddings that work well with cosine similarity, making semantic search practical at scale. Modern models such as BGE, E5, GTE and commercial embedding APIs extend this with larger training data and longer inputs. The sentence-transformers library makes generating them easy in Python. Choose models whose maximum input length fits your passages.

Sum vs mean of word embeddings: which is better for sentence similarity?

For cosine similarity, sum and mean give identical results, because the mean is just the sum divided by the word count, which changes length but not direction. For Euclidean distance or dot products, the mean is usually better, since sums grow with sentence length.

Also asked as: sum vs mean of word-embeddings for sentence similarity

Plain averaging treats every word equally, so common words dominate. Weighted averages work better: TF-IDF weighting or the smooth inverse frequency method from Arora and colleagues, which downweights frequent words and removes a common component, form a strong baseline. Still, averaged word vectors ignore word order, so modern sentence embedding models, trained specifically for similarity, clearly outperform them. Use averaging only for lightweight or educational setups.

When were word embeddings invented?

Neural word embeddings emerged in 2003, when Bengio and colleagues learned word vectors inside a neural language model. They became widely used after Word2Vec in 2013 and GloVe in 2014, which made training them fast and practical.

Collobert and Weston showed in 2008 that learned embeddings help many language tasks. Word2Vec, from Mikolov and colleagues at Google, popularised analogies such as king minus man plus woman approximating queen. FastText in 2016 added subword information for rare words. Contextual embeddings followed with ELMo in 2018 and BERT later that year, giving the same word different vectors in different sentences, which is the basis of modern language models.

Who invented vector embeddings?

No single person invented them. The idea grew from the distributional hypothesis of linguists such as Zellig Harris and J. R. Firth in the 1950s, Gerard Salton's vector space model for information retrieval, and latent semantic analysis in 1990.

Salton's work in the 1960s and 1970s represented documents as vectors of term weights for search, a direct ancestor of today's retrieval. Latent semantic analysis compressed those vectors to capture hidden topics. Neural approaches from Bengio, Mikolov and others then learned dense embeddings directly from data. Today's embedding models build on all of these ideas, combining distributional meaning, vector similarity and deep learning at massive scale. Understanding this history helps explain why retrieval ideas recur.

What do search embeddings do in OpenSearch?

OpenSearch's neural search features generate embeddings for documents during indexing and for queries at search time, using models hosted in OpenSearch or connected remote models, so you can search by meaning alongside traditional keyword search.

Also asked as: what does search embeddings do opensearch

An ingest pipeline with a text embedding processor converts fields into vectors automatically, and the neural query type embeds the user's query with the same model. Connectors link to external models from providers such as Cohere or Amazon Bedrock, useful for multilingual search. Hybrid search combines BM25 and vector scores with normalisation, similar in spirit to how RAG.NextUpgrad merges dense and BM25 results. Test relevance on real queries before switching from keyword search.

What is the difference between symmetric and asymmetric semantic search?

In symmetric search, queries and documents are similar in form and length, such as finding duplicate questions or paraphrases. In asymmetric search, short queries match longer documents, such as a question finding a relevant passage.

Also asked as: embeddings for symmetric vs. asymmetric semantic search

Use embedding models trained for the right case. Asymmetric models learn from query and passage pairs, such as search logs or question answering data, and some, such as E5, expect prefixes like query and passage to signal roles. Symmetric models are trained on paraphrase pairs. Using a symmetric model for search can rank passages poorly, and vice versa. RAG is almost always asymmetric, so choose retrieval-focused embedding models and test with real questions.

How are text embeddings trained?

Text embedding models are trained with contrastive learning: the model sees pairs of related texts, such as a question and its answer, and learns to place them close together while pushing unrelated texts apart.

Training typically starts from a pretrained transformer, then uses large collections of weakly related pairs, such as titles and bodies, followed by fine-tuning on high-quality labelled pairs with hard negatives, passages that look relevant but are not. Loss functions such as InfoNCE use other examples in the batch as negatives. Techniques like Matryoshka training let embeddings be shortened with little quality loss. Domain fine-tuning can improve retrieval for specialised vocabularies. Evaluate trained models on held-out queries from your own domain.

What does it mean to compute embeddings in AI?

Computing embeddings means running data, such as text, images or audio, through an embedding model to produce vectors. It is an inference step, done once for each stored item and again for each query at search time.

In practice, you batch inputs for efficiency, often on a GPU, and store results with IDs and metadata in an index or vector database. Computing costs scale with data size, so large collections may take hours and should be versioned by model. Recompute embeddings when you change models or update content. FaceVision computes face embeddings in the browser and stores only those vectors, never images, which shows embeddings can also support privacy-conscious designs.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail