What are embeddings in NLP? How they look, vs tokens and one-hot, and where to store them

In NLP, an embedding is a list of numbers, a vector, that represents a word, sentence or document so that items with similar meanings have similar vectors. Embeddings power semantic search, RAG, clustering and classification. RAG.NextUpgrad combines dense embedding retrieval with BM25, so this page explains what embeddings are, why they matter, what they look like, how they differ from tokens and one-hot encoding, where to store them, and how subword n-gram models relate to transformers.

What are embeddings in NLP?

Embeddings are dense numerical vectors that represent text. A model maps each word, sentence or document to a point in a high-dimensional space, typically hundreds to a few thousand dimensions, where distance reflects similarity in meaning.

Also asked as: what is embeddings in nlp

Early embeddings, such as word2vec and GloVe, gave each word one fixed vector learned from co-occurrence patterns, famously capturing relations like king minus man plus woman landing near queen. Modern contextual embeddings from transformers give the same word different vectors depending on context, so bank in river bank differs from bank account. Sentence embedding models, such as those in Sentence Transformers, produce one vector per sentence or passage, which is what semantic search and RAG use.

Why are embeddings important in NLP models?

Neural networks need numbers, and embeddings are the numbers that carry meaning. They let models generalise between similar words, compare texts by meaning rather than exact words, and reuse knowledge learned from huge datasets in smaller tasks.

Inside language models, the first layer converts tokens into embeddings, and every later layer refines these vectors. Outside the model, sentence embeddings enable semantic search, retrieval for RAG, clustering, recommendation, deduplication and classification with simple models on top. Their quality directly decides retrieval quality, which is why choosing and evaluating an embedding model matters so much in RAG systems. Good embeddings also handle synonyms and paraphrases that keyword search misses.

What do embeddings look like?

An embedding is just a long list of decimal numbers, such as [0.021, -0.113, 0.087, ...], often 384, 768 or 1,536 values long. Individual numbers have no readable meaning; the information lies in the pattern across all dimensions.

To see structure, reduce dimensions with PCA, t-SNE or UMAP and plot the points: sentences about sport cluster together, away from sentences about finance. Many embedding models output normalised vectors with length 1, so cosine similarity equals the dot product. In storage, a 768-dimensional float32 embedding takes about 3 KB, which adds up quickly for millions of chunks, so teams often use float16 or quantized vectors to save space.

Embeddings vs tokens: what is the difference?

Tokens are the pieces text is split into, such as words or subwords, each mapped to an integer ID. Embeddings are the vectors those IDs are converted into, or vectors representing whole sentences, which carry meaning numerically.

The pipeline is: text, then tokenizer, then token IDs, then embedding lookup, then model layers. Token counts determine model limits and API usage; embeddings determine how meaning is represented and compared. A tokenizer is a fixed vocabulary and rules, while embeddings are learned parameters. When people say they are embedding documents for RAG, they mean running an embedding model to turn each chunk into a single vector, which involves tokenising it first.

Embeddings vs one-hot encoding: what is the difference?

One-hot encoding represents each word as a vector with one 1 and zeros elsewhere, as long as the vocabulary. It is sparse, huge and treats every pair of words as equally different. Embeddings are short, dense and place similar words close together.

Also asked as: embeddings vs one hot encoding

With a 50,000-word vocabulary, one-hot vectors have 50,000 dimensions, and cat is as far from kitten as from carburettor. A 300-dimensional embedding is compact and captures similarity learned from data. In practice, an embedding layer is equivalent to multiplying a one-hot vector by a learned matrix, implemented as a fast lookup. One-hot encoding remains useful for small categorical features, such as colour or country, in traditional machine learning.

Where should you store embeddings for similarity search?

Store them in a system with an approximate nearest neighbour index: a vector database such as Qdrant, Weaviate, Milvus or Pinecone, a search engine such as Elasticsearch or OpenSearch, or your existing database with vector support, such as PostgreSQL with pgvector.

Also asked as: where to store embeddings for similarity search

For prototypes, a FAISS index or a NumPy file on disk works for up to hundreds of thousands of vectors. For production, choose based on scale, filtering needs and operations: keep vectors next to relational data with pgvector, or use a dedicated vector database for large, filter-heavy workloads. Always store the source text, document ID and metadata with each vector, and record the embedding model version, since vectors from different models are incompatible.

Can you store embeddings in PostgreSQL?

Yes. The pgvector extension adds a vector column type, distance operators for cosine, Euclidean and inner product, and HNSW and IVFFlat indexes for fast approximate search. It is available on most managed PostgreSQL services.

Also asked as: can we store embeddings in postgresql

Example: CREATE EXTENSION vector;, then CREATE TABLE chunks (id bigserial, content text, embedding vector(768));, an HNSW index with vector_cosine_ops, and queries ordered by embedding <=> query_vector with a limit. You get SQL filters, joins, transactions and backups for free, and PostgreSQL full-text search enables hybrid retrieval. Tune hnsw.ef_search to trade speed for recall. pgvector handles millions of vectors well on adequate hardware; for very large or filter-heavy workloads, compare with a dedicated vector database.

Can you store embeddings in MongoDB?

Yes. You can store embeddings as arrays of numbers in documents, and MongoDB Atlas Vector Search adds approximate nearest neighbour indexes and a $vectorSearch aggregation stage for similarity queries with filters.

Also asked as: can we store embeddings in mongodb

This suits teams already using MongoDB: embeddings live in the same documents as the content and metadata, so there is no separate system to sync. Define a vector search index specifying the field, dimensions and similarity metric, then query with $vectorSearch, passing the query vector, number of candidates and limit. Atlas also supports hybrid search by combining vector and full-text results. For self-managed MongoDB, check your version's vector search availability, since features have expanded over time.

Can SQLite store embeddings?

Yes. You can store embeddings as BLOBs or JSON arrays in SQLite, and extensions such as sqlite-vec add vector columns and similarity search, which suits local, mobile and small applications.

Without an extension, store vectors as binary blobs and compute similarity in Python with NumPy, which is fine for tens of thousands of vectors. sqlite-vec provides fast brute-force search with SQL syntax, and SQLite's FTS5 full-text search allows hybrid retrieval in a single file. This makes SQLite attractive for desktop apps, offline RAG, edge devices and tests. For large collections or many concurrent users, move to pgvector or a dedicated vector database.

Is there a pretrained transformer that uses subword n-gram embeddings like fastText?

Not in the same way. fastText builds word vectors from hashed character n-grams, so it can embed unseen words. Most transformers instead use learned subword tokenizers, such as BPE, WordPiece or SentencePiece, which also handle rare words by splitting them into pieces.

Also asked as: is there a pretrained (nlp) transformer that uses subword n-gram embeddings for tokenization like fasttext

Character and byte-level transformers come closest: CharacterBERT builds word representations from characters, CANINE works directly on characters with hashed embeddings, and ByT5 operates on raw bytes, removing the fixed vocabulary entirely. These models are robust to misspellings and rare words, like fastText, but are slower due to longer sequences. If you need fastText's robustness inside a standard pipeline, combining fastText features with a transformer, or using a byte-level model, are practical options.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail