What does embedding mean in AI? Dimensions, nn.Embedding, graph and face embeddings

In everyday English, embedding means fixing something firmly inside something else. In AI and machine learning, an embedding is a learned vector, a list of numbers, that represents an item such as a word, sentence, image, user or face, arranged so that similar items have similar vectors. Embeddings drive search in RAG.NextUpgrad and identity matching in FaceVision. This page covers the meaning, dimensions and attention heads, PyTorch's nn.Embedding, graph and face embeddings, training your own, and a common LlamaIndex configuration issue.

What does embedding mean?

Generally, to embed is to set something firmly within a surrounding mass, like a stone embedded in concrete or a video embedded in a web page. In machine learning, an embedding is a representation of data as a point in a vector space, learned so that geometry reflects meaning.

The terms connect: an embedding places items inside a mathematical space where distances are meaningful. "Embedding" can refer to the process, as in "embedding the documents", or the resulting vectors, as in "store the embeddings". In mathematics, an embedding is a structure-preserving map from one space into another, which inspired the machine learning usage.

What is an embedding in AI?

In AI, an embedding is a dense numerical vector produced by a model to represent an input's meaning or features. Similar inputs map to nearby vectors, so comparing embeddings measures similarity.

Also asked as: what is embedding in ai

Uses include semantic search, recommendations, clustering, classification, deduplication and retrieval for RAG. Language models also use embeddings internally: each token is converted into a vector before processing. Embedding models are trained on large datasets, often with contrastive objectives that pull related pairs together and push unrelated ones apart. The same idea applies across text, images, audio, users, products and graphs.

What is embedding dimension?

Embedding dimension is the number of values in each embedding vector, such as 384, 768 or 1,536. It sets how much information each vector can hold and how much memory and computation it costs.

Larger dimensions can capture finer distinctions but cost more storage and slower search; smaller dimensions are cheaper. Many modern models support shortened embeddings with little quality loss through Matryoshka-style training. In vector databases, the index must match the model's dimension exactly. In transformers, the embedding dimension, often called d_model, is also the width of the hidden layers throughout the model.

How do attention heads relate to embedding dimension in transformers?

In standard multi-head attention, the embedding dimension is split evenly across heads: each head works with d_model divided by the number of heads. With d_model of 768 and 12 heads, each head uses 64 dimensions.

Also asked as: what's the relationship between number of heads and embedding dimension in transformers

Each head projects the input into its own query, key and value spaces of that smaller size, attends independently, and the outputs are concatenated back to d_model and mixed by a final projection. This is why d_model must be divisible by the number of heads. More heads let the model attend to different patterns in parallel, but each head gets fewer dimensions, so architects balance head count and head size.

What is nn.Embedding?

nn.Embedding is a PyTorch layer that stores a trainable table of vectors, one row per item in a vocabulary. Given integer indices, such as token IDs, it returns the corresponding rows as embedding vectors.

import torch, torch.nn as nn
emb = nn.Embedding(num_embeddings=10000, embedding_dim=256)
ids = torch.tensor([[12, 503, 7]])
vectors = emb(ids)        # shape: (1, 3, 256)

It is the first layer of most language models and recommendation systems that start from IDs. Options include padding_idx, whose vector stays zero and is not updated, and from_pretrained to load existing vectors, optionally frozen.

How does nn.Embedding work?

nn.Embedding is a matrix of shape vocabulary size by embedding dimension, initialised randomly. A forward pass simply selects rows by index, which is equivalent to multiplying a one-hot vector by the matrix but much faster.

During training, gradients flow only to the rows that were used in the batch, and the optimiser updates those vectors so they become useful for the task. Over time, items used in similar contexts develop similar vectors. Because it is a lookup, it cannot handle unseen IDs; tokenisers map unknown text to known subword IDs to avoid that problem.

What are graph embeddings?

Graph embeddings represent nodes, edges or whole graphs as vectors that capture their structure and relationships, so connected or structurally similar nodes get similar vectors. They let standard machine learning work on network data.

Methods include random-walk approaches such as DeepWalk and node2vec, graph neural networks such as GraphSAGE and graph attention networks, and knowledge graph embeddings such as TransE, which model relations as vector translations. Uses include friend and product recommendations, fraud detection, drug discovery and link prediction in knowledge graphs. GraphRAG systems combine knowledge graphs with text embeddings for retrieval.

What is a face embedding?

A face embedding is a vector, often 128 or 512 numbers, that a face recognition model produces from a face image so that images of the same person are close together and different people are far apart. Recognition compares embeddings rather than raw photos.

Also asked as: what is face embedding

FaceNet popularised this approach in 2015 with a triplet loss; later methods such as ArcFace improved accuracy. In FaceVision, detection and embedding run in the browser with ONNX Runtime Web, and only embeddings are stored through a FastAPI and PostgreSQL back end, never images. Face embeddings are still sensitive biometric data, so store them securely and get clear consent.

How do I train my own fine-tuned text embedding model?

Start from a pretrained embedding model and fine-tune it with Sentence Transformers on pairs of queries and relevant passages from your domain, using a contrastive loss such as MultipleNegativesRankingLoss, then evaluate retrieval on held-out questions.

Also asked as: how to train your own fine-tuned text embedding model

Steps:

• Collect query and positive passage pairs; real search logs are best.
• Generate synthetic questions for passages with a language model if data is scarce, then review samples.
• Add hard negatives, similar but wrong passages, to sharpen the model.
• Fine-tune for a few epochs and track recall at k.
• Compare with the base model on the same evaluation set.

Re-embed your whole corpus with the fine-tuned model before deploying.

Why does LlamaIndex still ask for an OpenAI key with a local Hugging Face embedding model?

Because LlamaIndex uses separate defaults for the embedding model and the language model. Setting a local embedding model changes only embeddings; the LLM used for generating answers still defaults to OpenAI, which needs a key.

Also asked as: why does llama-index still require an openai key when using hugging face local embedding model

Fix it by setting both in global settings:

from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
Settings.llm = None   # or a local LLM such as an Ollama model

Setting the LLM to None disables generation for retrieval-only use; otherwise configure a local model. Also check components such as response synthesisers that may create their own default LLM.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail