A dense real-valued vector that represents a token, image patch, item, or sentence. Semantic similarity maps to geometric proximity.

Data Flow

token id          embedding layer           dense vector
   318          ------------->       [-0.21, 0.07, 0.92, ... , -0.15]   (d=768)

sentence / chunk  sentence-encoder         vector
"Unhappiness"   ------------->     [0.11, -0.33, 0.45, ...]
similar to        cosine = 0.86
"Depression"

What it is

A learned function f(item) -> R^d where related items land close and unrelated items land far.

Three types

• Token embeddings: every token has a vector; trained with the LM.
• Sentence embeddings (e.g. Sentence-BERT): one vector per chunk for retrieval.
• Multi-modal embeddings: image, audio, text in the same space (CLIP).

Code (sentence-transformer)

from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer("all-MiniLM-L6-v2")
vecs = model.encode(["cat sat on mat", "dog lay on rug"])
similarity = (vecs[0] @ vecs[1]) / (
    (vecs[0]**2).sum()**0.5 * (vecs[1]**2).sum()**0.5
)

Pitfalls

• Mixing embedding models in the same vector store: embedding drift.
• Embeddings leak information — don’t query on user PII unless you really need to and have consent.

Analogy

A map where every concept is a city; related cities are close.

Interview tip: When asked “what’s an embedding?”, answer with three properties: dense, learned, semantic.