Module 2 — Embeddings: Turning Meaning into Numbers

This is the most important idea in the entire course. If you only deeply understand one module, make it this one. The database is just plumbing; embeddings are the water.

The big idea (explained simply)

An embedding is a list of numbers that represents the meaning of something. A model reads your text (or image, or audio) and outputs a fixed-length list of numbers — say 768 of them. The trick that makes it useful:

Things that mean similar things get similar lists of numbers. Things that mean different things get different lists of numbers.

That’s it. An embedding is a “meaning fingerprint.” Just like a fingerprint identifies a person, an embedding identifies a meaning — and unlike fingerprints, similar meanings have similar fingerprints.

A concrete tiny example

Suppose we had a magical 3-number embedding model. It might produce:

"king"   → [0.99, 0.02, 0.85]
"queen"  → [0.97, 0.95, 0.83]
"man"    → [0.95, 0.05, 0.10]
"woman"  → [0.93, 0.97, 0.12]
"banana" → [0.10, 0.40, 0.05]

Notice: king/queen/man/woman all share a high first number (maybe “human-royalty-ish”), and the second number tracks something like gender. “Banana” is off on its own. The model wasn’t told these rules — it learned them from reading billions of sentences. Real embeddings have hundreds of dimensions and no single dimension means something this clean, but the intuition holds: dimensions capture features of meaning.

The famous party trick

In good word embeddings, you can literally do arithmetic on meaning:

king - man + woman ≈ queen
Paris - France + Italy ≈ Rome

You subtract the “man” direction and add the “woman” direction, and you land near “queen.” This blew people’s minds in 2013 (the word2vec paper) and is the clearest proof that the numbers encode real semantic structure.

How are embeddings actually made?

A neural network is trained on a huge amount of data with an objective that forces it to put similar things together. The most common training idea is contrastive learning:

• Show the model pairs that should be similar (a question and its correct answer; two sentences that paraphrase each other; an image and its caption).
• Show it pairs that should not be similar (random unrelated pairs).
• Adjust the network so similar pairs land close and dissimilar pairs land far apart.

Do this millions of times and the model learns a “meaning space” where distance = dissimilarity. You don’t train this yourself — you use a pre-trained model. You just feed in content and collect the vectors.

Types of embeddings you’ll meet

• Word embeddings (word2vec, GloVe) — one vector per word. Historic, mostly superseded, but great for intuition.
• Sentence / passage embeddings (Sentence-BERT, E5, BGE, GTE, OpenAI text-embedding-3, Cohere embed, Voyage) — one vector for a whole sentence or paragraph. This is what you’ll use 95% of the time for search and RAG.
• Image embeddings (CLIP, SigLIP, DINOv2) — one vector per image. CLIP famously puts images and text in the same space, so you can search images with text (“a dog on a skateboard”).
• Multimodal embeddings — text, image, audio in one shared space.
• Specialized embeddings — for code (CodeBERT), for proteins, for recommendation (user/item vectors).

The properties that matter when you pick a model

1. Dimensionality — how many numbers per vector (e.g., 384, 768, 1024, 1536, 3072). More dimensions can capture more nuance but cost more memory and compute, and beyond a point add little. 384–1024 is a sweet spot for many apps.
2. Max sequence length — how much text fits in one embedding (e.g., 512 tokens vs 8192 tokens). Too-long text gets truncated (silently!) — a classic bug.
3. Domain fit — a model trained on web text may be mediocre on legal, medical, or code. Match the model to your data.
4. Symmetric vs asymmetric — some models are tuned so that queries and documents are encoded slightly differently (asymmetric, good for search: short question → long passage). Others treat both sides the same (symmetric, good for “find duplicate sentences”). Many models want you to add a prefix like "query: " or "passage: " — forgetting this quietly wrecks quality.
5. Normalization — many models output vectors meant to be L2-normalized (scaled to length 1) so that cosine similarity and dot product behave nicely. Check the model card.
6. Open vs API — run an open model yourself (BGE, E5, GTE, Nomic, Jina) for control and privacy, or call an API (OpenAI, Cohere, Voyage) for convenience and often top quality.
7. Cost & latency — embedding millions of documents has a real dollar and time cost. Batch your requests.

Industry tip: Check the MTEB leaderboard (Massive Text Embedding Benchmark) to compare models on retrieval quality — but always validate on your own data (Module 11). A model that’s #1 on a benchmark can be mediocre on your specific documents.

The single most important practical rule

You must use the same embedding model for your stored documents and for your queries.

Embeddings from different models live in different, incompatible number-spaces. Searching a database built with model A using a query embedded by model B is like measuring distance between a point in Paris and a point on the Moon — the numbers are meaningless together. If you ever change embedding models, you must re-embed your entire dataset (a “re-index”). Plan for this.

Chunking: the unsung hero of good retrieval

You rarely embed a whole 50-page document as one vector — it would blur all its topics into mush, and most models can’t fit that much text anyway. Instead you chunk: split the document into smaller pieces (a paragraph, a few sentences, ~200–500 tokens) and embed each chunk separately.

Good chunking is half the battle in RAG:

• Too big → one vector mixes many topics; search becomes vague.
• Too small → chunks lose context; “It increased by 20%” is useless without knowing what “it” is.
• Overlap — let consecutive chunks share some text (e.g., 10–20% overlap) so an idea split across a boundary isn’t lost.
• Respect structure — split on paragraphs, headings, sentences, or code blocks rather than blindly every N characters. “Semantic chunking” splits where the topic actually shifts.
• Attach context — prepend the document title or section heading to each chunk so it stands on its own.

We’ll do real chunking in Module 8.

Hands-on: generate your first embeddings

# pip install sentence-transformers
from sentence_transformers import SentenceTransformer

# A small, fast, high-quality open model (384 dimensions)
model = SentenceTransformer("BAAI/bge-small-en-v1.5")

sentences = [
    "The cat sat on the mat.",
    "A kitten rested on the rug.",      # similar meaning to #1
    "Quarterly revenue grew 12 percent.", # different topic
]

embeddings = model.encode(sentences, normalize_embeddings=True)
print(embeddings.shape)   # (3, 384)

# Cosine similarity = dot product when vectors are normalized
import numpy as np
def cos(a, b): return float(np.dot(a, b))

print("cat vs kitten:", round(cos(embeddings[0], embeddings[1]), 3))  # high, ~0.8+
print("cat vs revenue:", round(cos(embeddings[0], embeddings[2]), 3)) # low, ~0.1-0.3

You just turned meaning into numbers and measured it. The cat/kitten pair scores high; the cat/revenue pair scores low. The database in later modules just does this comparison at massive scale.

Gotchas

• Silent truncation. If your text exceeds the model’s max length, the extra is dropped without warning. Long docs lose their endings. Chunk first.
• Forgotten prefixes. Models like E5/BGE expect "query: " / "passage: " (or instruction prompts). Skipping these can drop quality 10–30%.
• Mixing models. Querying with a different model than you indexed with → nonsense results. Pin the model + version.
• Forgetting to normalize when your similarity metric assumes it. Cosine on un-normalized vectors, or dot product when you meant cosine, gives subtly wrong rankings.
• Assuming embeddings are “objective.” They inherit the biases and blind spots of their training data. “Similar” reflects the model’s worldview.
• Embeddings drift with model versions. text-embedding-3-large ≠ text-embedding-ada-002. Treat the model+version as part of your data schema.

Check yourself

1. What property makes embeddings useful for search?
2. Why must queries and documents use the same embedding model?
3. What problem does chunking solve, and what’s the risk of chunks that are too small?
4. You change from a 768-dim model to a 1024-dim model. What must you do to your existing database?

Answers: (1) Similar meanings → numerically similar vectors, so distance measures dissimilarity. (2) Different models produce incompatible number-spaces; cross-model distances are meaningless. (3) It keeps each vector focused on one topic and fits model length limits; too-small chunks lose surrounding context and become ambiguous. (4) Re-embed (re-index) the entire dataset with the new model; old vectors are incompatible.

➡️ Next: 03-vectors-and-similarity-metrics.md — the gentle math of comparing vectors.