Module 10 — Production and Scaling

Anyone can run a vector search in a notebook. Running it for millions of users, reliably, fast, and affordably is the job. This module is the operations and systems-design layer that turns you from “can use a vector database” into “can own one in production.”

The four forces you’re always balancing

Every production decision trades among these. You can’t max all four; you pick targets and engineer to them.

        RECALL (accuracy)
              ▲
              │
LATENCY ◄─────┼─────► COST ($ / RAM / hardware)
              │
              ▼
        THROUGHPUT (queries & inserts per second)

Want higher recall? Raise efSearch/nprobe → higher latency, lower throughput.
Want lower cost? Compress (Module 6) → some recall loss; or use disk indexes → some latency cost.
Want lower latency? More RAM/replicas → higher cost.

Expertise is choosing deliberately where to sit, with numbers, not vibes.

Latency: what “fast” really means

Don’t measure averages — measure percentiles:

p50 (median): typical experience.
p95 / p99: the slow tail — the 1-in-20 or 1-in-100 request. This is what users actually complain about, and what dominates when one user makes many calls.

A system with great p50 and terrible p99 feels broken. Set a target like “p99 < 100ms for retrieval” and engineer to it. Sources of tail latency: cold caches, garbage collection, a query that probes many cells, large filtered scans, network hops, an overloaded shard.

Levers to reduce latency: keep hot vectors in RAM, tune the index dials down to the minimum recall you need, cache frequent queries and their embeddings, reduce k, co-locate services, use faster distance math (SIMD/quantized), and replicate to spread load.

Throughput and scaling out: sharding and replication

Two independent scaling axes — know the difference cold:

Sharding (partitioning) — for size and write/throughput

Split the dataset across machines; each shard holds a slice of the vectors. A query fans out to all shards (or a routed subset), each searches its slice, results are merged. This lets you hold more vectors than one machine’s RAM and parallelize the work.

Random/hash sharding: even load, but every query hits every shard (fan-out cost).
Semantic/routed sharding: group similar vectors (e.g., by cluster or tenant) so a query can skip irrelevant shards — less fan-out, but risk of hot shards and boundary misses.

Replication — for reliability and read throughput

Keep copies of each shard on multiple machines. If one dies, a replica serves. Replicas also share read load (more queries/second). Replication doesn’t increase capacity; it increases resilience and read QPS.

Rule of thumb: shard to fit the data and handle writes; replicate to survive failures and serve more reads. Real clusters do both: N shards × R replicas.

Separation of storage and compute

Modern systems (Milvus, Turbopuffer, LanceDB, many managed services) keep vectors in cheap object storage (S3) and spin stateless compute nodes up/down to search them. This decouples “how much data” from “how much query power,” making scaling and cost elastic. It’s a major architectural trend — understand it.

Memory hierarchy and cost (the real driver)

RAM is the dominant cost in vector search. The whole field’s cost-engineering is about not keeping everything in expensive RAM:

Fastest, $$$$  ──►  GPU memory   (huge throughput, costly)
                    RAM          (HNSW lives here; the default)
                    SSD/NVMe     (DiskANN serves billions cheaply)
Slowest, $     ──►  Object store (S3; cold tiers, separation of compute)

Strategies to cut cost: quantization (Module 6) to fit more per GB; disk-based indexes (DiskANN) to serve from SSD; tiered storage (hot data in RAM, cold on disk/S3); scale-to-zero serverless for spiky workloads. A 32× compression isn’t an academic trophy — it’s a 32× smaller RAM bill.

Data lifecycle: inserts, updates, deletes, re-indexing

Static datasets are easy; live ones are where it gets real.

Inserts: HNSW supports incremental adds; IVF may need periodic retraining of centroids as data grows/shifts.
Updates: usually delete + re-insert (re-embed if content changed).
Deletes: often soft deletes (tombstones) because removing a node from an HNSW graph cleanly is hard. Tombstones accumulate, wasting space and slowly hurting recall.
Compaction / re-indexing: periodically rebuild indexes to reclaim space, remove tombstones, and refresh clusters. Plan for it — schedule it, do it on replicas, swap in.
Distribution drift: if your data’s “shape” changes over time, IVF clusters and PQ codebooks go stale; recall silently decays. Re-train/re-index on a cadence.
Re-embedding migrations: the big one. Changing embedding models means re-embedding everything (Module 2). Build a pipeline that can re-embed and reindex a whole corpus without downtime (dual-write to old + new, then cut over).

Consistency, durability, and freshness

Durability: vectors must survive crashes — write-ahead logs, persistence to disk/object store, backups. A library (FAISS) gives you none of this; a database does.
Consistency: after you insert a vector, when does it become searchable? “Eventual” (a delay) vs “strong” (immediately). RAG over fast-changing data cares about freshness; pick a system whose guarantees match.
Backups & disaster recovery: snapshot indexes + source data so you can rebuild. Remember you can always re-embed from source text if you keep it — keep the source text.

Observability: you can’t run what you can’t see

Monitor at minimum:

Latency p50/p95/p99 for search and insert.
Throughput (QPS, insert rate) and error rates.
Recall — sample queries against ground truth periodically; recall drifts as data changes and nobody notices until users do.
Resource use: RAM (the big one), CPU/GPU, disk, index size growth, tombstone count.
Query patterns: filter selectivity, k distribution, slow-query log.
Cost per query / per million vectors.

Set alerts on recall drops and p99 latency, not just CPU. A vector system can be “healthy” by infra metrics while quietly returning worse results.

Security and compliance (often overlooked, sometimes fatal)

Access control / multi-tenancy: enforce tenant and permission filters centrally (Module 9); a dropped filter is a breach.
Encryption at rest and in transit.
PII & embeddings: embeddings can leak information about their source text (partial reconstruction is possible). Treat vectors of sensitive data as sensitive data. Consider where embeddings are generated/stored for compliance (data residency).
Deletion / right-to-be-forgotten: ensure deleting source data also purges its vectors (and tombstones eventually compact away).
Auditing: log who queried/changed what.

A reference production architecture

            ┌──────────── Ingestion pipeline (offline) ───────────┐
Sources ──► │ load → chunk → embed (batch) → upsert to vector DB  │
            └──────────────────────────────────────────────────────┘
                                   │
                          ┌────────▼─────────┐  N shards × R replicas
   User ──► API/Gateway ─►│   Vector DB       │  HNSW/IVF-PQ + filters
        (auth, tenant)    │  (hot: RAM/SSD,    │  + metadata + BM25
                          │   cold: object store)
                          └────────┬─────────┘
                                   │ top-k candidates
                   ┌───────────────▼───────────────┐
                   │ rerank → assemble context → LLM│ (for RAG)
                   └───────────────┬───────────────┘
                                   ▼
                          answer + citations
        + Observability (latency, recall, cost) and periodic re-index/compaction

Capacity planning, quickly

Estimate before you build:

RAM for vectors ≈ N × dim × bytes (4 for float32, 1 for int8, ~0.1–0.25 for PQ) × index overhead (HNSW adds ~1.5–2×). Example: 50M × 768 × 4 × 1.7 ≈ ~260 GB float32 HNSW → fits on a big box, or ~16× less with PQ.
Shards = total RAM needed ÷ RAM per machine (with headroom).
Replicas = for HA (≥2) and to meet read QPS.
Headroom for growth, compaction, and spikes (don’t run at 90% RAM).

Gotchas

Tuning for p50, suffering at p99. Users feel the tail. Optimize and alert on p99.
Forgetting index overhead in capacity math. HNSW graphs can nearly double vector RAM. Size for it.
Never compacting. Tombstones and stale clusters silently rot recall and waste RAM.
No recall monitoring. Infra dashboards look green while result quality decays. Track recall on a sample.
Ignoring re-embedding cost. Model upgrades mean re-indexing everything; if you didn’t keep source text, you’re stuck. Keep the source.
Single point of failure. A library on one node with no replica = data loss on crash. Use replication for anything that matters.
Running at the RAM cliff. Vector systems degrade hard when they spill out of RAM. Leave headroom.

Check yourself

What’s the difference between sharding and replication, and what does each buy you?
Why measure p99 latency, not just average?
Why is RAM the central cost concern, and name two ways to reduce it.
Why are deletes hard in HNSW, and what maintenance does that imply?
What must you do when upgrading embedding models, and how do you avoid downtime?

Answers: (1) Sharding splits data across machines for capacity and write/throughput (queries fan out); replication keeps copies for fault tolerance and more read QPS. Shard to fit/serve writes, replicate to survive failures and serve reads. (2) Averages hide the slow tail; the p99 request is what users complain about and dominates multi-call workflows. (3) Fast search wants vectors in expensive RAM, and vectors are large; reduce it via quantization/compression and disk-based or tiered storage. (4) Cleanly removing a node from the graph is hard, so deletes are soft (tombstones) that accumulate — implying periodic compaction/re-indexing to reclaim space and recall. (5) Re-embed the entire corpus with the new model and rebuild the index; avoid downtime by dual-writing to old + new indexes and cutting over once the new one is ready (and keep source text so you can re-embed at all).

➡️ Next: 11-evaluation-and-quality.md — measuring whether your system is actually good.