Choosing a Vector Database

What a vector database does

An embedding turns text (or images, or audio) into a vector of numbers that captures its meaning. A vector database stores those vectors and answers one central question very quickly:

That's the engine behind semantic search, RAG, recommendations and duplicate detection. Around that core, a good vector database adds the things any database needs: storing the original text and metadata, updates and deletes, filtering, persistence, backups and scaling.

Do you even need one?

Not always. For a few thousand to a few hundred thousand vectors, the simplest approach — keep them in a NumPy array and compare the query against every vector — is fast, exact and has zero infrastructure. Our RAG tutorial does exactly that.

Comparing against every vector is called exact or brute-force search. Its cost grows linearly with your data. Let's do the math for one million vectors from a 1,536-dimension model:

1,000,000 vectors × 1,536 dimensions × 4 bytes (float32)
  ≈ 6.1 GB of raw vector data
  and ~1.5 billion multiply-adds for every single query

At that scale — or when you need concurrent users, frequent updates, filtering and durability — you want proper indexing. That's what vector databases provide.

How they search fast: approximate nearest neighbors

The key trick is to give up a tiny bit of accuracy for a huge gain in speed. Approximate nearest neighbor (ANN) indexes find almost always the true closest vectors, while examining only a small fraction of the data. Accuracy is measured as recall: the share of the true top results the index actually returned. Well-tuned indexes commonly reach recall in the high 90s (%).

HNSW (Hierarchical Navigable Small World)

The most popular index today. It builds a multi-layer graph in which each vector is linked to its near neighbors. A search starts at a sparse top layer, hops greedily toward the query, and descends to denser layers to refine — a bit like zooming in on a map from country to city to street.

• Strengths: excellent speed and recall; handles inserts well.
• Weaknesses: memory-hungry (the graph lives alongside the vectors); slower to build.
• Knobs: M (links per node) and ef (how widely to search) trade memory and speed for recall.

IVF (Inverted File Index)

Clusters the vectors into many groups (using k-means). At query time, it finds the clusters closest to the query and searches only inside those.

• Strengths: lower memory, fast to build.
• Weaknesses: recall depends on how many clusters you probe (nprobe); clusters can drift as data changes.

Quantization

Compresses vectors to save memory and speed up comparisons — for example storing 8-bit integers instead of 32-bit floats (scalar quantization), single bits (binary quantization), or compact codes (product quantization, PQ). Compression costs some accuracy, so systems often search compressed vectors first and then re-score the top candidates with the full-precision vectors.

Disk-based indexes

For datasets too large for RAM, approaches such as DiskANN keep most of the index on SSD with only a compact structure in memory, cutting cost substantially at some latency.

Features that matter beyond search

• Metadata filtering — "find similar chunks, but only from documents this user may access, published after 2024". How well filtering combines with ANN search varies a lot between products and is worth testing.
• Hybrid search — combining vector similarity with keyword (BM25) search, so exact terms like product codes aren't missed.
• Updates and deletes — how quickly new data becomes searchable, and how deletes affect index quality.
• Multi-tenancy — isolating many customers' data efficiently.
• Operations — backups, replication, monitoring, scaling, security and compliance.

The main options

This market moves quickly, and features and pricing change often — check each project's current documentation before deciding.

Example: semantic search with pgvector

If you already run PostgreSQL, you can add vector search with a few SQL statements:

-- 1. Enable the extension
CREATE EXTENSION IF NOT EXISTS vector;

-- 2. Store text, metadata and a 384-dimension embedding
CREATE TABLE chunks (
  id        bigserial PRIMARY KEY,
  source    text,
  content   text,
  embedding vector(384)
);

-- 3. Add an HNSW index for fast cosine-distance search
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- 4. Find the 5 chunks closest to a query embedding ($1)
SELECT source, content, 1 - (embedding <=> $1) AS similarity
FROM chunks
WHERE source LIKE 'handbook%'          -- ordinary SQL filters work too
ORDER BY embedding <=> $1
LIMIT 5;

The <=> operator is pgvector's cosine distance (lower is closer), which is why the query orders by it ascending and computes similarity as 1 − distance. Your application computes the query embedding with the same model used for the stored chunks and passes it in as $1.

How to choose

1. Start with what you have. Already on Postgres, Elasticsearch or MongoDB? Their vector features are often enough, and one less system to run is a real advantage.
2. Estimate your scale. Count vectors × dimensions × 4 bytes. Under a few million vectors, almost any option works; beyond hundreds of millions, specialized systems earn their keep.
3. List your must-have features: filtering, hybrid search, multi-tenancy, update frequency, compliance.
4. Decide managed vs. self-hosted based on your team's operations capacity and data-residency needs.
5. Benchmark on your own data. Measure recall, latency and cost with your real queries and filters — published benchmarks rarely match your workload.
6. Keep it swappable. Hide the database behind a small retrieval function in your code, so you can change it later without rewriting the app.

Key takeaways

• Vector databases store embeddings and quickly find the nearest ones to a query.
• Small datasets don't need one — brute-force search in memory works fine.
• At scale, approximate indexes like HNSW and IVF trade a little recall for big speed gains.
• Quantization saves memory; re-scoring recovers accuracy.
• Filtering, hybrid search and operations often matter more than raw speed.
• Start with infrastructure you already have, and benchmark on your own data.