Vector Database Cluster

A vector database cluster is a group of nodes that together store an embedding index and serve similarity queries against it. Work is divided two ways, and they solve different problems.

Sharding

splits the data across the vector database cluster. Each shard holds a slice of your vectors, so a collection too large for one machine’s memory spreads across several. This is how you scale capacity — the classic database sharding pattern, applied to embeddings.

Replication

copies the data. Each shard has one or more replicas holding identical vectors, so a node failure doesn’t lose anything and read traffic can spread across copies. This is how you scale availability and throughput.

Query cost scales with shard count.

More shards means more parallel searches per query and more merging work. Beyond a certain point, adding shards makes queries slower rather than faster, because coordination overhead exceeds the parallelism gain.

Recall behaves differently.

Each shard runs its own graph index — usually HNSW, from the foundational paper — over a fraction of the data. Ten shards each returning their local top-10 gives the coordinator 100 candidates to merge, which is generally fine. But shard-local approximation errors compound differently than single-index errors, so measure recall on the clustered configuration, not on your single-node benchmark. Our nearest neighbor search guide covers why recall degrades silently without ever throwing an error.

Tail latency is set by your slowest shard.

Since the coordinator waits for all shards, one slow node — mid-compaction, mid-merge, or noisy neighbor — sets p99 for every query. In a vector database cluster, p99 is a property of the worst node, not the average one.

Start with raw vector size.

Dimensions × 4 bytes for float32. A 1,024-dimension embedding is about 4KB, so ten million vectors is roughly 40GB before anything else.

Add index overhead.

HNSW graph structures typically add 30–100% on top, depending on your M parameter. That 40GB becomes 55–80GB in practice, and this is the line most capacity plans forget.

Add metadata and payload.

Filterable attributes, IDs, and stored fields all consume memory or disk.

Multiply by replicas.

Two replicas doubles it. Three triples it.

Then leave headroom.

Indexes grow, catalogs expand, and merge operations need working space. Running above roughly 70–80% of available memory is where clusters start behaving unpredictably.

Replica count

determines failure tolerance. One replica per shard survives a single node loss. Two survives two. Most production clusters run two or three, balancing cost against risk.

Read distribution

is the underrated benefit — replicas serve queries in parallel, so replication scales throughput as well as availability. If your bottleneck is query volume rather than data size, adding replicas often helps more than adding shards.

Consistency during writes

deserves attention. When a vector is inserted, replicas must converge. Some systems replicate synchronously (slower writes, consistent reads), others asynchronously (faster writes, brief inconsistency). For ecommerce, the practical question is how long after a price or stock change the update is visible on every replica — which connects directly to the freshness concerns in our real-time indexing guide.

Rebalancing

happens when nodes join or leave, and it’s the operation most likely to cause trouble. Moving vector shards means moving large amounts of data and rebuilding index structures, which competes with query traffic for resources. Ask any platform how rebalancing behaves under load before you need to find out.

Pinecone

abstracts the cluster away — you choose a scale tier and the service handles sharding, replication, and rebalancing. Least operational burden and least control, which is the right trade for most teams.

Qdrant

offers explicit distributed mode with configurable shards and replicas via Raft consensus. Predictable and well documented, and you own the topology decisions.

Weaviate

supports horizontal scaling with sharding and replication, plus multi-tenancy that isolates tenants into separate shards — genuinely useful if you serve many customers from one deployment.

Milvus

is the most explicitly distributed of the group, separating compute and storage into distinct node types (query, data, index, coordinator). Considerable operational complexity in exchange for genuine billion-scale capability.

Elasticsearch / OpenSearch

apply their existing shard-and-replica model to dense-vector fields, which is a real advantage if you already run and understand an Elastic cluster.

pgvector

doesn’t cluster natively for vector workloads — you’d scale through Postgres replication or an extension like Citus, which works but wasn’t designed for this. Below a few million vectors it rarely matters; above that it becomes the reason teams migrate.

Per-node latency, not just cluster average.

Since the slowest shard sets your p99, cluster-wide averages actively hide the problem. Alert on individual node latency — and tie the numbers back to the quality metrics in our search relevance metrics guide, since infrastructure health and result quality are measured separately and both matter.

Memory headroom per node.

Vector clusters degrade sharply rather than gracefully when memory fills. Alert well before capacity, because there’s no gentle slope.

Recall against ground truth.

Run periodic brute-force comparisons on a sample of real queries. Recall drifts as data grows and index parameters stay fixed, and nothing in your monitoring will tell you — results just quietly get worse. This is the single most under-instrumented metric in vector operations.

Indexing lag distribution.

Median and p99 time from write to searchable, especially during bulk imports. Tail lag is where correctness problems hide.

Rebalancing events.

Log them, and correlate with latency spikes. Most mysterious performance incidents in a vector database cluster trace back to a rebalance nobody noticed.

Under roughly a million vectors,

a single well-provisioned node handles the workload comfortably. Clustering adds operational complexity and failure modes for no capacity benefit.

If your constraint is throughput rather than data size,

replicas alone may solve it without sharding at all — and replicas are far simpler to operate.

If quantization would fit your data on one node,

do that first. Cutting memory 4× is usually cheaper and simpler than distributing across machines, and it’s reversible.

If search isn’t your differentiator,

a managed platform absorbs all of this. That’s the honest calculus for most ecommerce teams: cluster topology, rebalancing behavior, and recall monitoring are specialized, permanent work that rarely differentiates a retail business. bCloud AI’s AI search engine handles the distributed layer entirely — sub-200ms at high SKU counts with real-time indexing — so the engineering question becomes what to build with search rather than how to keep search running. Our AI search for large catalogs guide covers the scale trade-offs, and the top semantic search solutions for e-commerce roundup compares managed options.

What is a vector database cluster?

A vector database cluster is a group of nodes that together store an embedding index and serve similarity queries. Sharding splits vectors across nodes to scale capacity beyond one machine’s memory; replication copies each shard to provide fault tolerance and additional read throughput.

How is vector sharding different from normal database sharding?

Traditional sharding routes a query to one shard using a key. Vector search has no key — nearest neighbors could be on any shard — so every query fans out to all shards, each searches its local index, and results are merged. Query cost therefore scales with shard count, and tail latency is set by the slowest node.

How do I size a vector database cluster?

Calculate dimensions × 4 bytes per vector, add 30–100% for HNSW index overhead, add metadata, multiply by replica count, then leave 20–30% headroom. Ten million 1,024-dimension vectors with two replicas realistically needs 180–200GB of provisioned memory before quantization.

How many shards and replicas should I use?

Start with the fewest shards that fit your data in memory with headroom, since extra shards add coordination overhead. Use two or three replicas for production availability. Add replicas for throughput problems and shards for capacity problems — they solve different things.

Does clustering affect search quality?

It can. Each shard runs its own approximate index over a fraction of the data, so recall behaves differently than on a single index. Always measure recall against brute-force ground truth on the clustered configuration rather than assuming single-node benchmarks transfer.

When do I actually need a cluster?

Generally above a few million vectors, or when you need high availability, or when query throughput exceeds one node’s capacity. Below roughly a million vectors a single well-provisioned node is usually sufficient — and quantization often defers clustering entirely by cutting memory several-fold.