Sharding
Also called horizontal partitioning, database sharding.
Splitting one large dataset across several databases, called shards, by a key such as customer ID, so each shard holds only part of the data.
Last updated
What it is.
Each row belongs to one shard, chosen by a shard key. For example, customers 1 to 1,000 on shard A and the rest on shard B, or by a hash of the customer ID. The app, or a routing layer, sends each query to the right shard.
Sharding spreads both storage and write load, which a read replica cannot do. In exchange, queries across shards, joins and moving data between shards all get harder.
Why it matters
Sharding lets a database grow past what one machine can handle. It is also a big, lasting design choice. Most products never need it, and those that do want a shard key that matches how data is used, often the tenant in multi-tenant SaaS.
In a product build
We design data so that a natural shard key exists, such as a tenant ID on every table, even when we start on one database. That keeps sharding possible later without rewriting the data model. Before sharding we use indexes, caching, replicas and a bigger primary first.
Common mistakes.
- 01
Sharding too early, adding complexity years before it is needed.
- 02
Picking a shard key that most queries do not use, so they hit every shard.
- 03
Hot shards, where one huge customer or date range takes most of the load.
Related terms.
- Read replicaA read-only copy of a database that is kept in sync with the main database and takes read queries off it.
- Multi-tenancyOne running copy of a software product serving many customers, called tenants, with each tenant's data kept apart.
- Connection poolingKeeping a set of open database connections and sharing them across requests, instead of opening a new connection for each one.
- Data warehouseA database built for analysis and reporting, where cleaned data from many systems is kept together in one place.
- MicroservicesA way of building an application as many small services, each owning one job and its own data, deployed independently.
