Partitioning and Sharding
Scaling a database
On large tables, usually an index is used to speed up a read query. Sometimes, a full index scan can also be slow as the index grows. This will increase the read latency. There are multiple ways to tackle this before we get to sharding.
We are trying to solve read latency. Writes are still alright, since we are writing to the end of the table but still rebalancing the index might be slow.
Usually, using LSM tree for write heavy database is more efficient. LSM trees are optimised for writes.
- Horizontal partitioning (single server)
- We split the single table into multiple parts called partitions.
- This in result in multiple indexes for these partitions. Since these partitions are smaller, the index is smaller, where the full index scan is much faster than the entire table’s index.
- We have smaller tables, client can let us know which partition to hit, smaller index → fast queries.
- Adding read replicas
- This is required if a single database node cannot handle so many connections made to it.
- In order to distribute the connections, we use read replication, where connections can be distributed/load balanced.
- Read replica will take on read requests only and sync with master server which accepts writes and critical reads (critical reads are those which require strong consistency.
- Master server will send updates to the replicas or backup servers.
- Multi master replication
- Database cluster contains multiple nodes with each node containing 100% of the data.
- Multiple masters accept writes and conflicts are resolved.
- This is commonly used, by deploying masters on different regions to cater to lower latency
- Sharding
- It is similar to horizontal partitioning but each partition lives in a different database node and not on the same server.
- Each shard (server) owns some amount of data. Data is partitioned across the shards. Each partition is mutually exclusive (sub-sets of complete data).