principlesqlMinor
What's the advantage of sharding in AWS Aurora?
Viewed 0 times
theadvantagewhatauroraawssharding
Problem
AWS' RDS docs on multi-master Aurora state the following:
In an Aurora multi-master cluster, each shard is managed by a specific DB instance, and a DB instance can be responsible for multiple shards.
Later in the same document, we read:
You can avoid resharding operations because all DB instances in a cluster can access all databases and tables through the shared storage volume.
So, a multi-master Aurora instance can be responsible for multiple shards. This is made possible in part because all DB instances access the same shared storage volume.
If multi-master Aurora instances can manage multiple shards, what's the advantage of using sharding at all? Why not just configure all instances to manage all shards? (Essentially obviating the need for sharding)
A theory
My suspicion is that not using sharding would lead to more deadlocks within Aurora's internals if different masters write to the same page at the same time. This, in turn, would lead to increased latency while Aurora retries the contentious write. Or, perhaps Aurora would generate an error and the application itself would have to retry the query. (I'm clearly not familiar enough with Aurora to know what happens, hence this question )
In an Aurora multi-master cluster, each shard is managed by a specific DB instance, and a DB instance can be responsible for multiple shards.
Later in the same document, we read:
You can avoid resharding operations because all DB instances in a cluster can access all databases and tables through the shared storage volume.
So, a multi-master Aurora instance can be responsible for multiple shards. This is made possible in part because all DB instances access the same shared storage volume.
If multi-master Aurora instances can manage multiple shards, what's the advantage of using sharding at all? Why not just configure all instances to manage all shards? (Essentially obviating the need for sharding)
A theory
My suspicion is that not using sharding would lead to more deadlocks within Aurora's internals if different masters write to the same page at the same time. This, in turn, would lead to increased latency while Aurora retries the contentious write. Or, perhaps Aurora would generate an error and the application itself would have to retry the query. (I'm clearly not familiar enough with Aurora to know what happens, hence this question )
Solution
What's the advantage of sharding in AWS Aurora?
What's the advantage of sharding in general?
It allows you to distribute your workload across multiple compute resources, each working with its own subset of data, thus giving you an option of horizontal scalability; you can add more shards if your data or your workload grow.
Traditionally a sharded application might look something like this:
Clients C1..3 would connect to servers S1..3 as appropriate, whereas each server would have its own isolated storage with a subset of data Dx.
It all works well until one of the servers fails. In that case the data it holds becomes unavailable to clients until the server is recovered. To improve availability replication is frequently used, placing redundant copies of data shards on other server' storage.
Thus, if a server should fail, other servers that store replicas of its data shards will pick up the load, so the clients continue to have access to all of the data. This comes at the cost increased network traffic due to replication and higher storage utilisation, because you need to store extra copies of data.
Multi-master Aurora solves the latter problem by providing a single storage volume that is simultaneously accessible by all servers.
This eliminates the need to send copies of data shards across the network and store their redundant copies elsewhere.
Under normal circumstances each server works only with the shards assigned to it, but if it fails, other servers can pick up the workload without extra cost.
What's the advantage of sharding in general?
It allows you to distribute your workload across multiple compute resources, each working with its own subset of data, thus giving you an option of horizontal scalability; you can add more shards if your data or your workload grow.
Traditionally a sharded application might look something like this:
Clients C1..3 would connect to servers S1..3 as appropriate, whereas each server would have its own isolated storage with a subset of data Dx.
It all works well until one of the servers fails. In that case the data it holds becomes unavailable to clients until the server is recovered. To improve availability replication is frequently used, placing redundant copies of data shards on other server' storage.
Thus, if a server should fail, other servers that store replicas of its data shards will pick up the load, so the clients continue to have access to all of the data. This comes at the cost increased network traffic due to replication and higher storage utilisation, because you need to store extra copies of data.
Multi-master Aurora solves the latter problem by providing a single storage volume that is simultaneously accessible by all servers.
This eliminates the need to send copies of data shards across the network and store their redundant copies elsewhere.
Under normal circumstances each server works only with the shards assigned to it, but if it fails, other servers can pick up the workload without extra cost.
Context
StackExchange Database Administrators Q#318134, answer score: 2
Revisions (0)
No revisions yet.