snippetMinor
How to ensure equitable distribution of SQS messages in a distributed systems setting?
Viewed 0 times
systemsensureequitablesettingdistributedmessageshowdistributionsqs
Problem
I have multiple servers, each having a script polling an SQS queue [all polling the same queue].
So, is there any way I can ensure an equitable distribution of messages to all those clients [i.e. my worker servers here]. Like for example, if there are 100 messages in the queue, then
Can AWS ELB (Elastic Load Balancer) help me do it? If yes, then how? If not, then is there an alternate service in the AWS ecosystem which can help me do it?
Or am I overthinking this? I mean, can this be solved straightforwardly in the polling script? [Please keep in mind the race conditions involved due to multiple clients polling a single queue]
So, is there any way I can ensure an equitable distribution of messages to all those clients [i.e. my worker servers here]. Like for example, if there are 100 messages in the queue, then
20-20-20-20-20 if there are 5 workers, and so on.Can AWS ELB (Elastic Load Balancer) help me do it? If yes, then how? If not, then is there an alternate service in the AWS ecosystem which can help me do it?
Or am I overthinking this? I mean, can this be solved straightforwardly in the polling script? [Please keep in mind the race conditions involved due to multiple clients polling a single queue]
Solution
If there are 100 messages in the queue and 5 consumers, the initial distribution will be no more than 10-10-10-10-10.
A single response can never return more than 10 messages.
This seems like a non-issue.
Race conditions related to multiple consumers should be a non-issue as well. SQS is designed for multiple simultaneous consumers.
Use long polls and a 20 second max wait timer and be amazed. (No, a 20 second wait does not delay messages by 20 seconds. It doesn't delay them at all. You sort of need to see it in action to really understand how it works.)
You're definitely overthinking some things, I suspect.
A single response can never return more than 10 messages.
This seems like a non-issue.
Race conditions related to multiple consumers should be a non-issue as well. SQS is designed for multiple simultaneous consumers.
Use long polls and a 20 second max wait timer and be amazed. (No, a 20 second wait does not delay messages by 20 seconds. It doesn't delay them at all. You sort of need to see it in action to really understand how it works.)
You're definitely overthinking some things, I suspect.
Context
StackExchange DevOps Q#1058, answer score: 6
Revisions (0)
No revisions yet.