AWS SQS (FIFO)

Overview

• Type of SQS queue (compare: Standard)
• Stronger guarantees on ordering and exactly-once processing
  • works under specific condidtions
  • exactly-once delivery is generally impossible in distributed systems

Ordering

• MessageGroupId parameter applied on SendMessage*
  • Partitions the messages into multiple groups (each group preserves ordering)
• Sender
  • To be meaningful each sender should be single-thread and use distinct MessageGroupId
• Receiver
  • ReceiveMessages may return multiple distinct MessageGroupIds
    • No control which ones are returned
    • These MessageGroupIds become "blocked"
      • Until they are deleted (acknowledged)  other Receiver  are "partially blocked"

Rate Limit

• Unlike Standard FIFO imposes rate limits
• Max 300 requests/second

Send Deduplication 

• SQS drops duplicate messages (from sender)
• Types
  • ContentBasedDeduplication
    • SQS calculates SHA256 over message content (not attributes)
  • MessageDeduplicationId
    • Explicitly passed by Sender
    • Overrides ContentBasedDeduplication
• Detection for 5 minutes

Receive Deduplication

• ReceiveMessageDeduplicationId
  • Can be passed in Receive
  • Useful when processing crashed and we retry
    • Allows to skip the normal VisibililtyTimeout

References

• https://aws.amazon.com/blogs/developer/how-the-amazon-sqs-fifo-api-works/