AWS SQS

Model

• Queue - identified by url
  • e.g.  http://sqs.us-east-1.amazonaws.com/123456789012/queue2
  • if idle may get deleted after 30 days
•  Message
  • Max: 256KB of text. Larger can be managed via S3
    • SQS message is a pointer to S3 object
  • Max 10 messages in single request (Send/Receive)
  • Has uniquely assigned MessageId (max 100 characters)
  • MD5 of the message is returned on Receive
  • Receipt handle
    • Returned when a message is received
      • If received many times: latest handle is valid
    • Needed to delete a message
    • Max 1024 characters
  • Retention Period
    • Default: 4d
    • Range: 60s-14d
• Visibility Timeout
  
  • After receive the message remains in the queue but is "invisible" for others to receive
  • Prevents multiple consumers processing the same message (i.e. reservation)
  • Invisible = In Flight
  • Timeout can be updated per queue/per message
  • VisibilityTimeout=0: "I do not want to process it, just peeking"
  • Default: 30s
  • Range: 0s-12h
• Types
  • Standard
    • Ordering: NOT guaranteed (message can arrive in any order)
      • Producer can include sequence# and consumer can reorder itself (like TCP does)
    • At-least-once delivery
      • Possible to get duplicates as messages stored on multiple servers and receive/delete may not reach all of them
      • Processing should be idempotent
  • FIFO
    • See SQS(FIFO)

Message Attributes

• Send along with message but separate from message body
• Max 10 attributes per message
• Can be used for structured metadata (timestamp, geospatia data, identifiers)
• Structure
  • Name
  • Type - String, Number, Binary
    • CustomType (e.g. Binary.gif, Binary.jpeg, Number.float) - type trait
  • Value

Pricing

• Requests
  
  • 1 request = max 64KB chunk so 256 kB message  = 4 requests
• Data transfer

Polling

• Short (standard)
  • Sample random servers (e.g. A,B)
  • May not retrieve messages even if they exist (e.g. on C)
  • WaitTimeSeconds = 0 or queue attribute ReceiveMessageWaitTimeSeconds = 0
• Long
  • wait until message is available or request times-out
  • checks ALL the servers (A,B,C) unlike "Short polling"
  • WaitTimeSeconds: 1-20 has priority over ReceiveMessageWaitTimeSeconds 
  • can be set on:
    • ReceiveMessage
    • CreateQueue
    • SetQueueAttribute
  • Default: 0
  • Range: 0-20s

Delivery Delay

• Default: 0
• Range: 0-15min
• Delayed Queue
• Message Timer

Batching

• Reduces cost (pricing based on requests not individual messsage)
• SendMessageBatch
• DeleteMessageBatch
• ReceiveMessage already processes up to 10 messages (no batch counterpart)
• AmazonSQSBufferedAsync in Java

Dead Letter Queue (DLQ)

• Enable Redrive Policy
  • Target queue ARN
  • Configure maximum number of receives before message is sent to DLQ
• Retention based on original creation date
  • DLQ should typically have longer retention
• Requires separate consumer process for this queue
• Allows to isolate failed messages - "poision pills"
  • Delete never happened for them
• AWS Console "peek" counts as Receive

SNS Integration

• Topic subscription
• Fan-outs
  • Image uploaded event sent to SNS
    • SQS: generate thumbnail
    • SQS: image recogntion
    • SQS: indexing

Encryption

• Stored in encrypted form on SQS Servers
  • Encrypted
    • Message body
  • Not Encrypted
    • Queue metadata
    • Message metdata (message Id, timestamp, attributes)
    • Per-queue metrics
• SSE-KMS
  • AWS-managed CMK
  • Custom CMK
• Data Key Reuse Period
  • "Data Key" caching - configurable
  • Shorter -> more expensive -> better protection
    • KMS has limit 100 TPS

Permissions

• Resource level permission (similar to bucket policy)
• e.g. Grant other AWS accounts access
  • Also anonymous access
  • Supports conditions