AWS IoT

Security and Identity

• Uses TLS 1.2
• Mutual authentication
• Each device has unique client certificate 
• Each device has a policy (what it is allowed to do)

Thing

• Representation of a device
• Attributes
• Certificate

Device Gateway

• Handles all the long-lived connections
• • Front-end for all AWS services
• Protocols
• • MQTT
  • • Lightweight TCP protocol (port 1883)
    • Designed by IBM
  • HTTP
• Brokering layer
• • Used to pass messages 
• Topic based architecture (publish/subscribe)
• • e.g. "building3/chillers/unit5"

Device Registry

• Tag devices with metadata
• • thingname
  • attributes (MD, SN, install_date)

Device Shadow

• Persistent device state
• You can program against this state
• • desired state
  • reported state
  • delta: difference between reported and desired states
  • • IoT sends the delta to the device
• It may be outdated

Rules Engine

• Transform messages based on rules
• Rule Structure
• • Attribute
  • Topic Filter
  • Condition
  • Action
• Examples
• • Insert into DynamoDB
  • Alert when coolant level on AC goes below 20%

Device SDKs

• C-SDK
• RTOS
• JS-SDK
• RaspberryPi
• Arduino SDK
• Mobile SDK (Android, iOS)

AWS ElastiCache

Cache Architectures

• Side cache
  • Most common
  • Independent from datasource
  • Can go out of sync
  • Requires development work
  • Examples
    • Typically used Memcached, Redis
• Write-through/Read-through
  • Integrated with data source (same API)
  • Always in sync
  • Examples
    • DynamoDB Accelerator (DAX)
• Write-behind
  • Integrated with data source
  • Writes happen asynchronously
    • Can go out of sync

Memcached

• Cache Cluster - cache nodes + cache parameters group
• Configuration Endpoint (Auto-discovery)
  • DNS name for an arbitrary cache node
  • Each node maintains the list of all other nodes in the cluster
  • Client (auto-discovery aware) uses this list to create connections
  • Client hashes the key (consistent hashing) and selects the node
  • AWS provides "auto-discovery" clients for .NET, Java, PHP

AWS SQS

Model

• Queue - identified by url
• • e.g.  http://sqs.us-east-1.amazonaws.com/123456789012/queue2
  • if idle may get deleted after 30 days
•  Message
• • Max: 256KB of text. Larger can be managed via S3 (SQS message is a pointer)
  • Max 10 messages in single request (Send/Receive)
  • Has uniquely assigned MessageId (max 100 characters)
  • MD5 of the message is returned on Receive
  • Receipt handle
  • • Returned when a message is received
    • • If received many times: latest handle is valid
    • Needed to delete a message
    • Max 1024 characters
  • Retention Period
  • • Default: 4d
    • Range: 60s-14d
• Visibility Timeout
• • After receive the message remains in the queue but is "invisible" for others to receive
  • Prevents multiple consumers processing the same message (i.e. reservation)
  • Invisible = In Flight
  • Timeout can be updated per queue/per message
  • VisibilityTimeout=0: "I do not want to process it, just peeking"
  • Default: 30s
  • Range: 0s-12h
• Guarantees
• • Ordering: NOT guaranteed (message can arrive in any order)
  • • Producer can include sequence# and consumer can reorder itself (like TCP does)
  • At-least-once delivery
  • • Possible to get duplicates as messages stored on multiple servers and receive/delete may not reach all of them
    • Processing should be idempotent

Message Attributes

• Send along with message but separate from message body
• Max 10 attributes per message
• Can be used for structured metadata (timestamp, geospatia data, identifiers)
• Structure
• • Name
  • Type - String, Number, Binary
  • • CustomType (e.g. Binary.gif, Binary.jpeg, Number.float) - type trait
  • Value

Pricing

• Billed for request
• • 1M = 0.05$
  • 1 request = max 64KB chunk so 256 kB message  = 4 requests
• Billed for data transfer

Polling

• Short (standard)
• • Sample random servers (e.g. A,B)
  • May not retrieve messages even if they exist (e.g. on C)
  • WaitTimeSeconds = 0 or queue attribute ReceiveMessageWaitTimeSeconds = 0
• Long
• • wait until message is available or request times-out
  • checks ALL the servers (A,B,C) unlike "Short polling"
  • WaitTimeSeconds: 1-20 has priority over ReceiveMessageWaitTimeSeconds 
  • can be set on:
  • • ReceiveMessage
    • CreateQueue
    • SetQueueAttribute
  • Default: 0
  • Range: 0-20s

Delivery Delay

• Default: 0
• Range: 0-15min
• Delayed Queue
• Message Timer

Batching

• Reduces cost (pricing based on requests not individual messsage)
• SendMessageBatch
• DeleteMessageBatch
• ReceiveMessage already processes up to 10 messages (no batch counterpart)
• AmazonSQSBufferedAsync  in Java

Dead Letter Queue

• Enable Redrive Policy
• • Target queue ARN
  • Configure maximum number of receives before message is sent to DLQ
• Requires separate consumer process for this queue
• Allows to isolate failed messages (delete never happened for them)
• AWS Console counts as Receive

SNS Integration

• Topic subscription
• Fan-outs
• • Image uploaded event sent to SNS
  • • SQS: generate thumbnail
    • SQS: image recogntion
    • SQS: indexing

Permissions

• Resource level permission (similar to bucket policy)
• e.g. Grant other AWS accounts access
• • Also anonymous access
  • Supports conditions

AWS Glacier

Model

• Vault
• • name may be the same across regions (unique per region)
  • container for archives
  • analogous to S3 bucket
  • max 1000 per region
• Archive
• • base unit storage in Glacier
  • immutable (create/delete only)
  • can be any data (photo, document, etc.)
  • • best practice: aggregate data into .zip or .tar
  • 32 kB metadata overhead
  • max 40 TB
  • Upload
  • • Single max 4GB
    • Recommended multi-part for > 100 MB
    • • Compute and supply tree-hash
      • • Hash for each megabyte segment and combine in tree fashion
• Inventory
• • Updated once per day
  • List of all archives
  • Inventory date not changed if no add/delete of archives
  • Format: CSV or JSON

Jobs

• Executed asynchronously (Job ID returned)
• Associated with vault
• • Multiple jobs may be in-progress
• Typically 3.5-4.5 hours
• When it completes user can download the output (available for 24h)
• Types
• • Archive Retrieval
  • • entire archive or subset of files in the archive
  • Inventory Retrieval (list of archives)
  • • filter can be applied (e.g. archive creation date)
• May have SNS notifications enabled

Tree Hash

• On upload include 2 headers
• • x-amz-content-sha256
  • • hash of entire payload used for signature calculation
  • x-amz-sha256-tree-hash
  • • specific to archive upload
    • main benefit - avoids re-reading a (potentially big) file to calculate its hash
    • • it's computed piece-meal
    • for each chunk of 1MB compute hash (last may be < 1 MB)
    • • build the next level of tree (compute hash again)
      • • repeat until you reach top (root)
    • Examples
    • • Single request (6.5 MB)
      • • 1 request (SHA256 computed 13 times)
      • Multi-part
      • • 2 requests each has hash-tree of corresponding parts
        • Complete Multipart Upload (tree hash of entire archive)

Vault Access Policy

• Resource-based policy (similar to bucket policy)

Vault Lock Policy

• Similar to vault access control
• Enforce compliance requirements
• • e.g. WORM (Write Once Read Many)
• Once policy is locked it cannot be edited
• • Stronger control than vault access policy
• Use case
• • time-based data retention rules (deny deletes) but allow read access
  • • Combine vault lock policy (deny delete) and vault access policy (read)
• Process
• • Initiate lock
  • • Sets to IN_PROGRESS and returns LockId
    • Validate and test your policy
    • 24 hours timeout 
  • Complete the lock process
• Policy elements
• • Resource (vault)
  • Conditions
  • • glacier:ArchiveAgeInDays, glacier:ResourceTag
  • Action

Pricing

• Allowed to retrieve 5% of data any month for free
• • More retrieval you pay per 1 GB
• Peak usage taken and applied to the whole month retroactively
• Data Retrieval Policy
• • Simplifies cost management by setting limits
  • • Free Tier Only
    • Max Retrieval Rate (GB/h)
    • No Retrieval Limit

S3

• Integrated with S3
• Lifecycle configuration can transition between S3 <-> Glacier

AWS EC2 - Instance Types

General Purpose

• t2
• • burstable performance
  • • credit = 1 vCPU running at 100% for 1 minute
    • • = 1 vCPU running at 50% for 2 minutes
    • example: t2.small
    • • vCPU: 1
      • • CPU: Intel Xeon with Turbo - up to 3.3 GHz
      • Initial CPU credit: 30 (when started)
      • Credits earned per hour: 12 (=1 per 5 minutes)
      • Maximum Balance: 288
      • Base Performance: 20% of the CPU core
    • stop-start
    • • Earned credits are lost
      • Initial balance is reset
    • expiry
    • • initial credits do not expire
      • earned credits expire
    • Metrics
    • • CPU Credit Usage
      • • Calculate: How may are earned per period (e.g. 5 minutes)
        • Compare: How many are used per period
        • Ensure earned > used over longer period
  • Use cases
  • • development environments, build servers, code repositories, low traffic websites, micro services, experiments, beta 
  • Restrictions
  • • EBS-only

• m4
• • 2.4 GHz Intel Xeon v3 (Haswell)
  • EBS-optimized by default
  • Support for enhanced networking
  • Restrictions
  • • EBS-only

• m3
• • High Frequency Intel Xeon (Ivy Bridge) or 2.6 GHz Intel Xeon (Sandy Bridge)
  • SSD-based instance storage (4-160GB)
  • Use cases
  • • small&mid-size db, SAP, SharePoint, cluster computing, enterprise apps, caching fleets, data processing tasks

• Previous Generation
• • t1, m1

Compute Optmized

Cheapest CPU

Use cases

• HPC
• Front-end fleets, batch processing, distributed analytics, science and engineering projects, ad serving, MMO, video encoding

• c4
• • High Frequency Intel Xeon (Haswell)
  • EBS-optimized by default
  • Support for enhanced networking
  • Support for Placement Groups
  • CPU:
  • • 18 cores with two hyperthreads each (36 vCPU)
    • Customized High Frequency Intel Xeon (Haswell)
    • Turbo boost: ability to increase Hz of 2 cores when other 16 are powered down
    • Ability to control
    • • C-state - sleep level when core is idle
      • • Cores in low power require lower energy BUT
        • It takes longer to wake them up
        • • Ability to create thermal headroom for turbo boost
      • P-state - clock frequency
      • Linux only
  • Restrictions
  • • EBS-only

• c3
• • High Frequency Intel Xeon (Ivy Bridge)
  • SSD-based storage (32-640GB)
  • Support for enhanced networking
  • Support for Placement Groups

• Previous Generations
• • c1, cc2.8xlarge, cc1

Memory Optimized

Cheapest RAM

• r3
• • Ivy Bridge
  • RAM 15.5-244GB
  • SSD Storage (32-640GB)
  • Support for Enhanced Networking
  • Use cases
  • • high performance databases, distributed memory caches, in-memory analytics, genome assembly and analysis, large SAP and Microsoft deployments

• Previous generation
• •  m2, cr1   

GPU

• g2
• • High Frequncy Intel Xeon (Sandy Bridge)
  • NVIDA GPU (1536 CUDA cores) + 4GB RAM
  • • 1-4 such processors
    • hardware video encoder
    • • 8 stream 720@30fps
      • low latency frame capture
  • Use cases
  • • 3D application streaming, machine learning, video-encoding and server-side graphics
      

• Previous Generation
• • cg1

Storage High I/O

• i2
• • CPU: Intel Xeon (Ivy Bridge)
  • SSD-based local storage 
  • • Disks: 1x800 - 8x800 GB
    • TRIM
  • Supported:
  • • Enhanced Networking
    • Placement Groups
    • EBS Optimization
  • Use cases
  • • Cassandra, Data warehousing, Hadoop, cluster file system
  • IOPS
  • • Read IOPS: 35,000 (i2.xlarge) - 365,000 (i2.8xlarge)
    • First Write IOPS: 35,000 (i2.xlarge) - 315,000 (i2.8xlarge)

• Previous Generation
• • hi1

Storage High Density

Cheapest HDD 

• d2
• • HDD Storage (6TB - 48 TB)
  • High disk throughput
  • EBS Optimized by default
  • Support for Enhanced Networking

• Previous Generation
• • hs1

Network Performance

• Low Performance: < 1Gbps
• Medium Performance: ~1 Gbps

AWS EC2 - ELB(CLB)

Overview

• Layer 4 (TCP/SSL) + Layer 7 (HTTP/HTTPS simplified)
• Use cases
  • Instances in EC2-Classic (other LBs can't do this)

Types

• Internet-facing
• Internal (VPC only)
  • Routes to instances in private subnets
  • DNS: internal-name-123456789.region.elb.amazonaws.com
    • Resolves to private IPs of ELB nodes

Nodes

• Contains multiple nodes (in selected AZs)
• ELB DNS maps to IP addresses of the nodes
  
  • IP addresses may change
  • DNS TTL 60 seconds
• ELB node sends traffic to specific backend instances

Listeners

• HTTP
• HTTPS (secure HTTP)
• TCP
• SSL (secure TCP)
  • session affinity not supported
• Not supported
  • UDP (ELB wouldn't know if backend accepted a connection)

Routing Algorithm

• TCP Listeners - round robin
• HTTP(S) - least outstanding requests ("least busy")

Backend Types

• EC2 instances
• Autoscaling Groups

Sticky Sessions (session affinity)

• Allowed for HTTP/HTTPS listeners
• Application Cookie
  
  • Follows duration of existing application session cookie
    • Name must be provided
  • If application cookie expires/gets removed the session stops being sticky
  • ELB inserts new stickiness cookie when a new application cookie is created
    • request routed to NEW instance
    • continues to route to NEW even if old comes back after failure
• ELB  Cookie
  • Cannot set  "secure" or "HttpOnly" on this cookie
  • Contains mapping to a specific instance
  • Duration
    • By default: browser session
    • Can be specified
  • Stickiness lost when switching to different listener (e.g. HTTP -> HTTPS)
• May cause scaling issue
  • Clients stay on the instances
  • Even though you may add a lot of backend instances they will not receive any traffic

Proxy Protocol

• Passes source/destination - (IP,port) for TCP/SSL requests
• Human Readable header added to the request

  • PROXY TCP4 198.51.100.22 203.0.113.7 35646 80\r\n

• Must be supported by backend instance
• Configured via Policy Type

Connection Draining

• Instance status: "InService: instance deregistration currently in progress"
• Default: 300s (5 minutes), Max 3600s (1h)

Dualstack

• EC2-Classic supported
• Communication to backend over IPv4
• DNS names

  • name-123456789.region.elb.amazonaws.com

  • ipv6.name-123456789.region.elb.amazonaws.com    

  • dualstack.name-123456789.region.elb.amazonaws.com

    • Returns both IPv4 and IPv6     


Security

• Security Policy 
  • SSL Protocol: SSL 2.0, SSL 3.0, TLS 1.0 (=SSL 3.1), TLS 1.1, TLS 1.2
  • SSL Ciphers
    • ECDHE for Perfect Forward Secrecy (ephemeral keys are not stored)
  • Server Order Preference
    
    • Enabled: ELB selects first cipher from its list that is on client list (i.e. ELB has the final say)
    • Disabled: client decides
  • Types of policy
    • Predefined (versioned)
    • Customer Defined
• Security Group - allows backend server to restrict access (i.e. traffic via ELB only)
• Supports certificates from IAM and AWS Certificate Manager

Metrics

• SurgeQueueLength - how many requests queued up on ELB waiting to be serviced
  • Queue length is 1024
• SpillOverCount - queue grew too much and requests were rejected by ELB
• BackendConnectionsErrors - how are backend servers behaving

Prewarming

• Takes time to scale-out ELB when flash traffic expected
• Possible to call support and ask for pre-warming

Backend Server Authentication Type

• Optional authentication when using HTTPS 
• Process
  • Add certificate name and "pem"-encoded certificate to ELB
  • Enable

Misc

• Cross Zone Load Balacing
  • evens out imbalance between AZs
  • ensures there is no 503 when there are no backend instances in the zone
• ELB scaling typically takes 1-7 minutes
• ELB  sends traffic to primary private IP address
• Sending traffic to instance in private subnet
  • For each subnet EC2 instance gets a public IP
    • hence: add public subnets to ELB only
• Single instance can be added to multiple ELBs

Subnets

• Public subnet for "public ELB" and private subnet for "internal ELB"
• Max 1 subnet per AZ
• ELB creates a load balancing node in each subnet
  • Specify at least 2 subnets

References

• https://aws.amazon.com/articles/best-practices-in-evaluating-elastic-load-balancing/
• http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/how-elb-works.html

AWS Disaster Recovery

Concepts

RPO - recovery point objective (how much data loss is acceptable, e.g. updates from last 5 minutes can be lost)

RTO - recovery time objective (how long should it maximally take to restore the data from backup, e.g. 5 hours)

Scenarios

• Backup&Restore ($)
• • Glacier
  • S3
  • Storage Gateway
  • • Cached volumes (S3 is authority)
    • VTL - backup target for tapes
  • Transfer
  • • DX
    • Import/Export
    • Internet
  • Long RTO, High RPO
• Pilot Light ($$)
• • Minimal version of critical core components always running
  • • Database server (e.g. asynchronous replication)
  • Preconfigured server AMIs - can be started quickly
• Warm standby ($$$)
• • Scaled down version of environment running all the time
  • Can be used for non-production work
  • • Test, QA, internal use
  • Patch on same schedule as primary
  • App tier & standby-by site write to both primary and stand-by
  • Upon recovery
  • • add more servers
    • resize instances 
    • Switch DNS to warm standby
• Multi-Site ($$$$)
• • active-active
  • Two regions at the same time
  • Route53 for weighted routing
  • Complext state management (database reconciliation)

RDS

• Multi A-Z synchronously replicates data between AZ
• • Eliminates I/O freezes during backup
  • Eliminates outage during maintenance windows
• Automatic failover
• • AZ outage
  • Primary DB fails
  • DB instance type is changed
  • OS of DB is undergoing patching
  • Manual failover initiated (reboot with failover)
  • Failover ~60s
• Daily backup (within 30 minutes "backup window")
• • Retention period configurable - number of days
  • When deleting instance all automatic backups are deleted (only manual snapshots remain)
  • Automated DB snapshot can be copied (retain indefinitely)
• RDS archives logs (point in time recovery)
• • Ability to recover to arbitrary point in time in the retention period (RPO 5 minutes)
• Use InnoDB (MyISAM will not work)
• On restore possible to change many options
• • engine (SQL Server Standard -> SQL Server Enterprise)
• Snapshots can be copied cross-region

S3

• Very high durability (11 9's)
• • Reducecd Storage cheaper but lower durability
• Replicated within region
• Possible to replicate cross-region
• • Asynchronous
• Access time: instant

Glacier

• Very high durability (like S3)
• Access time: a few hours

DynamoDB

• Backed by SSD
• Replicated in 3 AZs in a region
• DataPipeline supports
• • cross region DynamoDb copy (with filtering)
  • copy to S3

AMI

• Can be copied cross-region

EBS

• AFR - 0.5-1.0% (if < 20GB modified since last snapshot)
• Snapshots reduce AFR
• RAID does not offer that much help as all disks in array are likely to fail together
• Replicated within AZ (mirrored using proprietry technology)
• Snapshots can be copied cross-region

Business Continuity

• How business continous to operate in the event of disaster
• Fire, theft, flood