distributed-systems

DynamoDB Services

  1. Transaction manager
  2. Lock manager
  3. Request router
  4. Replicas/nodes
  5. Metadata service
  6. Autoadmin
  7. Backup and restore
  8. DynamoDB streams

Metadata service

Contains data on:

  • All partitions of a table
  • Key range of each partition
  • Storage nodes for each partition (leader, replica information)

Request router using metadata service to retrieve node data for keys. Data is locally cached on request router to reduce latency.

Cache data rarely changes, i.e, partitions rarely change, partition node data rarely changes, etc. Incase of huge cache misses, metadata service receives high traffic, which might lead to a crash.

Solution: MEM DS (In-memory distributed data store)

  • implemented using patricia + merkle trees.
  • Optimised for range queries (finding out which partition range a key falls in).
  • Request router now relies on MemDS if cache miss (less expensive than directly hitting the metadata service).
  • MEM DS is horizontally scalable. MEM DS is constantly told about incoming load (using async request from request router) in order to keep it well provisioned for incoming requests incase of sudden cache miss.
  • Storage nodes update MemDS on partition information.

AutoAdmin service

  • splits partitions based on load
  • Tackling throughput dilation
    • Bursting
    • Adaptive capacity: Adjusts the partition throughput in proportion.
      • Example: Node capacity: 1000 write units, contains 3 partitions. Each partition gets 33% write units. With adaptive capacity, the hot partition gets 90% write units and other 2 get 100 write units. AutoAdmin will ensure no 2 hot partitions are in the same node.