Raw reading list

exclude-master-tasklist

• Instagram sharding: https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a5c

• Dynamo: https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

• Dynamo DB: https://www.usenix.org/system/files/atc22-elhemali.pdf

• Anti-entropy without merkle trees: https://repositorium.sdum.uminho.pt/bitstream/1822/51489/1/DottedDB-SRDS2017.pdf

• P0 https://systemdesign.one/hinted-handoff/

• zab protocol

• Paxos, multi-paxos, paxo group (google spanner)

• Raft: https://www.youtube.com/watch?v=ro2fU8_mr2w

• Sparse indexes in database

• pack files ?

• Conflict free replicated data types: https://arxiv.org/pdf/1805.06358

• A commutative replicated data type for cooperative editing: https://inria.hal.science/file/index/docid/445975/filename/icdcs09-treedoc.pdf

• Database engineering knowledge base: https://arpitbhayani.me/knowledge-base/database-engineering

• Preventing Postgres SQL Race Conditions with SELECT FOR UPDATE: https://on-systems.tech/blog/128-preventing-read-committed-sql-concurrency-errors/

• Postgres anti patterns: https://www.enterprisedb.com/blog/postgresql-anti-patterns-read-modify-write-cycles

• https://use-the-index-luke.com/

• https://dicedb.io/

• https://www.uber.com/en-IN/blog/ringpop-open-source-nodejs-library/

• https://www.yugabyte.com/

• cassandra consistency levels: https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/dml/dmlConfigConsistency.html

• notion sharding: https://www.notion.com/blog/the-great-re-shard

• Advisory locking in postgres

• Distributed locks during transactions

• https://newsletter.systemdesign.one/p/whatsapp-engineering

• https://grafana.com/blog/2024/02/09/how-i-write-http-services-in-go-after-13-years/

• https://www.dragonflydb.io/guides/redis-sorted-sets

• https://www.dragonflydb.io/blog/transactions-in-dragonfly

• https://www.dragonflydb.io/guides/message-broker-vs-message-queue

• https://benjamincongdon.me/blog/2021/08/17/B-Trees-More-Than-I-Thought-Id-Want-to-Know/

• Go scheduler https://www.youtube.com/watch?v=YHRO5WQGh0k

• Lock free data structures/programming

  • https://www.youtube.com/watch?v=c1gO9aB9nbs
  • https://www.youtube.com/watch?v=lVBvHbJsg5Y
  • https://www.1024cores.net/home/lock-free-algorithms/introduction

• RAG

  • https://www.pinecone.io/learn/splade/
  • https://medium.com/@zilliz_learn/exploring-bge-m3-and-splade-two-machine-learning-models-for-generating-sparse-embeddings-0772de2c52a7

• https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction

• Distributed storage/systems

  • Erasure coding: https://www.backblaze.com/blog/reed-solomon/#:~:text=Reed%2DSolomon%20is%20an%20erasure,element%20are%20lost%20or%20unavailable.

• https://www.smashingmagazine.com/2017/04/guide-http2-server-push/

• https://www.confluent.io/events/kafka-summit-london-2023/an-introduction-to-kafka-cruise-control/

• Error wrapping in Go: https://bitfieldconsulting.com/posts/wrapping-errors

• https://www.scylladb.com/glossary/acid-database/

• DDD + SOLID:

  • https://www.youtube.com/watch?v=y1JiMGP51NE
  • https://medium.com/building-inventa/how-we-used-event-storming-meetings-for-enabling-software-domain-driven-design-401e5d708eb

• Google spanner: https://dl.acm.org/doi/pdf/10.1145/3035918.3056103

• Swiss hash tables

• Valkey hash table

• Papers

  1. Dynamo - Amazon’s Highly Available Key Value Store ✅ 2025-05-21
  2. Google File System: Insights into a highly scalable file system
  3. Scaling Memcached at Facebook: A look at the complexities of Caching
  4. BigTable: The design principles behind a distributed storage system
  5. Borg - Large Scale Cluster Management at Google
  6. Cassandra: A look at the design and architecture of a distributed NoSQL database
  7. Attention Is All You Need: Into a new deep learning architecture known as the transformer
  8. Kafka: Internals of the distributed messaging platform
  9. FoundationDB: A look at how a distributed database
  10. Amazon Aurora: To learn how Amazon provides high-availability and performance
  11. Spanner: Design and architecture of Google’s globally distributed databas
  12. MapReduce: A detailed look at how MapReduce enables parallel processing of massive volumes of data
  13. Shard Manager: Understanding the generic shard management framework
  14. Dapper: Insights into Google’s distributed systems tracing infrastructure
  15. Flink: A detailed look at the unified architecture of stream and batch processing
  16. A Comprehensive Survey on Vector Databases
  17. Zanzibar: A look at the design, implementation and deployment of a global system for managing access control lists at Google
  18. Monarch: Architecture of Google’s in-memory time series database
  19. Thrift: Explore the design choices behind Facebook’s code-generation tool
  20. Bitcoin: The ground-breaking introduction to the peer-to-peer electronic cash system
  21. WTF - Who to Follow Service at Twitter: Twitter’s (now X) user recommendation system
  22. MyRocks: LSM-Tree Database Storage Engine
  23. GoTo Considered Harmful
  24. Raft Consensus Algorithm: To learn about the more understandable consensus algorithm
  25. Time Clocks and Ordering of Events: The extremely important paper that explains the concept of time and event ordering in a distributed system

• Debezium

• CDC messages (change data capture)

• https://research.google/pubs/f1-the-fault-tolerant-distributed-rdbms-supporting-googles-ad-business/

• https://research.google/pubs/spanner-googles-globally-distributed-database-2/

• Demystifying distributed SQL and why companies choose CockroachDB? https://www.youtube.com/watch?v=OrZszgOt5Fs

  • Write behind logging: https://db.cs.cmu.edu/papers/2016/p337-arulraj.pdf

• Amazon aurora: https://assets.amazon.science/dc/2b/4ef2b89649f9a393d37d3e042f4e/amazon-aurora-design-considerations-for-high-throughput-cloud-native-relational-databases.pdf

• What Every Programmer Should Know about How CPUs Work: https://www.youtube.com/watch?v=-HNpim5x-IE ✅ 2025-05-19

• IPFS, IPFN, Orbit DB, Web3 storage, Stable coins, smart contracts

• DBOS

• SingleStore

• Zero copy I/O

• Timescale architecture for real-time analytics

• SpacetimeDB

• Multi tenent databases

• Serializability and Linearizability https://aerospike.com/blog/concurrency-control-in-dbms/

• https://qdrant.tech/articles/hybrid-search/

• https://qdrant.tech/articles/late-interaction-models/

• ZeroMQ

• Terragrunt

• Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics : https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf

• CXL for disaggregated memory in cloud native databases vs RDMA (Remote Direct Memory Access)

• Golden master approach

Data Engineering

• https://github.com/apache/hamilton
• https://keymakr.com/blog/seamlessly-integrate-annotation-stages-into-your-data-pipeline/