hard amazon · part of Practice Questions · Senior SWE Roadmap
Requirements to clarify
- Functional: store and retrieve arbitrarily large binary objects by key, with metadata.
- Non-functional: extreme durability (data must essentially never be lost), high availability, scales to exabytes, cost-efficient for cold/rarely-accessed data.
Core components
- Object chunking: large objects are split into fixed-size chunks distributed across many disks/nodes rather than stored as one giant blob on one machine.
- Durability via erasure coding or replication: replication (store N full copies) is simple but storage-expensive; erasure coding (split data into k data + m parity chunks, able to reconstruct from any k of the k+m) gives similar durability at much lower storage overhead — the standard tradeoff to discuss.
- Metadata service: tracks which chunks/nodes make up each object, separate from the data path itself — this index needs to be highly available since it’s needed for every read.
- Consistency model: many object stores offer strong read-after-write consistency for new objects but historically weaker consistency on overwrites/deletes — worth stating explicitly which model you’re designing for.
Key tradeoffs
- Erasure coding saves significant storage cost over full replication but costs more CPU to reconstruct data and is more complex to implement — the standard answer for “how would you reduce storage costs while keeping durability.”