hard general · part of Practice Questions · Senior SWE Roadmap · topic form: Message Queues & Event-Driven Architecture
Requirements to clarify
- Functional: producers publish messages to topics/queues, consumers subscribe and process them, in order (at least per-partition) and without loss.
- Non-functional: high throughput, durability (messages survive broker crashes), horizontal scalability, configurable delivery guarantees.
Core components
- Partitioning: a topic is split into partitions, each an ordered, append-only log — ordering is only guaranteed within a partition, which is what allows parallel consumption across partitions.
- Replication: each partition is replicated across multiple brokers (leader + followers) for durability; a broker failure triggers leader election among the in-sync replicas.
- Consumer groups: multiple consumers in a group split the partitions among themselves, each partition consumed by exactly one consumer in the group at a time — this is how horizontal consumer scaling works.
- Offset tracking: consumers track how far they’ve read (an offset) so they can resume after a crash without reprocessing everything (or, for at-least-once, resume slightly behind and reprocess a few messages).
Key tradeoffs
- At-least-once (simple, occasional duplicates, requires idempotent consumers) vs exactly-once (needs transactional writes/dedup, meaningfully more complex) — most systems default to at-least-once and push idempotency onto the consumer.