hard general · part of Practice Questions · Senior SWE Roadmap
Requirements to clarify
- Functional: schedule a job to run once at a specific time, or on a recurring cron-like schedule; execute reliably even if a worker crashes.
- Non-functional: a job must run exactly once (or at-least-once with idempotent handlers) even at scale, must survive node failures, must handle millions of scheduled jobs.
Core components
- Job store: durable storage of job definitions (schedule, payload, target) — often a DB indexed by “next run time” for efficient polling.
- Scheduler/dispatcher: periodically polls for jobs due to run, and hands them off to a worker pool (typically via a message queue, decoupling scheduling from execution — see Message Queues & Event-Driven Architecture).
- Leader election / partitioning: multiple scheduler instances must coordinate so the same job isn’t dispatched twice — either a single elected leader polls, or job ranges are partitioned across schedulers.
- Worker execution & retries: workers pull jobs from the queue, execute, and report success/failure; failed jobs get retried with backoff, with a dead-letter queue after repeated failures.
Key tradeoffs
- Exactly-once execution is very hard to guarantee end-to-end; most real systems provide at-least-once execution and require job handlers to be idempotent instead.