Design a Distributed Job Scheduler

Earlier designs kept needing "do this later": send a marketing push at 08:00 the user's time (article 33), release a ticket hold after 10 minutes (article 38), rebuild the autocomplete index every night (article 36), send a weekly digest (article 22). Each team could run its own cron jobs, and article 22 showed how that goes wrong: the job runs on every server, or on none when the one server dies.

A distributed job scheduler turns "run this at time T" and "run this every hour" into a reliable platform service: jobs run on time, at least once, never accidentally many times, and survive machine failures. It is a classic SDE-2 design because the hard parts are time, coordination and exactly the failure cases covered in Module 6.

• How to scope a scheduler and estimate dispatch rates, including spikes at round times.
• Why scheduling and execution should be separate.
• Time-bucketed job storage and partitioned dispatchers with leases.
• Atomic claims that prevent double dispatch, with a runnable simulation.
• Workers, heartbeats, retries and dead letters.
• Recurring jobs, misfire policies, time zones and daylight saving.
• Failure handling, security for HTTP callbacks, and observability.

Build a multi-tenant scheduling service. Internal services submit jobs to run once at a given time, after a delay, or on a recurring schedule (cron-like). When a job is due, the scheduler triggers it by putting a message on the tenant's queue or calling the tenant's HTTP endpoint. Jobs can be cancelled or rescheduled.

Functional requirements: create, cancel and reschedule one-time and recurring jobs; trigger each due run on time; retry failed triggers; record run history.

Non-functional requirements

One-time jobs: 100 million scheduled per day -> about 1,160 dispatches/s on averageRecurring jobs: 2 million definitions; on average each runs 24 times a day -> 48 million runs/day,  about 560/s on averageTotal average: about 1,700 dispatches/sSpikes: people schedule at round times. Assume 10% of a day's runs fall on the first second  of each hour, spread over 24 hours:  (148 million x 10%) / 24 = about 620,000 runs due in the same second at the top of each hourStorage: 100 million one-time jobs/day x 1 KB = 100 GB/day; with 7 days of history, about 1 TB,  plus recurring definitions (small)

Conclusions: the average rate is modest, but spikes at round times are enormous: hundreds of thousands of runs due in the same second. Storage needs time-based partitioning. Precision of seconds means we cannot scan the whole table every second.

POST /v1/jobsIdempotency-Key: digest-user-42-2026-10-12{  "schedule": {"run_at": "2026-10-12T08:00:00+05:30"},          (or {"cron": "0 8 * * MON", "timezone": "Asia/Kolkata"})  "target": {"type": "queue", "name": "digest-jobs"},              (or {"type": "http", "url": "https://..."})  "payload": {"user_id": 42, "week": "2026-W41"},  "retry": {"max_attempts": 5, "backoff": "exponential"},  "misfire_policy": "run_once"                                     (or "skip", "run_all")}201 Created  { "job_id": "job_5f2c", "next_run_at": "2026-10-12T02:30:00Z" }DELETE /v1/jobs/{id}                       -> 204 (cancel future runs)PATCH  /v1/jobs/{id}  {"schedule": ...}    -> 200 (reschedule)GET    /v1/jobs/{id}/runs?cursor=...       -> run history

Times are stored in UTC; the time zone is kept for recurring schedules so the next run is computed correctly.

jobs   (job_id PK, tenant_id, schedule (run_at or cron + timezone), target, payload,        retry_policy, misfire_policy, status (active, cancelled), next_run_at, version)due_index (bucket_minute, partition, next_run_at, job_id)   -> one row per upcoming run; partitioned by (bucket_minute, partition)   -> "all runs due in this minute for partitions I own" is a small range readruns   (job_id, scheduled_at, attempt, status (dispatched, acked, failed), dispatched_at,        finished_at, error)   PRIMARY KEY (job_id, scheduled_at)   -- one logical run per job per scheduled time

• The due index is the heart of the design: it turns "find due jobs" into "read this minute's bucket", instead of scanning millions of rows. A time-ordered structure such as a sorted set keyed by run time works the same way at smaller scale (Redis sorted sets support range queries by score (Redis sorted sets)).
• partition is a hash of job_id into, say, 1,024 partitions, so dispatch work can be split across machines.
• runs with primary key (job_id, scheduled_at) gives every run a stable identity, which is the idempotency key handlers use.

One scheduler process, every second:  SELECT * FROM jobs WHERE next_run_at <= now() AND status = 'active'  for each: trigger it, then update next_run_at (or mark done)

It works on one machine with a small table. Its problems are the subject of the next section.

1. Single point of failure: if the one scheduler dies, nothing runs.
2. Double dispatch: run two schedulers for availability, and both see the same due job.
3. Scanning cost: querying a table of hundreds of millions of rows every second.
4. Top-of-the-hour spikes: hundreds of thousands of runs due at once.
5. Slow or failing targets: if triggering means calling an HTTP endpoint inline, one slow tenant delays everyone.
6. Crashes between steps: a run triggered but its record not updated, or the reverse.

Tenants --API--> Job service --> jobs + due_index (partitioned DB)Dispatchers (several; each owns a set of partitions via leases)   every second: read due_index for (current minute, my partitions) where next_run_at <= now   for each run: claim it atomically -> write dispatch to the outbox -> nextOutbox relay --> target queues (tenant queues) or the HTTP-callback queueTenant workers (competing consumers) execute jobs, idempotently, keyed by (job_id, scheduled_at)HTTP callers (scheduler-owned workers) call tenant endpoints with timeouts and retries

Dispatchers do only cheap work: read, claim, enqueue. Running jobs is someone else's problem, which keeps dispatch fast and isolates tenants. Microsoft's Scheduler Agent Supervisor pattern describes a similar division between scheduling steps, agents that perform them and a supervisor that monitors and recovers failed steps (Scheduler Agent Supervisor pattern).

The 1,024 partitions are divided among dispatchers. Each dispatcher holds a lease for its partitions in a consistent coordination store (for example etcd, which provides leases built on Raft (Why etcd)), renewed every few seconds (article 22). When a dispatcher dies, its leases expire and the remaining dispatchers take over its partitions. Adding dispatchers rebalances partitions.

Leases alone are not enough: a paused dispatcher may wake up and keep dispatching after its lease has moved (article 22's process-pause problem, which Martin Kleppmann's analysis of distributed locking illustrates with long garbage-collection pauses (How to do distributed locking)). So the claim itself must be safe, as below. With safe claims, two dispatchers briefly touching the same partition is harmless.

A run is claimed with a conditional update against what the dispatcher read:

UPDATE jobsSET version = version + 1,    next_run_at = :next_run_or_null      -- for recurring jobs, the next occurrence, in the same updateWHERE job_id = :id AND version = :version_read AND status = 'active';-- 1 row: this dispatcher owns this run. 0 rows: someone else claimed it, or it was cancelled or changed.

In the same transaction the dispatcher inserts the runs row (job_id, scheduled_at) and an outbox row for the queue message (article 21), so a claimed run is always enqueued exactly once by the relay, and an unclaimed one never is.

A subtle trap is worth knowing. If you claim by status alone ("set PENDING to DISPATCHED") and a recurring job's reschedule sets its status back to PENDING, a second dispatcher holding a stale read can claim the same run again. Claiming against the version read (or against the specific scheduled_at) fixes it. This Java program simulates two dispatchers that scan the same due jobs before either claims, with version-based claims; a recurring job reschedules itself in the same step.

// File: SchedulerDemo.javaimport java.util.ArrayList;import java.util.List;import java.util.TreeMap;public class SchedulerDemo {    static class Job {        final String id;        final long everySeconds;     // 0 = one-time job        long nextRunAt;              // seconds since the start of the simulation        long version = 0;            // incremented on every successful claim        Job(String id, long nextRunAt, long everySeconds) {            this.id = id;            this.nextRunAt = nextRunAt;            this.everySeconds = everySeconds;        }    }    // What a dispatcher read when it scanned the due jobs.    record Seen(Job job, long version, long runAt) {}    // Due index: run time -> jobs due then (a stand-in for a time-bucketed table or a sorted set).    static final TreeMap<Long, List<Job>> due = new TreeMap<>();    static final List<String> queue = new ArrayList<>();    static void schedule(Job job) {        due.computeIfAbsent(job.nextRunAt, t -> new ArrayList<>()).add(job);    }    // Mimics: UPDATE jobs SET version = version + 1, next_run_at = ? WHERE id = ? AND version = ?    static boolean claim(Seen seen) {        Job job = seen.job();        if (job.version != seen.version()) {            return false;                                // someone else claimed this run first        }        job.version++;        queue.add(job.id + "@" + seen.runAt());          // enqueue the run for the workers        if (job.everySeconds > 0) {                      // recurring: the next run is scheduled            job.nextRunAt = seen.runAt() + job.everySeconds;  // in the same atomic update            schedule(job);        }        return true;    }    static void poll(String dispatcher, long now, List<Seen> scanned) {        for (Seen seen : scanned) {            String result = claim(seen) ? "dispatched" : "lost the race for";            System.out.println("t=" + now + " " + dispatcher + " " + result + " " + seen.job().id                    + " (run at " + seen.runAt() + ")");        }    }    public static void main(String[] args) {        schedule(new Job("send-reminder-77", 10, 0));        schedule(new Job("hourly-report", 10, 60));        for (long now = 0; now <= 75; now += 5) {            List<Job> dueNow = new ArrayList<>();            while (!due.isEmpty() && due.firstKey() <= now) {                dueNow.addAll(due.pollFirstEntry().getValue());            }            // Both dispatchers scan the same due jobs before either claims anything.            List<Seen> scanA = new ArrayList<>();            List<Seen> scanB = new ArrayList<>();            for (Job job : dueNow) {                scanA.add(new Seen(job, job.version, job.nextRunAt));                scanB.add(new Seen(job, job.version, job.nextRunAt));            }            poll("dispatcher-A", now, scanA);            poll("dispatcher-B", now, scanB);        }        System.out.println("queue: " + queue);    }}

t=10 dispatcher-A dispatched send-reminder-77 (run at 10)t=10 dispatcher-A dispatched hourly-report (run at 10)t=10 dispatcher-B lost the race for send-reminder-77 (run at 10)t=10 dispatcher-B lost the race for hourly-report (run at 10)t=70 dispatcher-A dispatched hourly-report (run at 70)t=70 dispatcher-B lost the race for hourly-report (run at 70)queue: [send-reminder-77@10, hourly-report@10, hourly-report@70]

Each run appears in the queue exactly once, even though both dispatchers saw every due job.

• Each dispatcher preloads the next minute's due runs for its partitions into memory (a priority queue or timing wheel keyed by run time), then fires each one when its time arrives and claims it then. The database is read once a minute per partition, not once a second.
• New jobs scheduled for "within the next minute" are also pushed to the owning dispatcher (or picked up by a short poll of the current bucket), so they are not missed.
• Use the database's clock (or one time source) for "now" in claims, not each dispatcher's clock (article 18).

620,000 runs due in one second is a lot to claim, enqueue and execute at once, and the tenants' downstream systems may not cope either.

• Spread dispatch over a window: allow jobs to declare a tolerance ("between 00:00 and 00:05"), and use deterministic jitter (based on job id) within it. Many "daily at midnight" jobs do not care about the exact second.
• Batch claims and enqueues (claim hundreds of runs in one statement per partition).
• Rate-limit per tenant at the queue or HTTP-caller stage, so one tenant's 300,000 midnight jobs do not starve others.
• Scale dispatchers before known peaks (the top of every hour is predictable).

• Queue targets: the run message carries job_id, scheduled_at and payload. Tenant workers process it as competing consumers (Competing Consumers pattern), extend the message's visibility while working on long jobs (SQS describes extending the visibility timeout for exactly this (visibility timeout)), and acknowledge on success.
• HTTP targets: scheduler-owned callers send a signed request with an idempotency header (job_id:scheduled_at), a timeout, and retries with backoff and jitter per the job's policy; after the last attempt, the run goes to a dead-letter store and the tenant is alerted.
• Idempotent handlers: at-least-once is the contract. A handler records (job_id, scheduled_at) as processed (a unique key) in the same transaction as its effect (article 20). The digest example from article 22 does exactly this.

• Next run is computed when a run is claimed, from the cron expression in the job's time zone, then converted to UTC for the due index.
• Daylight saving: in zones with daylight saving, some local times happen twice (clocks go back) and some never happen (clocks go forward). Define the rule (for example: a skipped time runs at the next valid instant; a repeated time runs once, at its first occurrence) and test it. India does not use daylight saving, but a global scheduler must handle zones that do.
• Misfires: if dispatch was down for 2 hours, a job that runs every 5 minutes missed 24 runs. Policy per job: run_once (run one catch-up run now), run_all (run each missed run, carefully paced), or skip (wait for the next scheduled time). Catch-up after an outage must be rate-limited, or recovery itself becomes a spike.

Cancellation sets status = 'cancelled' and increments the version; a dispatcher's claim then fails (the version changed). A run already dispatched may still execute; handlers that care should check the job's status (or a cancellation token) before doing irreversible work. Say this race out loud in an interview.

• Each run is claimed at most once (atomic conditional update), and every claimed run is enqueued (outbox). Execution is at least once (redelivery after crashes), and handlers make effects effectively once.
• Ordering between runs of the same recurring job is not guaranteed if a previous run is slow; jobs that must not overlap need a per-job concurrency limit (skip or delay a run while the previous one is still active).
• Cancellation is eventually effective for runs already dispatched.
• Run history is eventually consistent with execution (workers report completion).

• Tenant isolation: each tenant's jobs, queues and history are visible only to that tenant; per-tenant quotas on jobs and dispatch rate.
• HTTP targets are SSRF risks (article 28): a tenant could register http://169.254.169.254/... or internal addresses. Allow only registered, verified destinations (or an allow-list of domains per tenant), resolve and block private ranges, and run callers in an isolated network segment.
• Sign callbacks so tenants can verify requests came from the scheduler; include timestamps to prevent replay.
• Payloads may contain sensitive data: encrypt at rest, limit size, avoid logging them.
• Audit job creation, changes and cancellations.

"The scheduler accepts one-time and recurring jobs and triggers each run within a few seconds of its time, at least once. That's about 1,700 runs a second on average, but up to hundreds of thousands due in the same second at the top of each hour.

Jobs live in a partitioned store with a due index bucketed by minute and partitioned by a hash of the job id. Dispatchers own partitions through leases in etcd; each preloads the next minute's due runs and fires them from memory. A run is claimed with a conditional update on the job's version, which also computes the next occurrence for recurring jobs in the job's time zone, and in the same transaction writes the run record and an outbox message, so a run is never lost and never dispatched twice, even if a paused dispatcher wakes up with a stale lease.

The relay puts runs on tenant queues or an HTTP-callback queue. Tenants' workers execute idempotently, keyed by job id and scheduled time; HTTP callers sign requests, retry with backoff and dead-letter failures, and only call verified destinations to avoid SSRF. Round-time spikes are smoothed with allowed windows and per-tenant rate limits, and outages are recovered with per-job misfire policies and paced catch-up."

Dispatch work: 20,000,000 x 0.2 ms = 4,000 s of dispatcher timeAcross 10 dispatchers in parallel: about 400 s, roughly 7 minutesExecution: 20,000,000 / 50,000 per second = 400 s, also about 7 minutes (overlapping with dispatch)

• Next: Design a Ticket Booking System Without Overselling, which uses this scheduler to expire seat holds.
• Related: article 22 for leases and fencing, article 20 for idempotent handlers.