04QuestionsJob scheduler

04 · Worked prompt

Job scheduler

Design durable schedules, leases, worker heartbeats, retries, priorities, and exactly-once effects.

35 minInterview blueprint

INTERVIEW RUBRIC

What a passing answer must show

100 points · 45 minutes

20pts
Scope the problem
0–5 min
Prioritize the core flows, state the scale, and name the non-goals.
15pts
Define contracts
5–10 min
Identify durable entities, APIs, idempotency, and the source of truth.
30pts
Complete the diagram
10–25 min
Trace one write path and one read path. Label the commit boundary and async work.
20pts
Lead one deep dive
25–38 min
Choose the highest-risk trade-off and explain the mechanism, alternative, and cost.
15pts
Prove reliability
38–45 min
Walk a failure, recovery, metric, bottleneck, and evolution path.

DRAW THIS FIRST

One complete box-and-arrow design

Design durable schedules, leases, worker heartbeats, retries, priorities, and exactly-once effects.

Job scheduler system architecture. Durable occurrences and fenced leases separate scheduling precision from effect uniqueness. Request path: Job owners to Scheduler API to Schedule service to Schedule/job DB. Asynchronous path: Priority ready queues to Worker fleet. Read path: Status service to Attempt history. External dependency: Target systems. — Job scheduler · system architecture

DEFEND THE DIAGRAM

Explain every boundary before adding more boxes.

Durable occurrences and fenced leases separate scheduling precision from effect uniqueness.

INTERVIEW CONTRACT

Design durable schedules, leases, worker heartbeats, retries, priorities, and exactly-once effects.

CAPACITY QUESTIONS TO QUANTIFY

Millions timers · second precision · variable runtimes · tenant priorities. State average and peak load, stored bytes, bandwidth or open connections, and the growth horizon before choosing a partitioning strategy.

End-to-end walkthrough

Trace the architecture in this order.

01
Enter and classify the request
Job owners → Scheduler API
Schedule, pause, inspect enters over HTTPS / RPC. Scheduler API handles identity, admission, routing, and request context; it deliberately does not own domain truth.
02
Validate, then cross the commit boundary
Scheduler API → Schedule service → Schedule/job DB
Schedule service receives the command, checks invariants and retry identity, then uses create occurrence to update Schedule/job DB. The user-visible mutation is accepted only after this boundary succeeds.
03
Move replayable work off the request path
Schedule service → Priority ready queues → Worker fleet → Attempt history
Schedule service emits publish after commit; Worker fleet uses consume and record attempt to build Attempt history. Consumers must tolerate duplicate delivery and stale retries because this path is asynchronous.
04
Serve reads from the right authority
Scheduler API → Status service → Attempt history / Schedule/job DB
Status service uses optimized read for the common, read-optimized path and strong read when correctness or repair requires authoritative state. The API must state the freshness promise instead of hiding it.
05
Contain the dependency boundary
Worker fleet → Target systems
execute effect crosses into Target systems. Treat timeouts as ambiguous, use a deadline and idempotent retry or reconciliation, and keep the core state recoverable when the dependency is unavailable.

Ownership ledger

Why each box exists—and what it must defend.

Component	Owns	Why it exists	Interviewer probe
Scheduler APIAuth + tenant quotas	Identity, admission, routing	Protects the system edge and attaches trusted context before domain work begins.	Timeout budgets, quotas, regional routing
Schedule servicePersist next occurrence	Write invariants and retry identity	Serializes or conditionally applies state changes before acknowledging success.	Concurrent writes, deduplication, hot ownership
Schedule/job DBOccurrences + lease epochs	Authoritative durable state	Provides the one record used to resolve disputes, recover, and rebuild projections.	Partition key, replication, consistency
Priority ready queuesDue jobs by class	Durable asynchronous handoff	Absorbs bursts and lets slow or optional work retry independently of the request.	Ordering key, lag, retention, dead letters
Worker fleetClaim, heartbeat, execute	Replayable processing	Runs expensive, fan-out, or side-effecting work with leases and bounded retries.	Idempotency, poison work, autoscaling
Attempt historyProgress + terminal result	Rebuildable query state	Shapes data for the dominant reads without weakening the write-side invariant.	Freshness, versioning, rebuild time
Status serviceSchedules + attempts	Read composition and freshness policy	Chooses authoritative or derived state and returns a stable client contract.	Fan-out, cache policy, partial results
Target systemsIdempotent job effects	External capability, not local truth	Keeps a specialized or third-party concern behind a replaceable contract.	Ambiguous timeout, circuit breaking, fallback

Physical design

Name the database, shard key, indexes, and guarantees.

Database + storage: PostgreSQL/distributed SQL stores schedules, occurrences, leases, and attempts; a durable priority queue stores runnable work.
Partitioning / sharding: Hash schedules by tenant/schedule ID and bucket due work by time + shard so pollers do not contend on one global index.
Indexes: Due (bucket, next_run, shard, state), unique (schedule_id, occurrence_time), attempts by occurrence/state, and lease expiry.
Replication + consistency: Claims are strong and fenced. Execution is at least once; exactly-once effects require an idempotency key at the destination.
Cache, queue + recovery: A transactional outbox releases occurrences; visibility leases, heartbeat, jittered retry, dead letters, and reconciliation handle ambiguity.
Capacity math: Estimate schedules, timers/sec, precision, top-of-hour burst, duration distribution, and oldest-ready age.
Alternative rejected: One cron heap works initially but lacks failover and burst isolation; persisted next occurrences plus sharded pollers evolve safely.

Deep-dive candidates

Pick one risk and explain the mechanism, alternative, and cost.

Timer index

Bucket by due time and shard schedule ownership; maintain a short lookahead queue

A full-table poll is not a scaling plan

Lease fencing

Increment a token on every claim and reject stale completion

Expiration alone cannot stop a zombie worker

Recurring policy

Define skip, coalesce, or catch-up behavior for missed runs

Cron syntax does not define business semantics

Failure pressure test

Show detection, containment, recovery, and evidence.

Dispatcher restart

Rescan overlap and rely on unique occurrence identity

duplicate occurrence attempts

Worker partition

Expire lease and fence late completion

stale token rejections

Queue overload

Apply tenant fairness and delay low priority work

schedule delay by priority

Before you finish, explicitly cover

Functional requirements and non-goals
Peak traffic, storage, bandwidth, and growth
Entities, APIs, idempotency, and pagination
Source of truth and consistency promise
Partition key, replicas, caches, and hot spots
Retries, backpressure, failover, and reconciliation
Latency, saturation, correctness, and recovery metrics
Security, migration, cost, and multi-region evolution

SAY THIS WHILE YOU DRAW

A four-part talk track

Scope
“I’ll prioritize create one-time and recurring schedules and dispatch near the requested time.”
Scale
“The design changes around millions timers · second precision · variable runtimes · tenant priorities.”
Decision
“Indexed polling is simple; delay queues or time wheels scale huge timer populations.”
Risk
“The first failure I want to pressure-test is: Clock skew and lease races can run one job more than once.”

Reference details

Open these only after you can explain the diagram above without reading.

01Requirements and state lifecycle4 requirements

Create one-time and recurring schedules
Dispatch near the requested time
Support priorities, retries, cancellation, and history
Recover from dispatcher and worker failure

Job scheduler · state lifecycle

state lifecycle

Each transition must be durable, observable, and safe to retry.

scheduled

ready

leased

succeeded or failed

dead-lettered

02Data model and APIs4 entities · 3 interfaces

Core entities

Scheduleschedule_id, rule, next_run_at, policy, versionOwner: Schedule store

Jobjob_id, schedule_id, due_at, priority, stateOwner: Dispatcher

Attemptjob_id, attempt, lease_token, deadlineOwner: Worker system

ExecutionResultjob_id, outcome, effect_keyOwner: Result store

External interfaces

POST /v1/schedules

Create an idempotent schedule

PATCH /v1/schedules/{id}

Version-check pause, resume, or rule change

POST /internal/jobs/{id}/complete

Commit a result under the current lease token

03Deep dives and trade-offsChoose one

Timer index

Bucket by due time and shard schedule ownership; maintain a short lookahead queue

A full-table poll is not a scaling plan

Lease fencing

Increment a token on every claim and reject stale completion

Expiration alone cannot stop a zombie worker

Recurring policy

Define skip, coalesce, or catch-up behavior for missed runs

Cron syntax does not define business semantics

04Failures, recovery, and evidence3 scenarios

Dispatcher restart

Rescan overlap and rely on unique occurrence identity

duplicate occurrence attempts

Worker partition

Expire lease and fence late completion

stale token rejections

Queue overload

Apply tenant fairness and delay low priority work

schedule delay by priority

05What makes the answer seniorInterviewer signals

Exactly-once execution is unrealistic; exactly-once effect is the goal
Always define missed-run behavior
The lease token belongs in the completion transaction
Primary trade-off: Indexed polling is simple; delay queues or time wheels scale huge timer populations.

BEFORE THE NEXT QUESTION

Can you redraw it from memory?

Name the source of truth.
Trace the write and read paths.
Defend one trade-off.
Recover from one failure.

Scope the problem

Define contracts

Complete the diagram

Lead one deep dive

Prove reliability

Write

Propagate

Read

Trace the architecture in this order.

Enter and classify the request

Validate, then cross the commit boundary

Move replayable work off the request path

Serve reads from the right authority

Contain the dependency boundary

Why each box exists—and what it must defend.

Name the database, shard key, indexes, and guarantees.

Pick one risk and explain the mechanism, alternative, and cost.

Timer index

Lease fencing

Recurring policy

Show detection, containment, recovery, and evidence.

Dispatcher restart

Worker partition

Queue overload

Reference details

Core entities

External interfaces

Timer index

Lease fencing

Recurring policy

Dispatcher restart

Worker partition

Queue overload

Can you redraw it from memory?