03Building blocksRate limiters
03 · Building block
Rate limiters
Compare token bucket, leaky bucket, fixed window, and sliding window strategies for distributed enforcement.Architecture map
See where the component sits in a real system.
Write
API clients enters through Edge gateway. Limiter owns validation and commits the durable record to Policy store.
Propagate
Usage event stream separates the committed write from background work. Quota aggregators can retry safely while it builds Atomic counter store.
Read
Policy evaluator serves from Atomic counter store, then checks authoritative state whenever freshness, policy, or correctness requires it. It also consults Protected service as an explicit dependency.
Say this first: A written policy plus atomic counters decides admission before expensive work, with explicit fail-open/closed behavior.
Open the full whiteboard ↗Explain every boundary before adding more boxes.
A written policy plus atomic counters decides admission before expensive work, with explicit fail-open/closed behavior.
Compare token bucket, leaky bucket, fixed window, and sliding window strategies for distributed enforcement.
Limit dimensions · refill · burst · active keys · decision latency · overshoot. State average and peak load, stored bytes, bandwidth or open connections, and the growth horizon before choosing a partitioning strategy.
End-to-end walkthrough
Trace the architecture in this order.
- 01
Enter and classify the request
API clients → Edge gatewayTenant/user/IP traffic enters over HTTPS / RPC. Edge gateway handles identity, admission, routing, and request context; it deliberately does not own domain truth.
- 02
Validate, then cross the commit boundary
Edge gateway → Limiter → Policy storeLimiter receives the check + consume, checks invariants and retry identity, then uses commit to update Policy store. The user-visible mutation is accepted only after this boundary succeeds.
- 03
Move replayable work off the request path
Limiter → Usage event stream → Quota aggregators → Atomic counter storeLimiter emits publish after commit; Quota aggregators uses consume and project / update to build Atomic counter store. Consumers must tolerate duplicate delivery and stale retries because this path is asynchronous.
- 04
Serve reads from the right authority
Edge gateway → Policy evaluator → Atomic counter store / Policy storePolicy evaluator uses atomic decrement for the common, read-optimized path and strong read when correctness or repair requires authoritative state. The API must state the freshness promise instead of hiding it.
- 05
Contain the dependency boundary
Limiter → Protected serviceforward / 429 crosses into Protected service. Treat timeouts as ambiguous, use a deadline and idempotent retry or reconciliation, and keep the core state recoverable when the dependency is unavailable.
Ownership ledger
Why each box exists—and what it must defend.
| Component | Owns | Why it exists | Interviewer probe |
|---|---|---|---|
| Edge gatewayIdentity + route context | Identity, admission, routing | Protects the system edge and attaches trusted context before domain work begins. | Timeout budgets, quotas, regional routing |
| LimiterToken/window decision | Write invariants and retry identity | Serializes or conditionally applies state changes before acknowledging success. | Concurrent writes, deduplication, hot ownership |
| Policy storeLimits + versions | Authoritative durable state | Provides the one record used to resolve disputes, recover, and rebuild projections. | Partition key, replication, consistency |
| Usage event streamAccepted/rejected samples | Durable asynchronous handoff | Absorbs bursts and lets slow or optional work retry independently of the request. | Ordering key, lag, retention, dead letters |
| Quota aggregatorsRollups + abuse signals | Replayable processing | Runs expensive, fan-out, or side-effecting work with leases and bounded retries. | Idempotency, poison work, autoscaling |
| Atomic counter storeTokens/windows + TTL | Rebuildable query state | Shapes data for the dominant reads without weakening the write-side invariant. | Freshness, versioning, rebuild time |
| Policy evaluatorScope + cost + exemption | Read composition and freshness policy | Chooses authoritative or derived state and returns a stable client contract. | Fan-out, cache policy, partial results |
| Protected serviceAdmitted requests only | External capability, not local truth | Keeps a specialized or third-party concern behind a replaceable contract. | Ambiguous timeout, circuit breaking, fallback |
Physical design
Name the database, shard key, indexes, and guarantees.
- Database + storage
- Redis Cluster with Lua/transactions performs atomic token/window updates; SQL/config service stores policies; local buckets provide emergency capacity.
- Partitioning / sharding
- Hash normalized tenant/user/IP/endpoint key. Strict global limits use one owner; regional sub-budgets allow bounded overshoot.
- Indexes
- Direct counter key and TTL; policy by scope/version. Avoid scans or joins in the request path.
- Replication + consistency
- Atomic decrement/refill per key. Fail-open/closed and global overshoot depend on abuse, security, or financial risk.
- Cache, queue + recovery
- Gateways cache policies; usage samples stream asynchronously. The synchronous decision has a tight timeout and explicit fallback.
- Capacity math
- Estimate RPS, distinct keys/sec, hottest key, counter memory, burst, decision p99, and failover overshoot.
- Alternative rejected
- Fixed window is cheapest but bursts at boundaries; token bucket is the default for bounded burst plus average rate.
Deep-dive candidates
Pick one risk and explain the mechanism, alternative, and cost.
Token bucket
Tokens refill at a steady rate up to a capacity, allowing bounded bursts while enforcing a long-term average.
Tie the mechanism back to Policy store, Atomic counter store, and the stated limit dimensions · refill · burst · active keys · decision latency · overshoot envelope.Leaky bucket
Requests enter a bounded queue and leave at a uniform rate, smoothing output but adding wait and overflow behavior.
Tie the mechanism back to Policy store, Atomic counter store, and the stated limit dimensions · refill · burst · active keys · decision latency · overshoot envelope.Window counters
Fixed windows are cheap but allow boundary bursts; sliding logs are exact but expensive; weighted counters approximate a rolling window efficiently.
Tie the mechanism back to Policy store, Atomic counter store, and the stated limit dimensions · refill · burst · active keys · decision latency · overshoot envelope.Failure pressure test
Show detection, containment, recovery, and evidence.
The topic-specific correctness risk
Central counters bottleneck; local counters overshoot during partitions.
Track failed promises at Policy store and Atomic counter store.Usage event stream or Quota aggregators falls behind
Bound admission, scale on oldest-work age, retry with jitter, and isolate poison work before lag becomes unbounded.
Oldest event age · retry rate · dead-letter volume · projection freshnessPolicy store is slow or unavailable
Apply a deadline, preserve retry identity, fail over only within the stated consistency model, and reconcile any ambiguous result.
Commit p99 · timeout rate · replication lag · recovery time- Functional requirements and non-goals
- Peak traffic, storage, bandwidth, and growth
- Entities, APIs, idempotency, and pagination
- Source of truth and consistency promise
- Partition key, replicas, caches, and hot spots
- Retries, backpressure, failover, and reconciliation
- Latency, saturation, correctness, and recovery metrics
- Security, migration, cost, and multi-region evolution
Read the solid request path first, stop at the source of truth, then follow the dashed event path into workers and rebuildable read models. Every arrow names a contract you should be ready to defend.
- 01
Request — Carries identity and route Define the output contract before moving to the next owner.
- 02
Policy engine — Resolves quota Define the output contract before moving to the next owner.
- 03
Counter store — Updates budget atomically Define the output contract before moving to the next owner.
- 04
Decision — Allows or rejects Define the output contract before moving to the next owner.
- 05
Headers — Expose reset state Define the output contract before moving to the next owner.
- 06
Telemetry — Finds abuse Confirm the result and emit the evidence needed to reconcile it.
Lesson spine
What you need to understand.
Rate limiting converts product policy and capacity protection into a fast decision on a stable identity.
Token bucket
Tokens refill at a steady rate up to a capacity, allowing bounded bursts while enforcing a long-term average.
Leaky bucket
Requests enter a bounded queue and leave at a uniform rate, smoothing output but adding wait and overflow behavior.
Window counters
Fixed windows are cheap but allow boundary bursts; sliding logs are exact but expensive; weighted counters approximate a rolling window efficiently.
Distributed enforcement
Use an atomic store or partitioned counter owner. Regional budgets reduce coordination at the cost of bounded global overshoot.
Identity and policy
Limit by user, tenant, API key, IP, endpoint, cost unit, or a hierarchy of them. Return quota and reset guidance.
Failure mode
Fail open for low-risk availability, fail closed for financial or security limits, or reserve local emergency budgets.
Before the boxes
Frame the decision.
What must work
Compare token bucket, leaky bucket, fixed window, and sliding window strategies for distributed enforcement.
What changes the design
Limit dimensions · refill · burst · active keys · decision latency · overshoot
What owns the truth
Identify the component that commits authoritative state, then separate synchronous confirmation from derived work.
What stays simple
Do not add global coordination, multi-region writes, or a specialized store until a requirement earns the complexity.
Decision table
Make the trade-offs explicit.
| Decision | Defensible position | Cost to acknowledge |
|---|---|---|
| Primary mechanism | Token buckets allow controlled bursts; sliding windows improve fairness with more state. | The stronger guarantee usually adds coordination, latency, state, or operational work. |
| Sync vs. async | Keep only correctness-critical confirmation synchronous. Move derived views, notifications, analytics, and cleanup behind a durable boundary. | Async work needs idempotency, lag monitoring, replay, and a product definition for partial completion. |
| Simple vs. scaled | Begin with one logical owner and a clear API. Partition or replicate only the resource proven to be the first bottleneck. | Migration requires stable identities, versioned contracts, backfill, and a rollback path. |
Failure review
Design the recovery path.
Topic-specific risk
Central counters bottleneck; local counters overshoot during partitions.
ResponsePersist enough identity and state to distinguish retry, resume, compensation, and operator repair.
Dependency timeout
A timeout is ambiguous: the remote side may have failed, succeeded, or still be running.
ResponseUse deadlines, bounded backoff with jitter, idempotency keys, and a status or reconciliation path.
Overload or skew
Average capacity can look healthy while a tenant, key, partition, region, or expensive request saturates one owner.
ResponseExpose queue depth and hot-key share, apply backpressure, isolate tenants, and degrade optional work before correctness.
Evidence + level bar
Prove the design can be operated.
Health of the promise
Measure user-visible latency or freshness, correctness drift, saturation, retry volume, and time to recover. Alert on the failed promise—not only CPU.
Complete and clear
Finish the happy path, identify the state owner, choose reasonable building blocks, and explain one scale mechanism.
Trade-offs and failure
Separate read and write paths, define consistency, explain partitioning, and make duplicate or partial failure safe.
Evolution and operations
Discuss multi-region boundaries, migration, tenant isolation, capacity, observability, and how the architecture changes over time.
Interview language
Open the deep dive with a claim.
“For Rate limiters, the decision I want to make explicit is this: Token buckets allow controlled bursts; sliding windows improve fairness with more state. I’ll trace the state-changing path first, show where the result becomes durable, then test the design against the highest-risk failure and our target scale.”
08 · Retrieval check
Can you defend it without the page?
- For Rate limiters, where is the correctness boundary and which failure would you test first?
- Which component owns committed truth, and what event or response proves the commit?
- Where is the first scaling or coordination bottleneck under the stated envelope?
- What happens after an ambiguous timeout or duplicate operation?
- Which complexity would you remove at one hundredth of the scale?