03Building blocksSearch systems
03 · Building block
Search systems
Design ingestion, inverted indexes, ranking, sharding, and freshness for high-volume full-text retrieval.Architecture map
See where the component sits in a real system.
Write
Writers + searchers enters through Search API. Document service owns validation and commits the durable record to Source database.
Propagate
CDC / indexing log separates the committed write from background work. Indexer fleet can retry safely while it builds Index shard replicas.
Read
Query coordinator serves from Index shard replicas, then checks authoritative state whenever freshness, policy, or correctness requires it. It also consults Ranking service as an explicit dependency.
Say this first: The source database owns documents; a version-aware indexing pipeline builds distributed searchable projections.
Open the full whiteboard ↗Explain every boundary before adding more boxes.
The source database owns documents; a version-aware indexing pipeline builds distributed searchable projections.
Design ingestion, inverted indexes, ranking, sharding, and freshness for high-volume full-text retrieval.
Corpus · indexing rate · query QPS · top-K · freshness · deletion deadline. State average and peak load, stored bytes, bandwidth or open connections, and the growth horizon before choosing a partitioning strategy.
End-to-end walkthrough
Trace the architecture in this order.
- 01
Enter and classify the request
Writers + searchers → Search APIDocument updates + queries enters over HTTPS / RPC. Search API handles identity, admission, routing, and request context; it deliberately does not own domain truth.
- 02
Validate, then cross the commit boundary
Search API → Document service → Source databaseDocument service receives the command, checks invariants and retry identity, then uses source write to update Source database. The user-visible mutation is accepted only after this boundary succeeds.
- 03
Move replayable work off the request path
Document service → CDC / indexing log → Indexer fleet → Index shard replicasDocument service emits publish after commit; Indexer fleet uses consume and bulk index to build Index shard replicas. Consumers must tolerate duplicate delivery and stale retries because this path is asynchronous.
- 04
Serve reads from the right authority
Search API → Query coordinator → Index shard replicas / Source databaseQuery coordinator uses shard top-K for the common, read-optimized path and strong read when correctness or repair requires authoritative state. The API must state the freshness promise instead of hiding it.
- 05
Contain the dependency boundary
Query coordinator → Ranking servicererank candidates crosses into Ranking service. Treat timeouts as ambiguous, use a deadline and idempotent retry or reconciliation, and keep the core state recoverable when the dependency is unavailable.
Ownership ledger
Why each box exists—and what it must defend.
| Component | Owns | Why it exists | Interviewer probe |
|---|---|---|---|
| Search APIAuth + query normalization | Identity, admission, routing | Protects the system edge and attaches trusted context before domain work begins. | Timeout budgets, quotas, regional routing |
| Document serviceCommit source document | Write invariants and retry identity | Serializes or conditionally applies state changes before acknowledging success. | Concurrent writes, deduplication, hot ownership |
| Source databaseCanonical documents | Authoritative durable state | Provides the one record used to resolve disputes, recover, and rebuild projections. | Partition key, replication, consistency |
| CDC / indexing logDocument version changes | Durable asynchronous handoff | Absorbs bursts and lets slow or optional work retry independently of the request. | Ordering key, lag, retention, dead letters |
| Indexer fleetAnalyze + build segments | Replayable processing | Runs expensive, fan-out, or side-effecting work with leases and bounded retries. | Idempotency, poison work, autoscaling |
| Index shard replicasPostings + doc values | Rebuildable query state | Shapes data for the dominant reads without weakening the write-side invariant. | Freshness, versioning, rebuild time |
| Query coordinatorScatter, merge, rank | Read composition and freshness policy | Chooses authoritative or derived state and returns a stable client contract. | Fan-out, cache policy, partial results |
| Ranking serviceFeatures + reranking | External capability, not local truth | Keeps a specialized or third-party concern behind a replaceable contract. | Ambiguous timeout, circuit breaking, fallback |
Physical design
Name the database, shard key, indexes, and guarantees.
- Database + storage
- OpenSearch/Elasticsearch/Lucene shards are derived read storage; SQL/document DB remains truth; Kafka/CDC sends versioned updates.
- Partitioning / sharding
- Hash or route documents by tenant/category/locality. Queries fan out only to relevant shards; replicas add read QPS.
- Indexes
- Inverted postings/positions, doc values for sort/facets, geo/BKD structures, and optional vector indexes.
- Replication + consistency
- Search is eventual and monotonic by source_version. Read-after-write may query truth or wait for an indexing token.
- Cache, queue + recovery
- Bulk-index idempotently, expose lag, dead-letter invalid documents, cache normalized frequent queries by index generation.
- Capacity math
- Estimate documents, indexed bytes, updates/sec, query QPS, shard working set, fan-out, top-K, and freshness SLA.
- Alternative rejected
- SQL full-text is a strong start; corpus size, relevance features, and independent read scale earn a distributed index.
Deep-dive candidates
Pick one risk and explain the mechanism, alternative, and cost.
Inverted index
Map analyzed terms to document IDs so retrieval touches matching postings instead of scanning every record.
Tie the mechanism back to Source database, Index shard replicas, and the stated corpus · indexing rate · query qps · top-k · freshness · deletion deadline envelope.Analysis pipeline
Normalize case, tokenize, stem or lemmatize, handle stop words, and preserve fields needed for exact filters.
Tie the mechanism back to Source database, Index shard replicas, and the stated corpus · indexing rate · query qps · top-k · freshness · deletion deadline envelope.Query path
Parse the query, apply filters, retrieve candidates across shards, score with BM25 or custom features, then merge top results.
Tie the mechanism back to Source database, Index shard replicas, and the stated corpus · indexing rate · query qps · top-k · freshness · deletion deadline envelope.Failure pressure test
Show detection, containment, recovery, and evidence.
The topic-specific correctness risk
Out-of-order updates can resurrect deleted or unauthorized documents.
Track failed promises at Source database and Index shard replicas.CDC / indexing log or Indexer fleet falls behind
Bound admission, scale on oldest-work age, retry with jitter, and isolate poison work before lag becomes unbounded.
Oldest event age · retry rate · dead-letter volume · projection freshnessSource database is slow or unavailable
Apply a deadline, preserve retry identity, fail over only within the stated consistency model, and reconcile any ambiguous result.
Commit p99 · timeout rate · replication lag · recovery time- Functional requirements and non-goals
- Peak traffic, storage, bandwidth, and growth
- Entities, APIs, idempotency, and pagination
- Source of truth and consistency promise
- Partition key, replicas, caches, and hot spots
- Retries, backpressure, failover, and reconciliation
- Latency, saturation, correctness, and recovery metrics
- Security, migration, cost, and multi-region evolution
Read the solid request path first, stop at the source of truth, then follow the dashed event path into workers and rebuildable read models. Every arrow names a contract you should be ready to defend.
- 01
Source of truth — Commits records Define the output contract before moving to the next owner.
- 02
Change stream — Emits versions Define the output contract before moving to the next owner.
- 03
Indexer — Normalizes and tokenizes Define the output contract before moving to the next owner.
- 04
Shard set — Stores inverted lists Define the output contract before moving to the next owner.
- 05
Query service — Retrieves candidates Define the output contract before moving to the next owner.
- 06
Ranker — Scores and merges Confirm the result and emit the evidence needed to reconcile it.
Lesson spine
What you need to understand.
A search system builds a read-optimized, relevance-aware projection of canonical data.
Inverted index
Map analyzed terms to document IDs so retrieval touches matching postings instead of scanning every record.
Analysis pipeline
Normalize case, tokenize, stem or lemmatize, handle stop words, and preserve fields needed for exact filters.
Query path
Parse the query, apply filters, retrieve candidates across shards, score with BM25 or custom features, then merge top results.
Freshness
Use an outbox, change-data capture, or event stream to version index updates; never let an older retry resurrect deleted content.
Features
Fuzzy matching, autocomplete, facets, synonyms, vector retrieval, and business ranking each add different cost and evaluation needs.
Quality and operations
Measure relevance offline and online, index lag, shard skew, query tail latency, and failed document versions.
Before the boxes
Frame the decision.
What must work
Design ingestion, inverted indexes, ranking, sharding, and freshness for high-volume full-text retrieval.
What changes the design
Corpus · indexing rate · query QPS · top-K · freshness · deletion deadline
What owns the truth
Identify the component that commits authoritative state, then separate synchronous confirmation from derived work.
What stays simple
Do not add global coordination, multi-region writes, or a specialized store until a requirement earns the complexity.
Decision table
Make the trade-offs explicit.
| Decision | Defensible position | Cost to acknowledge |
|---|---|---|
| Primary mechanism | Asynchronous indexing protects writes; synchronous indexing improves read-after-write visibility. | The stronger guarantee usually adds coordination, latency, state, or operational work. |
| Sync vs. async | Keep only correctness-critical confirmation synchronous. Move derived views, notifications, analytics, and cleanup behind a durable boundary. | Async work needs idempotency, lag monitoring, replay, and a product definition for partial completion. |
| Simple vs. scaled | Begin with one logical owner and a clear API. Partition or replicate only the resource proven to be the first bottleneck. | Migration requires stable identities, versioned contracts, backfill, and a rollback path. |
Failure review
Design the recovery path.
Topic-specific risk
Out-of-order updates can resurrect deleted or unauthorized documents.
ResponsePersist enough identity and state to distinguish retry, resume, compensation, and operator repair.
Dependency timeout
A timeout is ambiguous: the remote side may have failed, succeeded, or still be running.
ResponseUse deadlines, bounded backoff with jitter, idempotency keys, and a status or reconciliation path.
Overload or skew
Average capacity can look healthy while a tenant, key, partition, region, or expensive request saturates one owner.
ResponseExpose queue depth and hot-key share, apply backpressure, isolate tenants, and degrade optional work before correctness.
Evidence + level bar
Prove the design can be operated.
Health of the promise
Measure user-visible latency or freshness, correctness drift, saturation, retry volume, and time to recover. Alert on the failed promise—not only CPU.
Complete and clear
Finish the happy path, identify the state owner, choose reasonable building blocks, and explain one scale mechanism.
Trade-offs and failure
Separate read and write paths, define consistency, explain partitioning, and make duplicate or partial failure safe.
Evolution and operations
Discuss multi-region boundaries, migration, tenant isolation, capacity, observability, and how the architecture changes over time.
Interview language
Open the deep dive with a claim.
“For Search systems, the decision I want to make explicit is this: Asynchronous indexing protects writes; synchronous indexing improves read-after-write visibility. I’ll trace the state-changing path first, show where the result becomes durable, then test the design against the highest-risk failure and our target scale.”
08 · Retrieval check
Can you defend it without the page?
- For Search systems, where is the correctness boundary and which failure would you test first?
- Which component owns committed truth, and what event or response proves the commit?
- Where is the first scaling or coordination bottleneck under the stated envelope?
- What happens after an ambiguous timeout or duplicate operation?
- Which complexity would you remove at one hundredth of the scale?