Infrastructure System Design · Complete Revision Atlas

Connected architectures · numbered data flows · purpose and tradeoffs · monitoring walkthrough added

STUDY FRAMEWORK: For each numbered arrow, explain what data moves, why the destination exists, what state it owns, and what happens when that edge fails. Green lines in component boxes indicate benefit; orange lines indicate tradeoff. Metrics & alerting are guided study, not an independently passed mock.

01 | Bitly / URL Shortener

GUIDED x2 — independent timed mock not verified

Fast, highly available redirects with unique short codes; analytics are nonblocking.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G client Clients + Create URLs and follow short links - Client retry may duplicate creates gw API Gateway + Auth, routing, abuse controls - Extra hop; must scale globally client->gw write Shorten Service + Validate URL and request custom alias - Validation adds create latency gw->write 02 create read Redirect Service + Resolve short code; HTTP redirect - Popularity spikes cause hot keys gw->read 03 resolve id ID / Code Generator + Assign collision-safe unique code - Coordination or collision handling cost write->id map Sharded Mapping DB + Authoritative code → long URL record - Shard routing / replication complexity id->map 05 persist cache Redirect Cache + Serve popular mappings from memory - Stale mappings until TTL/invalidation read->cache 06 lookup evt Click Event Queue + Async capture; no redirect blocking - Clicks may arrive late or duplicate read->evt 07 async cache->map 08 miss ana Analytics Workers + Aggregate counts and dashboards - Metrics are eventually consistent evt->ana

Numbered arrow / data-flow reference

01 · client → gw — passes data
02 · gw → write — create
03 · gw → read — resolve
04 · write → id — passes data
05 · id → map — persist
06 · read → cache — lookup
07 · read → evt — async
08 · cache → map — miss
09 · evt → ana — passes data

End-to-end: Create: API → code allocator → mapping DB. Redirect: client → cache → DB on miss → redirect; click events flow asynchronously.

Key architectural decisions

  • Unique code allocation and alias conflict handling
  • Hot URLs served from cache; TTL/invalidation on change
  • Protect redirect availability even if analytics is delayed

Repeat / gap: Cold test: collisions under concurrent creates; expired / malicious URL handling.

02 | Distributed Rate Limiter

INDEPENDENT REPEAT — 8.0/10 (Senior Lean Hire)

Distributed token buckets with atomic consumption; policy control plane separate from request path.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G client API Clients + Send authenticated requests - Retries can raise attempted traffic gw API Gateways + Enforce allow / HTTP 429 - Adds latency to every request client->gw near Local Deny Cache + Short-term blocked-until optimization - May reject after a policy increase gw->near 02 blocked? redis Sharded Redis Primaries + Authoritative token buckets; Lua atomicity - Hot key and central dependency gw->redis 03 atomic check svc Backend Services + Handle accepted requests - Fail-open can overload backends gw->svc 04 allow obs Metrics / Alerts + 429s, Redis latency, fail-open counts - Monitoring adds cardinality/cost gw->obs rep Redis Replicas + Failover copies across AZs - Lag/failover can lose token state redis->rep 06 replicate cfg Policy DB + Authoritative limit definitions - Policy edits need safe rollout cp Config Publisher + Version events + snapshot reconciliation - Stale versions require reconciliation cfg->cp 07 change cp->gw 08 versioned rules cp->redis 09 policy version

Numbered arrow / data-flow reference

01 · client → gw — passes data
02 · gw → near — blocked?
03 · gw → redis — atomic check
04 · gw → svc — allow
05 · gw → obs — passes data
06 · redis → rep — replicate
07 · cfg → cp — change
08 · cp → gw — versioned rules
09 · cp → redis — policy version

End-to-end: Request → gateway → version-aware local deny cache → Redis Lua token-bucket check → backend or 429; policy updates propagate independently.

Key architectural decisions

  • Single atomic owner per (user, endpoint) bucket
  • Hot users: gateway-local reject cache; no local extra grants
  • Fail-open/closed based on endpoint risk; capacity for AZ failover

Repeat / gap: Repeat: bounded outage fallback, stale cached rejects after limit increase, exact refill math.

03 | Distributed Cache

FULL MOCK — 7.0/10 (Borderline Lean No Hire)

Database-authoritative cache-aside store with consistent hashing, replicas, and versioned invalidation.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G app Application / Client + GET/SET using cache client - Must handle misses and stale values router Consistent Hash Ring + Virtual nodes route each key to owner - Membership movement remaps keys app->router owner Cache Primaries + In-memory authoritative cached copies - Evictions and node memory limits router->owner 02 GET / SET rep AZ Cache Replicas + Availability via async replicas / failover - Async copies can return stale data owner->rep 03 replicate fill Miss Coalescer + One bounded DB fetch per hot missing key - Coalescing contention / retry delay owner->fill 04 miss db Source-of-Truth DB + Authoritative records and versions - Miss storms overload source of truth inv Invalidation Events + Committed DB updates publish versions - Lost event causes stale cache db->inv 05 updates tomb Versioned Tombstones + Reject late stale fills after invalidation - Tombstone retention consumes space inv->tomb 06 version tomb->owner 07 invalidate fill->db 08 bounded read mon Membership / Health + Detect failures, epochs, ownership - Failover requires fencing + spare capacity mon->router 09 ring changes

Numbered arrow / data-flow reference

01 · app → router — passes data
02 · router → owner — GET / SET
03 · owner → rep — replicate
04 · owner → fill — miss
05 · db → inv — updates
06 · inv → tomb — version
07 · tomb → owner — invalidate
08 · fill → db — bounded read
09 · mon → router — ring changes

End-to-end: Read: cache client → hash ring → owner; on miss coalesce requests → DB → conditional fill. Write: DB commit → versioned invalidation.

Key architectural decisions

  • Hot keys: salted replicated read copies with update fanout
  • TTL + jitter; sharded/approximate LRU avoids one global lock
  • Failover must protect DB against cache-miss storms

Repeat / gap: Repeat: AZ loss capacity math, stampede coalescing, LRU contention and final summary.

04 | Distributed Message Queue

FULL MOCK — 6.5/10 (Borderline Lean No Hire)

Durable partition logs with leader fencing, consumer groups, replay, and explicit delivery guarantees.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G prod Producers + Publish keyed messages; handle retry - Retries may create duplicates route Partition Router + Hash key → ordered partition - Hot key limits partition throughput prod->route leader Leader Brokers + Append to replicated partition log - Leader failure interrupts writes route->leader 02 append follower Follower Brokers + Replicate across AZs; take over on failure - Replication consumes bandwidth/storage leader->follower 03 replicate log Append-only Log + Durable ordered records per partition - Retention requires disk capacity leader->log ctrl Controller / Metadata + Elect leader; assign epoch + ownership - Controller outage stalls reassignment ctrl->leader 05 epoch / fence grp Consumer Group + Assign partitions among consumers - Rebalance pauses consumption log->grp 06 poll offset Offset Store + Checkpoint progress; retention + replay - Ack loss causes replay/duplicates grp->offset 07 commit sink Idempotent Handlers + Apply side effects safely on redelivery - Dedup state/atomicity complexity grp->sink 08 process offset->grp 09 resume

Numbered arrow / data-flow reference

01 · prod → route — passes data
02 · route → leader — append
03 · leader → follower — replicate
04 · leader → log — passes data
05 · ctrl → leader — epoch / fence
06 · log → grp — poll
07 · grp → offset — commit
08 · grp → sink — process
09 · offset → grp — resume

End-to-end: Producer → partitioned leader log → replicated followers → consumers; offset commits follow successful processing; failed deliveries replay.

Key architectural decisions

  • Ordering is per partition, not global
  • At-least-once delivery requires idempotent consumers
  • Broker epochs fence stale leaders; partitions limit consumer parallelism

Repeat / gap: Repeat: backlog/retention arithmetic, producer backpressure, ordering vs throughput.

05 | Notification System

PRACTICED — exact transcript / mock score not available

Asynchronous multichannel notifications with preferences, retries, idempotency, and provider fallback.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G event Producer Services + Emit notification requests - At-least-once emits duplicates ing Notification API + Validate, authenticate, idempotency key - Idempotency bookkeeping cost event->ing prefs Preferences DB + Opt-in, channel, quiet-hour settings - Preference lookup adds latency ing->prefs 02 lookup temp Template / Priority + Render payload and choose urgency - Template/rules deployment risk ing->temp prefs->temp q Durable Priority Queues + Buffer send jobs; urgent before bulk - Backlog raises send latency temp->q 05 enqueue w Delivery Workers + Channel dispatch; concurrency limits - Provider quotas constrain throughput q->w 06 consume prov Push / Email / SMS + External providers and failover - Outages, rate limits, external cost w->prov 07 send dlq Retries / DLQ + Backoff, poison-message isolation - Delayed retry may miss relevance w->dlq 08 fail status Delivery State DB + Track attempts and final outcome - States may lag actual delivery w->status 09 state dlq->w 10 retry

Numbered arrow / data-flow reference

01 · event → ing — passes data
02 · ing → prefs — lookup
03 · ing → temp — passes data
04 · prefs → temp — passes data
05 · temp → q — enqueue
06 · q → w — consume
07 · w → prov — send
08 · w → dlq — fail
09 · w → status — state
10 · dlq → w — retry

End-to-end: Event → Notification API → user preferences and priority → durable queue → workers → provider; delivery state and retries recorded.

Key architectural decisions

  • Idempotency prevents duplicate sends where supported
  • Bounded retries / exponential backoff / provider failover
  • Urgency versus fairness; opt-out / quiet hours

Repeat / gap: REFERENCE RECONSTRUCTION: Known practiced themes; component-level design not fully recoverable.

06 | Distributed Job Scheduler

FULL MOCK — 7.5/10 overall · independent HLD + coached deep dives

Fast due-job discovery via Redis; durable claims and dispatch via PostgreSQL/outbox; leases, retries and quorum-fenced regions.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
scheduler_main client Client / API Gateway + Auth, quotas, CRUD of schedules - Burst traffic; validate requests config Job Management Service + Create / update / cancel jobs - Concurrent edits need version checks client->config db PostgreSQL Sharded State + Source of truth: configs + occurrences - 50K/s peaks require partitioning config->db 02 persist definitions relay Relay / Prefetch Service + Loads near-term scheduled runs - Refresh lag; overlapping windows db->relay 03 indexed upcoming runs dispatch Task Execution Service + Find due IDs; Lua reserve atomically - Reservations can expire or be stale db->dispatch 04 bounded fallback if Redis fails redis Redis Sorted Sets + Due-time index for next 5 minutes - Cache loss needs DB rebuild relay->redis redis->dispatch 06 due IDs claim DB Atomic Claim + Outbox + CAS SCHEDULED → CLAIMED + event - Durable transaction per occurrence dispatch->claim 07 only Lua winner proceeds claim->db 08 atomic claim outbox Outbox Relay + Reliably publish committed claims - At-least-once may republish claim->outbox queue RabbitMQ / SQS + Buffer job executions; isolate workers - Backlog adds start delay outbox->queue workers Execution Workers + Lease + heartbeat + idempotent retry - External side effects may duplicate queue->workers workers->db 12 status, lease logs Cassandra / Log Store + Store detailed attempt history - Eventual consistency, extra ops workers->logs monitor Metrics & Alerts + Lag, retries, lease expiry, DLQ - Monitoring is not automatic recovery workers->monitor

Numbered arrow / data-flow reference

01 · client → config — passes data
02 · config → db — persist definitions
03 · db → relay — indexed upcoming runs
04 · db → dispatch — bounded fallback if Redis fails
05 · relay → redis — passes data
06 · redis → dispatch — due IDs
07 · dispatch → claim — only Lua winner proceeds
08 · claim → db — atomic claim
09 · claim → outbox — passes data
10 · outbox → queue — passes data
11 · queue → workers — passes data
12 · workers → db — status, lease
13 · workers → logs — passes data
14 · workers → monitor — passes data

Failure recovery and cross-region fencing

scheduler_recovery region Quorum-backed Ownership + Lease + epoch per schedule partition - Partition may pause dispatch w US-West (old epoch 7) + Previously owned partition P42 - Stale owner after partition region->w old ownership expires e US-East (epoch 8) + Takes ownership with spare capacity - Promotion delay and higher load region->e new fenced lease claim Authoritative Claim Check + Reject stale epoch / canceled versions - DB coordination on dispatch w->claim epoch 7 rejected e->claim epoch 8 accepted q Queue + Worker Lease + At-least-once attempts + heartbeat - Expired lease may overlap attempts claim->q target External Task / Provider + Stable idempotency key per occurrence - Provider must honor dedup contract q->target recover Recovery Scanner + DLQ + Retry expired leases; retain failures - Backpressure may delay retries q->recover timeout / no ACK recover->q bounded retry

End-to-end: Client → Job Management API → PostgreSQL → Relay (5-minute lookahead) → Redis sorted sets → Task Execution Service (Lua reservation) → PostgreSQL atomic claim + outbox → queue → leased workers → completion history.

Correctness: One occurrence = unique (jobId, scheduledAt, version). A Redis reservation reduces contenders but never replaces the authoritative PostgreSQL claim. Claim and outbox insert commit together. The outbox relay may publish twice; consumers must deduplicate attempts.

Worker crash: RUNNING lease + heartbeat → expiry → recovery worker atomically acquires a newer attempt → same idempotency key to external provider → durable COMPLETED, then queue ACK. Never claim true exactly-once arbitrary side effects.

Redis outage: bounded indexed DB scan of due occurrences, then refill only eligible future SCHEDULED occurrences; throttle warmup to protect PostgreSQL.

Cancellation: update PostgreSQL config version/status; event invalidates Redis; DB claim validates version/status atomically. Running jobs use cooperative cancellation.

Capacity: 10M new jobs/day × 100KB ≈ 1TB raw/day; 100M active ≈ 10TB raw. 10M midnight jobs within +5s requires ≈2M starts/s (40× normal 50K peak). Prioritize durability, relax punctuality under overload.

Revision gap: Strong worker-crash reasoning during deep dive (9/10), but final summary forgot lease recovery and durable outbox. Rehearse both without prompts. Overall 7.5/10, borderline Lean Hire (Senior), Staff not yet proven.

07 | Google Drive / File Storage + Sync

GUIDED 8-PHASE WALKTHROUGH — independent mock not validated

Keep large file bytes off application servers; commit immutable versions atomically and sync by durable change cursors.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G dev Client Devices + Chunk upload/download; local sync cursor - Offline divergence and retry logic gw API Gateway / Auth + Authorize files and requests - Central auth adds latency dev->gw up Upload / Download API + Issue scoped signed URLs + sessions - Signed URL expiry / upload cleanup gw->up 02 upload / download meta Metadata Service + CAS current-version pointer + manifests - Atomic version checks may conflict gw->meta 03 commit metadata sync Sync Service + Cursor pull; notifications are hints - Devices can lag or miss notifications gw->sync 04 get changes obj Object Storage + Immutable checksum-verified chunks - Orphaned chunks and storage bills up->obj 05 signed direct bytes region Cross-region Durability + Protect committed versions on region loss - Cross-region commit latency and cost obj->region 06 replicated chunks db Metadata DB / Quorum + Authoritative file versions + folder state - Quorum can reduce write availability meta->db 07 CAS transaction chg Account ChangeLog + Ordered durable sync changes - History retention / compaction cost meta->chg 08 atomic change out Outbox / CDC + Propagate committed changes reliably - Events arrive late or duplicate db->out 09 commit events db->region 10 durable quorum chg->sync 11 since cursor idx Folder Index + Read-optimized paginated folder view - Eventually consistent folder listings out->idx 12 build index

Numbered arrow / data-flow reference

01 · dev → gw — passes data
02 · gw → up — upload / download
03 · gw → meta — commit metadata
04 · gw → sync — get changes
05 · up → obj — signed direct bytes
06 · obj → region — replicated chunks
07 · meta → db — CAS transaction
08 · meta → chg — atomic change
09 · db → out — commit events
10 · db → region — durable quorum
11 · chg → sync — since cursor
12 · out → idx — build index

End-to-end: Upload chunks directly to object storage via signed URLs; metadata CAS commits manifest + ChangeLog; devices sync missed changes by cursor.

Key architectural decisions

  • Acknowledge only when referenced chunks + metadata are durable
  • Stale edits get 409/conflict copy; never silently overwrite
  • Folder indexes eventual; authoritative versions strong

Repeat / gap: Cold test: multi-region split brain / quorum and hot-account sharding.

08 | Distributed Denylist / Blocking

FULL DESIGN WALKTHROUGH — Oct 19

Versioned immutable denylist snapshots push control-plane updates to low-latency regional matchers.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G src Gov / Trusted Sources + Publish IP / CIDR deny rules - Inputs can be malformed/outdated fetch Fetcher / Normalizer + Fetch, validate, normalize source entries - Source outages or format drift src->fetch db Canonical Rules DB + Source-specific rules + generations - Generation storage / validation work fetch->db 02 ingest generation snap Snapshot Builder + Compile CIDR rules / test full version - Build time delays urgent changes db->snap 03 latest rules obj Immutable Snapshot Store + Versioned artifacts + verified manifest - Download and storage replication cost snap->obj 04 artifact then manifest notify Version Notifications + Tell regions of new snapshot - Missed notifications must reconcile snap->notify 05 version ready reg Regional Updaters + Download, validate, stage next version - Rollout skew across regions obj->reg 06 download notify->reg 07 signal matcher In-memory Rule Matcher + Atomic v41 → v42 swap; trie lookup - Memory footprint; delayed version switch reg->matcher 08 atomic ref swap gw Edge / Gateway + Check IP and allow or block - Adds per-request lookup latency matcher->gw 09 allow / reject gw->matcher 10 check req Incoming Requests + High-rate traffic checks - Burst traffic pressures edge capacity req->gw

Numbered arrow / data-flow reference

01 · src → fetch — passes data
02 · fetch → db — ingest generation
03 · db → snap — latest rules
04 · snap → obj — artifact then manifest
05 · snap → notify — version ready
06 · obj → reg — download
07 · notify → reg — signal
08 · reg → matcher — atomic ref swap
09 · matcher → gw — allow / reject
10 · gw → matcher — check
11 · req → gw — passes data

End-to-end: Sources → canonical generation → snapshot compile/test → immutable manifest → regional updater → atomic in-memory rule switch → gateway checks.

Key architectural decisions

  • Readers capture one immutable snapshot; no partial trie mutation
  • Polling reconciles missed version notifications
  • Allow ~1 min propagation; preserve old version for rollback

Repeat / gap: Repeat: competing source contributions; immediate revocation semantics and stale region behavior.

09A | Google News — Publisher Ingestion

FULL MOCK — overall 7/10 (guided improvements)

Absorb article bursts durably; make indexed news visible before slower story clustering finishes.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G pub 50K Publishers + RSS, webhook / API, web crawling - Feed quality and publisher outages vary ing Publisher Ingestion + Validate links and normalize metadata - Input validation adds ingestion work pub->ing mq Durable Article Queue + Buffer spikes; isolate slow processors - Queue lag consumes freshness budget ing->mq wk Processing Workers + Exact dedup, parse, safety checks - CPU-heavy enrichment creates backlog mq->wk db Sharded Article DB + Authoritative article metadata; hash(articleId) - Shard hot spots / eventual index lag wk->db 04 persist media Object Storage + CDN + Serve thumbnail bytes independently - CDN invalidation and egress charges wk->media 05 thumbnails out Outbox / CDC + Reliable committed article events - Consumers see events eventually db->out idx Article Index + By publisher, category, time, region - Write amplification and index lag out->idx 07 index cl Story Clustering + Async group related items by storyId - Semantic grouping is costly/imperfect out->cl 08 group upd Feed Materializer + Refresh shared category / breaking feeds - Invalidation fanout and rebuild spikes out->upd 09 update cl->upd 10 story event cache Shared Feed Caches + Hot category + global news candidates - Stale news until refresh/event upd->cache 11 publish candidates

Numbered arrow / data-flow reference

01 · pub → ing — passes data
02 · ing → mq — passes data
03 · mq → wk — passes data
04 · wk → db — persist
05 · wk → media — thumbnails
06 · db → out — reads / writes
07 · out → idx — index
08 · out → cl — group
09 · out → upd — update
10 · cl → upd — story event
11 · upd → cache — publish candidates

End-to-end: Publisher → normalization → durable queue → workers → sharded DB → committed event stream → article index / clustering / feed caches.

Key architectural decisions

  • At 8k/s input vs 2k/s DB for 10s, backlog is 60k; ~30s more to drain
  • Exact dedup before write; cluster async so 60s freshness still possible
  • Index gives publisher/time reads without scatter across article shards

Repeat / gap: Repeat: derive event pipeline independently; bound end-to-end indexing lag.

09B | Google News — Feed Serving + Failover

FULL MOCK — overall 7/10 (guided improvements)

Hybrid feed construction, shared hot caches, stable cursor snapshots, and graceful regional degradation.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G cli Web / Mobile Clients + Refresh creates new feed; scroll uses cursor - Refresh resets pagination snapshot global Global Traffic Router + Healthy nearby region; failover with headroom - Failover adds latency/load to peers cli->global gw Regional API Gateway + Auth, limits, route to Feed Service - Rate limits can impact burst reads global->gw feed Feed Service + Coordinate candidates, ranking, pages - Ranking can make p99 unpredictable gw->feed 03 GET /feed prefs User Profile DB + Interests, locale, geo preferences - Privacy and dependency latency feed->prefs 04 lookup shared Shared Regional Caches + Category, region and breaking news lists - Hot keys and stale candidates feed->shared 05 hot candidates index Article Index + Fallback candidates and metadata lookup - Index replication lag affects freshness feed->index 06 miss / fallback rank Candidate Ranker + Lightweight per-user ordering - Compute cost; ranking volatility feed->rank 07 candidate set prefs->rank 08 features snap Snapshot Cache + Fixed ranked IDs and continuation cursor - Per-session storage and expiry rank->snap 09 materialize active session snap->feed 10 20 + cursor cdn Media CDN + Fast thumbnail delivery to clients - Stale thumbnails and egress cost cdn->cli 11 images

Numbered arrow / data-flow reference

01 · cli → global — passes data
02 · global → gw — passes data
03 · gw → feed — GET /feed
04 · feed → prefs — lookup
05 · feed → shared — hot candidates
06 · feed → index — miss / fallback
07 · feed → rank — candidate set
08 · prefs → rank — features
09 · rank → snap — materialize active session
10 · snap → feed — 20 + cursor
11 · cdn → cli — images

End-to-end: Client → global routing → regional feed → shared cache + preferences + optional index → rank → stable snapshot → 20 stories + cursor.

Key architectural decisions

  • Cursor pins one stable snapshot; refresh starts a newer version
  • Failover preserves user geography but may degrade personalization
  • Prewarm popular caches; coalesce misses; reserve failover capacity

Repeat / gap: Repeat: distinguish active-session snapshots from shared caches; explain hot keys and index lag.

10 | Food Delivery Cost Dashboard

SEPARATE DESIGN REQUEST — worked architecture not recoverable

Financial aggregate from deliveries, overlapping work time, and time-dependent driver rates.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G driver Delivery Sources + Send completed deliveries / timestamps - Late, duplicate, or corrected events api Ingestion API + Validate delivery and driver identifiers - Validation adds write path overhead driver->api q Delivery Event Queue + Durably absorb delivery updates - Queue lag makes dashboard stale api->q worker Cost Calculator + Apply duration × hourly rate per delivery - Rate changes require historical lookup q->worker ledger Cost Ledger + Immutable per-delivery calculated entries - Append-only entries require corrections worker->ledger 04 transactional cost rate Rate History DB + Versioned driver rates / effective times - Rate-version correctness complexity rate->worker 05 effective rate agg Aggregate Worker + Maintain total; correct reversals idempotently - Hot global total; replay handling ledger->agg 06 events total Total Cost Store + One authoritative dashboard aggregate - Single hot aggregate / failover cost agg->total 07 aggregate dash Accounting Dashboard + Read current total with freshness indicator - Displayed total may be delayed total->dash 08 read

Numbered arrow / data-flow reference

01 · driver → api — passes data
02 · api → q — passes data
03 · q → worker — passes data
04 · worker → ledger — transactional cost
05 · rate → worker — effective rate
06 · ledger → agg — events
07 · agg → total — aggregate
08 · total → dash — read

End-to-end: Completed delivery → durable event → per-delivery cost using rate history → idempotent ledger → total aggregate → accounting dashboard.

Key architectural decisions

  • Multiple simultaneous deliveries are billed independently per requirement
  • Idempotent delivery IDs prevent double-counting
  • Handle corrections/rate revisions and reconcile aggregates

Repeat / gap: REFERENCE DESIGN ONLY: Prompt was discussed; no independent architecture/mock record verified.

11A | Ticketmaster — Search + Seat Reservation

GUIDED FULL DESIGN + FINAL SYNTHESIS (~7/10) — previously omitted

Separate eventual event discovery from strongly consistent seat inventory and reservation ownership.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G client Buyer Client + Search, select seats, pay - Multiple buyers contend for one seat gateway CDN / API Gateway + Static assets, auth, controls - Traffic floods still need admission client->gateway search Event Search Service + Fast public search & listing - Index may show stale availability gateway->search 02 discover wait Waiting Room + Admission token; backpressure - Fairness and long user wait gateway->wait 03 hot sale redis Redis / Search Cache + Popular events; approximate seats - TTL and invalidation leave stale seats search->redis es Elasticsearch + Event search projection - Eventual index freshness / extra cost redis->es 05 miss events Event Catalog DB + Source of event details - Catalog writes + cross-index sync out Outbox / CDC + Project event updates - Index publication is asynchronous events->out out->es book Reservation Service + Validate token, reserve seats - High contention on hot events wait->book 08 admit inv Inventory DB (Postgres) + Atomic AVAILABLE → HELD - Atomic locks constrain hot-seat writes book->inv 09 conditional txn hold Reservation Hold + reservationId + expiresAt - Expired holds need safe reconciliation inv->hold 10 record hold delay Expiry Worker / Delay Queue + Cleanup hint, not correctness - Expiry jobs can arrive late hold->delay 11 expiry event delay->inv 12 CAS if expired

Numbered arrow / data-flow reference

01 · client → gateway — passes data
02 · gateway → search — discover
03 · gateway → wait — hot sale
04 · search → redis — passes data
05 · redis → es — miss
06 · events → out — passes data
07 · out → es — passes data
08 · wait → book — admit
09 · book → inv — conditional txn
10 · inv → hold — record hold
11 · hold → delay — expiry event
12 · delay → inv — CAS if expired

End-to-end: Search: client → CDN/Gateway → Search Service → Redis/Elasticsearch (eventual). Booking: waiting room → Reservation Service → authoritative Postgres conditional seat hold.

Key decisions:

  • Never use Redis seat-map as inventory source of truth; DB owns reservation state.
  • Atomic seat transition includes reservationId and expiration; cleanup worker is only a hint.
  • Hot event requires admission control and inventory partitions within an event, e.g. section/range.

Revision focus: Retest: articulate DB conditional update and expiry race without relying on distributed lock. Guided walkthrough followed by ~7/10 synthesis.

11B | Ticketmaster — Payment + Multi-Region Safety

GUIDED FULL DESIGN — correctness and failure recovery

Payment is external: durable intent, idempotent charging, reconciliation, and fenced regional ownership.

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G hold Existing HELD Reservation + reservationId, expiry, buyer - Hold expiry races with payment pay Payment Orchestrator + Durable PAYMENT_PENDING - External side effects are uncertain hold->pay psp Payment Provider + External charge; retries - Timeout may hide successful charge pay->psp 02 idempotency key recon Reconciliation Worker + Resolve ambiguous timeout - Slow settlement and refund paths pay->recon 03 ambiguous db Inventory + Orders DB + Conditional HELD → SOLD - Transactions serialize contested seats pay->db 04 finalize SOLD psp->pay 05 success / unknown recon->psp 06 query status out Transactional Outbox + Reliable post-commit events - Async confirmations may be delayed db->out rep Standby / Regional Replica + Replicate committed inventory - Replication lag risks failover data loss db->rep 08 replication notif Confirmation Worker + Send tickets / receipts - Email/push duplicates or outage out->notif home Event Home-Region Writer + Single authorized write owner - Hot home-region writer bottleneck home->db 10 owns writes quorum Quorum + Epoch/Fencing + Promote new owner safely - Minority region loses write access rep->quorum 11 on failure quorum->home 12 new epoch / fence old

Numbered arrow / data-flow reference

01 · hold → pay — passes data
02 · pay → psp — idempotency key
03 · pay → recon — ambiguous
04 · pay → db — finalize SOLD
05 · psp → pay — success / unknown
06 · recon → psp — query status
07 · db → out — reads / writes
08 · db → rep — replication
09 · out → notif — passes data
10 · home → db — owns writes
11 · rep → quorum — on failure
12 · quorum → home — new epoch / fence old

End-to-end: Reserve seat → persist payment intent → charge with stable idempotency key → reconcile uncertain outcomes → conditional SOLD commit → outbox confirmation. Failover needs quorum and fencing.

Key decisions:

  • PAYMENT_PENDING must not expire like an ordinary unpaid hold.
  • A timeout is not proof of payment failure: reconcile before refund/charge retry.
  • During region partitions, stop minority writes to avoid double-selling; accept reduced booking availability.

Revision focus: Retest: single-writer ownership, external-payment ambiguity, hot-event shard key and explicit write-unavailability tradeoff.

12A | Distributed Metrics — Complete Architecture

GUIDED DESIGN — independent initial HLD, guided deep dives

Full ingestion, storage, querying, alerting, and notification pipelines.

G agent Agents / Exporters + Collect & batch samples - Local buffers are bounded gw Regional Gateway + Route agents to healthy region - Failover increases regional load agent->gw 01 batched samples ing Ingestion Service + Validate & normalize metrics - CPU / network overhead gw->ing 02 push metrics quota Quota + Series Registry + Restrict new high-cardinality series - May reject useful new labels ing->quota 03 admit series kafka Durable Kafka Stream + Buffer burst and support replay - Finite capacity, retention lag quota->kafka 04 accepted samples writer TSDB Writers + Batch writes with backpressure - Can lag when database slows kafka->writer 05 TSDB consumer flink Flink Alert Evaluators + Evaluate windows independently - Checkpoint and replay complexity kafka->flink 09 independent alert consumer tsdb Recent TSDB / Cassandra + Indexed recent sample storage - Partition/hot-key tuning writer->tsdb 06 persist samples roll Flink Rollup Workers + Aggregate by time and labels - Late data needs correction tsdb->roll 07 historical rollups s3 Historical Rollup Store + Cheap 30-day+ query layer - Older points less detailed roll->s3 08 compact and tier state Durable Incident State + Own FIRING and RESOLVED episodes - Cross-region state coordination flink->state 11 transition incident notify Notification + Outbox + Retry & dedupe each incident - Exactly-once delivery needs provider state->notify 12 FIRING / RESOLVED dest PagerDuty / Teams / Email + Reach on-call responders - Delivery dependencies can fail notify->dest 13 deduped delivery rule Rule Config + PostgreSQL + Own alert definitions - Config propagation may lag rule->flink 10 CDC rule versions query Query Engine + Prune, aggregate, cache results - Expensive high-cardinality queries query->tsdb 15 recent time range query->s3 16 historical rollups dash Dashboards / Clients + View historical and live trends - Freshest bucket may change dash->query 14 range query

Numbered arrows — what crosses each connection

01 · Agents / Exporters → Regional Gateway: batched samples
02 · Regional Gateway → Ingestion Service: push metrics
03 · Ingestion Service → Quota + Series Registry: admit series
04 · Quota + Series Registry → Durable Kafka Stream: accepted samples
05 · Durable Kafka Stream → TSDB Writers: TSDB consumer
06 · TSDB Writers → Recent TSDB / Cassandra: persist samples
07 · Recent TSDB / Cassandra → Flink Rollup Workers: historical rollups
08 · Flink Rollup Workers → Historical Rollup Store: compact and tier
09 · Durable Kafka Stream → Flink Alert Evaluators: independent alert consumer
10 · Rule Config + PostgreSQL → Flink Alert Evaluators: CDC rule versions
11 · Flink Alert Evaluators → Durable Incident State: transition incident
12 · Durable Incident State → Notification + Outbox: FIRING / RESOLVED
13 · Notification + Outbox → PagerDuty / Teams / Email: deduped delivery
14 · Dashboards / Clients → Query Engine: range query
15 · Query Engine → Recent TSDB / Cassandra: recent time range
16 · Query Engine → Historical Rollup Store: historical rollups

Why this design works: Separate consumers of Kafka isolate Cassandra slowdown from alert evaluation. A series registry blocks unbounded label cardinality; rollups and query pruning keep 30-day dashboards fast.

Key guarantees: Alert state and notification delivery state are durable. Kafka buffers failures only up to provisioned retention and disk capacity. Critical alert recovery depends on checkpointed/replicated stream state.

Known practice gap: Multi-region stream failover and state reconciliation were taught, not independently tested.

12B | Distributed Metrics — Regional Failover

GUIDED DEEP DIVE — failover recovery reference

Recover metrics and alerts across regions while minimizing duplicates and loss.

G west US-West Agents + Buffer samples on failure - Bounded buffers may overflow lb Global Routing + Switch ingestion to US-East - Detection and propagation delay west->lb 01 region unavailable east US-East Ingestion + Kafka + Accept / replay durable events - Requires reserved surge capacity lb->east 02 reroute and resend f Flink Alert Evaluator + Replay samples and state - Late / duplicated samples east->f 03 resume Kafka offsets ts TSDB Backfill + Recover historical gaps - Some unreplicated samples lost east->ts 08 replay backfill ckpt Replicated Checkpoints + Restore operator state/offsets - Replication can lag ckpt->f 04 restore checkpoint s Durable Incident Store + Incident ID & generation - Must reconcile replicated state f->s 05 reconcile transitions o Delivery Outbox + One claim per transition - Redelivery remains possible s->o 06 durable notification claim n PagerDuty Dedup Key + Suppress replayed notifications - Depends on provider retention o->n 07 same incident key

Numbered arrows — what crosses each connection

01 · US-West Agents → Global Routing: region unavailable
02 · Global Routing → US-East Ingestion + Kafka: reroute and resend
03 · US-East Ingestion + Kafka → Flink Alert Evaluator: resume Kafka offsets
04 · Replicated Checkpoints → Flink Alert Evaluator: restore checkpoint
05 · Flink Alert Evaluator → Durable Incident Store: reconcile transitions
06 · Durable Incident Store → Delivery Outbox: durable notification claim
07 · Delivery Outbox → PagerDuty Dedup Key: same incident key
08 · US-East Ingestion + Kafka → TSDB Backfill: replay backfill

Failure sequence: Agents retry against US-East; surviving Kafka processes buffered and replicated records; Flink restores a durable checkpoint then replays from consistent offsets.

Alert correctness: An incident ID tracks FIRING through RESOLVED, and the same delivery idempotency key is reused across replay. A new firing after resolution gets a new incident generation.

Honest guarantee: No acknowledged metric loss requires explicit cross-region replication semantics. Duplicate-free external pages require provider-side deduplication or idempotency support.

DRILL 1 | Read-Heavy Hot Key

SCENARIO DRILL — not a full-system mock

Cache popular item across independent read owners

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G req Heavy Read Traffic + Skew on one key - High skew even if average traffic low router Read Router + Detect and spread hot reads - Routing table skew and control overhead req->router copies Replicated Cache Copies + Independent capacity per copy - Every update fans out to copies router->copies db Source of Truth + Fallback and updates - Fallback can become bottleneck copies->db 03 miss / refresh

Numbered arrow / data-flow reference

01 · req → router — passes data
02 · router → copies — passes data
03 · copies → db — miss / refresh

End-to-end: Replicate read-only hot data across nodes; avoid one owner taking all traffic.

Key decisions:

  • Version/TTL invalidation must reach every replica.
  • Replication helps reads; it does not multiply write authority.

Revision focus: Repeat briefly as scenario-specific correctness test.

DRILL 1b | Hot-Key Write Propagation

SCENARIO DRILL — not a full-system mock

One write authority and version-aware invalidation

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G writer Writer + Update value - Concurrent writers reorder events owner Authoritative DB/Leader + Atomic monotonic version - Leader / CAS cost per version writer->owner evt Invalidation Event Bus + Fan out version updates - Out-of-order or missed invalidations owner->evt replicas Replicated Caches + Reject stale version - Stale reads until copies converge evt->replicas reader Reader + Check current data - Freshness check costs a lookup reader->replicas

Numbered arrow / data-flow reference

01 · writer → owner — passes data
02 · owner → evt — passes data
03 · evt → replicas — passes data
04 · reader → replicas — passes data

End-to-end: Commit authoritative version → emit invalidation → replicas reject older versions.

Key decisions:

  • State where version check is atomic.
  • Recover after missed events via TTL/reconciliation.

Revision focus: Repeat briefly as scenario-specific correctness test.

DRILL 2 | Ticket Seat Reservation Drill

SCENARIO DRILL — not a full-system mock

Hold → pay → sold with database correctness

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G buyer Buyer + Select one seat - Double-clicks / competing buyers booking Booking API + Create reservation - Retry may create duplicate hold buyer->booking db Seat DB + Conditional AVAILABLE → HELD - Contention on one seat row booking->db payment Payment / Finalizer + Idempotent completion - Charge and inventory can diverge db->payment 03 held sold SOLD State + At most one winner - Winner finalization must be idempotent payment->sold 04 conditional txn

Numbered arrow / data-flow reference

01 · buyer → booking — passes data
02 · booking → db — reads / writes
03 · db → payment — held
04 · payment → sold — conditional txn

End-to-end: AVAILABLE → HELD → PAYMENT_PENDING → SOLD; unique DB constraints prevent double sale.

Key decisions:

  • Expiry is checked atomically, not solely via delayed cleanup.
  • Same payment ID cannot sell twice.

Revision focus: Repeat briefly as scenario-specific correctness test.

DRILL 3 | Profile Service Dependency Outage

SCENARIO DRILL — not a full-system mock

Bound dependency pressure with graceful fallback

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G api Gateway / Caller + Needs profile - Synchronous dependency adds latency cb Circuit Breaker + Open on sustained errors - Premature opening may block recovery api->cb dep Profile Service + Primary dependency - Failure cascades under retry storm cb->dep 02 healthy cache Last-Known Cache + Stale bounded fallback - Staleness vs availability cb->cache 03 open / fallback queue Bounded Work Queue + Buffer async-safe operations - Buffer must be bounded cb->queue 04 async-only

Numbered arrow / data-flow reference

01 · api → cb — passes data
02 · cb → dep — healthy
03 · cb → cache — open / fallback
04 · cb → queue — async-only

End-to-end: Trip circuit breaker → use allowed stale cache → bounded retries with jitter → recover gradually.

Key decisions:

  • Do not endlessly buffer synchronous requests.
  • Specify which operations may be stale or dropped.

Revision focus: Repeat briefly as scenario-specific correctness test.

DRILL 4 | Shard Growth 20 → 200

SCENARIO DRILL — not a full-system mock

Migrate ownership through versioned cutover

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G router Consistent-Hash Router + Virtual nodes + version - Mapping version split causes misroutes old 20 Existing Shards + Read/write owner - Dual load during migration router->old 01 old map copy Migration Workers + Copy + catch up changes - Concurrent writes require catch-up old->copy new 200 Target Shards + Verify copied state - Need verify completeness before cutover copy->new cut Cutover Controller + Atomic ownership epoch - Epoch / rollback complexity new->cut 04 verified cut->router 05 publish new map

Numbered arrow / data-flow reference

01 · router → old — old map
02 · old → copy — passes data
03 · copy → new — passes data
04 · new → cut — verified
05 · cut → router — publish new map

End-to-end: Consistent hashing limits remaps; verify data and switch write owner with epoch/version.

Key decisions:

  • Specify dual-write or log catch-up and rollback plan.
  • Old writer must be fenced after cutover.

Revision focus: Repeat briefly as scenario-specific correctness test.

DRILL 5 | Replica Lag (30s)

SCENARIO DRILL — not a full-system mock

Route correctness-sensitive reads appropriately

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G client Read Client + Needs fresh profile - Stronger guarantees increase latency routing Consistency Router + Choose freshness policy - Extra branching and coordination client->routing replica Lagging Replica + Fast, possibly 30s stale - Can violate read-your-writes routing->replica 02 stale OK primary Primary Database + Latest acknowledged state - Read overload on writer routing->primary 03 strong read fallback Wait / Error Path + When freshness required - Unavailability / long waits routing->fallback 04 no healthy source

Numbered arrow / data-flow reference

01 · client → routing — passes data
02 · routing → replica — stale OK
03 · routing → primary — strong read
04 · routing → fallback — no healthy source

End-to-end: Critical reads use primary or wait/error; noncritical reads may use replicas.

Key decisions:

  • Define maximum tolerated staleness.
  • What user-visible guarantee is relaxed?

Revision focus: Repeat briefly as scenario-specific correctness test.

DRILL 6 | Queue Ack Lost

SCENARIO DRILL — not a full-system mock

Idempotent effects with at-least-once delivery

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G queue Durable Queue + May redeliver after lost ack - At-least-once means replay worker Worker + Process messageId - Crash after side effect before ack queue->worker idem Idempotency Store + Atomic processed key - Needs atomic durable transaction worker->idem 02 check-and-reserve side Business Side Effect + Exactly once logically - External API may lack idempotency idem->side 03 atomic boundary ack Queue Ack + Only after commit - Lost acknowledgment repeats delivery side->ack

Numbered arrow / data-flow reference

01 · queue → worker — passes data
02 · worker → idem — check-and-reserve
03 · idem → side — atomic boundary
04 · side → ack — passes data

End-to-end: Lost ack → re-delivery; durable idempotency key prevents duplicate logical effect.

Key decisions:

  • Side effect + idempotency commit must share a safe boundary.
  • External APIs require stable idempotency keys.

Revision focus: Repeat briefly as scenario-specific correctness test.

DRILL 7 | Multi-Region Network Partition

SCENARIO DRILL — not a full-system mock

Keep a single safe write authority

+ Achieves / why it exists− Tradeoff / cost to defend in the interview
G client Clients + Write requests - Minority-side writes may be rejected quorum Quorum / Lease Authority + Issue epoch; elect owner - Availability loss without majority client->quorum leader Current Region Leader + May be isolated - Old leader may keep serving stale fence Epoch Validation + Reject stale old-leader writes - Every write must validate epoch leader->fence 02 old epoch new Surviving Majority + New authorized writer - Promotion adds temporary downtime quorum->new 03 majority promotion fence->new 04 reject stale owner

Numbered arrow / data-flow reference

01 · client → quorum — passes data
02 · leader → fence — old epoch
03 · quorum → new — majority promotion
04 · fence → new — reject stale owner

End-to-end: Surviving quorum promotes one writer and fences isolated region, avoiding split brain.

Key decisions:

  • Minority region must refuse writes.
  • Quantify recovery objective and replay lag.

Revision focus: Repeat briefly as scenario-specific correctness test.

Scenario drills (not full-system mocks)

Only 7 of the planned 25 drills plus 1b were verified: hot read key; hot-key writes/version invalidation; seat reservation; dependency outage; 20→200 shard migration; replica lag; duplicate queue delivery / lost acknowledgment; multi-region network partition.

Cross-mock repeat queue: capacity/backlog calculations; state ownership and atomicity; cache stampede and AZ failover; queue ordering versus concurrency; 90-second synthesis.

Reminder: understanding the architecture after coaching ≠ independently deriving it in a timed interview.