Every lesson so far gave you one pipeline, one feed, one database. This week the job gets bigger than the tools: four data sources, fifty thousand records an hour, a legacy API with opinions, strict PII rules — and a customer who wants the architecture drawn by tomorrow. Your instinct will be to start drawing boxes. Don't. There's an order to this, and the boxes come third.

Assumes: Stage 3 so far (especially Post 3 on backoff and Post 4 on receiving events) and Milestone 2 (the intake pipeline this design grows out of). Installs: none — every runnable below is stdlib-only.

Monday, 8:05 AM. Maria: "The county wants in. Not just our 311 feed — a regional consolidation. Four sources, about fifty thousand records an hour."

You (reaching for the whiteboard marker): "Okay, so we need a bigger pipeline, a queue, maybe—"

Maria: "Two of the sources carry PII — names and phone numbers in the complaint text. Lisa's rules apply. And one of them is the county's legacy API — Dev says it's slow and it throttles."

You (marker halfway to the board): "And the deadline?"

Maria: "The architecture decision is tomorrow. The council briefing is in three weeks."

Four sources. Fifty thousand records an hour. PII. A legacy API. Results tomorrow. This is the moment the job stops being "build the thing" and becomes "decide what the thing is" — and the most expensive mistake is deciding too fast.

Before you code: clarify the ask

You (marker down): "Before the boxes — what are the four sources, exactly?"

Maria: "Our 311 Socrata feed, the county's legacy API, nightly CSV dumps from the neighboring borough, and webhook pushes for status changes."

You: "And 'results' means what? A database the console reads, or answers to specific questions?"

Maria: "The console reads it. Same workflow as now, regional data."

You: "The PII — which sources, and what does Lisa's rulebook actually require?"

Maria: "The legacy API and the CSVs carry caller names and phone numbers. Lisa's rule: PII is redacted before it rests anywhere, and it never appears in logs."

You: "And the legacy API's manners — rate limit, page size, timeouts?"

Maria: "Ten requests a second, a hundred rows a page. Dev says it gets slow at night."

You: "Last one: fifty thousand an hour — average, or peak?"

Maria: "...I'll find out. That's a good question."

Input: four named sources, a PII rulebook, a legacy API's manners, one unanswered question about peak vs average
Output: an architecture decision Maria can defend at the council briefing — with numbers
Deadline: tomorrow for the decision, three weeks for the build

Notice what just happened: the marker went down and the questions came out. Every answer above deleted a wrong architecture. "The console reads it" deleted the data-warehouse detour. "Redacted before it rests" placed a redaction step before storage. "Ten requests a second" put a ceiling on one source. The design is already happening — it just doesn't look like boxes yet.

The minimal concept

System design, FDE-style, is ten steps in an order. Not a methodology — an order. Each step's output is the next step's input, and skipping one is how you get an architecture that fails at 2 AM for a reason nobody wrote down:

  1. Requirements — what, exactly? Sources, consumers, PII, deadline. Written down, not nodded at.
  2. Scale — the arithmetic. Records per second, GB per day, workers needed. Computed, not guessed.
  3. Data flow — where does one record travel, from source to screen? Draw the journey before the boxes.
  4. Components — now the boxes: what each one does, and only what it does.
  5. Sync vs async — who waits for whom? The slowest dependency sets the pace of everything it touches.
  6. Storage — where data rests, for how long, and how big that gets.
  7. Failure modes — what breaks, what's the blast radius, what's the mitigation? Named in advance.
  8. Security — where's the PII, who can see it, how is it proven gone?
  9. Observability — how do you know it's healthy at 8 AM without reading code?
  10. Tradeoffs — what you chose, what you rejected, and why. In writing.

The rest of this lesson runs the scenario through all ten. The runnable parts ran on this machine — their outputs are pasted from real runs. The diagrams and tables are conceptual: design artifacts, not code, labeled as such.

Build it, step 1: requirements, written down

The four sources, as a table — because "four sources" is a rumor and a table is a requirement:

SourceHow it arrivesVolume sharePII?Manners
A: 311 SocrataPoll, paged~40%NoWell-behaved; the feed we know
B: county legacy APIPoll, paged~25%Yes — names, phones10 req/s, 100/page, slow at night
C: borough CSV dumpsNightly file drop~25%Yes — names, phonesOne big file, midnight
D: status webhooksPush to our receiver~10%NoBursty; retries for a day

And the requirement list, one line each — the whole contract with Maria:

  • Ingest ~50K records/hour average from four sources into one regional store.
  • The operator console reads it — same workflow, regional data (Post 6).
  • PII redacted before it rests anywhere; PII never in logs. (Lisa's rule, verbatim.)
  • Every record accounted for — Milestone 2's reconciliation, at regional scale.
  • Survive a source being slow, a source being down, and a deploy at the wrong hour.

Build it, step 2: the arithmetic [runnable]

Maria came back with the answer: 50K/hour is the average; the evening rush runs about 3×. That one answer changes the sizing completely — so it goes into the script, not into memory. A back-of-envelope capacity script, every number computed:

AVG_PER_HOUR   = 50_000     # average intake across all 4 sources
PEAK_FACTOR    = 3.0        # evening rush multiplier
AVG_RECORD_KB  = 2.5        # measured from a 1,000-row sample of the feed
WORKER_REC_S   = 150        # one worker: validate + redact + write, measured locally
REDACT_REC_S   = 500        # PII redaction pass alone, measured locally
LEGACY_RPS     = 10         # legacy API rate limit (their docs)
LEGACY_PAGE    = 100        # legacy page size
RETENTION_DAYS = 90
INDEX_OVERHEAD = 1.3        # indexes + WAL ≈ 30% on top of raw
avg intake      :   50,000/h  =   13.9 rec/s
peak intake     :  150,000/h  =   41.7 rec/s
volume          :    0.1 GB/h  ->    3.0 GB/day
disk (90d retention, 1.3x index):     351 GB   (estimate: excludes replicas,
                                               backups, WAL retention)
workers for peak throughput: 1  (41.7 rps needed; 1 worker = 150 rps measured)
workers deployed           : 2  (1 for throughput + 1 spare for crash coverage)
legacy API effective need  : ~10.4 rec/s (25% of peak 41.7)
legacy API ceiling         : 10 req/s x 100 rows = 1000 rows/s theoretical;
                             effective rate depends on latency, page fill, and
                             rate-limit windows — measure, don't assume
30-min outage at peak needs queue depth 75,000 records (187.5 MB baseline;
  assumes uniform arrivals — the CSV burst needs its own math, below)

Read the decisions hiding in those numbers — and the qualifications. Workers: 1 for throughput, 2 deployed. One worker's measured 150 rps already covers the 41.7 rps peak three times over; the second worker is crash redundancy, not capacity. Saying "2 workers for peak" would imply the peak needs two — it doesn't. Disk: 351 GB, an estimate — decimal GB, 2.5 KB/record, 1.3× for indexes and WAL. It excludes replicas, backups, and queue storage; the real deployment number comes from the DBA, not this script. The legacy API: source B is ~25% of volume, so its share of peak is ~10.4 rps — the 1000 rows/s ceiling is theoretical; what matters is its effective rate under real latency and rate-limit windows, which is why it gets its own lane. And the queue math is a baseline, not a maximum — see the burst analysis below.

The CSV burst, modeled separately. Source C delivers ~25% of the day's 1.2M records — ~300,000 rows — in one midnight file. If the sanitizer processes it in 20 minutes, that's a 250 rps burst against the queue, six times the "peak" rate. The 75,000-record baseline assumes uniform arrivals; the real queue depth must cover the CSV burst plus a coincident outage. Model per-source bursts, retry traffic, and recovery drain — then size the queue. The uniform math is the floor; the burst math is the design.

Build it, step 3: the data flow [conceptual]

Now — and only now — the diagram. One record's journey, before any boxes get responsibilities:

[source A: 311 Socrata] ──┐
                                    │
[source B: legacy API]  ──┼──▶ [in-memory: validate + REDACT] ──▶ [durable queue]
                                    │         ▲                         │
[source C: nightly CSV] ──┘         │         │                         │
                        sanitize-at-boundary   │              [workers ×2: validate
[source D: webhooks] ──▶ [receiver] ─┘         │               contracts → dedupe
                                               │                → write]
                                     raw PII never            │
                                     touches disk             ▼
                                              [Postgres · 351 GB · 90-day retention]
                                                              │
                                                              ▼
                                              [read API] ──▶ [operator console]
                                                              │
                                              [audit log · metrics · alerts]

The correction that matters most in this lesson: the first draft drew the durable queue before redaction — which would persist raw names and phone numbers to disk, violating Lisa's rule ("redacted before it rests anywhere") before the worker even ran. The queue holds sanitized events only. Redaction happens in memory at the ingestion boundary, per source adapter, before anything durable is written. And the CSV boundary gets its own discovery question — see step 7.

The receiver is Post 4's verified receiver — signature, freshness, replay protection — feeding the same queue. The workers run Milestone 2's contract gate and DLQ per source. The read API is Post 5's FastAPI service. Nothing here is new technology — it's the last seven lessons, composed.

Build it, step 4: sync vs async [runnable]

The question: do the four sources get fetched one after another, or independently? The legacy API's bad days decide it. Simulated latencies (a stand-in for the vendor's moods), measured timings:

# simulated per-source latencies: the legacy API has a bad day
LAT = {"socrata": 0.4, "legacy": 4.0, "csv": 0.8, "webhook": 0.1}
sync chain :  5.3s for 5000 records (slowest source sets the pace: 4.0s)
decoupled  :  4.0s for 5000 records (slowest source hides behind the others)

The sync chain takes as long as the sum; the decoupled fetch takes as long as the slowest. At 2 AM, when the legacy API's 4 seconds become 20, the sync version's nightly window blows by the same 16 seconds per cycle — compounded across every page. The rule: slow dependencies get their own lane. Each source polls on its own schedule, into its own queue partition, so one's bad day is one's problem. The queue from step 3 isn't just outage insurance — it's what makes "independent lanes" a real thing instead of a diagram.

Build it, steps 5–6: storage and failure modes

Storage: 351 GB for 90 days, multiple concurrent writers, and a read API serving operators — that's Postgres, not SQLite. Milestone 2's SQLite was the right call for one pipeline on one box; the arithmetic now says otherwise, and the arithmetic is allowed to change the answer. Retention is a requirement, not a default: 90 days because the council's quarterly review needs it, and the disk math above is what 90 days costs.

Failure modes — named in advance, with the blast radius and the mitigation, because "we'll handle it" is not a mitigation:

What breaksBlast radiusMitigation (already in the design)
Legacy API slows to 20s timeouts at nightSource B stallsOwn lane + own queue partition; alert on queue depth, not on the API
Postgres down 30 min for maintenance at peak75,000 recordsDurable queue holds 187.5 MB; workers replay after
Worker crashes mid-batchOne batchIdempotent writes (PRIMARY KEY); reprocess from the queue
PII source adds a new field overnightUnredacted PII at restContract rejects unknown fields to the DLQ (Milestone 2's gate)
Webhook secret leaksForged status eventsDual-secret rotation (Post 4)

Build it, step 7: PII [runnable]

Lisa's rulebook, as code: redact before rest, and never in logs. But the first version of this section had a hole Lisa caught in review — the demo redacted SSNs, phones, and emails, and left "John Doe" sitting in the output. Names are explicitly in Lisa's PII list. Regex can't reliably find names in free text, so the design doesn't try:

# Structured fields: drop or hash by policy, never regex.
# Free text: allowlisted output schema — only known-safe fields survive.
def redact_record(rec):
    out = {k: rec[k] for k in SAFE_FIELDS}   # allowlist, not denylist
    out["caller_name_hash"] = sha256(rec["caller_name"].encode()).hexdigest()
    out["descriptor"] = redact_free_text(rec["descriptor"])  # SSN/phone/email
    # Free-text names: documented limitation. The policy says which sources
    # may carry names in free text, and those sources get manual sampling.
    return out
in : caller John Doe SSN 123-45-6789 phone (212) 555-0147
out: caller_name_hash=9f2c… SSN [REDACTED] phone [REDACTED]
assertions hold: no SSN, phone, email, or plaintext name survives

Three rules, stated plainly. Structured PII (name/phone fields): hashed or dropped by policy, never regexed. Free-text PII (SSN/phone/email patterns): regex redaction, with the documented limitation that names in free text can't be reliably caught — those sources get sampling audits. Placement: redaction runs in memory at the ingestion boundary, before the durable queue — so raw PII never rests on disk, not even briefly. (The first draft drew the queue before redaction. Lisa's rule forbids it. The diagram above is the corrected version.)

And the open discovery question this lesson should have asked sooner: the nightly CSV arrives as a file on disk — raw, with PII, at rest. "Redacted before it rests anywhere" and "the source drops a raw CSV at midnight" are in tension. The architecture can't resolve that alone; it's a contract question for the source owner: do they redact before delivery, or do we agree on an approved secure-ingest boundary (encrypted drop, immediate in-memory sanitize on pickup, no retention of the raw file)? That question goes to Maria before the build starts — not after the audit.

The pollers' credentials follow Post 2's rule: each source gets its own service account from env, short-lived tokens, no human passwords doing machine jobs. The architecture's security story is one sentence: PII is redacted at the door, and every machine knocks with its own identity.

Build it, steps 8–9: observability and tradeoffs

Observability is the 8 AM test: Maria opens one page and knows the system is healthy without reading code. Five metrics, three alerts, one report line:

  • Metrics: queue depth per source, worker lag (newest unprocessed record's age), DLQ count per source, redaction rate, read-API error rate.
  • Alerts: queue depth above 50K records, DLQ spike (schema drift's signature), disk above 80%.
  • The morning report: 50,012 in · 49,988 stored · 24 quarantined · hashes match — Milestone 2's reconciliation, regional.

Tradeoffs — the step most designs skip, and the one the council briefing actually needs [conceptual]:

DecisionChoseRejectedWhy
DatabasePostgresSQLite351 GB, concurrent writers; SQLite was right for one pipeline, wrong for a platform
QueueDurable, per-sourceDirect write187.5 MB buys survival of 30-min outages and slow sources
Redaction pointAt ingestAt readRaw PII never rests; Lisa's rule verbatim
Workers2 + 1 spare1 big boxPeak 41.7 rps vs 150 rps/worker; a spare survives a crash
Legacy sourcePoll, own laneSync chainTheir API only offers polling; its bad days stay its own

A tradeoff table is a decision with its reasoning attached. Six months from now, when someone asks "why Postgres?", the answer is a row in this table — not a memory.

Break it, three ways

1. The 100 GB disk. The arithmetic said 351 GB. The budget meeting said "we have a 100 GB disk, that's plenty." Day 33, 2 AM: the disk fills mid-batch, Postgres stops accepting writes, workers back up into the queue, the queue fills its partition, and the morning report reads ingest failed: disk full. The fix was never technical — it was running the script before the meeting. Every number in step 2 exists so that a future argument starts from arithmetic instead of vibes.

2. The queue-less "simplification." Someone proposes dropping the queue: "the workers are fast, the database is up — why pay for a middleman?" Then Postgres goes down for 30 minutes of maintenance during the evening peak. The math from step 2: 75,000 records with nowhere to go. Without the queue they're dropped (or held in worker memory until the OOM killer arrives — same thing, louder). With it: 187.5 MB on disk, replayed when Postgres returns, morning report green. The queue isn't a component; it's the answer to "what happens during the 30 minutes everything is down."

3. The helpful debug line. A developer adds one log line "for debugging" — the full raw record, PII and all. Lisa runs her check (the real run):

$ grep -oE '[0-9]{3}-[0-9]{2}-[0-9]{4}' app.log
987-65-4321

A real Social Security number, sitting in a log file, greppable by anyone with read access. The fix is two rules, both already in the design: redact at ingest (so there's no raw PII to log), and Post 2's rule — log about the data ("record 99100231 redacted, 3 fields"), never the data. Lisa's grep is the test; it should find nothing, forever.

Productionize: the one-page architecture doc

The decision Maria defends tomorrow isn't a diagram — it's a one-page document. This is its shape, and every section is one of the ten steps:

  1. What we're building, in one paragraph. Regional 311 intake: four sources, one store, console reads it.
  2. The numbers. 50K/hr avg, 150K/hr peak, 3 GB/day, 351 GB for 90 days, 2+1 workers. The script's output, pasted.
  3. The diagram. The data-flow sketch from step 3.
  4. The tradeoffs. The table from step 9 — chose, rejected, why.
  5. What breaks. The failure-modes table from step 6.
  6. The PII story. Redacted at ingest, never in logs, per-source service accounts.
  7. How we'll know it's healthy. The five metrics, three alerts, morning report line.

And three rituals that keep the doc honest after tomorrow:

  • Failure-mode review, monthly. Re-read the table; add the failures the month taught you. The table is a living document or it's fiction.
  • Load test at 2× peak before launch. 83 rps against the workers — the arithmetic says they hold; the test is the receipt.
  • Game day, quarterly. Kill a worker, slow the legacy API, fill the disk to 85% — on purpose, on a Tuesday, with everyone watching. The 2 AM version of these events doesn't schedule itself.

Explain it to the customer

"Maria — the architecture decision, in plain language. Four sources feed one durable queue, each on its own schedule so the county's slow API can't block anyone else. Two workers validate, redact, and write — sized for the evening rush with one spare, not for the average. The database holds 90 days, which is 351 GB — that's the number the budget meeting needs, computed not guessed. PII is redacted the moment it arrives and never touches a log file; Lisa's rule, in code. If anything breaks, the failure-modes table says what happens and what saves us — the queue alone covers a 30-minute outage. Tradeoffs are written down: Postgres over SQLite, queue over direct write, redact at ingest. If the council asks 'why,' every answer is in the doc. And the build order: queue and storage first, workers second, the read API third — the console keeps reading our current data until the cutover."

The pattern, one last variation: no code in the memo, no jargon without a translation, and the build order at the end — because "what do we do Monday" is the question the decision always ends with.

Must know

  • The order: requirements → arithmetic → data flow → components → sync/async → storage → failure modes → security → observability → tradeoffs
  • Do the arithmetic before you draw the boxes — 50K/hr average is not 50K/hr peak
  • Slow dependencies get their own lane; the slowest dependency sets the pace of everything it touches
  • Name failure modes in advance, with blast radius and mitigation — "we'll handle it" is not a mitigation
  • Redact PII at ingest; log about data, never the data
  • Tradeoffs in writing — chose, rejected, why — or the reasoning evaporates in six months

Useful later

  • Formal capacity planning — percentiles, growth curves, headroom policies across a fleet
  • Exactly-once semantics — when idempotent writes aren't enough and you need transactions across systems
  • Multi-region and disaster recovery — when "the database is down" means the building is down
  • Cost modeling — when the 351 GB gets a monthly price tag and someone asks about S3 tiers

Don't memorize this

  • The ten steps as a mantra — remember questions before boxes, arithmetic before opinions; look up the rest
  • Any vendor's rate limits or page sizes — remember manners go in the requirements table; look up the docs
  • Queue depth formulas — remember peak rate × outage window; recompute per system

Where this lands in CityOps

This lesson is the blueprint Milestone 3 builds from. The intake pipeline was one feed and one box; the regional system is four sources, a queue, workers, and a read API — and the framework above is how you get from one to the other without guessing. When the milestone's client interrupts arrive (and they will), the tradeoff table is where the answers already live: every "can't we just…?" gets checked against a written decision instead of relitigated from scratch.

And the signature line: do the arithmetic before you draw the boxes. Post 1's principle was the URL is a noun; the verb is the method — reading the contract before signing it. This is the same instinct, one level up: read the constraints before drawing the system.

Field check

  1. Maria says "fifty thousand records an hour." You ask "average or peak?" — why does the answer change the design, and what did it change here?
  2. The legacy API slows to 20-second timeouts at night. In the sync design, what happens to the Socrata source's data? In the decoupled design?
  3. Lisa's grep finds an SSN in app.log. Name the two design rules that should have prevented it, and where each one lives.
  4. The budget meeting offers a 100 GB disk "because 3 GB a day sounds small." Walk through the arithmetic that refuses.
  5. A teammate proposes dropping the durable queue: "the workers are fast and Postgres is reliable." What's your one-paragraph answer?
What good answers look like

1. Because you size for what the system must survive, not for what's typical — the evening rush runs 3× the average here (150K/hr, 41.7 rps vs 13.9). It changed the worker count (2 sized for peak + 1 spare, instead of 1 that "handles the average"), the queue depth math (75,000 records for a 30-min peak outage), and killed any design that assumed the average. Sizing for the average is how systems die at 7 PM. 2. In the sync design, everything behind the legacy call waits — the Socrata source's pages queue up behind a 20-second stall, and the nightly window blows by 16+ seconds per cycle, compounded per page. In the decoupled design, nothing happens to Socrata's data: the legacy source has its own lane and its own queue partition, so its bad night is its own problem — the measured demo showed the decoupled fetch bounded by the slowest source instead of the sum. 3. (a) Redact at ingest — PII is redacted inside the worker before it rests anywhere, so there's no raw PII in the system to leak into a log; it lives in the worker design (step 7). (b) Log about data, never the data — even redacted-system logs must carry record IDs and counts ("record 99100231 redacted, 3 fields"), never payloads; it lives in the logging discipline (Post 2's rule, applied). One prevents PII existing downstream; the other prevents the log becoming a credential store. 4. 3 GB/day × 90 days retention = 270 GB raw, × 1.3 for indexes and WAL = 351 GB. A 100 GB disk fills on day 33 — mid-batch, at 2 AM, with Postgres refusing writes and the morning report reading "ingest failed: disk full." The arithmetic was computed from a measured 2.5 KB average record; "sounds small" is not a measurement. 5. "The queue isn't a middleman, it's the answer to 'what happens during the 30 minutes Postgres is down.' At peak that's 75,000 records — 187.5 MB on disk, replayed when the database returns, morning report green. Without it those records are dropped or held in worker memory until the OOM killer arrives. It also decouples the sources: the legacy API's bad nights don't block Socrata's pages. 187.5 MB of disk is the cheapest insurance in the design — show me which failure mode you prefer instead."