Debugging through the CityOps 311 pipeline

Prerequisites: The Terminal and Git, Demystified (you'll run scripts from the command line) and Data Formats and Databases in One Sitting (CSV basics). Everything else is taught here.

Every FDE spends a large part of every deployment staring at something that broke. Debugging isn't a talent — it's a procedure. Here's the procedure, practiced on a real pipeline.

Clarify the ask: "it's broken" is not a bug report

Before touching code, pin down the symptom. "It's broken" could mean ten different things, and each one sends you down a different path. Ask:

• What exactly fails? The script crashes? The dashboard shows old numbers? The page won't load?
• When did it start? "Friday it worked" tells you to look at what changed since Friday — new data file, a deploy, a config edit.
• Can you make it happen again? A reliably reproducible failure is easier to investigate and verify. For intermittent failures, collect enough evidence to narrow the conditions under which they occur — an intermittent one without evidence is a ghost story.
• What changed? Start by asking what changed in code, data, configuration, dependencies, load, or operating conditions. If nothing obvious changed, investigate latent assumptions and intermittent conditions — an unchanged system can still fail when a latent bug meets a new input, a resource limit is reached, or a timing condition occurs.

This is the same discipline as Discovery: separate the complaint from the problem. Maria's complaint is "the dashboard is broken." The problem, as you'll discover, is one bad row in a CSV.

The minimum concept: the debugging loop

Debugging is a loop with six steps. Memorize it, because you will run it hundreds of times:

The FDE principle for this lesson:

Build: reproduce the crash

Here is the intake script — the teaching fixture for this lab. It reads the daily 311 CSV and computes the average age of open requests:

import csv

def load_requests(path):
    rows = []
    with open(path) as f:
        reader = csv.DictReader(f)
        for row in reader:
            rows.append({
                "id": row["request_id"],
                "borough": row["borough"],
                "days_open": int(row["days_open"]),
            })
    return rows

requests = load_requests("requests.csv")
print(f"Loaded {len(requests)} requests")
avg = sum(r["days_open"] for r in requests) / len(requests)
print(f"Average days open: {avg:.1f}")

And here is requests.csv — the file the city uploaded this morning:

request_id,borough,days_open,status
SR-10421,Queens,3,open
SR-10422,Brooklyn,7,open
SR-10423,Manhattan,N/A,open
SR-10424,Bronx,12,closed
SR-10425,Queens,5,open

Step one of the loop: reproduce. Run it:

$ python3 intake.py
Traceback (most recent call last):
  File "intake.py", line 15, in <module>
    requests = load_requests("requests.csv")
  File "intake.py", line 11, in load_requests
    "days_open": int(row["days_open"]),
ValueError: invalid literal for int() with base 10: 'N/A'

Good — it crashes on demand. Now step two: read the witness statement.

Read the traceback like a detective

Start with the exception message at the bottom, then walk backward through the traceback frames to understand how execution reached it. The last line is the punchline; the lines above it are the trail:

• Last line first: ValueError: invalid literal for int() with base 10: 'N/A' — the program tried to turn the text 'N/A' into an integer and gave up. That's what it choked on.
• The frames above: line 15 called load_requests("requests.csv"), and inside that, line 11 ran int(row["days_open"]). That's where.
• What's missing: which row. The traceback doesn't say. The witness saw the crime but not the criminal's face. That's normal — the next step is yours.

Hypothesize, then test small

Form one hypothesis: "some row has a non-numeric days_open." Test it with the smallest possible check — don't rewrite the loader yet, just interrogate the data:

import csv

with open("requests.csv", newline="", encoding="utf-8-sig") as f:
    reader = csv.DictReader(f)
    for n, row in enumerate(reader, start=2):  # header is line 1
        try:
            int(row["days_open"])
        except ValueError:
            print(f"Line {n}: request_id={row['request_id']!r}, days_open={row['days_open']!r}")

$ python3 inspect_row.py
Line 4: request_id='SR-10423', days_open='N/A'

Hypothesis confirmed. Line 4 of the CSV — request SR-10423 from Manhattan — has N/A where a number should be. The dashboard "stopped" because the loader crashes on the first bad row and never finishes.

Notice what you did not do: you didn't guess, you didn't rewrite the whole script, and you didn't "fix" it by deleting the row from the CSV by hand (that would just hide this morning's symptom and guarantee a repeat tomorrow).

Break: the error message can point at the symptom, not the cause

That was the easy version — the witness pointed near the crime. Often it doesn't. Here are misdirections you'll meet in the field:

Watch the first one happen. Same script, same files — just run from the wrong folder (create it first):

$ mkdir -p subfolder
$ cd subfolder
$ python3 ../intake.py
Traceback (most recent call last):
  File "../intake.py", line 15, in <module>
    requests = load_requests("requests.csv")
  File "../intake.py", line 5, in load_requests
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'requests.csv'

The witness says "no such file." The verdict: nothing is wrong with the file or the code — you are standing in the wrong room. open("requests.csv") looks in the process's current working directory, not necessarily beside intake.py. When an error makes no sense, widen the interrogation: what directory am I in, which Python is this, what changed since it last worked? (This is why knowing your terminal matters — half of debugging is orienting yourself.)

The later improvement, once you've felt the pain: anchor the path to the script's own directory instead of the caller's:

from pathlib import Path
DATA_FILE = Path(__file__).resolve().parent / "requests.csv"

This removes the accidental dependence on where the user launches the script.

Productionize: fix the cause, log the evidence

Now the fix. The root cause isn't "line 4 is bad" — it's "the loader assumes every row is clean, and real-world data never is." (You met messy real-world files in Data Formats and Databases; this is what you do about them.)

But hold on — before writing code, there's a decision here that isn't yours alone to make. Quarantining the bad row is a business policy, not just a technical fix. So you call Dev from the 311 team:

With Dev's approval, fix the assumption, not the row:

import csv
import json
import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")

def load_requests(path):
    rows, quarantined = [], []
    with open(path, newline="", encoding="utf-8-sig") as f:
        reader = csv.DictReader(f)
        for n, row in enumerate(reader, start=2):
            try:
                days = int(row["days_open"])
            except ValueError:
                logging.warning("line %d: quarantined %r (days_open=%r)",
                                n, row["request_id"], row["days_open"])
                quarantined.append(row)
                continue
            rows.append({"id": row["request_id"],
                         "borough": row["borough"],
                         "status": row["status"],
                         "days_open": days})
    return rows, quarantined

requests, quarantined = load_requests("requests.csv")

# Persist the quarantine so it survives the program exit
with open("quarantine.jsonl", "w", encoding="utf-8") as f:
    for row in quarantined:
        f.write(json.dumps(row) + "\n")

total = len(requests) + len(quarantined)
if total == 0:
    raise RuntimeError("empty input batch")
# Illustrative threshold, not a CityOps recommendation: the real
# number must be agreed with the data owner, and critical-field
# failures may need stricter rules regardless of percentage.
# (Set high enough that 1 bad row in a 5-row demo publishes with
# a warning; a 95%-bad batch still fails loudly.)
if len(quarantined) / total > 0.50:
    raise RuntimeError("quarantine threshold exceeded")

# The metric, defined explicitly: average age of currently OPEN
# requests with valid days_open. Closed requests are excluded —
# they measure completed lifecycle, not current backlog age.
# (Note: this assumes the source's days_open means elapsed age;
# confirm what the field actually measures before using it as a KPI.)
open_requests = [r for r in requests if r["status"] == "open"]
avg = sum(r["days_open"] for r in open_requests) / len(open_requests)
print(f"Loaded {len(requests)} valid records; quarantined {len(quarantined)}")
print(f"Open requests included in age metric: {len(open_requests)}")
print(f"Average age of valid open requests: {avg:.1f} days")

$ python3 intake_fixed.py
WARNING: line 4: quarantined 'SR-10423' (days_open='N/A')
Loaded 4 valid records; quarantined 1
Open requests included in age metric: 3
Average age of valid open requests: 5.0 days

The newline="" and encoding="utf-8-sig" are one-line insurance: they handle Windows line endings and strip a BOM if the city's export tool adds one — the same BOM that can cause a phantom KeyError: 'borough'.

Three decisions in that fix, all deliberate:

• Quarantine durably, don't silently drop. The bad row is logged, kept aside in the quarantined list, and written to quarantine.jsonl — Maria can see exactly which request needs a real value, even after the program exits. (Production pipelines also need run IDs, timestamps, secure storage, retention rules, and controlled access; a small persistent file is enough for this exercise.) Silently skipping the row would make the numbers wrong without anyone knowing.
• Log, don't print-debug-and-delete. logging.warning stays in the code and shows up in server logs at 2 AM. A print("HERE!!!") you delete tomorrow helps nobody. A useful production log answers three questions: what failed, on which input or entity, and what the system did about it — our warning names the line, the request ID, the bad value, and the quarantine decision.
• Catch only what you can handle. The try/except wraps exactly one conversion, and the handler does something useful: records the bad row and moves on. This version prevents non-numeric days_open values from aborting the batch — but a missing days_open column, a renamed header, or a None value raising TypeError would still fail. Other schema and data-quality failures require separate validation. Contrast that with the classic sin:

# Don't silently swallow broad exceptions.
try:
    process(row)
except Exception:
    pass   # the bug is now invisible AND still there

The sin isn't the broad except itself — at a top-level worker or process boundary, catching broadly to log, report, and shut down cleanly is legitimate. The sin is the silent pass: no log, no record, no signal. If you catch broadly, you must do something visible with it.

And one more assumption to check: what if every row is bad? The loader would quarantine everything, then crash dividing by zero:

$ python3 intake_fixed.py   # with a file where all rows have N/A
WARNING: line 2: quarantined 'SR-10001' (days_open='N/A')
WARNING: line 3: quarantined 'SR-10002' (days_open='N/A')
Traceback (most recent call last):
  File "intake_fixed.py", line 25, in <module>
    raise RuntimeError("no valid requests loaded")
RuntimeError: no valid requests loaded

That's the if not requests: raise RuntimeError(...) guard doing its job — failing loudly instead of crashing mysteriously on ZeroDivisionError. Recovery from one failure mode can expose the next assumption; check for it deliberately.

try:
    days = int(row["days_open"])
except ValueError as e:
    raise ValueError(f"bad days_open in row {row['request_id']!r}") from e

$ python3 context_demo.py
SR-10421 3
SR-10422 7
Traceback (most recent call last):
  File "context_demo.py", line 8, in process
    days = int(row["days_open"])
ValueError: invalid literal for int() with base 10: 'N/A'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "context_demo.py", line 13, in <module>
    for rid, days in process("requests.csv"):
  File "context_demo.py", line 10, in process
    raise ValueError(f"bad days_open in row {row['request_id']!r}") from e
ValueError: bad days_open in row 'SR-10423'

The original error is preserved (Python chains it with from e), and the new message names the row. That's catching with purpose.

Final loop step: verify. "It ran without crashing" is not verification. Separate what the script proves from what a production recovery would require.

Lab verification (demonstrated here):

• The original failing input now completes end to end.
• Exactly the 4 expected valid rows loaded.
• Exactly the 1 expected row (SR-10423) quarantined, named in the log and present in quarantine.jsonl.
• The metric matches the hand calculation: 5.0 days across 3 valid open requests, 1 quarantined — a number with its context, not a naked average.

Production verification (not implemented here):

• Output is written to the destination the dashboard reads.
• The dashboard reads the new output and its freshness timestamp advances.
• Maria can view the updated page from her environment.

Only after those additional checks could the incident report honestly say "dashboard updating."

One more thing a fix should leave behind — the smallest possible regression check, shown here as a preview of the testing lesson to come:

def test_bad_days_open_is_quarantined():
    rows, bad = load_requests("requests.csv")
    assert len(rows) == 4
    assert len(bad) == 1
    assert bad[0]["request_id"] == "SR-10423"

No framework needed for the idea: a bug fix is more trustworthy when the failure becomes a repeatable test. The full testing lesson arrives later; the habit starts now.

A fix you haven't re-tested against expectations is a hypothesis, not a fix.

Communicate: the bug report Maria actually needs

You're the FDE on site — Maria needs to tell the mayor's office what happened, and your team needs to know what changed. Write it up in four lines:

• Symptom: dashboard froze; intake script crashed with ValueError on this morning's CSV. (Last successful intake Friday 06:02; Tuesday intake FAILED; dashboard timestamp Friday 06:02.)
• Root cause: row SR-10423 contained days_open=N/A; the loader assumed every days_open value was numeric. (Source cause — why the upstream system emitted N/A — still being investigated. The failure mechanism is confirmed; the upstream origin is not.)
• Fix: loader now quarantines bad rows with a logged warning and a durable quarantine.jsonl instead of crashing; SR-10423 flagged for a real value. (Quarantine-vs-fail policy confirmed with Dev; illustrative 50% batch-fail threshold added.)
• Verified: re-ran on today's file — 4 valid rows loaded, 1 quarantined (SR-10423 in log and quarantine file); average age 5.0 days across 3 valid open requests. Dashboard recovery pending the production verification checks above.

Notice the shape: symptom, cause, fix, proof. That's the same shape as a good problem statement — and it's what makes people trust you with the next outage.

CityOps: put it together

Back to Tuesday morning. Here's the full play in order:

1. Reproduce: ran intake.py — crashed on demand. Not intermittent, not the dashboard, not the network.
2. Read: bottom-up traceback — ValueError on int('N/A') at line 11. Witness: bad value. Missing: which row.
3. Hypothesize & test: one bad row — confirmed with a 10-line probe: line 4, SR-10423.
4. Fix the cause: loader quarantines bad rows with logging and a durable quarantine file instead of assuming clean data; metric defined as average age of valid open requests.
5. Verify: re-ran against expectations — 4 valid rows loaded, SR-10423 quarantined and logged, metric 5.0 days across 3 valid open requests. Called Maria with the four-line report.

The important part wasn't speed; it was that every step reduced uncertainty. No guessing, no rewriting the world, no 2 AM heroics. That's what "experience the job before you get the job" feels like — the job is mostly this loop, run calmly.

A good FDE doesn't stop when the exception disappears. They explain why it happened, verify the expected business result, and leave behind evidence that the same failure will be detected — or handled safely — next time.