Debugging Like a Detective: Reading Errors, Logs & Finding the Real Cause

Prerequisites: The Terminal and Git, Demystified (you'll run scripts from the command line) and Data Formats and Databases in One Sitting (CSV basics). Everything else is taught here.

Every FDE spends a large part of every deployment staring at something that broke. Debugging isn't a talent — it's a procedure. Here's the procedure, practiced on a real pipeline.

Clarify the ask: "it's broken" is not a bug report

Before touching code, pin down the symptom. "It's broken" could mean ten different things, and each one sends you down a different path. Ask:

• What exactly fails? The script crashes? The dashboard shows old numbers? The page won't load?
• When did it start? "Friday it worked" tells you to look at what changed since Friday — new data file, a deploy, a config edit.
• Can you make it happen again? A bug you can reproduce on demand is a bug you can fix. An intermittent one is a ghost story.
• What changed? New data, new code, new machine, new user input. Something changed; your job is to find what.

This is the same discipline as Discovery: separate the complaint from the problem. Maria's complaint is "the dashboard is broken." The problem, as you'll discover, is one bad row in a CSV.

The minimum concept: the debugging loop

Debugging is a loop with six steps. Memorize it, because you will run it hundreds of times:

The FDE principle for this lesson:

Build: reproduce the crash

Here is the intake script, close to what Milestone 1 produced. It reads the daily 311 CSV and computes the average days a request stays open:

import csv

def load_requests(path):
    rows = []
    with open(path) as f:
        reader = csv.DictReader(f)
        for row in reader:
            rows.append({
                "id": row["request_id"],
                "borough": row["borough"],
                "days_open": int(row["days_open"]),
            })
    return rows

requests = load_requests("requests.csv")
print(f"Loaded {len(requests)} requests")
avg = sum(r["days_open"] for r in requests) / len(requests)
print(f"Average days open: {avg:.1f}")

And here is requests.csv — the file the city uploaded this morning:

request_id,borough,days_open,status
SR-10421,Queens,3,open
SR-10422,Brooklyn,7,open
SR-10423,Manhattan,N/A,open
SR-10424,Bronx,12,closed
SR-10425,Queens,5,open

Step one of the loop: reproduce. Run it:

$ python3 intake.py
Traceback (most recent call last):
  File "intake.py", line 15, in <module>
    requests = load_requests("requests.csv")
  File "intake.py", line 11, in load_requests
    "days_open": int(row["days_open"]),
ValueError: invalid literal for int() with base 10: 'N/A'

Good — it crashes on demand. Now step two: read the witness statement.

Read the traceback like a detective

Start with the exception message at the bottom, then walk backward through the traceback frames to understand how execution reached it. The last line is the punchline; the lines above it are the trail:

• Last line first: ValueError: invalid literal for int() with base 10: 'N/A' — the program tried to turn the text 'N/A' into an integer and gave up. That's what it choked on.
• The frames above: line 15 called load_requests("requests.csv"), and inside that, line 11 ran int(row["days_open"]). That's where.
• What's missing: which row. The traceback doesn't say. The witness saw the crime but not the criminal's face. That's normal — the next step is yours.

Hypothesize, then test small

Form one hypothesis: "some row has a non-numeric days_open." Test it with the smallest possible check — don't rewrite the loader yet, just interrogate the data:

import csv

with open("requests.csv", newline="", encoding="utf-8-sig") as f:
    reader = csv.DictReader(f)
    for n, row in enumerate(reader, start=2):  # header is line 1
        try:
            int(row["days_open"])
        except ValueError:
            print(f"Line {n}: request_id={row['request_id']!r}, days_open={row['days_open']!r}")

$ python3 inspect_row.py
Line 4: request_id='SR-10423', days_open='N/A'

Hypothesis confirmed. Line 4 of the CSV — request SR-10423 from Manhattan — has N/A where a number should be. The dashboard "stopped" because the loader crashes on the first bad row and never finishes.

Notice what you did not do: you didn't guess, you didn't rewrite the whole script, and you didn't "fix" it by deleting the row from the CSV by hand (that would just hide this morning's symptom and guarantee a repeat tomorrow).

Break: the error message can point at the symptom, not the cause

That was the easy version — the witness pointed near the crime. Often it doesn't. Here are misdirections you'll meet in the field:

Watch the first one happen. Same script, same files — just run from the wrong folder:

$ cd subfolder
$ python3 ../intake.py
Traceback (most recent call last):
  File "../intake.py", line 15, in <module>
    requests = load_requests("requests.csv")
  File "../intake.py", line 5, in load_requests
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'requests.csv'

The witness says "no such file." The verdict: nothing is wrong with the file or the code — you are standing in the wrong room. When an error makes no sense, widen the interrogation: what directory am I in, which Python is this, what changed since it last worked? (This is why knowing your terminal matters — half of debugging is orienting yourself.)

Productionize: fix the cause, log the evidence

Now the fix. The root cause isn't "line 4 is bad" — it's "the loader assumes every row is clean, and real-world data never is." (You met messy real-world files in Data Formats and Databases; this is what you do about them.)

But hold on — before writing code, there's a decision here that isn't yours alone to make. Quarantining the bad row is a business policy, not just a technical fix. So you call Dev from the 311 team:

With Dev's approval, fix the assumption, not the row:

import csv
import logging

logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")

def load_requests(path):
    rows, quarantined = [], []
    with open(path, newline="", encoding="utf-8-sig") as f:
        reader = csv.DictReader(f)
        for n, row in enumerate(reader, start=2):
            try:
                days = int(row["days_open"])
            except ValueError:
                logging.warning("line %d: quarantined %r (days_open=%r)",
                                n, row["request_id"], row["days_open"])
                quarantined.append(row)
                continue
            rows.append({"id": row["request_id"],
                         "borough": row["borough"],
                         "days_open": days})
    return rows, quarantined

requests, quarantined = load_requests("requests.csv")
if not requests:
    raise RuntimeError("no valid requests loaded")
print(f"Loaded {len(requests)} requests, quarantined {len(quarantined)}")
avg = sum(r["days_open"] for r in requests) / len(requests)
print(f"Average days open: {avg:.1f} across {len(requests)} valid requests; "
      f"{len(quarantined)} request quarantined")

$ python3 intake_fixed.py
WARNING: line 4: quarantined 'SR-10423' (days_open='N/A')
Loaded 4 requests, quarantined 1
Average days open: 6.8 across 4 valid requests; 1 request quarantined

The newline="" and encoding="utf-8-sig" are one-line insurance: they handle Windows line endings and strip a BOM if the city's export tool adds one — the same BOM that can cause a phantom KeyError: 'borough'.

Three decisions in that fix, all deliberate:

• Quarantine, don't silently drop. The bad row is logged and kept aside — Maria can see exactly which request needs a real value. Silently skipping it would make the numbers wrong without anyone knowing.
• Log, don't print-debug-and-delete. logging.warning stays in the code and shows up in server logs at 2 AM. A print("HERE!!!") you delete tomorrow helps nobody. A useful production log answers three questions: what failed, on which input or entity, and what the system did about it — our warning names the line, the request ID, the bad value, and the quarantine decision.
• Catch only what you can handle. The try/except wraps exactly one conversion, and the handler does something useful: records the bad row and moves on. Contrast that with the classic sin:

# Don't silently swallow broad exceptions.
try:
    process(row)
except Exception:
    pass   # the bug is now invisible AND still there

The sin isn't the broad except itself — at a top-level worker or process boundary, catching broadly to log, report, and shut down cleanly is legitimate. The sin is the silent pass: no log, no record, no signal. If you catch broadly, you must do something visible with it.

And one more assumption to check: what if every row is bad? The loader would quarantine everything, then crash dividing by zero:

$ python3 intake_fixed.py   # with a file where all rows have N/A
WARNING: line 2: quarantined 'SR-10001' (days_open='N/A')
WARNING: line 3: quarantined 'SR-10002' (days_open='N/A')
Traceback (most recent call last):
  File "intake_fixed.py", line 25, in <module>
    raise RuntimeError("no valid requests loaded")
RuntimeError: no valid requests loaded

That's the if not requests: raise RuntimeError(...) guard doing its job — failing loudly instead of crashing mysteriously on ZeroDivisionError. Recovery from one failure mode can expose the next assumption; check for it deliberately.

try:
    days = int(row["days_open"])
except ValueError as e:
    raise ValueError(f"bad days_open in row {row['request_id']!r}") from e

$ python3 context_demo.py
SR-10421 3
SR-10422 7
Traceback (most recent call last):
  File "context_demo.py", line 8, in process
    days = int(row["days_open"])
ValueError: invalid literal for int() with base 10: 'N/A'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "context_demo.py", line 13, in <module>
    for rid, days in process("requests.csv"):
  File "context_demo.py", line 10, in process
    raise ValueError(f"bad days_open in row {row['request_id']!r}") from e
ValueError: bad days_open in row 'SR-10423'

The original error is preserved (Python chains it with from e), and the new message names the row. That's catching with purpose.

Final loop step: verify. "It ran without crashing" is not verification. Check the specific expectations:

• The original failing input now completes end to end.
• Exactly the 4 expected valid rows loaded.
• Exactly the 1 expected row (SR-10423) quarantined, named in the log.
• The metric matches the hand calculation: 6.8 across 4 valid requests, 1 quarantined — a number with its context, not a naked average.
• The dashboard consumes the new output and shows fresh numbers.

A fix you haven't re-tested against expectations is a hypothesis, not a fix.

Communicate: the bug report Maria actually needs

You're the FDE on site — Maria needs to tell the mayor's office what happened, and your team needs to know what changed. Write it up in four lines:

• Symptom: dashboard froze; intake script crashed with ValueError on this morning's CSV.
• Root cause: row SR-10423 contained days_open=N/A; the loader assumed every days_open value was numeric. (Source cause — why the upstream system emitted N/A — still being investigated. The failure mechanism is confirmed; the upstream origin is not.)
• Fix: loader now quarantines bad rows with a logged warning instead of crashing; SR-10423 flagged for a real value. (Quarantine-vs-fail policy confirmed with Dev.)
• Verified: re-ran on today's file — 4 valid rows loaded, 1 quarantined; average 6.8 days across 4 valid requests with 1 quarantined; dashboard updating.

Notice the shape: symptom, cause, fix, proof. That's the same shape as a good problem statement — and it's what makes people trust you with the next outage.

CityOps: put it together

Back to Tuesday morning. Here's the full play in order:

1. Reproduce: ran intake.py — crashed on demand. Not intermittent, not the dashboard, not the network.
2. Read: bottom-up traceback — ValueError on int('N/A') at line 11. Witness: bad value. Missing: which row.
3. Hypothesize & test: one bad row — confirmed with a 10-line probe: line 4, SR-10423.
4. Fix the cause: loader quarantines bad rows with logging instead of assuming clean data.
5. Verify: re-ran against expectations — 4 valid rows loaded, SR-10423 quarantined and logged, metric 6.8 across 4 valid with 1 quarantined, dashboard live. Called Maria with the four-line report.

The important part wasn't speed; it was that every step reduced uncertainty. No guessing, no rewriting the world, no 2 AM heroics. That's what "experience the job before you get the job" feels like — the job is mostly this loop, run calmly.