Every team touching the 311 feed has written its own client by now — Tom's has the pagination loop, the sanitation analysts' has a different one, and the vendor's own SDK is a thin wrapper that turns every failure into None. Meanwhile Dev has spent the morning staring at an empty result set, convinced the API is broken. Two problems, one lesson: build the client library you wish the vendor shipped, and learn the debugging method that finds the real bug — which is usually yours.
Assumes: Stage 3 Post 1 (status codes, the retry rule), Post 3 (pagination, Retry-After, jitter), Python Post 7 (scripted fakes). Installs: pip install requests.
Monday, 9:20 AM. Dev, without looking up: "The 311 API is broken. Zero Brooklyn complaints. I've been staring at it for an hour."
You: "Show me the request."
Dev: "It's not the request, the endpoint is returning nothing — status 200, empty list. The feed must be down."
Maria (arriving with a deadline): "I need the Brooklyn open-complaints count for the council briefing at noon. Is it the API or is it us?"
And while Dev debugs, Tom's team is on their fourth copy of the same pagination loop — each one with slightly different retry behavior, none of them tested. The vendor never shipped a real SDK, so every team invented one. Badly.
Before you code: clarify the ask
You: "Dev — before 'the API is broken': what did you send, exactly? Show me the raw request."
Dev: "…I'll pull it up."
You: "And Maria — the briefing number. If the API is fine and it's our query, do you want the number or the postmortem?"
Maria: "The number. Then the postmortem."
You: "Then the deliverables are two: first, the number — by debugging the actual request instead of theorizing about the API. Second, one shared client every team uses, so we stop debugging four different copies of the same loop."
Input: Dev's empty result set and four hand-rolled 311 clients
Output: the real root cause before noon, plus one honest SDK every team adopts
Deadline: the council briefing
Notice the split: half of this lesson is a method (debugging someone else's API), half is an artifact (the SDK). The method finds today's bug; the artifact prevents next month's four copies of it.
The minimal concept
Two ideas, and the second one is the whole post:
An SDK is a translation layer: wire format in, domain language out. The vendor speaks $select, $offset, and query.soql.no-such-column. Your teams should speak iter_requests(borough="BROOKLYN"), "unknown column 'bogus_field'", and a for loop. A good SDK hides mechanics — pagination, retries, auth, encoding — and translates errors into the domain's vocabulary. A bad SDK hides the errors too, and that's worse than no SDK at all.
Debugging someone else's API is a method, not a talent. Four steps, in order: (1) reproduce it smallest — a script under twenty lines that shows the wrong behavior; (2) read the raw request and the raw response — the URL, the status, the body, not your parsed interpretation of them; (3) change one variable at a time until the behavior changes; (4) write the ticket that gets fixed — what you sent, what you expected, what you got, the minimal repro. Steps 1–3 find the bug. Step 4 is what makes the vendor fix theirs.
Build it, part 1: the SDK we wish existed
One class, one file. The public surface is two methods — everything else is private machinery:
"""The client library we wish the vendor shipped."""
import logging
import math
import random
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
import requests
MAX_WAIT = 300 # longest we'll honor from a Retry-After before failing loud
def parse_retry_after(value):
"""Seconds or HTTP-date → seconds; None if absent/invalid."""
if value is None:
return None
value = value.strip()
try:
seconds = float(value)
return seconds if math.isfinite(seconds) and seconds >= 0 else None
except ValueError:
pass
try:
dt = parsedate_to_datetime(value)
delta = (dt - datetime.now(timezone.utc)).total_seconds()
return max(0.0, delta) if math.isfinite(delta) else None
except (ValueError, TypeError):
return None
BASE = "https://data.cityofnewyork.us/resource/erm2-nwe9.json"
RETRYABLE = {429, 500, 502, 503, 504}
log = logging.getLogger("sdk311")
class SDKError(Exception):
"""Every failure names the vendor, the call, and what the caller
should do. No bare HTTPError leaks through."""
class NotFoundError(SDKError):
"""The row you asked for does not exist."""
class BadQueryError(SDKError):
"""Your query was wrong — bad column, bad syntax. Fix the request."""
class VendorError(SDKError):
"""The vendor failed after retries. Their problem, your pager."""
class ThreeOneOne:
"""311 client: pagination hidden inside iteration, vendor errors
translated into the domain's language, retries baked in."""
# Columns the SDK may select or filter on. SoQL has no parameter
# binding for expressions, so the allowlist is the injection defense:
# only these names ever reach the query string.
ALLOWED_FIELDS = {"unique_key", "created_at", "agency", "complaint_type",
"borough", "descriptor", "status"}
def __init__(self, session=None, page_size=1000, max_retries=4):
if page_size <= 0:
raise ValueError("page_size must be positive")
if max_retries < 1:
raise ValueError("max_retries must be at least 1")
# NB: `session or ...` would discard a session object whose
# truthiness is False — use an explicit None check.
self.session = session if session is not None else requests.Session()
self.page_size = page_size
self.max_retries = max_retries
@staticmethod
def _lit(value):
"""SoQL string literal: single quotes are doubled. Values are still
validated by type before this — escaping is the last line, not the
first. Dates must already be ISO strings; callers pass data, never
query fragments."""
if not isinstance(value, str):
raise BadQueryError(f"filter value must be a string, got {type(value).__name__}")
return "'" + value.replace("'", "''") + "'"
# -- the one place that touches the network ---------------------------
def _get(self, params):
delay = 1.0
for attempt in range(1, self.max_retries + 1):
try:
r = self.session.get(BASE, params=params, timeout=(5, 30))
except (requests.ConnectionError, requests.Timeout) as e:
if attempt == self.max_retries:
raise VendorError(f"311 feed unreachable: {e}") from e
self._wait(delay, attempt, "no response")
delay = min(delay * 2, 30)
continue
if r.status_code == 200:
try:
data = r.json()
except ValueError as e:
raise VendorError(
f"311 feed returned 200 with malformed JSON: "
f"{r.text[:200]!r}") from e
if not isinstance(data, list):
raise VendorError(
f"unexpected response shape (expected list): "
f"{str(data)[:200]!r}")
return data
if r.status_code in RETRYABLE and attempt < self.max_retries:
self._wait(self._delay_for(r, delay, attempt),
attempt, r.status_code)
delay = min(delay * 2, 30)
continue
raise self._translate(r)
@staticmethod
def _delay_for(resp, delay, attempt):
# Retry-After can be seconds OR an HTTP-date. Parse both; reject
# garbage. (Same parser as the pagination lesson — one shared helper
# would be better, which is the SDK's next refactor.)
retry_after = resp.headers.get("Retry-After")
if retry_after is not None:
parsed = parse_retry_after(retry_after)
if parsed is not None:
return min(parsed, MAX_WAIT)
return delay * (0.5 + random.random()) # our guess: jitter it
@staticmethod
def _wait(delay, attempt, status):
log.warning("311 feed -> %s: retrying in %.1fs (attempt %d)",
status, delay, attempt)
time.sleep(delay)
@staticmethod
def _translate(resp):
body = resp.text[:200]
msg = ""
try:
msg = resp.json().get("message", "")
except Exception:
pass
if "no-such-column" in msg:
col = msg.split("No such column:")[-1].split(";")[0].strip()
return BadQueryError(
f"unknown column {col!r} — check $select/$where")
if resp.status_code == 400:
return BadQueryError(f"bad query: {msg or body}")
if resp.status_code == 404:
return NotFoundError("311 dataset not found (wrong resource id?)")
return VendorError(f"311 feed failed: HTTP {resp.status_code}: {body}")
# -- the public surface ------------------------------------------------
def iter_requests(self, *, borough=None, agency=None, start=None,
end=None, fields=None):
"""Yield 311 rows. Pagination is hidden: this is just a for loop.
Best-effort iteration, not verified extraction: the feed can change
between pages (Post 3's moving-boundary lesson), so this makes no
completeness promise. For reconciliation-critical work, use a bounded
window with a source count, dedupe, and explicit completeness checks
— a simple generator cannot certify that it saw everything."""
fields = fields or ["unique_key", "created_date", "agency",
"complaint_type", "borough"]
unknown = set(fields) - self.ALLOWED_FIELDS
if unknown:
raise BadQueryError(f"unknown fields: {sorted(unknown)}")
clauses = []
if borough is not None:
# the feed stores BROOKLYN, not brooklyn — the SDK absorbs it
clauses.append(f"borough={self._lit(borough.upper())}")
if agency is not None:
clauses.append(f"agency={self._lit(agency)}")
if start is not None:
clauses.append(f"created_date >= {self._lit(start + 'T00:00:00')}")
if end is not None:
clauses.append(f"created_date < {self._lit(end + 'T00:00:00')}")
params = {"$select": ",".join(fields),
"$order": "created_date,unique_key,:id",
"$limit": str(self.page_size)}
if clauses:
params["$where"] = " AND ".join(clauses)
offset = 0
while True:
params["$offset"] = str(offset)
rows = self._get(params)
if not rows:
return
yield from rows
offset += len(rows)
if len(rows) < self.page_size:
return
def get_request(self, unique_key):
rows = self._get({"$select": "unique_key,created_date,agency,"
"complaint_type,borough",
"$where": f"unique_key={self._lit(unique_key)}",
"$limit": "1"})
if not rows:
raise NotFoundError(
f"no 311 request with unique_key={unique_key!r}")
return rows[0]
def count_requests(self, *, borough=None, agency=None, status=None):
"""Source-side count via $select=count(*) — the number comes from
the vendor, not from draining a paginated stream. (Counting the
stream would inherit the moving-boundary problem; the count query
is atomic from the source's side.)"""
clauses = []
if borough is not None:
clauses.append(f"borough={self._lit(borough.upper())}")
if agency is not None:
clauses.append(f"agency={self._lit(agency)}")
if status is not None:
clauses.append(f"status={self._lit(status)}")
params = {"$select": "count(*)"}
if clauses:
params["$where"] = " AND ".join(clauses)
rows = self._get(params)
try:
return int(rows[0]["count"])
except (IndexError, KeyError, ValueError, TypeError) as e:
raise VendorError(
f"unexpected count response shape: {rows!r}") from e
Read the design decisions before the output. _get is the only method that touches the network — one place for retries, one place for error translation, one place to log. The public methods never see an HTTP status code; they see rows or domain exceptions. The $order carries the :id tiebreaker from Post 3's moving-page-boundary war story. And borough.upper() is the SDK absorbing a vendor quirk so no caller ever has to — the same instinct as Post 5's API translating brooklyn to BROOKLYN for its own clients. Contracts absorb differences; they don't leak them.
Build it, part 2: the runs
Two days of the real feed, pagination invisible to the caller — and note the first line, because the demo got throttled for real:
WARNING 311 feed -> 429: retrying in 1.0s (attempt 1)
rows: 19500
first key: 70480014 | last key: 70497065
sample: NYPD - Illegal Parking
19,500 rows across twenty pages, and the caller wrote a for loop. The vendor 429'd the very first page — the SDK honored Retry-After: 1 to the millisecond and kept going. That warning line is the retry machinery from Post 3 earning its keep in someone else's codebase, which is exactly where SDKs live or die.
The quirk absorption, the translated errors — all real runs:
# lowercase in, uppercase stored — the caller never knows
>>> list(sdk.iter_requests(borough="brooklyn", start="2026-09-20",
... end="2026-09-22"))[:2]
rows returned: 2 | borough field: BROOKLYN
# known key: the row. unknown key: a named error, not an empty list
>>> sdk.get_request("70480014")["complaint_type"]
'Illegal Parking'
>>> sdk.get_request("no-such-key")
NotFoundError: no 311 request with unique_key='no-such-key'
# bogus field: the vendor's SoQL error, translated into the domain
>>> list(sdk.iter_requests(start="2026-09-20", end="2026-09-21",
... fields=["unique_key", "bogus_field"]))
BadQueryError: unknown column 'bogus_field' — check $select/$where
And the retry path, proven without angering the vendor — Post 7's scripted-fake pattern, a fake session playing [429 + Retry-After: 1, 200] (SIMULATED network, MEASURED timings):
recovered after 2 calls, total 1.0s, rows=[{'unique_key': 'abc'}]
Break it, three ways
1. The silent [] — Dev's "broken API," debugged live. Back to 9:20 AM. Here's the method, applied to the actual bug. Step one, the smallest repro — twelve lines, no SDK, no app code:
import requests
BASE = "https://data.cityofnewyork.us/resource/erm2-nwe9.json"
for borough in ["brooklyn", "BROOKLYN"]:
r = requests.get(BASE, params={"$select": "unique_key",
"$where": f"borough='{borough}'",
"$limit": "3"}, timeout=(5, 30))
print(f"borough='{borough}' -> status {r.status_code}, rows: {r.text[:80]}")
borough='brooklyn' -> status 200, rows: []
borough='BROOKLYN' -> status 200, rows: [{"unique_key":"67658444"}
,{"unique_key":"67660221"}
,{"unique_key":"67660222"}]
Step two, read the raw response: status 200, body []. Not an error — an answer. The API did exactly what it was asked: "rows where borough equals 'brooklyn'" — and the feed stores BROOKLYN. Step three, change one variable: the case. Behavior changes. Root cause found before 10 AM, and it's ours, not the vendor's. The API was never broken; the query was. That's the usual ending to "the API is broken" — which is why the method starts with your own request, not their server.
2. The SDK that lies. This is the client library the vendor actually shipped — a thin wrapper that "never bothers the caller with errors." Real run:
class BadSDK:
def get_request(self, unique_key):
try:
r = requests.get(BASE, params={...}, timeout=(5, 30))
rows = r.json()
return rows[0] if rows else None
except Exception:
return None # "never bother the caller with errors"
missing key -> None
vendor down -> None
Same None for "not found" and "the vendor is on fire." The caller cannot distinguish them, so it treats them identically — which means a vendor outage looks exactly like an empty result set, and Maria's briefing gets a confident zero instead of a paged engineer. Hiding the wire is the SDK's job; hiding the error is sabotage. Every exception our SDK raises carries the vendor, the call, and what to do — because the error is the most important part of the interface.
3. The ticket that gets ignored. Dev's first draft to the vendor's support desk:
Subject: 311 API broken
Body: "Hi, the API is returning no data for Brooklyn. Please fix ASAP, we have a briefing at noon."
That ticket gets a bot reply asking for the request details — which Dev didn't include because he was in a hurry. The ticket that gets fixed looks like this:
Subject: $where borough='brooklyn' returns [] while borough='BROOKLYN'
returns rows — is borough matching case-sensitive by design?
What I sent:
GET /resource/erm2-nwe9.json?$select=unique_key
&$where=borough='brooklyn'&$limit=3
What I expected: Brooklyn rows (or a 400 telling me the value is invalid)
What I got: HTTP 200 with body []
Minimal repro: (the 12-line script above — pasted verbatim)
Environment: requests 2.34.2, no app token, 2026-10-03 ~10:00 UTC
Question: is case-sensitive matching intended? If so, documenting
it would have saved us an hour.
The difference isn't politeness — it's answerability. The first ticket asks the vendor to debug your code. The second hands them everything needed to answer in one reply, and asks the one question only they can answer. (In our case the answer turned out to be "yes, it's case-sensitive, and the fix lives in our SDK" — which is why the ticket also documents the hour it cost, so the docs gap gets fixed for the next team.)
Productionize: the SDK as a dependency
Five habits turn the class into something four teams can share:
Version it and pin it. The SDK is a dependency like any other — it gets a version number, a changelog, and a pin in every team's requirements.txt (Post 8's rule). When the vendor changes something and the SDK adapts, teams upgrade deliberately — not by surprise at 2 AM.
Log the wire at debug, with the run id. Every request logs method, URL, and status at DEBUG; failures log at WARNING with the translated error attached. Tagged with the run id so the 6:20 rerun's lines don't interleave with the 6:00 attempt's. The SDK never logs response bodies at INFO — bodies can carry PII, and logs are forever.
Test it with fakes, not the vendor. Post 7's pattern: a scripted fake session plays 429s, 400s, and truncated pages, and the suite asserts the SDK retries, translates, and iterates. The real feed appears in exactly one slow, marked test — because a suite that needs the vendor is a suite that fails when the vendor sneezes.
Don't expose the raw session. The moment a caller reaches past the SDK to sdk.session.get(...), the translation layer has a hole in it — errors leak untranslated, retries get bypassed, and the next vendor change breaks that caller first. If a use case genuinely needs the raw wire, that's a missing SDK method, not a reason to break the abstraction.
Changelog the vendor, not just the code. When the vendor deprecates a field or tightens a limit, that goes in the SDK's changelog in plain language — "the feed now caps pages at 50k; iter_requests unchanged" — so the four teams learn it once, from you, instead of four times from their own 2 AM pages.
Explain it to the customer
Two handoffs. First, Maria's number — the debugging method did its job before noon:
"Maria — the API was never broken. Our query asked for borough='brooklyn' and the feed stores BROOKLYN — 200 OK with an empty list, which is why it looked like an outage. The briefing number, counted from the corrected query with the status filter the SDK now exposes:
>>> sdk.count_requests(borough="brooklyn", status="Open")
Brooklyn open complaints: 1,842
The shared SDK uppercases the borough for every caller, and count_requests uses the vendor's $select=count(*) so the number comes from the source, not from counting a paginated stream. This specific hour never happens again. The postmortem is one line: we debugged our request before blaming their server."
Then Dev's — the artifact and the method, together:
"Dev — two things. First, the SDK: ThreeOneOne is the one client now — iter_requests hides the pages, errors come back translated (BadQueryError names the column, NotFoundError names the key), and 429s are retried with the vendor's own Retry-After. Delete your loop; use this. Second, the method, for the next 'broken API': smallest repro, raw request and response, change one variable, then the ticket template — what you sent, what you expected, what you got, the repro pasted verbatim. The ticket that gets fixed is the one the vendor can answer in one reply."
The pattern, as always: the number first, then the fix that prevents the repeat — and the method, because the next bug won't be this bug.
Must know
- An SDK is a translation layer: wire format in, domain language out — hide mechanics, translate errors, never hide failures
- One method touches the network; everything public speaks the domain (
iter_requests,get_request) - Debugging someone else's API: smallest repro → raw request/response → change one variable → the ticket that gets fixed
- A 200 with an empty body is an answer, not an error — suspect your query before their server
- The ticket that gets fixed: what you sent, what you expected, what you got, the minimal repro pasted verbatim
Useful later
- Code-generated SDKs (OpenAPI → client) — when the vendor publishes a spec worth generating from
- Async clients — when one-at-a-time iteration is too slow and the vendor allows concurrency
- Response caching inside the SDK — when the same reads repeat and the data tolerates staleness
- SDK-level metrics — request counts, retry rates, and error classes as dashboards, not just logs
Don't memorize this
- Any vendor's query dialect — remember translate the wire into the domain, look up the dialect per vendor
- Retry constants — remember honor Retry-After, jitter your own waits, cap attempts, tune per vendor
- Ticket portal rituals — remember answerable tickets get fixed, look up the portal's fields
Where this lands in CityOps
The SDK becomes the shared client underneath everything that reads the feed: the intake pipeline's extractor, the weather enrichment, and every ad-hoc script Tom writes at 9 AM. One client, one retry policy, one error vocabulary — the four hand-rolled loops get deleted, and the next vendor change is a one-place fix with a changelog entry. The debugging method joins the toolkit permanently: it's what Dev reaches for the next time a vendor's API "breaks," and the ticket template is what turns a morning of staring into a one-reply answer.
And there's a turn coming, the same one Post 3 promised: in Milestone 3 you build the CityOps API — which means you become the vendor. Someone else will write the thin, silent SDK over your endpoints, file the vague ticket about your empty result sets, and wonder why your borough filter is case-sensitive. Every design decision in this lesson is one you'll soon make from the other side of the table: which errors your API translates, which quirks it absorbs, and whether its 200-with-[] is an answer or a trap.
The signature line for this one: hide the wire, not the error. Post 1 said the URL is a noun; the verb is the method — the SDK is what happens when you take that contract seriously enough to write the client you wish someone had written for you.
Field check
- Dev's repro shows
borough='brooklyn'→ 200 with[], andborough='BROOKLYN'→ 200 with rows. Is the API broken? What was the actual bug, and which debugging step found it? - Your SDK's
get_requestreturnsNonefor both a missing key and a vendor outage. What's wrong with that — and what does the translated-error design do instead? - A teammate reaches past the SDK and calls
sdk.session.get(...)directly "just this once." Name two concrete things that break the next time the vendor changes something. - The vendor 429s your extractor with
Retry-After: 30. Your SDK waits exactly 30 seconds. Why not the jittered 4 seconds from your backoff formula? - You file the good ticket from this lesson. The vendor replies: "working as intended, borough is case-sensitive." What do you change — in the SDK, and in the ticket's aftermath?
What good answers look like
1. No — the API answered exactly what it was asked. The bug was the query: the feed stores BROOKLYN, and the request asked for brooklyn. Step three of the method found it — change one variable (the case) until the behavior changes. The 200-with-[] was the clue: an answer, not an error, which pointed at the request rather than the server. 2. The caller can't distinguish "not found" from "the vendor is on fire" — so an outage presents as a confident empty result instead of a paged engineer. The translated design raises NotFoundError (naming the key) for the missing row and VendorError (naming the vendor and the status) for the outage — the error is the most important part of the interface. 3. (a) Error translation is bypassed — raw HTTP errors and status codes leak into the caller's code, so the vendor change breaks it in ways the SDK would have absorbed. (b) Retry policy is bypassed — the direct call has no backoff, no Retry-After handling, so it's the call that earns the stern email. Both are really the same break: the abstraction has a hole, and holes are where vendor changes enter. 4. Because Retry-After is the vendor's explicit instruction — honor it exactly. Jitter applies only to your own exponential guesses. Waiting less than asked is how you earn the second, sterner email; the SDK's _delay_for encodes exactly this split. 5. In the SDK: nothing about the request changes — but the borough.upper() normalization (already there) gets a comment citing the ticket, and the SDK's docs note the case-sensitivity explicitly. In the aftermath: you reply to the ticket asking them to document it, and you add the quirk to the SDK changelog in plain language so the other three teams learn it from you instead of from their own lost morning.
Comments
Post a Comment