REST, Deeply

Every lesson so far has called APIs from the Python side — you were the customer of someone else's contract. This stage flips the table. Before you build CityOps' own API in Milestone 3, you learn to read any API's contract the way a lawyer reads the fine print: what the nouns are, which verbs each one speaks, and what the status codes are really telling you. Ten minutes with any docs, dangerous by the end of it.

Assumes: Posts 1–9, especially Post 6. One install: pip install requests.

Before you code: clarify the ask

Input: the 311 dataset's API docs, plus Tom's half-understood requirement
Output: one readable GET request Tom can reuse, and a reader that fails loudly instead of silently
Deadline: the dashboard refreshes Thursday

Notice that "just reading" did more work than any code will. It ruled out every verb except one before a single request was fired.

The minimal concept

Four ideas, and the first one is the whole lesson in a sentence:

The URL is a noun; the verb is the method. A REST API is sentences. GET (give me) /tickets (the tickets). Resources are nouns that live at URLs; the HTTP method says what you're doing to them: GET reads, POST creates, PUT replaces, PATCH partially updates, DELETE removes. Read any API's docs as sentences and the docs start making sense.

Status codes are the API's vocabulary for what happened. 2xx — success. 3xx — go look over there. 4xx — the request is the problem: 400 malformed, 401 missing credentials, 403 insufficient permission, 404 not found, 429 slow down. 5xx — the server is having a bad day: 500, 502, 503, 504. Read the code before you touch the body — parsing the body of a failed request is where the confusing errors come from.

Query parameters are the fine print. Filters, field lists, and limits ride in the URL after the ? — but you build them with params={...}, never by concatenating strings. The library encodes; your fingers don't.

Idempotency is the retry rule. GET and HEAD are designed safe to repeat — asking twice changes nothing, so retrying them is usually fine. POST usually isn't idempotent: asking twice can create twice. PUT and DELETE are designed repeat-safe — repeat them and the net effect is the same. This is why Post 6's fetch_json only ever retried GETs. It was already living this lesson's rule; today the rule gets a name.

Build it, part 1: read the docs like a contract

Take the endpoint Tom was staring at and decode it like a contract. Every REST endpoint decomposes the same way:

Piece	What it is	311 example
Base URL	Whose API this is	https://data.cityofnewyork.us
Resource	The noun — which dataset, which flavor	/resource/erm2-nwe9.json (the 311 dataset, as JSON)
Method	The verb — what you're doing	GET: we're only reading
Parameters	The fine print — fields, filters, limits	$select, $where, $limit (Socrata's dialect)

Now Tom's ask as a sentence — "GET (give me) the 311 resource, but only these columns, where the agency is DSNY and the borough is Brooklyn, and only three rows to start":

import requests

s = requests.Session()
s.headers["User-Agent"] = "cityops-lesson/1.0 (field training)"

r = s.get("https://data.cityofnewyork.us/resource/erm2-nwe9.json",
          params={"$select": "unique_key,created_date,agency,complaint_type,status,borough",
                  "$where": "status='Open' AND agency='DSNY' AND borough='BROOKLYN'",
                  "$limit": "3"},
          timeout=(5, 30))
print(r.status_code, r.headers["Content-Type"])
for row in r.json():
    print(row)

200 application/json;charset=utf-8
{'unique_key': '45838074', 'created_date': '2020-03-16T21:34:00.000', 'agency': 'DSNY', 'complaint_type': 'Electronics Waste Appointment', 'status': 'Open', 'borough': 'BROOKLYN'}
{'unique_key': '45838075', 'created_date': '2020-03-16T20:51:00.000', 'agency': 'DSNY', 'complaint_type': 'Electronics Waste Appointment', 'status': 'Open', 'borough': 'BROOKLYN'}
{'unique_key': '45841003', 'created_date': '2020-03-16T15:16:00.000', 'agency': 'DSNY', 'complaint_type': 'Electronics Waste Appointment', 'status': 'Open', 'borough': 'BROOKLYN'}

That's a real run against the real feed — 200, JSON, three rows. And here's the ten-minutes-with-any-docs checklist this decomposes into. Run it against unfamiliar docs and you're dangerous fast:

1. Base URL — whose server am I talking to? (Copy it exactly; http vs https matters.)
2. Resource path — which noun? The path names the thing; the extension or Accept header names the flavor.
3. Verbs — which methods does this resource speak? A read endpoint that only speaks GET will not negotiate with your POST.
4. Parameters — which filters, which field selection, which pagination? Required vs optional.
5. Auth — does it need a key, and where does the key go? (Stage 3's next lesson lives here.)
6. Status codes — what does this API return when it fails, and in what format?
7. Rate limits — how fast may I ask? What happens when I ask faster?

Build it, part 2: status codes are the vocabulary

Vocabulary is learned by hearing it spoken. Three clean examples, all real runs:

>>> requests.get("https://httpbin.org/status/200", timeout=10).status_code
200
>>> requests.get("https://httpbin.org/status/404", timeout=10).status_code
404
>>> requests.get("https://httpbin.org/status/503", timeout=10).status_code
503

200: success, carry on. 404: the thing isn't there — check your resource path. 503: the server is having a bad day — this one may be worth retrying. Same call shape, three different meanings, all in the code before you ever look at a body.

And the Socrata feed speaks the vocabulary fluently — including in failure. Typo the dataset id and the API tells you exactly what went wrong, in JSON, no guessing:

>>> r = requests.get("https://data.cityofnewyork.us/resource/erm2-nwe.json", timeout=10)
>>> r.status_code
404
>>> r.json()
{'code': 'dataset.missing', 'error': True, 'message': 'Not found',
 'data': {'id': 'erm2-nwe'}}

dataset.missing — not "something broke," but "that dataset doesn't exist." Good APIs narrate their failures. Your job is just to read the narration before doing anything else — which is exactly what the next section is about not doing.

Build it, part 3: verbs in the lab

To feel the verbs, you need an API you can safely poke — so this section runs against a tiny local lab server (http.server, on your own machine) with four endpoints: GET/POST on /tickets, PUT/DELETE on /tickets/0. Never experiment with verbs against a production API; that's what labs are for.

Watch what each verb means, by repetition:

base = "http://127.0.0.1:8811"
p = {"title": "pothole on 5th", "status": "open"}

# the same POST, sent twice:
>>> requests.post(base + "/tickets", json=p, timeout=5).json()["id"]
0
>>> requests.post(base + "/tickets", json=p, timeout=5).json()["id"]
1
>>> len(requests.get(base + "/tickets", timeout=5).json())
2          # two tickets. the retry created a second one.

# GET twice: identical, nothing changed
>>> a = requests.get(base + "/tickets", timeout=5).json()
>>> b = requests.get(base + "/tickets", timeout=5).json()
>>> a == b
True

# PUT the same update twice: one ticket, same final state
>>> requests.put(base + "/tickets/0", json={"title": "pothole on 5th", "status": "closed"}, timeout=5).status_code
200
>>> requests.put(base + "/tickets/0", json={"title": "pothole on 5th", "status": "closed"}, timeout=5).json()
{'title': 'pothole on 5th', 'status': 'closed', 'id': 0}

# DELETE twice: 200, then 404 — but the ticket stays gone either way
>>> requests.delete(base + "/tickets/0", timeout=5).status_code
200
>>> requests.delete(base + "/tickets/0", timeout=5).status_code
404

All real runs. The pattern: POST created something new every time it was asked. GET asked twice and changed nothing. PUT landed in the same state twice. DELETE's second call complained (404, "already gone") but the net effect — ticket gone — was identical. That's idempotency, not as a definition but as observed behavior: which verbs can you safely repeat, and which one punishes you for it.

Break it, three ways

1. The 404 nobody checked. Tom's first script read the docs, built the request — and treated every response as data. Watch what happens when the resource path is wrong and nobody reads the status (real run, typo'd dataset id):

r = requests.get("https://data.cityofnewyork.us/resource/erm2-nwe.json",
                 params={"$limit": "3"}, timeout=15)
print("status was:", r.status_code, "(nobody checked)")
rows = r.json()
print("first row keys:", list(rows[0].keys()))

status was: 404 (nobody checked)
KeyError: 0

The error body is a dict describing the failure; the script treated it as a list of rows. The KeyError surfaces three lines later, pointing at nothing. Post 6's rule stands: status first, body second — the failure you check for at the boundary is the failure you can diagnose.

2. The hand-built URL. Tom's second attempt built the query string by concatenation — because the value contained an &, the server saw two parameters where he meant one (real runs):

base = "https://data.cityofnewyork.us/resource/erm2-nwe9.json"
hand = base + "?$select=unique_key&$where=complaint_type='Noise & Construction'&$limit=1"
>>> requests.get(hand, timeout=15).status_code
400
>>> requests.get(hand, timeout=15).json()["message"]
"Unrecognized arguments [ Construction']"

>>> requests.get(base, params={"$select": "unique_key",
...                              "$where": "complaint_type='Noise & Construction'",
...                              "$limit": "1"}, timeout=15).status_code
200

The server is not being difficult — & means "next parameter" in a query string, so the hand-built URL genuinely asked for a parameter named Construction'. params= encodes the value; concatenation trusts your fingers. One of those is a strategy.

3. The retried POST. The lab already showed you: same body, sent twice, two tickets (201 id=0, 201 id=1). Now put it in production clothes. Your network hiccups after the server created the ticket but before the response reached you. Your retry logic — the good, Post 6-style retry logic — fires the POST again. The server, doing exactly what POST means, creates it again. In the lab that's two rows. In production it's two work orders dispatched, two charges on the card, two "your request is confirmed" emails to the customer. The retry wasn't wrong and the server wasn't wrong — the verb was wrong for the retry.

Productionize: the idempotency rule, as code

Everything above compresses into one helper: rest_call. It bakes the retry rule into the call itself — safe methods (GET, HEAD) may be retried with backoff on 429/5xx; unsafe methods (POST, PUT, DELETE) are attempted exactly once unless you explicitly pass allow_retry=True. And every failure raises an ApiError carrying the status, the URL, and a hint about what the code means — caught only to add that context, per the exception rule:

import logging, random, time
import requests

SAFE_METHODS = {"GET", "HEAD"}          # designed safe to repeat
RETRYABLE = {429, 500, 502, 503, 504}

class ApiError(RuntimeError):
    """An HTTP failure with the status, the URL, and what it usually means."""
    HINTS = {
        400: "bad request — the request is malformed (do not retry as-is)",
        404: "not found — wrong resource, or it was deleted",
        429: "too many requests — back off, then retry",
    }
    def __init__(self, method, url, status, body=""):
        hint = self.HINTS.get(status,
            "server-side failure — retry may help"
            if str(status).isdigit() and int(status) >= 500
            else "request problem — fix the request, don't retry")
        super().__init__(f"{method} {url} -> {status}: {hint} | body: {str(body)[:120]}")

def rest_call(method, url, params=None, json=None, tries=3, allow_retry=False):
    """REST with the idempotency rule baked in.

    Safe methods retry (bounded, backoff+jitter) on 429/5xx.
    Unsafe methods run exactly once unless allow_retry=True —
    because a retried POST can create twice.
    """
    method = method.upper()
    attempts = tries if (method in SAFE_METHODS or allow_retry) else 1
    delay = 1.0
    for attempt in range(1, attempts + 1):
        try:
            r = requests.request(method, url, params=params, json=json,
                                 timeout=(5, 30),
                                 headers={"User-Agent": "cityops-lesson/1.0 (field training)"})
        except (requests.ConnectionError, requests.Timeout) as e:
            if attempt == attempts:
                raise ApiError(method, url, "no response", e) from e
        else:
            if r.status_code in RETRYABLE and attempt < attempts:
                wait = delay * (0.5 + random.random())
                logging.getLogger("cityops").warning(
                    "%s %s -> %s: retry in %.1fs", method, url, r.status_code, wait)
                time.sleep(wait); delay = min(delay * 2, 30)
                continue
            if r.status_code >= 400:
                raise ApiError(method, url, r.status_code, r.text)
            return r.json()

Three behaviors, all verified against live servers — note what the log lines prove:

# GET recovers from a flaky endpoint (two 503s, then healed):
WARNING GET http://127.0.0.1:8811/flaky -> 503: retry in 0.9s
WARNING GET http://127.0.0.1:8811/flaky -> 503: retry in 2.1s
{'ok': True}

# POST to the same failing endpoint: ZERO warning lines — one attempt, then out:
ApiError: POST http://127.0.0.1:8811/flaky -> 503: server-side failure — retry may help | body: {"er...

# POST with allow_retry=True: retries twice, then gives up honestly:
WARNING POST http://127.0.0.1:8811/flaky -> 503: retry in 0.8s
WARNING POST http://127.0.0.1:8811/flaky -> 503: retry in 1.1s
ApiError: POST http://127.0.0.1:8811/flaky -> 503: server-side failure — retry may help | body: {"er...

And the hint, doing its job on the typo'd dataset — the real message, unabridged:

ApiError: GET https://data.cityofnewyork.us/resource/erm2-nwe.json -> 404:
not found — wrong resource, or it was deleted | body: {
  "code" : "dataset.missing",
  "error" : true,
  "message" : "Not found",
  "data" : {
    "id" : "erm2-nwe"
  }
}

Read the discipline in the code: we catch ConnectionError and Timeout only — the network failed, so a retry might help — and re-raise as ApiError with the method, URL, and status attached. raise ... from e keeps the original chained. Non-retryable failures (400, 404) raise immediately with the hint; nothing is swallowed, nothing retried blindly. This is Post 6's fetch_json grown up: same backoff-and-jitter DNA, now with the verb-awareness this lesson taught it.

Explain it to the customer

"Tom — the dashboard query is one readable call: rest_call("GET", ...) with the columns, the status != 'Closed' filter, and the Brooklyn filter as parameters — params= builds the URL, so the &-in-a-value bug can't happen. 'Open' is now defined in writing as status != 'Closed', not the English word — that's a different number by hundreds of thousands of rows, so the definition lives in the code, not in a Slack thread. If the endpoint ever 404s, you get one error naming the URL, the status, and what it means — not a KeyError three lines later. And the retry rule is in the helper now: reads retry themselves, writes never retry unless you explicitly say so."

The pattern: what you built, the definition that changed the number, the failure mode that's now loud, and the rule that's now code instead of tribal knowledge.

Where this lands in CityOps

This lesson is the hinge of the whole stage. Everything so far, you've been the caller; in Milestone 3 you build CityOps' own API with FastAPI — the other side of the table. Every rule in this lesson becomes a decision you'll make: which nouns your API exposes, which verbs each one speaks, which status codes it returns, and what its failure bodies look like. The dataset.missing JSON you admired today is the standard you're about to hold yourself to.

And the intake pipeline from Milestone 2 gets its reader upgraded: rest_call replaces the raw session.get, so the nightly pull now fails with a named, hinted error instead of a KeyError at 6 AM. Post 7's suite can pin the new rule — a POST that retries without allow_retry is a test that should fail.

The signature line for this one: the URL is a noun; the verb is the method. Post 6 said don't trust the happy path — reading the contract is how you stop trusting it and start verifying it, before the first request is ever sent.