APIs from the Python Side

So far every lesson has started with a file somebody handed you. Out in the field, the freshest data isn't handed to you — it lives behind a URL, and it only answers if you ask correctly. This week Python learns to ask: real API calls against real weather and 311 data, and everything you need to survive when the network doesn't cooperate.

Assumes: Posts 1–5. One install: pip install requests.

Before you code: clarify the ask

Input: Open-Meteo hourly precipitation + open-data 311 daily counts, Sept 18–22
Output: one joined table and a one-line verdict Lisa can carry into the briefing
Deadline: the briefing is at 2 PM

That last clarification is doing quiet work: you've already spotted the confound (weekends) before touching the keyboard. The verdict will need it.

The minimal concept

An HTTP request is a question with three parts: a method (GET means "give me data"), a URL (where to ask), and query parameters (the details of the question). A response has three parts back: a status code (did it work?), headers (metadata), and a body (the answer — usually JSON).

Four ideas carry the whole post:

Status codes are the API's first sentence. 2xx means success. 4xx means the server is refusing this request for a client-side or request-state reason — 401 needs authentication, 403 needs permission, 404 means the thing isn't there, 429 means you're asking too fast. Blindly retrying the identical request usually won't help, with important exceptions such as 429. 5xx means the server is having a bad day. Read the code before you touch the body — parsing the body of a failed request is where the confusing errors come from.

params={...} beats hand-built URLs. Let requests encode the query string. Hand-concatenated URLs break on spaces, commas, and the first & inside a value.

Timeouts are not optional. Without one, a stalled server can hang your script indefinitely — and a hung script inside a scheduled pipeline is a silent outage. Always pass timeout=(connect, read): how long to wait to reach the server, and how long to wait between bytes once talking.

GET is usually retry-friendly; POST needs more care. GET is designed to be safe and idempotent — asking for the same data twice changes nothing — so it's normally fine to retry. POST often isn't idempotent, so retrying it requires more care: a retried POST can create, charge, or send twice, and a network failure can leave you unsure whether it reached the server at all. (Stage 3 teaches the idempotency-key discipline that makes some POSTs safely retryable.) This post retries GETs; the POST rules come later.

Build it: ask the sky, then ask the city

The weather call first. One choice matters here: we're analyzing a past event, so we use Open-Meteo's archive endpoint (ERA5 reanalysis), not the forecast endpoint — past observations come from the historical API, future weather from the forecast one. The archive lags real time by a few days, which is fine for a retrospective and worth knowing before you build a dashboard on it:

import requests

session = requests.Session()
session.headers["User-Agent"] = "cityops-lesson/1.0 (field training)"
# Identifying your script in User-Agent is basic API citizenship:
# when something goes wrong, the operator can tell a script from an attack.

weather = session.get(
    "https://archive-api.open-meteo.com/v1/archive",
    params={"latitude": 40.71, "longitude": -74.01,
            "hourly": "precipitation",
            "start_date": "2026-09-18", "end_date": "2026-09-22",
            "timezone": "America/New_York"},
    timeout=(5, 30),
)
# For this client, make status handling explicit before treating
# the body as successful data:
weather.raise_for_status()
data = weather.json()
# ...then validate the shape before the pipeline trusts it.
# The HTTP helper checks transport; the caller checks meaning.
if "hourly" not in data or "precipitation" not in data["hourly"]:
    raise ValueError("weather response missing expected fields")
hourly = data["hourly"]

rain = {}
for ts, mm in zip(hourly["time"], hourly["precipitation"]):
    rain[ts[:10]] = rain.get(ts[:10], 0) + (mm or 0)

Then the 311 feed. The portal accepts a SQL-like query right in the parameters — if Post 3's SQL survived contact with reality, this will feel familiar:

calls = session.get(
    "https://data.cityofnewyork.us/resource/erm2-nwe9.json",
    params={"$select": "date_trunc_ymd(created_date) AS day, count(*) AS n",
            # half-open interval: [Sep 18, Sep 23). "Through 23:59:59" misses
            # 23:59:59.500 — Post 5's boundary lesson, applied to queries.
            "$where": "created_date >= '2026-09-18T00:00:00' "
                      "AND created_date < '2026-09-23T00:00:00'",
            "$group": "day", "$order": "day", "$limit": "10"},
    timeout=(5, 30),
)
calls.raise_for_status()
counts = {row["day"][:10]: int(row["n"]) for row in calls.json()}

# Two successful API calls do not guarantee a valid joined dataset:
if set(rain) != set(counts):
    raise ValueError(
        f"date mismatch: weather={sorted(rain)}, 311={sorted(counts)}")

print(f"{'date':<12}{'requests':>10}{'rain_mm':>9}")
for day in sorted(rain):
    print(f"{day:<12}{counts[day]:>10}{rain[day]:>9.1f}")

date          requests  rain_mm
2026-09-18     11964      0.2
2026-09-19     10742      0.0
2026-09-20      8839     10.9
2026-09-21     10661      2.4
2026-09-22     11036      2.1

September 20th was the wettest day by a distance — 10.9 mm, rain on and off through the day, heaviest mid-morning — and it had the fewest requests. So what can we actually say? The data does not support the specific claim of a same-day 311 spike on September 20. That's the verdict, and notice its shape: it rules out the claimed spike without replacing it with a counter-claim. September 20 was also a Sunday, and weekend volume runs lower regardless of weather — so this five-day slice can't tell us whether rainfall caused volume to rise or fall, only that the spike the story needed isn't there. It refuses to replace one unsupported story with another.

This is the job. Not confirming the story someone wanted — returning what the evidence actually supports, and stopping where the evidence stops.

Break it, three ways

The code above works on a good day. Field code has to survive the other days. Here are the three failures that actually happen, in the order you'll meet them.

1. The hang. The first version of this script had no timeout at all. Against a stalled endpoint it doesn't error — it just sits there, forever, holding your pipeline hostage. Watch what a timeout buys you:

# a test endpoint that waits 10s before answering; we give up after 2
requests.get("https://httpbin.org/delay/10", timeout=2)

ReadTimeout

The client stops waiting around the configured timeout instead of hanging indefinitely. A loud failure in 2 seconds beats a silent hang at 2 PM when Lisa is waiting. timeout=(5, 30) says: 5 seconds to connect, 30 seconds between bytes. Tune the numbers per API; never delete the parameter.

2. Trusting 200. The second version assumed every response was a success and called .json() directly. Then the API rate-limited us mid-run and the "JSON" was an HTML error page. Real behavior:

r = requests.get("https://httpbin.org/status/429", timeout=10)
r.raise_for_status()

HTTPError: 429 Client Error: TOO MANY REQUESTS for url: https://httpbin.org/status/429

429 means "slow down," not "you're wrong" — the correct response is to wait, not to fix your parameters. Many APIs tell you exactly how long via a Retry-After header. And raise_for_status() before .json() is the order that keeps the error message pointing at the real problem instead of a JSON parse error three frames away from it.

3. The hammer loop. The third version retried — but as while True with no delay and no limit. During a real outage, that loop turns one script into a small denial-of-service attack: retrying every second means ~86,000 requests a day from a single hung script, which is how you get your IP throttled and an angry email from the API operator. Retries need three properties: bounded (a max number of attempts), backing off (wait longer each time: 1s, 2s, 4s…), and jittered (add randomness so a fleet of your scripts doesn't retry in lockstep and thundering-herd the recovering server).

Productionize: one helper, honest retries

Everything above compresses into a single helper. Read it as a checklist made executable:

import logging, random, time
import requests

log = logging.getLogger("cityops")
RETRYABLE = {429, 500, 502, 503, 504}

def fetch_json(session, url, params=None, tries=4):
    """GET a JSON endpoint with bounded retries.

    Retries connection errors, timeouts, and 429/5xx with exponential
    backoff + jitter. Honors numeric Retry-After exactly; date-form
    Retry-After falls back to backoff. Other 4xx raise immediately:
    retrying a 400 is just asking twice.
    """
    delay = 1.0
    for attempt in range(1, tries + 1):
        try:
            resp = session.get(url, params=params, timeout=(5, 30))
        except (requests.ConnectionError, requests.Timeout) as e:
            last, wait = e, None            # network failed or timed out -> retry
        else:
            if resp.status_code in RETRYABLE:
                last = RuntimeError(f"HTTP {resp.status_code}")
                try:
                    wait = float(resp.headers.get("Retry-After", ""))
                except (TypeError, ValueError):
                    wait = None              # date-form header: fall back to backoff
            else:
                # Status handling is explicit before the body is treated as
                # data: other 4xx (400/401/403/404) are your bug, not the
                # network's.
                resp.raise_for_status()
                return resp.json()
        if attempt == tries:
            raise RuntimeError(f"GET {url} failed after {tries} tries") from last
        if wait is None:
            # Jitter the backoff — never the server's explicit order.
            wait = delay * (0.5 + random.random())
        log.warning("attempt %d/%d failed -> retry in %.1fs", attempt, tries, wait)
        time.sleep(wait)
        delay = min(delay * 2, 60)

Three behaviors, all verified against scripted servers rather than asserted:

# flaky server: 503 + Retry-After: 1, twice, then 200
WARNING attempt 1/4 failed -> retry in 1.0s     # the server said 1s: honored exactly, not jittered
WARNING attempt 2/4 failed -> retry in 1.0s
{'rain_mm': 10.9}                                # recovered on attempt 3

# bad parameters (400): exactly ONE http call, raises immediately
HTTPError                                       # no pointless retries

# server down: exactly `tries` attempts, then a clear error
RuntimeError: GET https://x.test/weather failed after 3 tries

Notice the exception discipline: we catch ConnectionError and Timeout — the network failed or timed out, so a retry might help — and nothing broader. The final raise ... from last keeps the original error chained, so the log shows both what we did and what the network did. The numeric Retry-After is obeyed to the second; only our own backoff gets jittered, because the server's explicit "wait 120 seconds" is an order, not a suggestion. And the shape check from the build section still sits at the caller — the HTTP helper validates transport, the weather adapter validates weather shape. Different responsibilities.

Explain it to the customer

"Lisa — on the storm story: the data doesn't support the claimed same-day spike. September 20th was the wettest day in the window (10.9 mm, rain on and off through the day) and it had the lowest 311 volume: 8,839 vs ~11,000 on the surrounding days. The caveat for the briefing: the 20th was a Sunday, and weekend volume runs lower regardless of weather — so this slice rules the spike out but can't say whether rain caused volume to rise or fall. And the method is rerunnable: one script pulls the archive weather and the 311 feed, validates both shapes, reconciles the join, and prints the table — so the next storm story gets checked in minutes, not argued about for a week."

The pattern by now: the verdict, then the caveats, then the rerunnable method. The middle sentence is the one that keeps you honest — it refuses to replace one unsupported story with another.

Where this lands in CityOps

fetch_json joins the boundary toolkit next to parse_ts and parse_money. In Milestone 2, the intake pipeline pulls the weather feed every night and stores it alongside the 311 data — the enrichment step that turns "complaints vs storms" from a one-off briefing scramble into a standing dataset. The join you wrote by hand today becomes Post 4's merge, rerunnable.

Post 4's principle was if you can't rerun it, you didn't clean it. Post 5's was a number without its unit is a rumor. Post 6's: don't trust the happy path — and it applies twice today. Don't assume the customer's hypothesis is true: check it against the data, and stop where the evidence stops. And don't assume the network will behave: code for its bad days, not just its good ones.