SkyWatch #1: Polling Live Skies

A customer wants a live dashboard over data you don't own, from an API with a quota you can't negotiate. This post builds the ingestion pipeline: bounding-box polling, a quota budget, dedup by icao24, and the nulls that are perfectly normal. The one principle for this post: you don't control the source — design for its constraints.

The customer problem

Priya's readers want to know what's overhead right now. "Right now" is doing heavy lifting in that sentence: it rules out yesterday's schedule data, it rules out a static map, and it means whatever we build is only as live as its slowest dependency. The data exists — OpenSky Network publishes live aircraft state vectors from a community receiver network, free, no API key for anonymous access. The catch is the one every free API has: you are a guest on someone else's infrastructure.

Clarify the ask

Before any code, the questions that shape the whole design:

• Which sky? "Near our city" becomes a bounding box — say, the metro area plus approach corridors. The box size directly sets the API cost, so this is a budget question disguised as a geography question.
• How live is "live"? A reader checking "what was that plane" tolerates a 3–5 minute delay. An air-traffic display would not. Our tolerance sets the poll cadence — and the cadence must fit inside the quota.
• What does a flight mean here? A state vector is a momentary observation of a transponder, not a flight record. Same aircraft, new poll, new vector — dedup is our job.

The minimum concept

One poll. One bounding box. Parse the response, print the callsigns. The entire pipeline in its smallest shippable form:

import requests

BBOX = {"lamin": 37.0, "lamax": 38.0, "lomin": -123.0, "lomax": -122.0}  # Bay Area, ~1 sq deg

r = requests.get("https://opensky-network.org/api/states/all",
                 params=BBOX, timeout=15)
r.raise_for_status()
body = r.json()

print(f"API time: {body['time']}, vectors: {len(body['states'] or [])}")
for v in (body["states"] or [])[:5]:
    icao24, callsign = v[0], (v[1] or "").strip()
    print(f"{icao24}  {callsign or '(no callsign)'}  {v[6]:.2f},{v[5]:.2f}")

A real response looks like this (trimmed from a live call): ["a14b1a","UAL301 ","United States",1792124736,1792124744,-122.3801,37.616,null,true,0.06,208.12,...]. Eleven-plus positional fields, no names. That positional array is the state vector — everything downstream starts here.

Build: the poller

The poller has four jobs: ask politely, parse defensively, dedup, and stamp every snapshot with when we saw it.

Anatomy of a state vector

The fields that matter and their traps:

Dedup and the snapshot store

def parse_vector(v):
    """One state vector -> one dict. Nulls preserved, never coerced."""
    return {
        "icao24": v[0],
        "callsign": (v[1] or "").strip() or None,
        "origin_country": v[2],
        "time_position": v[3],      # may be None
        "last_contact": v[4],
        "longitude": v[5],
        "latitude": v[6],
        "baro_altitude_m": v[7],    # None on ground — normal
        "on_ground": v[8],
        "velocity_ms": v[9],        # None when stationary — normal
        "true_track": v[10],
        "observed_at": time.time(), # OUR clock: when we saw it
    }

def merge_snapshot(prev, vectors):
    """Dedup by icao24; keep the freshest vector per aircraft."""
    merged = dict(prev)
    for v in vectors:
        p = parse_vector(v)
        old = merged.get(p["icao24"])
        # Freshness = last_contact; fall back to our observation time
        new_ts = p["last_contact"] or p["observed_at"]
        old_ts = (old["last_contact"] or old["observed_at"]) if old else -1
        if new_ts >= old_ts:
            merged[p["icao24"]] = p
    return merged

Note observed_at: the API tells you when it saw the aircraft. Your pipeline must record when you saw the API's answer. When the dashboard later says "3 minutes old," that number must come from your clock, not the API's — the difference is the first thing you'll need when debugging staleness in monitoring.

Break: the quota does the designing

Here's where the free tier stops being a footnote and starts being the architecture. The arithmetic, from documented figures:

Three consequences fall out of that diagram:

1. Poll cadence is a budget decision, not a preference. With a 1-credit box, anonymous quota buys a poll every ~3.6 minutes. Priya's readers tolerate that. If she'd asked for per-second tracking, the honest answer is "not on this quota" — the rate-limits lesson's clarify-before-calculate rule, applied to someone else's API.
2. The 10-second floor matters too. Even with credits to spare, anonymous requests are throttled to ~1 per 10 seconds. A retry loop that fires immediately on failure will eat 429s. Backoff with jitter, always.
3. 429s are a normal operating condition. Quota exhaustion isn't an exception — it's the predictable end of every day if your cadence is wrong. The poller treats 429 like weather: expected, handled, logged.

Productionize: the poll loop

def poll_forever(bbox, credits_per_poll=1, daily_budget=400):
    """Poll on a cadence the quota can sustain. 429s handled, not feared."""
    interval = 86400 / (daily_budget // credits_per_poll)  # seconds between polls
    interval = max(interval, 11)  # respect the ~10s anonymous floor
    snapshot = {}
    while True:
        try:
            r = requests.get(API, params=bbox, timeout=15)
            if r.status_code == 429:
                wait = int(r.headers.get("Retry-After", 60))
                log(f"429: backing off {wait}s (+jitter)")
                time.sleep(wait + random.uniform(0, 10))
                continue
            r.raise_for_status()
            snapshot = merge_snapshot(snapshot, r.json()["states"] or [])
            store_snapshot(snapshot)  # sqlite, one row per icao24
        except requests.RequestException as e:
            log(f"poll failed: {e}; keeping last good snapshot")
        time.sleep(interval)

Packaging follows the packaging lesson: a real package, a requirements.txt, structured logs with the poll timestamp, credit spend, vector count, and HTTP status on every cycle. The credit spend per day is a metric from day one — Post 3 will alert on it.

Communicate: what "live" means, in writing

Priya's site gets a one-line honesty label under the dashboard, decided now, not later: "Positions update every few minutes from OpenSky Network's community receivers. Coverage and freshness vary." That sentence is a contract with the reader. It prevents the support email that starts "your site said the plane was HERE but it was THERE." The dashboard doesn't exist yet — but its truthfulness policy already does.

Next: SkyWatch #2 — flights meet weather. Two live sources, two different freshness guarantees, one dashboard that has to be honest about both.