Packaging, Envs, Logging & CLIs

Everything you've built so far runs on one machine: yours. This week that ends. Tom needs to run your pipeline on the city's server, Maria needs proof it ran last night, and neither of them should have to call you. The "runs anywhere" toolkit: environments that travel, logs that remember, and a command line that explains itself.

Assumes: Posts 1–6. A terminal and pip. No new concepts from outside this series.

Before you code: clarify the ask

Input: the 311/weather pull logic from Post 6
Output: an installable project Tom runs as cityops pull --date 2026-09-20, with logs Maria can check the next morning
Deadline: the nightly pull starts Friday

Notice the shift: nobody asked for a new analysis. The ask is entirely about operability — who runs it, where, and how they know it worked. That question should have been asked before the first line of pipeline code. Better late than at 8 AM on a Monday.

The minimal concept

Three ideas, and they're all contracts:

An environment is a contract: this interpreter, these packages, these versions. A virtual environment (venv) is a private Python installation for one project — its own interpreter link and its own package directory, isolated from every other project on the machine. requirements.txt is the written form of the contract: every package, pinned to an exact version. "It works on my machine" is what you say when the contract was never written down.

Logging is the script's memory. print() talks to whoever happens to be watching the terminal. logging writes timestamped, leveled records to both the terminal and a file — so Monday-morning Maria can reconstruct Friday night's run. Levels are a triage system: DEBUG for developers, INFO for "what happened," WARNING for "something's off but we continued," ERROR for "this failed."

A CLI is the script's handshake. argparse turns "edit the script and rerun" into cityops pull --date 2026-09-20 — named commands, named arguments, and a --help page Tom can read instead of reading your code. If the only way to run your script is to open it in an editor, it's not a tool yet.

Build it, part 1: an environment that travels

Start where Tom started: a fresh machine. A virtual environment is one command, and it costs nothing:

$ python3 -m venv .venv
$ .venv/bin/python --version
Python 3.12.3

Note the trick worth keeping: .venv/bin/python uses the environment directly — no activation step, no shell magic. That matters for scheduled jobs, where there is no interactive shell to activate anything.

Now watch Tom's Monday morning reproduce itself, on purpose:

$ .venv/bin/python -c "import requests"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'requests'

That's the whole bug. Not a logic error — a missing contract. The fix is the contract, written down:

$ .venv/bin/pip install requests
Successfully installed certifi-2026.7.22 charset_normalizer-3.5.2 idna-3.20 requests-2.34.2 urllib3-2.8.0
$ .venv/bin/pip freeze > requirements.txt
$ cat requirements.txt
certifi==2026.7.22
charset-normalizer==3.5.2
idna==3.20
requests==2.34.2
urllib3==2.8.0

pip freeze writes down exactly what got installed — including the dependencies of your dependencies (urllib3, certifi), because those can break you just as thoroughly. Tom's install becomes two lines:

$ python3 -m venv .venv
$ .venv/bin/pip install -r requirements.txt

And here's the detail that justifies the ceremony: the laptop this post was written on has requests 2.31.0; the fresh environment just installed 2.34.2. Same pip install requests, different answers, depending on when you ask. An unpinned requirement is a promise the future gets to renegotiate. The pin is what makes "Tom's install" and "your install" the same install.

Build it, part 2: logging that outlives the terminal

Maria's question — "did the pull run last night?" — needs an answer that survives the night. Two handlers: one for the human watching now, one for the human on Monday:

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
    handlers=[
        logging.StreamHandler(),            # terminal, for the human watching
        logging.FileHandler("cityops.log"), # file, for the human on Monday
    ],
)
log = logging.getLogger("cityops")

log.debug("connection pool warmed")         # filtered out at INFO
log.info("pulling 311 data for 2026-09-20")
log.warning("archive API slow: attempt 1 took 4.2s")
log.info("311 rows=8839 rain_mm=10.9")
log.error("reconciliation failed: 3 rows missing from 311 feed")

$ .venv/bin/python logdemo.py
2026-10-03 04:56:12 INFO    cityops: pulling 311 data for 2026-09-20
2026-10-03 04:56:12 WARNING cityops: archive API slow: attempt 1 took 4.2s
2026-10-03 04:56:12 INFO    cityops: 311 rows=8839 rain_mm=10.9
2026-10-03 04:56:12 ERROR   cityops: reconciliation failed: 3 rows missing from 311 feed
$ cat cityops.log
2026-10-03 04:56:12 INFO    cityops: pulling 311 data for 2026-09-20
2026-10-03 04:56:12 WARNING cityops: archive API slow: attempt 1 took 4.2s
2026-10-03 04:56:12 INFO    cityops: 311 rows=8839 rain_mm=10.9
2026-10-03 04:56:12 ERROR   cityops: reconciliation failed: 3 rows missing from 311 feed

Two things to notice. First, the DEBUG line never appears — level=logging.INFO filters it, which is the point: verbosity is a dial, not a rewrite. Second, those console lines went to stderr, not stdout — that's StreamHandler's default, and it's a feature: stdout stays clean for data (pipes, > out.json), while the narrative of the run goes to stderr and the file. Post 6's retry helper already logged this way — same logger, same habit, now with a file behind it.

Level	Meaning	Example
DEBUG	For the developer diagnosing	retry delays, raw response sizes
INFO	What happened, the normal story	rows pulled, files written
WARNING	Off, but we continued	slow API, fallback used
ERROR	This failed	reconciliation mismatch, bad input
CRITICAL	The run cannot continue	config missing, disk full

If everything is INFO, nothing is signal. If everything is ERROR, nothing is either. The levels are a triage system — spend thirty seconds choosing, and Maria can skim a month of logs without calling you.

Build it, part 3: a CLI Tom can run

The pull logic from Post 6 becomes two named commands. argparse handles the parsing, the help text, and the error messages — you only write the functions:

"""CityOps intake CLI: pull 311/weather data, then report on it."""
import argparse
import logging
from datetime import date

log = logging.getLogger("cityops")

def cmd_pull(args):
    try:
        day = date.fromisoformat(args.date)
    except ValueError:
        # not "except: pass" — bad input gets a clear message and a nonzero exit
        raise SystemExit(f"bad --date {args.date!r}: use YYYY-MM-DD")
    log.info("pulling 311 data for %s", day)
    log.info("311 rows=8839 rain_mm=10.9")
    log.info("pull complete: wrote data/%s.json", day)

def cmd_report(args):
    log.info("report: 5 days, 54342 requests, wettest day 2026-09-20 (10.9 mm)")

def main():
    p = argparse.ArgumentParser(prog="cityops",
                                description="CityOps 311/weather intake pipeline")
    sub = p.add_subparsers(dest="command", required=True)
    pull = sub.add_parser("pull", help="pull one day of 311 + weather data")
    pull.add_argument("--date", required=True, help="day to pull, YYYY-MM-DD")
    pull.set_defaults(func=cmd_pull)
    rep = sub.add_parser("report", help="summarize what's been pulled")
    rep.set_defaults(func=cmd_report)
    args = p.parse_args()
    args.func(args)

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO,
                        format="%(levelname)-7s %(name)s: %(message)s")
    main()

Now Tom never opens the file. He asks it what it does:

$ .venv/bin/python cityops.py --help
usage: cityops [-h] {pull,report} ...

CityOps 311/weather intake pipeline

positional arguments:
  {pull,report}
    pull         pull one day of 311 + weather data
    report       summarize what's been pulled

options:
  -h, --help     show this help message and exit

$ .venv/bin/python cityops.py pull --date 2026-09-20
INFO    cityops: pulling 311 data for 2026-09-20
INFO    cityops: 311 rows=8839 rain_mm=10.9
INFO    cityops: pull complete: wrote data/2026-09-20.json
$ .venv/bin/python cityops.py pull --date yesterday
bad --date 'yesterday': use YYYY-MM-DD

The if __name__ == "__main__": guard is doing quiet, essential work: importing the module (for tests, for reuse) doesn't run the CLI. Verified, not asserted — import cityops completes silently. Post 1 introduced the guard; this is the post where it earns its keep.

And the date validation is Post 5's boundary instinct in CLI form: fromisoformat rejects yesterday with a ValueError, and we catch it only to add context — which value was bad and what format was expected — then exit nonzero. That's the exception rule: catch to add context or to recover, never to swallow.

Break it, three ways

1. The phantom environment. You already watched this one happen: a fresh venv has no requests, and the script dies with ModuleNotFoundError before doing anything useful. The break isn't the error — it's that Tom had to discover your dependencies by trial and error. The requirements.txt from part 1 is the fix, and it's a two-line install. Write it down before the first handoff, not after the first failure.

2. The drifting pin. A requirements.txt that says requests with no version is a suggestion, not a contract. The laptop this post was written on carries requests 2.31.0; the fresh environment installed 2.34.2 — same command, different answers, months apart. Most days the drift is harmless. The day it isn't, Tom rebuilds the environment, something subtle changes, and the failure looks like your code broke. pip freeze output — exact versions, transitive dependencies included — is what makes a rebuild a rebuild rather than a lottery ticket.

3. The void. Run the CLI and throw away stdout:

$ .venv/bin/python cityops.py pull --date 2026-09-20 > out.txt
INFO    cityops: pulling 311 data for 2026-09-20
INFO    cityops: 311 rows=8839 rain_mm=10.9
INFO    cityops: pull complete: wrote data/2026-09-20.json
$ wc -c < out.txt
0

out.txt is empty — every log line went to stderr and the terminal, nothing to stdout. That's the design working: stdout is reserved for data. But flip it around and you see the failure mode this whole post exists to kill: a script that print()s its only evidence into a terminal nobody watches, run by a scheduler that discards the output. Maria's "did it run last night?" gets answered by cityops.log, not by anyone's memory of a terminal.

And the last break is the smallest: run it with no arguments at all.

$ .venv/bin/python cityops.py
usage: cityops [-h] {pull,report} ...
cityops: error: the following arguments are required: command

Exit code 2, with usage instructions attached. Compare that to the script that requires editing a variable at the top of the file and rerunning — or worse, silently pulls the wrong date. A CLI that explains itself is a script Tom can operate without you.

Productionize: the nightly-run checklist

Four habits turn the demo into something you'd schedule:

Project layout. Flat and boring beats clever:

cityops/
├── .venv/               # the environment (never committed, never copied)
├── cityops.py           # the CLI
├── requirements.txt     # the contract (pip freeze output)
├── cityops.log          # the memory (created at runtime)
└── data/                # pulled JSON lands here

Log rotation. A nightly pull writing one log file will eventually fill a disk — the most boring outage in the catalog. RotatingFileHandler caps it:

from logging.handlers import RotatingFileHandler
handler = RotatingFileHandler("nightly.log", maxBytes=400, backupCount=3)

$ ls -la nightly.log*
-rw-rw---- 1 root root 396 Oct  3 04:56 nightly.log
-rw-rw---- 1 root root 396 Oct  3 04:56 nightly.log.1
-rw-rw---- 1 root root 396 Oct  3 04:56 nightly.log.2
-rw-rw---- 1 root root 396 Oct  3 04:56 nightly.log.3

Four files, oldest quietly retired. (The 400-byte limit here is a demo to make rotation visible; in production you'd use something like 10 MB.) Maria's month of history stays readable; the disk stays unfilled.

Exit codes that mean something. Measured, not assumed:

$ .venv/bin/python cityops.py pull --date 2026-09-20 >/dev/null 2>&1; echo $?
0
$ .venv/bin/python cityops.py pull --date yesterday >/dev/null 2>&1; echo $?
1
$ .venv/bin/python cityops.py >/dev/null 2>&1; echo $?
2

0 means success, 1 means the run failed (bad input, failed pull), 2 means argparse rejected the invocation itself. Schedulers, cron, and Tom's wrapper scripts all read exit codes — a script that exits 0 on failure is lying to its operator. (One honesty note: SystemExit with a string message prints the message to stderr and exits 1 — that's the behavior you just measured.)

No blanket excepts. The only try/except in the CLI wraps date parsing, and it converts a ValueError into a clear message plus a nonzero exit. Everything else — network errors, bad responses — propagates as a traceback into the log, which is exactly where you want an unexpected failure to land: loud, timestamped, and attributable. A bare except: around main() would turn every failure into silence with exit code 0. Don't.

Explain it to the customer

The handoff note to Tom — this is the actual deliverable. Everything else was preparation:

Notice what's in there: install steps, run steps, where the evidence lives, what the exit codes mean, and how to ask for help usefully. That's the whole post in one paragraph. A handoff note that doesn't include "how do I know it worked" isn't a handoff — it's an abandonment.

Where this lands in CityOps

The venv, the requirements.txt, the logging setup, and the CLI are the scaffolding Milestone 2 stands on. The intake pipeline runs nightly on a schedule — which means no human watches it, which means the log file is the interface. Every lesson so far produced code that works; this is the lesson that makes it operable: installable by Tom, runnable without you, and auditable by Maria the morning after.

Post 4's principle was if you can't rerun it, you didn't clean it. Post 6's was don't trust the happy path. Post 8's: "it works on my machine" is not a deployment strategy — the environment, the logs, and the CLI are what turn a script that works into a system that runs.