Deploying Behind Enterprise Proxies & Firewalls

Nothing reaches the open internet from the client network. pip install fails. Container pulls fail. TLS verification breaks in ways that look like the code is wrong. This is the lesson for the day you deploy somewhere the internet doesn't reach — and still ship.

The customer problem

Clarify the ask

"The network is locked down" is not one problem — it's a list. Before touching anything, enumerate what the deployment actually needs from outside the client's walls, split by when it needs it:

Two things fall off the list immediately: DNS and NTP are usually served inside the client network — don't assume they need the internet. And "the internet" is never the requirement; a short list of named hosts is. That list is what you hand Tom when you ask for firewall exceptions — specific, justified, minimal.

Minimum concept: proxies, TLS interception, and trust

Three mechanisms explain nearly every "works on my machine, fails on theirs" network incident:

1. The proxy and its env vars

An HTTP proxy is a middleman: instead of connecting to pypi.org directly, your tools connect to the proxy and ask it to fetch on their behalf (the CONNECT method for HTTPS). Tools discover the proxy through environment variables:

export HTTP_PROXY=http://proxy.city.gov:8080
export HTTPS_PROXY=http://proxy.city.gov:8080
export NO_PROXY=localhost,127.0.0.1,.city.gov,.svc.cluster.local

• HTTP_PROXY / HTTPS_PROXY — where the proxy lives. Note the scheme is http:// even for HTTPS_PROXY: that's the proxy's own address, not the target's.
• NO_PROXY — hosts that must bypass the proxy: localhost, the city's internal domains, the cluster's service names. Forgetting this is the classic failure — internal database traffic gets sent to the proxy and dies.
• Set both cases (HTTP_PROXY and http_proxy). Most tools honor lowercase; some only read uppercase. Setting both costs nothing and ends the debate.

1b. What the proxy actually does with HTTPS

For HTTPS, the proxy doesn't read your traffic and re-send it — it opens a blind tunnel. Your tool sends CONNECT pypi.org:443, the proxy opens a TCP pipe to the target, and your TLS handshake runs through that pipe. That's why a proxy can't "fix" a broken TLS chain: with interception, the proxy terminates the tunnel and re-encrypts with the corporate CA; without interception, it just passes bytes and the original certificate must verify normally.

The practical takeaway: if the proxy requires authentication, the credentials ride in the proxy URL (http://user:pass@proxy:8080) — which is exactly why they belong in the vault, never in a file. And if a tool "doesn't support proxies," what it usually means is it doesn't read the env vars — check its own config file before declaring it broken.

2. TLS interception and the corporate CA

Many enterprise proxies re-sign TLS: the proxy terminates your HTTPS connection, inspects it, and re-encrypts it with a certificate signed by the corporate CA — a certificate authority your laptop has never heard of. Your tools see a certificate chain they can't verify and fail with CERTIFICATE_VERIFY_FAILED.

The fix is not --insecure and not "disable verification" (both train the team to ignore the exact attack TLS exists to stop). The fix is to install the corporate CA certificate into the trust store so the chain verifies properly. Lisa will ask you to confirm this, and she'll be right to.

3. Every tool has its own trust store

This is the part that burns hours: there is no single "system trust." Each layer keeps its own list of trusted CAs:

When TLS fails for one tool but works for the others, you're not looking at a network problem — you're looking at that tool's trust store.

Build: making CityOps proxy-ready

Step 1 — Diagnose before you fix

On the locked-down host, run the diagnosis in order. Don't skip to step 3:

# 1. Is a proxy even configured?
env | grep -i proxy

# 2. Can the proxy be reached, and does plain HTTP through it work?
curl -x http://proxy.city.gov:8080 -I http://example.com

# 3. Does HTTPS through it verify? (Expect failure here first.)
curl -x http://proxy.city.gov:8080 -I https://pypi.org/simple/

Step 2 passing while step 3 fails is the signature of TLS interception: the proxy works, but nothing trusts its re-signed certificates yet.

Step 2 — Install the corporate CA

Get the CA certificate file from Tom or Lisa — it usually arrives as city-ca.crt. On Debian/Ubuntu hosts and containers:

sudo cp city-ca.crt /usr/local/share/ca-certificates/city-ca.crt
sudo update-ca-certificates
# Expected: "Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done."

Verify it took effect — this time the HTTPS check should pass:

curl -x http://proxy.city.gov:8080 -I https://pypi.org/simple/
# Expected: HTTP/1.1 200 ... (via the proxy, certificate verified)

If your Python uses its own bundled CA file (some distributions and virtualenvs do), point it at the system bundle explicitly:

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
python3 -c "import urllib.request; print(urllib.request.urlopen('https://pypi.org/simple/', timeout=15).status)"
# Expected: 200

Step 3 — Teach pip the network

# One-time host config (writes to ~/.config/pip/pip.conf)
pip config set global.proxy http://proxy.city.gov:8080
pip config set global.cert /etc/ssl/certs/ca-certificates.crt

# Then the install that failed on Tuesday works:
pip install -r requirements.txt

Step 4 — The air-gap escape hatch: vendored wheels

Sometimes the answer is "no proxy for builds either." For that case, build a vendored wheel bundle on a connected machine, carry it in, and install with no index at all:

# On a machine WITH internet: download everything, wheels included
pip download -r requirements.txt -d ./wheels

# Carry ./wheels to the locked-down host (USB, internal file share, artifact repo)
# On the locked-down host: install with no index, no network
pip install --no-index --find-links ./wheels -r requirements.txt

This is fully offline and fully reproducible — the same wheels install the same bytes everywhere. Commit the requirements.txt with pinned versions and hashes (from pip hash or pip-compile --generate-hashes) so the bundle is auditable. Lisa likes this option: nothing is fetched at deploy time from anywhere.

Step 5 — Containers through the proxy

The Docker daemon needs the proxy for pulls, and builds need it for RUN pip install steps. Two configs:

# /etc/systemd/system/docker.service.d/http-proxy.conf  (daemon pulls)
[Service]
Environment="HTTP_PROXY=http://proxy.city.gov:8080"
Environment="HTTPS_PROXY=http://proxy.city.gov:8080"
Environment="NO_PROXY=localhost,127.0.0.1,.city.gov"

sudo systemctl daemon-reload && sudo systemctl restart docker

# Build args (RUN steps inside the build)
docker build \
  --build-arg HTTP_PROXY=http://proxy.city.gov:8080 \
  --build-arg HTTPS_PROXY=http://proxy.city.gov:8080 \
  --build-arg NO_PROXY=localhost,127.0.0.1,.city.gov \
  -t registry.city.gov/cityops/api:1.4.2 .

# And the corporate CA must be installed INSIDE the image too,
# or every HTTPS call the container makes will fail verification:
#   COPY city-ca.crt /usr/local/share/ca-certificates/
#   RUN update-ca-certificates

The better long-term move — ask Tom for it — is an internal container registry: base images and your app images get mirrored inside once, and the cluster pulls from there with no proxy at all.

Step 6 — The preflight script

Wrap the diagnosis in a script that runs in CI and on every deploy target. It uses only the standard library, honors the proxy env vars automatically via urllib, and exits non-zero when the network path is broken:

#!/usr/bin/env python3
"""preflight.py - verify the outbound network path for a CityOps deploy.

Checks, in order: proxy env vars, DNS resolution, and the TLS trust
chain to the package index and container registry. Prints PASS/FAIL
per check and exits non-zero if anything the deploy needs is broken.

Run:  python3 preflight.py
"""
import os
import socket
import sys
import urllib.request
from urllib.parse import urlparse

TARGETS = {
    "package index": "https://pypi.org/simple/",
    "container registry": "https://registry-1.docker.io/v2/",
}


def check_proxy_env():
    proxy = os.environ.get("https_proxy") or os.environ.get("HTTPS_PROXY")
    no_proxy = os.environ.get("no_proxy") or os.environ.get("NO_PROXY")
    ok = bool(proxy)
    print(f"[{'PASS' if ok else 'FAIL'}] proxy env: "
          f"https_proxy={proxy or '(unset)'}")
    print(f"       no_proxy={no_proxy or '(unset)'}")
    return ok


def check_dns(host):
    try:
        socket.getaddrinfo(host, 443)
        print(f"[PASS] dns: {host} resolves")
        return True
    except OSError as e:
        print(f"[FAIL] dns: {host} did not resolve ({e})")
        return False


def check_tls(url, name):
    try:
        with urllib.request.urlopen(url, timeout=15) as r:
            print(f"[PASS] tls+http: {name} -> HTTP {r.status}")
            return True
    except Exception as e:  # want the specific failure surfaced, not hidden
        print(f"[FAIL] tls+http: {name} ({type(e).__name__}: {e})")
        return False


def main():
    results = [check_proxy_env()]
    for name, url in TARGETS.items():
        host = urlparse(url).hostname
        results.append(check_dns(host))
        results.append(check_tls(url, name))
    if all(results):
        print("\nAll checks passed - this host can reach what the deploy needs.")
        return 0
    print("\nOne or more checks failed - fix the network path before deploying.")
    return 1


if __name__ == "__main__":
    sys.exit(main())

Example output on a healthy host (illustrative — your hostnames will differ):

[PASS] proxy env: https_proxy=http://proxy.city.gov:8080
       no_proxy=localhost,127.0.0.1,.city.gov
[PASS] dns: pypi.org resolves
[PASS] tls+http: package index -> HTTP 200
[PASS] dns: registry-1.docker.io resolves
[PASS] tls+http: container registry -> HTTP 401

All checks passed - this host can reach what the deploy needs.

(HTTP 401 from the registry's /v2/ endpoint is the healthy answer — it means TLS verified and the registry responded; it just wants credentials for the catalog.) Run this in CI before every deploy and the Tuesday-morning surprise becomes a Monday-afternoon warning.

Break: four ways the locked-down network bites

Productionize: the locked-network deploy checklist

Before CityOps ships into the city's network, these are settled — not improvised on deploy day:

• Preflight in CI — preflight.py runs against a host with the client's network profile before every deploy. Red means stop, not "retry harder."
• Dependency strategy chosen — either proxy-configured installs (pip config committed, CA installed) or a vendored wheel bundle with pinned versions and hashes. Not both, not neither.
• Container strategy chosen — proxy-configured daemon and builds, or (better) an internal registry mirror Tom maintains.
• CA distribution solved — the corporate CA is installed in host images and container base images, from a known source, with an owner and an expiry date on your calendar.
• NO_PROXY reviewed — internal domains listed, tested against the real internal endpoints.
• Proxy credentials in the vault — rotated on the client's schedule, never in files or chat.
• Egress allowlist requested — the exact host list from your build/deploy/run table, sent to Tom with a business justification per host (template below).
• Rollback doesn't need the internet either — the previous image and wheel bundle are retained inside the network, so a rollback works when the proxy is the thing that's broken.

The allowlist request — specific, justified, minimal:

Subject: Egress allowlist request - CityOps deployment

Hi Tom — for the CityOps deploy behind the city proxy, we need
egress to these hosts (HTTPS only):

  pypi.org / files.pythonhosted.org - Python package index (build)
  registry-1.docker.io / auth.docker.io - container registry (deploy)
  api.modelprovider.example - model inference endpoint (runtime)

Internal DNS/NTP we use the city's own servers - no exception needed.

Could you confirm these are reachable through the proxy, or let us
know the internal mirror equivalents if you'd rather we use those?

Thanks!

Notice the last line: you're offering Tom the better answer (internal mirrors) instead of demanding the internet. Infra teams remember who asks like that.

Communicate: the network, translated

• Maria ("Can we still deploy Friday?"): "Yes. The city's network blocks direct internet access, which is normal for a government network — we've configured our deploy to work through the city's proxy and verified every connection it needs. Friday holds; here's the one-page list of what we asked IT to allow." She needed a yes and a reason, not a proxy tutorial.
• Lisa ("How are you handling the proxy credentials and the TLS interception?"): "Proxy credentials come from the vault at deploy time — nothing in code or config files. For TLS, we install the city's CA into the trust stores instead of disabling verification, so the chain still validates end to end. Here's the preflight output proving it." Mechanism, not reassurance.
• Tom ("Why is the deploy pulling from the internet at all?"): "Fair question — it shouldn't, long term. Short term we need three hosts through the proxy (list above). Longer term we'd rather use your internal PyPI mirror and registry if you have them — point us at them and we'll switch." You validated his instinct instead of fighting it.

CityOps: the deployment behind the firewall

Read it as the runbook: the container talks to the outside world only through the proxy, only to allowlisted hosts, with the corporate CA in its trust store. If the proxy path fails, the wheel bundle and internal registry are the fallbacks — both inside the walls. The preflight script guards the whole path, and the checklist above is what "done" looks like.

The FDE principle for this lesson: the network is part of the system. "It works on my machine" was never a deploy plan; "it works through their proxy, with their CA, to their allowlist" is.