Every system you'll ever touch as an FDE runs on secrets — API keys, passwords, tokens. Handle them wrong once and you'll learn why the hard way. This is the short, practical guide to never being that story.
Monday, 9:12 AM. Your CityOps 311 pipeline finally works end to end — it pulls live complaint data from the city's open data portal and lands it in your database. You're proud. You push the repo to GitHub so your teammate can review it, grab coffee, and come back to a Slack message from a stranger:
"Hey — your repo has a live API key in pull_311.py. Bots scan GitHub for these within minutes. You should revoke it."
Your stomach drops. The key is the CityOps application token, issued to your team to identify your integration to the city's open data API. It's now public. It doesn't automatically grant access to private data — the 311 dataset is public — but someone could misuse it, exhaust its quota, or disrupt your integration. You revoke it and investigate the exposure.
This post exists so that Monday never happens to you.
Clarify the ask: what counts as a secret?
Before fixing anything, get precise about what you're protecting. A secret is any string that proves you are allowed to do something: read private data, write to a system, spend someone's money. If someone else learns it, they can impersonate you.
Not every key has the same blast radius. An FDE must assess the actual permissions of an exposed credential, not assume every key is administrator-level:
| Credential type | What exposure may allow |
|---|---|
| Public-data API application token (like the CityOps 311 token) | Unauthorized use of the application's quota or identity, depending on provider controls |
| Privileged API access token | Access to protected data or operations within the token's permissions |
| Secret — guard it | Not a secret — safe in code |
|---|---|
| API keys and tokens | Public API endpoint URLs |
| Database passwords and connection strings | Database hostnames (without credentials) |
| Private SSH keys | Public SSH keys (*.pub) |
| OAuth client secrets | OAuth client IDs (usually) |
| Signing keys, encryption keys | Timeouts, retry counts, feature flags |
The rule of thumb: if leaking it would let a stranger act as you, it's a secret. When in doubt, treat it as a secret — over-protecting a timeout value costs nothing; under-protecting a key costs everything.
The minimum concept: secrets are config, not code
Here is the entire mental model this post hangs on:
The one principle
- A secret in code is a secret already shared. Code gets copied, screenshotted, pasted into chat, committed, pushed, and backed up. Every one of those is a place your secret now lives that you don't control.
- Secrets are configuration, not code. Code describes what to do; secrets describe who is allowed to do it. They change for different reasons, on different schedules, and they belong in different places.
- So: code goes in Git. Secrets go in the environment. Your program reads them at runtime and never contains them. One honest caveat: environment variables keep credentials out of source code, but they are not encryption — they can leak through process inspection, crash reports, debug output, or poorly configured containers. Protect local files, limit process access, avoid logging values, and use an approved secret manager for production. Likewise,
.envis a plain-text local file, not an encrypted vault.
Build it: the right way, step by step
Let's wire the CityOps 311 pull script the right way. First, the lazy way — so you can see exactly what's wrong with it:
# pull_311.py -- THE LAZY WAY (don't do this)
API_KEY = "cityops_live_9f3a2b7c4d1e"
def fetch_complaints():
print("Fetching 311 complaints using configured credentials")
fetch_complaints()
$ python3 pull_311.py
Fetching 311 complaints using configured credentials
It works. That's the trap — the lazy way always works right up until it destroys you. The key is sitting in plain text, and anyone who can read the file owns it:
$ grep -rn "cityops_live" .
./pull_311.py:2:API_KEY = "cityops_live_9f3a2b7c4d1e"
Now the right way. The script reads the key from the environment — per-process variables the OS hands to your program — and refuses to run without it:
# pull_311.py -- THE RIGHT WAY
import os
api_key = os.environ.get("CITYOPS_API_KEY")
if not api_key:
raise SystemExit(
"CITYOPS_API_KEY is not set.\n"
"Copy .env.example to .env, add your key, then load it:\n"
" set -a; source .env; set +a"
)
print("Fetching 311 complaints using configured credentials")
This small script demonstrates loading a required credential and failing safely when it's absent — it does not yet perform the actual 311 API request. (A full integration would pass the token according to the provider's documented authentication mechanism, never in a URL or a log line.) Notice there's no try/except here — and that's deliberate. A missing secret isn't something to recover from; it's a configuration error the human must fix. Failing fast with a message that says exactly what to do is kinder than crashing ten lines later with a confusing authentication error. (That's the standing rule: only catch an exception when you can add useful context or recover from it.)
$ CITYOPS_API_KEY=demo_key_abc123 python3 pull_311.py
Fetching 311 complaints using configured credentials
Warning: use inline environment-variable assignments only with fake demonstration values like this one. Don't type real secrets directly into commands — depending on your shell and tooling, typed commands can be saved in shell history or captured elsewhere.
The normal workflow loads the key from your .env file instead of the command line:
$ cp .env.example .env
# ... populate .env with your real key, using an appropriate local method ...
$ set -a; source .env; set +a
$ python3 pull_311.py
Fetching 311 complaints using configured credentials
(That's Bash syntax. In Windows PowerShell the loading procedure differs — you'd set the variable with $env:CITYOPS_API_KEY = "..." or use a dotenv loader — but the principle is identical: the secret enters the process environment, never the command history.)
$ python3 pull_311.py
CITYOPS_API_KEY is not set.
Copy .env.example to .env, add your key, then load it:
set -a; source .env; set +a
Two files make this work in a real project. First, .env — your personal secrets, one per line, never committed:
# .env -- YOUR secrets. Never commit this file.
# Illustrative replacement value, not a real credential.
CITYOPS_API_KEY=REPLACEMENT_KEY_FROM_PROVIDER
If the original key was exposed, moving it into .env does not make it safe again — revoke it and obtain a replacement. A new file doesn't un-leak a compromised key.
Second, .gitignore — the bouncer that keeps .env out of Git. (Gitignore patterns were covered in The Terminal and Git, Demystified — this is where that lesson pays for itself.)
# .gitignore
.env
__pycache__/
*.pyc
$ git status --short
?? .gitignore
See that? Git sees .gitignore but not .env — for untracked files, the ignore rule keeps it out of git add. But be precise about what .gitignore is: it helps prevent accidental additions of untracked secrets. It does not protect a file that was already tracked, and it doesn't stop anyone from force-adding an ignored file. It is not a security boundary — it's a guardrail, not a vault. It is not a substitute for secret scanning or access controls.
Check whether a sensitive file is already tracked:
$ git ls-files .env
.env
If that prints .env, the file is tracked despite the ignore rule. To stop tracking it while keeping your local copy:
$ git rm --cached .env
$ git commit -m "Stop tracking local environment file"
But if that file contained an exposed credential, rotate the credential first — removing it from the latest commit doesn't erase it from history.
And third, .env.example — a committed template with fake values showing teammates what they need:
# .env.example -- commit this. It shows WHAT is needed, not the values.
CITYOPS_API_KEY=your_key_here
The three-file pattern — memorize it
.env— real secrets. Gitignored, never committed..gitignore— lists.envso normal Git commands skip it. (Guardrail against accidents, not a security boundary.).env.example— committed template with placeholder values, so the next person knows what to fill in.
Break it: why "just delete it" doesn't work
Suppose the lazy way already happened — the key got committed. You delete the line and commit the fix. Safe now? Let's check:
$ git log --all -p -- pull_311.py | grep -c "cityops_live_9f3a2b7c4d1e"
2
The key appears twice in history even though the current file is clean. Deleting a secret from the latest file does not remove it from Git history. Technically, history can be rewritten — but you cannot reliably recall copies that have already been cloned, forked, cached, or downloaded, and repository hosts may retain their own copies. Treat the credential as compromised and revoke it. There is no "oops" button; there is only the revoke playbook:
When a secret leaks — do this immediately
- Contain. Disable the exposed credential and restrict further exposure — revoke it now. Don't clean up first; a live leaked key is being abused while you tidy.
- Coordinate and preserve evidence. Notify the designated security contact promptly and follow their containment, investigation, and disclosure requirements. Record when the exposure was discovered, which repository and commit were involved, which permissions the credential had, and what logs are available. Avoid destroying evidence while cleaning up.
- Restore. Issue a least-privileged replacement and update every authorized consumer — your
.env, the server, your teammate's machine. For critical production credentials, coordinate containment and restoration to minimize disruption. - Investigate. Review usage logs for the exposure window, the credential's permissions, and which systems were reachable. Do not independently decide the incident is closed because no unusual usage appeared.
- Prevent recurrence. Add secret scanning, review controls, and an approved secrets workflow so the same mistake can't recur.
Productionize: least privilege and secret managers
The .env pattern covers you on your own machine. Two more ideas carry you into real client work.
Least privilege: keys should be weak on purpose
Least privilege means every secret gets the minimum power it needs — nothing more. Use the narrowest permissions and quotas the provider actually supports. For a privileged credential, that might mean read-only access to one dataset instead of admin access to the whole portal — then, when (not if) a key leaks, the blast radius is a bruise instead of a crater. One caveat: not every provider supports per-token scopes. If the CityOps token only identifies a public-data application, document that limitation rather than inventing read-only scopes that don't exist. A read-only key that leaks is an incident; an admin key that leaks is a disaster.
Secret managers: the team-scale answer
.env files don't scale to teams — you can't Slack secrets around (that's just leaking with extra steps). For shared and production systems, teams use a secret manager: a locked vault (like HashiCorp Vault, AWS Secrets Manager, or Doppler) where secrets live encrypted, access is logged, and rotation is a button press instead of a scavenger hunt. You don't need to operate one today — just know the shape of the answer: local dev → .env; teams and production → a secret manager.
Later, not now
- Running your own Vault cluster, key rotation automation, short-lived dynamic credentials — real skills, but they're Stage 5 (Deploy) territory.
- Today: the three-file pattern, least privilege, and the revoke playbook. That's 95% of the protection for 5% of the effort.
Communicate: telling the customer a key leaked
This is the part nobody teaches and everybody needs. If a secret leaks on your watch, the customer hears it from you — fast, specific, and without minimization:
"Lisa — heads up on something I need to own. This morning I found that the CityOps 311 application token was committed to our repo in plain text for about three hours before I caught it. I've already revoked the token and issued a replacement, and I've verified the pipeline runs cleanly on the new token. Our initial review found no unusual usage in the available logs for that window; we are continuing the assessment with the security team. Here's what I'm changing so it can't happen again: secrets move to environment variables with a .env + .gitignore pattern starting today, and I'm adding a pre-commit check that scans for key patterns. Happy to walk through any of this — and I'm sorry this happened."
Note the shape: what happened, what you already did, what the impact was, what changes next. No excuses, no jargon, no "it was just a dev key." And note what it doesn't do: it doesn't claim clean logs prove the key was never misused, and it doesn't declare the pipeline healthy on the new key until the post-rotation test actually passed. Customers forgive mistakes; they don't forgive cover-ups.
Break it on purpose: catch a secret before it commits
The lesson mentions a pre-commit scanner — now make it real with a five-minute exercise. You don't need a full security platform; any approved secret scanner works (your team or provider will have a recommended one).
- Add a fake credential pattern to a disposable test file — something obviously fake like
FAKE_TEST_KEY=abc123_do_not_useinscratch_test.txt. - Run the secret scanner against the repository.
- Observe that the scanner flags the value.
- Delete the test file and rerun the scan to confirm it's clean.
Two honest caveats: scanners can miss credentials (novel formats, encoded values) and they produce false positives (test fixtures, documentation examples). A clean scan is evidence, not proof. The scanner is one layer — the three-file pattern and code review are the others.
CityOps: wire it into the project
Back to the milestone. Your CityOps scaffold from Milestone 1: The CityOps Scaffold now gets its secrets discipline:
cityops/
├── .env # real key -- gitignored, on your machine only
├── .env.example # CITYOPS_API_KEY=your_key_here -- committed
├── .gitignore # contains: .env
└── scripts/
└── pull_311.py # reads os.environ, fails fast if unset
Every future CityOps script that needs a credential follows the same three-file pattern. When a teammate joins, they copy .env.example to .env, paste their own key, and everything works — no secrets in chat, no secrets in Git, no Monday-morning Slack messages from strangers.
Field check
- You find
DB_PASSWORD="s3cret!"hardcoded in a client's legacy script that five people share over email. What's your first move, and what do you set up so it stays fixed? - A teammate says ".env is annoying — let's just put the staging key in the repo, it's only staging." How do you respond?
- Lisa asks: "If our 311 API key leaked yesterday, how would we know, and what would you do in the first 30 minutes?" Draft the reply.
What a good answer looks like
1. First move: rotate that password immediately — it's been emailed around, so assume it's compromised. Then move it to an environment variable, add the three-file pattern (.env + .gitignore + .env.example), and get everyone off the emailed copy. To keep it fixed: the committed .env.example documents what's needed, and a note in the repo README explains the pattern so the next person doesn't regress.
2. Staging keys leak into production more often than anyone admits — configs get copied, repos get made public, contractors keep clones. The habit has to be unconditional: no secrets in repos, ever. The three-file pattern costs thirty seconds; a leaked staging key that turns out to also work on production costs a career story.
3. How we'd know: the provider's usage dashboard — unexpected quota burn or calls from unfamiliar IPs during the exposure window. First 30 minutes: contain (revoke the token), coordinate (notify the security contact, preserve evidence), restore (issue the replacement, verify the pipeline on it), then tell Lisa what happened, what's known, and what's changing — without claiming clean logs prove no misuse.
A good FDE treats credentials as customer trust. Keep them out of source code, give them only the permissions they need, and when one leaks, contain it quickly, investigate the impact, and communicate what is known — without pretending the risk disappeared when the file was deleted.
Comments
Post a Comment