The bug report comes in at 2 PM on a Tuesday: the payment service is returning 500s in production. You check staging. It works fine. You SSH into the production pod and look at the config. The PAYMENT_GATEWAY_TIMEOUT_MS environment variable is set to 3000. In staging it's 10000. Nobody knows when that changed or why.
This is environment drift. Not the catastrophic kind — no data lost, service recovered in 20 minutes. But this is how the mechanism works, and I want to trace it from its origin rather than just documenting what it looks like at the point of failure.
The First Edit Nobody Documented
Drift doesn't start with malice or laziness. It starts with a legitimate fix under time pressure.
Scenario: three months ago, production was having intermittent timeout errors with the payment gateway. An engineer SSHed in, increased PAYMENT_GATEWAY_TIMEOUT_MS from 3000 to 10000, confirmed the timeouts stopped, wrote a Jira ticket to "update config permanently." The ticket sat at the bottom of the backlog. Two sprints later the engineer who filed it left for another company. The Jira ticket is now "In Review" because someone moved it there by accident during a sprint planning clean-up.
The staging environment never got updated because staging didn't have the timeout problem. Staging uses a mock payment gateway that responds in under 100ms. The timeout value is irrelevant there. So the fix was applied, the immediate problem was solved, and the config divergence was invisible for three months.
This is not unusual. It's the default trajectory for production config changes made without automated enforcement. The first manual edit that doesn't get backfilled to all environments is the origin event.
How Drift Accumulates: Three Vectors
Manual SSH edits are the most obvious vector but not the only one. Let me trace the three patterns we see most often.
Vector 1: The manual fix that never got codified. Described above. One environment gets a live edit. The IaC (Terraform, Helm values, Kubernetes ConfigMap) never gets updated to reflect it. Subsequent deployments don't overwrite the manual change if they're additive rather than declarative (e.g., kubectl apply preserves fields not in the manifest if the field existed before). The drift persists through deployments.
Vector 2: Dependency version pinning divergence. Staging gets a dependency upgrade as part of testing. The upgrade goes well in staging. A production deployment happens the next day with the old version because nobody bumped the production Helm values file. Now staging is on Postgres client library 15.3 and production is on 15.1. These are nominally compatible. Then a query behavior changed in a minor version between the two, and production is executing that query differently without error — just with different results in a corner case that only shows up at 3 AM under specific load patterns.
Vector 3: Feature flag divergence. Feature flags are supposed to make environment differences explicit and controllable. In practice, they often make drift harder to detect because the drift is now intentional-looking. If staging has NEW_AUTH_FLOW=true and production has NEW_AUTH_FLOW=false, that difference is by design during rollout. But now you have two code paths running in parallel, and bugs in the disabled path accumulate undetected until the flag flips in production. The flag mechanism that was supposed to reduce risk has inadvertently created a category of configuration divergence that's nearly impossible to test comprehensively.
Why Automated Drift Detection Is Harder Than It Sounds
The intuitive solution: compare environment configurations on a schedule, alert on differences. Several tools do this — config drift detection is a recognized category. The problem is that not all config differences are drift. Some are intentional:
- Staging has different resource limits (4 CPU / 8GB) than production (8 CPU / 32GB). Expected.
- Staging points at a mock third-party API. Production points at the real one. Expected.
- Staging has verbose logging (
LOG_LEVEL=debug). Production hasLOG_LEVEL=warn. Expected. - Staging has a feature flag enabled for testing. Production doesn't. Expected.
A naive diff tool generates so much expected-difference noise that engineers stop reading the alerts. The alert for "PAYMENT_GATEWAY_TIMEOUT_MS is different across environments" gets mentally filed under "probably intentional" and ignored, right alongside the legitimate staging-vs-production resource difference alerts.
Useful drift detection requires a model of what differences are expected. Which means you need to explicitly declare your expected environment differences — and maintain that declaration as your configuration evolves. This is infrastructure work that often doesn't get done until after the first painful incident.
The Terraform / Helm Values Paradox
Infrastructure-as-code was supposed to solve this. Define your config in code, apply it consistently, version it in git. And it does solve the problem — when applied consistently. The failure mode: a mix of IaC-managed and manually-managed configuration within the same environment.
The paradox arises because Terraform and Kubernetes are inherently stateful. terraform apply reconciles declared state with actual state, but only for the resources declared in your configuration. If you created a resource manually (or via some other mechanism), Terraform doesn't know about it, doesn't touch it, and doesn't complain. Your Terraform plan is "green" and your environment has manual configuration that Terraform will never reconcile.
The same pattern appears in Kubernetes: a kubectl apply on a Deployment manifest will update the fields specified in the manifest. Fields that exist in the live Deployment but aren't in the manifest — say, a spec.template.spec.containers[0].env entry added manually — survive the apply unchanged. Your manifest is the source of truth according to your CI pipeline. But it isn't the actual source of truth.
The correct fix is kubectl apply --prune (for resource-level reconciliation) combined with strict policy enforcement preventing direct cluster mutation except via the CI pipeline. Both of those require organizational discipline that's hard to maintain as team size and time pressure increase.
What Makes It Hard to Stop
Once drift exists, there's a second-order problem: fixing it requires confidence that the production configuration is wrong, not the staging configuration. This is a judgment call, and it's often genuinely unclear.
In our payment timeout example: is the staging value (10000ms) correct, or is the production value (3000ms) correct? Well, 10000 was the emergency fix applied to production months ago — which means the production environment (confusingly) has the wrong value even though the emergency fix was applied to production. If you're investigating the incident cold, without the history, the "correct" value isn't obvious from the config alone.
This is why incident post-mortems that surface drift issues often result in extended investigation time: you have to reconstruct the history of which environment made the deliberate choice and which got the unintentional divergence. Without good change audit trails in both your IaC and your CI pipeline, this reconstruction is done via Slack search and git blame, neither of which is comprehensive.
We're not saying IaC solves everything — it solves the mechanical part of consistent application. The harder part is the cultural and process part: the discipline to route all configuration changes through the declarative path even when it's slower than SSH. Teams that build that discipline early spend significantly less time in post-mortems. Teams that let it slip start needing dedicated drift remediation work 12-18 months in.
The Minimum Viable Drift Prevention Stack
If you're a growing engineering team and you want the simplest defensible approach to environment drift prevention, here's what we consider the minimum:
1. Track config values explicitly in version control, separate from secrets. A config/ directory in your repo with per-environment values files. These are not used directly — they're the source of truth that gets applied by your CI pipeline. Any config change that bypasses this directory is undocumented drift.
2. CI pipeline enforces the apply, not humans. No direct cluster access for config changes in production. All changes go through: PR → review → merge → CI applies to staging → CI applies to production. Human access to production is read-only (exec for debugging) but not write.
3. Explicit expected-difference declarations. A small YAML file listing the config keys that are expected to differ between environments, and what the valid values for each environment are. Any drift outside this declared set is an alert.
# config/expected-env-diff.yaml
expected_differences:
LOG_LEVEL:
staging: debug
production: warn
RESOURCE_LIMITS_CPU:
staging: "2"
production: "8"
PAYMENT_GATEWAY_URL:
staging: "https://mock.payment.internal"
production: "https://api.payment-provider.com"This file is the documented contract between environments. Anything not on the list should be identical. Your drift detection runs against this contract, not against an arbitrary diff of all config keys. The signal-to-noise ratio becomes useful.
None of this is exotic tooling. All of it requires upfront investment that feels slow when you're moving fast. The teams that skip it discover in month 18 why the investment would have been worth it.