There's a particular kind of engineering debt that Kubernetes introduced almost by accident. Not the cluster config debt — everyone talks about that. I mean the CI pipeline debt that grew up in K8s's shadow: the Helm chart test jobs, the multi-namespace staging harnesses, the kubectl diff steps bolted onto pipelines that started life as simple build-and-push workflows.
We've been talking to platform teams about this for the past year, and the pattern is consistent enough that I want to put numbers on it. Not fabricated numbers — patterns from conversations, plus our own experience building SuperPlane's pipeline infrastructure on EKS. The picture is not flattering for the average K8s shop.
How Kubernetes Made CI Complicated
Before Kubernetes, a typical CI pipeline had a clear shape: run tests, build artifact, push to registry, deploy via SSH or a simple API call. Maybe 150-300 lines of YAML if you were being thorough. The deploy step was a rounding error on total pipeline time.
Kubernetes changed the deploy step from a line of shell into a sub-discipline. You now have:
- Manifest templating (Helm or Kustomize or both)
- Image tag substitution across environments
- Namespace-scoped resource validation (kubectl apply --dry-run)
- Rollout status polling (kubectl rollout status, with a timeout you picked somewhat arbitrarily)
- Post-deploy smoke tests that hit the cluster endpoint
- Cleanup jobs for ephemeral preview environments
- Secret management that's handled differently per environment because you set it up before you had a good pattern
That's not a deploy step anymore. That's a second pipeline inside your pipeline. And it needs maintenance.
What Platform Teams Actually Do Each Week
When we ask platform engineers to time-box their pipeline maintenance work — not pipeline development, just keeping existing pipelines working — the answers cluster around 6-10 hours per week per team. That's for teams with 3-8 engineers running 15-40 services on EKS.
The work splits roughly like this:
- Flaky environment issues: ~3 hours. A pod didn't come up before the rollout timeout, so the pipeline failed. The deploy actually succeeded. Someone has to investigate, retrigger, verify.
- Helm chart updates propagation: ~2 hours. A shared base chart gets a version bump. Now you're diffing N service pipelines to make sure nothing broke the image tag substitution pattern.
- Debugging kubectl authentication issues in CI: ~1 hour. IRSA token refresh, serviceaccount binding changes in the cluster, GitHub Actions' OIDC configuration drifting from what your cluster expects.
- Pipeline YAML changes from infra PRs: ~2 hours. Someone adds a new environment variable requirement at the cluster level, now 12 service pipelines need to add an
env:block.
That's a rough 8-hour median. One engineer-day per week that isn't building product.
The EKS Authentication Problem Specifically
I want to dwell on the kubectl authentication issue because it's the most invisible until it bites you.
When you're running CI on GitHub Actions and deploying to EKS, the typical setup uses OIDC federation: GitHub's OIDC provider issues a token, your IAM role trusts GitHub's issuer, and your workflow assumes the role via aws-actions/configure-aws-credentials. This works reliably. Until something changes.
What changes:
- The IAM role's trust policy has a condition on the
subclaim. Someone renames a branch or changes the workflow file path. Now the token's subject doesn't match the condition and you get an access denied that looks like a network error. - The EKS cluster's aws-auth ConfigMap gets edited during a node group rotation. The IAM role mapping that your CI assumed got removed because the person doing the rotation was looking at a doc that predated your setup.
- Your kubeconfig file in the GitHub Actions environment caches a cluster endpoint that's been recycled after a cluster upgrade.
None of these failures are caught by a normal "is CI passing" check. They surface as intermittent kubectl errors that look indistinguishable from transient network problems. The pipeline retries, sometimes succeeds (if the issue was timing-related), and nobody investigates the root cause until it happens three times in a week and an engineer actually sits down with CloudTrail.
Helm Complexity as a Force Multiplier
Helm is a perfect example of a tool that solves one hard problem and creates three medium problems. The hard problem it solves: parameterized Kubernetes manifests. Genuinely useful. The medium problems:
1. values.yaml inheritance hierarchies that nobody fully understands. Most K8s shops end up with a base values file, a per-environment override, and a per-service override. Sometimes a per-deployment override on top of that. The merge semantics for nested maps in Helm are non-obvious, and the effective values for any given deployment are spread across three to four files that only get merged at helm install / upgrade time.
In your CI pipeline, this means your "what will actually get deployed" question requires running helm template locally and diffing the output — which almost nobody does on every PR, because it's slow and requires cluster connectivity.
2. Chart version bumps are mini-migrations. When a shared chart increments a major or minor version, you're often looking at changed default values, renamed keys, or deprecated configurations. The correct response is careful testing. The common response is updating the chart version in 12 places, watching CI run, and hoping the smoke tests catch anything significant.
3. The --wait flag is doing work you can't see. helm upgrade --wait --timeout 10m is in almost every K8s deployment step. What it's actually doing: polling the cluster's rollout status every few seconds, checking readiness probes, waiting for pods to replace. If your readiness probe is too sensitive (returns 503 under normal startup load), your pipeline "fails" on perfectly good deploys. You start tuning initialDelaySeconds based on CI failure patterns rather than actual application behavior, and now your pipeline configuration is driving your runtime configuration rather than the other way around.
The Rollout Status Polling Problem
Here's a concrete scenario. A team running a monorepo on EKS with 18 services. Each service has its own GitHub Actions workflow. Deployments to staging use helm upgrade --wait --timeout 8m. Their services typically come up in 60-90 seconds.
What they observe: about 12% of staging deployments time out. Investigation reveals: the timeout isn't because pods failed to start — it's because a transient issue in cluster DNS resolution delayed the readiness probe from returning a 200 on first contact. Pods are healthy within 95 seconds. But the readiness check fires at 90 seconds, returns a temporary 503, and the Helm wait loop records that as "not ready" and starts its countdown from there.
The fix teams reach for: increase the timeout. Now it's --timeout 15m. This means pipelines that do fail legitimately take 15 minutes to fail rather than 8 minutes. At 30 deploys per day across 18 services, most of which succeed, the average pipeline time increases by 0. But when something actually breaks, feedback latency doubles. That's a meaningful developer experience hit, and it came entirely from an EKS pipeline detail that has nothing to do with application code quality.
What This Costs Beyond Engineer Time
There's a second-order effect that's harder to measure but probably larger in impact: pipeline complexity correlates with deployment frequency decay.
Teams that started deploying multiple times per day often drift toward once-a-day or less when their K8s pipeline becomes fragile. The reasoning is explicit: "We don't want to trigger a 20-minute pipeline run for a one-line change, especially if there's a 1-in-10 chance it fails on something unrelated to our code." So teams batch changes. Batching changes means merging more things at once. Merging more things at once means harder debugging when something breaks. The pipeline complexity doesn't just waste time — it changes behavior in ways that compound.
We're not saying Kubernetes is wrong here, or that the complexity is avoidable given what K8s is doing. The abstraction genuinely earns its cost when you need fine-grained deployment control, horizontal autoscaling, and multi-region topology. But the pipeline complexity that comes along with it deserves its own budget line and its own measurement discipline.
If your platform team can't tell you how many engineer-hours per week go to pipeline maintenance (not pipeline feature work — maintenance), that number is almost certainly higher than you'd want it to be. Six to ten hours per week is common. That's one engineer for a week every month who isn't working on the platform capabilities your product teams are waiting for.
The Measurement Starting Point
If you want to put a real number on this, start here:
# Export your CI failure data by failure type
# For GitHub Actions, query the Actions API:
gh api /repos/{owner}/{repo}/actions/runs \
--field status=failure \
--field per_page=100 \
--jq '.workflow_runs[] | {id, name, conclusion, created_at, updated_at}'Then tag each failure as: code failure (test/lint), infrastructure failure (kubectl/Helm/auth), or flake (retry passed). If you've never done this, expect to find that 25-40% of CI failures in a mature K8s shop are infrastructure category, not code category. Those are the ones your platform team is firefighting. That's the number that should go into your quarterly planning conversation.
SuperPlane classifies failure category automatically, which is why this was one of the first things we built — not because it's the most sophisticated problem, but because without the classification you're flying blind on where maintenance time is actually going.