After shepherding over a thousand GitOps deployments, the patterns that separate resilient pipelines from brittle ones become clear. Here's what nobody tells you about managing IaC at enterprise scale while keeping engineers sane.
GitOps is a good idea with a deceptively simple pitch: the desired state of your infrastructure lives in Git, and a controller continuously reconciles reality against it. Everything is reviewed, everything is auditable, rollback is a revert. It works, and at a certain scale it becomes close to essential.
It also has failure modes that only appear once you are past a few dozen services, and they are rarely discussed by the people advocating for it. These are the ones that cost us the most time.
The repository structure decision is permanent
You will choose between one repository for everything, one per team, or one per environment. Whatever you pick becomes extremely expensive to change within about six months, because tooling, permissions, CI configuration and muscle memory all calcify around it.
Our recommendation, having lived with all three: one repository per team for application manifests, plus one central repository for shared platform infrastructure. A monorepo makes access control nearly impossible to express and turns every change into a merge conflict. Per-environment repositories guarantee that staging and production drift, because nothing structurally forces a change to propagate.
Environments are branches — no, they are directories
Using Git branches to represent environments is intuitive and wrong. Promoting a change becomes a merge, merges accumulate conflicts, and cherry-picking a hotfix into production means the branches diverge permanently. Directories with explicit overlays are less elegant and vastly easier to reason about, because the difference between staging and production is a file you can read rather than a merge history you have to reconstruct.
If you cannot answer 'what is different between staging and production' by reading a diff, your GitOps setup is decorative.
Drift is not the enemy you expect
The stated benefit of continuous reconciliation is that manual changes get reverted automatically. In practice this is occasionally catastrophic. An engineer scales a deployment up during an incident, the controller notices the divergence and scales it back down, and the incident gets worse for reasons nobody immediately understands.
The fix is a documented, fast break-glass procedure — a way to suspend reconciliation for a specific workload, that everyone on call knows about and that automatically re-enables after a set period. Without it, engineers learn to fight the controller, and the controller always wins at the worst moment.
Secrets are the part everyone underestimates
Declarative infrastructure in Git is straightforward right up until the configuration includes a database password. Every available answer involves a trade-off and none of them is clean.
- Encrypted secrets committed to the repository keep everything in one place, but rotation requires a commit and the encrypted blobs are opaque in review.
- An external secret store referenced by name keeps credentials out of Git entirely, at the cost of a runtime dependency that can fail during a deployment.
- Operator-injected secrets from a cloud provider's manager are usually the best fit for teams already committed to a single cloud, and awkward for anyone hybrid.
Reconciliation time is a scaling limit
With fifty applications, the controller reconciles everything in seconds and nobody thinks about it. With a thousand, full reconciliation takes minutes, and an urgent deployment sits in a queue behind hundreds of no-op checks. Teams notice this as 'deployments got slow' long before anyone connects it to the reconciliation loop.
Shard the controllers by team or by cluster before you need to. Watch reconciliation duration as a first-class metric alongside deployment frequency, and treat a rising trend as a capacity signal rather than a curiosity.
The cultural part
GitOps moves infrastructure changes into code review, which is the point, and it also means an infrastructure change now waits for a reviewer. If pull requests routinely sit for a day, you have added a day to every infrastructure change, and engineers will route around it.
- 1Auto-merge low-risk, well-tested changes — image tag bumps that passed CI do not need a human.
- 2Require review proportional to blast radius, not uniformly across all changes.
- 3Give every repository a named owner with a response-time expectation, so review is somebody's job rather than everybody's.
- 4Measure time-to-merge for infrastructure pull requests and treat a rising number as an incident in its own right.
The teams that succeed with GitOps are not the ones with the most sophisticated tooling. They are the ones that made the reviewed, declarative path faster than typing a command against a cluster. Once that is true, adoption stops needing to be enforced.