The Terraform State That Drifted for Six Months

A client's Terraform code said one thing. Production said another. It took a routine capacity change to reveal that 40% of their infrastructure had been modified by hand and nobody knew.

The request sounded routine. A mid-size e-commerce company needed to bump their RDS instance from db.r6g.xlarge to db.r6g.2xlarge ahead of a seasonal sale. They had Terraform. They had a CI pipeline that ran plan and apply. One variable change, one PR, done by lunch.

The terraform plan output was 247 lines long.

Instead of the single RDS modification they expected, Terraform wanted to recreate two security groups, modify three IAM policies, destroy and recreate an ElastiCache cluster, and change the instance type on a load balancer that nobody remembered provisioning through Terraform in the first place. The engineer who ran the plan stared at it for a minute, closed the terminal, and called me.

The archaeology begins

I was four weeks into an engagement focused on their deployment pipeline. Infrastructure wasn't originally in scope, but the Terraform situation was blocking the RDS upgrade, and the sale was two weeks away. So I started digging.

The state file told a messy story. Their AWS account had about 180 resources managed by Terraform — EC2 instances, RDS databases, VPCs, security groups, IAM roles, the usual mid-complexity setup. When I ran a drift detection pass, 73 of those resources had attributes that didn't match what the code declared. Forty percent.

Some of the drift was cosmetic. Tags that got added by AWS Config rules. Description fields that someone updated in the console because it was faster than a PR. But plenty of it was not cosmetic at all.

The most concerning find was a security group attached to their primary application load balancer. The Terraform code allowed inbound traffic on ports 80 and 443. The actual security group in AWS also allowed inbound on port 8080, with a source CIDR of 0.0.0.0/0. Wide open. An engineer had added it during an incident three months ago to test a direct connection to an internal service, and never removed it.

How it happened

Nobody set out to let the infrastructure drift. It happened the way it always happens — gradually, one "quick fix" at a time.

The pattern was predictable. An incident occurs at 11 PM. The on-call engineer needs to open a port, increase a timeout, or add an IP to an allow list. They could write Terraform, open a PR, wait for review, merge, and let the pipeline apply. Or they could click three buttons in the AWS console and fix it right now. At 11 PM, with a Slack channel full of people asking when the service will be back, the console wins every time.

The problem isn't the emergency change. It's what happens the next morning: nothing. The engineer moves on to the next thing. The incident gets a postmortem, maybe, but "backport the console change to Terraform" is a follow-up action that lives on a Jira ticket that gets deprioritized into oblivion.

I found tickets for seven of these manual changes. All marked "low priority." The oldest was from eleven months ago.

# What Terraform thought was running
resource "aws_security_group_rule" "alb_https" {
  type              = "ingress"
  from_port         = 443
  to_port           = 443
  protocol          = "tcp"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = aws_security_group.alb.id
}
 
# What was actually running: the above PLUS port 8080
# open to the entire internet, added during an incident
# in April and never cleaned up

The fix that wasn't straightforward

You might think the solution is simple: update the Terraform code to match reality, commit it, and move on. But that approach has a problem. Some of the drift was intentional and correct — things that needed to stay. Some of it was accidental and dangerous — things that needed to be reverted. And some of it was ambiguous — changes where the person who made them had already left the company, and nobody knew whether removing them would break something.

We spent three days on the triage. I wrote a script that exported the drift report into a spreadsheet with three columns: resource, attribute, and a status field. Then I sat with their infrastructure lead and went through all 73 resources, one by one, classifying each discrepancy as "adopt into code," "revert to code," or "needs investigation."

Thirty-one changes got adopted into Terraform. Twenty-eight got reverted — including the open port 8080. Fourteen required further investigation, which mostly meant pinging former employees on LinkedIn.

What actually prevented it from happening again

After the cleanup, we put two things in place. First, a scheduled drift detection job. A nightly CI run that executes terraform plan against production and posts the diff to a Slack channel. If the plan shows anything other than "no changes," someone has to look at it before the next business day. Not a fancy tool — just a cron job, a shell script, and a Slack webhook.

Second, we locked down console write access. Not completely — the on-call engineer still needs to be able to make emergency changes. But now those changes trigger a CloudTrail alert that creates a ticket automatically. Not a ticket that sits in the backlog forever — one that blocks the next sprint's deployment if it's not resolved. Harsh, but it works because the pain is immediate and visible.

The RDS upgrade eventually went through cleanly. One variable change, one plan output showing exactly one modification. Done in twenty minutes. That's what it should have looked like from the start.

The thing nobody wants to hear

Infrastructure as code only works if the code is the source of truth. The moment you let manual changes accumulate without reconciling them, your Terraform isn't managing your infrastructure — it's a fiction that happens to live in a Git repository. And the longer the drift goes undetected, the scarier it becomes to run apply, which means people avoid it, which means more changes happen manually, which means more drift. It's a feedback loop that only gets worse.

I keep thinking about that open port on 8080. Three months, exposed to the entire internet, on a load balancer handling credit card transactions. Nobody malicious found it, as far as we can tell. But "we got lucky" is not a security posture. How many other teams are sitting on similar surprises, buried in infrastructure that nobody's compared to the code recently?