The DNS Migration We Should Have Tested

dns infrastructure consulting reliability devops

A "30-minute" DNS provider switch turned into a 14-hour partial outage. Stale TTLs, an undocumented CNAME chain, and monitoring that only checked from the US — all the ingredients for a problem nobody could see.

The client's CTO called it a "30-minute task." They wanted to move their DNS from an old provider to Cloudflare. A handful of records, nothing exotic. The SRE copied the zone file, validated the entries, flipped the nameservers, and went to lunch.

Four hours later, their European customers couldn't reach the API.

I was six weeks into an engagement helping this company — a B2B SaaS platform with about 2,000 business customers across Europe and North America — improve their deployment pipeline. DNS migration wasn't part of my scope. But when half your customer base can't log in, scope becomes flexible.

The 24-hour TTL nobody checked

The first thing I looked at was the TTL on the old records. It was set to 86400 seconds — 24 hours. That's the default their old DNS provider had set when the records were first created three years ago, and nobody had ever changed it.

This meant that every resolver that had cached the old records would keep using them for up to a full day after the nameserver switch. The SRE had assumed propagation would take minutes. For some resolvers it did. For others — especially ISP resolvers in parts of Europe that honor TTLs more strictly — the old records stuck.

The fix for this is well-known: lower the TTL to 300 seconds a day or two before migration, let the short TTL propagate, then switch. It's step one in every DNS migration checklist. The SRE knew this in theory. They just forgot to do it.

# What should have happened 48 hours before the switchover
dig api.example.com
 
# Before: api.example.com.  86400  IN  A  203.0.113.10
# After:  api.example.com.  300    IN  A  203.0.113.10

The CNAME chain nobody documented

Lowering the TTL earlier would have shortened the blast radius. It wouldn't have prevented the second problem.

The primary API endpoint, api.example.com, was a CNAME pointing to api-gateway.example.com, which pointed to a load balancer hostname at their cloud provider. Standard stuff. But there was a third hop. Two years earlier, someone had added a CNAME from api-gateway.example.com to api-primary.internal.example.com as part of a blue-green deployment experiment that was later abandoned. The experiment ended. The DNS record didn't.

The zone file export from the old provider included all three records. But when the SRE imported them into Cloudflare, internal.example.com wasn't part of the zone being migrated — it lived in a separate hosted zone on AWS Route 53. The old provider's resolvers happened to have the Route 53 records cached, so the chain resolved fine. On Cloudflare, it broke. The chain hit api-primary.internal.example.com and got a SERVFAIL.

Here's what made this worse than it needed to be: their uptime monitoring checked https://api.example.com/health from three locations, all in the US. The American nameservers picked up the new Cloudflare records quickly. From the monitoring dashboard, everything looked green.

The first sign of trouble came from a customer support ticket, not an alert. A German customer reported they couldn't access the dashboard. Then another. Then twelve more in thirty minutes.

The team's instinct was to check application logs. Nothing. No errors, no latency spikes, no dropped connections. Because the requests never reached the application — they were failing at DNS resolution before any HTTP traffic was generated. The monitoring was resolving through a completely different path than the affected customers.

What a dig would have told us

Once someone thought to check DNS from European vantage points, the problem was obvious.

# From US — working
dig +short api.example.com @8.8.8.8
# 203.0.113.42 (Cloudflare IP, correct)
 
# From DE — broken
dig +short api.example.com @dns.telekom.de
# 203.0.113.10 (old provider IP, stale cache)
 
# After stale cache expired, the CNAME chain broke
dig +trace api.example.com
# api.example.com.           300 IN CNAME api-gateway.example.com.
# api-gateway.example.com.   300 IN CNAME api-primary.internal.example.com.
# api-primary.internal.example.com.  SERVFAIL

The fix took five minutes once they identified it: add the missing internal.example.com record directly to Cloudflare as an A record, bypassing the broken CNAME chain. Then wait for the stale TTLs to expire across European resolvers.

Total time from nameserver switch to full resolution: about 14 hours. Roughly 4 hours of real customer impact before the CNAME fix, then a long tail of stale-cache stragglers trickling in.

The .de outage put it in perspective

This engagement wrapped up in March. Two months later, on May 5th, the .de top-level domain had a DNSSEC outage that took down millions of German domains for three hours. Amazon.de, Deutsche Bahn, Spiegel — all unreachable. A routine DNSSEC key rollover produced invalid signatures, and validating resolvers started returning SERVFAIL for the entire TLD.

Same fundamental lesson: DNS is invisible until it isn't, and most teams treat it as someone else's problem.

My client's CTO messaged me that day: "At least we only broke it for our own customers."

Note

Before any DNS change, run dig +trace from multiple geographic regions and follow every CNAME to its final resolution. If your monitoring only checks from one country, you're only monitoring one country.

What we changed

After the incident, the team added four things to their infrastructure change process. Lower TTLs to 300 seconds at least 48 hours before any DNS modification. Document the full resolution chain — run dig +trace and follow every CNAME hop. Add health checks from at least three geographic regions that match where your customers actually are. And treat DNS changes like application deploys: staged, verified, with a rollback plan that accounts for propagation delay.

None of this is novel. It's all in the playbook. But this team had a rigorous deployment pipeline for application code — feature flags, canary releases, automated rollback — and still did DNS changes by hand, validated by feel, and assumed they'd work because they usually do.