Unit 5: How DNS answers · Lesson 6 of 6 · 20 min
Case file: the DNS record that went empty
Tell an empty answer from a missing name, read a synthetic check log and size the outage it caused.
case file
AWS US-EAST-1 DynamoDB DNS Outage
Amazon Web Services · 2025
From late October 19 to the afternoon of October 20, 2025 (Pacific time), a latent race condition in DynamoDB's automated DNS management left the regional DynamoDB endpoint in US-EAST-1 with an empty DNS record. The failure cascaded into EC2 instance launches, Network Load Balancers, Lambda, ECS/EKS, IAM/STS sign-in and many other services, taking down major websites for hours. AWS disabled the DNS automation worldwide pending fixes and added safeguards.
What went wrong
Two automation components applied DNS plans out of order, and a cleanup step deleted the active plan, removing all IP addresses for the endpoint; dependent services then entered congestive recovery states.
What would have stopped it
- Add guards in automation so stale changes cannot overwrite newer state, and block plans that would empty a critical DNS record.
- Design applications for regional failure (multi-region or graceful degradation) for critical workloads.
- Rate-limit automated capacity removal (velocity controls) during health-check-driven failover.
- Regularly exercise recovery workflows for backlog/congestion scenarios.
What defenders could have seen
- Synthetic DNS resolution checks failing for a critical service endpoint.
- Spike in API error rates for a dependency across many services simultaneously.
mission
- You are
- The engineer on call for corp.example's internal services.
- Situation
- Orders started failing at 09:43. The resolver says the name
api.corp.exampleexists. A synthetic check has been running every minute from~/dnscheck/. This lab is modelled on the AWS DynamoDB DNS outage of October 2025. - Goal
- Confirm what the resolver returns, measure the outage window and recommend checks that catch it in a minute.
- Hand in
- The failure type, the window and the check you would add.
A signpost with a blank arm
Run the health check on critical names
check.sh asks the resolver for each name and reports OK, an error code, or an empty answer.
cat critical.txt www.corp.example portal.corp.example api.corp.example mail01.corp.example ./check.sh $(cat critical.txt) www.corp.example OK 1 address(es) portal.corp.example OK 1 address(es) api.corp.example FAIL empty answer mail01.corp.example OK 1 address(es)
you should see api.corp.example returns no addresses at all.
Read the raw answer
NOERROR with zero answers is different from NXDOMAIN. The name exists, but its address list is empty.
dig @127.0.0.1 api.corp.example A +noall +comments | grep -oE 'status: [A-Z]+|ANSWER: [0-9]+' status: NOERROR ANSWER: 0 dig @127.0.0.1 nosuch.corp.example A +noall +comments | grep -oE 'status: [A-Z]+' status: NXDOMAIN dig @127.0.0.1 api.corp.example TXT +short "owner=platform team"
you should see Status NOERROR, ANSWER: 0. The name still has a TXT record, so the zone knows it, but the A records are gone.
defender tip Many monitoring tools only alert on NXDOMAIN or timeouts. An empty answer looks healthy to them.
Measure the window
The synthetic log has one line per minute.
grep -c FAIL synthetic.log 47 grep FAIL synthetic.log | sed -n '1p;$p' 09:41 api.corp.example FAIL empty answer 10:27 api.corp.example FAIL empty answer
you should see 47 minutes of empty answers, from 09:41 to 10:27.
Follow the damage
One empty record stops every service that depends on it, and recovery leaves a backlog.
cat app-errors.log
09:43 orders-api ERROR cannot resolve api.corp.example, retrying
09:44 checkout ERROR upstream orders-api timeout
09:47 orders-api ERROR cannot resolve api.corp.example, retrying
09:52 reports ERROR upstream orders-api timeout
10:28 orders-api INFO resolved api.corp.example, 2 addresses
10:31 checkout INFO backlog 18233 requests, draining
you should see Orders failed, checkout timed out, and a backlog of 18233 requests drained after recovery.
What a defender takes away
- 1Check that critical names resolve to addresses, not just that DNS answers.
- 2NOERROR with no answer is a failure for the caller.
- 3Guard automation that edits DNS so it can never empty a critical record.
Questions
0 of 3 solved
Answer each one from what you find on your lab machine.