Unit 9: Scheduled tasks and persistence · Lesson 6 of 6 · 20 min
Case file: a scheduled job that prevents an outage
An expired certificate cut mobile data for millions. Write the cron job that would have warned in time.
mission
- You are
- You are adding a defensive job to web01.
- Situation
- The shop's certificates are in /srv/certs. One belongs to a vendor appliance and was installed years ago.
- Goal
- Write a script that flags expired and soon-to-expire certificates and schedule it daily.
- Hand in
- certcheck.sh and its crontab line.
case file
Ericsson Expired Certificate Mobile Outage
Ericsson (O2 UK, SoftBank and others) · 2018
On December 6, 2018, an expired certificate in Ericsson SGSN-MME core network software caused mobile data outages for customers of several operators in multiple countries, including O2 in the UK (and its resellers) and SoftBank in Japan. Millions of O2 customers lost data service, affecting services such as TfL's timetable information. Ericsson apologised and decommissioned the faulty software versions.
What went wrong
A certificate embedded in installed vendor software expired without being tracked or renewed, and the software failed hard when it did.
What would have stopped it
- Inventory all certificates, including those embedded in vendor appliances and software, with owners and expiry dates.
- Automate renewal and alert well ahead of expiry (for example 30/14/7 days).
- Ask vendors to document embedded certificates and design software to warn, not fail, as expiry approaches.
What defenders could have seen
- Certificate-expiry monitoring alerts on core network elements.
- Sudden, simultaneous failure across nodes running the same software version at a specific time.
Read each expiry date
for c in /srv/certs/*.pem; do echo "$(basename $c) $(openssl x509 -in $c -noout -enddate)"; done
api.shop.example.pem notAfter=Oct 1 00:00:00 2026 GMT
gateway-internal.pem notAfter=Mar 31 00:00:00 2027 GMT
status.shop.example.pem notAfter=Jun 11 00:00:00 2028 GMT
www.shop.example.pem notAfter=May 3 00:00:00 2027 GMT
you should see One date has already passed.
Write the check
-checkend N fails if the certificate expires within N seconds.
cat > ~/certcheck.sh <<'EOF' #!/bin/bash for c in /srv/certs/*.pem; do n=$(basename "$c") if ! openssl x509 -in "$c" -noout -checkend 0 >/dev/null; then echo "EXPIRED $n" elif ! openssl x509 -in "$c" -noout -checkend $((180*86400)) >/dev/null; then echo "RENEW $n" else echo "OK $n"; fi done EOF chmod 700 ~/certcheck.sh ~/certcheck.sh EXPIRED api.shop.example.pem RENEW gateway-internal.pem OK status.shop.example.pem OK www.shop.example.pem
you should see One expired, one inside the 180-day window, two fine. Results move as the calendar does.
Schedule it
Cron mails any output to the account, so printing only problems means mail arrives only when there is something to do.
(crontab -l 2>/dev/null; echo '15 6 * * * $HOME/certcheck.sh | grep -v ^OK') | crontab - crontab -l 15 6 * * * $HOME/certcheck.sh | grep -v ^OK
you should see A daily job at 06:15.
Who owns the vendor certificate?
openssl x509 -in /srv/certs/gateway-internal.pem -noout -subject -startdate
subject=O = Shop Example Ltd, CN = gw01.internal.shop.example
notBefore=Nov 20 00:00:00 2016 GMT
you should see Issued in 2016 for the gateway. Embedded certificates are the ones nobody remembers.
defender tip Keep a certificate inventory with an owner per certificate, and alert at 60 and 30 days.
What a defender takes away
- 1Every certificate needs an owner and an alert.
- 2Embedded vendor certificates expire too.
- 3Cron jobs can watch for problems as well as cause them.
Questions
0 of 2 solved
Answer each one from what you find on your lab machine.