Let's Encrypt renewals that never surprise you

Certificates from Let's Encrypt are valid for 90 days and certbot renews them at 30 days left. In theory, you never think about it. In practice, every few years renewal silently breaks and you find out from a browser warning.

The three times it happened to me:

1. A firewall change closed port 80, so the HTTP-01 challenge could not reach the webroot.
2. Renewal worked, but nothing reloaded the web server, so it kept serving the old certificate from memory.
3. A second service used the same certificate files and was never restarted.

All three are fixable with a little discipline.

Use the webroot, and test it

certbot certonly --webroot -w /var/www/acme -d example.com -d www.example.com
certbot renew --dry-run

--dry-run talks to the staging environment and goes through the full challenge. Run it after any change to the firewall or the web server config. It takes ten seconds.

Deploy hooks

Anything in /etc/letsencrypt/renewal-hooks/deploy/ runs after a successful renewal. Mine is three lines:

#!/bin/sh
systemctl reload nginx
systemctl restart my-other-thing

Make it executable. certbot ignores files without the x bit, silently.

Monitor from the outside

The real fix for "I find out from a browser warning" is a check that looks at what clients actually see, not at files on disk. I wrote certpeek for that: it connects to each host, reads the certificate that is actually served and exits non-zero if it expires in less than N days.

$ certpeek --warn 21 example.com seyshee.dev
example.com     OK    67 days  (Let's Encrypt R11)
seyshee.dev     OK    74 days  (Let's Encrypt R10)

It runs from a systemd timer on a different machine and sends me a message if the exit code is not zero. That covers case 2 and 3 above, which no amount of looking at /etc/letsencrypt/live/ would have caught.