The certificate that renewed and still expired
The common failure is not a failed renewal — it is a successful one that never reached the served certificate. Certbot writes a new file under /etc/letsencrypt/live/, but nginx keeps the certificate it loaded at start, so the browser sees the old expiry while the disk shows a fresh one. You find it by comparing two dates: notAfter of the file on disk against notAfter of the certificate the server actually presents.
A certificate does not usually break because renewal failed. Renewal fails loudly: cron mails you, the dry run exits non-zero, the log line says Failed authorization procedure.
The failure that reaches users looks like this instead: renewal succeeded, the file is new, and the server still presents the old certificate.
What the browser actually sees
From outside, on a public host, 2026-11-18:
$ echo | openssl s_client -connect github.com:443 -servername github.com 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates
subject=CN=github.com
issuer=C=GB, O=Sectigo Limited, CN=Sectigo Public Server Authentication CA DV E36
notBefore=Oct 30 00:00:00 2026 GMT
notAfter=Jan 27 23:59:59 2027 GMT
Two dates, one number that matters. From the check date to notAfter is 70 days. That is the only date a visitor’s browser will ever look at.
The same command works on a file, and then it is answering a different question:
$ openssl x509 -in /etc/letsencrypt/live/example.com/fullchain.pem -noout -dates
notAfter=...
Two commands, two dates, and no reason for them to match. The first reads memory (what nginx loaded), the second reads disk (what certbot wrote). Every stale-certificate incident lives in the gap between them.
Why the gap opens
- Certbot renews, writes
live/<domain>/fullchain.pem, exits 0. - nginx reads certificates at start or at reload — not per request.
- Nothing reloaded nginx, so it keeps presenting the certificate from before.
- Sixty days later the old one expires, and the disk still says “valid for another 30”.
The other three silent shapes, for completeness:
| Shape | Symptom | Why it stays silent |
|---|---|---|
| Webroot moved / vhost changed | http-01 challenge gets 404 | Renewal task’s log goes to a file nobody reads |
| Port 80 blocked in the security group | challenge timeout | Same; and the failure is intermittent if only one of several A records fails |
.well-known caught by a redirect rule |
challenge follows a 301 to HTTPS and fails | Works for months, breaks the day a redirect is added |
| Timer disabled after a migration | nothing runs at all | The certificate stays valid for 60 more days, so nothing looks wrong |
All four share the same property: the failure is invisible until the certificate expires, which is up to 90 days after the cause.
The two checks that close it
Outside — what is served:
served=$(echo | openssl s_client -connect "$HOST:443" -servername "$HOST" 2>/dev/null \
| openssl x509 -noout -enddate | cut -d= -f2)
echo "$served"
Inside — what is on disk:
openssl x509 -in "/etc/letsencrypt/live/$HOST/fullchain.pem" -noout -enddate | cut -d= -f2
Assert three things:
servedis in the future (with a margin — 14 days is comfortable).served == disk(modulo re-issue timing). A difference means the reload never happened.certbot renew --dry-runexits 0 — that proves the challenge path works today, not the day it matters.
The dry run is the only one of the three that tests the part which breaks silently: DNS, port 80, and the webroot. It does not consume rate limits, and it takes seconds.
How you know it worked
- The served date moved. After adding a deploy hook that reloads nginx, force one renewal and re-read the served certificate. If the date did not move, the hook did not run — check its path and its exit code, not the certbot log.
certbot renew --dry-run; echo $?prints 0. Run it from the same timer context (same user, same environment). A dry run that passes in your shell and fails under cron is an environment difference:PATH, working directory, or a missing--deploy-hooktarget.- The timer actually fires.
systemctl list-timers | grep certbot(orcrontab -l). A disabled timer is not an error anywhere — it is an absence. - Your monitor reads the served date, not the disk date. Monitoring the file is how you get a green dashboard next to a red padlock.
How to undo it
- Adding a reload hook:
--deploy-hook "systemctl reload nginx"— if the reload fails, certbot reports a hook failure but the certificate is already renewed. Re-running renew will not retry the hook; run the reload manually, then verify the served date. - A bad vhost change: keep the previous vhost file and the previous
live/symlink target.certbotstores every issuance underarchive/; pointing the symlink back is reversible, deletingarchive/is not. - Rate limits: five duplicate certificates per week is the ceiling. If you are debugging by forcing renewals, use
--dry-run(no issuance) and--staging(separate limits) — otherwise you can lock yourself out of issuing the real certificate for a week exactly when you need it. - Never “fix” it by disabling the monitor. The monitor is the only component still telling the truth once the disk and the served certificate disagree.
Related
Questions people actually ask
How do you read the expiry of the certificate being served?
From outside: echo | openssl s_client -connect host:443 -servername host 2>/dev/null | openssl x509 -noout -enddate. That is what the browser sees. On disk, the same command against /etc/letsencrypt/live/<domain>/fullchain.pem is what certbot thinks it has.
Why do those two dates differ?
Because nginx reads the certificate once, at start or reload. A renewal that writes the file but does not reload leaves the old certificate in memory and keeps serving it until the next restart — which may be months away.
What proves renewal will work before it matters?
certbot renew --dry-run and its exit code. It exercises the same challenge path without consuming rate limits. A green dry run plus a reload hook covers both halves: the new file exists and the server picks it up.
Do expiry emails save you?
Only if they reach someone. They go to the account registered with the ACME client, arrive 20 and 10 days before expiry, and say nothing about a server that is serving a stale certificate while the disk copy is fine.
Sources
- Certbot documentation: renewal and hooks
- Let’s Encrypt: challenge types
- Let’s Encrypt: expiration emails
Last checked: 2026-11-18 · The outside checks were run for real on 2026-11-18 against a public host (openssl s_client output pasted below, notAfter Jan 27 2027, 70 days out). The renewal-side commands (certbot renew --dry-run, the hook, the disk-vs-served diff) are the documented procedure from our operations record; we did not run a certbot renewal inside this session, and the article says so rather than implying it.