How automatic certificate renewal works

Automatic renewal sounds like magic, but the mechanism is straightforward: a program proves to a certificate authority that it controls a domain, and the authority issues a certificate in return. The protocol is called ACME (Automatic Certificate Management Environment), and Let's Encrypt made it famous. Several other authorities now speak it too, which is why the same client software can often work with more than one of them.

The reason this matters is that certificates used to be a yearly chore. Someone generated a request, pasted it into a web form, waited for an email, installed the files and set a calendar reminder. Reminders were missed, people left the company, and sites went down with a browser warning that looked like a hack. ACME replaced that ritual with a conversation between two programs that takes a few seconds and can run on a schedule.

Understanding the conversation pays off the day it breaks. Renewal failures are rarely mysterious once you know which step failed, and nearly all of them are caused by something someone changed months earlier for an unrelated reason.

What actually happens during issuance

There are four parties and a short script. Your server runs an ACME client. The client holds an account key, which identifies you to the authority without a password. The authority runs the ACME server. And the domain's DNS and web server are what the authority checks.

1. The client tells the authority which names it wants on the certificate, for example example.com and www.example.com.
2. The authority replies with a challenge for each name: a random token the client must publish somewhere only the real owner could.
3. The client publishes the token and says it is ready.
4. The authority checks from the outside, from several network locations, and confirms it can see the token.
5. The client sends a certificate signing request containing a fresh key pair's public half, and the authority signs and returns the certificate.
6. The client saves the files and reloads the web server so it starts using them.

Proving control: HTTP, DNS and the others

There are a few challenge types, and which one you use decides what can go wrong.

In the HTTP method, the client places a small file under /.well-known/acme-challenge/ on your site, and the authority fetches it over port 80. If the file is there, you control the site. It needs no DNS access and is what most shared hosting uses behind the scenes. It cannot issue wildcard certificates, and the server must be reachable from the public internet on port 80.

In the DNS method, the client adds a TXT record named _acme-challenge to your domain:

_acme-challenge.example.com.  60  IN  TXT  "gfj9Xq...Rg85nM"

This is the only way to get a wildcard certificate such as *.example.com, and it works for servers that are not reachable from outside at all, such as an internal admin tool with a public name. The price is that the client needs credentials to change your DNS, usually an API token, and a token that can edit DNS is worth protecting. Scoped tokens that can only touch one zone are better than an account-wide key.

A third type, TLS-ALPN, validates over port 443 and suits setups where port 80 is closed on purpose. It is less common and mostly built into particular web servers and proxies.

One name can have its CNAME for _acme-challenge pointed at a zone you control, so the DNS client only needs access to that small zone. That is a neat way to avoid giving a tool the keys to your whole domain, though it adds a record that must not be deleted by a later tidy-up.

The rhythm of renewal

Let's Encrypt certificates last 90 days, and clients usually try to renew when about a third of that remains, so at around 60 days old. That deliberate margin means a failed attempt is retried many times before anything expires. A client that runs twice a day, which is common, gets dozens of chances in a month.

Most hosts run the client for you. If you have your own server, a scheduled job does the work: certbot installs a systemd timer or a cron entry, and other clients such as acme.sh or Caddy's built-in logic have their own. On a VPS, confirm it exists with systemctl list-timers | grep -i certbot or crontab -l. The cron helper is useful if you need to write the schedule by hand.

There is a practical wrinkle: authorities limit how often you may request certificates for the same names, to keep abuse down. A script that retries every minute in a loop can lock you out for days. Test with the authority's staging environment, which has generous limits and issues untrusted certificates, and move to production once the process works.

Why renewals fail

A redirect or firewall rule that blocks the challenge path. A new "force HTTPS" rule that sends the challenge request to a login page. A CAA record that does not list the authority. Port 80 closed because somebody read a hardening checklist. DNS credentials that changed. A web application firewall that treats the authority's validation servers as a scanner and blocks them. A server that obtained the new certificate but was never reloaded to use it. Most of these are changes someone made months earlier for another reason, which is why the failure feels sudden when it surfaces.

The CAA case deserves an example, because it is easy to miss. A record like this tells authorities which of them may issue for the domain:

example.com.  3600  IN  CAA  0 issue "letsencrypt.org"

If you later switch to a different authority and forget to add it, or add a CAA record for the first time and list only one, issuance from the others is refused. Check with dig CAA example.com +short. The DNS cheatsheet covers record types if CAA is new to you.

Diagnosing a failed renewal, step by step

1. Read the client's own log. With certbot it is /var/log/letsencrypt/letsencrypt.log. The error normally names the challenge and the reason, such as a 404, a connection timeout or a DNS lookup failure.
2. Reproduce the HTTP check yourself from outside your network: curl -I http://example.com/.well-known/acme-challenge/test. A 404 is fine here (there is no such token), but a 301 to a different host, a 403 or a timeout is the problem.
3. Check that port 80 answers from the internet, not just from the server itself. A cloud firewall or security group is easy to forget.
4. For DNS challenges, query the record directly with dig TXT _acme-challenge.example.com against the authoritative nameserver, not only your local resolver.
5. Run a dry run, which exercises everything except saving the result: certbot renew --dry-run.
6. After a successful run, confirm the new expiry on the live site, as described below.

On shared hosting you may have no access to the client at all, only a panel page that shows certificate status and a "reissue" button. In that case the useful move is to check the things you do control, namely redirects in your .htaccess, DNS records and CAA, and then ask support with the exact error text.

What to monitor

Do not rely on the renewal job alone. Set up an external check that reads the certificate your site actually serves and warns you at two weeks before expiry. The point of automation is to stop thinking about this, and the monitor is what lets you stop with confidence. Fourteen days leaves time to fix a DNS token or an unexpected firewall rule without a scramble.

A good monitor checks every name you serve, including subdomains and the www variant, because each may have a separate certificate or a separate failure. It checks from outside your network. And it alerts someone who will actually read the alert; a mailbox nobody opens is not monitoring. The SSL expiry tool shows how many days a site has left, and works for a one-off check.

Verifying it

See what the server presents, not what the disk holds:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -dates -issuer

The output shows notBefore and notAfter. A certificate renewed at about day 60 shows a notBefore roughly a month old on a healthy site. If it is nearly 90 days old and the expiry is close, renewal has been failing for a while, whatever the logs claim.