Hosting Autopsy / The CAA record that blocked its own certificate

The CAA record that blocked its own certificate

HOSTING AUTOPSY

7 min read · 1,491 words

This is a composite case written by the editors. It is built from patterns that come up often in support work and is not the account of a particular named person or company.

The IT administrator at a small accountancy firm read a security checklist and took it seriously. One item said to publish a CAA record, which tells the world which certificate authorities are allowed to issue certificates for your domain. He added one, listing the single authority the firm had bought certificates from for years. It was a good idea and it worked as intended for a year.

Then the hosting company switched its automatic certificates to a different authority. The next renewal attempt was refused, because the DNS record said only the old authority could issue. Nobody saw an error until the certificate neared expiry and the warning emails began.

Adding the new authority to the record cleared it up, with two days to spare. Two days is not a comfortable margin when the alternative is a browser warning page in front of every client who logs in to the portal. This is a composite, but it comes from a pattern that has caught out far more careful people than the one in the story.

What a CAA record does

CAA stands for Certification Authority Authorisation. It is a DNS record type that lists which authorities may issue certificates for a domain. Before issuing, a public certificate authority is required to look up the CAA records for the name being requested and refuse if its own identifier is not on the list. If there are no CAA records, any authority may issue. If there are some, only those named may.

The firm's record looked like this:

example.com.  3600  IN  CAA  0 issue "ca-one.example"

The three parts after CAA are a flag (nearly always 0), a tag (issue, issuewild or iodef), and a value. The issue tag names an authority allowed to issue ordinary certificates. issuewild does the same for wildcard certificates. iodef gives an address or URL to report violations to.

It is a restriction, not a configuration. That distinction matters: the record does nothing when everything is as expected, and silently turns every other authority away.

Renewal request to an authority Authority reads CAA in DNS Listed: certificate issued Not listed: request refused
The authority makes the decision at issuance time, using whatever the DNS says at that moment.

Why it worked for a year

For twelve months the hosting platform requested renewals from the same authority the firm had always used, so every check passed. The record fell out of sight and out of mind, as security measures do when they work. The administrator had written the change in his own notes. The notes were on his own laptop.

The trouble began when the hosting company, for its own commercial and technical reasons, changed the authority behind its automatic certificates. Hosts do this with little fanfare; to most customers it is an improvement they never see. The firm's portal certificate was due for renewal about five weeks later. Automatic renewals normally start well before expiry, often at around a third of the lifetime remaining, so the failures began long before anyone should have worried.

Weeks of failed renewals

Each renewal attempt failed in the same way, and the hosting platform logged it. The message from the authority said, in effect, that a CAA record for the domain does not allow it to issue. That message went to a log in the hosting control panel. It did not go to an email address, it did not appear on the dashboard, and the site continued to work perfectly because the old certificate was still valid.

Issued automatic renewal attempts each refused by CAA Warning emails Expiry certificate valid, site fine fixed here Illustrative: about 90 days end to end, failures start around day 60, fix lands 2 days before expiry.
For weeks the only evidence of the problem was a log nobody read, until the expiry warnings began.

The warning emails

The first people to know were not on the IT team. The certificate authority's notification emails, which are sent as expiry nears, went to the contact address recorded when the certificate was first ordered. That was a former employee's mailbox, now forwarded to a shared account. A bookkeeper saw "Your certificate expires in 14 days" and assumed it was marketing.

A week later a second notice arrived with the word "urgent" in it. Someone forwarded it to the administrator, who spent a morning checking the host's control panel, found the certificate listed as "renewal failed", and opened a support ticket with the hosting company.

Diagnosis

The support engineer's first question was whether the domain had a CAA record. It took one command to find out.

$ dig +short CAA example.com
0 issue "ca-one.example"

The hosting company's new authority was ca-two.example. The administrator had forgotten adding the record. When he looked at the zone he recognised it at once, and the whole story fell into place.

One extra point is worth knowing. CAA lookups climb the tree: the authority checks the exact name first and, finding none, looks at the parent, and so on up to the registered domain. A record at example.com therefore applies to portal.example.com unless the subdomain has its own set. That is how a record added years ago on the apex came to govern a certificate issued for a subdomain.

The fix

The fix was one added record. A domain can have several CAA records, and any listed authority may issue:

example.com.  3600  IN  CAA  0 issue "ca-one.example"
example.com.  3600  IN  CAA  0 issue "ca-two.example"
example.com.  3600  IN  CAA  0 iodef "mailto:[email protected]"

He also added an iodef line pointing at a shared mailbox, so that refused requests might be reported (not every authority sends these, but some do). Then he asked the hosting platform to retry the renewal, which succeeded within minutes, and replaced the stale contact address for expiry notices with a role mailbox read by three people.

Dropping the CAA record altogether would also have worked. The firm chose to keep it, which is reasonable, so long as someone owns the list of permitted authorities. The SSL expiry checker shows the date you are working against, and the same tool is a good thing to keep bookmarked.

Wrong turns on the way

Before anyone looked at DNS, the firm tried the usual remedies. They suspected the validation step, so they checked that the well-known validation path on the web server was reachable and that no redirect was getting in the way. It was fine. They suspected the hosting plan, so they asked whether the certificate feature had been switched off. It hadn't. They even considered buying a certificate by hand, which would have worked but meant a yearly manual renewal nobody would remember either.

The giveaway had been in the log all along. The error text names the cause quite directly, but it sits several lines below the part of the message that says "validation failed", and people read the headline and stop. When a renewal fails, read the whole message from the authority, not the summary line the control panel shows.

Common mistakes with CAA

MistakeEffect
Listing only the authority you use todayA host-side switch breaks renewal
Forgetting issuewild when you use wildcardsWildcard requests fall back to the issue list, which may be wrong for them
Setting a record on a subdomain that overrides the apexThe subdomain follows its own list, not the parent's
Typing the authority's identifier wronglyNo one can issue, including the one you meant
Leaving CAA behind after leaving a providerNew provider refused, as in this story

The identifier is a domain name chosen by the authority and published in its documentation, so copy it exactly and do not guess it from the company name.

Checking it yourself

dig +short CAA example.com
dig +short CAA www.example.com
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null | openssl x509 -noout -issuer -enddate

The first two show whether any restriction applies. The third prints who issued the current certificate and when it ends. If the issuer is not on your CAA list, your next renewal is likely to fail unless the issuer stays the same. Ask the host which authority they use for automatic certificates now, and whether it has changed or may change.

Things people ask

Do I need a CAA record?

No. It is optional hardening. It reduces the chance of a mistaken or malicious issuance by an authority you never use, at the price of one more thing to maintain.

Does CAA protect against fake certificates?

Only partly. It binds well-behaved authorities. It is not a defence against an attacker who can change your DNS.

Why did the host not warn us?

Many hosts assume customers have no restrictions. The change of authority was announced, but to a list the firm did not read.

What would have caught it

PreviousThe sale that started at the wrong hourNextThe redesign that lost its old addresses

More from Hosting Autopsy

Autopsy

The autoscaler that scaled to meet the bots

A startup used auto-scaling so the site would handle spikes. During one weekend a scraper began requesting...

Autopsy

The launch-day database that ran out of connections

A local theatre put tickets for its autumn season on sale at noon and announced it on social media a day...

Autopsy

The plugin update that took down the store

A shop running WooCommerce had automatic updates switched on, which is usually a good default. One afternoon...