Hosting Autopsy / The certificate that expired on a Friday evening

The certificate that expired on a Friday evening

HOSTING AUTOPSY

6 min read · 1,313 words

This is a composite case written by the editors. It is built from patterns that come up often in support work and is not the account of a particular named person or company.

An online shop selling hand-made furniture ran for three years on a certificate bought once and renewed by hand each time. The reminder emails went to the address of the person who set it up, who had left the business the previous spring. Nobody had looked at that mailbox since, and nobody had thought to ask whether anything important still arrived there.

On a Friday at about six in the evening, the certificate passed its expiry time. The next visitor to the site saw a full-page browser warning telling them the connection was not private, with a button that made proceeding feel dangerous. Most people did what you or I would do and left. Orders over that weekend fell to almost nothing. The owner assumed it was a slow weekend until a customer phoned on Monday morning to ask whether the shop had been hacked.

The fix itself took twenty minutes: generate a request, validate the domain, install the new certificate. Finding out was the hard part. Everything else about the site was fine, which is why the weekend looked ordinary in the analytics, just quieter. This is a composite story, assembled from several support cases, but the shape of it is very common.

The weekend, hour by hour

The shop was run by two people: the owner, who made the furniture and answered the phone, and a part-time helper who packed orders. Neither was technical. The website had been built by a freelancer, who bought the certificate, installed it by hand and then moved on to other work. The renewal was a yearly chore that the freelancer did, then a friend of the owner did, and the last time round the owner's brother-in-law did it, using the freelancer's login to the host.

WhenWhat happenedWhat anyone could see
Thursday, 30 days beforeAutomatic reminder sent to the old addressNothing; mailbox unread
Friday, about 18:00Certificate passes its "not after" timeBrowsers begin showing a full-page warning
Friday eveningNormally the busiest browsing period of the weekVisitors leave at the warning page
Saturday and SundayOrders drop to almost nothingAnalytics show fewer visits, so it looks like a quiet weekend
Monday morningA customer phones to ask if the shop was hackedThe owner opens the site and sees the warning for the first time
Monday, about 10:30New certificate installedPadlock back, orders resume

The owner had not opened the site from a normal browser since Thursday. The helper packed boxes from a printed list and the order emails, which had stopped, were not missed because there was no baseline in anyone's head. Two days of silence was within what a small shop might expect.

Certificate validExpired, unnoticedFriday 18:00Monday morning30 days out: reminder sent to unread mailboxabout 64 hours of warnings
The reminder went out a month early, to nobody, and the outage lasted from Friday evening until Monday morning.

What visitors saw, and why they left

Modern browsers do not show a small, apologetic note about an expired certificate. They replace the page with a full-window warning, usually in red or amber, with wording such as "Your connection is not private" and an error code such as NET::ERR_CERT_DATE_INVALID. The way forward is hidden behind an "Advanced" link, and the language on it discourages anyone who is not sure what they are doing.

For a shop, this matters more than for a brochure site. A person shopping for a dining table is about to type an address and perhaps card details into a page. A warning that says the connection is not private is a fair reason to stop. Search engines also tend to drop pages that cannot be fetched securely, and some mail clients and payment pages refuse to load embedded content from a site with a broken certificate, so the damage is wider than the front page alone.

Some visitors did click through. Those with a recent browser had to find the Advanced link and accept a risk, and there is no reason to think many of them then entered card numbers. Repeat customers who had bookmarked the site mostly assumed it was down and tried again later, which is more or less how the outage stayed invisible.

Why nobody noticed

Three things combined. First, the reminder emails: certificate authorities send notices to the contact address on the order, and that address belonged to a person who had left. The mailbox still existed, so nothing bounced, which would at least have been a signal. It just collected mail.

Second, analytics. A visitor who sees the browser warning and leaves never loads the page, so the analytics script never runs. The graph shows fewer visits, not a block of errors. A drop from forty visits a day to a handful on a weekend looks like a quiet weekend, and for a furniture shop that is not unusual.

Third, nobody was checking from outside. The owner tested the shop from the workshop's own laptop, where the browser had a saved exception for the site from an earlier certificate problem, so it opened without complaint. That is a surprisingly common reason for an owner to be the last to know.

The twenty-minute fix

Once the problem was known, the repair was routine. The helper's brother-in-law was called, found the old login, and did the following:

  1. Generated a new private key and certificate signing request on the server, with the shop's domain and the www name as the names to cover.
  2. Submitted the request to the certificate authority and chose email validation, then discovered the validation email went to the same abandoned mailbox. The owner had to reach the old address through the hosting control panel and read the link from there.
  3. Completed validation, downloaded the issued certificate and the intermediate chain, and installed both in the control panel.
  4. Reloaded the web server and checked the site from a phone on mobile data.

The step that takes the time is not technical. It is finding out which mailbox, which login and which account. If the certificate had been issued through an ACME client the whole routine would have run by itself some thirty days before expiry.

# Generate a key and request (example names)
openssl req -new -newkey rsa:2048 -nodes \
  -keyout shop.example.com.key -out shop.example.com.csr \
  -subj "/CN=shop.example.com"

The aftermath

The owner moved the certificate to automatic issuance through the hosting panel, which handles renewal without any human action, and put a shared role address on the account as the contact. The former employee's mailbox was closed after anything useful had been forwarded out of it. A free external monitor now checks the certificate every day and sends alerts at 30, 14 and 7 days, to two people.

The lost weekend cost a few orders, plus the harder-to-count cost of customers who now associate the shop with a warning page. Several regular customers wrote to say they had thought the shop had closed.

Commands worth running

You can read the expiry date of any site's certificate from a terminal in a few seconds:

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates -subject

The output includes notAfter= followed by the date. If that is within a month and you have not set up automatic renewal, act now. The SSL expiry tool does the same from a browser, and the troubleshooting guide lists the other common certificate errors.

Manual renewalReminder emailOne person readsRenews by handfails if the person leavesAutomatic renewalTimer on serverACME checkNew certificateno human step
The manual path has one person in the middle; the automatic path has none.

What would have caught it

NextThe migration that forgot the cron jobs

More from Hosting Autopsy

Autopsy

The autoscaler that scaled to meet the bots

A startup used auto-scaling so the site would handle spikes. During one weekend a scraper began requesting...

Autopsy

The redirect loop nobody could see

A developer put a site behind a CDN to speed it up and switched on the CDN's setting for "flexible"...

Autopsy

The plugin update that took down the store

A shop running WooCommerce had automatic updates switched on, which is usually a good default. One afternoon...