A local theatre put tickets for its autumn season on sale at noon and announced it on social media a day earlier. At 12:03 the site began returning "Error establishing a database connection".
The database was healthy. It allowed a fixed number of simultaneous connections, and the sudden crowd, plus the ticketing plugin holding connections open while it waited for the payment provider, used them all. New visitors were turned away until older ones finished.
The host raised the limit temporarily. Later the theatre enabled page caching for everything except the checkout and spread the on-sale across two days.
What the first twenty minutes looked like
The theatre's box office was one full-time manager and two part-time volunteers. The website was a WordPress site on a mid-sized VPS with a ticketing plugin bolted on. In a normal week it served a few thousand page views and rarely noticed.
The on-sale post went out at 11:55 on the theatre's social accounts, a day after the announcement. By noon several hundred people had the page open. At 12:00 they refreshed together. For about two minutes the site was merely slow, then the first error pages appeared. By 12:10 the volunteer on the phones was telling callers to try again later, and the callers were telling her the website was down. It was not exactly down; it was answering some visitors and refusing others, which is more confusing to everyone involved.
How a connection limit works
Every time a PHP page needs data, it opens a connection to the database server, runs its queries and closes the connection. The server only accepts so many at once. MySQL and MariaDB call the setting max_connections, and the stock value is often 151, though hosts tune it up or down to match the machine's memory. Each open connection costs a little RAM on the server, which is why the number is finite.
When the limit is reached, the database answers new arrivals with error 1040, "Too many connections". WordPress cannot tell the difference between that and a dead database, so it prints the generic message about establishing a connection. The same words appear when the password is wrong or the server is off, which is why the message sends so many people hunting in the wrong place.
On shared hosting there is usually a second, smaller limit per account, sometimes called max_user_connections, so one busy site cannot starve its neighbours. That limit is often 15 to 50. A VPS owner has the whole server's limit to play with, and also the whole server's memory to protect.
Why the ticketing plugin made it worse
A normal page lives for a fraction of a second, so a hundred connections can serve a lot of visitors. Checkout is different. The plugin opened a connection, wrote a pending order, then called the payment provider's servers and waited for a reply, still holding the connection. If the provider took three seconds instead of three hundred milliseconds, each checkout held its slot ten times longer.
The arithmetic is unforgiving. With 100 connections and checkouts that hold one for three seconds, the site can complete roughly 33 checkouts a second at best, and every ordinary page view competes for the same pool. A crowd of a few hundred hitting refresh together exceeds that in an instant. Nothing was broken; the system was asked for more than it had.
Diagnosing it while it happens
The administrator's first instinct, restarting MariaDB, cleared the error for under a minute and then it returned. That was the clue that the database was fine and demand was the problem. Logging in and running one query told the story:
mysql -e "SHOW GLOBAL STATUS LIKE 'Threads_connected'; SHOW GLOBAL VARIABLES LIKE 'max_connections'; SHOW FULL PROCESSLIST;"Threads_connected sat at the maximum. The process list showed dozens of connections in the Sleep state, each belonging to a checkout script, and a few real queries. Sleeping connections are the giveaway: they are doing nothing, just occupying seats.
Wrong turns on the day
Three theories were tried before the connection count was checked. The first was a failed update, because a plugin had updated itself on Sunday night. Rolling it back changed nothing. The second was the payment provider being down; their status page was green. The third was an attack, which cost twenty minutes of reading access logs and finding only ordinary visitors, many of them refreshing the same page over and over.
Refreshing is worth a thought. When a site slows down, people hit reload, and every reload is another request the server must answer. A site under stress generates extra load from its own impatient audience, so the crowd you planned for is not the crowd you get.
The fix, in the order it happened
The host raised max_connections from 100 to 250 to get through the afternoon. That bought time but also cost memory, and the VPS began to swap, so it was a loan rather than a cure. Raising the limit is a dial that works until the machine runs out of RAM, at which point the failure is worse than a polite error page.
The lasting changes were three. Page caching went on for the whole public site, with the cart, checkout and account pages excluded. The ticketing plugin's payment call was moved to run after the connection was released, a setting the vendor documented. And the theatre split the on-sale into two days, one for members and one for everyone else, which also made the phones calmer.
What changes between hosting types
| Setup | Who sets the limit | What you can do |
|---|---|---|
| Shared hosting | The host, per account | Cache hard, ask support for the figure, upgrade the plan |
| VPS | You | Tune max_connections against RAM, add a cache, add a database server |
| Dedicated server | You | Same as a VPS with more headroom and more to monitor |
| Managed WordPress | The host | Usually caching is built in; ask how checkout pages are treated |
On shared hosting the surprise is that your neighbours do not matter much for this particular failure, but your own per-account ceiling does. A site that is fine at 15 simultaneous connections on a Tuesday is not fine on a launch day, and the error looks identical.
The aftermath
By the afternoon about two thirds of the first-night seats had sold, and a few customers had been charged twice after pressing the pay button again when nothing happened. The box office refunded the duplicates by hand and sent a short, plain apology. The volunteer who had spent the lunch hour on the phones asked for a written note she could read out next time, which the manager wrote and pinned beside the till.
The second on-sale, three weeks later, saw peak connections of 38 on the same server. The difference was almost entirely the cache. The tools page has a page weight checker for the other half of the problem, which is how much each cached page still asks the visitor to download.
Checking it yourself
Ask the host for the number and write it down. On a VPS, SHOW VARIABLES LIKE 'max_connections' tells you, and SHOW STATUS LIKE 'Max_used_connections' tells you the high-water mark since the last restart. If that mark is more than about 70 percent of the limit on an ordinary day, a launch will break you.
Then test. A load-testing tool such as ab or hey pointed at a staging copy shows when errors begin. Do it a week before the event, not the morning of. The uptime calculator shows what a bad hour costs in a month's availability figure.
What would have caught it
- Ask your host what the connection limit is and what happens when you hit it.
- Load test before a known busy moment.
- Cache everything that does not need to be live.
- Stagger announcements if you can.
- Look at the high-water mark for connections after every busy day, and treat 70 percent as the warning line.