The Host's Casebook / The shop that moved on a Friday afternoon

The shop that moved on a Friday afternoon

CASEBOOK

6 min read · 1,392 words

A note on authenticity. This is a composite story written by the editors, built from situations that come up again and again. It is not the account of a particular named person or business. Real reader stories go through the submission page and are marked as reader-submitted.

A small online shop selling hand-thrown ceramics decided to change hosts. The owner was happy with the site itself but the old plan had become slow and the renewal price had jumped. She found a better offer, and picked Friday afternoon for the move because that was when she had time. The files and database were copied over without trouble, the new copy looked right, and DNS was switched at around four o'clock.

It is hard to think of a worse hour. Over the next several hours some customers reached the new server and some the old one, depending on which DNS answers their provider had cached. Orders arrived at both. When the owner checked on Monday, the old server held a dozen orders that never appeared in the new shop's dashboard, and several customers had been charged without a confirmation email.

The plan as she made it

Her checklist was reasonable for a brochure site. Export the files, export the database, import both at the new host, check that the homepage loads, then change the nameservers or the A record at the domain registrar. She tested the new copy by looking at it through a temporary address supplied by the new host, and every page looked right, including a test checkout in sandbox mode.

What the checklist lacked was any thought about the middle. A DNS change does not flip like a light switch. The old answer sits in caches around the world for as long as its time to live, and that figure had been left at the default of four hours, or 14400 seconds. Nobody had told her to reduce it. Nothing in the process asked what should happen to an order placed halfway through.

She also left the old hosting account untouched and live, which was the right instinct, since you should never cancel the old plan on the day of the move. It turned out to be part of the trouble, because the old shop kept happily accepting orders.

What happened between four o'clock and Monday

At four the registrar showed the new address. For some visitors that was effective straight away: those whose internet provider had no cached answer, or whose cache had expired, were sent to the new server. Others were sent to the old one for as long as their resolver held the old record. The cache lifetime counts from the moment each resolver last fetched the record, so the expiry times were scattered across the whole four-hour window, not lined up at four o'clock.

A Friday evening shopper at home on one broadband provider might therefore see the old shop, while their neighbour on another saw the new one. Both shops looked identical, so neither visitor could tell. Both shops took payments through the same payment provider, which meant money arrived correctly in her account every time. What differed was where the order was recorded.

Customer A cache expired Customer B still cached New server new database Old server old database Payment provider Both servers take payment. Only one records the order where she looks.
For several hours the shop existed twice, with a separate set of orders on each side.

Finding the missing orders

The first sign came on Saturday morning, from a customer who wrote to say she had paid but received nothing. The owner looked at the shop's dashboard, found no such order, and assumed it was a one-off email glitch. A second message followed in the afternoon. By Sunday evening there were four, and she began to worry.

On Monday she logged into the payment provider's dashboard and compared it against the new shop's order list. The payment side showed more successful charges than the shop had orders. The difference was the missing set. Then she remembered that the old plan was still active, logged into the old control panel, opened the old database through the admin tool, and found the dozen orders sitting in a table that nobody had touched.

The confirmation emails had not gone out for a different reason. The old server's mail setup had been tied to the previous hosting account's sending reputation, and several of the messages had landed in spam, while some had not been sent at all after the domain's records changed. Customers who had paid saw nothing, and a few filed disputes with their card issuers before she could reach them.

The clean-up

She spent a good part of the week cross-referencing payment records against the two databases and apologising by email. For each order, the steps were the same:

  1. Match the payment reference in the provider dashboard to the row in the old database.
  2. Re-create the order in the new shop by hand, with the correct items, delivery address and status.
  3. Send a personal email confirming the order and the expected dispatch date.
  4. Note the order in a spreadsheet, so that nothing was counted twice.

Stock levels were off as well. Items sold on the old server had not been deducted on the new one, so for two days the new shop showed ceramics as available that were already promised. She made a small number of customers wait for a second firing of a popular mug, and offered them a discount for the delay.

The financial damage was modest. The reputational side took longer: the reviews page picked up a couple of grumbling comments about not hearing back, which stayed visible for months.

The next move, done properly

Several months later a different host change came along, and this time she planned it. The procedure she followed is a decent template for any small shop:

  1. Several days before the move, lower the TTL on the relevant records to something like 300 seconds. Wait out the old TTL, so every resolver in the world has the short value. The TTL planner works out the date.
  2. Prepare the new server and test it through its temporary address or a hosts file entry on her own computer.
  3. Choose a quiet time. For her, the shop's traffic is lowest on weekday mornings, so she picked a Tuesday.
  4. Put the old shop into maintenance mode, so it stops taking orders, then take a final copy of the files and database.
  5. Restore that final copy on the new server, check the latest order is present, and change DNS.
  6. Keep the old shop in maintenance mode for at least a day, or for as long as the old TTL, and watch for any stragglers.

The whole switch had a twenty-minute window when ordering was paused, with a banner on the shop saying it would be back shortly. A few customers returned later in the day. Nobody was charged without an order.

Days before Lower TTL Tuesday am Pause orders +10 min Final copy +20 min Switch DNS Next day Old site off times illustrative
The order of steps matters more than the speed: no two copies of the shop ever take orders at the same time.

Look at your own configuration

Before any move, find out what the current TTL is. From a terminal:

dig example.com A +noall +answer
dig @8.8.8.8 example.com A +noall +answer

The number in the second column of each answer is the seconds remaining on that resolver's cached copy. Repeat the query a few minutes apart and the countdown shows how the cache ages. After the switch, query a few public resolvers and compare them with your registrar's DNS to see how far the change has spread. If you can edit your computer's hosts file, you can also point the domain at the new server for testing before anybody else sees it.

A shop that takes orders needs a freeze period. A static site can be moved on a whim, because both copies show the same content. Anything that writes data (orders, comments, bookings, form submissions) needs one writer at a time.

What else people want to know

Is Friday really that bad?

Mid-week mornings give you the whole working day to notice and repair problems, with support teams at full strength. A Friday afternoon move means the first symptoms land over a weekend.

Can the host do the move for me?

Many hosts offer a free migration, and it is worth taking. They still cannot know when your orders arrive, so the freeze is your decision.

What if I use a payment link rather than a database?

Then the order may only exist at the payment provider, which makes the clean-up easier, but check stock and confirmation emails anyway.

How long should the old plan stay open?

At least a week, in maintenance mode, as a safety net. Cancel after you have checked for stragglers.

NextThe blogger and the vanishing domain

More from The Host's Casebook

Composite case

The hobbyist and the home connection

A keen hobbyist ran a small website from a Raspberry Pi at home and pointed his domain at the home router's...

Composite case

The photographer and the 4 MB hero image

A wedding photographer was disappointed that her new site loaded slowly. She had upgraded her hosting twice...

Composite case

The estate agent and the twelve-megabyte slideshow

An estate agent's homepage opened with a full-screen slideshow of eight property photos. Each one was a...