A developer copied a production database to a staging site to test a new order workflow. The copy contained real customers and real email addresses, and the staging site was connected to a working mail service.
When she tested the new "your order has shipped" message, it went to several hundred customers for orders that were delivered months before. Replies and confused phone calls followed.
The shop now scrubs customer data from staging copies and disables outgoing mail there by default. The shop in this story, a small online seller of kitchenware, is a composite, and the details are illustrative. What is not invented is the sequence: a perfectly reasonable testing habit, one forgotten setting and an afternoon spent apologising.
Why staging copies hold real data
Testing an order workflow with three hand-made orders proves little. Real data has odd shapes: customers with no phone number, addresses with accents, orders with refunds and partial shipments, discount codes from years ago. A workflow that works on tidy test data fails on the first awkward real one. So developers copy the production database to staging, and they are right to want to.
On this occasion the developer did it the usual way. She exported the database, imported it into a second database on the same hosting account, copied the site files into a subfolder, changed the database name in the configuration and pointed a spare subdomain at it. The whole job took twenty minutes.
Everything came across, including things she was not thinking about. The settings table carried the mail service credentials. The scheduled task list carried the jobs that send reminders. The orders table carried statuses, so several hundred orders sat in the state "delivered". And the customer table carried every address the shop had ever collected.
The afternoon it happened
The new feature sent a message when an order's status changed to "shipped". To test it, the developer selected a few orders in the staging dashboard and ran the bulk action that marks them shipped, expecting to look at the first few messages in her own inbox, because she had set the first test order to use her address.
She had not realised that the bulk action applied to the whole filtered list, which was the entire order history. The status change fired the message for each order in turn. Within a couple of minutes, 340 emails had left the building.
The first reply arrived inside ten minutes: "I received this months ago, is something wrong with my parcel?" Another asked whether they were being charged again. The shop's phone, which usually rang a few times a day, rang constantly. The customer service person, who had no idea a test was running, spent an hour reassuring people before anyone connected the calls to staging.
Why mail left a site that was only a test
A staging site is not a model of the real site; it is a second copy of the real site. It will do what the real site does unless something stops it. Sending mail is the standard example because mail leaves the system and cannot be recalled.
On this shop the application sent mail through an outside email service, using an API key stored in the settings table. The copy brought that key along, so staging authenticated to the same service and sent through the same account, from the shop's real sender address. The recipients' mail programs had no way to tell staging mail from the real thing. The messages passed SPF and DKIM because, from the mail service's point of view, they were legitimate.
It was also not the only pathway. The same copy carried scheduled jobs. If the developer had left staging alone overnight, the abandoned-basket reminder, the review request and the weekly newsletter would all have run against the same list. Payment gateways, SMS providers and webhooks to other systems are in the same category: anything that reaches outside the site.
Cleaning up
The immediate steps came in a deliberate order. First, stop it: the mail service API key was revoked from the service's dashboard, which instantly cut off staging and, for a few minutes, the live shop as well, until a new key was entered there. Second, find out who received what, from the service's sending log, which listed every recipient and time.
Third, tell the customers. A short, plain message went to the 340 recipients: the earlier email was sent in error during a system test, no action is needed, no payment was taken, and apologies for the confusion. People were mostly kind about it. A few asked whether their data had been exposed; the true answer was that it had not left the shop's own systems, and the message said so.
Finally, the shop checked whether the data protection rules where its customers live required anything further. In this case the exposure was an unwanted message to the customers' own addresses, with no disclosure to third parties, and the shop recorded the incident internally.
Making staging safe by default
The shop changed its copy procedure so the safe state is automatic. Three layers, each independent of the others:
- Outgoing mail is redirected. On staging, the application is configured with a mail catcher or a transport that writes messages to a log instead of sending them. If the application cannot do that, set the SMTP host to a local capture tool, or use a mail plugin's "send everything to this address" option.
- Personal data is scrubbed after the import and before anything else runs.
- Scheduled jobs and third-party keys are switched off or replaced with test keys.
A scrub script can be as short as this, run immediately after the import:
UPDATE customers SET email = CONCAT('customer', id, '@example.invalid'),
phone = NULL,
name = CONCAT('Test Customer ', id);
UPDATE settings SET value = '' WHERE name IN ('mail_api_key','sms_token');
UPDATE users SET email = '[email protected]' WHERE role = 'developer';The .invalid top-level domain is reserved so that it can never resolve, which makes it a safe placeholder. Any order that does reach the mail layer will bounce at once without leaving the server.
A banner finished the job, because a copy that looks identical to the live site invites exactly the confusion that caused this. Staging pages now show a coloured strip across the top and the title begins with "STAGING", and the site refuses to be indexed by search engines.
Common questions
Is it enough to put staging behind a password?
It stops visitors, but it does not stop the site from sending mail or running jobs. The password protects the front door only.
Can I just ask the mail provider to block staging?
Some let you restrict by sending address or IP. That helps as a second layer, but depends on a setting nobody remembers to review.
Should developers ever see real customer data?
Sometimes a bug only reproduces with a particular real record. If so, copy that one record, scrubbed, and not the whole table. Fewer rows mean less to lose and less to explain.
What about test environments in the cloud?
The same rule holds. The environment copies configuration as well as data, so scrub both.
What would have caught it
- Disable or redirect email on staging.
- Anonymise personal data in test databases.
- Mark staging clearly so nobody mistakes it for the real thing.
- Replace live keys for mail, SMS and payments with test ones as part of the copy script.
- Turn off scheduled jobs on staging, and check that bulk actions show how many records they will touch before they run.