A volunteer-run local news site enabled comments on every story. For a year it was lively and polite. Then spammers discovered it, and each article collected dozens of links to gambling sites.
Moderating by hand took up a volunteer's evenings. Some spam slipped through and got the site flagged by a security filter, which put a warning in front of its visitors.
The editors turned on a spam filter, required a first approved comment before automatic posting, and closed comments on older articles by default. The paper is a composite and the figures are examples, but the course of events will be familiar to anyone who has left a comment box open on the internet for long enough.
The site and its comments
It was a website for one small town, run by about eight volunteers: a retired sub-editor, two students, a parish councillor who liked to cover planning meetings, and a few people who took photographs. It published four or five short stories a day on WordPress, on an ordinary shared hosting plan paid for out of donations. There was no advertising and no budget, and the editors liked it that way.
Comments were switched on from the start, because the founders believed a local paper should be a conversation. Anyone could type a name, an email address and a message under any story, and it appeared immediately. For a year this worked better than anyone had expected. People argued about a bus route, corrected the spelling of a street, thanked the fundraisers, and told stories about the swimming pool that the reporters could not have found themselves.
There were a few rude comments, which the editors removed, and a very occasional advert for a local plumber, which they let stand.
When it turned
The change was gradual and then sudden. In the second year, a trickle of comments appeared that were obviously not from locals: a phrase of broken English praising "this informative post", followed by a name that was really a link. A few days later there were more. Then one Friday the editors opened the dashboard and found 340 comments awaiting moderation overnight, nearly all linking to online casino and betting sites.
Automated tools find comment forms the way a search engine finds pages. They crawl, spot a form that accepts a name, a message and a web address, and try it. If the post appears, the address goes onto a list, and the list is sold. A comment form with no obstacles will be found, and the finding takes days to weeks, not years. A small audience is no protection, because the tools do not care who reads the site. They care only that the page is on the open web, with a form on it.
What it cost to clean up by hand
The first response was to moderate harder. One volunteer, who had joined to photograph the carnival, took on the comment queue. Each evening she opened the list, read the lot, deleted the junk and approved the real ones. At the start it was ten minutes. By the third month it was two hours, and it never finished, because the queue refilled overnight.
She also found out what everyone who does this finds out: the dull spam is easy and the clever spam is not. Some comments were tailored, with a first line that referred to the article's actual subject, and a link tucked into the name field or a polite sentence at the end. A few passed her eye and went live.
The cost was not only her time. The comment queue became the reason she stopped taking photographs, and the editor noticed that no one new wanted to take it on.
The warning page
What forced the issue was a Sunday evening message from a reader: when she clicked a link to the site from her search results, her browser displayed a full-page red warning that the site may be deceptive or contain harmful content.
Browsers use shared lists of dangerous sites, and search engines and security filters feed those lists. A page that carries a lot of outgoing links to disreputable destinations, or hosts content inserted by third parties, can be added. The spam that had slipped past, along with older approved comments, gave the site dozens of links to gambling sites with poor reputations. The filter had treated the site as a neighbour of those sites, not as a victim of them.
The editors found the details in the webmaster tools of the search engine, where the site had been registered when it was set up. It listed the affected addresses and the type of problem. The remedy was to clean every one of those pages and then request a review. They did, and it took about three days for the warning to go. For those three days, visits from search were close to zero.
# With WP-CLI: how many comments, and how many contain a link?
wp comment list --status=approve --format=count
wp db query "SELECT COUNT(*) FROM wp_comments WHERE comment_approved='1' AND comment_content LIKE '%http%'"
# Delete everything already marked as spam
wp comment delete $(wp comment list --status=spam --format=ids) --forceThe fix, in three parts
They did not switch comments off. The editors thought the comments were worth saving, and they were. They changed three things.
- A spam filter. They enabled the spam-filtering service that works with the publishing software. It compares each comment against a large shared database of known abuse and puts suspect ones aside before any human sees them.
- A first approved comment. A new commenter's first message is held until a moderator approves it. After that, the same name and email can comment freely. Real readers see a short "awaiting moderation" notice once, and then forget about it.
- Comments closed on older stories. Anything older than 30 days stops accepting comments automatically, with a note that says so. This took the biggest slice of the spam away at a stroke, since the old stories were where most of it landed.
They added two small extras. Comments containing more than one link are held for review, and outgoing links in comments are marked rel="ugc nofollow", which tells search engines not to treat them as the site's endorsement and removes much of the point of posting them.
Weighing the trade-offs
| Measure | What it cuts | What it costs |
|---|---|---|
| Spam filter | Most automated junk | Occasional false positive, so check the spam folder |
| First-comment approval | Nearly all that gets past the filter | A short delay for a newcomer, once |
| Auto-close after 30 days | Spam on old stories | Late replies on old stories are lost |
| Hold messages with several links | Link-stuffed spam | Genuine posts that cite sources wait |
| Registration required | Casual abuse | Fewer comments overall |
The editors also wrote a short comment policy and linked it under the form. This does nothing to spam, but it helps with the humans, who now have a stated reason for any removal and a person to contact.
Questions that come up
Should we turn comments off?
If you cannot moderate, yes. A dead comment box is better than a full one of gambling links. If readers value the discussion, filter and hold.
Does nofollow stop spam?
It removes the search benefit, which is why it is done, but it does not stop the attempts. Filters and moderation do that.
How long does a browser warning last?
Until the pages are clean and a review is granted. In this telling that was a few days, but it can take longer.
Could the site have been hacked instead?
It is worth ruling out. Spam in comments is a different problem from injected code, but check for unknown admin users and changed files if the warning names pages you did not write.