Over the past few weeks, especially since the

However, a recent post on the Akismet Blog particularly caught my eye. The post, entitled “Is It Spam?” details the shift in spammer tactics and how some users have fallen for it, marking spam comments “not spam” despite the better judgment of Akismet.

However, it was one of the last paragraphs that really got me thinking:

In the case of the pingbacks (the ones that start [...]) the spammers are actually stealing your work…

Though that fact is clear to anyone who reads this site regularly, it occurred to me that our aggressive blocking of trackback and pingback spam may be preventing us from identifying spam bloggers and preventing us from shutting them down.

Curious, I decided to check my own spam files and was more than a little surprised at what I found.

The Problem

As wonderful as Digital Fingerprints and other anti-splog features are, trackbacks and pingbacks remain one of the most common ways for a spam blog to be identified.

The reason is simple, spammers, usually in a bid to either appear legitimate or obtain some additional incoming links, often provide links to the original post. Those links, due to the nature of the software they use, cause trackbacks to be sent to the original site and those trackbacks can lead Webmasters right to the people misusing their content.

Another alternative is that, in many cases, bloggers will link to other articles on their site, as I’ve done with this one, and those links often get picked up when the article is scraped. Those links, in turn, produce trackbacks that can be easily followed up on.

The problem is that anti-spam solutions such as Akismet and Defensio aggressively filter out and stop trackback spam. Most Webmasters, just happy they aren’t being inundated with junk comments, never check their spam folders. This means that those trackbacks, which can be very useful in detecting scraping and plagiarism, are often filtered out before anyone, including the blogger, sees them.

Down the Rabbit Hole

Curious to see if this was a real problem or simply an academic issue, I delved into my Akismet spam folder to see what was there.

The sample was relatively small, approximately 1600 comment spams. This is mostly due to me switching back and forth between Akismet and Defensio over the past week and that my blog automatically discards spam comments on posts older than one month.

However, when I did a filter search for “[...]” I found thirteen trackbacks, all of them containing various amounts of scraped content (Note: I found another thirteen in my Defensio folder, which had approximately 900 spam messages).

In every case the site was hosted on a “.info” domain, had scraped a excerpt of one of my stories and introduced it with a generic statement such as “While looking through the blogosphere we stumbled on an interesting post today. Here’s a quick excerpt.” None carried my digital fingerprint.

In most cases, the excerpts were fairly short though, in a few cases, they spilled over into a few paragraphs. In all cases, they were surrounded with advertising from a variety of sources.

Of the thirteen sites in my Akismet folder, about half were down and the remainder appeared to belong to the same spam blog network. However, these are thirteen scrapers I would never have known about if I hadn’t manually reached in and filtered through my trackback spam.

It’s a scary thought, but it makes one wonder how many I have missed up until now and, even worse, how many do I never get the chance to see?

The Good News

On the upside, these particular sites, though definitely scrapers, are not what I would call “high priority”. Since they only reposted an excerpt of the feed and did offer a link back, the damage they can cause is somewhat minimized. However, it is still annoying that this has been going on right underneath my nose and I never would have found out about it had it not been for manual intervention.

The fear isn’t so much that spam blogs like these will stay hidden, but that Akismet might bury someone scraping the full post or even the full feed. That seems to be somewhat rare, possibly an indication that Akismet considers the amount of reuse when analyzing whether a comment is spam or not, but without more Webmasters looking through their Akismet spam, there is no real way to know.

However, none of this is to say that Akismet, or any other spam plugin, is helping spammers out by assisting them in escaping detection. I would imagine the net effect of the plugin is still very bad for the spammers as they depend on these trackbacks and comment spams to build their networks.

Furthermore, any spammer caught up in your Akismet spam folder is likely an ineffective one to start with. If Akismet has already pegged the site as garbage, you can say with little doubt that Google, Technorati and others probably have as well.

Still, it may be worth a few moments to check your Akismet spam folder and see what you find.

Checking For Scraping

The process for checking your Akismet folder for potential scrapes is actually fairly simple. Visit your Akismet folder in your WordPress panel, it can be found under the “comments” section, and then, using the search box up top, type in “[...]“. It should take you to a list with the suspicious trackback posts.

This system is far from perfect as not all trackback spam seems to include that intro, despite it being something of a standard. Unfortunately, neither Akismet nor Defensio offer a means to simply filter spam based upon spam type, this makes the above search, though somewhat simplistic, the best alternative at the moment.

If you are using Defensio and you perform the check, be certain to tick the box that includes “obvious” spam as the majority of trackbacks in my folder, ten total, were labeled as such.

Conclusions

This situation is very frustrating. We, as bloggers, are forced to make a choice between being inundated with comment spam and being able to effectively follow up on scrapers and spam bloggers. Of the two, comment spam certainly seems to be the most annoying, especially considering the ratio of scraping to comment spam, and the most time-consuming to fight.

In short, the time it would take to deal with comment spam without Akismet far outstrips the time it takes to reach into the spam folder once every few weeks and search for suspicious pings.

Still, it is frustrating that Akismet does not offer an easier way to track these cases, either by enabling filtering on spam type or offering a special folder for suspicious blogs.

Though I definitely would rather have the protection than not, it would be nice if these plugins could help us stop other kinds of spam than just comments to our blogs.

For me, I’m going to debate what action to take against these scrapers. Though I certainly can and probably will notify their advertisers, I am unsure about taking additional action due to the nature of the reuse.

In the meantime, I’m encouraging everyone to delve into their spam folders and see what they find. Be sure to let me know if you find anything exceptionally interesting in there. I’ll be eager to hear about what turns up.

Leave a comment below if you want to share.

reCAPTCHA is still holding the comment spam at bay and I have swapped out Akismet for Defensio in hopes of reducing the number of trackbacks that seep through.

So far that move has been a stellar success, in the first four hours of operation, Defensio has had a perfect accuracy. However, what is stunning is the statistics it has also collected.

Striking Numbers

I installed Defensio on this blog at about 8:30 AM central time this morning. Now, less than four hours later, Defensio has already caught over 200 spam comments.

If this pace is maintained, then that will represent well over 600 spam comments and trackbacks today. Though it is reCAPTCHA, and not Defensio, stopping the comments. what is stunning is the increase in activity this represents.

Typically this site has gotten its fair share of comment spam, but it has usually been well under a hundred per day. By my estimation, with a daily average of about 50 per day before, this represents a 12-fold increase in comment spam, almost overnight.

Conclusions

The defenses are holding now but I will keep a close watch on Defensio to make sure that it doesn’t trap legitimate comments or let any spam slip through. I’ll definitely report back, likely after the holiday, about how it has done.

Hopefully though this 100% accuracy trend will continue.

Though it doesn’t have much to do with content theft, I have several reasons for wanting to cover this. First many of the RSS scrapers and spam bloggers also use this technique to supplement their work. Second, in some cases, the spammed comment contains scraped content, either from your site or others making it an infringement. Finally, it is an issue that is dear to the target readers of this site, bloggers and Webmasters.

Though WordPress’ reputation against spam blogs has been almost impeccable, it has proved to be very vulnerable to comment spam. This has given rise to an entire cottage industry of anti-spam plugins and most of them, in my experience at least, have been ineffective.

This lead me, about a month ago, to disable comments on all old posts. However, I have since backed away from that position because, among other reasons, it simply was not working.

The most effective comment spam plugin I know of, Akismet, is made by Automattic, the operators of WordPress.com. It is a generous gift to the community and it comes at what must be great expense to Automattic since it works by letting their servers filter the millions of comments that get submitted. However, it is not perfect, by Automattic’s own admission, and it does not stop the comment spam from going through, just from appearing on the site.

Unfortunately, WordPress’ problem with comment spam runs much deeper and it renders nearly 99% of all anti-spam methods useless. However, a change on the backend could, potentially, fix that and change the comment spam game forever.

How a Comment Gets Posted (and The Problem With It)

A comment in WordPress works just like any other form.

You most likely have a comments.php file in your template that represents the actual comment form. That is embedded into your post pages via a template call. The comments.php form, upon submission, sends the comment to another file wp-comments-post.php, which sends the comment up the chain of commands and, eventually, places it in the database.

It is a simple form that works like any other. It is also the same process as when you send an email via a Web form or post a forum. However, the problem is that, with WordPress and other spam-prone applications, the backend does not know what the frontend is doing.

Basically, with the default install, wp-comments-post.php has no way to confirm that the comment has come from comments.php or anywhere else on the domain.

Spammers, being the clever lot that they are, simply started calling the wp-comments-post.php without ever visiting the site itself. They simply call the file with a specially-formatted address and, magically, a comment is submitted though the bot never set foot on the actual post page.

This is bad news for WordPress users as nearly all spam counter measures rely on modifications to the comments.php file to work. This includes most captchas, spam questions and even some comment disabling plugins. The spammer simply bypasses those measures, leaving only post-submission filtering to weed out the junk from the real comments.

Though, on most sites, that is a fairly effective approach, sites with large volumes of spam, such as this one, might find it unacceptable. Not only does it mean that some spam is destined to escape the filters and go live, but it can put a strain on the sever, even if, as with Akisment, most of the filtering is done elsewhere.

Furthermore, if email spam has taught us anything, filtering systems are prone to the “better mouse” problem. If one clever spammer finds a way to game the system, the hull will have been breached and all could be flooded.

How Bad Is It?

The problem is rampant. Consider this screenshot taken from my own site stats yesterday.

You can see that the wp-comments-post.php file is the fourth most called file on my server (Note: Both share-this.php and the ajax-edit-cooments files are often called multiple times in a single page, thus why they are so high.). A quick check of the comment count shows that there is no reason for that to happen.

There are hundreds of hits per day on that file, most of which never access the site itself.

This has led to a whole slew of solutions to the problem. The first is to simply rename the file. However, spammers have grown wise to that method, detecting the new name in as little as ten hours.

Another is to edit your .htaccess file to block visitors from accessing the wp-comments-post.php file without first visiting your domain. I implemented this myself on Plagiarism Today but, while my comment spam volume decreased some, it did not stop. Spoofing a referrer is pretty trivial and it seems that most comment spammers are already doing that.

Yet another hack involves increasing the time between comment submissions, a method that works to stop spammers that “flood” your comments, but does nothing to stop spammers who post once and then come back at a random time later to post again.

One final method, which I ran across some time ago but have been unable to locate again for this article, involved inputting code into the comments.php file that would then be verified by the wp-comments-post.php file. Though it was a messy edit that involved hacking both files, it would have been, theoretically, effective. Once I locate the hack again, I will try it and see if it does indeed work.

In the end though, short of hacks and server alterations, there is no way to prevent this kind of injection. Since almost all plugins deal only with the comments.php file, there is no simple way to effectively block this kind of abuse.

Fixing the Problem

This problem is not unique to WordPress by any stretch of the imagination. None of this should be taken as a criticism of WordPress or its developers. This problem is present on other blogging platforms, message board applications and nearly anything that accepts input from the outside world and posts it to the Web. WordPress merely happens to be what I use and what I am most familiar with.

That being said, there needs to be a fix for this problem. There needs to be some way for the backend, wp-comments-post.php, to ensure that the comment actually came from the frontend, comments.php.

One solution involves using a generic anti-spam question in the comments file but then hacking the wp-comments-post.php file to die if the answers to not match. Thus, anyone calling the backend directly without knowledge of the question would get an error.

However, a static method, like the one described in the post, could be easily beaten by a spammer just adding the variable to their software. A more random implementation, such as the one described in the comments, would provide more protection but could still be figured out if needed since computers are very good at math.

I am not a programmer, but what seems to be needed is a means for the two files to handshake with one another in a way that a spammer can not crack. One example might be to create a hash of the comment using a key that exists only on the server. Another would be to use a pseudorandom variable such as a random number generator, the time on the CPU clock or anything else the two files could share. Another idea would be to have the backend check the WP log and ensure that, at the very least, the IP address involved visited the post page in question before commenting.

(Note: The above suggestions are offered “off the cuff” and probably would not work. Please post suggestions and ideas in the comments.)

This would not be easy. It might require rethinking the entire comment posting process, but certainly there has to be a way to at least improve the situation so spammers can not, with easy, abuse the system.

I am open to any and all suggestions on the process. Please comment below if you have any thoughts.

Some Brief Good News

I did, recently, run across some good news in this fight. I installed reCAPTCHA on my blog a few days ago as an experiment. Though it didn’t stop the flow of spam comments, it did improve Akismet’s accuracy greatly.

It appears that, for whatever reason, Akismet has an easier time dealing with comment spam when it comes almost solely from the backend. Since I installed reCAPTCHA, I have not had any spam comments go live or enter the moderation queue.

I plan to continue the experiment for at least a few more days to see if that trend continues.

(UPDATE: I just received an email from Ben Maurer, the tech lead on the reCAPTCHA project, he said that reCAPTCHA counts the spam as it eats as spam in WordPress, that could explain why Akismet seems to be so accurate. Still, what intrigues me most is that no spam has gotten all of the way through. It seems logical that reCAPTCHA is blocking the spam that actually uses the form, which was the spam getting through from time to time, while Akismet easily handles the spam directly injected through the backend.)

(UPDATE 2: As my education on reCAPTCHA continues, it appears that the plugin DOES validate against comments injected into the backend. That officially makes this my favorite anti-spam plugin.)

Conclusions

Closing this backdoor will not be easy nor will it obliterate comment spam. However, channeling it through the traditional forms makes it possible to apply various Turing tests to weed out the bots.

In short, it won’t put an end to comment spam or replace filtering, but at least it will add an extra line of defense.

Right now, WordPress users are just one clever spammer away from a tidal wave of spam. If someone can find a way to beat Akismet and other spam filtering plugins, there is no backup plan.

Perhaps now, while the situation is somewhat in hand, it is time we started working on one.