Though these spam blogs will take many different approaches, inevitably, many of these spam blogs will use scraping as a means to fill their pages and appear more authentic to both Google the search engine and Google the host administrator.

Bloggers, especially those that frequently have spam-friendly keywords in their sites, should be aware of the likelihood of increased scraping on the Blogger service. Now would be an excellent time for sites that offer email subscriptions to check for any @blogger.com accounts and everyone to consider taking feed protection steps such as installing Antileech, creating a feed header/footer or using a digital fingerprint. 

Sadly though it recently seemed as if Google was on the offensive against spam, it now appears as if the tables have turned.

While the new spam wave is still ramping up, now is the best chance for bloggers to be aware of the issue and be prepared to take action as needed. Hopefully, Google will fix this issue soon and the impact of the problem will be limited.

If not, then Blogspot could easily become even more of a spam wasteland than before, making it even more difficult for legitimate bloggers to get noticed on the service and for Webmasters everywhere to keep their content out of spammer’s hands.  

These days, even death does not put an end to content theft. It merely opens up new avenues for it.

As a recent article on Blue Hat SEO pointed out, nothing is ever really deleted from the Web.

Caching sites and archives hold on to your content long after the page has been removed and, as the article demonstrates, anything that is available can be scraped.

It is the equivalent of picking a dead man’s pocket, but it is a type of plagiarism that can and does happen. It is also a kind of plagiarism that raises a whole slew of new questions and concerns.

How it is Done

The process for scraping a dead site is surprisingly simple, involving only five steps.

1. Visit an archiving site such as The Internet Archive
2. Lookup an old site that you know to be deceased
3. Find an old, but still relevant, article
4. Check to make sure that the article has not been posted elsewhere
5. Copy/paste it onto your site

As the original article points out, there are ways to automate this process, such as using sitemaps, but the process clearly works best by hand. With that in mind, this method is unlikely to be used by professional spam bloggers, who need more content than this method could provide, but could be favored by human-powered spam sites that are trying to appear legit.

To those Webmasters, the unethical ones catering to humans and the search engine bots, this type of content theft can seem like a dream come true, especially when one weighs the advantages that come with it.

The Advantages for Scrapers

When looking at why a scraper or a plagiarist would find this kind of content appealing, consider the following:

1. Since the content has been removed from the search engines, there is no possible duplicate content penalty and no original site to compete with in the search engine rankings
2. If the original site is shut down, most likely, the people charged with protecting the content are no longer looking for infringements. Thus, the odds of finding legal trouble are slim to none.
3. Finally, there is a wealth of dead information out there, ready to be copied. Though the Web is growing, sites are also dying every day, thus ensuring a steady stream flowing into a large, growing pool.

Of course, taking advantage of such a system requires a certain amount of cleverness and work ethic, things that are in short supply for most that would steal content, but if one sees the advantages as outweighing the drawbacks, it would still be an appealing option.

However, given the wide range of legitimate, easily-accessed content available to scrape, it seems unlikely that most infringers would take this route.

Despite that, there is little doubt that at least some do scrape dead sites for at least part of their content and that fact raises some difficult questions that don’t come with easy answers.

Questions to Ponder

Obviously, given the current length of copyright law, any such scraping would likely be an infringement. Unless the work was placed under a Creative Commons license or donated to the public domain, the work is almost certainly still protected and the copying is illegal.

However, with the business interest in protecting those works gone, any copyright is likely to go unenforced. Most won’t care to check for any infringement and those who do will likely be unaware of the potential danger. Most feel that, when the site is removed, the risk of plagiarism disappears.

Worse still, the authors of the work, who often created the works as a work for hire and hold no copyright interest in it, would be the ones holding the greatest continued stake in the work. Though their employers have long since closed up shop, their name and reputation remains affixed to the works.

But even if the copyright holder does detect such post-mortem plagiarism and expresses an interest in defending those works, going through the motions could be difficult. Though the DMCA does not state that URLs must be provided with a DMCA notice, some hosts have that requirement and it is the most common way of identifying the original works. With the site down, that element could prove difficult.

Furthermore, since there would be no economic damages and it is unlikely that the work was registered, suing for said infringement would be almost financially impossible. In short, though stopping such plagiarism would be possible, it would be more difficult and less worth the time spent.

It may not be the perfect crime, but it is certainly pretty close. It’s very hard to envision a scraper paying for such an infringement, no matter how unethical or illegal it is.

Prevention is the Key

Since enforcing the copyrights of a dead site is, in many cases, impractical, the best approach is to look at preventing the infringement from happening in the first place. Doing that involves having a sound shut-down strategy that includes the following steps:

1. Handle all existing plagiarism cases: Deal with all ongoing plagiarism and content theft cases that you can. Ensure that the content is removed completely before beginning to take content offline.
2. Block known archiving sites: Edit your robots.txt file to exclude the Internet Archive and other archiving services. All cached copies of your site should be removed within a few days. (Note: You can skip this step in favor of step three but it is worth being certain that these archiving sites drop your cached copies.)
3. Block All Search Engines: Since search engine caches can remain active for some time after a site goes down, once you are certain the archive sites have removed the work, edit the robots.txt file to exclude all search engines. You can also use meta tags to prevent indexing, or both to ensure that all spiders stop indexing your site.
4. Move Content to a Hidden Location: Move a copy of your site to a hidden, but accessible, location. Consider making the site password protected to ensure that it can not be indexed or visited by anyone you do not personally authorize.
5. Remove the Original Site: Take down the site and close up shop, being certain to leave behind a means of contact in the event a problem is discovered later.

Though the process seems lengthy and difficult, it can easily be done over a couple of days with most of the time spent waiting on the spiders to re-index your site. Once they do, the cached copies should be dropped almost immediately.

The key thing in all of it is to ensure that the cached copies of your work are removed BEFORE you take down the site. As long as you ensure that the major caching services, especially the Internet Archive, have removed your work before you shut down, you can greatly reduce the risk of post-mortem plagiarism.

Conclusions

The good news is that, though there is no way to tell how common this kind of content theft is, it almost certainly is very rare. In most cases, the meager rewards do not justify the effort required. Also, when it does happen, it will largely limited to the larger, better-known sites that are now defunct as it requires at least some advanced knowledge of the site’s existence.

The bottom line is that the content on any Web site, even a closed one, still has value. This is especially true if the rightsholder is looking to start up another venture in the future or is pursuing off-line methods of distribution. The fact that the site is down does not mean it is acceptable to exploit the effort and expense of the original author.

Perhaps the strangest side effect of this is the damage it does to the classic classic scraper mantra “If you don’t want your content scraped, get off the Web”. If shutting down a site does not put a definite end to content theft, then dealing with plagiarism on a live sight becomes an even more practical solution.

In the end, it seems that the only sure fire way to avoid scraping is to not put it on the Web in the first place. As it sits right now, once the content is on the Web, there is no way to erase it.

Though the attention to this issue is welcome, many of these guides contain false, misleading or incomplete information. Though they are produced by smart, well-intentioned people, their errors can lessen the effectiveness of their strategies and, in some cases, expose the person following it to legal danger.

For example, one guide I encountered a year ago encouraged people to take their grievances public immediately and post them to a special forum. Not only is this time-consuming and unlikely to succeed in many cases, but it opens up the person doing the posting to a libel suit if their information is wrong.

However, such dangerous mistakes are relatively rare and are usually limited to small and obscure sites. Instead, the most common mistake made when crafting an anti-plagiarism strategy is something much more simple: Forgetting about the host.

As strange as it may sound, the most common omission in many of thse guides is the most effective tactic of all, getting the site shut down.

Skipping a Step

It seems that most guides on plagiarism fighting are pretty good at telling you ways to detect the infringement and to contact the plagiarist. Many provide stock cease and desist letters to send to the infringing Webmaster and advice on how to deal with different kinds of plagiarists.

However, should that step fail, a majority of these guides will then offer advice on how to get the content removed from the search engines or get the site’s advertising cut.

Though targeting advertisers can be a very effective way of dealing with profit-motivated plagiarism, such as with scrapers, neither that nor targeting search engines is as useful for deflecting the potential problems that come with being plagiarized as getting the site or the content removed.

Interestingly enough, many guides will include DMCA contact information and stock letters for contacting the search engine, but will completely omit any information about sending such a letter to the host.

Whenever I see such an omission, I comment on it and, in most cases, it is corrected fairly quickly. I have only seen a few such guides remain for a long period of time without this critical information.

Still, the frequency of this mistake has made me wonder why so many people overlook it. However, it didn’t take me long to think of a few potential answers as to why.

The Hardest Button to Button

The problem with filing a DMCA notice with a host is that it can be a very daunting challenge. Even if you have the template handy, you have to first know how to determine who the host is, then, if they are in the U.S., locate their DMCA contact information and then contact via the means they specify.

That, in turn, requires a level of research many people are not comfortable with. If you are unfamiliar with networking tools, ill at ease reading through terms of use or only have limited knowledge about how the Internet works, sending a DMCA notice to a host can be a very daunting challenge.

Sending notices to Google and the other search engines, by comparison, is very easy. If you have the template in hand, there is only one page you need to know for each search engine. It is pretty trivial, from there, to send out the notices without doing any research and not wading into any uncomfortable waters.

However, this is dangerous for several reasons:

1. Doesn’t remove the content: Though it won’t turn up in the search engines you send the notice to, the plagiarized copies are still available on the Web and other search engines as well as human visitors can still access it.
2. Turns the search engines into the copyright police: This concentrates all of the responsibility for policing copyright into three or four search engines. This was not the goal of the DMCA and it gives those companies too much power and responsibility in this matter. A change in policy of just one search engine could, potentially, have drastic implications on the Web.
3. Can harm innocent bystanders: Search engine DMCA bans work differently from site to site but, in some cases, it is possible that more than the pages than intended can be banned, including pages written by other people.

While there is definitely a place and a time for using search engine DMCA bans, immediately following a cease and desist is not it. Typically, I only turn to search engine bans when everything else has failed and I am prepared to give up.

In my mind, it is a way to do something when it seems nothing can be done at all.

Correcting the Problem

Since fewer people are comfortable with sending DMCA notices to host, fewer people use them. Since fewer people use them, fewer people write about them and that means that fewer people know about their existence.

This in a brutal cycle where more and more people get incomplete information. Not only does this lead people to use less effective tactics, but leads to mistakes down the road when they attempt to contact the host, often resulting in false or incomplete notices.

The key, then, becomes to make sure that more people are aware this method of dealing with plagiarism and push them to take advantage of it. It also means working to ensure that they have the tools available to file a proper notice and send it to the correct person.

If we can do that, along with providing basic copyright information, we can go a long way to reducing the copyright drama that exists on the Web.

Conclusions

The vast majority of people who post anti-plagiarism guides are good, well-intentioned people that are trying to help others. Unfortunately, they are not always right and sometimes that advice can lead people astray.

It is important, when researching an anti-plagiarism strategy, not to just read one guide, but two or three. Don’t take any one person’s word, including mine, as gospel. Seek out other opinions, views and strategies. There is a constant dialog going on and, though I try to report on it, the Web is a big place and I don’t see absolutely everything.

Build your own strategy based upon your needs, time constraints and skills. When appropriate, experiment. If you learn something that works or see something new, share it with others and drop me a note as well.

I am always on the lookout for new techniques and strategies in prevention, detection and cessation that can help myself and other Webmasters. Input and feedback is always appreciated.

Note: I have not linked any of the guides that inspired me to create this story. My goal with this is not to call anyone out or embarrass anyone. These are complicated issues and mistakes are understandable. I want to encourage others to create more guides, not shame people that make simple mistakes. Furthermore, nearly all of the guides that I’ve seen with this error have since been fixed.

Radison, having heard about these new Web sites called “blogs” decided to create one herself and obtained a WordPress.com account. However, her ignorance about blogs quickly caught up with her. Shortly after her first post, she checked her feed statistics and found that a whopping eleven people had subscribed to her feed.

She then quickly learned they weren’t interested in her feed at all, just her content.

Fortunately, Rachel Radison does not exist. She is a figment of my own imagination. I created both her and her site as an experiment to see both how common scraping is and how long it would take for scrapers to find a blog.

The answer surprised even me.

Background

The idea for the experiment came from an article on A Daily Rant. There, the owner took an ongoing WordPress.com blog and shut it down, leaving only a “this blog has been moved” post to let subscribers know. He then tracked the subscribers to the feed and found that, even after most subscribers had moved over, eighteen still remained.

The experiment was interesting but flawed. Many humans change RSS readers and often forget to unsubscribe from old feeds everywhere they’ve been. For example, I am certain there are many dead feeds in my old Rojo account. It is entirely possible that, of the readers they had prior to the shut down, eighteen were simply old, but legitimate, RSS readers continuing to check the dead feed.

However, the idea seemed valid enough, take a dead feed and monitor the traffic on it. If you can set it up so that no humans should be subscribed to the feed, you can be reasonably sure that all traffic to the feed is from bots.

So, that is exactly what I did.

The Experiment

Using an old Gmail account, I created a new WordPress.com blog. I then gave the blog a theme and gave it a name “Rachel Radison’s Mortgage Blog”. I specifically chose mortgage a topic matter because it is both a spam-friendly keyword and it is a topic I have at least a little knowledge about after buying my new home.

In order to avoid being caught in WordPress.com’s spam filters, I decided to write the posts myself, using my extremely limited knowledge of the subject and a large amount of fluff. After a few moments of faking my way through the first post (being sure to add a footer disclaiming the site as rubbish), I made sure that the blog was going to ping all of the usual notification services and published it.

I then returned the next day to do another post but stopped off to check the feed stats, what I saw is below:

Even I was stunned by this. I had expected the feed to be scraped. But to have eleven “subscribers” after less than 24 hours was stunning.

Some of these subscribers could be easily explained. Technorati and Google both picked up the feed almost immediately. Likewise, WordPress seems to have its own crawler. However, no others have picked it up as of this writing and that leaves at least eight “subscribers” unaccounted for.

But Wait, It Gets Worse

I posted again on the seventeenth but, due to difficulties with my move, was very late in posting on the eighteenth. However, before I posted, I checked the feed stats again, what I saw is below:

The second day had seen a whopping sixteen subscribers, though no new search engines had picked up the feed. Even more strange, day three had dropped to only three subscribers, marking an over 80% drop in subscribers.

However, shortly after I posted the third, and final, article, the subscriber count more than doubled, reaching eight. Though still a marked drop from the day before, it showed that the feed subscribers were prompted by my posting and not creation of the blog.

To test this, I then let the blog lapse for a few days. Very quickly, the feed subscriber count completely flat-lined, reaching zero.

This does not follow a “human” pattern. Though feed counts rise and fall, as anyone at FeedBurner will tell you, but they do not follow this pattern. This is, almost certainly, the work of bots, both good and bad.

The outcome is pretty damming, it is obvious that there are at least some scrapers waiting for your site from the very first post and that being an unknown blogger is no protection against RSS abuse.

Problems with the Study

This isn’t to say that there aren’t problems with the study. There are several.

First and foremost, the study, by itself, means nothing. It is just one site on one service and on one topic. A more complete study would try more blogs on a variety of topics and services.

But a bigger problem is that I can not account for all of the subscribers of the feed. I have done several searches for the scraper sites but have had no luck in locating them. Odds are, it is simply too early for them to have been picked up by the search engines. Even the original site is only in Technorati and Google as of this writing.

Also, with most search engines running spam filters, it is very likely that they would catch the scraped blogs before they were indexed. In fact, I have a feeling that several have even determined that the original blog is spam, which it technically is, and refused to index it as well, thus why Icerocket, Sphere, Yahoo! and others have not added the original either.

But even if we consider all of the legitimate blog search engines that would likely be looking at the feed, it doesn’t account for all of the subscribers.

That fact is further supported by the fact that WordPress could not identify most of the readers of the feed and, most that it did identify, were deemed “Web browsers”.

Also, as you can see in the image below, traffic to the site itself never reached anywhere near traffic to the feed (save today where I’ve been visiting the site). Most search engines, including Technorati, also visit the site when indexing the feed to ensure they get the full post (it is a way to guard against partial feeds limiting blog search engines).

If the uses of the feed were legitimate, then traffic to the site would, initially, either meet or exceed the traffic to the feed. With no long-term subscribers that may not visit the site every day, the fact that over a dozen “subscribers” accessed the feed without accessing the site is very suspicious.

Conclusions

Though it is hard to draw any solid conclusions from this study, there are at least three things that are obvious:

1. Suspicious use of a feed begins, literally, with the first post.
2. Being an unknown blogger is no defense against scraping.
3. Spammers are basing much of their scraping on the notification services most blogs ping. If a pinged post has a keyword they are targeting, it seems that they then visit the feed to grab the content.

What isn’t clear at this time, and likely won’t be until search engines update or drop their spam filtering, is how many of these suspicious visitors were truly scrapers. Almost certainly some were, but but some were also likely pinging services and search engines.

But if simple math is a clue and we believe that most legitimate services would also look at the site, it seems that the vast majority have less than honest intentions.

The bottom line is that something rotten is going on, it is just a matter of how rotten it is.