When it comes to things crawling your site, there are good bots and bad bots. Good bots, like Google’s spider, crawl your site to index it for search engines or provide some other symbiotic use. Others spider your site for more nefarious reasons such as stripping out your content for republishing, downloading whole archives of your site or extracting your images.

The question is simple. How do you block the bad bots while welcoming the good ones in? Fortunately there is a standard, robots.txt, that can do just that.

However, editing your robots.txt file can be a daunting task, even for experienced Webmasters, as it requires mastering a very specific and somewhat unusual format. To make matters worse, even those who understand how such files work rarely know the names of all the bots to let in or block, limiting what they can do with it.

Fortunately, there is both a WordPress plugin and a robots.txt generator that can help. What is unclear, however, is if they will actually work.

A Quick Word About Robots.txt

The basic idea behind robots.txt is that it is meant to be guide for robots (or spiders are they are often called) when they visit your site. It tells the bot the pages and directories they can and can not visit and can either set up blanket instructions for all bots or instructions for specific bots.

This is done by, appropriately, including a file titled robots.txt in the root of your server. For example, Plagiarism Today’s (deliberately open) robots.txt file can be found at http://www.plagiarismtoday.com/robots.txt.

Since robots.txt makes it possible to filter bots by their identifier, it can be used to let only certain spiders into your site while keeping others out. The problem is that there are, quite literally, hundreds of different spiders out there and new ones being written all the time. It is almost impossible to keep on top of what spiders you should allow and those that you should banish.

Fortunately, there are two tools that may be able to help you do exactly that by giving you the information you need to build a robots.txt file that’s capable of keeping at least some of the bad bots at bay.

Building a Better Robots.txt

To help you build an effective robots.txt file, there are two tools you may wish to look at:

1. Robots.txt WordPress Plugin: Written by Peter Coughlin, this WordPress plugin largely automates the process of building a robots.txt file that can keep bad bots out while letting in the good ones. Though largely hands-free, you can tweak your own robots.txt file through this plugin, eliminating the need to access it via FTP. Also, the plugin is careful not to overwrite any existing file and will uninstall gracefully if requested.
2. Robots.txt Bulder: Provided by David Naylor, this robots.txt builder lets you select from categories of bots, including search engines, archivers and, of course, “bad robots” and decide the level of access each group has. When done, simply paste the results from the builder into your robots.txt file and upload it to your server.

The two tools are actually related as Coughlin’s WordPress plugin actually uses Naylor’s list of bad bots to help it determine which bots to keep out. So, in that regard, the WordPress plugin is essential an automated install of Naylor’s list for WordPress users.

This also means that both methods function largely the same way, by compiling a list of bad bots and, using robots.txt, telling them to keep out while throwing the doors wide open to other kinds of spiders that may wish to crawl your site.

However, it is unclear exactly how effective this method is and the reason is the nature of the bad bots themselves.

Limitations and Concerns

The problem with using robots.txt in this way is that, for it to work, it requires the cooperation of the robots themselves. Robots.txt doesn’t do anything to actually restrict the bots from accessing various parts of your site, merely tell them where you wish to allow them to go or not go. In short, if a bot wants to ignore your robots.txt guidelines, they can.

Legitimate bots obey robots.txt for a variety of legal and ethical reasons. Googlebot, for example, will always adhere to your robots.txt rules. Bad bots, however, are free to ignore them and often do.

This doesn’t mean using robots.txt to block bad bots is completely ineffective. The reason is that many “bad” bots are actually good or neutral ones used for bad reasons. For example, there are many download spiders that can be used for good or bad reasons but, unless you have a specific reason to allow them, you should probably block them as they serve little positive use for you. Those will, by in large, obey robots.txt.

Also, many bad bots will obey robots.txt simply because so few sites bother to block them and there is little reason for them not to. Since not following the standard can open up potential legal challenges against them, including potential copyright issues and tresspass to chattels, though such a theory presents its own dangers and challenges. there is great risk in not following them and little to gain by ignoring them.

There are still other ways around this. Some nefarious bots will change their name, even randomize it to avoid being targeted by robots.txt (instead gaining the rights granted all other bots not specifically listed, which is usually more liberal) but most that don’t want to obey robots.txt will simply not do so and unless the server takes some additional effort to block them, such as using an IP or user-agent block, both of which can also be mitigated against, there isn’t much to stop them.

One other possibility is to use cloaking to trick scrapers into grabbing the wrong content, but that possibility also carries its own risks with it.

Bottom Line

Given the fact that there is very little risk with manipulating your robots.txt file to block bad bots, there is very little reason to not do it. It may not block all or even most, but it can block some and that can make it of some small benefit.

The problem is that, if more webmasters start blocking bad bots via robots.txt, the method will become less effective as more and more decide the risk of ignoring it is worthwhile. However, that would require a very large number engage in the practice, which is unlikely considering that many sites don’t even have control over their robots.txt file.

In short, if you can do it, you probably should. It’s an extra layer of protection against those who might wish to misuse your site or your content. It is not much, but certainly every little bit helps.

The quiz seemed to generate some interest, just not the kind I had expected. A few people wrote me to ask if one of the questions was possible and, if so, how to do it. Still the, first poster, Andy Murdoch of MMMeeja, a very neat looking Web design and development company, was the one to get the questions right.

So what were the correct answers and why? We’re going to take a look at the questions one at a time to see what the right answer was and

The Answers

What are three ways you can request your site be removed from Google search? (Note: This is your own site or a page on it, not an infringer)

Not surprisingly, there was more than three correct answers. The most common ways involve using robots.txt or meta tags. But you can also use Google Webmaster tools to remove the site or you can direct your server to respond to Google with an error code.

There are other ways, but those are the most common.

What is the email address for the Google DMCA agent? (please only include the part before the @).

The answer is DMCA-agent. This was actually something of a trick question. Google recently changed its DMCA information, the former address was amac. However, since I’ve mentioned the previous address on this site and it appears to still work, I would have accepted it as an answer.

What, according to Matt Cutts, is the most effective way to report spam to Google?

The answer was right here on Plagiarism Today. Matt Cutts has said that spam reports filed through Google Webmaster Tools are given much more weight than those files through other means.

What is the name of the Google service that will email you search results as they are picked up by Google?

Google Alerts. I’ve mentioned this service many times on this site and it remains one of their best anti-plagiarism/copyright infringement tools.

What is the name of the Google spider, or rather, the name you need to refer to it as when you are trying to block it?

Googlebot. This one was pretty simple actually and you can find it on any number of sites.

What is the meta tag command to prevent Google from displaying a sample of your content in their results pages?

This was the one that seemed to pique people’s interest. I got a couple of emails asking me if this was even possible. It’s actually pretty simple. The meta tag command “NOSNIPPET” will prevent Google from displaying the snippet in their results pages.

You can read about this and other neat Google Meta Tags here.

What search command lets you see approximately how many pages Google has indexed on a site?

site:yourdomain.com. Important to note that this method is rather dubious in its effectiveness. The results tend to differ wildly from search to search but can still give you a rough idea of the range of indexed pages.

Conclusions

All in all, most of the questions in this quiz were pretty basic. Outside of the Matt Cutts and the NOSNIPPET questions, most of the questions could be answered any number of places.

The goal was to get people thinking about how Google users their content and what tools they are provided with to control that.

I hope everyone enjoyed this quiz and look forward to another one in the near future!

Special Thanks: To Red Vs. Blue for the CTRL+ALT+BINGO Joke. Too good not to reuse…

These days, even death does not put an end to content theft. It merely opens up new avenues for it.

As a recent article on Blue Hat SEO pointed out, nothing is ever really deleted from the Web.

Caching sites and archives hold on to your content long after the page has been removed and, as the article demonstrates, anything that is available can be scraped.

It is the equivalent of picking a dead man’s pocket, but it is a type of plagiarism that can and does happen. It is also a kind of plagiarism that raises a whole slew of new questions and concerns.

How it is Done

The process for scraping a dead site is surprisingly simple, involving only five steps.

1. Visit an archiving site such as The Internet Archive
2. Lookup an old site that you know to be deceased
3. Find an old, but still relevant, article
4. Check to make sure that the article has not been posted elsewhere
5. Copy/paste it onto your site

As the original article points out, there are ways to automate this process, such as using sitemaps, but the process clearly works best by hand. With that in mind, this method is unlikely to be used by professional spam bloggers, who need more content than this method could provide, but could be favored by human-powered spam sites that are trying to appear legit.

To those Webmasters, the unethical ones catering to humans and the search engine bots, this type of content theft can seem like a dream come true, especially when one weighs the advantages that come with it.

The Advantages for Scrapers

When looking at why a scraper or a plagiarist would find this kind of content appealing, consider the following:

1. Since the content has been removed from the search engines, there is no possible duplicate content penalty and no original site to compete with in the search engine rankings
2. If the original site is shut down, most likely, the people charged with protecting the content are no longer looking for infringements. Thus, the odds of finding legal trouble are slim to none.
3. Finally, there is a wealth of dead information out there, ready to be copied. Though the Web is growing, sites are also dying every day, thus ensuring a steady stream flowing into a large, growing pool.

Of course, taking advantage of such a system requires a certain amount of cleverness and work ethic, things that are in short supply for most that would steal content, but if one sees the advantages as outweighing the drawbacks, it would still be an appealing option.

However, given the wide range of legitimate, easily-accessed content available to scrape, it seems unlikely that most infringers would take this route.

Despite that, there is little doubt that at least some do scrape dead sites for at least part of their content and that fact raises some difficult questions that don’t come with easy answers.

Questions to Ponder

Obviously, given the current length of copyright law, any such scraping would likely be an infringement. Unless the work was placed under a Creative Commons license or donated to the public domain, the work is almost certainly still protected and the copying is illegal.

However, with the business interest in protecting those works gone, any copyright is likely to go unenforced. Most won’t care to check for any infringement and those who do will likely be unaware of the potential danger. Most feel that, when the site is removed, the risk of plagiarism disappears.

Worse still, the authors of the work, who often created the works as a work for hire and hold no copyright interest in it, would be the ones holding the greatest continued stake in the work. Though their employers have long since closed up shop, their name and reputation remains affixed to the works.

But even if the copyright holder does detect such post-mortem plagiarism and expresses an interest in defending those works, going through the motions could be difficult. Though the DMCA does not state that URLs must be provided with a DMCA notice, some hosts have that requirement and it is the most common way of identifying the original works. With the site down, that element could prove difficult.

Furthermore, since there would be no economic damages and it is unlikely that the work was registered, suing for said infringement would be almost financially impossible. In short, though stopping such plagiarism would be possible, it would be more difficult and less worth the time spent.

It may not be the perfect crime, but it is certainly pretty close. It’s very hard to envision a scraper paying for such an infringement, no matter how unethical or illegal it is.

Prevention is the Key

Since enforcing the copyrights of a dead site is, in many cases, impractical, the best approach is to look at preventing the infringement from happening in the first place. Doing that involves having a sound shut-down strategy that includes the following steps:

1. Handle all existing plagiarism cases: Deal with all ongoing plagiarism and content theft cases that you can. Ensure that the content is removed completely before beginning to take content offline.
2. Block known archiving sites: Edit your robots.txt file to exclude the Internet Archive and other archiving services. All cached copies of your site should be removed within a few days. (Note: You can skip this step in favor of step three but it is worth being certain that these archiving sites drop your cached copies.)
3. Block All Search Engines: Since search engine caches can remain active for some time after a site goes down, once you are certain the archive sites have removed the work, edit the robots.txt file to exclude all search engines. You can also use meta tags to prevent indexing, or both to ensure that all spiders stop indexing your site.
4. Move Content to a Hidden Location: Move a copy of your site to a hidden, but accessible, location. Consider making the site password protected to ensure that it can not be indexed or visited by anyone you do not personally authorize.
5. Remove the Original Site: Take down the site and close up shop, being certain to leave behind a means of contact in the event a problem is discovered later.

Though the process seems lengthy and difficult, it can easily be done over a couple of days with most of the time spent waiting on the spiders to re-index your site. Once they do, the cached copies should be dropped almost immediately.

The key thing in all of it is to ensure that the cached copies of your work are removed BEFORE you take down the site. As long as you ensure that the major caching services, especially the Internet Archive, have removed your work before you shut down, you can greatly reduce the risk of post-mortem plagiarism.

Conclusions

The good news is that, though there is no way to tell how common this kind of content theft is, it almost certainly is very rare. In most cases, the meager rewards do not justify the effort required. Also, when it does happen, it will largely limited to the larger, better-known sites that are now defunct as it requires at least some advanced knowledge of the site’s existence.

The bottom line is that the content on any Web site, even a closed one, still has value. This is especially true if the rightsholder is looking to start up another venture in the future or is pursuing off-line methods of distribution. The fact that the site is down does not mean it is acceptable to exploit the effort and expense of the original author.

Perhaps the strangest side effect of this is the damage it does to the classic classic scraper mantra “If you don’t want your content scraped, get off the Web”. If shutting down a site does not put a definite end to content theft, then dealing with plagiarism on a live sight becomes an even more practical solution.

In the end, it seems that the only sure fire way to avoid scraping is to not put it on the Web in the first place. As it sits right now, once the content is on the Web, there is no way to erase it.