As such, it’s important for all webmasters and content creators to be aware of what RSS scraping is, how it works and where it’s going in the future. Even though RSS as a protocol may be on the ropes, RSS scraping is not a problem that’s going away and, in fact, may be getting a lot worse in the coming years.

With that in mind, here is a quick FAQ on some of the more common questions asked about RSS scraping and what can be done about it.

What is RSS?

RSS, sometimes referred to as Really Simple Syndication or Rich Site Summary, is a protocol that makes it easy for other sites and tools to access the content in your site by formatting your content in a consistent, easy-to-parse way.

Contrary to an HTML document, which could have the content be anywhere on the page, RSS indicates clearly what is the headline, body and other elements of the content. This makes it easy to grab the content and display it elsewhere without the surrounding formatting and HTML code.

How is RSS Normally Used?

Traditionally, RSS has been used to enable readers to subscribe to a site using various RSS readers such as Google Reader, Feed Demon and even many mail clients.

However, RSS has also been used to power other services, such as email newsletters and even Facebook integration.

What is RSS Scraping?

RSS scraping is when a third party, usually a spammer, grabs the content in an RSS and republishes it wholesale on another site.

In this regard, RSS scrapers work a great deal like Google Reader, grabbing your site’s content and displaying it on a site but, where Google Reader places the content behind a password protected wall that can only be accessed by the subscriber (or those who are shared the individual story), scrapers instead place the content on a public site for anyone to view, including search engines.

Why do People Scrape RSS Feeds?

Spammers seek high rankings in search engines so they can get traffic to display their ads against or sell products with. To do this, they need content but creating content by hand is time-consuming and difficult, especially when much of it is going to make no difference in the search engines.

RSS scraping is an easy way for spammers, and other sites, to quickly fill their pages with content, even if the content comes solely from other sites.

How Can RSS Scraping Hurt Me?

In most cases, RSS scraping doesn’t hurt. Google and other search engines have become savvy enough about spam that most of the time, they don’t give much credence to spam sites, keeping them from getting a lot of traffic or harming you in the rankings.

However, the system is far from perfect and there are many times spammers outrank the sites they scrape from for relevant terms. This is especially true with new sites or those that don’t have a strong search engine presence.

Less likely is that others may confuse the spam site as either being the original site or as being one endorsed by you, thus actively taking traffic from you. Few people, however, make this mistake with spam sites as the distinction is usually very clear.

All in all, the risk from an individual case of RSS scraping is actually fairly low, but the problem is that there is rarely just one or two such scrapers working at any given time.

What Can I Do About RSS Scraping?

Dealing with RSS scraping starts with good SEO practices. If you link between your posts, get good inbound mentions and earn social networking shares, odds are that RSS scraping won’t greatly impact you.

If it does, you can alway seek to have the content removed by either filing a DMCA notice with the spammer’s host or, if that fails, sending one to Google.

If RSS scraping becomes a more serious and more recurring problem, you may want to consider truncating your feeds or eliminating them. Though that would be an extreme last resort.

Is RSS Scraping Illegal?

Some have made arguments that distributing your content via an RSS feed, even if you didn’t realize you were doing it, creates an implied license to use it in this manner. However, there are many problems with that and other related arguments on RSS scraping.

Generally, RSS scraping is considered to be copyright infringement, though there are other legal arguments against RSS scraping as well.

What if I Want to Encourage RSS Scraping and Reuse

If you want others to scrape your RSS feed, you can actually give blanket permission to do that by inserting a Creative Commons license into your feed. This will let bots that do scraping know your intentions and, those that are complying with the law should be able to follow your wishes.

How Can I Track RSS Scraping?

Many people will find RSS scrapers on accident when they search for keywords relevent to their blog or site. However, you can keep track of your content using automated tools like Fairshare that are designed for tracking dynamic content.

In the end though, its best to keep an eye on the search engines for terms that others commonly find your site through as scrapers will often show up for those same results though, initially, they will likely be lower than your site.

What is the Future of RSS Scraping

Though it’s difficult to predict what spam tactics will be popular in the coming years, RSS scraping has been a problem for at least six years and is continuing today.

That being said, it has fallen out of favor with many spammers, who prefer content generation or scraping excerpts from feeds to avoid duplicate content penalties in the search engines. Still, many active spammers use the method though spammers have clearly become more diversified in this area.

Bottom Line

There’s no doubt that RSS scraping can be and often is very annoying and very problematic. That being said, there’s no reason that it should be a major headache or that it should become a reason to walk away from your site. Most cases of RSS scraping don’t have a major impact on a blog and those that do can usually be dealt with.

That being said, if you are having a serious problem with RSS scraping, please feel free to drop me a line or, if you think you may need outside help, feel free to see if I can help via my consulting services.

All in all, RSS scraping is a reality most bloggers and webmasters will have to deal with, but it’s not one that should sink your site if you’re savvy about how to handle it.

The policy change, will remove all of the automated content posting features from free user accounts, which make up the “vast majority” of WPD members, according to Marty Rozmanith, the creator of WPD.

The tools, however, will remain available for all paid members of the service, regardless of the level they choose.

Previously, unpaid members had limited access to some of the content posting tools, including the Yahoo! Answers, article database and RSS posting tool, enabling free members, who were limited to only three blogs, to automatically post content from a variety of sources, typically without permission.

Whether this does anything to stem the vitriol that has been directed at the service remains to be seen, but I can’t see how many will be convinced, especially when there are so many difficult questions to be answered.

WordPressDirect Recap

For those who did not read the previous articles about WPD, the service promotes itself as a “WordPress deployment and maintenance service that helps people especially those with very little technical experience) create a search-optimized WordPress blog.”

In short, it is a one-click install program that not only sets up the software, but also adds a theme, optimizes the permanlinks and makes a handful of other SEO-oriented changes. In that regard, it is much like Fantastico, but with added features to help get the blog started.

However, WordPressDirect stepped into controversy with its add-on tools, which allow users to automatically update the blogs they create using content from a variety of sources including RSS feeds, online article databases and more.

This caused many, especially on the Mashable article, to accuse the site of being a spam service. In that regard, it does share many traits, especially when you look at how the tools work and where they pull their content from.

WordPressDirect attempted to defend itself against the accusations, blaming much of the problem on their marketing, but the attempts to make peace fell on deaf ears for the most part. This, in turn, led to the recent changes they just announced.

Fixing the Problem?

Most likely, these changes are going to do little to nothing to placate the mob that has formed around WordPressDirect. Though the changes mean that the 9000+ free members of the site will no longer have the ability to automatically scrape and repost content, it says nothing about the paid members. The limitations on free accounts, including just three blogs per user, effectively meant that no one could actually be a master spammer with a free account (unless they spammed WPD and set up thousands of accounts).

To many, including myself, this sounds like a very shrewd maneuver. Though it removes most of the users from the ability to do spam-like things, it does not affect the paid ones and the email contained several pitches for the paid packages. It seems not like an attempt to shed the spam-related but to profit from it.

This move does not remove these tools from the power users nor does it impact their bottom line in any meaningful way, other than perhaps adding a few new paid members.

WPD, as a service, is walking a very thin line. It is trying to proclaim itself to not be a spam tool while offering many of the exact same features that are found in spam applications. Though, as I said in my Blog Herald article, it would make a very poor spamming program, it is completely foreseeable and almost certain that users, likely even most users, would use it for that purpose.

Furthermore, issues such as the trademark concerns over the use of the WordPress name, the lack of attribution of copied works, etc. remain unaddressed. Though it is a good step, it seems to be one either too small or in an unrelated direction.

Conclusions

Shortly after my Blog Herald article was released, WPD sent out an email to all members saying that it was “most balanced article” he could find.

Though I try to balanced with all of my coverage, I can not hide the fact that WordPressDirect has me very uneasy and nervous. The service has far too much use for evil and, even though I don’t know if its creators built the service with such intentions, that is the use that instantly springs to mind for myself and many others.

The problem is that a service such as what WPD proclaims to be, a WordPress installation aid that auto-optimizes the blog, could be very useful. I could even see someone such as myself using it rather than keeping a WordPress checklist for every new blog I install (I routinely get recruited to help with WP installations).

But as useful as that could be, the service, is too hot to touch right now and I seriously doubt that is going to change with these recent revisions to their policies. Though I am going to keep an eye open on the marketing changes they mentioned, I don’t see WPD becoming any less of a tainted name anytime soon.

To repair its name, WPD is going to have to make sacrifices that may hurt its business. Sadly, it doesn’t seem to be what they are doing right now.

The results were stunning. According their report, on average the sites that they studied had 140% more views of their content on other sites than the original. This meant that well over half of all views of the content took place on sites other than the creator’s and were unavailable for either monetization or, in many cases, attribution.

Though the results are interesting, they likely are not at all surprising to many who deal with copyright and plagiarism issues on the Web. With human copying, RSS aggregation (both good and bad) and other republication as common as it is, many had already suspected that the audience of a content was much larger off the site than on it. Attributor is simply one of the first to conduct a study that shows it.

However, there are several elements of the study that are interesting beyond the initial findings and may offer clues as to what Webmasters are most at risk of having their content misused.

How it Was Done

According to the report (PDF), they analyzed 100 publishers from the Compete Top 1000, first discarding their existing customers and sites with partial feeds, and added some publishers into the list on top of that to ensure a good mix of different topics.

They then ran the sites through the content matching service and analyzed all of the copies that had higher than a 50% match and more than 125 words the same. After removing known licensees, they looked at the sites that had information on and used traffic estimates from Compete to get an approximate idea of how large the viewership was on these match sites.

After that, they then broke down the results by broad category to see which kinds of publishers had the largest “Audience Multiplier”, meaning viewership of their content on other sites.

The results were that, on average, the publishers tested had nearly 60% of their content viewership on other sites. This leads to missed opportunities both for linkbuilding and for monetization as well as possible causes for removal requests.

This obviously will be of great interest to many Web publishers, who are looking for ways to maximize their audience in the face of an economic slow down, but may not come as a surprise to those that have studied these issues.

However, other findings of the study are potentially even more useful to Webmasters, especially those in high-risk fields.

High-Risk Topics

The study, in addition to looking at publishers in general, broke down their results by content category and the results there were staggering.

Of the sites listed, automotive sites fared the worst. For them, they had nearly seven times the audience on other sites than they did on their own. Travel sites also had a high multiplier, over five times the amount and movie reviews had just under five.

In each of the cases above, the sites have audiences on other parts of the Web that easily dwarf their own traffic, meaning they are experiencing the highest level of unlicensed copying and the sites that are copying them have the highest amount traffic levels.

The topics that fared best were politics and health, both of which had barely over one. However, in both cases, the audience is still larger on the rest of the Web, only in these cases it is by a very small margin.

Why there is such a wide divide between the different topics is very difficult to say. Without the full statistics, which were not available in the report, it could be due to a variety of reasons. Though it seems unlikely that one site would be scraped and republished significantly less than another, especially since nearly all of the topics have a strong spammer following, it could be a sign that copying and pasting has had a higher degree of success in certain categories.

For example, with political sites, most people visit the one or two sites that they trust rather than performing blind searches. However, when automotive problems arise, people tend to put queries into Google and trust the search results. Even though both sites likely have many copies of their content, one is able to rely on their brand loyalty to keep much of their audience close by.

However, this is only a guess, but it is clear that it is time to think about the sites in danger slightly differently. Technology and health were two categories with very low multipliers, though they both have a very high tendency to attract spammers.

Some Caveats

It is worth noting that the study, while useful, should not be considered scientifically valid. The sampling size is too small and the traffic statistics, though likely about as good as possible, leaves room for error.

It is very difficult to imagine a more thorough study being performed without the backing of a major university, but any study in this area is likely to face similar challenges and limitations.

The other element is that this study focuses on large publishers and not regular bloggers. Whether this means that bloggers would have a much higher audience multiplier due to their smaller initial audience or a smaller one due to less copying and scraping remains to be seen.

Though, likely, the results on the different categories of content and their relative risk may transfer well to smaller publishers, a separate study would likely be needed for smaller bloggers to see how their audience compares.

Still, the purpose of the study is not to necessarily achieve these goals, but to illustrate the possibility of a much larger audience outside of the original site. This is something many have suspected but, to my knowledge, this is the first study to attempt to discuss the issue.

Conclusions

The bottom line is simple, most likely the audience for your content is much larger off your site than it is on it. What you do about that is completely up to you. Whether you attempt to monetize it, turn it into promotion or request removal of it (or a combination of all three) is up to the individual author and the course they want to take.

No matter what though, it is clear that this audience and its potential (both for harm and for good) is too big to ignore and it is important to start tracking and understanding what is going on. Whether it is through a professional service such as Attributor, one targeted at individuals such as Copyscape or even simple Google searches, the time to understand these uses is now.

What the Attributor study illustrates, more than anything, is the need for an even deeper understanding of how this copying takes place, what it means for publishers and what strategies could they use to maximize their benefit from it.

This is an area ripe for exploration moving forward and one that will require a great deal of creativity and work.

Disclosure: I work as a consultant for Attributor.

So, with that in mind, I’m taking a look at the year ahead with seven new predictions about what is in store for us in 2008 when it comes to content theft and copyright issues.

Hopefully these predictions will strike a good balance between the dead obvious and the completely insane as I try to figure out what is coming up in this clearly unpredictable field.

Spinning Spam Increases

I made this prediction a few days ago on the Blog Herald but it is worth repeating here.

Spammers are facing increased competition for search engine results from both legitimate sites and other spammers. Couple that with duplicate content penalties, copyright issues and smarter search engines, and innovation is essential for their survival.

One of the forms that innovation will likely take is the form of spinning content, through a combination of synonymizing and translating. This type of content theft is harder to detect and will require new techniques to detect and stop.

Expect more on this here in the coming weeks.

New Technologies Change the Game

Another repeat from the aforementioned Blog Herald article, but yet another important one. Companies, such as Attributor, are poised to really change the game by providing powerful, easy to use tools that can not only improves the effectiveness of content theft fighting, but makes it accessible to a broader audience.

However, even if Attributor doesn’t work out, other companies, some of whom are stealth, will likely be coming onto the scene this year and could have similar results.

In short, this is going to be the year in which companies realize the potential market in helping bloggers protect their content and start to exploit it.

False DMCA Notices

Hopefully I am allowed to repeat a prediction from last year as this one is almost a given. With thousands and thousands of DMCA notices being hurled each day, it is a certainty that some will be off target.

Expect several more DMCA notice controversies over the next year, the perfect storm of rabid copyright holders, remix culture and lawsuit-frightened hosts is just too much to ignore.

RIAA Loses Traction

Though 2007 was a bad year for both the RIAA and the music industry, 2008 will likely be even worse. This may well be the year that the record industry starts to distance itself from its boogeyman and the RIAA will start to loose support even among its members.

Some of this has already happened with EMI planning on slashing funding, but with the record labels turning against DRM, I doubt they will be the only ones.

No matter the wins or losses they receive in the courtrooms, expect the RIAA to leave 2008 even weaker than it went in.

Perez Hilton Loses

Though I honestly didn’t expect these lawsuits to reach 2008 but, since they have, it seems likely that they will come to some kind of a conclusion this year. If they do, expect Perez Hilton to suffer a few grave setbacks.

Granted, the case will probably not be over in its entirety in the next 365 days, appeals and so forth can take many years, but the writing should be on the wall for Hilton well before the ball drops on 2008.

Anything else would be the product of either a stunning upset, or an unbearable delay.

Old Media Joins the Fun

Though the record and movie studios have been fighting to protect their content on the Web for years now, old media, namely magazines and newspapers, have not been extremely active in this field.

Expect that to change in the coming year but, unlike RIAA, expect them to better understand how to leverage the Web. I suspect that their delay in entering the fray has taught them a great deal and, looking at the New York Times shift away from paid content, it seems likely that newspapers “get it” at least a little bit more.

Perhaps this is a case of wishful thinking, but it isn’t without at least some grounding in reality.

Image Search Grows Up

Image search, especially when dealing with content theft, is an ineffective black art. I am hoping, perhaps against my better judgment, that 2008 will be the year that image search grows up and we can actually detect duplicate images on the Web even if the file has been renamed, has had the EXIF data changed or has been altered.

This is easily my farthest step out on the limb as I have little reason to believe this, but the market here is just too big to ignore and. with more and more photography winding up on the Web and so much content theft/plagiarism in this area, image protection is easily the next big market for protection.

That is, if it isn’t already the current big market.

Conclusions

As far as copyright issues go, 2007 was a wild year and 2008 is poised to be even more crazy.

However, no matter what happens in 2008, or how wrong these predictions turn out, stay tuned to this site as I will do my best to keep you up to date on what is going on and, hopefully, share some useful information along the way.

Thank you all for making 2007 such a great year for Plagiarism Today and I am looking forward to a very exciting 2008.

Disclosure: I am a consultant for Attributor.

I recently ran across this link on the social news site Reddit. It is a submissions listing for the user “lecoq”, who has submitted hundreds of entries from various Blogspot Blogs. All of the entries that I checked contained content found first on other sites and all were surrounded by different ad units, usually from several different ad networks.

The end result is that these blogs appear to be nothing more than the traditional definition of spam. However, the list, which appeared on the front page of Reddit a few days ago, has not resulted in any removals by Google.

It does appear though, at this moment, that this is the work of a human and not of an automated scraper. There seems to only be about six spam blogs involved and they pull from a variety of sources.

Though this definitely does not follow the typical format for a spam blogger, it is worrisome that, even after all of the attention this list received, that almost none of the blogs have been shut down. It is obvious, given the submission pattern, that the intent here is to spam social news sites with copied content, and spam is a violation of Google’s Blogger Content TOS.

Hopefully, the offensive is ongoing and this is just a hiccup in the system, probably caused by the human element in the posting, and that Google will address it soon. In the meantime, I’m going to keep an eye on this list and see about contacting some of the content owners to alert them of the misuse.

I encourage others to do the same.

Generally, however, that power is best left unused. For the most part, restricting people’s access to your site is a bad move. Though you can use your powers to carve out a members-0nly area or prevent others from accessing administrative areas of the site, turning people away from the door is usually unwise.

Still, there are some people that you want to keep out. RSS scrapers and image hotlinkers, for example, offer nothing to your site but instead only steal your content, your bandwidth and your other resources. If you can prevent them from accessing your site in the first place, without impacting other users, it is probably in your best interest to do so.

Fortunately, with Apache’s .htaccess file, it is possible to do all of those things and more. All one has to do is understand a few basics and get the code that they need.

A Quick Primer

According to the Apache Software Foundation, .htaccess is a distributed configuration file that provides “a way to make configuration changes on a per-directory basis”. It is most commonly used when a Webmaster has access to the server, but not the core configuration files for that server. This is typical of most shared hosting environments.

When editing an .htaccess file, there are three important things to remember:

1. .htaccess is the name of the file: In short, htaccess is the extension and there is no file name. This can make editing the file difficult on some computers, but it is important that the convention be followed. If needed, name the file something else and rename it after uploading it to your server.
2. It is an ASCII file: .htaccess is a plain text file and should only be edited in a text editor such as Notepad.
3. It only works with Apache: Though other servers, such as Microsoft’s IIS Server, offer similar features. .htaccess itself is only for Apache-based servers. If you are unsure of what kind of server you have, check with your hosting provider.

Finally, it is important, when working with .htaccess, to back up well and be careful with your edits. A poorly-constructed .htaccess file can render your site useless.

But despite these warnings, .htaccess files are, generally, very easy to edit and manipulate. Furthermore, there is a lot of very good free code ready for you to copy, paste and manipulate to fit your needs.

Stop Image/File Hotlinking

One of the easiest and most basic tasks that can be performed with .htaccess is stopping image/file hotlinking. This is the process by which other sites link directly to your files, either having them display or download directly from their site. This not only amounts to content theft, as well as often plagiarism, but also bandwidth theft as your server spends the resources to serve the file everyone someone on their site calls for it.

According to Zann Marketing, the process is very simple. All one has to do is navigate to their images folder and either create a new .htaccess file or add the following code to their existing one:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yoursite.com.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yoursite.com.*$ [NC]
ReWriteRule .*\.(gif|jpg|png)$ – [F]

The first line tells the server to turn the Rewrite engine on, the second line instructs the server to check and see if the referrer is blank, the third and fourth line check to make sure that is not from your own site and the fifth line instructs the server to disallow the request for the selected file types if none of the above statements are true.

With this code, you can easily modify it several different ways including:

1. Add New Domains: You can add new domains and sites to allow hotlinking from. The original example from Zann Marketing includes the IP address for Google Images, for example. You can include other search engines as well.
2. Add New File Types: By editing the last line, you can modify your rules to include any kind of file necessary including movie files, documents and anything else you wish to have protected from hotlinking.
3. Disable Access to Blank Referrers: By removing the second line, you can prevent access from browsers and tools that return a blank referrer. Though some scrapers and black hat spiders do this, so do many visitors in a bid to protect their privacy.

Though this method will not stop people from saving your images to their hard drive and uploading it where they please, it can prevent people from stealing both your image and your bandwidth at the same time.

Also, on the original Zann Marketing page, there are examples for blocking just one hotlinker and to redirect hotlinkers to another image, thus pulling the famous “switcheroo”.

Finally, if the process of editing the code seems too daunting, you can also use HTML Basix’s .htaccess code generator to create an .htaccess code set for you to copy and paste into your file.

Blocking RSS Scraping

Equally easy, or in some cases even easier, than blocking hotlinking is blocking RSS scrapers. All you need to do so is determine the IP address of the RSS scraper. (Note: You can use domains if you wish. However, since not all scraping software is located on the domain itself, IP addresses are more reliable).

The easiest way to determine the IP address of a scraper is to use the Copyfeed plugin and have it place the IP address in the scraped content. This not only eliminates the need to translate domain names into IP addresses, but also works in case where the scraping software is located on another server or computer.

However, if that fails or is not an option and the scraped site is hosted on its own domain, you can simply use the IP address for the server itself. To determine the IP address for a domain, simply enter it into a site like Domain Tools and let it get the IP for you. It only takes a few seconds.

If the scraper is using a free service such as Blogspot, you will likely have to look into your server logs and attempt to find traffic on the feed that times out with when the posts go up on the scraper site. It is a risky task to undertake as you can accidentally block legitimate users and it can be very time-consuming on larger sites, but it is the only option in some cases if you wish to use blocking techniques.

No matter what method you use, once you have the IP address, all that is required, according to JavascriptKit, is the following code in the .htaccess file of your feed’s directory:

order allow,deny
deny from xxx.xxx.xxx.xxx
allow from all

Editing the code is easy, all you have to do is replace the Xs with the IP address of the scraper. You can add more lines to as new scrapers emerge and you can also use wildcards by leaving off numbers. For example 123.123.123., would block all IP addresses that start with 123.123.123. This can be useful if a scraper has an IP that changes, but only within a certain range.

It is important to note that this code will block ALL access to your site for that IP address. However, there is very little reason to allow a scraper onto your site as, most likely, they are only accessing the feed anyway.

Also, if you want to redirect scrapers to a fake feed, you can use the method discussed on Hung Truong, which often generates very humorous results.

Finally, once again, if you are uneasy with editing the files yourself, HTML Basix offers another .htaccess generator, this one to block users. It is also useful to stop RSS scraping.

The bottom line is that, once you obtain the IP address of the scraper, it is trivial to block them using .htaccess. All you need is a little bit of understanding and some freely-available code.

Conclusions

Though .htaccess editing can seem very intimidating to a novice, it is actually very easy to do. With the proper tools and a few fundamentals, anyone can manipulate their .htaccess files and use it to their advantage.

Though these manipulations won’t do anything to stop human plagiarism it can stop some of the more common types of plagiarism before they happen, all without impacting legitimate users at all. It makes sense, if possible, to use these methods to your advantage.

However, it is important to note that you are unlikely to find a free host that allows manipulation of .htaccess files. This is, predominantly, the feature of paid hosting companies. Also, it will not work if your images and RSS feeds are on another server, such as Flickr or FeedBurner.

But if you’ve paid for your hosting, it makes sense to use the tools that come with that kind of an upgrade. One of those is the ability to protect your content at the server level.

It is a great power and one that is sorely underused on the Web.

As with most replies from Google about such matters, it was very vague and short on details. However, it did state the following:

“I can’t really comment on anything further than what you’ve noticed, but you’re right: 1) some spammy blogs are showing the Terms of Service warning; and 2) this is a good thing.”

He went on to say that many spam blogs, possibly most, are currently being shut down even before the search engines have a chance to crawl them.

Though I will have to continue to dig for more information. It is nice to confirm that Google is taking strong action and that, perhaps for the first time, it seems to be fairly effective. We will have to see how and if spam bloggers change their game and, if they do, how Google will respond.

The cat and mouse game, most likely, has just begun.

However, looking at the Google results, you’d never know that. The top search result belongs not to Now Smell This, but a fashion site called Stylefeeder (nofollowed).

Now Smell This is at the very bottom of the front page, tenth over all.

But if you click the Stylefeeder link, something strange happens, you get taken not to Stylefeeder’s site, but rather, to Now Smell This. Though the domain in Google clearly reads Stylefeeder.com, you land on Now Smell This’ Ralph Rocks page.

A simple look at the source code of the Stylefeeder page reveals the problem, it’s not a page at all. It’s a redirect. What happened is that Now Smell This has fallen victim to an almost ancient form of search engine spam, the 302 referrer hijack.

It’s an old threat, but as this case proves it is still around and it is a way for a spammer to steal your content and your ranking without ever copying a single word.

Hijacking 101

The 302 hijack is actually pretty straightforward. As Claus Schmidt explains in his paper on the subject, the 302 redirect is supposed to be used to temporary redirect users and search engines to a new site.

To a search engine or a browser, it is a way of saying that the content you seek is no longer here, but that it is, for the moment, at this new link. However, since the redirect is temporary, search engines hang on to the original link as it may change. Google, generally, continues to spider and index the 302 page in case it changes or redirects elsewhere.

The problem is that, since no page actually exists for the 302 referral, it is not a page but simply a script, search engines, in some cases at least, index the content from the new site and attribute it to the non-existent 302 page. This appears to be what happened with the “Ralph Rocks” case above.

In short, what happens is this:

1. Google stumbles across the hijacker’s 302 redirect. Interprets it as saying “content over there for now”.
2. Since Google sees that it is a temporary redirect, indexes the page as if it exists and then uses the content from the page it points to index the content.
3. The original page, which is now the target of a 302 redirect, gets less weight with Google since it could, theoretically, change at any time and is supposedly just a temporary home.
4. The redirect page, even though it doesn’t exist, can get moved up in the results for all of the keywords present in the original site.

When it is all said and done, the spammer has tricked the search engine into thinking that the original page is just a temporary site, a stop gap of sorts, and that its non-existent page is the original work.

It’s a devious trick and it gives spammers a means to scrape content without ever copying a single word. In the eyes of the search engines, they completely replace the original work.

Taking the Plane to Cuba

Once the spammer has control of the site’s search engine presence, he or she can do what they want with it.

Where traditional hijacking differs from what is going on with Stylefeeder is that, often times, the spammer will attempt to cloak their real intentions, offering the 302 redirect to the search engines, but sending human visitors to another site altogether.

However, the dangers of this redirect go well beyond mere spam. As Schmidt explains, it can also be used to redirect visitors to adult sites, set up false bank/credit card site or create false storefronts.

It is a very dangerous exploit and it is one that, theoretically, was closed of years ago. However, as cases such as this one and sites such as Google Jacking prove, this problem is still very real and very present.

It’s a scary problem for Webmasters, unlike scraping, which can be blocked, or plagiarism, which can be detected, 302 referral spam can not be blocked effectively and can not be detected until the spammer has already achieved their goal. According to Schmidt, once it has taken place, the damage is done and there is no easy way to claw back out.

That, in turn, makes taking precautions against such attacks very important. Something that is easier said than done.

Precautions

Schmidt goes on to recommend a series of steps that Webmasters can take to guard themselves against this kind of hijacking, they include the following.

1. Redirecting non-www pages
2. Use absolute internal linking on your site (full links with domain names)
3. Have random, updated content on each page
4. Use the “base” meta tag

Most of these precautions are simple to do, especially if you control your own server, and have no real impact on the end user. Thus, it makes sense, even if the potential reward is very small, to take the steps.

Some Good News

The good news is that, though it is clear 302 redirect spam is still a problem, it is also definitely on its way out. Most of the problem was dealt with by the search engines years ago and, though some are still able to exploit it, those who do so successfully seem to be few and far between.

In the case of Now Smell This, it is unlikely that the site in question would have gotten away with it if they had attempted to cloak their intentions. Google, by all accounts, has gotten a great deal better about detecting cloaking and that, in turn, has made this kind of spam much more difficult to execute.

Clearly, the heyday for this kind of spam is over. As devastating as it can be for content owners, it is a less of a concern now than it was a year ago and much less than it was three or four.

This is one area where the search engines have truly gotten smarter, just not smart enough to stamp out the problem completely as of yet.

Conclusions

Page hijacking, at least via 302 redirects, is not the problem it used to be. It’s harder than ever to get away with and it seems the major search engines have all done a decent job keeping it from overrunning their results.

However, the problem is not gone completely. 302 redirects can, and still do, affect search engine results and allow people to use your content against you, even if they never copy a single word.

Though it is not the concern it once was, it is still worthwhile to take a few simple precautions to prevent it from becoming a much larger issue. After all, only one site has to be successful with it to severely impact your own search engine ranking for a keyword and only a few have to be successful before your entire domain suffers.

But the real onus to stop this problem, as Schmidt points out, lies not with Webmasters but search engines. Though we can take precautions to help search engines tell the difference between a real redirect and a spam one, if the search engines don’t pick up on that, there is nothing that can be done.

Smarter search engines, ones not easily fooled by simple tricks, will be the real solution. Then again, that’s the same thing people have been saying about spam blogs for years and that problem has not gone away either.

Note: I attempted to contact Stylefeeder for this article and did not hear back in time to go to press. If I do hear from them, I will update this page.

The goal is to not simply help bloggers discover if and where their content is being scraped, but also give them information to help them track down the scraper and, when used in conjunction with plugins such as Antileech, stop the infringement outright.

It is a potentially exciting plugin that could replace a other extensions that are currently available, effectively combining the functionality of at least two indispensable plugins into one, and holds a lot of potential to help WordPress bloggers stop infringement and prevent spam bloggers from taking their content.

How it Works

Copyfeed is something of a swiss army knife when it comes to extending your feed. It has several different functions, many of which are already available.

First, the plugin can add a digital fingerprint to the plugin and provides easy links to search for copies of your work. In that regard, it works a great deal like MaxPower’s Digital Fingerprint Plugin and can serve much of the same purpose.

Second, the plugin adds a customized feed copyright notice, similar to Angsuman’s Feed Copyrighter Plugin. This plugin, however, takes it a step farther making it easier to add your custom copyright information and then and even making it possible to add HTML to the footer.

However, where the plugin goes above and beyond the currently available tools is by adding the ability to include the IP address of the feed reader in the feed itself. This means that, if the feed is scraped, the scraper will also be publishing the IP address of the computer they used to scrape the feed, making it very easy to either block the address, or input it into Antileech for redirection.

Finally, Copyfeed also offers the ability to whitelist certain domains from seeing the above information. This means that it is would be possible to protect your regular feed, but allow BlogBurst or another desired syndication service to bypass that content without having to point them to a separate feed.

It’s a very powerful set of features that, when combined with Antileech, can provide almost total protection of the feed from scrapers and spammers. All of this also says nothing about the other attributes that Copyfeed can add, including comments and related posts, that have nothing to do with content theft preventions.

Some Problems

However, this isn’t to say that the plugin is perfect, there are several issues with it that may stop it, in some cases at least, from being ready for full use.

First off, both the plugin and the site are currently only available in German. Though you can use tools to translate the site, the plugin administration panel rests in the admin area of the WordPress install and can’t be easily viewed by such tools.

Fortunately, Bueltge has promised to include a translated version of the plugin with his next release, along with “more features” but it has not been released as of this writing.

Second, it is uncertain how legitimate readers will feel about having their IP address printed on their feed’s entries. Some, especially those who are unaware of how the Internet works and what an IP address really is, might cite privacy concerns. Others, especially those who use Web-based readers, might be confused as the IP address would not be theirs, but that of the site they read the feeds on.

It would be up to the user of the feed to explain these issues if they chose to use that feature.

Finally, aspects of the plugin, obviously, will not work well with FeedBurner. Though the copyright notice and the digital fingerprint will both work correctly, the IP feature will not. The plugin would just report the IP address of FeedBurner and not the person reusing the feed, making it useless in that case.

However, there are many people that will still likely find the plugin very useful and many more will do so once the plugin has been translated into other languages.

Conclusions

All in all, this plugin is an impressive effort to stop feed scraping and will almost certainly become an essential tool once translations of it are offered. In fact, it is possible with just a few extra features that it could become something of a one-stop shop for battling feed scrapers in WordPress, becoming almost the only plugin needed.

However, until the translated versions are released, all most of us can do is sit and wait in anticipation for the tool to become available.

It seems almost certain that this plugin, for many WordPress users, will be a nice step forward in protecting their feed. Hopefully, once I am able to use it, it will live up to those expectations.

In the meantime, I’d be very interested in hearing comments from those who are using Copyfeed on their sites. I’d love to hear how it is working out for you.