Defending against scraping, however, is incredibly difficult. Though some plugins and tolls like Bad Behavior for WordPress and simple blocking of bots can help, they aren’t perfect or complete solutions and in some cases, can deeply drain both your time and your site’s resources.

However, the team over at Distil thinks they have found a better way. By acting as an intermediary between the Web and your site, they claim to not only be able to filter out most scrapers and infringers, but also to speed up your site and improve its performance.

How it works is by combining the their anti-scraping and bad bot technology with a robust content delivery network, this enables them to not only filter out threats to your site, but also serve much of your static content quickly and from servers located nearest to your visitors.

But is Distil worth the time and money? I decided to give it a trial and see what I found.

What is Distil?

The closest comparison one can make to Distil is Cloudflare as both use DNS changes to better protect and speed up your site.

With Distil (or Cloudflare) you edit your DNS settings, which can usually be found at your domain registrar or in your site’s control panel, to direct visitors not to your server, but to a custom nameserver from Distil. Visitors will then query Distil for your site, which first filters out any malicious users and then delivers any content it can from its servers, which are spread all across the world. Anything it can’t deliver, it queries from your server and then provides to the user directly.

The end result, if all goes well, is that most of the content of your site is delivered directly from Distil’s servers, which should be faster than coming from your own, and most malicious users, including scrapers, are filtered out before they ever reach your site or your content. Best of all, the process is completely invisible to end users (other than the potential speed increase).

To find out, if it works as advertisers, I switched Plagiarism Today over to Distil last weekend and, as of this writing, have been using it for the better part of a week.

Setting Up and Using Distil

To start using Distil, you have to first sign up for an account and have it activated. Once that’s done, you’ll be given an address that, using your DNS settings, you will direct both your www.domain.com and domain.com (as well as any other subdomains you want to redirect).

Then, after the DNS servers propagate, you should be using Distil’s service. From there, you can log into the Distil dashboard, which lets you configure a variety of options including:

• Site Acceleration Settings (if available)
• Rate Limiting
• Blocking Known Violators
• Blocking Bad User Agents
• Browser Integrity Checks
• Filter By Country
• Block Bad Referrers
• Whitelist/Blacklist
• WWW/Non-WWW Routing

You also get a bevy of statistical data including information about the number of unique sessions, the total number of requests, total human requests and the total bot requests. Bot requests are then further broken down by the number of search engine requests (which are always allowed) and the number of blocked requests (as well as the reasons for being blocked). The blocked bots are then further broken down by bot type, IP address and more.

The result is that you get an overall perspective of what’s going on with your site, both in terms of human traffic but, more directly, the security threats you’re facing.

But does that make Distil worth trying? A lot of it depends on your needs and what you’re looking to get out of it.

The Good of Distil

The one thing that immediately struck me about Distil is the granular level of control it gives you over security issues. Though Cloudflare offers a good deal of site security, it’s focused on spammers and attackers and only lets you set a broad level of security (low, medium, high or basically off). With Distil, you can set individual options to your liking both to target the threats most relevant to your site and, more importantly, make sure you don’t interfere with legitimate users.

Over the past few days I’ve had no reports of legitimate visitors being hassled by Distil, something that was an occasional problem with Cloudflare, especially for visitors from outside the U.S. and Europe.

So, even though Distil did not block as many bots as Cloudflare (likely because I have the security settings for most features turned down or off), it did a better job staying out of the way and still seemed to stop the most egregious offenders. Over time, I plan on slowly increasing the settings to see if they block more and continue to be non-intrusive.

Beyond security, my first concern after switching to Distil was that my site might take a performance hit. Having been a Cloudflare user for many months, I was used to the power of a robust CDN. However, I did a series of tests both before and after the change and found that Distil was usually slightly faster than Cloudflare, often shaving off 30% of the site’s loading time.

Compare these two example results, first before:

And then after:

(Note: While this example isn’t an apples-to-apples test due to differing endpoints, the results were consistent regardless of endpoint. Also, obviously there were other changes made in the four days between the tests, though no major alterations, frontend or back, were made.)

Finally, the support team at Distil is, simply put, the best of any company I’ve worked with. They answered every question I had very promptly, usually within 15 minutes and it didn’t seem to matter what time of the day I was asking it. This enabled me both to get my site set up quickly with Distil despite some confusion and questions and deal with an issue with Google Analytics (that turned out to be my own fault).

All in all, Distil did a good job in providing granular security control, a site performance boost and great support.

The Problems with Distil

The biggest initial problem with Distil is that, in its current form, it is not very simple to use. Not only do you have to wait for your account to be activated by a human, but the process of switching over your DNS is not as straightforward as Cloudflare.

If you aren’t comfortable working with DNS and aren’t familiar with how to edit CNAME and A records, the process is going to be intimidating. Sadly, unlike Cloudflare, there isn’t a great deal of hand holding unless you contact support. While I agree with Distil that’s better to not hand over total DNS control to a third party, as you have to do with Cloudflare, it’s also the much more difficult route for the user.

Another issue I have with Distil is the current pricing structure. The free account, which does not have content acceleration, offers only 5 GB of traffic per month, an amount even a modest blogger will likely blow through quickly. A site Plagiarism Today’s size fits (barely) under the cap for the small account, which offers 50 GB of transfer for $29 per month. However, Cloudflare’s free plan allows for unlimited traffic and it’s pro account, which offers additional statistics and monitoring, is only $20 per month. Other CDNs, such as MaxCDN, charge only $50 for 1 TB (1000 GB) of data.

Distil told me that they are considering restructuring their pricing in the coming weeks, a move that, most likely, will help with this problem.

For now at least, Distil is a terrible deal as CDN though its security features may help to make it more compelling to webmasters concerned about scraping and content misuse.

Finally, Distil, obviously, won’t be able to help with at least some kinds of scraping. RSS scraping likely won’t be blocked unless the bot doing it is already in the system and it is unclear just how many are. However, if you know the bot you can add it yourself in your control panel. Also, any human copying won’t be blocked because the system is designed precisely to allow humans to access your site.

Despite these limitations, there’s still a lot of webmasters who would likely benefit from Distil, even if that number could be a great deal larger down the road.

Bottom Line

Distil isn’t perfect. It’s a new company and it’s product certainly has its share of flaws. Right now, it’s aimed at a fairly niche market of webmasters who are technically savvy, want a great deal of granular control over their site’s security and are willing to pay extra to make it happen.

However, with some changes to its setup procedure, pricing and control panel, it could become a compelling option for many more sites.

In short, Distil is going to be a company to watch in the coming months and years. As it refines its tools and pricing, it could become a major force for helping content creators protect their work.

In the meantime though, other webmasters just wanting a CDN to improve their site’s performance will, most likely, want to look up other solutions, such as Cloudflare and MaxCDN as they are significantly cheaper and, in the case of Cloudflare, provides better analytics, easier setup and at some decent, if simplified, security features.

Still, if you’re in Distil’s niche, which is likely to grow, I can see why it would be a very powerful solution to a complex problem.

Today, after some serious thought, I’ve decided to show exactly how it’s being done and highlight two services that, along with providing a service to end users who are tired of partial RSS feeds, are also helping to feed spammers as well.

These services are RSS expanders, meaning they convert short RSS feeds to full ones. Both of these services are public, free to use and have been in operation for some time. However, more recently, I’ve been seeing them used by spammers as clients come to me confused as to how a site is scraping their full content when they have a partial feed.

To understand how they work and what they mean, we have to take a look at two example services and what they can do.

What is a Feed Expander

The idea behind a feed expander is fairly simple, it takes a short RSS feed, one that either has truncated content only or is headline-only, and converts that into a full RSS feed. This is done by looking at the URLs in the feed, extracting the content from the Web page and then creating a new feed out of that.

Traditionally, spammers have performed feed expansion on their servers, reading the RSS feed and doing the scraping themselves. However, several new tools have been made available to the public, including FeedEx and FeedExpander that provide this service for free and to anyone willing to paste in a short RSS feed.

In short, there’s no need for a potential spammer to set up software of their own. They can simply feed your partial feed into the feed expander and scrape the full one it produces, no work or expense required.

THough the services don’t work on all sites, in particular those with unusual formats, they do work on most and, even though they aren’t perfect, they are already more than reliable enough for spammers, as evidenced by the ones I’ve seen using these and other services.

Is Feed Expansion Legal?

To help with this article, I reached out to both sites I mentioned above. I only heard back from FeedEx, where I got a response from Nikolay, who says his site respects robots.txt and doesn’t scrape content where robots are barred.

However, it’s unclear if such action is enough to make these services completely legal. The reason is that, while robots.txt work for search engines, search engines don’t redistribute the content and distribution is one of the rights that copyright protects. Furthermore, any implied license argument about this kind of use would be weak at best as the webmaster, by having a truncated feed, indicated pretty clearly that they don’t want their content distributed via that means.

While these services could mitigate this by truncating their outgoing feeds (as Google did with its service to produce feeds from any page), that would defeat the purpose of the service.

In short, these services take the content from other websites, copies it and posts it on another page (remember, an RSS feed is fundamentally a specially-formatted webpage). This is, quite frankly, the very definition of what copyright infringement online is, however, if these services are used by non-spammers, the rightsholder is unlikely to know or care.

However, this just deals with the copyright issues. As discussed previously, the scraping of a site brings about a variety of other issues including trespass to chattels, Computer Fraud and Abuse Act violations and more (Note: The original PDF linked to is offline, I’m working to find a replacement, in the meantime, see this article as well.).

All of this combines to paint a pretty bleak legal picture, but yet these services soldier on.

FeedEx Responds

As mentioned above, I reached out to both sites before writing this article but only FeedEx responded (However, I will update this article should I hear back from FeedExpander).

When told that some webmasters are upset at his service, Nikolay responded that the complaints may not be as high as some would expect, saying that, “During all years of feedex.net presence, I have received just 2 complaints. And something around 100 of improvements requests.”

Nikolay also stated that, even though his site has a DMCA policy, he handled the requests with a simple email, blocking his bots from accessing those feeds.

When asked why he created the service, Nikolay, said that he did it first for himself as he wanted a more mobile way to view websites and was tired of sites with partial feeds and of alternatives such as Readability, especially on his tablet and phone.

That being said, Nikolay did acknowledge that spammers have used his service but that he has no means of stopping them, “I know that some spammers using my service bad way. At the moment I have no automated methods to ban them all and I cannot do that manually. So, I ban only those feeds, for which I have received complaint.”

FeedExpander, in its FAQ, says something similar, calling itself a “double edged sword” that is used both for legitimate purposes and for scraping.

Both sites claim that they designed their service for legitimate uses only and that the misuse of it is a side effect of the intended purpose.

Whether you feel that’s true or not, it’s clear that these services and ones like them are here to stay and the legal issues are, for the most part, purely hypothetical.

Fighting Back

As I mentioned in my article earlier this month, there are ways you can fight back. This includes linking to yourself regularly, including footers in your posts and breaking apart content.

However, my goal with this post is not to cause these services to get flooded with removal requests. I really don’t think it would do much good. While I believe they will block your feeds (if they are even using them), they are only two services and most spammers that use this method and most spammers still prefer to use their own technology rather than rely on a third party.

What’s important to note is that the use of truncated RSS feeds is an almost complete waste. The only way it will help protect your content is by stopping those too lazy to use a feed expander. Given that there are so many full feed RSS sites out there, that protection might be worth something, but a dedicated scraper can easily get your content if motivated.

That being said, there is another benefit: Time. If you truncate your feed and someone passes it through one of these services, there’s going to be a delay in when it appears on their site. According to Nikolay, that time can be between 5-10 hours if your feed isn’t popular with the service. That, hopefully, will be plenty of time for Google to spot your site as the original and treat the scrapers as the spammers they are.

Bottom Line

As I said back in 2006, I have always felt that partial RSS feeds were a bad trade. These services do little more than highlight how ineffective it is against spamming.

That being said, there are better way to track and prevent reuse of your content. All you have to do is plan in advance and not believe truncated feeds to be a silver bullet against the problem.

This is why my goal isn’t so much to take these services to task but to merely highlight that they exist. The spammers already know about them and you should too. Their existence doesn’t change much on the front of content theft and spamming, but hopefully they will raise awareness to what has been around for many, many years.

The reason for feed truncation is simple, most RSS scraping takes place through the RSS feed so truncating it, or showing only the first few paragraphs of it, meant scrapers couldn’t grab the whole post and could only do limited damage to your site and its content.

However, as I pointed out in my article, the technology was already available (and already a decade or more old) to scrape content out of the Web page itself. This meant that truncating an RSS feed, while useful against most scrapers (at the risk of angering readers) was a temporary solution until spammers started using mare advanced methods.

Now, it appears I might have been ahead of the curve. At least two clients and several others I’ve talked with have reported that, despite either having no RSS feed or only a truncated one, that their site’s full content is being scraped. The problem seems to be growing and it seems likely that it will get a lot worse before it gets any better.

Why RSS Scraping Was (IS) King

RSS Scraping became popular because of how simple it was. RSS feeds, unlike regular HTML, have a predictable format and structure that makes it easy to extract the content from them. It’s how RSS readers work as well as RSS scrapers.

RSS feeds are plentiful and their ease of use also opens up the doors to other kinds of manipulation, such as changing out words, inserting links and clipping off unwanted portions.

However, RSS feeds also have a serious problem. For one, webmasters, after they learn about the scraping often truncate the feed or insert warnings into it to make it useless to the spammer. Second, webmasters often alert Google and other search engines via RSS when a new post goes live, making it so that the spammer has a difficult time getting to the search engines first.

Combine that with the fact that recent Google updates, including Panda and Farmer, have been pounding sites seen as content farms or spam, many spammers have been seeking alternate ways to get their content in recent months.

Non-RSS Scraping Comes to Town

Non-RSS scraping came to the forefront in January of last year when Google announced that Google Reader could track changes on any site. Though the feature wasn’t evil in and of itself, only displaying truncated content, it brought the issue to the forefront and proved that the technology was practical, scalable and functional.

Even though it was shut off just nine months later, the proof of concept was still there and it seems at least some spammers took notice.

However, Google wasn’t the first nor was it the last, it was merely the biggest. Yahoo! Pipes and Page2RSS have long done the same thing for years (Note: Once again, I’m not saying these are the services the spammers are using, they are once again mere proofs of concept) and there are countless downloadable applications that can run on a server. This is, most likely, the approach being taken by spammers.

Obviously though, the tech is there and has been for some time, but now it seems to have the attention of at least some spammers and that, in turn, is going to change the game for content creators sooner rather than later.

How Webmasters Can Fight Back

These tools work because A) It’s trivial to detect changes in a website and B) even though sites are different from one another, the pages within a site tend to remain fairly consistent.

This is a big part of how sites are operated today as most are template-driven. To make matters worse, with CMSes like WordPress, Drupal, Joomla, etc., there’s often a lot of consistencies between sites that use the same platform, making it even easier.

This makes fighting back against this kind of scraping much more difficult. However, the usual tips for fighting against scraping remain relevant, just no longer solely for the RSS feed:

1. Link to Your Content Regularly: Try to include one or two links in each of your posts that reference a page or a post on your site. This helps tell the search engines which site is the authentic one and pass along the credibility accordingly.
2. Include a Footer: Including a footer in the content area of your site may cause it to get picked up along with your text, especially with highly automated scrapers. You can also include a digital fingerprint in the content area, though when searching for it later you’ll have to omit your site.
3. Breaking Apart Content: Breaking content apart across multiple pages is a controversial strategy (one that your readers will likely not care for) but it can also hinder this style of scraping similar to how truncated RSS feeds work.

All in all, this type of scraping is going to be much more difficult to combat as there is no feed you can simply disable, alter or truncate if things get out of hand. It’s a problem that will have to be met head on and one webmasters will have to be vigilant about.

Bottom Line

The good news in all of this is that this type of scraping is still not very common, at least not that we know of. However, if you have a full RSS feed, it may be difficult to know where spammers are getting the content from.

Still, this type of scraping does appear to be on the rise. The only question is if it will be a long-term trend or a short-term fad. Right now, it’s looking more likely it will be the former.

This is why webmasters need to be aware of this problem so they can be vigilant against it and, when needed, deal with the problem.

In the end though, the only real question I have is why did it take so long for spammers to pick this up? Usually on the cutting edge, they seem to be at least five years behind the curve on this one, if not more.

Maybe someone can provide an explanation for that one below…

However, times have definitely changed. Last year I wrote about how the role of RSS was changing. By most accounts, the use of feed readers peaked in 2008 at about 11% and has been declining since. The broader public found feed readers too complicated and not useful enough for regular consumption.

But at the same time, RSS usage has grown in very big ways. Currently millios of people are reading RSS feeds without realizing they’re doing so. Countless Twitter accounts and Facebook Pages are being fed via RSS and are serving them much like a feed reader was supposed to, sending people near-instant updates and letting them read all of their content in one place.

This shift is changing what RSS is and means, turning it away from being a means to read a site and into the engine that enables sharing and content discovery.

This, in turn, is impacting how webmasters and bloggers use and interact with RSS and is also shifting the ways in content creators protect their works and how users interact with it.

Here are just five examples of how that is happening right now.

1. Fewer, If Any, RSS Buttons

If you go to TechCrunch, you won’t find a single RSS button on their home page. Since their recent redesign, the RSS link has been moved to the footer, three little letters at the bottom of their site.

Meanwhile, their Facebook “Like” box is prominently displayed in their sidebar and Twitter sharing buttons line the entire site. Webmasters have been steadily downplaying RSS subscription in favor of social networking.

RSS just doesn’t have the “cool” factor any more and it’s been moved to a behind-the-scenes player in content distribution. This is why many webmasters, myself included, have been slowly scaling back RSS subscription efforts in lieu of other, more popular alternatives.

2. Better RSS Control

RSS by its nature has historically been completely open. Anyone could be accessing it. A visitor to an RSS feed could be single user looking at it in Outlook or it could be Google Reader preparing to send it to hundreds of subscribers. This opened the door for scrapers and others who wanted to misuse the content in the feed as everyone had to be let in.

However, the number of distribution channels is dropping. This makes it possible to limit who has access to the feed and only let in permitted clients. Though you’ve always been able to block scrapers, this would change the system from one where everyone has access until they’re booted to one where only the permitted users are let in at all.

This could stop scrapers before they start, or at least force them to pull from other channels to get the content.

3. Greater Tolerance of Truncated Feeds

Five years ago, having a truncated feed was a sure-fire way to turn away potential subscribers. The issue was such a hot-button topic that a petition was circulated around against partial feeds and it gained a bit of traction.

However, with the new subscription channels, people are more used to getting a preview and clicking through. They are more about content discovery than content consumption, making partial feeds roughly as useful as full ones.

4. Loss of Platform Control

While the ability to control access and the ability to greater openness to the use of partial feeds gives webmasters more control, it also comes with drawbacks.

Previously, if a single RSS reader or site using your content did something you didn’t like you could always block them, file a takedown notice or take other action. However, if Facebook decides to display RSS feeds in an in appropriate or controversial way, there’s not much one can do as that is a large percentage of the audience.

The good news is that Facebook and Twitter both don’t integrate RSS directly and, instead, use third party apps to do it. However, that’s no guarantee in and of itself as decisions by these two can impact and even cut off how RSS flows through their systems.

In short, even though you can always switch apps, Facebook and Twitter are still very much in control.

5. Losing Sight of What RSS Even Is

With RSS disappearing from sites and fewer bloggers even using them, it seems likely that even fewer people will be aware of RSS in just a few years’ time. Even those who know of it and use it somewhat now will, with time, probably forget about it as both the name RSS as well as the famous icons will be all-but-meaningless to end users.

This also means that fewer webmasters will be thinking about it and fewer will be weighing the issues and decisions that come with having an RSS feed on your site.

This may, in turn, open the doors for others with less-than-pure intentions to exploit the naivete of webmasters, who are unaware of how they are gaining access to their site’s content.

Bottom Line

All in all, the changing role of RSS is a mixed bag for webmasters and content creators. While it will make it easier to block and reduce the impact of traditional scrapers, the loss of control over the platform and lack of front-of-mind understanding of what RSS is and how it works still opens up some serious vulnerabilities.

However, this is a transition that is happening slowly and will continue to do so for some time. Most likely we still have several more transition years before we truly reach the point with RSS where it is meaningless to users.

That being said, with so many major blog eschewing or downplaying RSS, it may be that the transition is happening much faster than once thought possible. It may simply be that the simplicity and large presence of Facebook, Twitter and other social networks are just overpowering to the traditional RSS model and we may be mourning RSS’ demise as a destination sooner rather than later.

Either way though, RSS will live on, behind the scenes, driving social media and marketing for content creators of all stripes. That much is definitely certain.

Have any suggestions for the 3 Count? Let me know via Twitter @plagiarismtoday.

1: EU Backs Fans Watching Soccer on Cheap Decoders

First off today, a ruling by the EU’s highest court may change the way fans all over the continent view soccer matches. The case centers around a British pub owner, Karen Murphy, who was sued by the England’s Premier League after she used a cheap Greek decoder to view league matches in her pub. The decoder cost a fraction of what it would have cost to subscribe to Sky TV, the broadcaster with the UK rights. That prompted the suit from the league. Though a lower court ruled against Murphy, the higher court said that a local law barring the importation and sale of such decoders could not be justified and that the matches themselves do not qualify for copyright protection, though pre-game material, overlays and other elements might. The decision now goes back to the lower court, which has to take the advisement and apply it.

2: Legal Expert Says Anti-counterfeit Deal Should Be Scrapped

Next up today, a Douwe Korff, professor of international law at London Metropolitan University, was recently hired by the Greens in the EU Parliament to study the human rights implications of the Anti-Counterfeiting Trade Agreement (ACTA). He concluded that the agreement may negatively impact free speech and personal excpression by not allowing trivial or otherwise beneficial infringements. However, the EU Parliament has asked its own legal committee, the JURI committee, to look into the legality of ACTA as well.

3: Century 21 Canada wins lawsuit against Rogers subsidiary Zoocasa

Finally today, in a story that I missed when it first broke, Century 21 Canada has won its lawsuit against Zoocasa, a real estate searching site that Century 21 accused of scraping content from their Web presence. According to the ruling, the scraping was a violation of both copyright law and Century 21′s terms of service. The court ordered Zoocasa to stop misusing the site and to pay $1,000 in damages.

Suggestions

That’s it for the three count today. We will be back tomorrow with three more copyright links. If you have a link that you want to suggest a link for the column or have any proposals to make it better. Feel free to leave a comment or send me an email. I hope to hear from you.

Want the Full Story?

Tune in every Wednesday evening at 5 PM ET for the live recording of the Copyright 2.0 Show or wait and get the edited version Friday right here on Plagiarism Today.

The 3 Count Logo was created by Justin Goff and is licensed under a Creative Commons Attribution License.

Basically, at that time, RSS scrapers were arguing that, by putting content into an RSS feed, one was giving permission to use it on other sites, essentially creating an implied license to republish it. However, as I talked about in the previous article, the legal realities are much different and RSS scraping without per mission is, almost certainly, a copyright infringement.

However, while the legal realities haven’t changed much in the past five years, the people doing the scraping have. Spammers and sploggers, now wary of duplicate content issues, have largely abandoned RSS scraping in favor of other techniques. Today, the scrapers are fewer but place themselves as editors, curators and collectors, people building moderated lists of great content.

This shift hasn’t done much to alter the legal realities of scraping nor has it done much to placate creators who still see this as one of the most common issues they face.

The truth is that, even with this new veneer, RSS scraping is still not legally or ethically acceptable. Whether it’s curators or spammers, those who scrape from RSS feeds are in a dubious position and one that seems to be getting worse every day.

The Past Five Years Of Law and Scraping

The past five years of legal history have been strangely quiet on the issue of RSS scraping. Despite how common the behavior is, very few suits have dealt with the issue.

The best known of those cases was Gatehouse Media vs. The New York Times. Which saw Gatehouse Media, the owners of “Wicked Local” brand sites as well as hundreds of smaller papers, sue the New York for the Times’ scraping of their RSS feeds for inclusion on Boston.com’s “Your Town” section.

The suit only centered around the headlines and excerpts from the stories involved but the Times felt their position was weak enough to warrant settling the matter publicly and quickly. In the end, the New York Times agreed to stop scraping Gatehouse feeds and respect restrictions placed by Gatehouse Media on the content.

Related cases on the issue of data scraping, sometimes called data mining, have largely been equally negative for the scrapers. Though only at the summary judgment phase at last report, the Snap-on Business Solutions Inc. v. O’Neil & Assocs., Inc, highlights the other legal perils of scraping.

In that case, Snap-on produced and maintained a database of auto parts for Mitsubishi. After two years, Mitsubishi began to look at other vendors for the contract but Snap-on would not give up control over the data. Mitsubishi eventually hired an outside contractor, O’Neil, to scrape the content out of the database and bring it into a new system. When Snap-on learned of the scraping, they filed suit.

In the summary judgement phase of the case, the judge ruled that Snap-on likely had arguments regarding the Computer Fraud and Abuse Act (CFAA), Trespass to Chattels and Breach of Contract. The court rejected a copyright infringement argument, but only because the content copied did not qualify for copyright protection, unlike with RSS feeds.

The case shows, as I pointed out years ago, that scraping is a legal minefield. Even cases that seem to go the way of the scraper, such as the Cvent, Inc. v. Eventbrite, Inc. case, are highly fact-specific and seem to hinge more on poor case preparation than the law itself. (Note: Even in that “victory” the copyright claims and the unjust enrichment claims survived dismissal.)

Instead, most seem to follow the route of the Ticketmaster L.L.C. v. RMG Technologies, Inc. case, a 2007 win for Tickemaster against a sniping service that was snatching up popular tickets using an automated process. In that case, the court ruled RMG was infringing copyright by merely browsing the relevant pages since they were doing so in violation of Tickemtaster’s “browserwrap” license.

In short, the legal realities for scraper are even more bleak than they were five years ago. The implied license argument that’s so popular among scrapers has been eroded and, all in all, it’s almost impossible to scrape legally, RSS or otherwise.

Yet, what’s changed in the last five years isn’t so much the law, but the scrapers themselves and that’s where things have truly gotten interesting.

The Death of the Spammer Scraper

Back in 2006, your “typical” RSS scraper was probably a spammer, someone seeking a quick, hands off way of filling a large number of sites with search engine friendly content to rise in the rankings and, eventually, usurp the original work for certain keywords.

Those days, however, are gone. Though scraping spammers still exist, most spammers moved on from this method as Google and the other search engines improved their duplicate content detection, making it a less effective technique. Methods such as content spinning, content generation and even cheap outsourcing have proved to be more effective and equally reliable.

This decline has largely mirrored the overall decline in traditional (reader-based) RSS usage. RSS is falling out of vogue, at least as a tool for scraping and reading, but not as a tool for “curating”.

The reason is that tools for integrating RSS into existing websites have grown much more common and easier to use in the past five years. Though some were developed for the use of spamming, other tools were meant to allow authors to integrate all of their sites in one place. However, some authors have latched onto these tools as a way of bringing in the work of others without permission.

This has created a situation where the people doing the scraping are fewer in number, but likely much more dangerous. Where search engines were relatively effective at filtering out spammers, these sites tend to appear to be much more legitimate, increasing the likelihood they could be mistaken as originals.

Fortunately, the law doesn’t make a great distinction between spammers and those who scrape with less nefarious intentions, but many who engage in this practice have, according to emails I’ve seen, have claimed to have an ethical or even legal right to engage in the scraping, calling themselves “editors”.

This has set the stage for some ugly battles that, while they haven’t reached the courtroom yet, have certainly been heated on the Web.

Indeed, this argument seems to be one that’s moving out of the courtroom and into the court of public opinion, a place where it’s likely to stay given how straightforward the legal issues seem.

Bottom Line

In the end, consider that the New York Times Company, one of the most powerful media institutions on the planet, couldn’t or didn’t want to defend scraping of just headlines and summaries, there’s little hope for a successful defense of full RSS scraping. This is especially true in the light of other, related scraping cases.

However, those who want to scrape and those who are willing to allow their feeds to be scraped do have options. Creative Commons, for example, has modules for RSS feeds that enable applications to detect what they are allowed to do with a feed.

To those who don’t wish to allow it, I encourage you to put in your feed itself a notice stating that you do not wish to allow republishing and that the feed is for private personal use only. Though it shouldn’t be necessary under the law, it’s a wise move that blocks many of the potential arguments a scraper might raise. Furthermore, such footers can greatly help with the detection of scrapers.

All in all, RSS scraping has definitely changed in terms of who is using it and why, but the threat isn’t all that different and the legal realities have hardly changed at all.

This means that RSS scraping can be easily fought, just that the people you’re moving against may be a bit more vocal in their views.

As such, it’s important for all webmasters and content creators to be aware of what RSS scraping is, how it works and where it’s going in the future. Even though RSS as a protocol may be on the ropes, RSS scraping is not a problem that’s going away and, in fact, may be getting a lot worse in the coming years.

With that in mind, here is a quick FAQ on some of the more common questions asked about RSS scraping and what can be done about it.

What is RSS?

RSS, sometimes referred to as Really Simple Syndication or Rich Site Summary, is a protocol that makes it easy for other sites and tools to access the content in your site by formatting your content in a consistent, easy-to-parse way.

Contrary to an HTML document, which could have the content be anywhere on the page, RSS indicates clearly what is the headline, body and other elements of the content. This makes it easy to grab the content and display it elsewhere without the surrounding formatting and HTML code.

How is RSS Normally Used?

Traditionally, RSS has been used to enable readers to subscribe to a site using various RSS readers such as Google Reader, Feed Demon and even many mail clients.

However, RSS has also been used to power other services, such as email newsletters and even Facebook integration.

What is RSS Scraping?

RSS scraping is when a third party, usually a spammer, grabs the content in an RSS and republishes it wholesale on another site.

In this regard, RSS scrapers work a great deal like Google Reader, grabbing your site’s content and displaying it on a site but, where Google Reader places the content behind a password protected wall that can only be accessed by the subscriber (or those who are shared the individual story), scrapers instead place the content on a public site for anyone to view, including search engines.

Why do People Scrape RSS Feeds?

Spammers seek high rankings in search engines so they can get traffic to display their ads against or sell products with. To do this, they need content but creating content by hand is time-consuming and difficult, especially when much of it is going to make no difference in the search engines.

RSS scraping is an easy way for spammers, and other sites, to quickly fill their pages with content, even if the content comes solely from other sites.

How Can RSS Scraping Hurt Me?

In most cases, RSS scraping doesn’t hurt. Google and other search engines have become savvy enough about spam that most of the time, they don’t give much credence to spam sites, keeping them from getting a lot of traffic or harming you in the rankings.

However, the system is far from perfect and there are many times spammers outrank the sites they scrape from for relevant terms. This is especially true with new sites or those that don’t have a strong search engine presence.

Less likely is that others may confuse the spam site as either being the original site or as being one endorsed by you, thus actively taking traffic from you. Few people, however, make this mistake with spam sites as the distinction is usually very clear.

All in all, the risk from an individual case of RSS scraping is actually fairly low, but the problem is that there is rarely just one or two such scrapers working at any given time.

What Can I Do About RSS Scraping?

Dealing with RSS scraping starts with good SEO practices. If you link between your posts, get good inbound mentions and earn social networking shares, odds are that RSS scraping won’t greatly impact you.

If it does, you can alway seek to have the content removed by either filing a DMCA notice with the spammer’s host or, if that fails, sending one to Google.

If RSS scraping becomes a more serious and more recurring problem, you may want to consider truncating your feeds or eliminating them. Though that would be an extreme last resort.

Is RSS Scraping Illegal?

Some have made arguments that distributing your content via an RSS feed, even if you didn’t realize you were doing it, creates an implied license to use it in this manner. However, there are many problems with that and other related arguments on RSS scraping.

Generally, RSS scraping is considered to be copyright infringement, though there are other legal arguments against RSS scraping as well.

What if I Want to Encourage RSS Scraping and Reuse

If you want others to scrape your RSS feed, you can actually give blanket permission to do that by inserting a Creative Commons license into your feed. This will let bots that do scraping know your intentions and, those that are complying with the law should be able to follow your wishes.

How Can I Track RSS Scraping?

Many people will find RSS scrapers on accident when they search for keywords relevent to their blog or site. However, you can keep track of your content using automated tools like Fairshare that are designed for tracking dynamic content.

In the end though, its best to keep an eye on the search engines for terms that others commonly find your site through as scrapers will often show up for those same results though, initially, they will likely be lower than your site.

What is the Future of RSS Scraping

Though it’s difficult to predict what spam tactics will be popular in the coming years, RSS scraping has been a problem for at least six years and is continuing today.

That being said, it has fallen out of favor with many spammers, who prefer content generation or scraping excerpts from feeds to avoid duplicate content penalties in the search engines. Still, many active spammers use the method though spammers have clearly become more diversified in this area.

Bottom Line

There’s no doubt that RSS scraping can be and often is very annoying and very problematic. That being said, there’s no reason that it should be a major headache or that it should become a reason to walk away from your site. Most cases of RSS scraping don’t have a major impact on a blog and those that do can usually be dealt with.

That being said, if you are having a serious problem with RSS scraping, please feel free to drop me a line or, if you think you may need outside help, feel free to see if I can help via my consulting services.

All in all, RSS scraping is a reality most bloggers and webmasters will have to deal with, but it’s not one that should sink your site if you’re savvy about how to handle it.

It is Friday again and that means that it is time for another episode of the Copyright 2.0 Show.

It is our second week with our “longer story” format and we have a lot of great news for you. There’s an update to the Global Grind case, a major plagiarism controversy and perhaps the most meta copyright infringement ever.

All delivered with our usual mix if reporting, in-depth discussion and humor. It’s a show that, hopefully, is as entertaining as it is thought-provoking and informative.

This week’s stories include:

• Global Grind Stops Scraping
• A Plagiarism Controversy Shakes a Major Art Contest
• RapidShare Scores a Big Win in Germany
• Can You Pirate a Pirate Ship?

You can download the MP3 file here (direct download). Those interested in subscribing to the show can do so via this feed.

Show Notes

About the Hosts

Jonathan Bailey

Jonathan Bailey (@plagiarismtoday) is the Webmaster and author of Plagiarism Today (Hint: You’re there now) and works as a copyright and plagiarism consultant. Though not an attorney, he has resolved over 700 cases of plagiarism involving his own work and has helped countless others protect their work and develop strategies for making their content work as hard as possible toward their goals.

Patrick O’Keefe

Patrick O’Keefe (@iFroggy) is the owner of the iFroggy Network, a network of websites covering various interests. He’s the author of the book “Managing Online Forums,” a practical guide to managing online communities and social spaces. He maintains a blog about online community management at ManagingCommunities.com and a personal blog at patrickokeefe.com.

Last week, I covered a controversy involving the well-known hip-hop site Global Grind. The controversy, originally reported by my friend and Copyright 2.0 Show co-host Patrick O’Keefe, had become something of a large-scale controversy, especially in the rap and hip-hop community, garnering many mentions on popular blogs and Twitter accounts.

In short, what O’Keefe accused Global Grind of doing, and the site later admitted, was scraping content from various sources, including one of O’Keefe’s blog, and publishing the content to Google. Since Global Grind had been accepted to Google News, the scraped content appeared there as well.

However, today I have an update on the case. After a week of the firestorm, it seems that Global Grind has not only stopped scraping, but has also removed all old scraped content from its site and stopped using its frame for outbound links.

All in all the news is extremely good. Though it is upsetting that Global Grind engaged in these practices in the first place and didn’t respond to O’Keefe’s initial, private, inquiries on the issue, it is clear they are taking the right steps now.

Currently, on their home page, all the stories pulled from other sources, of which there are only a few, cite just a few words and link directly to the original source, without the frame. Only one story contains more than an acceptable level of copied content and, judging from the way others were handled, it seems that it was likely an error made by a staff member, not a change in policy.

So, for now, this case seems to be largely resolved. Though I am still hoping for further changes from Global Grind, including a public statement on this issue and a designation of an actual DMCA agent, the major issues have been dealt with.

Hopefully this case will serve as a warning to other sites that may try a similar tactic, bloggers and Webmasters do notice and are not happy about this kind of infringement.

It is Friday again and that means that it is time for another episode of the Copyright 2.0 Show.

It is a very special week for the Copyright 2.0 Show as spend the hour on just one news story, the Global Grind controversy originally reported on by Patrick O’Keefe, the esteemed co-host of the show. We also debuted a new chatroom, which can be found here and had a very long, involved discussion with those who dropped by for the show.

It was a great show and we hope to see you there every Wednesday at 6 PM ET for the live recording!

In this show we covered:

• The Background of the Global Grind Case
• What Has Been Done About It
• How Global Grind Has Responded
• What Affected Webmasters Can Do
• What’s Next for the Case
• And Many more…

You can download the MP3 file here (direct download). Those interested in subscribing to the show can do so via this feed.

Show Notes

About the Hosts

Jonathan Bailey

Jonathan Bailey (@plagiarismtoday) is the Webmaster and author of Plagiarism Today (Hint: You’re there now) and works as a copyright and plagiarism consultant. Though not an attorney, he has resolved over 700 cases of plagiarism involving his own work and has helped countless others protect their work and develop strategies for making their content work as hard as possible toward their goals.

Patrick O’Keefe

Patrick O’Keefe (@iFroggy) is the owner of the iFroggy Network, a network of websites covering various interests. He’s the author of the book “Managing Online Forums,” a practical guide to managing online communities and social spaces. He maintains a blog about online community management at ManagingCommunities.com and a personal blog at patrickokeefe.com.