Last week, @melebeth introduced me to CPedia, a new “encyclopedia” by the makers of Cuil, a search engine that was initially greeted with much fanfare before seemingly flaming out.

Cpedia is not an encyclopedia in the strictest sense as it is not written by human beings. Unlike traditional encyclopedias, which are written by paid experts, or Wikipedia, which is written largely by volunteers, Cpedia is written automatically from the search result pages creating an automated, and often wildly inaccurate encyclopedia-like page.

For example, I have a Cpedia, page as well as one for this site, though my personal page doesn’t actually say anything about me and the one for PT seems to discuss random people/items only tangentially related to the site.

However, the concern Melebeth approached me with was not just about the accuracy of Cpedia, but about the way it used content from other sources. According to her, the search engine was lifting text directly from third-party sites but not properly quoting or citing it.

So, I delved into Cpedia and found, unfortunately, that her fears were largely founded.

How CPedia Works

The basic idea behind CPedia is that it combs through the search results for a relevant term and tries to build out an encyclopedia entry automatically. The results, visually, are very similar to Wikipedia but the content is generally more jumbled and difficult to read.

Cpedia does attribute the content it uses, but in a very strange way. If you click or hover your mouse over the text within an article, but not the link, you will be given a sidebar that shows the text that’s been quoted and a link to the source in the sidebar. If you click the inline text link, you are instead taken to a references page that then links to the original source.

Usually, the individual copied passages are very short though, sometimes, the word count of the passage approached 100 words, especially when the source package was broken up into multiple parts.

CPedia seems to have a nearly unlimited number of topics covered, likely aided by the fact it is automatically generating results, and has many pages that Wikipedia does not, including one for me.

All in all, CPedia is fairly straightforward but that does not mean it isn’t a problem. In fact, in this case, it means quite the opposite.

Problems With Cpedia

Apart from the questionable accuracy of Cpedia, the entire operation, to me, seems highly suspect. The idea of creating new pages of content using snippets from dozens, even hundreds of other pages seems to be a very poor way to do business.

But even discarding the way the content is created, there are several issues with the attribution issue alone. Consider the following two problems:

1. Always One Step From Source Link: Whether you hover over the text or click to the references page, you are always one action away from the source link. This means users and other search engines alike are always two steps from the source site even though it would be trivial to make it one.
2. Lack of Clear Quotes: The entire entry is made up of short verbatim quotes from various sources but it is not clear where the quotes begin and end without hovering over the text. The goal is to make the entire work seem like an original creation, an actual encyclopedia entry, without much in the way of visible quotes, just traditional footnote citations.

However, the bigger problem is actually very simple. There are already many sites that build thousands and thousands of entries using snippets from various other pages. They’re called spam blogs and they use a variety of article generation and spinning technology to build new articles out of hodgepodges of existing ones.

And Cpedia is acting very much like a spam blog. Entries from CPedia are appearing in Google, which currently has about 177,000 entries indexed, and though, Cpedia’s robots.txt disallows the wiki directory, it doesn’t seem to be stopping search engines from indexing the entries.

When you factor all of this together, it becomes clear that Cpedia is acting exactly like a spam blog and less like an encyclopedia. Was the intention? Probably not. But it is how the site is functioning, pumping thousands of pages of poorly-written duplicate content into the major search engines.

If that is not the hallmark of a spam blog, I’m not sure what is.

Making it Stop

To be clear, what Cpedia is doing isn’t, most likely, illegal. Fair use would likely protect their very limited use of the content from each individual source. This is one of the reasons this technique is so common among spam blogs is that it makes them almost immune to copyright disputes as a means of closure.

In short, even though the ethics of Cpedia can be hotly debated, most likely they are on the right side of the law.

That being said, if you want your work removed from Cpedia, all you have to do is remove it from Cuil and that can be done by using robots.txt to block “twiceler”.

Also, you can block the IP range that Cuil uses for crawling, which is also listed on the link above.

It is a fairly simple change to make and one that is relatively easy to make. (Note: I have not and will not make it on PT, I keep my robots.txt open intentionally to help observe various issues, like these).

All in all, though I disagree strongly with what Cpedia is doing, they do have the right to do it. This makes fighting back trickier, but far from impossible.

Bottom Line

What Cpedia is doing, in my opinion, is unethical. They are using quotes from various sites without adequate clarity or attribution. They are pumping thousands of pages of admittedly duplicate content into other search engines and are producing and encyclopedia that, by their own admission, is wildly inaccurate.

Though copyright may not be a viable litigation route, I have to wonder how libel will work in this case as repeating libel is, generally, the same as making the libelous statement. In short, those admitted inaccuracies in Cpedia could, in theory, come back to bite the company at a later date.

Considering that search engine liability in cases of libel is still being settled around the world, Google won such a claim in the UK but republishing this information on your own site and admitting it is inaccurate seems to be opening up new avenues for liability.

Would this be a likely claim against Cuil/Cpedia? Probably not. But only because the audience for the site is so small that it seems unlikely many will care. The fact that Cuil/Cpedia has seen so little success is a big part of why webmasters haven’t noticed the spammy nature of the issue and taken up arms.

To be certain, Cpedia flew under my radar until Melebeth asked me about it. I can imagine it is doing the same for many others right now as well.

It is Friday again and that means that it is time for another episode of the Copyright 2.0 Show.

It is a very special week for the Copyright 2.0 Show as spend the hour on just one news story, the Global Grind controversy originally reported on by Patrick O’Keefe, the esteemed co-host of the show. We also debuted a new chatroom, which can be found here and had a very long, involved discussion with those who dropped by for the show.

It was a great show and we hope to see you there every Wednesday at 6 PM ET for the live recording!

In this show we covered:

• The Background of the Global Grind Case
• What Has Been Done About It
• How Global Grind Has Responded
• What Affected Webmasters Can Do
• What’s Next for the Case
• And Many more…

You can download the MP3 file here (direct download). Those interested in subscribing to the show can do so via this feed.

Show Notes

About the Hosts

Jonathan Bailey

Jonathan Bailey (@plagiarismtoday) is the Webmaster and author of Plagiarism Today (Hint: You’re there now) and works as a copyright and plagiarism consultant. Though not an attorney, he has resolved over 700 cases of plagiarism involving his own work and has helped countless others protect their work and develop strategies for making their content work as hard as possible toward their goals.

Patrick O’Keefe

Patrick O’Keefe (@iFroggy) is the owner of the iFroggy Network, a network of websites covering various interests. He’s the author of the book “Managing Online Forums,” a practical guide to managing online communities and social spaces. He maintains a blog about online community management at ManagingCommunities.com and a personal blog at patrickokeefe.com.

Article Updated: See Below.

Fav.or.it, a content aggregation service, had earned a great deal of controversy among many bloggers. The site would collect content from various RSS feeds, at least in some cases including the full content, and display it on their site as well as offer visitors the chance to comment and discuss the news, away from the original site. This caused some to accuse Fav.or.it of using splogging as a business model and earned it several mentions on this site, including this one.

However, earlier this week, Fav.or.it went down. The initial message said that the site had been taken down for “maintenance”. However, after a few days, the message was changed to read the following:

“Recently we decided that we would not continue with the fav.or.it service on our site. For more details about this please take a look on our blog. As a result we are replacing the site with our company site!”

In short, the Fav.or.it service is no more. It is down never to return, and the site is being replaced by a home page for the company, also called Fav.or.it, which also runs products such as Tweetmeme, which I was unaware of until the site changed, and TweetTabs (Note: The link to TweetTabs on the Fav.or.it site currently is broken).

The sudden closure of the site seemed odd and I emailed Fav.or.it to ask what had happened. However, I am yet to receive a response.

(Note: I have a suspicion as to why the site might have gone down but do not wish to say anything more until I get confirmation.)

Fav.or.it Moves On

Though the closure of its flagship service may seem like a major blow, it could be a very good thing for the company in the long run. They have already moved on to other, more-popular and less copyright-questionable products the best-known of which is Tweetmeme, which is used on this site.

This may help Fav.or.it focus their time and resources on those projects, rather than a small, legally-dubious and much less popular content aggregator.

In the end, I don’t harbor any real ill feelings toward the company as they seem to have created new products that manage to be both useful and raise far fewer copyright issues. Abandoning Fav.or.it as a service was a good move for both themselves and Webmasters everywhere.

As such, I have no qualms about keeping the Tweetmeme feature on this site and don’t think others should either.

I just hope that this serves as a learning experience for Fav.or.it as a company and they are able to build and grow from this controversy.

Update: 08/10/09

Fav.or.it, the company, has posted a short blog entry about the closure of the site that talks about the various reasons for its closure including a shift in the commenting marketplace that made it difficult for them to compete, a lack of updates to the service and a series of implementation mistakes that caused it to lag behind other competitors, including Lazyfeed.

The post did address the content reuse issues and said the following:

The site has also not been without controversy for re-use of content (through public RSS feeds), and although we put massive effort into support of licensing models (such as auto-detection of creative commons) our approach to aggregation of content for which we could not detect a license, and that required the publisher to opt-out (rather than opt-in) was in hindsight misguided.

In short, Fav.or.it, the company, acknowledges they made mistakes with the way they used content within the system and seem to have learned from them with Tweetmeme. However, they did stop short of issuing a full apology. Though I don’t think that will satisfy the most disgusted at Fav.or.it, there still seemsto be much rejoicing that the site is done for, both due to its poor reuse of content, but that it also increases the focus on Tweetmeme. .

All in all, it was a busy week in copyright news, a total of sixteen stories were covered including the following:

• SCO lost much of its Linux copyright infringement suit
• Eight more dogpile onto YouTube
• Veoh launches a preemptive strike
• Perfect 10 is at it again
• Google mistakes its own blog as spam, deletes it
• And Many more…

You can download the MP3 file here. Those interested in subscribing to the show can do so via this feed.

Show Notes

I also want to take a moment to link to this story on Ars Technica dealing with plagiarism issues in the scientific community. It is a great read and I wanted to cover it on the broadcast but there simply wasn’t any time.

[audio:http://go.numly.com/1847107081310184589]

As part of running this site, I subscribe to many different Technorati Watchlists. They help me keep up to date on the latest in content-theft and plagiarism-related issues.

Unfortunately, I see a great deal of spam blogs on these watchlists. What’s worse, it can be hard to tell, when looking at my RSS reader, which blogs are legitimate and which are junk. Thus, I often end up clicking through to the splogs that successfully penetrate Technorati’s armor.

Most of those spam blogs have, traditionally, been on Blogspot. However, over the past week or so, I’ve noticed that a lot of the Blogspot links have been returning results like this indicating that the blog has been locked down for “Possible Blogger terms of service violations”.

It appears that, at least based upon the sample I have, that Blogger is on a major offensive against spam blogs and that their effectiveness has gone up drastically over the past week or so. If true, this could be great news for bloggers, especially those on Google’s service, but more research is needed before a victory can be claimed.

I am looking into this matter and am trying to find out exactly what is going on. It could just be that Google has discovered the network responsible for most of the spam targeting my keywords and all of this is a fluke. However, I wanted to pose the question to everyone reading this: Have you noticed a reduction in spam from Blogspot?

I’ll be interested to hear if others are having similar experiences.

The problem seems to be straightforward. If a blog licenses all of their content under a CC license, then a scraper that follows the terms of said license is just as protected as a human copying one or two works. This may be within the letter of the license, but it violates the spirit of Creative Commons.

However, after talking with Mike Linksvayer, the Vice President of Creative Commons, I’m relieved to say that is not the case. CC licenses have several built-in mechanisms that can prevent such abuse.

In fact, when one looks at the future of RSS, it is quite possible that using a CC license might provide better protection than using no license at all.

Against the Spirit: A Crisis with the Commons?

Whether or not some scrapers target CC licensed material or not is up for debate, what is clear is that, when they do, it is often a source of frustration.

People choose CC licenses because they want to share their work with others. They want to participate in a cultural revolution and give their ideas new wings. They do not, generally, want to see their entire site mirrored elsewhere, surrounded by Adsense ads and depriving them of traffic.

Ideally, a CC license is supposed to be symbiotic. The licensor gives up certain rights to their work and the licensee, in exchange for use of the work, makes certain the original author gets due credit and is rewarded for his or her effort. Spam bloggers, however, approach the CC license in bad faith, taking as much as they can while giving as little as possible back.

This has prompted many CC license users to either drop or alter their license. It has become common for sites that are being scraped to change their licenses to “non-commercial”, stop using CC licenses or even shut down their sites altogether.

However, though these spam blogs do seem to be following the terms and conditions of the Creative Commons Licenses, even if by accident, the vast majority do not. In fact, even enabling commercial use of your work is not an open invitation to be scraped.

As it turns out, CC licenses have built in mechanisms that can be used to fight that kind of abuse.

Where Computers Fear to Tread

For the use of a CC licensed work to be valid, according to Linksvayer, the following terms must be met among others:

1. The work must be attributed and it must provide a link back to the copyright holder.
2. If the license is non-commercial, then the work must be used accordingly.
3. If a license has a share-alike term attached to it, then the copied work must express the same license.
4. All CC licensed material must state that it is licensed as such, usually with a statement that says “This work is licensed under a Creative Commons License”. Failure to do so puts the reuse in violation.
5. Finally, with Creative Commons, the licensor has the right to request removal of their name from any reused content, failure to comply puts the reuse in violation of the license.

The problem with all of this is that it is almost impossible for an automated scraping system to comply with all of these elements.

Though some spam bloggers do attribute and link back, most do not. All spam blogging, at least in theory, is a violation of the non-commercial license and, since no spam blogs I have seen carry over CC information, they are in violation of the attribution and share-alike attributes of the Creative Commons License.

Even if a scraper manages to comply with the first four mechanisms above, it is unlikely that, when asked, they would remove the name from any work they reused. Spammers, seeking to automate their operations, are unlikely to edit their spam blogs by hand to appease one copyright holder.

The result is that virtually all automated scraping and spam blogging is a violation of the Creative Commons License, regardless of what license is used.

Technicalities and Human Error

Some of these attributes, however, are relatively unknown. Though most people understand what is and is not acceptable with the various CC licenses, many of the nuances of using CC licenses, such as the fourth mechanism, are little known or followed, even by humans seeking to play fair.

However, most copyright holders, often in the dark about the requirements themselves, do not hold human copiers to these standards. So long as they get the attribution and reuse that they envision, they typically do not raise any alarms if there isn’t a “This work is licensed under…” statement in the reused content.

The question is whether or not it is fair to hold scrapers to a higher standard than we generally hold other people. While it certainly is the right of the copyright holder to determine which misuses of their work they follow up on and deal with, many will, likely, feel uneasy about using largely unenforced technicalities against spam bloggers.

But even those who feel uneasy about enforcing those elements of the Creative Commons license may still benefit from applying one, especially to their feed. With some very difficult questions about copyright and RSS feeds unanswered, having a defined license on your feed could be come critical.

Implied Licenses and RSS

Though most attorneys I know and have spoken with feel that there is no implied license to scrape and republish RSS feeds, the question has not yet come before a court and the outcome, as with all cases pushing new territory, is unpredictable at best.

However, if it is determined that RSS scraping and republishing is legal and that there is such an implied license with posting an RSS feed, attorney Denise Howell feels that any implied license can be overwritten by a defined one, such as a Creative Commons License.

This makes sense consider that an implied license is one designed to operate when there is no actual agreement exists between the parties. If a specific license is posted, it would override the implied license.

We see this already on the Web. By posting a Web page to the Internet, the courts have found that there is an implied license for it to be indexed and cached by the search engines. However, once you state your intention for the page to not be used in such a manner, either through meta tags, robots.txt or manual opt-out, the implied license is dropped and the search engines, legitimate ones at least, have to comply with your requests.

The Creative Commons Organization is working on means of integrating CC licenses into RSS feeds. Hopefully this issue will garter more attention as the legal issues mount and a more final draft can be fleshed out.

Conclusions

The bottom line is that Creative Commons does not encourage or permit blind RSS scraping and spam blogging. Though it might be useful for legitimate aggregation, Creative Commons provides a great deal of protection against scraping, much more than previously thought.

Whether or not these mechanisms prove useful in fighting scraping remains to be seen. However, there is no longer a reason to hold back on a DMCA notice or a copyright complaint just because your commercial CC license seems to permit the use. Unless the scraper followed all of the requirements above, the use is still invalid.

Hopefully this will encourage the wider use of CC licenses, specifically the use of more liberal ones. I myself have removed the non-commercial requirement from my CC license as, like many others, my primary concern was commercial use by scrapers.

In the end, this is just another example of how Creative Commons, when used correctly, can work well for everyone and, in many cases, is good copyright policy.