Last week, @melebeth introduced me to CPedia, a new “encyclopedia” by the makers of Cuil, a search engine that was initially greeted with much fanfare before seemingly flaming out.

Cpedia is not an encyclopedia in the strictest sense as it is not written by human beings. Unlike traditional encyclopedias, which are written by paid experts, or Wikipedia, which is written largely by volunteers, Cpedia is written automatically from the search result pages creating an automated, and often wildly inaccurate encyclopedia-like page.

For example, I have a Cpedia, page as well as one for this site, though my personal page doesn’t actually say anything about me and the one for PT seems to discuss random people/items only tangentially related to the site.

However, the concern Melebeth approached me with was not just about the accuracy of Cpedia, but about the way it used content from other sources. According to her, the search engine was lifting text directly from third-party sites but not properly quoting or citing it.

So, I delved into Cpedia and found, unfortunately, that her fears were largely founded.

How CPedia Works

The basic idea behind CPedia is that it combs through the search results for a relevant term and tries to build out an encyclopedia entry automatically. The results, visually, are very similar to Wikipedia but the content is generally more jumbled and difficult to read.

Cpedia does attribute the content it uses, but in a very strange way. If you click or hover your mouse over the text within an article, but not the link, you will be given a sidebar that shows the text that’s been quoted and a link to the source in the sidebar. If you click the inline text link, you are instead taken to a references page that then links to the original source.

Usually, the individual copied passages are very short though, sometimes, the word count of the passage approached 100 words, especially when the source package was broken up into multiple parts.

CPedia seems to have a nearly unlimited number of topics covered, likely aided by the fact it is automatically generating results, and has many pages that Wikipedia does not, including one for me.

All in all, CPedia is fairly straightforward but that does not mean it isn’t a problem. In fact, in this case, it means quite the opposite.

Problems With Cpedia

Apart from the questionable accuracy of Cpedia, the entire operation, to me, seems highly suspect. The idea of creating new pages of content using snippets from dozens, even hundreds of other pages seems to be a very poor way to do business.

But even discarding the way the content is created, there are several issues with the attribution issue alone. Consider the following two problems:

1. Always One Step From Source Link: Whether you hover over the text or click to the references page, you are always one action away from the source link. This means users and other search engines alike are always two steps from the source site even though it would be trivial to make it one.
2. Lack of Clear Quotes: The entire entry is made up of short verbatim quotes from various sources but it is not clear where the quotes begin and end without hovering over the text. The goal is to make the entire work seem like an original creation, an actual encyclopedia entry, without much in the way of visible quotes, just traditional footnote citations.

However, the bigger problem is actually very simple. There are already many sites that build thousands and thousands of entries using snippets from various other pages. They’re called spam blogs and they use a variety of article generation and spinning technology to build new articles out of hodgepodges of existing ones.

And Cpedia is acting very much like a spam blog. Entries from CPedia are appearing in Google, which currently has about 177,000 entries indexed, and though, Cpedia’s robots.txt disallows the wiki directory, it doesn’t seem to be stopping search engines from indexing the entries.

When you factor all of this together, it becomes clear that Cpedia is acting exactly like a spam blog and less like an encyclopedia. Was the intention? Probably not. But it is how the site is functioning, pumping thousands of pages of poorly-written duplicate content into the major search engines.

If that is not the hallmark of a spam blog, I’m not sure what is.

Making it Stop

To be clear, what Cpedia is doing isn’t, most likely, illegal. Fair use would likely protect their very limited use of the content from each individual source. This is one of the reasons this technique is so common among spam blogs is that it makes them almost immune to copyright disputes as a means of closure.

In short, even though the ethics of Cpedia can be hotly debated, most likely they are on the right side of the law.

That being said, if you want your work removed from Cpedia, all you have to do is remove it from Cuil and that can be done by using robots.txt to block “twiceler”.

Also, you can block the IP range that Cuil uses for crawling, which is also listed on the link above.

It is a fairly simple change to make and one that is relatively easy to make. (Note: I have not and will not make it on PT, I keep my robots.txt open intentionally to help observe various issues, like these).

All in all, though I disagree strongly with what Cpedia is doing, they do have the right to do it. This makes fighting back trickier, but far from impossible.

Bottom Line

What Cpedia is doing, in my opinion, is unethical. They are using quotes from various sites without adequate clarity or attribution. They are pumping thousands of pages of admittedly duplicate content into other search engines and are producing and encyclopedia that, by their own admission, is wildly inaccurate.

Though copyright may not be a viable litigation route, I have to wonder how libel will work in this case as repeating libel is, generally, the same as making the libelous statement. In short, those admitted inaccuracies in Cpedia could, in theory, come back to bite the company at a later date.

Considering that search engine liability in cases of libel is still being settled around the world, Google won such a claim in the UK but republishing this information on your own site and admitting it is inaccurate seems to be opening up new avenues for liability.

Would this be a likely claim against Cuil/Cpedia? Probably not. But only because the audience for the site is so small that it seems unlikely many will care. The fact that Cuil/Cpedia has seen so little success is a big part of why webmasters haven’t noticed the spammy nature of the issue and taken up arms.

To be certain, Cpedia flew under my radar until Melebeth asked me about it. I can imagine it is doing the same for many others right now as well.

The policy change, will remove all of the automated content posting features from free user accounts, which make up the “vast majority” of WPD members, according to Marty Rozmanith, the creator of WPD.

The tools, however, will remain available for all paid members of the service, regardless of the level they choose.

Previously, unpaid members had limited access to some of the content posting tools, including the Yahoo! Answers, article database and RSS posting tool, enabling free members, who were limited to only three blogs, to automatically post content from a variety of sources, typically without permission.

Whether this does anything to stem the vitriol that has been directed at the service remains to be seen, but I can’t see how many will be convinced, especially when there are so many difficult questions to be answered.

WordPressDirect Recap

For those who did not read the previous articles about WPD, the service promotes itself as a “WordPress deployment and maintenance service that helps people especially those with very little technical experience) create a search-optimized WordPress blog.”

In short, it is a one-click install program that not only sets up the software, but also adds a theme, optimizes the permanlinks and makes a handful of other SEO-oriented changes. In that regard, it is much like Fantastico, but with added features to help get the blog started.

However, WordPressDirect stepped into controversy with its add-on tools, which allow users to automatically update the blogs they create using content from a variety of sources including RSS feeds, online article databases and more.

This caused many, especially on the Mashable article, to accuse the site of being a spam service. In that regard, it does share many traits, especially when you look at how the tools work and where they pull their content from.

WordPressDirect attempted to defend itself against the accusations, blaming much of the problem on their marketing, but the attempts to make peace fell on deaf ears for the most part. This, in turn, led to the recent changes they just announced.

Fixing the Problem?

Most likely, these changes are going to do little to nothing to placate the mob that has formed around WordPressDirect. Though the changes mean that the 9000+ free members of the site will no longer have the ability to automatically scrape and repost content, it says nothing about the paid members. The limitations on free accounts, including just three blogs per user, effectively meant that no one could actually be a master spammer with a free account (unless they spammed WPD and set up thousands of accounts).

To many, including myself, this sounds like a very shrewd maneuver. Though it removes most of the users from the ability to do spam-like things, it does not affect the paid ones and the email contained several pitches for the paid packages. It seems not like an attempt to shed the spam-related but to profit from it.

This move does not remove these tools from the power users nor does it impact their bottom line in any meaningful way, other than perhaps adding a few new paid members.

WPD, as a service, is walking a very thin line. It is trying to proclaim itself to not be a spam tool while offering many of the exact same features that are found in spam applications. Though, as I said in my Blog Herald article, it would make a very poor spamming program, it is completely foreseeable and almost certain that users, likely even most users, would use it for that purpose.

Furthermore, issues such as the trademark concerns over the use of the WordPress name, the lack of attribution of copied works, etc. remain unaddressed. Though it is a good step, it seems to be one either too small or in an unrelated direction.

Conclusions

Shortly after my Blog Herald article was released, WPD sent out an email to all members saying that it was “most balanced article” he could find.

Though I try to balanced with all of my coverage, I can not hide the fact that WordPressDirect has me very uneasy and nervous. The service has far too much use for evil and, even though I don’t know if its creators built the service with such intentions, that is the use that instantly springs to mind for myself and many others.

The problem is that a service such as what WPD proclaims to be, a WordPress installation aid that auto-optimizes the blog, could be very useful. I could even see someone such as myself using it rather than keeping a WordPress checklist for every new blog I install (I routinely get recruited to help with WP installations).

But as useful as that could be, the service, is too hot to touch right now and I seriously doubt that is going to change with these recent revisions to their policies. Though I am going to keep an eye open on the marketing changes they mentioned, I don’t see WPD becoming any less of a tainted name anytime soon.

To repair its name, WPD is going to have to make sacrifices that may hurt its business. Sadly, it doesn’t seem to be what they are doing right now.

All in all, it was a busy week in copyright news, a total of sixteen stories were covered including the following:

• SCO lost much of its Linux copyright infringement suit
• Eight more dogpile onto YouTube
• Veoh launches a preemptive strike
• Perfect 10 is at it again
• Google mistakes its own blog as spam, deletes it
• And Many more…

You can download the MP3 file here. Those interested in subscribing to the show can do so via this feed.

Show Notes

I also want to take a moment to link to this story on Ars Technica dealing with plagiarism issues in the scientific community. It is a great read and I wanted to cover it on the broadcast but there simply wasn’t any time.

[audio:http://go.numly.com/1847107081310184589]

As part of running this site, I subscribe to many different Technorati Watchlists. They help me keep up to date on the latest in content-theft and plagiarism-related issues.

Unfortunately, I see a great deal of spam blogs on these watchlists. What’s worse, it can be hard to tell, when looking at my RSS reader, which blogs are legitimate and which are junk. Thus, I often end up clicking through to the splogs that successfully penetrate Technorati’s armor.

Most of those spam blogs have, traditionally, been on Blogspot. However, over the past week or so, I’ve noticed that a lot of the Blogspot links have been returning results like this indicating that the blog has been locked down for “Possible Blogger terms of service violations”.

It appears that, at least based upon the sample I have, that Blogger is on a major offensive against spam blogs and that their effectiveness has gone up drastically over the past week or so. If true, this could be great news for bloggers, especially those on Google’s service, but more research is needed before a victory can be claimed.

I am looking into this matter and am trying to find out exactly what is going on. It could just be that Google has discovered the network responsible for most of the spam targeting my keywords and all of this is a fluke. However, I wanted to pose the question to everyone reading this: Have you noticed a reduction in spam from Blogspot?

I’ll be interested to hear if others are having similar experiences.

The problem seems to be straightforward. If a blog licenses all of their content under a CC license, then a scraper that follows the terms of said license is just as protected as a human copying one or two works. This may be within the letter of the license, but it violates the spirit of Creative Commons.

However, after talking with Mike Linksvayer, the Vice President of Creative Commons, I’m relieved to say that is not the case. CC licenses have several built-in mechanisms that can prevent such abuse.

In fact, when one looks at the future of RSS, it is quite possible that using a CC license might provide better protection than using no license at all.

Against the Spirit: A Crisis with the Commons?

Whether or not some scrapers target CC licensed material or not is up for debate, what is clear is that, when they do, it is often a source of frustration.

People choose CC licenses because they want to share their work with others. They want to participate in a cultural revolution and give their ideas new wings. They do not, generally, want to see their entire site mirrored elsewhere, surrounded by Adsense ads and depriving them of traffic.

Ideally, a CC license is supposed to be symbiotic. The licensor gives up certain rights to their work and the licensee, in exchange for use of the work, makes certain the original author gets due credit and is rewarded for his or her effort. Spam bloggers, however, approach the CC license in bad faith, taking as much as they can while giving as little as possible back.

This has prompted many CC license users to either drop or alter their license. It has become common for sites that are being scraped to change their licenses to “non-commercial”, stop using CC licenses or even shut down their sites altogether.

However, though these spam blogs do seem to be following the terms and conditions of the Creative Commons Licenses, even if by accident, the vast majority do not. In fact, even enabling commercial use of your work is not an open invitation to be scraped.

As it turns out, CC licenses have built in mechanisms that can be used to fight that kind of abuse.

Where Computers Fear to Tread

For the use of a CC licensed work to be valid, according to Linksvayer, the following terms must be met among others:

1. The work must be attributed and it must provide a link back to the copyright holder.
2. If the license is non-commercial, then the work must be used accordingly.
3. If a license has a share-alike term attached to it, then the copied work must express the same license.
4. All CC licensed material must state that it is licensed as such, usually with a statement that says “This work is licensed under a Creative Commons License”. Failure to do so puts the reuse in violation.
5. Finally, with Creative Commons, the licensor has the right to request removal of their name from any reused content, failure to comply puts the reuse in violation of the license.

The problem with all of this is that it is almost impossible for an automated scraping system to comply with all of these elements.

Though some spam bloggers do attribute and link back, most do not. All spam blogging, at least in theory, is a violation of the non-commercial license and, since no spam blogs I have seen carry over CC information, they are in violation of the attribution and share-alike attributes of the Creative Commons License.

Even if a scraper manages to comply with the first four mechanisms above, it is unlikely that, when asked, they would remove the name from any work they reused. Spammers, seeking to automate their operations, are unlikely to edit their spam blogs by hand to appease one copyright holder.

The result is that virtually all automated scraping and spam blogging is a violation of the Creative Commons License, regardless of what license is used.

Technicalities and Human Error

Some of these attributes, however, are relatively unknown. Though most people understand what is and is not acceptable with the various CC licenses, many of the nuances of using CC licenses, such as the fourth mechanism, are little known or followed, even by humans seeking to play fair.

However, most copyright holders, often in the dark about the requirements themselves, do not hold human copiers to these standards. So long as they get the attribution and reuse that they envision, they typically do not raise any alarms if there isn’t a “This work is licensed under…” statement in the reused content.

The question is whether or not it is fair to hold scrapers to a higher standard than we generally hold other people. While it certainly is the right of the copyright holder to determine which misuses of their work they follow up on and deal with, many will, likely, feel uneasy about using largely unenforced technicalities against spam bloggers.

But even those who feel uneasy about enforcing those elements of the Creative Commons license may still benefit from applying one, especially to their feed. With some very difficult questions about copyright and RSS feeds unanswered, having a defined license on your feed could be come critical.

Implied Licenses and RSS

Though most attorneys I know and have spoken with feel that there is no implied license to scrape and republish RSS feeds, the question has not yet come before a court and the outcome, as with all cases pushing new territory, is unpredictable at best.

However, if it is determined that RSS scraping and republishing is legal and that there is such an implied license with posting an RSS feed, attorney Denise Howell feels that any implied license can be overwritten by a defined one, such as a Creative Commons License.

This makes sense consider that an implied license is one designed to operate when there is no actual agreement exists between the parties. If a specific license is posted, it would override the implied license.

We see this already on the Web. By posting a Web page to the Internet, the courts have found that there is an implied license for it to be indexed and cached by the search engines. However, once you state your intention for the page to not be used in such a manner, either through meta tags, robots.txt or manual opt-out, the implied license is dropped and the search engines, legitimate ones at least, have to comply with your requests.

The Creative Commons Organization is working on means of integrating CC licenses into RSS feeds. Hopefully this issue will garter more attention as the legal issues mount and a more final draft can be fleshed out.

Conclusions

The bottom line is that Creative Commons does not encourage or permit blind RSS scraping and spam blogging. Though it might be useful for legitimate aggregation, Creative Commons provides a great deal of protection against scraping, much more than previously thought.

Whether or not these mechanisms prove useful in fighting scraping remains to be seen. However, there is no longer a reason to hold back on a DMCA notice or a copyright complaint just because your commercial CC license seems to permit the use. Unless the scraper followed all of the requirements above, the use is still invalid.

Hopefully this will encourage the wider use of CC licenses, specifically the use of more liberal ones. I myself have removed the non-commercial requirement from my CC license as, like many others, my primary concern was commercial use by scrapers.

In the end, this is just another example of how Creative Commons, when used correctly, can work well for everyone and, in many cases, is good copyright policy.