Copyscape, which finished third in the previous testing, had its premium service rated first this year with its free service taking the third slot. Both version handily beat out popular academic plagiarism checkers such as Turnitin and SafeAssign.

Though the results may come as a shock to many in the academic community, especially those that pay for accounts at services targeted at universities, those that have followed Dr. Weber-Wulff’s work will likely not be surprised at all. Many will remember her work on exposing iPlagiarismCheck following the previous rounds of testing.

Nonetheless, her study offers a great deal for those interested in plagiarism to think about.

Methodology

To conduct the tests, Dr. Weber-Wulff, along with her team, compiled 31 papers with known amounts of plagiarism and ran them through each of the applications and then rated their effectiveness on a scale of 0-3. They then rated their usability against a set of criteria, including Website of the company and cost transparency, the page layout and captions, workflow and navigation, on a similar scale. The points were then totaled up and ranked on the following scale.

• Very good: 72-80
• Good: 60-71
• Satisfactory: 48-59
• Sufficient: 40-47
• Poor: 0-39

Also listed in the results were cancelled tests, meaning tests that could not be completed for some reason, and products that could not be easily compared against other plagiarism checkers, such as those designed to check medical databases.

Results

In the end, none of the tests came away with a “Very Good” rating. However, Copyscape Premium did come very close, earning 70 out of a needed 72 points. Copyscape Premium earned the maximum number of points in a total of 23 out of the 31 tests.

Plagiarism Detector, another relative unknown, came in second, sandwiched between the two versions of Copyscape. Ephorus, last year’s top application, came in seventh this year, at the very bottom of the “Good” systems.

SafeAssign was number 8, coming in at the top of the “Satisfactory” systems and Turnitin was at the top of the “Adequate” list at 13.

However, several systems were not tested this year. Article Checker and CheckforPlagiarism were aborted, the first not fitting the criteria for a plagiarism checker and the latter not allowing free access for testing purposes.

Also, several commercial services were not slated for testing at all, including Attributor and iCopyright.

Caveats

Though the results of the test certainly should give Copyscape’s developers a reason to cheer, there are several caveats, in addition to the services that were tested, that are worth mentioning.

First, since the test papers were in German, it remains to be seen how well the results would translate to English and other languages. Though I wouldn’t anticipate too much difference since the languages are similar, it could have a bearing on the results, especially with some of the more complex tools that may be language-specific.

Second, the usability tests were geared more toward teachers than they were Webmasters. The test is designed for the academic arena so this makes sense, but as Webmaster, it means that the usability results need to looked at in a slightly different light as our workflow needs are different.

Third, once again, it is important for Webmasters to realize that the results are geared for academic plagiarism, where it is important to trace one plagiarized work to its source, not one source to its many plagiarized copies. Though the search techniques are largely the same, many of the apps best geared at finding academic plagiarism don’t always detect all of the potential sources.

Finally, it is important to remember that the study itself is not an endorsement of any one plagiarism checking system nor is it a recommendation, it is only an analysis of how the various systems did on these specific tests, nothing more.

Conclusions

Though the results are interesting, what I glean most from this comparison is that, even though the standards were lowered for this round of tests, none of the services were able to achieve “Very Good”. This means that even the best plagiarism detection tools have flaws and need improvement.

Personally, I’ve never relied on just one means of detecting plagiarism, instead pulling my information from multiple sources. This study seems to reinforce that idea and further encourages me to back up any system I choose with alternates.

The simple fact is that no plagiarism system is going to be 100% effective and, as such, it makes sense to double up your net, especially since so many of the services are both free and easy to use.

Though I’ve been very critical of Copyscape in the past on this site, it is clear that they are the only company currently serving this niche and many people are excited about their services.

However, the biggest criticism they’ve faced is that their free service is too limited (only ten results per search, a few searches per month) and that their main paid service, Copysentry, is too expensive.

To answer those criticisms, Copyscape recently began offering a series of premium services, targeted in the space between their free and automatic offerings. The idea to be affordable enough that more bloggers and Webmasters would upgrade to their paid offerings and not rely so heavily on the free service.

Unfortunately, as great as the idea is, the execution of this service severely limits its effectiveness, thus only prolonging the agonizing wait until a real plagiarism detection/cessation service comes to light.

What You Get

At the core of Copyscape Premium is an unlimited version of its free service. For a rate of five cents per search (minimum $5 purchase required), you get unlimited searches with unlimited results. You also get the ability to paste text directly into a text box for searching, instead of just submitting Web pages, and an API that you can use to automate and refine submissions to the service.

However, the service goes above and beyond that as well. It offers a plagiarism case tracking service that lets you add discovered cases to an inbox, add notes to them and close them out when they’re handled. Users also have the ability to blacklist certain sites from the results, useful if your own site keeps popping up, and, according to the site, a more robust detection algorithm.

On the surface, it sounds like a great deal. Five cents per search is very reasonable and the case tracking technology alone, if done right, could be worth the price of admission by itself.

That’s why it is so frustrating, and heartbreaking, to report that the service itself is a monumental letdown in its current form. Perhaps not much in the way of a waste of money, but a tremendous waste of time.

My Experience

The process of setting up an account with Copyscape Premium was simple enough. I registered my username and deposited $5 into my account for a total of 100 credits.

I then sought out one of my earlier poems, one that I knew was plagiarized and copied heavily, and searched for it using the URL. Copyscape spun on it for a few seconds and returned me to the results page, which had only one sentence on it “No results found…” (see screenshot above).

Confused, I figured Copyscape was struggling with the large number of comments on my URL. I copied and pasted the text of the poem into the text search option but it was to the same result (see second screenshot).

I then exited Copyscape, performed a simple Google search for the poem and instantly found dozens of suspect pages. Copyscape was simply not picking up any of the copies of the work.

I repeated the experiment with four or five other works, using a combination of URL and text searches. After ten searches, Copyscape failed to find even one plagiarist, even though all of the works I searched for were confirmed to have plagiarized copies on the Web.

Copyscape did return a single result on at least one of the text searches, but the result lead to a legitimate use of my work, not a plagiarist. It would be a theme that would repeat itself over and over in my attempts to use Copyscape Premium.

But Wait… There’s More

Adding to the ineffectiveness of their search is the fact that Copyscape does not allow you to add text searches into your “View Cases” inbox. As a result, I could not test their tracking system with the results I had on hand.

To do that, I had to punch in the URL for Plagiarism Today. That turned up 17 results but almost all of them were junk. The results included people quoting from my sidebar, sites using similar plugins (such as the Share This! plugin) or sites legitimately citing articles. The only one with a substantial amount of reproduced content was a Numly Verification Page, which I created myself through my Numly plugin. Still, it was good enough to test their case tracking service with for the moment.

Unfortunately, that service left a great deal to be desired as well. You create a case by clicking on a button in the Copyscape frame that’s visible when you visit a similar site. Once you create a case, you can add notes to it, close the case and reopen it. However, there isn’t much else you can do. Closed cases remain in the inbox, there are no sorting tools, and the notes are not lengthy enough to add any correspondence or information beyond a few words.

A simple spreadsheet or, even better, a full database could serve this purpose significantly better. It is convinient, but ultimately useless.

All in all, I spent about 20 of my 100 searches. I discovered no actual plagiarists and the vast majority of my results came back with no matches found. That is especially worrisome since my text searches should have, in theory, turned up my own site at the very least. After all, I only blacklisted my own domain for the last few searches.

Fortunately, the process only cost me $5 ($1 in actual searches) but it did eat up over one hour’s worth of time spread across two days. In that time, I could have easily set up dozens of Google Alerts and received automated, free and effective plagiarism detection.

Conclusions

In the end, there is very little good that I can say about Copyscape Premium. The results were ineffective, the tracking system was almost completely useless and the entire service seemed inept at finding plagiarists even when they were hidden in plain sight.

I really wanted to enjoy and rave about this system. I wanted it to be the great comeback for Copyscape in my eyes. Sadly, it was nothing of the sort.

The price point is great, the searches are lightning quick and the idea itself is wonderful. If Copyscape Premium actually did what it promised to do, it would easily be the best solution available right now.

However, in its current state, I can not recommend it at all. It is nothing more than a waste of money and time.

Hopefully, one of the companies that is developing plagiarism-detection products will step up and produce a real service that will fill this obvious void. The technology seems to exist, we’re just waiting for someone to put together all of the pieces.