The site currently is just a “parked” domain page running ads for the domain’s registrar, GoDaddy. According to the whois information for the site, the domain was “updated” on the eighth, indicating that it possibly expired and was transferred to another owner.

Workfriendly attempted to disguise Web surfing as a Microsoft Word document by formatting Web pages to appear as text in a Word file while bordering the site content with a fake border designed to look like the application. This was supposed to make it “safer” to surf at work as it would raise less suspicion should anyone see your monitor.

The site created problems, however, when it allowed search engines to index its modified pages, injecting many thousands of of pages worth of duplicate content into Google. It also created headaches by not obeying certain meta tags, causing links to break on some sites and for Google to report those errors as broken links on the original domain.

It is unclear at this time if the outage is temporary or permanent, however, the site has been down for at least two days, making a temporary outage increasingly unlikely.

Hat tip: Special thanks to David Bradley of Sciencebase (stupid typos, thanks for the catch!) for letting me know that Workfriendly is not working

The site, which is simply a script that modifies other sites to look like a document in Microsoft Word, so that one can surf the Web at work without raising suspicion, has nearly a quarter of a million URLs referenced in Google, even though only one page, the home page, contains original content.

However, I recently discovered that Workfriendly has another issue with it, one that causes, in some cases, both users and the search engines to seek out nonexistant URLs, causing 404 errors in very large numbers.

Though it is a problem caused by Workfriendly, it is one that Webmasters and bloggers need to take action to correct if they are vulnerable. Otherwise, the search engines could be steered toward hundreds of non-working URLs on your site, potentially hurting your ranking in them.

Discovering the Problem

I discovered the problem with Workfriendly over the weekend by accident. I logged into my Google Webmaster Tools account to check on any errors I had and was stunned to find over 150 file not found errors.

WordPress typically does a pretty good job avoiding file not found errors so to discover so many on my site, especially with no other errors found, was surprising.

Thinking that, perhaps, my recent update had caused an issue with my permalinks, I looked at the errors themselves. One was caused by me changing the date on a post, another was a server error where the URL worked fine, but the other 149 pointed to a directory that does not and has never existed on this server “/browse/Office2003Blue/”.

I remembered that Workfriendly used a similar link structure when you browsed the Web through it. I hopped onto the site and pulled up Plagiarism Today and watched as Workfriendly pulled up the site successfully. Clearly, the ban I had put in place a few months ago had stopped working, likely due to the plugin I was using not being compatible with newer version of WordPress.

After pulling up Plagiarism Today in Workfriedly, I hovered my mouse over one of the links and looked at the URL, indeed, it was pointing to URLs on this server in the non-existant “browse” directory. Clicking the link resulted in chaos in Workfriendly and, in most cases, led to the site loading up without Workfriendly’s obfuscation.

I immediately set out to block Workfriendly, this time using a hand-coded .htaccess block, but not before trying to figure out what was causing the problem.

Understanding the Issue

What made the problem perplexing was that it seemed to only be this site that was having the issue. Other sites I tested with Workfriendly worked fine.

However, after I looked at the source code for the page that Workfriendly created, the problem became almost immediately clear.

Plagiarism Today uses a “base” meta tag. It is a tag used to tell search engines and Web browsers what the “base” URL of your site is so that, when you use relative links (links that do not begin with an “http://”), the browser knows what URL you are pointing to.

It is a good practice for SEO reasons and to help with preventing 302 hijacking. Still, most sites do not have one and, in many cases, it isn’t necessary.

The problem was that Workfriendly, despite having manipulated all of the links on my site, was using relative links for everything. Rather than saying “http://www.workfriendly.net/browse/…” the links simply said “/browse/…”.

When it was combined with the base tag by the browser, that converted all of the links to “http://www.plagiarismtoday.com/browse/…”, a link that does not exist.

The combination of the base tag and Workfriendly’s use of relative links was causing the site to throw back URLs that did not exist and, due to the poor use of robots.txt, causing the search engines to pick up those bad links as well.

An Inconsiderate Script

My issue with Workfriendly has never been the service itself. Though some could argue that it creates a derivative work of the sites it processes, since the works are never saved, but are rather created dynamically, it is a difficult case to make.

However, more to the point, I am not upset about sites that want to remix or alter the site to make it easier to read. I would not oppose a version better suited for the visually impaired, for mobile browsers or other formats as needed, so long as the site showed basic respect for the content it was displaying.

And that is the problem with Workfriendly. The service shows no consideration for the Webmasters whose content it uses.

For one, the site allows the search engines to index the scraped pages, even though the pages do not exist and are, instead, dynamically-generated.

Second, sloppy programming on the site causes it to generate artificial 404 errors that could hurt Webmasters when dealing with the search engines. Fortunately though, since the bad links are on an external site, they likely won’t have much impact.

However, if Workfriendly had simply used a correct link format, including the “http://www.workfriendly.net” before each link or stripping out the Base tag, the issue would not be a problem at all.

But what is perhaps strangest of all is that Workfriendly offers you a script that you can put on your site to direct your visitors to their version of your site. However, in addition to letting your visitors use the Workfriendly service, you may be helping the search engines find your content in their links.

It seems unlikely that is worth the trade off.

Conclusions

Personally, I decided it was time to be done with Workfriendly. I edited my .htaccess file and have banned the server from accessing this site. So far it is the only IP to be completely banned from this domain. If you attempt to access the site from Workfriendly, you will get the message displayed to the right.

If anyone is looking for the code I added to my .htaccess file, I simply put this before any of my WordPress code:

order allow,deny
deny from 66.226.27.21
allow from all

This certainly isn’t the type of steps I wanted to take, but it was I felt I was forced to do and, sadly, what I have to encourage others to look at doing to.

But the problem is that, in their bid to create something simple and fun, the creators of Workfriendly made something that poses a real danger to Webmasters and bloggers. Though simple changes to the system could remedy these problems easily, the authors have either neglected or refused to do so.

The result, on this site at least, is that Workfriendly is banned. I have attempted to contact the creators several times in the past but have never received a response. Considering all of the attention that has been paid to scraping issue, it seems that either the creators are ignoring the criticism, or have abandoned the project.

Either way, right now Workfriendly is just another problem for Webmasters and bloggers to worry about.

The idea is pretty simple, you punch in a URL that you want to visit and workFRIENDLY pulls up the site in a format that resembles a Microsoft Word document (see Blog Herald on workFRIENDLY). The idea is that, if you use the site to surf the Web while at work, it will be less suspicious than having a regular browser open should your boss walk by.

However, the simple and somewhat tongue-in-cheek nature of the site belies a potential threat to Webmasters. A Google search of the domain reveals hundreds of thousands of indexed pages, only one of which, the home page, is original in its content. The rest are cached versions of the pages entered into the system.

The result is that workFRIENDLY, probably by accident, has become one of the most prolific scraping sites I’ve seen and certainly one of the best at getting their results listed in Google.

What is Going On

Fundamentally, workFRIENDLY is little different than other proxy services on the Web including Anonymouse and even Google cache. What separates workFRIENDLY from these services is that it modifies the look of the page so that it appears to be in the format of a Word document.

This modification of the text raises copyright questions of its own, especially in light of the Google cache ruling, which hinged in part on Google lack of modification of the original site, and section 512(b)which protects transitory services from infringement claims for hosting infringing material so long as the data is “transmitted through the system or network without modification of its content.”

However, for most Webmasters, it is likely to be a technical issue that raises the most concern.

The problem is that most sites like workFRIENDLY use a robots.txt file to block search engines from indexing the pages viewers pull up. For example, Anonymouse has a robots.txt file that blocks access to their cgi-bin directory, which is where all anonymous browsing takes place. Likewise, Google has a robots.txt that blocks the search directory, which is where it displays its cached copies.

However, workFRIENDLY does not have such a robots.txt file, in fact, as of this writing, the site doesn’t have one at all. This has caused search engines to index virtually everything it can get its hands on, including over a quarter of a million pages according to Google’s admittedly flawed estimate and over one million in Yahoo.

Though not everything workFRIENDLY has ever visited has been indexed, it appears that a good percentage of it has and that the site is not doing anything to stop it. Worse still, some of these pages have started turning up in organic search results, especially for more obscure keywords, and some Webmasters are getting upset about it.

What To Do About It

According to the person who brought this to my attention, who wishes to remain anonymous, she attempted to contact the host of the site, WebHost4Life, but was told that the data used to create the cached copies was stored elsewhere.

However, after further investigation, it appears that workFRIENDLY doesn’t use cached copies at all. Rather, the pages are created dynamically with each visit. I was able to test this by visiting their copy of Plagiarism Today and refreshing the page every few minutes to see if the time of the workFRIENDLY page had changed, even though my site had not.

If the site had used a cache, the timestamp would have stayed the same, as you can see in the images below, it did not.

First capture:

Second capture:

This creates a strange problem in that there is no content that the host of the site can takedown. The site itself is little more than the homepage and the needed scripts to pull down the data for display.

With no page to take down, the host is in a difficult situation. Though they can disable the whole domain, there is no easy way for them to remove just the infringing work. Worse yet, there is no way to simply block workFRIENDLY using meta tags or robots.txt as there is no documentation for their spider.

Instead, the focus shifts to blocking workFRIENDLY from accessing your site, which I was able to achieve using the plugin WP-Ban and the IP address of the site.

You can also edit your .htaccess file to achieve the same effect.

If that is not practical and you find that your search engine results are threatened or usurped by the duplicate content, you can always file a DMCA notice with Google to get the pages removed. You can also report spam results if you see workFRIENDLY urls in the results of other searches you perform.

Though not an ideal solution, it is at least a workable one and can serve as a stopgap until a more permanent answer, or at least some decisive action from Google, takes place.

Conclusions

I want to be clear that I do not think workFRIENDLY is doing anything malicious and, in truth, it may not even be illegal. I believe that they set up this site never expecting these results to be indexed. There is currently no advertising on the site, save the home page, and the site has hardly achieved what one would call great success with the search engines.

Unfortunately, a letter to workFRIENDLY went unanswered for quite some time so I do not have any word from them on this matter.

If the site would simply create a two-line robots.txt file that blocked indexing of the “browse” directory, the whole matter would be resolved in a very short period of time. However, as of this writing, they have not done that.

The result is that this site has become not just a prolific scraper, but one of the most difficult ones to deal with.

But what is worse about this case is the black eye that this gives Google when it comes to dealing with duplicate content. The fact that a site such as workFRIENDLY can, without any real effort, push hundreds of thousands of duplicate pages into the search results is very worrisome.

Hopefully Google can get around to fixing this issue soon. We have enough to worry about with the malicious spammers to spend too much time on those who simply make a mistake.