1:
Internet Archive wants book copyright indemnity like Google

The Internet Archive, famous for the Wayback Machine, which displays old versions of Web sites, is hoping that it can receive some of the same protections Google has with its recent Google Book Search settlement.

Google was sued in 2006 over its book scanning project by the Author’s Guild. The two recently reached a settlement where Google paid out over $125 million for the rights to continue scanning and displaying out of print books. The suit also protects Google in cases of orphan works, works where the author is unknown, should the author come forward later.

It is the orphan works issue that has the Internet Archive worried. They have written a letter to a Federal judge asking to intervene in the Google case, saying that the lack of such protection for them would put them at a disadvantage.

2: $74k Judgment Against Craigslist Prankster

Next up today, Jason Fortuny, the famous Craigslist prankster, has been ordered to pay a total of $74,000 to one of his victims. This includes damages for copyright infringement, damages for invasion of privacy as well as court costs and attorney fees.

Fortuny posted personal ads on Craiglist in various cities pretending to be a woman. As men responded to the ads, he posted their responses, including images, on a site he had created.

Unsurprisingly, he was sued for this behavior and, with the default judgment being filed, it appears that this case is over.

3: CNN makes copyright claim on video critical of reporter’s ‘Tea Party’ interviews

Finally today, CNN seems to have become the latest to step in the “takedown to block unpleasant” speech mess on YouTube, taking down a clip of a reporter being combative with tea party protesters.

The clip had become very popular with conservative bloggers as many felt it showed CNN’s political bias. Though the clip was pulled it has been re-posted in several other places on YouTube as well as on other video sharing sites.

Suggestions

That’s it for the three count today, we’ll be back tomorrow with three more copyright links. If you have a link that you want to suggest a link for the column or have any proposals to make it better. Feel free to leave a comment or send me an email. I hope to hear from you.

Want the Full Story?

Tune in every Saturday morning for the live recording of the Copyright 2.0 Show or wait and get the edited version Monday morning right here on Plagiarism Today.

photo credit: billaday

In order to keep things interesting here, I’ve decided to start an irregular series dealing with your content, copyright law and all things related. This series, entitled “Pop Quiz” will ask seven questions in a particular area.

For those interested in playing, the goal is simple: To be the first to get all the questions right.

If you want to play, simply leave a comment below with your answers to the questions. I’ll confirm the correct answers as soon as practical. The prize for winning, other than a feeling of lukewarm accomplishent, is a link to your home page in the follow-up post, if desired.

If you’re ready to give it a try, this week’s questions deal specifically with your content in Google and how to control where your content appears in Google.

1. What are three ways you can request your site be removed from Google search? (Note: This is your own site or a page on it, not an infringer)
2. What is the email address for the Google DMCA agent? (please only include the part before the @).
3. What, according to Matt Cutts, is the most effective way to report spam to Google?
4. What is the name of the Google service that will email you search results as they are picked up by Google?
5. What is the name of the Google spider, or rather, the name you need to refer to it as when you are trying to block it?
6. What is the meta tag command to prevent Google from displaying a sample of your content in their results pages?
7. What search command lets you see approximately how many pages Google has indexed on a site?

Bear in mind that some of these are designed to be brain-dead simple and others are meant to be a bit more challenging. However, all of them are more than easily researched and looked up.

And yes, it is legal to use Google.

I hope you have fun with this and I look forward to your answers!

Search engines, however, can provide a much better set of statistics than a site’s own timestamps. The only issue is that gleaning the needed information can be difficult. Fortunately, a relatively new Firefox plugin entitled Link Diagnosis helps with that by taking the dirty work out of determining when a page was indexed by Google.

The tool, while not perfect, can be a valuable asset when trying to determine approximately when a page appeared on the Web.

How it Works

Link Diagnosis is actually a robust plugin designed to analyze incoming links to a URL for SEO purposes. However, as one of its “hidden features” it is able to deteremine, approximately, the day the URL appeared in Google.

It works simply by having the user right click the page they want to check, select the “Get Page Age” option and, after a few seconds they are greeted with a JavaScript popup containing the date the script detected the site appeared.

It works by using Google’s INURL command which, when used in conjunction with a date filter, causes Google to display a date by each resulting URL. What the plugin does is take the URL you wish to check, create the search query and then automatically extract the applicable date, thus turning a multi-step process into a one-click solutions.

For anyone seeking to find out the date of a site, this could prove to be both a powerful tool and a good time saver as well.

Why to Use It

There are many reasons why you might want to check out the age of a particular page.

For one, you can use it to check if a spam blog or a plagiarist was indexed by Google before or after your original post (provided it was indexed at all). This can help determine what action you should take against the site.

However, many will also find its non-repudiation services to be very useful. If there ever is a dispute about who posted an article or an image first, this tool can help resolve it by providing an independent view on which went up first.

Though certainly not as accurate as Numly or MyFreeCopyright, using Google is far more accurate than looking at the Web Archive, especially considering that the latter can take over six months to display any information about a URL.

Still, Link Diagnosis is still far from perfect in this area. there are many issues one will have if one tries to rely upon this for non-repudiation.

Limitations

Before you begin to make heavy use of this service bear in mind the following caveats:

1. Google’s Limitations: The biggest issue of using the INURL method is that Google is not always index a site or a page immediately after it goes up. There are often delays. Also, the service can only work with pages already in the Google database, anything that has been blacklisted, either by the creator or by Google, will return no results.


2. URLs and Not Content: The function will tell you when the URL appeared in Google, not the content on the page. For permalinks that may be acceptable but dynamic pages, such as the front page of Plagiarism Today, it can create a problem.


3. Different Owners: Also, the system detects when a URL was first indexed by Google, not who owned it at the time. If a site changes ownership, even if it is taken out of Google during the transition, the date shown for the home page will be long to the original owner.

In short, the tools is subject to the exact same gaming and manipulation that Google and the other search engines are. As such, it can provide some quick and dirty information, especially on permalinks, but should never be taken as the ultimate gospel on the age of a page.

Link Diagnosis is no substitute for a true non-repudiation service and it does not claim to be.

Conclusions

Personally, I find the other features of Link Diagnosis much more compelling than its “page age” feature. Though it is great for a quick analysis, especially of a spam blog permalink, it may not always tell the complete truth or have the information you are seeking.

It is a great analysis tool but it should not be assumed to be the plain truth. There are plenty of ways that it could be wrong.

So, as with every tool, be sure to use it in conjunction with common sense and logic. Have it available, use it if needed, but don’t use it as a replacement for your own judgment.

No tool is that powerful.

Yet, out of the last 40 spam blogs I’ve looked at, 32 of them were located within the United States. The few that remained were in countries such as Iran, Russia and Ukraine, where copyright law makes it hard to stop them.

So why are so many spam blogs American despite the obstacles to setting up shop in the United States? The answers are surprisingly simple but do not bode well for the future of fighting spam here or abroad.

An American Tradition

The beautiful thing about the Internet is that it is possible to set up shop just about anywhere in the world and, in turn, have anyone else in the world come visit you. Geographic borders are meaningless, until you look at legal issues.

Legally-speaking though, the U.S., much like the EU and similar legal climates, seem to be very hostile places for spammers. Not only does copyright law give easy recourse for those who are scraped, but hosts are very well versed at dealing with spam and often take action without prompting.

It seems logical that spammers would start to take their operations and move them overseas, into countries with weaker laws and enforcement. A stable home would mean longer-running spam operations, that would mean more trust with the search engines and, theoretically, more money.

But despite this, spam seems to be an almost purely American tradition. American hosts are rife with spam blogs and there seems to be no rush on the part of spammers to move to other nations. Despite greener pastures, the purveyors of junk are fine working with in the United States and, when one looks at the reasons, it is clear why that isn’t going to change any time soon.

Sticking Around

Spammers aren’t staying within the U.S. to make it easier for American bloggers to shut them down, rather, they have their own interests in mind. Consider the following:

1. Cost: Hosting is extremely cheap in the United States and, with the dollar falling, it is only getting cheaper. For six dollars per month you can get a hosting account that lets you host unlimited domains. It is cheaper to risk having accounts cut than to pay a premium for hosting elsewhere.
2. Free Hosts: The vast majority of free hosts are located within the United States. Spammers that want to target sites such as Blogspot are pretty much forced to stay within the country.
3. Still Vulnerable to DMCA: Even if a spam blog sets up shop in another country, they would still be vulnerable to American laws with regards to the search engines. Since Google, Yahoo!, MSN and Ask are all American companies, a copyright holder can still use the DMCA to effectively blacklist them from search.
4. Search Engine Trust: Though I have not been able to find evidence of this, it seems only logical that search engines would put more trust into sites closer to their searchers. A site hosted in Iran may do well in Iranian searches, but would not likely perform well in the bulk of search results.
5. Cooperative Hosts: Despite the laws in the U.S. that prohibit this, there are still no shortage of lesser-known hosts that will gladly turn a blind eye to spam and copyright infringement. These hosts can typically get away with it because U.S. law makes it so difficult to sue for copyright infringement.

The end result is that the life of a U.S.-based spammer is far from easy, but it certainly is fruitful and, with a little bit of creativity and effort, the obstacles can be easily overcome.

Mitigating the Problem

With the current legal and hosting climate in the United States, there is very little Webmasters can do to outright stop this problem. However, there are steps that we can all take to minimize this issue.

1. Use The Laws: The fact that most spam is within the United States is something of a gift. Most hosts respect the DMCA and will remove infringing works. Taking advantage of that is a powerful start. The DMCA has been around in the U.S. for ten years and most hosts have effective policies for dealing with such notices.
2. Target Search Engines: Hosts that are uncooperative need to be pointed out to the search engines in the form of spam reports. The reason is that, if enough spam is reported in an IP range, Google will start to distrust the entire host and that will affect their bottom line, both driving away spammers and legitimate customers.
3. Use Search Engine DMCAs: Though working with Google may be tricky, once a spam report has been filed sending a DMCA notice to the search engines can decapitate the spam attack, making it useless.

These are not perfect solutions, but they are ways that everyday Webmasters can hit back at the spam problem without having to go through the hassle of finding an attorney.

Conclusions

The U.S. is in no danger of being dethroned as the Web spam king. Though spammers will diversify as the Web becomes even more international, like wolves hiding in a flock, they will follow the rest of us in a bid to fit in.

But as frustrating as this is, it does serve to our advantage and give us some powerful tools to fight back. All it takes is the no-how and willingness to take action.

Fortunately, it seems that more and more are fed up with Web spam and are doing something about it.

The site, which is simply a script that modifies other sites to look like a document in Microsoft Word, so that one can surf the Web at work without raising suspicion, has nearly a quarter of a million URLs referenced in Google, even though only one page, the home page, contains original content.

However, I recently discovered that Workfriendly has another issue with it, one that causes, in some cases, both users and the search engines to seek out nonexistant URLs, causing 404 errors in very large numbers.

Though it is a problem caused by Workfriendly, it is one that Webmasters and bloggers need to take action to correct if they are vulnerable. Otherwise, the search engines could be steered toward hundreds of non-working URLs on your site, potentially hurting your ranking in them.

Discovering the Problem

I discovered the problem with Workfriendly over the weekend by accident. I logged into my Google Webmaster Tools account to check on any errors I had and was stunned to find over 150 file not found errors.

WordPress typically does a pretty good job avoiding file not found errors so to discover so many on my site, especially with no other errors found, was surprising.

Thinking that, perhaps, my recent update had caused an issue with my permalinks, I looked at the errors themselves. One was caused by me changing the date on a post, another was a server error where the URL worked fine, but the other 149 pointed to a directory that does not and has never existed on this server “/browse/Office2003Blue/”.

I remembered that Workfriendly used a similar link structure when you browsed the Web through it. I hopped onto the site and pulled up Plagiarism Today and watched as Workfriendly pulled up the site successfully. Clearly, the ban I had put in place a few months ago had stopped working, likely due to the plugin I was using not being compatible with newer version of WordPress.

After pulling up Plagiarism Today in Workfriedly, I hovered my mouse over one of the links and looked at the URL, indeed, it was pointing to URLs on this server in the non-existant “browse” directory. Clicking the link resulted in chaos in Workfriendly and, in most cases, led to the site loading up without Workfriendly’s obfuscation.

I immediately set out to block Workfriendly, this time using a hand-coded .htaccess block, but not before trying to figure out what was causing the problem.

Understanding the Issue

What made the problem perplexing was that it seemed to only be this site that was having the issue. Other sites I tested with Workfriendly worked fine.

However, after I looked at the source code for the page that Workfriendly created, the problem became almost immediately clear.

Plagiarism Today uses a “base” meta tag. It is a tag used to tell search engines and Web browsers what the “base” URL of your site is so that, when you use relative links (links that do not begin with an “http://”), the browser knows what URL you are pointing to.

It is a good practice for SEO reasons and to help with preventing 302 hijacking. Still, most sites do not have one and, in many cases, it isn’t necessary.

The problem was that Workfriendly, despite having manipulated all of the links on my site, was using relative links for everything. Rather than saying “http://www.workfriendly.net/browse/…” the links simply said “/browse/…”.

When it was combined with the base tag by the browser, that converted all of the links to “http://www.plagiarismtoday.com/browse/…”, a link that does not exist.

The combination of the base tag and Workfriendly’s use of relative links was causing the site to throw back URLs that did not exist and, due to the poor use of robots.txt, causing the search engines to pick up those bad links as well.

An Inconsiderate Script

My issue with Workfriendly has never been the service itself. Though some could argue that it creates a derivative work of the sites it processes, since the works are never saved, but are rather created dynamically, it is a difficult case to make.

However, more to the point, I am not upset about sites that want to remix or alter the site to make it easier to read. I would not oppose a version better suited for the visually impaired, for mobile browsers or other formats as needed, so long as the site showed basic respect for the content it was displaying.

And that is the problem with Workfriendly. The service shows no consideration for the Webmasters whose content it uses.

For one, the site allows the search engines to index the scraped pages, even though the pages do not exist and are, instead, dynamically-generated.

Second, sloppy programming on the site causes it to generate artificial 404 errors that could hurt Webmasters when dealing with the search engines. Fortunately though, since the bad links are on an external site, they likely won’t have much impact.

However, if Workfriendly had simply used a correct link format, including the “http://www.workfriendly.net” before each link or stripping out the Base tag, the issue would not be a problem at all.

But what is perhaps strangest of all is that Workfriendly offers you a script that you can put on your site to direct your visitors to their version of your site. However, in addition to letting your visitors use the Workfriendly service, you may be helping the search engines find your content in their links.

It seems unlikely that is worth the trade off.

Conclusions

Personally, I decided it was time to be done with Workfriendly. I edited my .htaccess file and have banned the server from accessing this site. So far it is the only IP to be completely banned from this domain. If you attempt to access the site from Workfriendly, you will get the message displayed to the right.

If anyone is looking for the code I added to my .htaccess file, I simply put this before any of my WordPress code:

order allow,deny
deny from 66.226.27.21
allow from all

This certainly isn’t the type of steps I wanted to take, but it was I felt I was forced to do and, sadly, what I have to encourage others to look at doing to.

But the problem is that, in their bid to create something simple and fun, the creators of Workfriendly made something that poses a real danger to Webmasters and bloggers. Though simple changes to the system could remedy these problems easily, the authors have either neglected or refused to do so.

The result, on this site at least, is that Workfriendly is banned. I have attempted to contact the creators several times in the past but have never received a response. Considering all of the attention that has been paid to scraping issue, it seems that either the creators are ignoring the criticism, or have abandoned the project.

Either way, right now Workfriendly is just another problem for Webmasters and bloggers to worry about.

At the forty minute mark in the presentation, Cutts said something that was interesting to those of us who deal with spam blogs but has been largely overlooked. When discussing Google’s Webmaster Center, he mentioned that you can report spam through their Webmaster Tools feature and that they “give more weight” to those reports than the ones made through their public form.

In short, if you have access to Google’s Webmaster Tools, which is free and easy to register for, you can use the form in there to file a more meaningful spam report. Best of all, the form is identical to the public one and and should not seem foreign to anyone used to filing spam reports.

This is assumedly because the spam form in the Webmaster Tools is not anonymous, unlike the public one. Google, understandably, gives more significance to reports where they know the party providing the information.

To file the report, simply log into the Webmaster tools dashboard and click the “Report spam in our index” link on the right hand side. You report paid links.

This may resolve many of the claims that Google does not respond (see comments) to spam reports.

All in all, while this is a very simple trick, it might help with the reporting of spam in cases where a DMCA notice is simply not practical.

Note: In a strange coincidence, I found the video on John Pozadzides blog, who will be speaking directly before me at WordCamp Dallas.

However, unlike most victims of scraping, she knew exactly why her blog was being targeted heavily, she had recently read an article about the highest-paying AdSense keywords and knew that “Austin DWI” was commanding, at that time, well over $80.00 per click.

The results were predictable. As the term rocketed up in value, scrapers and other spam bloggers settled in to try and lure visitors for lucrative clicks and, along the way, ended up grabbing her keyword-rich content.

Her story illustrates a strange problem when dealing with spam blogs, that it is impossible to predict exactly where and what they will seize upon. They are, like anything else on the Web, highly prone to trends and changes in the climate making it virtually impossible to guess what will and will not be targeted tomorrow.

EBay on Steroids

According to Spencer, the spike in the price of the keyword was caused by a “bidding war” between attorneys in the Austin area. Since many lawyers in the region use Google to promote and they each wanted to be number one in the bidding, like any other online auction, they kept raising the price for the keyword until the cost they paid was far higher than what the term was actually worth.

The keyword, according to CyberWire, has come down in price, to approximately $55, still well above the average keyword cost and still in the top twenty.

However, these kinds of bidding wars can happen anywhere to almost any keyword. All that it takes is for two or three well-financed advertisers to seek out the top spot and keep trumping one another until they drive the price up past what the keyword is worth.

Though these bidding wars usually flare up and flame out quickly, they can last more than long enough for spammers to target the term and start scraping related content. This means that, if you happen to have a blog or site in the area, while your AdSense revenue may go up if you run ads, your content is likely to be scraped and reposted much more than usual, putting your search engine ranking in jeopardy.

The end result, your site can, almost overnight, go from experiencing only a minimal amount of content theft to being a prime target.

All it takes is a bidding war between a couple of advertisers in your field.

The Good News

Fortunately, while these bidding wars seem to be fairly common, they rarely lead to increased scraping.

The first reason for that is that the bidding rarely reaches heights that would attract the attention of scrapers. Though a leap in price from one dollar to four dollars per click might quadruple your AdSense revenue while the bidding war is going on, it will not attract many new spammers as there are still many keywords that are much more valuable.

The second reason is that spammers do not base their decision on what to target by keyword cost alone. If you look at the list of most expensive keywords, you see that the vast majority are for attorneys. But while spam about legal issues is present and common, it pales in comparison to gambling, pornography and other traditional spam targets.

The reason for this is that high-paying keywords are worthless if the terms are not regularly searched for and visitors are not likely to click the ads. A spammer can make more money off of fifty $2 clicks than just one $80 click.

The keywords that spammers target will have a balance between price, search frequency and click ratio. A bidding war may motivate spammers to target a borderline term they wouldn’t have otherwise, but only in the most extreme conditions would cause them to take heavy interest in a keyword that was, previously, completely ignored.

Conclusions

The bottom line for bloggers is three-fold.

1. All Blogs are Scraped: All blogs, regardless of their content and popularity, are going to be scraped at some point. It is an inevitability.
2. Some Blogs Will Be Targeted: Some blogs, such as those in traditional spam fields, will be heavily targeted as their content is more appealing. They may see many times the scraping of blogs in other fields.
3. Price Fluctuations Can Impact Scraping: Finally, price fluctions, both up and down, can impact the amount of scraping your site sees. Though the price is not the primary determiner of whether your blog will be targeted or not, it can cause problems in some circumstances.

As a Webmaster, even if you do not advertise, you should probably be at least somewhat aware of the value of your keywords and at least understand if they would pay well or not. Doing so may not help you prevent scraping but it at least gives you an idea of how much you should be on the lookout for it and how much time you should dedicate to it.

After all, knowing your enemy is half the battle and there is no better way to understand how they might respond than by knowing what they want and how they plan to obtain it from you…

Bloggers, in turn, have used these tools to keep on top of the various topics, check for misuse of their content, seek out related posts and look for other sites referencing them.

Many even use the tools without realizing. WordPress users, for example, use blog search every time they load their dashboard and see their “incoming links”. Others use it when they embed Technorati Tags into their posts.

However, blog searching is not working and it is getting worse. Where it once returned respectable results, today it throws out far more noise than signal. This has made it almost impossible to use the blog search for almost any practical use, making the only effective use of the technology to search for content theft and, even then, only in certain situations.

Unless something is done to fix blog search, it is only a matter of time before the technology is left completely by the wayside.

My History

When I first started Plagiarism Today nearly three years ago, I started using blog search extensively. I subscribed to a series of related Technorati watchlists and used my RSS reader to keep track of the latest happenings in the field of copyright and content theft. As time went on, I added RSS feeds form other search engines including Icerocket and Google Blog Search.

Initially the system worked pretty well. Though there were some garbage results, for the most part every item in my RSS reader warranted opening in a browser. However, over time, the amount of noise grew at a much faster rate than the number legitimate posts, eventually outpacing it three or four times over.

Now, I feel as if I am inundated by these watchlists, getting dozens of results per hour, but only a fraction of which are actually original, human-written articles.

As a result, I spend as much time per day filtering out the junk from my feeds as I do responding to and bookmarking legitimate articles.

The system has become the model of inefficiency and is borderline useless. However, looking through my feeds, I think I know where much of the noise is coming from and why this problem has crept up on us.

Issues With Blog Search

When performing an autopsy on my blog search results recently, I noticed that there were six types of posts that were cluttering up my feeds and suffocating the original content.

1. Spam Blogs: Splogs are an easy target to blame for blog search clutter but they are not the worst players, truth be told. Since most simply parrot posts already available elsewhere in the feed, they are easy to ignore. Still, the nature of spam blogs pretty much ensures that every post I want to read is repeated two or three more times over the coming hours and days.
2. Non-Blogs: When blog searching started, pretty much only blogs had RSS feeds. Today, RSS feeds are included in forums, twitter pages, social networking profiles and more. Sadly, all of these things are routinely showing up in blog search results. While often relevant in terms of keywords, they are difficult to comment on and rarely provide any news.
3. Comment Feeds: It seems that none of the blog search engines have been good at recognizing the difference between the main feed and the comment feed. I routinely see “Comment On” posts in my RSS reader. Typically they are posts I’ve already posted a comment to or the post is very old and is simply receiving a trickle of comments or trackbacks.
4. Old Posts: It is a strange but increasingly common issue where the results that get returned are anything but timely. I regularly get posts marked as “new” that are really several weeks old. This morning, for example, one of my blog search feeds had two posts talking about the candidated “fighting for Texas” and reporting the allegations of plagiarism against Oboma as new. The posts were dated in late February.
5. Repeated Posts: I have RSS feeds from three of the major blog search engines so I fully expect to see the same post a few times on the different feeds. However, I regularly see the same post repeated on the same feed, sometimes even days or weeks later. Many times I click on a post, thinking it is new, and am stunned that I commented on it weeks ago and a quick check shows that it showed up on the same feed when it was brand new.
6. Foreign Language Posts: Finally, even though my search terms are in English, I regularly get results where the bulk of the post is in another language. Though these blogs are often merely spam blogs using automatic translation, they are still of little use to me as I can’t read them, at least not without some translation, which none of the blog search engines seem to provide.

This is not to say that all of the search engines suffer from the problems above equally. Google Blog Search, for example, is inundated with spam blogs and non-blog results, Technorati is worst about repeating posts and delivering content too late while Icerocket seems to return a large number of foreign language posts.

Of course, these aren’t the only things that have been limiting the usefulness of my blog search tools, but they are five of the biggest players to be certain. If I were going to start fixing blog search, these are the places I would start.

Stinky Swiss Cheese

Of course, the problem blog search isn’t just that its results are cluttered. I’d be willing to deal with a large amount of clutter in order to stay on top of the relevant news. Unfortunately, it is a regular occurrence that I miss stories, including those that the feed should have picked up.

I can offer no explanation for why this has happened, many of the sites are available if you perform a manual search so there should be no reason they are missed.

Fortunately, since I do use mulitple search engines, the problem is not as severe as if I had used only one. But even with all of these layers, some sites do not show up and the problem baffles me.

Of the three I use regularly, Icerocket seems to be the worst at picking up all of the articles I need, even regularly missing updates from Plagiarism Today, but the problem, most of the time, seems somewhat random and hard to pin on one or two engines.

Effects on Content Theft

When it comes to detecting content theft, many of the most popular tools, including Copyfeed and the Digital Fingerprint Plugin, both use blog search engines to detect misuse of the RSS feed.

In that regard, the fact that spam blogs and duplicate posts routinely show up in blog search engines is a good thing. It means that the detection is more reliable and that the odds of the bad guys showing up is good.

Unfortunately, the fact that they can reliably appear in the results also encourages spam blogging as an activity. Though blog search engines are not the ultimate target of sploggers, the fact that junk content routinely gets indexed in them does not discourage spammers. This means that, while we will see a large percentage of the scrapers, it means that there will be many more of them.

Furthermore, with the other issues raised with blog search engines, it appears that we’ll be stumbling through the clutter far more than we’ll be battling the bad guys. This could, in the long run, actually hinder our ability to locate and deal with spam bloggers, by not only throwing up excesses of noise, but making it impossible to target the ones that pose the greatest threat.

Conclusions

What is striking is how different the blog search results are compared to the traditional search engines. Though Google and Yahoo! may take a few more days to pick up their results, they are almost completely free of spam blogs and seem, overall, to present relevant content in an organized manner.

Where Google Blog Search is a dismal failure, the main Google search engine is a triumph. This shows not just the different challenges each search engine faces, but a lack of focus and effort on creating the best blog search results possible.

It is clear that blog search has not been given the attention it needs to thrive. The companies involved, even those who have blog search as their sole business, have decided that it is not worth dedicating the resources toward fixing these issues.

Sadly, these are not new problems that only unveiled themselves recently, they have been ongoing issues that have simply grown larger and larger to the point that they now drag the entire system down. Where once they were minor nuisances begging to be nipped in the bud, now they are huge problems that feel almost too big to tackle.

Sadly, it is only a matter of time before blog search becomes completely useless, if it isn’t there already, and it is a problem that many of us have seen coming for a very long time.