Update: ScrapeGoat Responds
ScrapeGoat, the service custom-scraper service mentioned previously here, responded to the article earlier today. In a phone call with the Aaron Willis, one of the staff at ScrapeGoat, he clarified both the his company’s policy and their product.
First, according to Willis, the company does not do any content theft work. If they feel that a job might run afoul of copyright laws, they ask the person requesting the job for written permission. They have turned down several jobs, including one that arose from the previous article, due to concerns about copyright.
He went on to say that most of their work is done with the blessing of the copyright holder. They do a lot of work for affiliate sites, helping them scrape, legitimately, from the site they are promoting, and for government agencies. Sometimes they are even called upon to scrape content that a company already owns but can’t access because it is in the wrong format.
In regards to the terms of service (TOS) on the site, which forbids scraping of ScrapeGoat’s own site, Willis said that the TOS was a stock template, not one they had written. They will be changing the TOS shortly to avoid any confusion.
Willis seemed to understand the confusion though and apologized for not getting back in touch. Apparently my letter to them was eaten up by their spam filters, which his email reply does seem to validate. (Note: His email reply was sent this Wednesday, after they rescued my letter from their spam filters, and was subsequently chewed up in my own spam filters).
All in all, he answered my concerns very thoroughly and was very patient with me as I asked some very tough and pointed questions. It definitely appears to me now that ScrapeGoat is “above the boards” and legitimate. I apologize for the confusion.
However, this does point to a significant problem with scraping software. It is a very powerful tool and can be used for a great deal of good or a great deal of evil. It’s getting harder and harder to tell the good guys from the bad guys as both tout their scrapers much the same way. If I were a marketer for a legitimate scraping application or service, I would be very worried about this.
The term “scraper”, much like the word “hacker”, have earned such a negative reputation that the original meaning is almost completely lost. It will be interesting to see how this is handled in the months ahead, especially if more black hat scrapers garter public attention.
Want to Reuse or Republish this Content?
If you want to feature this article in your site, classroom or elsewhere, just let us know! We usually grant permission within 24 hours.