Comments on: Scraping Starts from the Very First Post http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/ Content Theft, Plagiarism, Copyright Infringement Fri, 10 Feb 2012 02:15:56 +0000 hourly 1 By: An Eclectic Mind » Interesting Links, May 2007 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/comment-page-1/#comment-130119 An Eclectic Mind » Interesting Links, May 2007 Wed, 12 May 2010 01:22:41 +0000 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/#comment-130119 [...] Scraping Starts from the Very First PostA fascinating experiment to see how quickly sploggers start scraping a feed. The results are rather shocking. On PlagiarismToday. [...] [...] Scraping Starts from the Very First PostA fascinating experiment to see how quickly sploggers start scraping a feed. The results are rather shocking. On PlagiarismToday. [...]

]]> By: PlagiarismToday » A Scrape of a Scrape http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/comment-page-1/#comment-62640 PlagiarismToday » A Scrape of a Scrape Tue, 07 Aug 2007 15:08:09 +0000 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/#comment-62640 [...] get asked by reporters and bloggers alike exactly how bad scraping is on the Web. I discuss my past experiments on the topic and how, depending on your keywords, suspicious traffic starts showing up with the [...] [...] get asked by reporters and bloggers alike exactly how bad scraping is on the Web. I discuss my past experiments on the topic and how, depending on your keywords, suspicious traffic starts showing up with the [...]

]]> By: engtech @ internet duct tape http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/comment-page-1/#comment-59811 engtech @ internet duct tape Thu, 31 May 2007 20:11:46 +0000 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/#comment-59811 I think you're misinterpreting your data source. The wordpress.com feed stats always follow the ebb and flow of your posting frequency. I have a popular wordpress.com blog, and my feed readers are split between the wordpress.com feed (http://engtech.wordpress.com/feed or http://internetducttape.com/feed) and the FeedBurner feed (http://feeds.feedburner.com/engtech). Wordpress.com doesn't let me redirect to my feedburner feed. About 624 of my readers are in FeedBurner, there's another 400-700 who grab the feed directly. Here are screenshots of my stats from wordpress.com and from FeedBurner. As you can see, there are serious discrepancies. I trust the FeedBurner stats much more. http://i115.photobucket.com/albums/n296/engtechwp/special/feedburner.png http://i115.photobucket.com/albums/n296/engtechwp/special/wordpresscom-feeds.png http://i115.photobucket.com/albums/n296/engtechwp/special/post-freq.png To make it worse, the wordpress.com stats seem to be pretty dumb in that they count feed reader hits even if it's just someone clicking on your link from another feed. Not an issue for this experiment, but something to note. Bottom line: no conclusions can be drawn from using wordpress.com feed stats. Set up a blog somewhere that let's you use FeedBurner stats and you'll have a *much* better data sample. Interesting idea, but the data you're basing it off of is so questionable to start with. I think you’re misinterpreting your data source. The wordpress.com feed stats always follow the ebb and flow of your posting frequency.

I have a popular wordpress.com blog, and my feed readers are split between the wordpress.com feed (http://engtech.wordpress.com/feed or http://internetducttape.com/feed) and the FeedBurner feed (http://feeds.feedburner.com/engtech). WordPress.com doesn’t let me redirect to my feedburner feed.

About 624 of my readers are in FeedBurner, there’s another 400-700 who grab the feed directly.

Here are screenshots of my stats from wordpress.com and from FeedBurner. As you can see, there are serious discrepancies. I trust the FeedBurner stats much more.

http://i115.photobucket.com/albums/n296/engtechwp/special/feedburner.png
http://i115.photobucket.com/albums/n296/engtechwp/special/wordpresscom-feeds.png
http://i115.photobucket.com/albums/n296/engtechwp/special/post-freq.png

To make it worse, the wordpress.com stats seem to be pretty dumb in that they count feed reader hits even if it’s just someone clicking on your link from another feed. Not an issue for this experiment, but something to note.

Bottom line: no conclusions can be drawn from using wordpress.com feed stats. Set up a blog somewhere that let’s you use FeedBurner stats and you’ll have a *much* better data sample.

Interesting idea, but the data you’re basing it off of is so questionable to start with.

]]> By: engtech @ internet duct tape http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/comment-page-1/#comment-121791 engtech @ internet duct tape Thu, 31 May 2007 20:11:46 +0000 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/#comment-121791 I think you're misinterpreting your data source. The <a href="http://wordpress.com">wordpress.com</a> feed stats always follow the ebb and flow of your posting frequency.<br><br>I have a popular <a href="http://wordpress.com">wordpress.com</a> blog, and my feed readers are split between the <a href="http://wordpress.com">wordpress.com</a> feed (<a href="http://engtech.wordpress.com/feed">http://engtech.wordpress.com/feed</a> or <a href="http://internetducttape.com/feed">http://internetducttape.com/feed</a>) and the FeedBurner feed (<a href="http://feeds.feedburner.com/engtech">http://feeds.feedburner.com/engtech</a>). <a href="http://Wordpress.com">Wordpress.com</a> doesn't let me redirect to my feedburner feed.<br><br>About 624 of my readers are in FeedBurner, there's another 400-700 who grab the feed directly.<br><br>Here are screenshots of my stats from <a href="http://wordpress.com">wordpress.com</a> and from FeedBurner. As you can see, there are serious discrepancies. I trust the FeedBurner stats much more. <br><br><a href="http://i115.photobucket.com/albums/n296/engtechwp/special/feedburner.png">http://i115.photobucket.com/albums/n296/engtech...</a><br><a href="http://i115.photobucket.com/albums/n296/engtechwp/special/wordpresscom-feeds.png">http://i115.photobucket.com/albums/n296/engtech...</a><br><a href="http://i115.photobucket.com/albums/n296/engtechwp/special/post-freq.png">http://i115.photobucket.com/albums/n296/engtech...</a><br><br><br>To make it worse, the <a href="http://wordpress.com">wordpress.com</a> stats seem to be pretty dumb in that they count feed reader hits even if it's just someone clicking on your link from another feed. Not an issue for this experiment, but something to note.<br><br>Bottom line: no conclusions can be drawn from using <a href="http://wordpress.com">wordpress.com</a> feed stats. Set up a blog somewhere that let's you use FeedBurner stats and you'll have a *much* better data sample.<br><br>Interesting idea, but the data you're basing it off of is so questionable to start with. I think you’re misinterpreting your data source. The wordpress.com feed stats always follow the ebb and flow of your posting frequency.

I have a popular wordpress.com blog, and my feed readers are split between the wordpress.com feed (http://engtech.wordpress.com/feed or http://internetducttape.com/feed) and the FeedBurner feed (http://feeds.feedburner.com/engtech). WordPress.com doesn’t let me redirect to my feedburner feed.

About 624 of my readers are in FeedBurner, there’s another 400-700 who grab the feed directly.

Here are screenshots of my stats from wordpress.com and from FeedBurner. As you can see, there are serious discrepancies. I trust the FeedBurner stats much more.

http://i115.photobucket.com/albums/n296/engtech…
http://i115.photobucket.com/albums/n296/engtech…
http://i115.photobucket.com/albums/n296/engtech…

To make it worse, the wordpress.com stats seem to be pretty dumb in that they count feed reader hits even if it’s just someone clicking on your link from another feed. Not an issue for this experiment, but something to note.

Bottom line: no conclusions can be drawn from using wordpress.com feed stats. Set up a blog somewhere that let’s you use FeedBurner stats and you’ll have a *much* better data sample.

Interesting idea, but the data you’re basing it off of is so questionable to start with.

]]> By: JB http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/comment-page-1/#comment-59718 JB Tue, 22 May 2007 20:07:25 +0000 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/#comment-59718 Elf's DH, I'd heard of that but had not seen an actual case of it taking place. Sure I've gotten the spam with the text in it, but I've never seen my own work used in that way. Sadly though, you may be very right. If that's the case, the odds of me finding this text is slim to absolutely none. Elf’s DH,

I’d heard of that but had not seen an actual case of it taking place. Sure I’ve gotten the spam with the text in it, but I’ve never seen my own work used in that way.

Sadly though, you may be very right. If that’s the case, the odds of me finding this text is slim to absolutely none.

]]> By: Elf's DH http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/comment-page-1/#comment-59717 Elf's DH Tue, 22 May 2007 20:00:50 +0000 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/#comment-59717 <i>I have done several searches for the scraper sites but have had no luck in locating them.</i> Not all scraping is done for splogs. I've gotten (and I'm sure everyone else has gotten) spam emails that have scraped sentences from random websites in order not to be filtered out as gibberish by Bayesian filters. (A particularly amusing one I got reconstituted the descriptions of birds from the Audubon Society). I have done several searches for the scraper sites but have had no luck in locating them.

Not all scraping is done for splogs. I’ve gotten (and I’m sure everyone else has gotten) spam emails that have scraped sentences from random websites in order not to be filtered out as gibberish by Bayesian filters. (A particularly amusing one I got reconstituted the descriptions of birds from the Audubon Society).

]]> By: JB http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/comment-page-1/#comment-59716 JB Tue, 22 May 2007 19:56:20 +0000 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/#comment-59716 Morin, As I said in the article, I'm not sure. There are two things that do disturb me, the first is that Wordpress could not identify most of the feed readers. I would thing that it would recognize one from an obvious source such as Weblogs. Second, those it DID identify were listed as "Web Browsers" and there should not have been any human subscribers to the feed (I didn't even subscribe). Many scrapers hide their bots by having them identify themselves as Web browsers, it is a well-known trick. I would say that about 80% of the subscribers were listed as "unknown" and the rest were Web Browsers. I wish I had taken a screenshot of that as well but I was in a rush due to the move. I might reignite the experiment later today and see what happens. WillMacc, Thanks for providing further confirmation to my theory. If you have any statistics on that, I would love to see them, perhaps we should work together and form a more thorough study? This was just quick and dirty to get a feel for the problem. Obviously more research needs to be done as the problem is greater than even I imagined... Morin,

As I said in the article, I’m not sure. There are two things that do disturb me, the first is that WordPress could not identify most of the feed readers. I would thing that it would recognize one from an obvious source such as Weblogs.

Second, those it DID identify were listed as “Web Browsers” and there should not have been any human subscribers to the feed (I didn’t even subscribe). Many scrapers hide their bots by having them identify themselves as Web browsers, it is a well-known trick.

I would say that about 80% of the subscribers were listed as “unknown” and the rest were Web Browsers. I wish I had taken a screenshot of that as well but I was in a rush due to the move. I might reignite the experiment later today and see what happens.

WillMacc,

Thanks for providing further confirmation to my theory. If you have any statistics on that, I would love to see them, perhaps we should work together and form a more thorough study? This was just quick and dirty to get a feel for the problem.

Obviously more research needs to be done as the problem is greater than even I imagined…

]]> By: WillMacc http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/comment-page-1/#comment-59714 WillMacc Tue, 22 May 2007 18:41:33 +0000 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/#comment-59714 Also... :) A lot of "pseudo" feeders are attached and monitor other ping services. I've seen countless visits from known crawlers with "bad intentions" hit the site as soon as a ping is sent out. If you have a blog hosted on your own domain, you can issue a ping (to only one service - say; pingomatic) and then sit back and watch who starts hitting the site. You'll see quickly a boat load of crawlers come and a lot of them will not appear as crawlers, but as regular user-agents. If you follow the trends of the crawlers/visitors after a ping, you'll probably start noticing some visitors will not pull any graphics on the blog; or only pull one hit as where most visitors will have line upon line of various content, items, and graphics that's embedded into the blog themes and within the articles. Those that do that are Usually bots and not legit users, but having said that, you'll have to be careful and pick out the rss readers from the bots and crawlers. Thanks, WillMacc Also… :)
A lot of “pseudo” feeders are attached and monitor other ping services.
I’ve seen countless visits from known crawlers with “bad intentions” hit the site as soon as a ping is sent out.
If you have a blog hosted on your own domain, you can issue a ping (to only one service – say; pingomatic) and then sit back and watch who starts hitting the site.
You’ll see quickly a boat load of crawlers come and a lot of them will not appear as crawlers, but as regular user-agents. If you follow the trends of the crawlers/visitors after a ping, you’ll probably start noticing some visitors will not pull any graphics on the blog; or only pull one hit as where most visitors will have line upon line of various content, items, and graphics that’s embedded into the blog themes and within the articles.
Those that do that are Usually bots and not legit users, but having said that, you’ll have to be careful and pick out the rss readers from the bots and crawlers.

Thanks,
WillMacc

]]> By: WillMacc http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/comment-page-1/#comment-59713 WillMacc Tue, 22 May 2007 18:30:53 +0000 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/#comment-59713 The moving of the blog and all that came after really wasn't to see who was scraping; but it did provide somewhat confirmation that probably a lot of those still visiting the site and the site feed was more than likely bots that are still habitually visiting the site/feed daily. I understand that people don't regularly check RSS readers - as in Google Feed Fetcher - but I would dare to guess that's only a small percentage of the visits. Moving the blog also provided me a chance to see who/what visits the site; as where with my WordPress.com blog, I could only see hits and not the actual visit information. Since the move; probably 10 crawlers have been shotdown from scraping the content of the blog, BUT, scraping a blog like mine isn't that big of a deal since it's the information on the blog that's important. So, if my content gets scrapped and ends up up another blog - fine; the information is still valid and people still get to see who's doing what with what and whom.. :) Thanks, WillMacc The moving of the blog and all that came after really wasn’t to see who was scraping; but it did provide somewhat confirmation that probably a lot of those still visiting the site and the site feed was more than likely bots that are still habitually visiting the site/feed daily.
I understand that people don’t regularly check RSS readers – as in Google Feed Fetcher – but I would dare to guess that’s only a small percentage of the visits.
Moving the blog also provided me a chance to see who/what visits the site; as where with my WordPress.com blog, I could only see hits and not the actual visit information.
Since the move; probably 10 crawlers have been shotdown from scraping the content of the blog, BUT, scraping a blog like mine isn’t that big of a deal since it’s the information on the blog that’s important. So, if my content gets scrapped and ends up up another blog – fine; the information is still valid and people still get to see who’s doing what with what and whom.. :)

Thanks,
WillMacc

]]> By: WillMacc http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/comment-page-1/#comment-121786 WillMacc Tue, 22 May 2007 18:30:53 +0000 http://www.plagiarismtoday.com/2007/05/22/scraping-starts-from-the-very-first-post/#comment-121786 The moving of the blog and all that came after really wasn't to see who was scraping; but it did provide somewhat confirmation that probably a lot of those still visiting the site and the site feed was more than likely bots that are still habitually visiting the site/feed daily.<br>I understand that people don't regularly check RSS readers - as in Google Feed Fetcher - but I would dare to guess that's only a small percentage of the visits.<br>Moving the blog also provided me a chance to see who/what visits the site; as where with my <a href="http://WordPress.com">WordPress.com</a> blog, I could only see hits and not the actual visit information.<br>Since the move; probably 10 crawlers have been shotdown from scraping the content of the blog, BUT, scraping a blog like mine isn't that big of a deal since it's the information on the blog that's important. So, if my content gets scrapped and ends up up another blog - fine; the information is still valid and people still get to see who's doing what with what and whom.. :)<br><br>Thanks,<br>WillMacc The moving of the blog and all that came after really wasn’t to see who was scraping; but it did provide somewhat confirmation that probably a lot of those still visiting the site and the site feed was more than likely bots that are still habitually visiting the site/feed daily.
I understand that people don’t regularly check RSS readers – as in Google Feed Fetcher – but I would dare to guess that’s only a small percentage of the visits.
Moving the blog also provided me a chance to see who/what visits the site; as where with my WordPress.com blog, I could only see hits and not the actual visit information.
Since the move; probably 10 crawlers have been shotdown from scraping the content of the blog, BUT, scraping a blog like mine isn’t that big of a deal since it’s the information on the blog that’s important. So, if my content gets scrapped and ends up up another blog – fine; the information is still valid and people still get to see who’s doing what with what and whom.. :)

Thanks,
WillMacc

]]>