Comments on: Shmoocon 2006: Covert crawling: a wolf among lambs

By: jaded

jaded — Tue, 31 Jan 2006 09:12:29 +0000

A lot of people have legitimate needs to crawl a site. Think about a site that carries the text of a book but has a strict “no spiders” policy (so they can shut you off when you stop paying, for example.) If you’re a legitimate user but need an offline copy of the book (for field work or whatever), you’re out of luck. Their server will spot a spider instantly, and shut you down.

But if you have a smart spider that skips around, reads chapters here and pages there, then they’re not likely to notice you or ban you. And you can still get the text of the book you need. Just make sure you delete your local copies of the book once your subscription has ended.

By: Aaron

Aaron — Mon, 30 Jan 2006 21:21:55 +0000

cuba: No point worrying about that now, is there?

By: Cuba

Cuba — Mon, 30 Jan 2006 08:45:16 +0000

Is there a legitimate use for this?

I am not usually one of those people that criticises stuff like this, but do we really want to make email address harvesting and exploit finding easier?

By: Eliot Phillips

Eliot Phillips — Sun, 29 Jan 2006 03:02:01 +0000

No, it isn’t broken because I see at least two comments show up in my email everyday. What is broken is the post not showing more then 250 comments. Sounds like a good enough excuse to me to lock the thread.

By: jared

jared — Sun, 29 Jan 2006 02:32:52 +0000

looks to me like the “most commented on (past 60 days) isn’t working properly. as of today, it’s been 4 months since the psp 2.0 to 1.5 downgrade was posted, and no one has commented on it since Oct 16th, 2005.

Also, very interesting article!