web crawler

3 Articles

How Film Industry Data Website The-Numbers.com Got Mauled By Bots

July 25, 2026 by 20 Comments

A lot has been made about the increase of automated traffic on the Internet, with the past years LLM-related crawlers having quite literally swarmed the picture here. Not only does this drive up traffic, it also increases load on web servers, whose owners find themselves faced with increased hosting costs. This recently led to The-Numbers.com going offline for a while as automated traffic was quite literally destroying their bottom line.

This saga is covered by [Stephen Follows], who had a chance to talk with the founder and CEO of the site, [Bruce Nash], after the site went basically offline for a few months. Since the website both licenses data for commercial purposes as well as offering the free access on its website, there were accusations of this being a ‘rug pull’.

The site was started in 1997, as a static HTML site on Geocities where [Bruce] provided box office analyses for investment purposes. Since that beginning traffic was generally polite, with human visitors and usually well-behaved search engine crawlers. Then around 2024 the first wave of scraper bots arrived, followed by a larger wave around December of 2025.

Despite implementing a few mitigations, such as LLM-targeted text, the increased traffic and the resulting load on a site architecture that was never designed for this ultimately led to a collapse. One of the major sources of traffic turned out to be from so-called ‘prediction markets’, like Polymarket, whose bots absolutely hammered the site.

Fortunately for [Bruce] and his team they do not rely on the free website for income, but they have had to massively rework the site’s architecture to bring back a semblance of the original features. As noted in the article, the amount of crawling traffic by these LLMs and ‘agentic AI’ tools is exponentially more than that for search engines, which makes this a major challenge.

Issues like these is why services such as Cloudflare are offering blocking features for such automated traffic. After all, unless such traffic is of use to you, you may as well treat it like a DDoS attack and cut it off at the root.

Thanks to [Ben] for the tip.

Trap Naughty Web Crawlers In Digestive Juices With Nepenthes

January 23, 2025 by 45 Comments

In the olden days of the WWW you could just put a robots.txt file in the root of your website and crawling bots from search engines and kin would (generally) respect the rules in it. These days, however, we have especially web crawlers from large language model (LLM) companies happily ignoring such signs on the lawn before proceeding to hover up every scrap of content on websites. Naturally this makes a lot of people very angry, but what can you do about it? The answer by [Aaron B] is Nepenthes, described on the project page as a ‘tar pit for catching web crawlers’.

More commonly known as ‘pitcher plants’, nepenthes is a genus of carnivorous plants that use a fluid-filled cup to trap insects and small critters unfortunate enough to slip & slide down into it. In the case of this Lua-based project the idea is roughly the same. Configured as a trap behind a web server (e.g. /nepenthes), any web crawler that accesses it will be presented with an endless number of (randomly generated) pages with many URLs to follow. Page generating is deliberately quite slow to not soak up significant CPU time, while still giving the LLM scrapers plenty of random nonsense to chew on.

Considering that these web crawlers deemed adhering to the friendly sign on the lawn beneath them, the least we can do in response, is to hasten model collapse by feeding these LLM scrapers whatever rolls out of a simple (optionally Markov-based) text generator.

This Machine Learning Algorithm Is Meta

September 3, 2019 by 20 Comments

Suppose you ran a website releasing many articles per day about various topics, all following a general theme. And suppose that your website allowed for a comments section for discussion on those topics. Unless you are brand new to the Internet, you’ll also imagine that the comments section needs at least a little bit of moderation to filter out spam, off topic, or even toxic comments. If you don’t want to employ any people for this task, you could try this machine learning algorithm instead.

[Ladvien] goes through a general overview of how to set up a convolutional neural network (CNN) which can be programmed to do many things, but this one crawls a web page, gathers data, and also makes decisions regarding that data. In this case, the task is to identify toxic comments but the goal is not to achieve the sharpest sword in the comment moderator’s armory, but to learn more about how CNNs work.

Written in Python, the process outlines the code itself and how it behaves, setting up a small server to host the neural network, and finally creating the webservice. As with any machine learning, you need a reliable dataset to use for training and this one came from Wikipedia comments previously flagged by humans. Trolling nuance is thrown aside, as the example homes in on blatant insults and vulgarity.

While [Ladvien] notes that his guide isn’t meant to be comprehensive, but rather to fill in some gaps that he noticed within other guides like this, we find this to be an interesting read. He also mentioned that, in theory, this tool could be used to predict the number of comments following an article like this very one based on the language in the article. We’ll leave that one as an academic exercise for now, probably.