A lot has been made about the increase of automated traffic on the Internet, with the past years LLM-related crawlers having quite literally swarmed the picture here. Not only does this drive up traffic, it also increases load on web servers, whose owners find themselves faced with increased hosting costs. This recently led to The-Numbers.com going offline for a while as automated traffic was quite literally destroying their bottom line.
This saga is covered by [Stephen Follows], who had a chance to talk with the founder and CEO of the site, [Bruce Nash], after the site went basically offline for a few months. Since the website both licenses data for commercial purposes as well as offering the free access on its website, there were accusations of this being a ‘rug pull’.
The site was started in 1997, as a static HTML site on Geocities where [Bruce] provided box office analyses for investment purposes. Since that beginning traffic was generally polite, with human visitors and usually well-behaved search engine crawlers. Then around 2024 the first wave of scraper bots arrived, followed by a larger wave around December of 2025.
Despite implementing a few mitigations, such as LLM-targeted text, the increased traffic and the resulting load on a site architecture that was never designed for this ultimately led to a collapse. One of the major sources of traffic turned out to be from so-called ‘prediction markets’, like Polymarket, whose bots absolutely hammered the site.
Fortunately for [Bruce] and his team they do not rely on the free website for income, but they have had to massively rework the site’s architecture to bring back a semblance of the original features. As noted in the article, the amount of crawling traffic by these LLMs and ‘agentic AI’ tools is exponentially more than that for search engines, which makes this a major challenge.
Issues like these is why services such as Cloudflare are offering blocking features for such automated traffic. After all, unless such traffic is of use to you, you may as well treat it like a DDoS attack and cut it off at the root.
Thanks to [Ben] for the tip.
It’s by definition a ddos. Driving so much automated traffic to one site that it’s unable to operate is a ddos attack. At least when it happens by human traffic to some small site that has content that suddenly gets huge attention it’s understandable, but unfortunate. This is a ddos attack without the specific intention to take the site down, but having that affect anyway.
As someone who runs a reasonably sized site with tens of thousands of user posting valuable content, I can confirm that normal users only make up less than 1-2% of the traffic. On a good day. When it gets really bad, its about 0,02%, the rest are search engines, AI scrapers, etc. And they completely ignore the robots.txt, follow nofollow links, but the worst part is they use huge pools of residential proxies. They can easily make 10 million requests where each comes from a different IP. And for some bizarre reason they download the whole site multiple times daily.
Back many years ago I was an ISO for an organization that had a publicly available database of information that was of use to a wide variety of users from law enforcement to real estate agents to people checking on their friends, neighbors and dating partners. Many entities tried to download the whole thing at once, which of course looked like a DDOS. My joke was that even before IP address checking, I could tell foreign attempts from domestic. The technique of foreign actors in their their programming/scripting was superior. US scrapers used clumsy methods, dowmnloading the entire large database every time, and often didn’t analyze the numbering scheme, leading to literally millions of not founds (8 digit numbering started in the 10s of millions). The Russians, Georgians and Ukrainians far more skilled.They adapted our alphabetical search to get the whole list of targets,, downloaded them all with pictures, and the next time only downloaded new records, becoming undetectable by simple numeric analysis.
It seems brute force tools and the BFMI (Brute Force and Massive Ignorance) techniques have won the day. D a r n glad I retired 12 years ago!!
If they’re disregarding the robots.txt file so brazenly, set a trap for them.
On my system, I have a link on the first page to a target named “dontgohereoryoullbebanned”. Requesting that from my web-server will add the caller’s IP address to the firewall to drop all traffic. Those that respect my robots.txt file don’t fall in that trap.
To implement this, I have NGinX and a little Python webserver that’s NGinX will reverse-proxy to. The Python webserver actually gets all traffic that NGinX can’t identify as a request that it can handle. The Python webserver has a list of URL requests that include the things that bots scanning for exploits look for, along with the poison URL link. Anything on the list; requestor into the firewall. Things not on the list get logged for me to review later to identify new script-kiddy scanning.
Great system. And thanks for sharing.
Not going to work at all. As I said, they use a pool of 10 million IP addresses, it will be extremely rare for one to visit the trap resource. Especially because what they do is each IP only makes maybe 5-10 requests a day to completely random pages.
What works is CF WAF captcha.
Related issue:
https://www.currentaffairs.org/news/who-will-save-the-internet-from-disappearing
Bots are only speeding up the digital decay that was already happening.
Changing the past by editing old newspaper articles was Winston’s job in the book 1984. Now we can do it on the web, much more easily.
A few weeks back I decided to get back into webwork, just a site on vps, but far removed from the old Win NT box I ran a site on 20+ years ago. To my horror within days of getting the domain name connected the log files were filling up with script-kiddy, AI probes, bot attack attempts. It was awful, I don’t even launch till a week or two later. Fire up fail2ban and associated jails, kick ’em off at the firewall level.
Ah right, what if the attack is coming from a NAT in a coffee shop? or a phone network? then I’m aggressively blocking real users too. I am lucky, I only have one variable driven endpoint with a simple expected code pattern from a QR code, and a shortish list of urls, Hence I was able to white-list those urls, mostly return 444s and/or lob 404s into the logfiles and ban, but not a ban for too long.
Log files largely clear now, few more patterns to match ’tis all. Have to say, I used copilot using claude to talk me through all of this new world, but the white-list was still my idea. (“good catch!” it says, yeah, why didnt you think of it, chump?)
It was always the wild-west out there, but this another planet, Wild-Westworld.
lol yeah, in 2014 I had a small mac with an open port (8085 I think) that I just used to log incoming requests. Various windows and linux script attacks, like clockwork. Cant imagine what it would be like now
I would’ve had a little bit of fun with the Polymarket bots and make a honey pot full of false information to screw with them.
Yeah, if you could find a way to feed only the polymarket bots actual data + random noise it would be amusing
Did you perhaps mean “exponentially”?
Ask Wolfram Alpha to plot a logarithm and an exponential on the same scale and you’ll see what I mean.
That’s what I meant, yes. Thanks for the catch :)
This site being human-written sadly means that you lot have to put up with our human flaws :)
In seperate news, the amount of people asking AI to do math and Science when Wolfram Alpha exists. It’s like trying to carry building supplies on a bicycle when Home Depot will rent you a truck 🤦🏼♂️
Lol. I’m old so I don’t care much but the Internet will be utterly f*cked for gen Z, alphas, betas and whatever comes after.
Oh well. Unlike me, they’ll be able to get consuelling, mental health support, thoughts and prayers… and if that doesn’t work, maybe also eat some antidepressants to turn into feelgood zombies for a few hours.
“ok boomer”
Roughly 70-30, non-human to human website visitors, with various Microsoft ASN global locations being by far the worst offenders.
Of the ‘human’ traffic, the better part comprises requests for files I don’t host (vulnerability scans?).
Next up: Polymarket bets on how long a poorly-resourced web site will stay up under sustained attack, er, I mean AI training.
Shameless plug, that’s exactly the use case for my (open source) software botfreak : https://freehackers.org/orzel/botfreak