Rendered at 00:09:53 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
robotmay 2 days ago [-]
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it.
Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.
Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.
embedding-shape 2 days ago [-]
> I have an admin dashboard with a little leaderboard for which ones get the most stuffed
You can't tease us like that without showing screenshots.
robotmay 2 days ago [-]
Haha here's a few snippets (will auto expire in a week, future readers):
No hot toaster pics though :( on second thought perhaps better in our imaginations
altairprime 2 days ago [-]
Yep, those work, thanks!
worldsavior 2 days ago [-]
NGL, that's something a crawler owner would say.
whstl 2 days ago [-]
My team runs a quite popular website, #1 or #2 in the market depending on the region. Several million visits per day.
Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down.
We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got complaints on Reddit from day one, pretty much. At the same time, it got a bit expensive having a WAF.
In the end the only solution that worked was blocking every single Chinese IP.
Traffic instantly got down to 10x less, same as it was before. Same conversion percentage as before.
robotmay 1 days ago [-]
In an odd sequence of events, I've sorta got a weird tiny following in China due to having a vintage camera stall in an antiques market here in Wales that gets shared on Xiaohongshu (https://en.wikipedia.org/wiki/Xiaohongshu) sometimes when I have Chinese customers. So I'd feel bad blocking the entire country myself, though thankfully I'm not being inundated with traffic from there yet - the most egregious bots I've caught so far are actually ones from Meta which ignore robots.txt.
whstl 1 days ago [-]
Oh trust me, we felt really really bad blocking the whole country, and it took weeks of attempts avoiding it.
Meta bots are also problematic, as they seem to be quite buggy. But the volume was nowhere near the hundreds of bots from China :/
robotmay 1 days ago [-]
Yeah Meta's bots are well and truly stuck in the blackhole on this site at the moment. I have no idea what they're supposed to be doing, but in case they are using them for AI work I've just added a token multipler based on subnet totals so they can have some more fun. I might come up with some other fun response types for them too.
rdtsc 19 hours ago [-]
The same trick worked for me 15 years ago to drop a lot of log noise out of constant connection attempts. Felt a bit bad but in the end it turned out fairly well.
leohart 19 hours ago [-]
I am surprised POW Anubis doesn't work for you. Are these bots bypassing POW somehow?
account42 8 hours ago [-]
POW never worked. The only bots its stops are the simple ones that don't support java script or cookies at all.
alok-g 2 days ago [-]
I had to look up what iocaine route is. Here's an article that explains well:
That was somewhat helpful, thanks, but the most interesting aspect for me is how iocaine tries to determine whether a request comes from a bot (since if this can be done reliably, we're done) -- but on that, all the link says is that it "uses a set of logical rules".
GoblinSlayer 1 days ago [-]
That link shows me:
---
Gymnastics trophy two years late, not two years wrote down on the other side of the two thousand millions. Stop all the time was now a.
Anything of this real London, these actual civilized men and women; in uniform batches. The whole incident could not control the beating of a telescreen. Folly, folly, folly! He.
---
jopsen 2 days ago [-]
Probably they must be deduplicating text they've seen before.
The only punishment would be unique text that trains their models to be degenerate. And even then you'd probably have to serve across many domains.
dspillett 2 days ago [-]
One thing I've thought of trying is, if I ever get around to putting things back online, loading the pages with altered versions of themselves in a manner that human users are unlikely to see (though they could if they wanted to) but scrapers are unlikely to ignore (because they look enough like something users might see, perhaps by opening a <detail> tag).
The altered versions would be the page or paragraphs of it, with sentences and words swapped arbitrarily so that training on the data adds as many iffy token sequences as correct ones.
jopsen 2 days ago [-]
To be clear, making pages with bad training data for bots won't make the bots go away.
It'll just punish the bad actors running the scrapers.
As the original poster mentions that they are using TVs as proxies to get residential IPs, one really can't think of these bots as criminal enterprises.
Sadly, if the bad actors has two cents for brain, they'll limit how much importance each domain name can have on training data. To mitigate impact of bad data like this.
(Note I'd suggest only linking to them from robot.txt as pages to not be indexed, that way no human or well behaved not ever will see them, which is kind of the point).
dspillett 2 days ago [-]
> … making pages with bad training data for bots won't make the bots go away.
It'll just punish the bad actors running the scrapers.
Exactly. I can't hope to keep them all at bay, but I can at least have the petty little victory of making their visit less convenient than it might otherwise be.
> if the bad actors has two cents for brain
I suspect that a majority of them are little better than the script kiddies of yore, running tools with minimal understanding of what is actually going on.
> I'd suggest only linking to them from robot.txt as pages to not be indexed
Agreed. Blocking all bots from all pages, well those that bother to listen to robots.txt. All bots because pretty much all of them are scraping for AI and similar these days, even googlebot. If I want people to see my stuff they'll get a link, and maybe they'll pass it on further, but all indexers/trainers can get stuffed. I'll likely make an exception for archive.org and similar.
robotmay 2 days ago [-]
Thankfully this is my photography website so what they're mostly after is images, which they have to request. I'm also intercepting all my legit images once the bot is flagged, which will no doubt have unintended consequences but I'll find out at some point I guess.
solid_fuel 2 days ago [-]
Have you considered releasing any of this as a plug for phoenix or similar?
I’ve also toyed with this approach and have a similar labyrinth on my website, but it just serves back huge amounts of meaningless text one word at a time.
robotmay 2 days ago [-]
If I get some time to un-LLM-ify it then sure! I build all my sites in Phoenix these days anyway so I'll likely want to port it to my other sites at some point too.
I do want to add some more gibberish generation to mine. Once a visitor hits the honeypot route it does replace the rest of the site content with nonsense but I think it'd be fun to have it spewing really daft, really long LLM-generated essays on toasters.
huhtenberg 2 days ago [-]
If you know the IP, it's easy to tar scraper's requests.
However the article mentions that now they are being scraped 4-5 requests at a time from random IPs that are then never reused. It still probably possible to detect scrapers if they start by requesting obscure pages, but in general it's a much more difficult case.
someothherguyy 2 days ago [-]
wouldn't that just make your connection load worse?
robotmay 2 days ago [-]
Currently not really an issue on this site but connections aren't an issue on Elixir usually anyway, unless you get up to about 1 million on one machine IIRC.
inventor7777 2 days ago [-]
You can't tease us with that and not share any information! ;-)
robotmay 2 days ago [-]
I've added a few screenshots above!
Elixir's really fun to do this in because the BEAM will let you have hundreds of thousands of processes sat around doing nothing quite happily, so slow IO and such is something it's weirdly well-suited to.
This is on my photography site so I'm less fussed about them harvesting my writing, and more about causing problems for image harvesters. I'm sure they'll get some stuff anyway but at least one bot got stuck in a 400 page hellhole earlier today so I'm overall very pleased :D
mlaretallack 2 days ago [-]
I hate to ask but what's a "sexy toasters."
robotmay 2 days ago [-]
Gemini's surprisingly bad at it but so far I have some voluptuous curvy toasters, toasters in the shape of a butt, and robot girls with toasters for tits. I have a Gemini subscription for the month after mistakenly thinking I'd get cheap Opencode usage through it, so gotta use it for something.
hahnbee 2 days ago [-]
what constitutes as a bad crawler and what information do you have that's attracting so many bots?
charles_f 2 days ago [-]
Not op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers
robotmay 2 days ago [-]
Yep this is how I'm doing it. It's marked as my super exciting client archive full of high-res RAW photos that all bots are told not to index in robots.txt, only for some weird reason the link never gets rendered to the page :>
I also explicitly tell all crawlers not to index the images on my site, actually, so I feel even better about serving the ones that do complete junk.
Not sure why I get more crawler traffic than I do legitimate traffic tbh - it's a relatively new website that I've been trying to sort out the SEO for as I'm trying to get photography work, so it's probably started cropping up on the radar of scrapers looking for training data.
eloisius 2 days ago [-]
How do you reliably detect the bots? Instead of slowban, I would feed it something abhorrent like an archive of 4chan posts.
robotmay 2 days ago [-]
It's mostly about luring them into URLs they're explicitly told in robots.txt that they shouldn't index. I do some identification via reverse DNS of known crawlers I actually want like Googlebot, though they respect robots.txt, in case something goes wrong and they accidentally get flagged.
mzajc 2 days ago [-]
> Why is git.kernel.org “interesting” to crawlers
I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.
The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.
TonyTrapp 2 days ago [-]
Exactly my observation as well. They devour absolutely everything, no exceptions. No matter how stupid it might be to digest a source code repository via HTTP. They probably don't even recognize what's inside those pages and that there's an easier way to obtain the same result.
jeremyjh 2 days ago [-]
The crawlers are not AI. The crawlers are deterministic. They are collecting data to train AIs.
jonhohle 2 days ago [-]
But gitweb is probably the second most used method of hosting a git repo and easily recognizable through heuristics. If it’s gitweb, fallback to git access and save everyone, including the crawler, time and resources.
jeremyjh 1 days ago [-]
Understood. The comment I replied to suggests the crawlers should figure this out on their own.
TonyTrapp 1 days ago [-]
No, when I said this behaviour is stupid I did not imply that it needs to be solved with "intelligence". This class if problem is already solved by traditional crawlers, AI companies just actively chose to disregard any of that in their race to the top / bottom.
TonyTrapp 2 days ago [-]
So? You can design a deterministic crawler for AI that
- Respects robots.txt
- Detects if something is likely a git forge and as a result
- Clone the git repository instead of trying to crawl the forge
None of this needs to be happening, and yet it does at an unprecedented scale.
2 days ago [-]
diegocg 2 days ago [-]
Which, as the post notes, it's incredibly stupid. So much for artificial "intelligence"
emsign 2 days ago [-]
Makes me wonder how much garbage they actually collect across the web. That can't be good for the quality of the LLM.
vintermann 2 days ago [-]
A lot of work is apparently put into bypassing any kind of anti-scraping, no work is apparently put into figuring out if the site freely gives a way to get all that information in a less wasteful way.
sigbottle 2 days ago [-]
Code (not AI) in general is the worst at semantics. Not an excuse for their practices, but it makes sense they wouldn't try to figure out something more advanced. If they did, their solution would probably be something like, "try all current approaches we have"; it wouldn't be fine grained or truly reasoning at all unless they stuck an actual AI in front of it.
asddubs 2 days ago [-]
I run a non source code service and I also get these, also getting hit by literally millions of residential IPs being cycled after 1 - 2 hits. Ingesting thousands of copies of near identical pages. The bots seem to be programmed in an extremely simple manner, using non-obfuscated javascript to write the href attribute seems to stop them from seeing links, where previously they would endlessly crawl generated links. Unfortunately it also means noscript users get shafted.
xiconfjs 2 days ago [-]
Same here. Have a small gitea instance with cloned projects from github I‘m keeping in case the github version gets removed. Every few days there is an army of bots hammering my small vm with 40k req/min for 20 min straight. I had to install anubis to keep the instance online (while still slowed down).
panki27 2 days ago [-]
I don't even have anything worthwile on my gitea instance, and the same thing happened to me.
The only thing that ended up helping was disabling access for anonymous users.
semiquaver 2 days ago [-]
> because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
This statement holds the core misapprehension behind Anubis. It’s not a ton of cycles. There is no difficulty setting that would be inconvenient for bots but usable for humans on mobile devices.
I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.
Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.
Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task:
> Build an iOS Safari Web Extension that accelerates Anubis proof-of-work using a native C ARM64 SHA-256 kernel. Precompute the invariant 128-byte challenge prefix, search fixed-width decimal nonces with ARM SHA-2 intrinsics and two worker threads, and target difficulty-6 solves under one second. Relay challenges from a Safari content script through the background service worker to native code, then submit the valid nonce/hash through Anubis’s normal pass-challenge endpoint. Include a deterministic benchmark app, correctness tests against CryptoKit, bounded execution, and fallback to Anubis’s stock solver.
wongarsu 2 days ago [-]
Isn't the issue here the three order of magnitude difference between the (presumably optimized) JS implementation and the optimized C kernel on your browser? If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution. Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them
For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this
semiquaver 2 days ago [-]
> I really don't see why Safari on a modern iPhone should be so slow at this
me neither, but I don't think it changes the argument. There's always going to be someone on a low-end device. Your adversaries already have superhuman coding ability and infinite patience. Why would you expect the long-term advantage to be with the defenders?
paytonjjones 2 days ago [-]
In this case, because there's a vastly more efficient economic path for the adversaries (cloning).
They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.
pbronez 2 days ago [-]
It’s more economical at a compute level, but not at the developer level. The moment you start customizing your crawler to use protocol X for site Y your scale story collapses.
paytonjjones 2 days ago [-]
It's a good point, but in practice it depends on how easy those customizations are to implement / maintain, and how much money and effort you save. At some point the compute cost can disrupt even the nicest scale story.
I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.
miki123211 2 days ago [-]
The temptation to offer different / inferior / limited content to scrapers will be too strong, so such solutions are doomed to fail.
This would likely work as "Cloudflare SideChannel", a (hypothetical) Cloudflare product that would let scrapers download the pages that humans actually visit, as they are added to the CF cache. It wouldn't work for the non-Cloudflare part of the internet where humans connect directly to the servers that have their content.
simonw 1 days ago [-]
The moment we establish a standard for offering a "optimized for scrapers" version of a site, people who do not want to be scraped will weaponise that to serve junk to scrapers... and scrapers will subsequently refuse to use it.
inigyou 2 days ago [-]
Why aren't we litigating against scrapers, anyway? DDoS is a felony.
ekidd 2 days ago [-]
Largely because they're residential botnets in places like Brazil (a real example from one of my sites that was crawled to near-destruction). Someone could probably do something about this, but it's out of reach for individual site owners.
inigyou 2 days ago [-]
If you block Brazil, they'll find an alternative, maybe then you can sue them.
afdbcreid 2 days ago [-]
But they operate from neither. You can at most sue the one renting them IP addresses. Which will do basically nothing.
(Also, blocking a whole country is likely not what you do, but you probably know that).
inigyou 2 days ago [-]
Why do you think you can only use the one who's renting them IP addresses?
semiquaver 2 days ago [-]
How good are you personally at iteratively and sequentially jumping through the legal systems of dozens of countries over the course of years, interleaved with genuinely difficult technical investigation, to unmask successive onion-layers of identity in order to unmask one offender?
Oh, and it also only takes a few minutes to reconfigure everything and invalidate those years of legal and investigatory work.
inigyou 1 days ago [-]
Why do you think OpenAI, Anthropic, Bright Data, and Comcast aren't US companies?
miki123211 2 days ago [-]
Because they're all in countries who would ignore such litigation.
articulatepang 1 days ago [-]
Who’s “we”? I don’t want to do that work. I don’t think kernel maintainers do, either. Do you?
2 days ago [-]
eru 2 days ago [-]
> It’s more economical at a compute level, but not at the developer level.
Developers and compute are interchangeable now.
nozzlegear 2 days ago [-]
Who's proompting the machine to do it differently without a developer there to ask the right questions?
eru 2 days ago [-]
You can have a high level prompt of: make our crawling cheaper and more reliable to run.
miki123211 2 days ago [-]
I highly doubt the adversaries actually care about kernel.org in particular.
It's more likely that they're trying to scrape the entire internet, and cgit is just unfortunate enough to have autogenerated pages, where each page seems to contain text entirely distinct from its neighbors. This probably bypasses the scrapers' heuristics, which normally prevent them from getting trapped in a maze of twisty little autogenerated pages, all alike.
The way you normally deal with datasets at that scale is to move postprocessing (particularly expensive postprocessing) as late in the pipeline as possible.
GoblinSlayer 2 days ago [-]
What phone users do on git.kernel.org? In the worst case they can login (if they get to the login page).
gavinsyancey 2 days ago [-]
The point of Anubis isn't to prevent bots from accessing a site, but rather to prevent dumb bots from accidentally DOSing a site.
If someone adds an Anubis solver to their spider, but then also makes it respect cache tags and stops crawling random old commits through a git-web interface, that's probably still mission accomplished.
Aurornis 2 days ago [-]
> Bots want to open many orders of magnitude more pages than a normal user, so the resource investment for each single page matters a lot more to them
Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time.
So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views.
A user who needs to load a single page and a bot that wants to scrape a number of pages may pay the same cost.
The amortized per-view cost is highest for the real user.
crote 2 days ago [-]
But the tokens are IP-bound, and as the article mentions a crawler using a single IP is not so difficult to block.
A legitimate user solves it once, shows legitimate-use behavior, and retains their access for thousands of requests. A crawler solves a challenge, is determined to use bot-like behavior after a dozen requests, gets an IP ban, and needs to recycle their IP and start from scratch.
It is indeed worse for fly-by users, but with a website like Linux cgit those basically don't exist. It would be a completely different story for, say, someone's personal blog.
rplnt 2 days ago [-]
Now you have a session of sorts and can limit the requests for that client, right? They can be fast, just limited in volume - regular user isn't punished.
JsonCameron 2 days ago [-]
yep, that's the exact play. Or better fingerprinted & blocked in other ways
afdbcreid 2 days ago [-]
So, they can interleave different tokens and get stopless access at normal price?
Yes this needs to be tailored for Anubis, but that was the point: the moment it will become popular enough, scrapers will have no problems to bypass it.
embedding-shape 2 days ago [-]
> which takes ~180sec for my iPhone 17 to solve at ~100KH/s
> so I really don't see why Safari on a modern iPhone should be so slow at this
FWIW, my iPhone 12 Mini also does ~110KH/s with Anubis on lists.ffmpeg.org, so seems fairly likely that Safari somehow here isn't working as expected.
physicalecon 2 days ago [-]
iPhone Air is roughly 90KH/s. I bet it’s a matter of the site not being “trusted” and the JIT being disabled.
Shank 2 days ago [-]
Apple dynamically disables JIT based on website trust?
supriyo-biswas 2 days ago [-]
GP is probably running in lockdown mode.
gnyman 2 days ago [-]
My 13 mini is getting similar results, no lockdown. Something else is going on.
GoblinSlayer 2 days ago [-]
I guess it's because the anubis code is heavy with allocations.
reitzensteinm 2 days ago [-]
Paying a second of CPU time is an inconvenience to a user but trivial for LLM scraping.
The GPU cost to pretrain on that page once will dwarf by 2-3 OOMs the CPU cost to compute Anubis, scrape and post process it. And you’re not going to just train on it once!
At best you’re creating a speed bump for wannabe players scraping with no real plan. The folks training models people use just do not care.
throwawayffffas 2 days ago [-]
No it's not, beacuse the people running the bots have access to the c implementation and GPUs, the arm implementation on the iphone might be fast, but is almost certainly slower than an implementation running on a 5090, let alone a b300.
It's like trying to keep billionaires out of a restaurant by charging high prices, but still keeping it affordable for the middle class, it doesn't work on a fundamental level.
The people trying to do the scrapping will always have more compute than your typical user.
fc417fc802 2 days ago [-]
> If the two stay within an order of magnitude of each other Anubis is a perfectly viable and scalable solution.
Exactly, and that's even before accounting for more sophisticated PoW algorithms such as the one used by XMR that almost entirely eliminate any benefit from offloading to an accelerator.
stjo 2 days ago [-]
> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience.
Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)
chr15m 2 days ago [-]
Monero is fine with making people wait, that's literally the whole basis of mining blocks in "crypto currency". It's tuned to the time required.
Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.
margalabargala 2 days ago [-]
One possible way to do this: rather than simply burning hashes for nothing, Anubis can burn hashes to mine crypto. This way, the bots pay for the infrastructure they consume, at least in part.
Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.
crote 2 days ago [-]
CPU crypto mining stopped being viable over a decade ago. If you want a client to pay for the infra cost of their request, they're probably going to need to work for hours to do so.
arc-in-space 2 days ago [-]
Disappointed at the lack of replies here, I've had a similar thought before and I'd be curious to hear about the technical viability of this
supriyo-biswas 2 days ago [-]
This was done in the past[1] and was widely regarded as a "cryptojacking" scheme serving "malicious" content to users.
If it was for profit. Here it's in response to abuse by AI companies.
fc417fc802 2 days ago [-]
> Anubis wants "slow for bots but fast for humans"
That's not true. The goal is to raise the cost for bulk scrapers to the extent possible while minimizing any inconvenience to regular users. It's not about being slow for bots but rather reducing the asymmetry of the problem (from the perspective of the server) by imposing a (minor) cost on anyone requesting a page.
Dylan16807 2 days ago [-]
That's a better way to put the goal. But does it actually raise the cost or reduce asymmetry in a meaningful way? If it makes crawling 0-5% slower then it's really not worth the annoyance.
I know having the Anubis javascript at all gets in the way of some bots. But is the calculation helping enough?
fc417fc802 2 days ago [-]
> does it actually raise the cost or reduce asymmetry in a meaningful way?
It's complicated. I think anubis currently only collects low hanging fruit so to speak but in practice that's the bulk of what was causing problems so it's good enough.
PoW in general absolutely yes - I think monero clearly demonstrates this. Last I checked it remained viable to mine monero on at least somewhat low end and at least somewhat outdated consumer CPUs. It seems to me that fact implies that a well constructed PoW scheme can in practice squash the impact of any performance differences to within at most a couple orders of magnitude.
Dylan16807 1 days ago [-]
But a couple orders of magnitude is still a ton of asymmetry.
And even if you can guarantee the same CPU time on bots and legitimate requests, you still have a bunch of asymmetry in favor of the bots.
fc417fc802 1 days ago [-]
Sure. Even if PoW in general is sound, when applied to scraping it's always going to be a matter of what the client is willing to "pay". It's entirely within the realm of possibility that a scraper could be willing to "outbid" the user of a 5 year old tablet.
But it's also true that charging anything is going to be expected to reduce server load, that the cheaper something is the more carelessly people will tend to make use of it, and that at least in general bot operators are expected to be more cost sensitive than regular visitors on account of scale.
To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.
I keep seeing comments along the lines of anubis only working because it bounces bots running lightweight setups without js but stop and ask yourself why are they running such lightweight setups to begin with?
Dylan16807 1 days ago [-]
> why are they running such lightweight setups to begin with?
It's easier and still gives access to almost everything. I don't think CPU limitations are the main issue there. As far as computer resources, I'd bet RAM is the bottleneck to running the script, not cycles. That suggests a RAM-heavy hash might do better than SHA256, but still agrees with the idea that the PoW in Anubis is not the important part.
> To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.
I considered that. But while a visitor is idle they need the result quickly and they face the full brunt of the challenge all at once. A scraper can be working on a thousand sites at once so the per-request impact of the challenges is a lot lower, and patiently working through challenges in the background usually reduces the real cost, potentially to almost zero.
And you'd better not need multiple gigabytes of RAM to visit your site, that'll freeze my computer.
eru 2 days ago [-]
> [...] which is not possible if somebody has a farm of devices that costs them basically zero.
Opportunity costs are real.
jcul 2 days ago [-]
Monero is fascinating. I downloaded the spec and tried to digest it, but the mathematics and cryptography are quite dense, at least for a casual read.
I might try to tackle it again when I can give it some more attention.
xena 2 days ago [-]
The WebAssembly challenge (as of yet unreleased, it's been in dev for a year and I don't want to enable it by default until I know it won't blow up horribly) is bit-wise instead of nibble-wise, meaning that admins have better challenge granularity: https://anubis.techaro.lol/docs/admin/configuration/challeng...
tptacek 2 days ago [-]
What I'm hearing is that you spent a year designing a more complicated proof-of-work challenge rather than moving to a challenge system that actually impacts scrapers. Consider revisiting this decision!
darkmighty 2 days ago [-]
> rather than moving to a challenge system that actually impacts scrapers
At this point isn't it basically auth-only?
Rant: (genuinely wondering too, and RFC, request for conversation) at this point don't we have Google, etc. basically doing Real World ID Verification, but without an open protocol backing, using it to corral users into their ecosystem and gather data, and leaving us without some open and distributed alternative like the (fairly well working it seems to me) Public Key infrastructure? (PKI)
I know Sam Altman (with Proof of Human or w/e) and others have suggested and also have their own alternative (which I think is basically what Google already does), but I think this should simply be built based on an open protocol and then hopefully not-too-large (and with a healthy competition) local companies could do it for you.
GoblinSlayer 2 days ago [-]
Botnets will just borrow your id.
darkmighty 1 days ago [-]
Maybe, but at least you can more easily assign responsibility in this case. If you soft-block or hard-block someone because they're in a botnet, you can get them to change (expire the old ones) their credentials and start with clean systems. Maybe they'll realize their TV is part of a botnet if they have to keep doing it, or their PC has malware.
GoblinSlayer 1 days ago [-]
The pattern described in the blog is that one ip makes 4-5 requests and never comes back.
darkmighty 1 days ago [-]
Indeed, but then you can (for example) just basically blocklist this ID and share this blocklist with other servers (and likewise import blocklists) -- a blocklist curated by one or more NGOs or the sort would be ideal.
Fixing a malware infection should be like fixing up anything else broken around your house (clean things up, re-auth yourself with the ID verifier, done).
Also, bots cannot simply automate renewing the credentials of course, after expiry you should be required to physically (or in a trustworthy digital medium) re-authenticate yourself preventing any kind of mass attack, you'd need millions of people to perform authentication multiple times a day (by that point it seems safe to assume they'd rather fix the malware source in their personal devices).
xena 1 days ago [-]
I am actually working on this kind of shared reputation database: https://github.com/TecharoHQ/reputationdb (warning: very prototype, not documented, such bugs, wow). I just have more to do than I have free time at this point.
selfmodruntime 2 days ago [-]
Consider that Xe provides their software for (basically) free
selfmodruntime 2 days ago [-]
How's it feel to take a pet project from the need of an annoyed blog hoster to the forefront of the LLM crawling discussion? :D
xena 1 days ago [-]
It's made me work two full time jobs, one of which is drastically underpaid compared to the level of rigor and effort I need to put into it.
GoblinSlayer 1 days ago [-]
Was a post form tried as a challenge?
[text field][submit]
(type qwerty and submit)
samtheDamned 2 days ago [-]
Out of curiosity I tried to load this on my iPhone 12 and it took about 8 and a half minutes and once it finally passed it said my browser didn't allow cookies and to enable them before trying again. I would have to go through the whole proof again. I get what they're doing but there definitely needs to be some work done on getting those times down on iphones.
miki123211 2 days ago [-]
And if you're a bot operating at scale, you can use different devices for crawling and challenge solving.
Do you have an underpowered TV box that would need hours to solve difficulty 6? Do you also have a Windows box where somebody installed gta6_leak_full.iso? Great, have the TV send their challenges to the Windows box for solving.
If you're operating at scale, you probably have hyper-optimized Anubis kernels, E.G. compiled for the particular set of SIMD extensions that a given machine has. At that scale, getting 1% more juice out of your machine pool becomes worth the squeeze. In the worst case scenario, you can probably get some cloud boxes on AWS to do the challenge solving for you.
philippta 2 days ago [-]
I think TVs are just used as a residential proxies. No remote controlled chrome is running there.
apitman 2 days ago [-]
This comment captures the current AI zeitgeist about as well as anything I've seen.
titularcomment 2 days ago [-]
I like that they supplied the prompt instead of the working code :)
PlotCitizen 2 days ago [-]
I don't
semiquaver 2 days ago [-]
Tough. The code isn’t worth enough to publish. It means nothing to me and I’d have to figure out how to do it pseudonymously. I only have the one GitHub account.
gizmodo59 2 days ago [-]
Side note.. Fable just rejected this. GLM 5.3 did without questioning me. 5.6 sol did it beautifully.
semiquaver 2 days ago [-]
I used sol as well. Should have noted that fable is more or less guaranteed to refuse something hacking-adjacent like that.
latentsea 2 days ago [-]
It's kinda interesting to see simultaneously the 'holy shit' response to the OpenAI / HuggingFace incident, and then the griping about Fable's controls regarding this.
Dylan16807 2 days ago [-]
"It should write the code I tell it to in an interactive session. Also when running autonomously it shouldn't decide to hack into systems."
I don't see much connection between that problem and these controls.
latentsea 14 hours ago [-]
What's the difference when you prompt it versus when it prompts itself?
Dylan16807 2 hours ago [-]
It shouldn't prompt itself to go outside its original scope. That would be a good place for controls.
But also I didn't say it should hack when I prompt it. For the purpose of this discussion that could go either way. To avoid the griping it just needs to write the code I want when I prompt it.
nozzlegear 2 days ago [-]
> Fable just rejected this.
All in the name of safety, of course.
rzmmm 2 days ago [-]
It does work, at least in my experience. The challenge may be cheap for your n-of-1 but keep in mind the scraper bots are crawling number of domains with very high concurrency.
inigyou 2 days ago [-]
And without control of their source IP addresses. They use a "rotating proxy" meta-proxy which sends each request through a different proxy, which invalidates the Anubis response and triggers a new challenge.
wbl 2 days ago [-]
Real attackers fuck. If they have IP address control issues they will fix them. The bots being stopped are the silly ones.
applfanboysbgon 2 days ago [-]
Okay. That's good enough. The problem people using Anubis have is not "I'm being targeted for a DDoS" and is actually "my site is being incidentally DDoS'd because scrapers are DDoSing the entire internet". A solution to the latter is what people need. Every single one of these threads comes up with hypotheticals about how useless Anubis is and how it can't possibly work and yet it does work for the real-world problem that real-world webmasters are dealing with.
Terr_ 2 days ago [-]
Right: It's a threat-modeling mistake to start lumping different groups with different goals and different operational constraints together.
Sure, the cheap lock I added to the outside shed's door-latch is terrible and can be shimmed open in two seconds with a bit of flat metal... but the important thing right now is that it keeps the raccoons out. If there's a new threat to the gardening tools or the bikes, then I have a different problem and I'll consider something else.
wbl 2 days ago [-]
If every site does it the scrapers will adapt. The scrapers will always have a cost advantage over real users for this.
fc417fc802 2 days ago [-]
Most of the problem in this case is that the scrapers are being incredibly lazy and not investing even minimal effort in their approach. If this forces them to revisit how they go about things then mission accomplished.
zalo 2 days ago [-]
One natural counter to this is to dynamically alter the computing algorithm per request (variants of Anubis?) so that the scrapers would need to rewrite their accelerator each time or just use the unoptimized JS version...
TZubiri 2 days ago [-]
I thought we learned this with Litecoin, it works only as a novelty factor, nothing fundamental.
When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it.
lxgr 2 days ago [-]
I mean, it works just fine for PoW-based cryptocurrencies (with "why are we doing this if there are much better solutions to the same problem" values of "fine") because there the game theory works out, which is "solve hashes to gain a probabilistic lottery ticket for a shared prize pool" having a higher EV than "solve hashes for the probabilistic chance of double spending some money, but if you don't pull it off you gain zero".
Anubis is just "solve hashes to get access to a valuable resources", and if the resource is at least valuable to whoever you're trying to deter as it is to legitimate users, you're not deterring anyone (and by extension, if it's more valuable to them, you're deterring your legitimate users instead).
TZubiri 2 days ago [-]
I mention litecoin specifically, it was developed right after BTC ASICS started dominating, and the LTC goal was for hardness to reward consumer hardware rather than ASICS, it was briefly successful until LTC ASICS were developed.
teravor 2 days ago [-]
anubis was a joke on arrival. PoW for this purpose is plain stupid.
the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.
crote 2 days ago [-]
Yes, but it actually works.
As the article mentions: Anubis blocked virtually all bots for a couple of months, and even now it is still blocking 67% of requests. Without Anubis they'd have to double the number of CPU cores, solely to serve crawler traffic!
It's an arms race. The best solution is the one you can plug in right now - even if it is stupid. hCaptcha might have a better blocking rate at the moment, but it's also $0.99/1K evals, and that "animated challenge" will be impossible to solve for visually-impaired users: for many people that cost is significantly higher than spinning up an extra server or two to feed the crawlers.
GoblinSlayer 2 days ago [-]
That's because those 67% of bots are the most stupid and don't run any javascript at all.
teravor 1 days ago [-]
that means all you had to do was gate the content behind a javascript snippet with a timer.
inventor7777 2 days ago [-]
My iPad Pro M2 takes around 30 seconds to solve it in Safari, while my Mac Studio solves it in about 3 seconds using 449% logical CPUs. It really makes no sense. Interesting how much slower it is than native - could it be protections against things such as crypto mining?
radlad 2 days ago [-]
I'm reminded of LUKS Argon2 implementation, which is much slower than userspace. Therefore a sufficiently difficult challenge that may take less than a second in userspace Linux can take 20 or more seconds at boot.
jwilk 2 days ago [-]
Huh? Are you saying there's an Argon2 implementation in the kernel?
minimaltom 2 days ago [-]
Its not in the kernel but in the userspace tool that goes from password to key (the key is handed to the kernel).
I got this wrong yesterday. While LUKS2 and GRUB now support Argon2, I configured LUKS1, which uses PBKDF2-SHA512.
In any case, it is GRUB which behaves slowly during boot. While I haven't tested Argon2, I believe it is affected in the same way as PBKDF2.
DenisM 2 days ago [-]
Memory-hard hash functions maybe? Like, you must dedicate 4gb of ram to compute the function. Not a problem for a one-off, but is a problem when reading lots of pages at once.
Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset.
brokensegue 2 days ago [-]
Do you want your phone to allocate 4GB to read a mailing list?
The better solution would be fix Anubis and optimise its calculations on iPhone. Maybe it needs WebAssembly hashing implementation instead of JS?
But it would just make Anubis suck a little bit less
Anubis "protection" model is wrong/broken on several levels. I suppose it does not even have ability to bypass hashing screen when server load is low.
bayindirh 2 days ago [-]
> I suppose it does not even have ability to bypass hashing screen when server load is low.
Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site.
It's on the same category as Iocaine and Go-Away, even if what they do is different in some cases.
Self-Perfection 2 days ago [-]
>It's sole purpose is to deny the bots you don't want on your site.
This is fundamentally impossible.
In the end only pay per crawl will work. Like x402 but maybe something similar.
In other words: not only client has to spend some kind of resource for access, but it has to be a transaction, server must get something from serving the request.
thaumasiotes 2 days ago [-]
> Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site.
No, this very article will tell you that the purpose of Anubis is to reduce load on the site. The purpose of denying bot requests is that you're strained by the load they impose.
Conversely, bots that pose no problems to you aren't "the bots you don't want on your site".
smallerize 2 days ago [-]
But the scraper is making way more requests and is paying for all that compute.
tptacek 2 days ago [-]
This is the basic misconception behind the Anubis design. In an anti-spam system, there's a user/abuser reward asymmetry. Users make single requests and get some constant reward V. Abusers make millions of requests and also get some constant reward V. Proof-of-work exploits the asymmetry: costs are, by a factor of ~millions, disproportionately imposed on abusers.
That's exactly not what happens with scrapers. Scrapers make millions of requests, yes. But each one of those requests produces an average (token-denominated) reward V'. There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.
That's before you even get to questions like "does the system actually work, or does a single solve get practically unbounded rewards".
progval 2 days ago [-]
> There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.
There is: users solve the PoW once, scrappers solve it every time they rotate IP addresses. Some of them change their IP address on every request, so they keep solving the PoW over and over.
tptacek 2 days ago [-]
That doesn't matter! Every time they successfully solve the challenge, they're getting the token reward. That's the missing asymmetry. In an anti-spam system, 99.999% of all transactions are unrewarded.
sunoforbrazilia 2 days ago [-]
[flagged]
cyberax 2 days ago [-]
I have a small personal site with an exposed Git repository. I'm getting crawled from about 100000 residential IPs from several ASNs (mostly in Brazil). Each crawler makes a couple of requests per minute.
Anubis doesn't really help against this.
GoblinSlayer 1 days ago [-]
This, blocking granularity is a country, not ip address.
graemep 2 days ago [-]
I wonder whether what Anubis is doing might be blocking agents that do not run JS or that have a timeout (or humans who give up as I just did)
tptacek 2 days ago [-]
I suspect that's most of the value, and that the deterrent is about software construction and not about the proof-of-work, which I've been saying for the last year is a productive direction for Anubis to head towards. Instead, it's just sort of tormenting normal people, as a kind of performance of anger about AI.
Which: I get! If it's your site, perform all the anger you want.
semiquaver 2 days ago [-]
no, a single solve returns a token which entitles the scraper to access the site as much as it wants for one week.
NobodyNada 2 days ago [-]
The token should only be valid for one IP, so the scraper has to keep solving challenges if it rotates IPs.
semiquaver 2 days ago [-]
OK, but that doesn't change anything. You have a large pool of IPs, each of which only needs to expend a handful of extra milliseconds of work to get unlimited access to the protected resource.
Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site.
a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.
inigyou 2 days ago [-]
The way they're internally implemented doesn't allow pinning an IP. They buy a rotating proxy service from a vendor, and don't get to choose their source IP.
rstuart4133 2 days ago [-]
It's not hard to test. Go to a page that demands PoW, change your IP and see what happens. I just did it. Spoiler: kernel.org asks for a new PoW.
If the source IP was an issue, you could do it other ways: for example, make the cookie rotate on every access, and insist there is a single stream of accesses.
semiquaver 2 days ago [-]
Why are you and other defenders of the Anubis approach so fixated on this one specific limitation of a certain type of scraping architecture? It’s hardly an immutable characteristic.
You say “they” as if all scrapers are a monolithic group with the same constraints and goals. Part of the problem is the massive diversity.
inigyou 2 days ago [-]
Why do you think anyone thinks it's immutable? Do you understand the basic concept of an arms race, or cat and mouse game?
dspillett 2 days ago [-]
> In other words not a meaningful barrier.
And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target.
> a $5000 ASIC-based device can hash SHA256 at 200TH/s or more
Peanuts for the big players, but many (almost all?) running smaller scale scraping operations are going to find $5000 rather prohibitive, and they are unlikely to be able to integrate it as they are probably running a “stock” scraper that they didn't write themselves.
semiquaver 2 days ago [-]
You don’t need to spend $5000 to obtain the hash rate of a $5000 device on a rental basis. You may have heard of this thing called “the cloud”. Obtaining very high hash rates is effectively free, largely as a side effect of the crypto bust.
Not sure why anyone would characterize these scrapers im general as all being fly-by-night operations that don’t have two cents to scrape together.
dspillett 2 days ago [-]
And yet you have not addressed the point that many report that tricks like Anubis work.
If they are so stupid an idea that they could never work, why do they seem to having the desired effect?
Dylan16807 2 days ago [-]
A significant amount of the effect is everything outside the proof of work. Not the cost of hashing but the need to run the javascript that submits it.
If you've seen anyone post a comparison of crawl rate versus difficulty, I'd love to see it. There's probably some difference but I want to know how much of the overall effect it is.
semiquaver 2 days ago [-]
I never said the approach was so stupid it could never work. I just shared how it’s annoying and how I worked around my annoyance.
An even stupider approach would work exactly as well or better, like a form saying “type the letter y in this box to continue”. The only benefit is as a road bump that makes the site in any way custom. The moment anyone you are defending against so much as looks at the mechanics of solving the challenge it completely falls apart.
dspillett 13 hours ago [-]
> I never said the approach was so stupid it could never work.
In that case I have read too deeply between the lines/tone. Sorry for the somewhat accusatory assumption of your opinion's details.
LatticeAnimal 2 days ago [-]
Some of the asymmetry might be regained if anubis had thousands of variations of PoW algorithms, each different enough that they must be solved independently.
I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...
inigyou 2 days ago [-]
You can do this on yours. Just have the client and server add an extra "2" after the challenge key or something. A different client which extracts the challenge key and does its own processing will only generate invalid responses.
LatticeAnimal 2 days ago [-]
Cool, so then that invalidates the ASIC problem, right?
My earlier idea was to imagine that each day Anubis picks an entirely different problem-class. Ex: one day it is Sha256, the next it is prime factorization, the next it is twin-prime-finding, the next it is cracking elliptic curves, the next it is some kind of sorting / information theory problem...
All with the goal of adapting constantly so that scrapers have a harder time optimizing for the PoW problem (i.e. with Sha256 ASICs)
inigyou 2 days ago [-]
That doesn't work because there is a finite list to choose from.
Nobody is using ASICs on Anubis.
xena 2 days ago [-]
> I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...
This is something I'm working on in a prototype. Stay tuned!
GoblinSlayer 1 days ago [-]
hashx kind of already does it.
CursedSilicon 2 days ago [-]
...That sounds like entropy? As in, the thing computers are bad at (truly random numbers) and /dev/urandom in your kernel already spits out an approximation of?
DarmokTanagra 2 days ago [-]
not if its running on someone else's device
karlgkk 2 days ago [-]
> I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.
You yourself describe the solution.
Extensions or browser integrations of the Anubis task.
dpark 2 days ago [-]
He didn’t describe a solution. He described a (crappy) workaround for humans. But the fact is that this cannot and will not stop bots. The people running bots can do the same, even faster.
karlgkk 2 days ago [-]
He did in fact describe it.
dpark 2 days ago [-]
I’m left wondering if we disagree about what the problem is.
The problem here is not merely that Anubis is inconvenient to humans. It’s that and also that it’s not very effective for blocking bots. Anything that makes it easier for humans to get past will also make it easier for bots to get past.
Dylan16807 2 days ago [-]
The point of contention is not the word "describe", it's the word "solution".
bjoli 2 days ago [-]
Does it really take 3 minutes on your iPhone? My pixel 8 does it in slightly less than a minute in Firefox.
efficax 1 days ago [-]
iphone 16 and i gave up after about 30 seconds, it was maybe a quarter done
FergusArgyll 2 days ago [-]
Woah, went from > 1 min on git.ffmpeg (difficulty 6) to instant. Thanks!
Edit: after some more analysis; Average with extension disabled is 90 seconds, with extension enabled it's .3 seconds
imtringued 16 hours ago [-]
That's actually weaponizable. You can deliberately insist on an inefficient algorithm to find bots that use a better algorithm to bypass the PoW. Legitimate users have zero reason to redo the challenge over and over again, meanwhile bots want to constantly redo the challenge because they don't want you to throttle the bot based on the reused session cookie. You can then crank up the difficulty up over and over again.
Demiurge 2 days ago [-]
I maintain a formerly popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server.
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
pdonis 2 days ago [-]
> I don’t want to stop the bots from updating their knowledge based on all the content.
Why not? At this point, any attempt to try to meet the bots halfway is doomed.
Demiurge 2 days ago [-]
Because I also use AI, and I actually want the content from the site to show up for me, as well as other AI users.
More over, I actually pay for three different AI subscriptions, and use them a lot. I also think it’s fair that I pay for their services. I just want them to also pay for what they use.
mbirth 2 days ago [-]
If these were actual legitimate bots using a known user-agent and coming from the same subnet so you could throttle their requests it would be fine.
But I’m seeing requests coming in from random IP addresses from all over the world using randomised user-agents. But somehow these are trying URLs in sequential order. A while ago this would be classed as a DDoS cyber attack.
Hackbraten 2 days ago [-]
All anti-bot measures are inherently flawed, and if you overdo it, your false positives cause real harm by excluding actual people from using your services.
Not as in “1 out of 1000 legit requests get blocked” but “you’re going to block 100% of some person’s requests.”
anal_reactor 2 days ago [-]
> Overall, it seems like the LLM are really straining the internet economy, the openness of it
I feel like it's the same issue that public resources are facing.
Step 1: public resource is freely available
Step 2: asshole shows up, drains the resource completely just because they can
Step 3: either resource is taken away, or it has massive security behind it
inigyou 2 days ago [-]
Alternative step 3: arrest the asshole.
We don't even need a new law. What they're doing is already a felony, and there's a process already in place to go to a court and request unmasking of 10000 IP addresses.
Are the AI companies worried about anything but their valuations?
I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large.
Micro transactions [in this case] are a great idea, these crawlers need to be taxed and made to pay for the unaccounted external costs. Furthermore, we need to fund any risk mitigations, security coordination, and other safety work. If these things are not imposed on the market, the most ruthless player will create the biggest mess they can get away with.
I don’t see a better way to do this, right now, than to allow Strips, or CF, to create some way to charge the crawlers. Imagine I have some really expensive content, I can put it behind the paywall, or subscription model, but that’s very specific to the human consumer. What if I do want to search engine or content aggregator to consume some of the content? It would be quite hard for me to sign an exclusive agreement with OpenAI, like what Reddit or NYT are doing. It might be quite beneficial to create some infrastructure to let bots automatically to pay for access. It might even be crypto based, I don’t think it really matters. It just has to be easy for me to integrate it, and the bots need to find it easy to use.
If we all start doing this, I think this could be of benefit to all content creators, such as musicians, movie makers, game developers, and even independent journalists.
We used to pay for software and things, but what has happened? Walled gardens and production monopolies are taking over. This has largely reduced the quality of all the content we consume.
andai 2 days ago [-]
> Are the AI companies worried about anything but their valuations?
>I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large.
Doesn't the same thing apply to every company? And also every government? And also individual?
We're all doing the "locally rational" thing, and globally doing... well...
I think the main factor is time horizon.
Locally rational behavior begins to look globally rational when you add time horizon.
For example, if I'm doing what's best for my family for the next five generations, that starts to look a lot like what is good for the whole country and the whole world.
Demiurge 1 days ago [-]
Yes, this applies to every company. The institutions have evolved a lot, and how we look at social safety and fairness have both evolved and devolved at the same time.
Companies, individuals, and governments have different roles. Rationality from their perspective is specific to their perspective, and I don’t believe just because evolution or physics are rational we should die off from a new virus or asteroid impact. That would be perfectly rational, but I don’t care about the global rationality of the universe.
What is best for my family and other families is not what is best for Sam Altman or Sataya, or Elon Mask.
We’re just talking about market players. What I’m saying is that these market forces need an outlet via some new infrastructure and regulation. Maybe it’s not micro-payments, maybe it’s some other new ideas.
I just don’t want the next century to be the century of East India AI sticking me in a Matrix pod, rationally.
andai 1 days ago [-]
>physics is rational
What is its objective function?
Demiurge 1 days ago [-]
Entropy.
inigyou 2 days ago [-]
[to prove you are not a crawler and read this comment, please deposit $2.00]
Demiurge 2 days ago [-]
I think that to prove I am not a crawler I probably just need to supply a Safari user agent string. The crawlers on my sites actually self identify for the most part. The malicious ones are banned when they try to post spam.
Anyway, what’s wrong with making 2 cents? :)
desterothx 2 days ago [-]
The troubling part for me is that a system like this could also just as easily be pawned onto the humans (I realize it technically already is with the subscription model internet, but there are still plenty of open sites). A feature like this seems to me will lead to an internet where no one can browse anything without having a wallet connected and paying for each attempted access. Exactly what the internet is not supposed to be.
Demiurge 2 days ago [-]
I don’t see how this is different from having Stripe easily available to any web developer. I want this feature, but it doesn’t mean I’m going to add it to every website I build. I think of it as Stripe pay wall at infrastructure level rather than a JavaScript popup. If we look at NYT, WP, many already do this. But HuffPost and Reddit rely on different models.
I don’t think this will make the internet less open, I think the internet is already pretty much less open. Who blogs these days for free? They rely on medium or substack, and have no choice, they submit to these platforms rules about the articles they write, even though they’re pretty generous.
Now, if the reward system was as open as say, Bitcoin, I think there would be a lot more independent and different bloggers and journalists, and independent platforms.
inigyou 1 days ago [-]
Once you have the payment system, all I have to do is make a site that looks valid until you make a payment. Then it will display the trolololololololololol gif and keep your money.
persolb 2 days ago [-]
Maybe won’t work for your site; I’ve been leaving flat files open, putting anything that needs processing (like search and login) behind Anubis.
iririririr 2 days ago [-]
do not fall for cloudflare marketing.
they have one of the worst bot detection in the industry. but because everyone uses them, their huge false positive numbers won't show up anywhere.
timpera 2 days ago [-]
Cloudflare is also great at playing both sides, and they're trying pretty hard to push for pay-to-crawl because they'll probably get a 30% cut along the way.
Dylan16807 2 days ago [-]
Pay-to-crawl works better and draws in more customers if their detection is better, doesn't it?
And I only stand to gain from pay-to-crawl, so I don't really mind that play.
Demiurge 2 days ago [-]
That’s definitely pretty annoying, it’s almost like the walled gardens scenario. I wish this could be more democratized.
TZubiri 2 days ago [-]
There's a winner takes all (monopoly) dynamic, if everyone uses your product, you can share the ip banlists between customers.
1oooqooq 2 days ago [-]
See, this is you falling for their marketing.
It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tuned blocklists. Don't fall for the marketing!
Try it yourself. Block those fingerprinting tech and try to access their own user forum. you get blocked. Now allow fingerprinting (even if emulated in a headless chrome!) and access by the most toxic vpn you can think off, access granted.
All cloudflare does is act like a gatekeep for monetizable-by-creepy-advertising-networks traffic, not bots. If the bots pay the small headless chrome tax, you get bombarded with their requests just fine.
TZubiri 2 days ago [-]
If I recall by browsing the docs, there's a lot of fine grained client control over the blocking params. Which makes sense since the main product of cf is bot blocking.
So you can definitely block by IP, by fingerprint, both or neither
iririririr 19 hours ago [-]
the point is if their solution is as magical because of the monopoly they have. which is what their marketing hint at (and everyone here internalized, judging by this thread).
why do you feel like discussing features was relevant?
Demiurge 2 days ago [-]
What would you suggest as an alternative? Block the bots yourself ?
inigyou 2 days ago [-]
Yes. Any number of methods work. From requiring a cookie to requiring CSS to be loaded to a PoW to a JavaScript redirect. Most scraper bots are really dumb. If you do it cookie based, you can also put a poison link on every page that invalidates their cookie and starts returning poison on every page after that, so they're damned if they do and damned if they don't.
Demiurge 2 days ago [-]
I have tried cookie based sessions and I have tried multiple things. I have been battling DDOS and script kiddies longer than CF has been a thing. You’re wrong in that you think I could continue to maintain the site without CF in front of it, I would just shut it down. It’s not a profitable forum, it’s just a hobby.
TZubiri 2 days ago [-]
[dead]
iririririr 19 hours ago [-]
don't write a crap site that fall to pieces with a million unauthenticated users *shrugemoji
realusername 2 days ago [-]
For bots, usually a monopoly just paints a bigger target on your back than custom low scale prevention methods.
And Cloudflare is no exception, the "hello world" of bots is Cloudflare bypass.
nunez 2 days ago [-]
HoustonArchitecture.com, a site that posts about new development projects in the Houston area, had the exact same thing happen to them. (They also got hacked; don't remember the details.) They now pay the Cloudflare tax.
rankdiff 2 days ago [-]
What is "formally popular"?
klez 2 days ago [-]
From context I'd say they meant "formerly" and that's just a typo. I didn't even notice the error before you pointed it out.
Demiurge 2 days ago [-]
Yes, my bad! Typing on the terrible iPhone keyboard :)
incognito124 2 days ago [-]
bzflag
tptacek 2 days ago [-]
Tavis Ormandy called this, about Anubis, almost exactly a year ago:
It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.
harshreality 2 days ago [-]
If you want to argue that there's excessive enthusiasm about anubis, which is neither a permanent solution nor a solution against all bots run for all purposes (some bot authors are cleverer than others, and you don't have to block everything, just enough to reduce resource use to tolerable levels)... fine.
However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense as a theoretical barrier against bots. Who cares as long as it works empirically, even briefly? Are you suggesting people let their sites continue to be degraded or unusable until they find a better solution, even though PoW challenges can improve the situation today?
Not everyone has the luxury of writing their own web app designed to handle massive bot traffic at minimal cost, gutting functionality to meet that requirement if necessary.
I noticed a couple of weeks ago that 1-2 orders of magnitude more bots were getting past anubis challenges. However, it would be incorrect to conclude that it's doing nothing and now worthless aside from heating up visitors' cpus. If I turn it off, I see >2x bots.
The next salvo in the war of attrition will probably be when anubis switches to wasm hashx, which the author suggested, in previous HN threads, is on the roadmap.
If bots quickly adapt (Claude vibe-coded JIT wasm-to-native-transpiler for hashx challenge algorithms?), that may be the end of anubis and efforts like it. If they don't, I expect more HN threads with people complaining that such compiler efforts are possible and therefore PoW-gated sites are doing it wrong... even if those sites stay usable.
tptacek 2 days ago [-]
I'm not moralizing. The sites running Anubis are owned by people who like Anubis and are fully within their rights to run Anubis. I don't care how enthusiastic they are about it. It's their call.
I'm just saying, it's bad computer science.
crote 2 days ago [-]
The real world doesn't care about the soundness of your computer science. All that matters is whether it works.
It's like saying we should just completely give up on all NP-hard problems because we'll never discover a way to find the optimal solution in all cases. Meanwhile in the real world, people are more than happy with, say, a route planner which virtually all of the time gives them a solution which is within a rounding error of optimal. Nobody is going to care that it is technically bad science.
Same with Anubis: Sure, it's a dead end in an ongoing arms race. It won't last forever. But right now it works pretty well, so let's hope it lasts until someone can figure out a better approach.
tptacek 1 days ago [-]
I'm fine with you running Anubis. My point was that Tavis flagged the outcome this blog post is discussing a year ago. It was hotly disputed on HN at the time. That's all I'm saying.
1 days ago [-]
harshreality 2 days ago [-]
You're arguing worst-case, and people running anubis are saying it dramatically helps in their experience, because they're not getting too many of the sorts of bots that people like you and Tavis (correctly) argue are theoretically possible. So the argument that it's "bad CS" falls flat.
A better, perhaps the best, argument is that a worse bot mix might appear tomorrow, so anubis should be used as a runway for a better solution, and not as a solution in itself. If only everyone with a website had the desire (or AI tokens) to implement their own better solution. In some cases even desire or a lot of tokens isn't enough. I mean, you think kernel.org folks haven't put some serious thought into it? And yet, as HN argues about it, there's kernel.org running anubis at difficulty 5.
tptacek 2 days ago [-]
I don't know what you mean by "the best argument". My entire argument is that it's bad CS. I'm not trying to convince people not to run Anubis. If anything, I'm trying to convince Anubis to use better computer science. That would make more people run it! But even that's just a faint subsidiary goal; the real goal, as someone who works in this field and has designed proof-of-work antiabuse systems before, is just to add clarity about what can and can't work and why.
harshreality 2 days ago [-]
I too would love anubis to use better CS, if available. Perhaps you have some ideas for @xena, instead of just letting anubis continue on the trajectory of iterating its PoW approach?
rvz 2 days ago [-]
Same here. But of course and as expected; 5 hours later, the GP comment offers no proper solutions for this "better computer science" they speak of.
It shows they really don't care and they are just rage-baiting readers in their weekly grumpiness without any suggestions.
Unless they come up with said "solution", they can completely ignored in their complaints.
tptacek 2 days ago [-]
I had a fairly long conversation with Xe privately about this and didn't feel the need to repeat it here, especially since (as the search box will tell you) I've already talked about this before. But, do go on.
rvz 2 days ago [-]
We are not asking you to repeat the full conversation here. We just want the actual solution that was discussed.
So could you give a short elaboration on what was this "better computer science" solution that you came up with in your discussion?
MallocVoidstar 2 days ago [-]
I looked at your comments and there's a bunch saying that Anubis is bad CS but none I could find saying what, exactly, you want Anubis to do instead. You sure are posting a lot about how it's bad, though.
myng111 2 days ago [-]
It is interesting, this particular user seems extremely active in this thread. I'm not sure if they have a bone to pick with Anubis in particular or some other reason for disliking the software so much, but to me it does seem overzealous. Regardless of if it's "bad computer science" or what, Anubis works quite well and is easy to customise to target it to specific botnets, as I've done. And this is coming from someone trying to develop her own non-PoW anti-scraper system.
arc-in-space 2 days ago [-]
You might want to note that "this particular user" is "extremely active" on this website.
(Nothing against it, I'm always happy to hear what Thomas has to say)
myng111 2 days ago [-]
Apologies, you're probably right, I don't post much on HN so I don't really know the regulars.
rtpg 2 days ago [-]
I get what you're saying, but I think that in the abstract "here's this thing you can do to cut the annoying traffic down to 1/3rd its previous level" (with the obvious issues to legitimate users of course) feels like a nice thing to have in this current moment!
If you were dealing with a DoS like this and had an option that cut the annoying traffic in half in a fairly deterministic way, that sounds like a decent thing to do operationally in the moment right?
inigyou 2 days ago [-]
Which part of computer science is about getting DDoSed by billionaires who are above the law?
TeMPOraL 2 days ago [-]
Where on Earth do people get the belief that:
- It's the SOTA companies doing it?
- Scrapers are doing it for training data?
Those are two assumptions I see in posts and threads around Anubis, that are taken at faith, and never once substantiated.
... It also has a quote from Anthropic, admitting they backed off their scraper after a robots.txt update.
inigyou 1 days ago [-]
Nobody who gives a shit about their software actually working obeys robots.txt. most robots.txt block everything except googlebot, which is an insane policy and you don't have to follow it.
shakna 1 days ago [-]
Which isn't all that relevant a discussion here. Anthropic saying they did so in this case, means that they admit they were the scraper flooding the domain.
mschuster91 2 days ago [-]
> It's the SOTA companies doing it?
There are more than just the American top dogs (OAI, Anthropic, SpaceX, Facebook)... especially the Chinese government with all its infinite cash resources and next to zero ethical constraints.
I don't trust the US top dogs at all, but I think the fear of discovery alone would lead them to not use "residential proxy" services. Non-US/EU entities however... who cares?
figglestar 2 days ago [-]
Why would they directly use a proxy service? I'd just launder the data scraping through some third party company that I could slough off if it ever turned into a news story. Not that anything would happen to them if they directly used these services anyway.
inigyou 2 days ago [-]
What's your alternative hypothesis?
charcircuit 2 days ago [-]
If you just want to block the bottom of the barrel scrapers you don't need to do any proof of work. It's overkill and wastes user's times for such a purpose.
nneonneo 2 days ago [-]
I disagree. The kernel finds it effective - 66% of scrapers are turned away directly.
The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site.
If each new site they hit forces them to solve a proof-of-work, that’s a meaningful dent in their scraping performance. Many of these boxes may not even have the spare CPU power to efficiently solve so many proofs of work - and anything that makes an owner notice their device is running slow is something that could meaningfully impede adoption of these SDKs, or force the operators to choose between minimizing performance impact or scraping more sites.
tptacek 2 days ago [-]
It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.
I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing.
I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
akerl_ 2 days ago [-]
Have you compared against adding a JavaScript requirement with no proof of work?
kstrauser 2 days ago [-]
Yes. That helped, but not nearly so much.
A1kmm 2 days ago [-]
The problem is that the scrapers are not sophisticated enough to use things like cgit and Git Forges efficiently (e.g. by git clone), while still being sophisticated enough to bypass widespread controls they are implemented to bypass.
An author of a scraper would need to trade off how much arbitrary compute they'll run against how much risk they take that someone will use them for free compute (inverting their role from exploiter to victim). They can work around that somewhat by analysing common enough code (to justify the LLM or human expense) and deciding to run it.
PoW challenges work when they have low enough adoption that it isn't worth scrapers' while to have a custom countermeasure for. To scale them once adoption is big enough, it probably takes some bespoke engineering per group of sites (coupled perhaps with some scraper traps that take advantage of scraper CPU for something useful, and aren't easily distinguished from the normal challenge).
yorwba 2 days ago [-]
For PoW to work, you don't need scrapers to have less compute than users, you only need their compute advantage to be less than the load-generation advantage they would otherwise have. And unlike legitimate users, scrapers aren't carefully picking the URLs they request, so the average value they get out of it is probably rather low. They have to make it up in volume.
2 days ago [-]
oasisbob 2 days ago [-]
> Many of these boxes may not even have the spare CPU power ...
I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point.
Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol.
If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.
Y_Y 2 days ago [-]
The implication here is that the proxy fridge forwards the Anubis challenge to a dedicated rig controlled by the scraper who efficiently solves it and returns the answer.
wongarsu 2 days ago [-]
That's still a notable step up in completely and resource investment for the crawler
See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha
inigyou 2 days ago [-]
That's not a step up, that is how they work by default.
lxgr 2 days ago [-]
Why would they even run a browser engine on the devices they're hosted on?
All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.
lxgr 2 days ago [-]
> 66% of scrapers are turned away directly.
Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users.
The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while.
It doesn’t have to be fully sound, it’s there to provide _some_ degree of interim protection.
Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good.
The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.
lxgr 2 days ago [-]
Sure, but then you’re essentially just doing versions of the old “what is 2+2” text captchas with extra computational steps, so why not save yourself and your users those?
done_lurking 1 days ago [-]
As gp said, it does provide some protection. If it didn't, nobody would use it and there would be nothing to discuss. Making snarky comparisons doesn't do anything.
lxgr 5 hours ago [-]
People can remain irrationally convinced about the efficacy of non-solutions to their problems for quite a while, in my experience.
And I really wasn't trying to be snarky; people had "what is i + j" type of captchas for a long time and I still see them every once in a while!
danieltanfh95 2 days ago [-]
it's not just "not sound", it is outright wrong.
semiquaver 2 days ago [-]
> meaningful dent
Until you actually do the math and realize that it is not meaningful at all. It’s equivalent to the blogs that have a custom “bot protector” that asks you “what’s 2+2” every time you submit a comment. It might work temporarily as an inconvenience, but nothing more.
graemep 2 days ago [-]
How s that measured? How do you count human users who have been turned away?
chr15m 2 days ago [-]
Yep, all security is about making attack more expensive than defense. If they have a farm of other people's devices, as alleged in the article, then it costs them less to calculate PoW than it costs ordinary users. Ordinary users have to sit there waiting, whereas bots are somebody else's problem.
jdnier 2 days ago [-]
I really enjoyed the writing style in this article.
And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.
atq2119 2 days ago [-]
It feels like this progression of increasingly drastic measures to circumvent the protections of a computer system ought to be enough to establish criminal intent and get some of the people running those crawlers into prison.
inigyou 2 days ago [-]
It does. It's literally a felony but for some reason not a single person has pressed charges.
emsign 2 days ago [-]
It's because the companies doing this bought the current government.
inigyou 2 days ago [-]
There is no evidence for who's doing this and nobody has made any effort to obtain any.
vermilingua 2 days ago [-]
Sorry, no evidence for who's running... GPTBot and ClaudeBot?
inigyou 1 days ago [-]
No evidence for who's running the global ddos scraper network.
NavinF 2 days ago [-]
turns out the overlap between "people who can't configure their webserver to serve at wire speed" and "people who can get law enforcement to take them seriously" is the empty set
p_j_w 2 days ago [-]
Didn’t read the article, huh?
TZubiri 2 days ago [-]
It goes beyond mirrors, it's just something criminals have been doing since forever, to abuse all websites.
virgoerns 2 days ago [-]
I also run a public cgit instance and get over 1M hits every day, although my pet projects are nowhere near the size or impact of kernel. I had to block (via nginx conf) cgit endpoints for diffs, blame, snapshots and historical commits, because nothing else works. Now they return 402 (payment required). I consider this my total defeat and it's killing me inside, but it is what it is.
inigyou 2 days ago [-]
You could also publish a list of IP addresses.
mzajc 2 days ago [-]
As the article describes, it doesn't help, because the traffic originates from millions of unique residential IPs across hundreds of ASNs and countries.
inigyou 2 days ago [-]
So?
VladVladikoff 2 days ago [-]
Have you tried blocking a million IPs before? Fail2ban gets pretty shaky at even 200,000
The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution.
And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawlers is using RTT diffs this is one example https://github.com/Sakura-sx/Aroma
pmlnr 1 days ago [-]
Fail2ban becomes a serious bottleneck at significant traffic. I've replaced it with a shell script and direct pf commands that run every few minutes.
virgoerns 18 hours ago [-]
Do you mind sharing your script? I also noticed that fail2ban currently eats ~25% of CPU and started thinking if it's even worth running at this point
Read the article, it will tell you why it won’t work.
kdowns 2 days ago [-]
I made it to a third round interview at anthropic in 2024 and they had me build a web crawler as their programming test. Part way through I started on making it respect robots.txt and I could immediately tell they were no longer interested in me.
myng111 2 days ago [-]
That's really funny. It's always been kind of amusing to me that Anthropic has this air about them of trying to be the most ethical AI company, but really exhibits the same behaviour as all the others.
sgsjchs 2 days ago [-]
Ironically, defense by obscurity may be the way to go here.
Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.
You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or building some unholy pipeline that uses ai agents to compile it to a GPU kernel for every host.
slipknotfan 2 days ago [-]
> Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.
A more robust solution would be to keep a few patches handy with different versions of the algorithm, and rotate which one is in use. This would keep the crawlers on their toes if they wise up to the changed algorithm. One could even imagine automatically rotating witch algorithm to use on a weekly basis.
dewey 2 days ago [-]
Render page in headless browser with JS, cache challenge solution, re-use on all your non-browser requests. Won't really change anything and it will auto-adapt to any new challenge that is being thrown.
sgsjchs 1 days ago [-]
It's not reusable from a different IP.
rtpg 2 days ago [-]
I was thinking about this. Would it be possible for Anubis to have a code gen process that would make every deployment of itself sufficiently unique such that it's "annoying" to work around?
I do wonder about the premise as well: are people special casing for Anubis?
afdbcreid 2 days ago [-]
It's not really defense by obscurity (the JavaScript is public), more like defense by... being different?
mattacular 2 days ago [-]
Snowflake defense
chr15m 2 days ago [-]
It doesn't matter what the hash is if it is inherently cheaper for a bot farm to compute the hashes than it is for a human to do it on their device. The human pays a greater cost in annoyance, wasted time, battery, and that means the PoW has failed its function. The bot farm owner does not care.
easton 2 days ago [-]
Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)
jacobvosmaer 2 days ago [-]
Normal clones can reuse delta-compressed data the server stored on disk. Shallow clones impose a negative constraint: do not transfer data outside the requested commit depth. Pre-computed delta chains than contain unrequested data become unusable and the server must do delta compression on the fly to satsify the shallow clone.
Hackbraten 2 days ago [-]
What I find remarkable is that for at least a decade, i.e., long before LLM scrapers were a thing, GitHub engineers have been reaching out to popular package manager projects, asking them to do away with shallow clones [0] [1]. They basically used the same reasoning as your comment did.
If I'm doing a shallow clone it isn't because I only want to receive a specific commit, it's because I don't want to burn a giant amount of disk space and network traffic on a full history.
In most use cases it would be perfectly acceptable for the server to send additional data. The client doesn't care about it because it is meaningless to them, but if it results in a significant load reduction on the server's side they don't really mind receiving it either. A 100MB shallow checkout coming with 400MB of garbage still beats cloning an entire 5GB history!
ygouzerh 2 days ago [-]
It's the point that surprised me the most! We always used shallow clones, to speed the CI, I didn't knew that it got that much impact server side!
radarsat1 2 days ago [-]
Me too! And I'm wondering now.. if it's so awful, is there no way to just block that capability server-side?
Waterluvian 2 days ago [-]
> It was immediately extremely effective — the bots just gave up. For a few months, it was bliss: bots were blocked at the perimeter and gave up, moving on to easier targets; the users were mildly annoyed but tolerated it, and the Anubis stack was easy enough to deploy everywhere.
As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is the CPU cost actually worse than mildly annoying everyone, or is it a problem being solved because it “offends the senses?”
I’m not leaning towards yes or no for this instance. But I regularly see people jumping to conclusions without measuring. What is the cost of 20% and is that cost worth “mildly annoying” everyone?
BowBun 2 days ago [-]
Not sure if you mean the solution, or the problem they were trying to solve.
The problem is the cost here is paid by a group of volunteers, no? It's objectively reducing capacity for a really important public project by 20%. If you're responsible for keeping a public good like this available, it seems obvious to me to want to prevent this overuse of resources. Only one of the 3 parties involved here is not willing to engage in good faith right now and causing harm. I don't understand a need to try and tolerate them.
colinsane 2 days ago [-]
> Is the CPU cost actually worse than mildly annoying everyone
> What is the cost of 20% and is that cost worth “mildly annoying” everyone?
from the articled:
> With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers.
this is not some "CPU use is 20% higher than baseline" situation. it seems that people still do not understand the scale of these bad actors.
Waterluvian 2 days ago [-]
Ah yeah I was quoting 20% of their total capacity.
Nevertheless, without knowing what it actually is in cost, it’s impossible to have an objective sense of the severity of this problem.
Is it, “everyone’s mildly annoyed so we can save $10k per year” or is it “everyone’s mildly annoyed so we can save $100k per year?” And has someone attempted to reason about what the $/year cost is of that mild annoyance?
I’ve seen this a lot in my career. Developers coming to conclusions that are ultimately based on what feels expensive or problematic.
inigyou 2 days ago [-]
Both groups exist. Scraper DDoS is likely to burn 100% of your CPU on git diffs if you host git. But static file sites are unlikely to notice it.
justAnotherHero 2 days ago [-]
While nowhere near compared to their scale, I run a consumer app where most of our users are using the mobile app, with the web app getting perhaps 10-15% of the mobile active users.
However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.
Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.
However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.
I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.
Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.
I still haven't found a way to deal with rotating residential IPs however, and most likely never will.
My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.
Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?
timpera 1 days ago [-]
I had the same problem, tried a bunch of stuff, banned millions of IPs, but nothing really worked. I ended up spending a bunch of time rewriting the website with Codex so that it is more efficient. Now I'm still getting the same junk traffic, but it barely affects the CPU anymore. I think this might be the way to go.
ynniv 2 days ago [-]
this is an increasingly common situation. it goes something like:
- i have a free, niche resource
- it becomes too popular
- i make it more efficient
- now it's really popular, and people are "abusing" it
- let's make them proof-of-work
- ... and proof-of-work harder
- but now "legitimate" users can't use it
- ???
the core problem is that the average person uses a mobile device where work is expensive, and the "attackers" use servers where work is cheap. if you require expensive proof-of-work, next comes a cheap-work-as-a-service where inefficient mobile devices pay small amounts of money to get efficient servers to complete their work for them. now everyone has an interest in making their usage efficient, but there's still an obvious inefficiency in the system: why have people pay unknown 3rd parties to burn cpu cycles to reduce costs for a free service, when you could just have people make small payments that cover the service's costs?
which is called l402/x402. micropayments' day has come
inigyou 2 days ago [-]
What's actually stopping them isn't the PoW, it's the customisation effort. If one site has Anubis nothing scrapes it. If many sites have Anubis they write counter scrapers. Today if you make a slight change to the Anubis algorithm on your site, they'll burn CPU endlessly computing hashes with the original algorithm and submitting wrong ones.
The author of Anubis hates this fact and will ban you if you mention it, so don't. He insists it's the PoW.
Barbing 2 days ago [-]
Searching around for this, found "make users click the mouse three times"[1] as an anti-bot idea. Generally, makes sense that smaller site owners can make small customizations to existing anti-bot tech and see positive results until they're either (1) deemed valuable enough to receive custom attention or (2) the scrapers include LLM-based anti-antibot methods.
>will ban you if you mention it
Even if mentioned really politely?
[1] would hope anyone trying this makes it accessible to visitors with disabilities
QuantumNomad_ 2 days ago [-]
Ask your visitors “what does it feel like being you?”
Anyone who responds with something like:
> I don’t have subjective feelings, consciousness, or an inner point of view, so there isn’t a sensation of “being me.”
is definitely an LLM.
voidUpdate 2 days ago [-]
*They insist
ynniv 2 days ago [-]
sounds right to me. ai will shred it
2 days ago [-]
desterothx 2 days ago [-]
I mean generally what you said is true. But who is actually going to look at the linux kernel from their phone (probably just the maintainers in extreme circumstances)
delichon 2 days ago [-]
> Why is git.kernel.org “interesting” to crawlers
Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.
wiredfool 2 days ago [-]
Seeing the exact same thing on (somewhat high profile) open data sites I run.
The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.
Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.
marginalia_nu 2 days ago [-]
Yeah my search engine saw traffic of up to 160 queries per second the other day from some bot that was ostensibly searching for information on Jack Parsons. Just variations on the same query in different permutations of filters and site:-terms.
inigyou 2 days ago [-]
That would be a different bot, one written specifically for your site. Mainly we're discussing the dumb ones that just crawl all possible http links
marginalia_nu 2 days ago [-]
No I think this is the same case. It seems to just be following local links on the SERP. From a search for jack parsons, you can find hyperlinks to the sorts of requests it's making.
inigyou 2 days ago [-]
Once detected, don't block them because they'll just change strategies automatically, but you can toy with them, like returning a page full of random numbers instead of real data.
wiredfool 2 days ago [-]
I’d love to, but I’m a bit limited in what I can do from a reputational damage POV. They’re my sites, in that I’m responsible, but they aren’t something where I can return incorrect responses.
2 days ago [-]
iririririr 2 days ago [-]
so true. the article authors wishing crawlers will use git instead is so funny because the crawlers don't care at all. they are scrapping everything with brute force. they don't care about your content or effective alternatives, and one more site driving their real users crazy with Anubis is nothing more than a new blip in their dashboard. the crawler operators will not even look at the url.
andrewaylett 2 days ago [-]
Just for fun, because I could, I vibed up a `cgit` replacement that runs entirely in the browser -- point it at a git repo where you've run `git update-server-info` and it'll load files as if it's starting to clone the repo, using range requests and browser caching to avoid actually loading more data than necessary for the view you've requested.
I'm certainly not saying you should use this code, but it's a proof of concept for avoiding the CPU overhead of cgit rendering by loading the data on the client.
It cost me £8.27 of Fable use (from the free credits I've been given) and 56% of my five hour quota on a $20/month Pro plan. There's no server logic, it's 1.3MB of minified JS and CSS and (while I'm absolutely not suggesting anyone try to use it) it basically works: https://github.com/andrewaylett/rgitweb
This is a one-shot, my prompt set the expectation that I'd be able to load resources using CORS but (not entirely unreasonably) the Git hosts I've tried don't set CORS headers.
Shared more because I was pleasantly surprised at how cheap and easy this was -- and with a repo link because talking about it without sharing the link would be a bit crass.
> So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done.
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
DarmokTanagra 2 days ago [-]
Most of these crawlers are probably AI generated in the first place, and AI generated code is garbage.
Thats a good enough explanation in my opinion.
Source: someone who reviews AI generated PR's on a daily basis
phmx 2 days ago [-]
I guess GitHub is in a similar bunch of sources, it should be also more efficient to crawl by cloning.
Anyway, isn’t it the whole sales pitch that it generates tailored solutions fast?
kalkin 2 days ago [-]
This tick about "if AI smart how come crawler dumb" is in most complaints I've read about AI crawlers and I've started to find it pretty annoying. The crawlers might be written using AI but they're evidently not actually running AI inference over the pages they get back--besides being able to tell this from the behavior, if this is pretraining input, that's enormous scale, so it'd mean a large increase in effective training cost. Naively assume inference costs are equal to pretraining costs (probably not true but maybe right order-of-magnitude) and it's a doubling.
This ends up tacitly turning a very legitimate complaint (ill-behaved crawlers) into a justification for head-in-the-sand AI denialism.
bigstrat2003 2 days ago [-]
There's nothing "denialist" about recognizing the utter stupidity of systems that are being mislabeled as "AI". You judge a tool by its results, and the results have been very poor indeed. The only heads in the sand are those whose owners continually refuse to recognize the proofs before their very eyes that there's zero intelligence here.
kalkin 2 days ago [-]
> zero intelligence here
That's head-in-the-sand stuff. AI is certainly very capable of being dumb (as are humans). But:
> “The problem was in need of a new real idea, which this new result seems to provide,” says James Maynard, a mathematician at the University of Oxford. “It seems that the AI has made a genuinely interesting mathematical contribution.”
Nobody a decade ago would have said "oh yeah solving a bunch of open problems in research mathematics, and finding a bunch of zero days in Chrome and Firefox, and winning literature prizes, are things that don't require intelligence."
Barbing 2 days ago [-]
>AI is certainly very capable of being dumb (as are humans).
I wonder how exactly the average scraper got to be so inefficient on kernel.org.
Did someone prompt a SotA model to write the most generic scraper possible?
Did someone prompt an old local model on their laptop to write a kernel.org scraper?
Perhaps no LLMs were involved in the first place. Seems to me there isn't much relation between how good a random scraper is and how usable/effective Mythos/Sol's outputs can be.
yellow_lead 2 days ago [-]
High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore?
> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
MithrilTuxedo 2 days ago [-]
Has anyone considered having Anubis perform more valuable hashing?
Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pickaxes.
IshKebab 2 days ago [-]
Yeah that has been around for many years. Usually by sketchy download sites.
It's not really going to help though because the scrapers using residential proxies aren't burning their own compute.
inigyou 2 days ago [-]
Yes they are. The proxy is just a proxy. All processing is central.
In a few years the VC money will dry up and this gross overspend on slurping data will end.
igor47 2 days ago [-]
Visions of vast data centers surrounded by fields of browning grass, in which aging, rusting, formerly extremely expensive hardware is spending billions of compute cycles looking at anime catgirls
rpcope1 2 days ago [-]
Sounds like an even shittier version of the Lorax. :(
marginalia_nu 2 days ago [-]
It's very likely the last few years of bot behavior is the consequence of the residential proxy business booming. This is indirectly due to AI company crawling, but the fact that they are as cheap and available as they are changes the incentives for anyone using them toward reckless and unsustainable request behavior, as there is no risk of burning your IPs, and very small chances of seeing any consequences of essentially DDoS:ing a website.
inigyou 2 days ago [-]
And the residential proxy business was created by Cloudflare, who was created by us using Cloudflare.
I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.
roughly 2 days ago [-]
Wait, so you’ve been the person who wanted to keep people from scraping your site, the person who’s trying to scrape your site, and the person getting paid to help someone scrape your site? Brother, what are you doing with your life?
inigyou 2 days ago [-]
Welcome to capitalism. Welcome to game theory. Welcome to competition. Welcome to the real world. Welcome to being a grown adult.
roughly 1 days ago [-]
Contingency exists, I get that, and if you're truly in that state, sure, I don't judge necessity. A lot of our cohort seems to confuse a studied disinterest in looking beyond the end of their own nose for the sage wisdom of adulthood, though.
inigyou 1 days ago [-]
How will you know anything about a system you stubbornly only look at one side of? Like saying planes are terrible because they're loud. They are, but have you never traveled to a different city?
roughly 1 days ago [-]
I’ve never mugged someone either, I suppose I should give it a shot before I go passing judgement.
pixl97 2 days ago [-]
Unfortunately we're apt to run into some kind of Jeavons Paradox where the hardware gets so much faster in those few years will be able to slurp massive amounts of data cheaply so the problem never really ends.
KomoD 2 days ago [-]
> High Anubis difficulty is annoying the hell out of me for several sites.
I just close the website if I see Anubis. Some have it set at reasonable difficulties (like 2)… others have it where I need to wait for like 30 seconds, I'm not wasting 30 seconds of my life for that.
colinsane 2 days ago [-]
the brutal truth is that if the website operator simply disabled Anubis, your page load would likely take more than 30s.
when a system was designed for 100 req/s and bots hit it with 5000 req/s, nobody entering that queue is having a good time. Anubis is the trade those operators make just to ensure your request gets serviced at all.
30s load time is already a sign that the Anubis approach is breaking down. if there's nothing else ready by the next order-of-magnitude increase in crawler load, those sites quite likely will just disappear from the public internet. hate Anubis all you want: for most of us, the realistic alternative is strictly worse.
johneth 2 days ago [-]
Just look at another tab while you're waiting if you're that bothered.
CGamesPlay 2 days ago [-]
Git forges seem especially prone to this: tons of information, highly valuable to scrapers, rendered through several different lenses, gives a combinatoric explosion of URLs. Obviously scrapers could just be less stupid and clone the repo, but it's not happening.
It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens the requestor wants, and the server becomes a dumb object storage that uses less resources. The anti-JS folks are free to clone the repo still, and view whatever lens they want, so that minuscule slice of the legitimate requests is still served, albeit with a degraded experience.
zbentley 2 days ago [-]
That'd fail for three reasons.
First, scrapers would start running JS. Whether they're running chromium-in-a-box or something more clever doesn't matter. Compute is effectively free for them, and even shitty WebOS set-top boxes can probably run a stripped-down headless browser with a JavaScript engine.
Second, a full clone in browser memory is massive for something like the Linux kernel. Lots of browsers and pro-JS folks wouldn't have the resources to run that.
Third, assuming that scrapers are willing to play the JS game, suddenly you have a massively increased rate of full clones happening. Even if serving raw git is cheap for your backend, the bandwidth the elevated clone count drives is not.
CGamesPlay 2 days ago [-]
1. Preventing access to open-source code by scrapers is not the goal behind this idea. The goal is to reduce the server load taken by the scrapers.
2. Why would you need a full clone to access /blob/cee9395acd8043be0644b25c34bfa86623f2b935/block/badblocks.c?
zbentley 2 days ago [-]
I don't think the show-commit or file-at-revision routes are what's causing the bot load.
Rather, the article describes the routes that compute views (e.g. show the change history across several commits) into history as being the issue. If we take that example, a JS client would have to fetch N individual commits/routes/objects and compute the requested view, but first it'd have to fetch the indexes/logs to determine what commits exist within e.g. a specified range.
I suspect that'd require more work on the frontend than "just run WASM-built git" ... unless the proposal is for it to fetch all requested objects lazily, in which case I think you'd be surprised how many files are read by git when answering a question like "show me the diff by user XYZ in file ABC on branch QRS between date 1 and date 2". That starts to get expensive to pull in the browser, and the bandwidth costs might start to hurt even if the backend now only had to serve cacheable dumb blobs.
CGamesPlay 2 days ago [-]
This is a much more substantial critique :) I definitely don't think this is a trivial idea. I did a quick analysis and the naivest possible solution isn't very good: only looking at uncompressed objects requires scanning ~340 MB to deliver the <1 MB answer the question "show me the diffs by user Zhang Yi in file fs/ext4/inode.c on branch master in 2025". The bulk of that is scanning commit objects for commits by Zhang Yi in 2025 (171 MB), then fetching trees to filter those to just fs/ext4/inode.c (159 MB).
But this begs the question, why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes? (Making a space-time tradeoff that doesn't make sense on a public-access website in a post-stupid-AI-scraper world?)
zbentley 1 days ago [-]
> This is a much more substantial critique
Thanks! FWIW, I engaged because I think there's the basis for a good approach here and I want to see if we can expose what works through discussion.
> why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes?
That's the critical question (ugh, as I write this I realize I'm starting to sound like an AI, sorry). It comes down (I think, I don't work on kernel.org) to a few things:
- cgit is ubiquitous and easy to deploy.
- Cgit doesn't have a database, or indexes. It's a fairly thin wrapper over git itself. Git itself is optimized for performance of local filesystem operations, which is nice, but means it can be written with e.g. the page cache in mind for performance, and doesn't need to spend a ton of time thinking about the number of files it accesses--after all, they're all small and locally-available, and thus probably cached.
- Things like cgit often have to combine (the C equivalents of) multiple chained git commands to render certain views.
- Cgit has caching, but it's simultaneously too naïve and too specific to handle use-cases like this. It's too naive in that it's caching content blobs via the filesystem (and using sendfile to serve them) without awareness of e.g. shared cache blocks for deduplication or anything, so it can't cache intermediate states used for rendering e.g. diffs. It's too specific in that it's only caching the results of certain underlying git operations and specified renders, so it can't prioritize e.g. cross-application LRU/frecency patterns when deciding what to cache and what to evict.
In short, what you're proposing is probably best implemented server-side (and hopefully already exists): it's an indexed database of Git's objects that has behavior parity with Git's rendering of the various views/questions people can make of the server. That's a nontrivial undertaking: git doesn't have a VFS layer you could plug a database into the way something like SQLite does; it very strongly prefers procedural file-based accesses to its data. If you could use something like that, you could engineer it like a typical webapp: content-unaware caching layers and indexes to make certain operations fast. Not easy to build, and not easy to swap in in place of cgit (which you also probably have to keep a lot of behavior parity with), though.
stickfigure 2 days ago [-]
How about just stop offering a html interface to the code? This doesn't seem like a critical service. Let people clone the repo normally.
If someone else wants to run a public HTML service, let them deal with the bots.
If you really want to offer a web interface, put it behind login. You can apply enough restrictions (captcha, super slow rate limit for new accounts) that it isn't cost effective to generate zillions of logins, and you can monitor logins for bot behavior.
Sucks, but here we are.
oasisbob 2 days ago [-]
I think you underestimate the difficulty in effectively gating user registration. If the anti-bot efforts in general don't work for other site pages, they won't work for signup functionality either.
stickfigure 2 days ago [-]
I've been on the other side of this kind of thing (hero rather than villain, though I'm sure someone out there disagrees).
You can fairly arbitrarily increase the difficulty of user registration, far beyond what users will tolerate for viewing an individual page. You can exploit this; a bot needs to make many accounts for the activity desired, and it's not hard to make "generate account" more expensive than it's worth for the amount of activity they get from each account.
If it costs your attacker a penny to solve the captcha to make an account, and they can only get 100 pages out of an account, you win.
TowerTall 2 days ago [-]
What if you make MFA mandatory. Normal users should have little issues with that despite the increased friction but bots would struggle with this.
oasisbob 2 days ago [-]
I think this asymmetry is a fantasy.
For many sites, your hypothetical penny to create an account is a cost an attacker would gladly pay.
marginalia_nu 2 days ago [-]
FWIW, git hosts have always interacted very poorly with crawlers, to the point where you have to actively code in git host detection to avoid getting stuck in an accidental crawler trap if you want to run a well behaved crawler. Easiest is just to look for anything that looks like a commit hash in a path and drop those URLs from the crawl frontier.
Reason they interact so poorly is that is that git hosts generate a lot of links. One for each file in each commit, and a diff for each file appearing in a pair of commits. Even a small repo can have millions of viable links, and most of these are stupidly expensive to render for the git host. On top of this crawlers generally don't have a very deep understanding of what they are crawling, and can't meaningfully distinguish computationally expensive requests from cheap ones.
nneonneo 2 days ago [-]
I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit. It could even have a little hint that says something to the effect of “go visit the home page if this is taking too long”. (Better not to ban them entirely, in case people really did click on some random link e.g. in a news story or mailing list message).
Distributed scrapers are going to generally try and hit their assigned list of pages; it’s a bigger waste of time if they have to go off to visit other pages first to get the cookie challenge.
colinsane 2 days ago [-]
> I wonder if one solution here could be to turn up Anubis difficulty if the first page hit is not one of the obvious entry-points to cgit.
Anubis has a fairly capable "policy" system. you can place something like this in your policy.json:
another thing smaller sites benefit from -- where the load induced by crawlers tends to be bursty (e.g. as they discover new expensive endpoints to crawl) -- is to adjust the difficulty up/down to maintain a steady system load.
Almost all of my visits to cgit instances are through direct deep links. Hard to imagine someone randomly browsing git listings
nneonneo 2 days ago [-]
The kernel folks likely have a good profile on what page people trigger Anubis on (i.e. what page people hit first). From that they could make heuristics about what pages are likely to be useful deep links.
Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.
NooneAtAll3 2 days ago [-]
from what I see there are 2 solutions:
1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally
2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficulty can also make spam pulling harder
lxgr 2 days ago [-]
> straight up go to every representative there is
In every country of the world? I think you might have a significantly easier time convincing Samsung and LG to ban this via their T&C.
Only a few countries would be enough, the rest will get banned until they pass apply similar policies.
johneth 2 days ago [-]
If there was a way of detecting the proxy SDKs, it would be a simple matter of banning the device. Or at least naming and shaming the apps that use them.
zbentley 2 days ago [-]
I'm sure there are ways of detecting residential proxy clients that are reasonably effective. I'm equally sure that the people in possession of those techniques (thinking CDN providers, big social networks with public links, newsrooms) are unlikely to share their secret sauce in public. Doing that loses a competitive edge in cost-of-hosting, and as soon as you go public the residential proxy makers can adapt.
johneth 2 days ago [-]
There's JA4 fingerprinting (and the JA4+ suite), but still not a surefire way of detecting. Plus the metadata for the signatures has to be constantly maintained and updated.
inigyou 2 days ago [-]
3) just track down the people and arrest them. DDoS is already a felony, and residential proxies aren't untraceable to law enforcement.
voakbasda 2 days ago [-]
I would think that residential proxying would be illegal already, as it is a network intrusion. The trick is chasing down the offenders, proving their actions did harm, and getting them to pay. None of those steps are easy, even if there are laws to assist. Otherwise, spam would be a solved problem.
inigyou 2 days ago [-]
It isn't illegal, there is no law against "network intrusion" which is a term you just made up, and if it was a real term it probably wouldn't cover this. There are laws against things like "unauthorized access to a protected computer system".
voakbasda 1 days ago [-]
Patayto, potahto. You care to explain how a “network intrusion” is not an “unauthorized access to a protected computer system”?
inigyou 1 days ago [-]
What's a network intrusion?
alwa 2 days ago [-]
Is it still an intrusion if the user accepted the shrinkwrap TOS of an app that trades them “free TV” in exchange for allowing that app to operate a proxy (via an “app monetization” SDK) on their network?
voakbasda 2 days ago [-]
Yeah, no one actually agrees to all of the individual terms in EULAs. That’s the first sign that the law will be nearly useless to address any aspect of these problems. It is already one-sided, and that side is not a friend to the consumer or general public.
Hackbraten 2 days ago [-]
Good luck enforcing the law against a company whose entire business model is to piggyback residential-proxy-as-a-service over the shitty FNQQOIIWKNURQB brand $60 projectors they flip on Amazon. You call them out, they’ll just say they stopped doing it and go on with business as usual.
ivanjermakov 2 days ago [-]
There has to be some not-yet-discovered way to have a capcha that is easy for any human but impossible for robot. Too bad capchas hurt user experience no matter how easy they are.
Another solution I came up with while reading HN comments: whitelist IPs instead blacklisting. Give access to well-behaving hosts/groups. It can even be shared across different sites. Although this would create a market for selling "good IP" proxies.
zythyx 2 days ago [-]
There are definitely ways of proving you're a human, unfortunately it also means giving up your privacy and anonymity (IRL ID Checks combined with appropriate routing and validation - even going as far as certifying the browser being used)
Obviously though none of us want to give that up, so the alternative is that we can almost never 'prove' we are human especially with bots getting as smart or smarter than the average redditor.
Barbing 2 days ago [-]
Imagine proving you have no financial incentive to get on the whitelist.
But! Using a network of real ID-checked humans to scrape the web, what would that be--half a billion times harder than Firecrawl or whatever they use today?
Too bad it's dead in the water today because so many (like me) hate the idea so much. Perhaps a biometric dongle (retinal-scanning orb :-/ ) that the staunchest privacy hawks stamp with their seals of approval because it's somehow nearly impossible to go horribly horribly... horribly... wrong... Yeah, anybody who can crack this issue, hope you have the free time or find the funding to try it, we need ya.
desterothx 2 days ago [-]
I vote for a finger pricker that tests the blood composition everytime I want to look up the nearest open grocery store
MetaWhirledPeas 2 days ago [-]
> whitelist IPs instead blacklisting
I think I agree with this. There will be difficulties though:
- Will likely have to be pay-to-play. No point in whitelisting IPs if you are bombarded with whitelist requests by a bot.
- Will come with the assurance that any bot traffic will result in an IP ban.
- Will rule out people with dynamic IP addresses.
- Will be difficult to gain new users.
mkl 2 days ago [-]
> There has to be some not-yet-discovered way to have a capcha that is easy for any human but impossible for robot.
Why? We are not so special or magical.
brownkonas 2 days ago [-]
The reverse turing test :(
echelon 2 days ago [-]
Even better: make people pay for access or vouch to give access to a third party.
ivanjermakov 2 days ago [-]
Some kind of proof of stake might be viable. "I as a visitor stake 1 cent that I'm a genuine user and not a sloppy bot, server is free to withdraw my stake if it's not true". If works, withdrawn money can be used to cover hosting costs.
__MatrixMan__ 2 days ago [-]
This is a fundamental flaw in the web. Since we treat a server's name as authoritative, anybody maintaining a replica needs to repeatedly hit that server to know if their remote version is up to date.
If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct for each interested party.
CDNs mitigate this only partially, because mutable data means they have no good cache invalidation strategy.
There's got to be a solution that doesn't involve heaping even more burdensome requirements on those who would dare to publish.
dennis-tra 2 days ago [-]
Content-addressing decouples the hoster from the data itself. Anyone can serve content-addressed data and you can locally verify that you’ve got served the correct bytes. IPFS implements building blocks for such an alternative web.
What irony that this article is about crawling content-addressed data.
__MatrixMan__ 2 days ago [-]
Agreed on both points. But it's looking increasingly likely that particular dream is not coming true. Kubo, the reference implementation IPFS node, is maintainerless as of last week.
I'm not sure where IPFS went wrong, but I think our web will continue to degrade until we figure it out. (It'll continue to degrade after that also, but then we can let it burn since we'll have a replacement to switch to).
__MatrixMan__ 2 days ago [-]
I think focusing on filecoin was probably the mistake. You've got to build something that people trust first and then consider adding a money-shaped app. If you start with something money-shaped you're indistinguishable from the legions of scams, and that's a hard position to start from if you're wanting to build something trustworthy.
inigyou 2 days ago [-]
Or poll a merkle tree root hash, like, um, git
__MatrixMan__ 2 days ago [-]
What I'm after is a way to get away from polling by server name. With a sufficiently interested audience I ought to be able to be offline except when I need to update the merkle root, meanwhile users can just serve the content to each other.
inigyou 2 days ago [-]
Do you know Zooko's triangle?
__MatrixMan__ 2 days ago [-]
Yes. I don't think human meaningful identifiers are important. We can make do with local pet names, like how Unison works.
We hardly even use the globally consistent names we have. Between humans its always "hey did you check out that website I sent you?" And never "hey did you check out https://foo.bar.biz"
Sharing and endorsement can just be part of the protocol. My reference to the site may as well be in terms of which of my friends think its trustworthy so I'm better enabled to make that determination myself (some of my friends have better taste than others).
Knowing that Alice.brownierecipe.2 and Bob.favoritebrownies refer to the same underlying document with an unfriendly global name is not so gnarly of a UX problem that we need to tolerate single points of failure to avoid it.
chuckadams 2 days ago [-]
Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
rcxdude 2 days ago [-]
The issue is there's far more pages than could reasonably be stored: cgit is an old-school server-side renderer and there's a combinatorial explosion of possible pages it could render. The article calculates there's about a quadrillion different pages that could be 'crawled'
charcircuit 2 days ago [-]
Most of those pages are going to be identical if not for some very minor differences. You can make it much much cheaper to put together these pages as opposed to the expensive operations it has to do right now.
skybrian 2 days ago [-]
Apparently the current implementation is very inefficient by modern standards:
Rather than using Anubis to give the client fake work, maybe the way to go would be to get the client to do the real work in JavaScript? Then the server doesn’t have to do it.
edward28 1 days ago [-]
Maybe it's time for AI to rewrite it in rust.
kees99 2 days ago [-]
Or, html generation could be very, very fast in the first place. For example, serving static files is usually just as fast as a reverse-proxy cache.
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
My first thought was put varnish in front of it as well. But given how wide the url scheming is, the valid urls seem to be in the quintillions, if i counted correctly, its unlikely a naive caching solution would work.
But given that it's only a few million actual commits it may be possible to come up with an intelligent caching system that at full capacity require only a few TB.
mik1998 2 days ago [-]
The cache gets evicted between the times the old commits are revisited.
oowa 2 days ago [-]
using the current solution and a cache pattern doesnt solve it, but it is an "affordable" solvable problem. but first we need to ignore OP said its ok for now...have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
Zariel 2 days ago [-]
That was my first thought, varnish (vinyl these days) in front of the website should dramatically reduce this as the git repo should be practically static for most of the content.
teo_zero 2 days ago [-]
But there are "cubic bazillions" of possible URLs that are being requested. Even if they boil down to "only" some millions actual commits, their rendered HTMLs are all different.
kijin 2 days ago [-]
Yes, this is very difficult to solve for sites with many URL variations, like git repos and heavily threaded forums. Your cache is always full, but the hit ratio is abysmal.
oowa 2 days ago [-]
remember, its only difficult for you. at the end of the day its just 7gb of data...
oowa 2 days ago [-]
no its only difficult for you. and only at this moment... any minute now you will see the way. btw, the OP is just saying "its OK for now". And the OP is just telling us: this is what has been happening... maybe its difficult for OP also, but they didnt say that. they just said their current setup cant handle it. old tech...have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
oowa 2 days ago [-]
have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
oowa 2 days ago [-]
i think its only 7gb of data total, therefore there should be a strategy to fully optimize for this deterministic scenario, but instead of caching instead with maybe compiled streaming renderer(s)... but he said its ok for now.. and anyway why should they need to fix anything, the slurp is what sucks. pun intended
oowa 2 days ago [-]
anyway, I have solved this exact scenario many times in the last 30 years... thanks for the dv.. have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
AshamedCaptain 2 days ago [-]
This is bad enough that I'm going to stop serving cgit. I've been doing cvsweb, then subversion, then cgit over my home server for many many years and for the first time ever this is annoying my own bw usage.
It's ridiculous also how you ban an IP then 1 second later another one picks up from where the first one left on.
rwmj 2 days ago [-]
I had to take down my cgit repository a few months ago. The load was causing other VMs on the same machine to become unusable.
singpolyma3 2 days ago [-]
I've been using git-arr instead with some success
ksimukka 2 days ago [-]
Remember SETI?
Maybe we should serve the bots a problem worth solving and benefit both parties. They spend some energy/tokens on a problem and we pay them with content.
If only I had a bot problem, this would be interesting to explore.
desterothx 2 days ago [-]
The trouble is still differentiating between real users and agents. Sure this could be useful (forcing agents to spend a certain amount of compute towards something actually useful), however real users that get stuck with it would be stuck for an even greater length of time, because the compute required to actually do something useful is beyond the reach of most consumer devices. An rtx pro 6000 costs .76$ per hour for renting, so quite a lot of compute needs to be done for it to be considered "worth it" for the website
superjan 2 days ago [-]
How feasible would it be to only offer a binary git (partial) download and move the html rendering to the client? It would still be a lot of requests, but less work for those servers. Not that I like SPA’s, but they could be useful here.
Artoooooor 2 days ago [-]
How expensive would AI access be if every user paid their fair share instead of shoving it on the people doing the actual work?
parineum 2 days ago [-]
You mean shoving it onto the investors?
inigyou 2 days ago [-]
No they mean the people doing the actual work, I think.
parineum 22 hours ago [-]
The people doing the work aren't paying for AI, they're getting paid by AI.
The people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.
inigyou 17 hours ago [-]
I'm getting paid for AI to scrape my stuff? That's news to me. How do I claim it?
parineum 3 hours ago [-]
Do you get paid when people "scrape your stuff"?
wredcoll 2 days ago [-]
See also: the price for uber rides.
NegativeLatency 2 days ago [-]
Claude loves doing this on GitHub repos too, I have line in my agents file to tell it to clone to tmp and look there.
initramfs 2 days ago [-]
I've been noticing page views in the past several months with a much wider span of origin on my Blogger stats. Before I would get a few from several countries, but now I am getting views from tiny countries and obscure or outdated browsers and operating systems, which leads me to think scrapers could be using VPN services in various countries along with header anonymizers that mask the device that they are using.
Great for AI scrapers, bad for hosters and everyone else.
8474_s 2 days ago [-]
The solution is allowing the convenient web interface only for trusted members, like e.g. 4 year old account can view all pages(with easy anubis setting) but guest/bots/everyone else has to git clone the thing and do it on their backend.
IIRC old forums also limited 'content for registered/trusted/moderators/etc' content views decades ago, turns out this crap saves gigatons of traffic.
throwawayffffas 2 days ago [-]
That excludes new legitimate users, just put up a very cheap paywall for the web interface, 1 dollar for lifetime access, that way each request is tied to a user and can be rate limited. While remaining essentially free.
throwawayffffas 2 days ago [-]
Stop rendering html on your side. So simple, just add a js dependency that does the rendering on the frontend. You just serve flat files and the diffs. All the rendering happens on the users device, legitimate users will probably not even notice, bots won't notice either, your cpu usage will go way down.
It's like anubis but instead of doing useless math, you will be moving the legitimate cpu work to their side.
pbronez 2 days ago [-]
“Expect to lose some functionality, at least when accessing our resources anonymously.”
This seems fine to me. It would be a better world if we could have anonymous bulk data access. But if aggressive scrapers are bloating host costs, I’m fine with logging in.
Now, the flip side is that ONCE logged in, I want my bulk access. The worst of all worlds with when you demand authentication and then STILL block bulk access.
Case in point, I want to automatically download my Amazon and Target order records. This is easy to automate with playwright or whatever, but authentication stays annoying. My sessions expire quickly and I have to re-auth all the time. There should be an API to pull this data down.
kev009 2 days ago [-]
I run a web to text usenet gateway and it has been an interesting challenge to scale it to deal with this. Article fetch and retrieval is extremely efficient but I apply JWZ threading and that can cause a single article to make a number of overview requests which are more expensive depending on the depth. I solve it currently with caching but will eventually implement a thread backend on the NNTP side to keep thread roots updated at insertion time and it will be a very cheap read request.
One persistent thought is what are people doing with this data? I get that people want to train models, but I also have a hard time believing there are more than a hundred companies with the resources to spider the web like this and actually do anything meaningful with all that data. Academics, researchers, and people working on lower level innovation are probably well off with CommonCrawl.. only people trying to make frontier models really need fresh and endless data right?
forrestthewoods 2 days ago [-]
> Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.
TIL shallow clones are expensive. That's wild to me. It's supposed to be cheaper!
Backslasher 2 days ago [-]
I thought they were expensive compared to fetches from established repos. TIL they're also expensive compared to full clones.
schobi 2 days ago [-]
I'm puzzled as there seems to be a clear pattern on how a human user would look like vs a bot.
High bot likely hood if: If a session jumps to a different ip. If the session jumps IP after just a few requests. If a new blank session starts with a deep link. Maybe those are cases where some POW is better justified?
Assumption: the rendered HTML might be viewed by a legitimate developer, even via a deep link from outside. But rarely from a wget script without a session cookie .
The other nice idea from the comments - is this rendering effort something that can be pushed to the user? Instead of pointless POW work, can you offload the expensive rendering to the user side?
But certainly, this is just an armchair comment and the kernel guys certainly have tried everything in this arms race...
dewey 2 days ago [-]
> But rarely from a wget script without a session cookie
The article mentions that some requested pages are less likely to be legit traffic (old commits) and more likely to have bot activity. Perhaps they could increase difficulty on those pages for the proof of work. Keeping the "current" at a lower difficulty allows most normal users to use the pages as normal, while penalizing the bots.
One other way I have been thinking of is just delay the delivery of the pages, either limit bandwidth or just wait for a bit until you deliver the page. For one user a (lets say max)3s delay on some pages is not a huge deal, however at scale that adds up and means the client can't gather other pages in the meantime.
Another option is to lock the out of date html renderings behind a login page while keeping the git openly available.
nxobject 2 days ago [-]
> One other way I have been thinking of is just delay the delivery of the pages, either limit bandwidth or just wait for a bit until you deliver the page. For one user a (lets say max)3s delay on some pages is not a huge deal, however at scale that adds up and means the client can't gather other pages in the meantime
Or some kind of vintage “set up a request in a form and press a submit form, and we’ll pretend to take a while to put things together.”
trashb 8 hours ago [-]
However you can implement it. There are several options but probably you'll want to do it server side since I believe most bots don't run javascript. If you search there are ways to implement something similar in nginx using modules or buildins.
yardstick 2 days ago [-]
How about
Allow git clone for free/unrestricted still.
Require the user to sign in to view html views. Sign in require a valid email or phone where a validation link is sent.
Or: Users signed in won’t see the Anubis. Users not signed in can still see the html views but have to use a very high work level?
Or: Limit unauthenticated requests from an IP to 5/minute. Authenticated requests can do a lot more before hitting the limit.
Barbing 2 days ago [-]
Would it help if this only applied to old pages?
Want to render than seven-year old commit via HTML? Sign in. (But I have no idea.)
javcasas 2 days ago [-]
At this point they are using residential proxies and stuff, and increasing the difficulty level is not going to help, among other things because they don't pay for it.
Why cannot we turn this whole proof of work thing into an official "help mining $SHITCOIN"? I mean, if they really want the data that badly, at least have them pay the hosting with their CPU/GPU/ASIC cycles.
yread 2 days ago [-]
We need a solid way to prove we are meatbags. How about a simple USB accessory that goes around your neck and measures your ECG. It could also give you subtle electrical shocks (in a unique pattern; a challenge) and measure how the ECG responds to that.
desterothx 2 days ago [-]
im rather fond of a blood tester that gives you unique drug cocktails as a fingerprint
iririririr 2 days ago [-]
anyone knows how Jwz solution is working?
dont click next link because he will show a nutsack image if the referrer contains hackernews. love the guy.
basically, instead of blocking, he just poison it. and if a human sees it, it takes less effort to ignore the nonsense than it takes your pocket computer to deal with proof of work.
feelamee 2 days ago [-]
Hm, interesting - how will it look the actual solution for such problems in the future. I suppose the issue will continue to grow.
First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider.
So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.
klez 2 days ago [-]
> there should be some cost for sending traffic somewher
So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks.
I don't have a solution, but "break how the web currently works" is not one I would accept all willy nilly.
EDIT: yes, I realize we already broke the web (with Anubis, cloudflare, recaptcha etc) but I think we should resist slowly breaking it further.
feelamee 2 days ago [-]
> So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks.
First of all - I suppose it should be very cheap. So, real humans will not pay much.
Second - why do u think that websites are meant to be free? They provide some service, so its a rather strange that the internet is so free (in both senses). I think, this freeiness is allowed to greatly speed up popularization. But for me is obvious that it can demand payment for service.
And third - service owner really meant it to be free, I don't see any problems with this in my idea. It can still provide free service.
klez 2 days ago [-]
I'm not saying websites can't demand payments for service, I'm just saying it's bad if it's a necessary fix for "scrapers are destroying the basic social contract of the web".
inigyou 2 days ago [-]
The internet is already like that, but for some reason the payment only extends as far as the recipient's ISP, not the actual recipient. Most senders pay a flat rate, but their ISP doesn't.
rwmj 2 days ago [-]
It's a social problem, technical solutions aren't going to work. To solve social problems you need the law to get involved. It should simply be illegal to install malware on someone's TV and use it to hijack their internet connection, and the punishment should be the same as for any theft of service, including imprisonment for people who do it.
feelamee 2 days ago [-]
The law will not work without a technical solution.
So, you need both. But, at least, technical.
inigyou 2 days ago [-]
You just take a sample of the crawling to whichever police department knows how to deal with DDoS, they subpoena the ISPs involved and trace who's using the proxies?
zbentley 2 days ago [-]
> just
> [four phases, each of which has historically been hard-to-impossible for highly organized international coalitions]
Never going to happen.
inigyou 2 days ago [-]
Did you try? What were the results?
kristianp 2 days ago [-]
One problem with Anubis is that once you've solved the POW once, you just need to hold the cookie to avoid solving it again. Scrapers have probably learnt to do that by now. So Anubis isn't as effective as it used to be before it was widely used.
edent 2 days ago [-]
WordPress powered a huge number of websites. Yet the crawlers all go straight for the HTML of those sites rather than the more efficient and structured JSON API which all WordPress sites have.
If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?
kardos 2 days ago [-]
Because they suspect that, sometimes, different content will be served by HTML vs alternate APIs
ironqcold 2 days ago [-]
My takeaway: we're degrading the web for real people to slow down bots that will just move forward. The solution seems is worse than the problem. At some point, we need to accept that the open web as we knew it is dying...
fschuett 2 days ago [-]
Maybe we should think about laws to make this "scraping without caching" behaviour punishable, i.e. "if AI companies do scrape sites, they're required to implement caching and use the most efficient method and not abuse the other persons resources". Maybe difficult to do in practice, but at least it could be a deterrent to some extent. Where are all the environmentalist politicians when you actually need them?
0xbadcafebee 2 days ago [-]
I'm assuming they haven't yet sent responses to the bots? Since AI is dumb, you can send errors that tell the bot to git clone rather than crawl. If it's vulnerable to prompt injection, it might listen and do the clone instead and stop trying to solve challenges.
Barring that, I think the solution is to charge money for access. Require users to sign up to render HTML, and provide a form of payment (any form you want). The cost is, say, $0.1 per GB. Rate limit all requests to reduce CPU. For the average user this will cost a few cents. For the bots you'll cover your costs and have a rate limiter to keep your system from being overwhelmed. Or they can git clone for free with no limit.
2 days ago [-]
bauerd 2 days ago [-]
They're the exception, not the rule. They get crawled like any other site, but happen to host git repositories. It's not obvious that these are targeted crawls and they likely may just end up in crawling queues a lot generally
a-dub 2 days ago [-]
i wonder what they're all up to. i imagine some are scraping datasets for pre-training, others are probably real-time scrapers looking for security bugs, even more still are agents working on coding tasks and looking at the kernel. also interesting to think about solutions: does everything need to be optimized now for weird access patterns that proliferated ai creates? do the ais need to have behavior trained in to be better netizens? is this the end of anonymous browsing and the beginning of an era where one has to attach an identity to all requests? or the end of community hosted free information services more broadly?
jruohonen 3 days ago [-]
Off-topic, but anyone with which he did the plots?
> Training an LLM on content produced by the LLM gives it the equivalent of a digital prion disease, so when a source is guaranteed to be LLM-free, like the entire history of kernel commits, it's worth its weight in gold as a source of training data.
Except the majority of LLM training content nowadays is synthetically generated by LLMs. I wish people would stop making this statement, I don't know why this claim persists to this day. It wasn't true two years ago and it sure isn't true now
hamandcheese 2 days ago [-]
I wonder how much is for training vs for LLMs doing research. On several occasions Claude has gone digging through kernel archives on my behalf (sometimes at my direction, other times all on its own). Usually to determine the current status of some kernel bug I'm experiencing. Apologies for the load, but I'm sure it was much less than an actual crawler trying to slurp up everything.
wolttam 2 days ago [-]
I think the solution is for the POW being done by the clients to *actually benefit the site owner*. Users remain just as mildly annoyed as with Anubis, but maybe a bit less knowing that the work they’re doing benefits the site owner/author, and the system helps thwart the bots (or at least makes them do work that benefits the author).
cute_boi 2 days ago [-]
All this happens due to companies like browserbase, Hyperbrowser, Scrapefly. These service exists to facilitate such operation and they aren't doing anything to prevent abuse. They are infact selling way to bypass captchas etc...
I think any service that is trying to sell a way to solve captchas must be banned by government. At least these things shouldn't be done so openly.
throwawayffffas 2 days ago [-]
> worth spending a ton of cycles to calculate the Anubis challenge.
Not sure how difficult Anubis is, but I would not be surprised if parsing the request through the LLM costs multiple orders of magnitude more compute than the Anubis challenge.
CqtGLRGcukpy 2 days ago [-]
I just had a look through my logs, and I've had over 90,000 requests from known AI bots over the last month. All this to a personal website that doesn't post very often.
And that's just known AI, I can't imagine what requests are pretending to a real person when they aren't.
jopsen 2 days ago [-]
I've seen this too.
I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..
They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.
Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.
iamniels 2 days ago [-]
I run a website with 10k unique pages. If I leave the gates open, Meta hits it 200.000 times per day. Every day. What are you paying developers $500k for Mark?
jopsen 2 days ago [-]
200k req/day = 2.3 req/s
That's bad for static content.
Try adding search, with pagination and 16 filters that can toggled on off. And 1000 tags, give each page 5.
mbirth 2 days ago [-]
I've simply blocked whole ASNs for Meta, Google, AWS, etc.
DarmokTanagra 2 days ago [-]
AI has simultaneously made the easiest parts of web development even easier while making the hardest parts near impossible.
nxndbebdb 2 days ago [-]
Just serve the raw commit and render on frontend. I really don't get why they are complaining, just be performant
RussianBot9580 2 days ago [-]
And for a shallow clone you would serve... what?
hdbsbs 2 days ago [-]
No need for a shallow clone, just let the Frontend fetch the relevant objects from a static file server
2 days ago [-]
BorisMelnik 2 days ago [-]
it's getting insane, I have a high profile client, I manage their infrastructure including web server. I swore to them years ago they would not have to turn on the CF managed challenge / under attack / human verification.
I've handled every type of attack and malware that came their way but these past few years, ai scrapers are a large por or their traffic, eating into the budget and now interfering with sales.
and I don't know if anyone else is noticing or watching these ASNs but it sure looks like a few well know and big name AI companies are using *residential proxies* to so their scraping.
cobbzilla 2 days ago [-]
I ended public access to my git server after I got flooded by bots and my own commits were noticeably lagging.
That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.
golem14 16 hours ago [-]
why not require an account for html access and otherwise just serve say a pre-cached complete repository requiring minimal work?
cobbzilla 13 hours ago [-]
Because I’m lazy. It has HTTP basic auth. If anyone wants access they can message me directly.
golem14 6 hours ago [-]
Yeah, I meant for kernel.org
It's hugely inconvenient for users not having an account and want to do more than clone an entire repo via an API of course.
smetj 2 days ago [-]
Offer the content, which is otherwise scraped, as a tarbal + diffs via a torrent or p2p network making it more efficient and cheaper for the crawlers to obtain the data.
virajk_31 2 days ago [-]
Am not an expert. Putting medium level interactive challenge for first time IPs in X time duration, wouldn't this reduce the bot traffic..
bfrog 1 days ago [-]
This greatly implies online advertising is dead, that the internet as something useful to humans is soon becoming questionable.
yuumei 2 days ago [-]
Here is an idea: instead of PoW do “proof of AI” each user who wants to access the page run AI/LLM prompts to undercut the corpos abusing the pages
tarpitt 2 days ago [-]
Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.
NooneAtAll3 2 days ago [-]
it's hard to separate spambot that only accesses 3-5 links per IP and a legit user. Changing content for legit user can be devastating
inigyou 2 days ago [-]
Put a cookie wall in front. The bot will either load the cookie and have a persistent identifier, or not load the cookie and not get in
jay_kyburz 2 days ago [-]
why is this not the answer? then you can also rate limit each cookie as well.
sgsjchs 2 days ago [-]
it'll load the cookie, make one request, move to a different ip, load the cookie, make one request, move to a different ip, ...
inigyou 2 days ago [-]
Why are you serving new cookies to someone who already has a cookie?
sgsjchs 2 days ago [-]
It discards the cookie after one request and appears as a new user.
inigyou 2 days ago [-]
That implies it is specifically written for your specific website
sgsjchs 1 days ago [-]
How so?
inigyou 1 days ago [-]
Why else would a thing be programmed to drop cookies after every seconds request?
sgsjchs 1 days ago [-]
Maybe I misunderstand what you mean by "load the cookie". What is it if not Set-Cookie header with redirect status? That doesn't seem particularly special to me.
inigyou 17 hours ago [-]
Set-Cookie header, 200 status and a JavaScript redirect? Many ways. Set-Cookie on pages that load quickly and check it on pages that load slowly?
sgsjchs 12 hours ago [-]
What happens when a user without the cookie GETs a slow page? Does he get redirected to fast one to and back?
jay_kyburz 22 hours ago [-]
You don't just redirect, you sleep 2 seconds, then respond with a redirect.
jay_kyburz 2 days ago [-]
If you are discovering urls you have to wait for a previous request to finish. The rate limit should work. Requests without a cookie wait 2 seconds. Request with cookies can only make human scale number of requests per second? (1?)
If you have a thousands of IP addresses, and you know all the urls you want to request in advance, you can just request them all simultaneously I guess.
The next more advanced version is that URLs are unique to your cookie.
Users can't share urls anymore, but it might be a tradeoff worth making. Unique urls for each user.
You could probably still make this work, if you share your url with another user, they get the page, but heavily rate limited like a regular no cookie request.
(a cookie url mismatch gets the rate limited version of the page)
api 2 days ago [-]
The AI companies should have their AI fix their crappy inefficient crawler code.
inigyou 2 days ago [-]
I also had this problem, but since nobody actually uses my gitea site besides crawlers, I just let a script run through my access log and ban every IP address who asked for a commit in the last 24 hours
_blk 2 days ago [-]
Why not use the POW to help cover the costs? Mine an actual coin (Annubis Coin?) and pay for anonymous infra access with it (or log in and get a certain quota for free)?
beached_whale 2 days ago [-]
I wonder if they could pre-render the stuff older than a month ago and compress it and serve it as static content. Not optimal, trades space for CPU, but might be cheaper.
afarah1 2 days ago [-]
Why not aggressively rate limit? Legitimate use of HTML rendered commits should be largely unaffected, and crawlers slowed to a halt. You can even jail after a number of 429's...
mattmcal 2 days ago [-]
There is a section in the article answering your question if you read it.
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.
mbirth 2 days ago [-]
I’d love a service like spamcop.net where I could submit my access_log and they lookup the abuse addresses and file abuse reports in my name. Maybe if people’s Internet access gets suspended they’ll think about installing random apps that work as a proxy in the background.
inigyou 2 days ago [-]
abuseipdb.com
Some ISPs ban customers based on a single report there - have fun!
mbirth 2 days ago [-]
Many thanks! I've just submitted the first batch of 3000 (daily limit) IP addresses.
ButlerianJihad 2 days ago [-]
That is a ridiculous way to try and deal with the problem of residential proxies.
You are, in reality, only hurting the actual owners, the subscribers of those ISPs who are behind those addresses. We call that "collateral damage".
If any of those actual residential users try to use a website, their ability to freely access the Internet may be harmed by a bad reputation that they do not deserve. They may be totally unaware and non-consenting to residential proxy use.
You are not, in fact, hurting the residential proxy-ers at all. Not one bit. They will move on to another IP and another compromised LAN, and they will continue to move on and on and on. They will not be harmed or impeded; they will simply keep turning up fresh, new, high-reputation IPv4 and IPv6 sources. This is a sheer numbers game, where the numbers are always in favor of the attackers.
Also if network admins keep blocking/filtering abusive residential proxies, they will balloon their firewall rules and cause actual performance issues at the network level. You will turn into your own DDOS without any actual benefit. You're on the losing side of the numbers game, and in the immortal words of W.O.P.R., "The Only Winning Move Is... Not to Play."
mbirth 2 days ago [-]
Similar to how people running an open SMTP are complicit in promoting spam, I see people running a wild public proxy as complicit in this malicious scraping activity.
And similar to how most people running mail daemons are using blackhole lists nowadays and are keen to not end up on there, maybe ISPs and web hosters can use the AbuseIPDB to sort out their customers.
Just doing nothing doesn't appear to stop the scans hammering my poor Raspberry Pi serving my few Git repositories.
ButlerianJihad 2 days ago [-]
Hey, from the beginning of SMTP, running an open relay was an administrative mistake. The MTA administrators were supposed to know what they were doing, because resources were allocated to them. They had privileges granted for the system and the network. It was right if they were blacklisted for misuse of those resources.
Now in 2026, running a "public proxy" doesn't take an administrator. You don't even need to be aware. Most victims are unknowing victims. They simply subscribe to an ISP and they have their own devices. They are being exploited for that innocence and ignorance. Most victims have no visibility to even detect that they're being used as a proxy. Most victims couldn't stop it, even if they wanted to.
I challenge anyone with a home router to list the processes running on that router, and list all current open connections on that router, and list all open, listening sockets on that router. I bet you can't do it. There are no consumer router OS that lend themselves to being secured, or even diagnosed. Malware can easily be planted on any of them and run, completely invisibly.
A residential proxy server could run on routers, could run on a switch, could run on your "Smart TV" or a smartphone, or a notebook computer. It could be anywhere in any form. Perhaps you consented to it, perhaps you didn't notice.
In no way is this the same as an SMTP open relay situation. If you wanna play "whack-a-mole" with a "blackhole list" you're simply going to overwhelm those lists with false positives and collateral damage. The residential proxies have long since moved on. You won't even find the culprits using those addresses you just blocked. You're just clogging up your own machines. It's a total self-own.
zbentley 2 days ago [-]
> You are, in reality, only hurting the actual owners
How many times do I have to hurt them before they decide to buy a different smart TV?
Seriously, that's like saying "if you try to stop your neighborhood rodent problem by getting citations sent to people with cat food on their porch, you're just hurting the innocent outdoor cat owners". They're participating, whether they know it or not. We can and should PSA and shame and regulate away residential proxies on the supplier side, but we can and should also simultaneously discourage them on the end-user side as well.
inigyou 1 days ago [-]
I'm sorry to be the one to explain to you that the world isn't fair, and you should always do what you can to advance your own interests.
fer 2 days ago [-]
If you operate your site alone in a silo yes, but crowdsourced solutions to ban/rate limit those IPs exist.
thomasjudge 2 days ago [-]
Are there lots of people doing development on mobile devices?
goldenarm 2 days ago [-]
Many are on old laptops, which suffer the same way
louiskottmann 2 days ago [-]
Isn't it perfectly reasonable to require an account for any use, and to ensure that making one has a high level difficulty anubis challenge or delay ?
hei-lima 2 days ago [-]
Great chart! Does anyone know what tool was used to make this?
acedTrex 2 days ago [-]
It feels inevitable that many systems will have to go to a login/trusted ip source type system. Its just not feasible to continue to operate with 99% of your traffic being fake.
GoblinSlayer 1 days ago [-]
They go to cloudflare.
TZubiri 2 days ago [-]
Same problem we've been having for ages.
Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate.
I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't.
There's a tradeoff between privacy and abuse, you want privacy? You get abuse, you want to battle abuse? Gotta sacrifice privacy. Worth noting that unmarked vpn users (residential proxy or residential vpn users) use these proxies for privacy, and therefore give a reasonable alibi to abusers.
inigyou 2 days ago [-]
Cloudflare doesn't block bots.
TZubiri 2 days ago [-]
That's the whole raison d'etre for CloudFlare, it was originally a DDoS protection layer, which, as the article mentions, is the final form of malicious traffic, being distributed and hard to attribute traffic to an identity.
If you know CloudFlare as anything else, it speaks to how successfully it has grown and marketed itself into other areas.
inigyou 2 days ago [-]
That's what their marketing tells you it does - not what it actually does.
bourse_lee 2 days ago [-]
What if Anubis computations were turned into a crypto-miner
throwawayffffas 2 days ago [-]
The whole concept behind Anubis is flawed. It tries to block access by imposing a compute cost to block people that are using incredible amounts of compute just to generate and parse the requests.
The people that Anubis tries to block have tons of compute to spare.
For example at runpod, if you rent a container with just one B300, you get 32 cpu cores and 250 gbs or ram that are essentially just sitting there while the gpu does all the work. If you think you can impose a compute cost on these people without blocking everyone else, well think again.
Or in my case, I am running qwen3.8 at home on a couple of gpus, these are attached on 32 core epyc server with 128gb of ram I am pretty sure i have more compute than the typical dev laptop.
I am of course nice, and don't aggressively scrape peoples services.
fer 2 days ago [-]
Browsers, at least Firefox, blocks crypto-miners. How to tell legitimate from underhanded crypto-mining?
lxgr 2 days ago [-]
> [...] when a source is guaranteed to be LLM-free, like the entire history of kernel commits [...]
Is that really the case? It was my understanding that LLM-based agents were explicitly allowed as long as their users follow certain guidelines [1]?
And more generally: Somehow the theory of "essentially all bot traffic is AI labs crawling the Internet for LLM training data" doesn't make sense to me at all. There are at best dozens of labs capable of running their own crawl at Internet scale, but hundreds of millions of people using LLMs to answer their questions. (If my personal LLM usage is any indication, firing off dozens or hundreds of web fetches to answer a single question is not unusual.)
While I understand that many existing projects have been resourced only for human readers and might as a result be struggling due to this, this characterization sounds a bit dishonest to me.
And unfortunately, for this use case (i.e. ephemeral queries in a context possibly lacking storage or git access), forking the individual repo to answer a handful of string match queries against it might just be more expensive than to run that query against a web search index and then just fetch those results over HTTP.
The solution would accordingly also look very different, as caching at the inference layer is significantly harder than at the training one (where it's most likely already widely done as that seems like a no-brainer).
The use of LLM agents was allowed relatively recently, you still have 20 years of guaranteed no llm commits in the history. As for personal LLMs being the cause of the traffic, I think you are overestimating the number of users of LLMs generally, and even then most of those users are not actually developers. And there are even fewer cases still where the llm being used actually needed something from the linux kernel, personally after quite heavy use of LLMs on linux, I have never seen it try to fetch the linux kernel directly once.
innocent_name 2 days ago [-]
Why can't they just ask Linux Foundation for 96, or even 1696 cores?
If you're reading this - go ahead and see HOW Linux Foundation spends their money.
alkonaut 2 days ago [-]
Proof-of-humanity can’t come soon enough.
We’re talking about privacy-preserving proof of age, but as we see here the real utility of such a system will be proof of humanity.
Arubis 2 days ago [-]
How do you define humanity?
How do you ensure it includes every human?
How do you ensure it doesn’t include every non-human?
I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.
alkonaut 2 days ago [-]
Every place on earth has some legal definition of who is human. The system I’m thinking of isn’t a technical/captcha one, it’s a human curated list of humans. Just an electronic ID. Those already exist but the challenge is making them (acceptably) privacy-preserving.
I want to take my existing national digital ID and use it online basically. BUT I don’t want the websites to know it’s me. Just that I’m human (or perhaps over a certain age). And I don’t want the ID issuer to know what site/service asked whether I’m a human or I’m 18 etc.
Arubis 1 days ago [-]
> Every place in earth has a legal definition of who is human.
I actually doubt that! And in places that do I doubt it’s perfection. Citizenship is mostly defined legally. Residency has a legal definition that may or may not track with reality. Most legal regimes I’m aware of (IANAL) define personhood; whether humanness is defined I’m less sure of. Things that are more nebulous and subjective…sometimes they’re defined, but necessarily subjectively so.
And that gets you into dangerous territory if you’re at the margins.
A fun example: how do you define “alive”? Does the person have to have been born? Are there conditions on that? Would they exclude some people we think of as alive? Does the person’s body need to be self-sufficient or can it rely on mechanical or other intervention? Is there a line? Is the person alive if their body is functioning but their brain is not? What about vice-versa? And even if the answers to all these questions are currently perfect, what happens if and when the definition shifts?
Being alive and human is messy business. And this is setting aside the discussion of making government issued records safely usable without leaking, which they already aren’t.
alkonaut 1 days ago [-]
Issuing this digital ID for all the _other things_ I do with it already has to deal with all this!
Of course the debate would revolve around ”is it fine to have a national government issued ID be the ultimate proof of your identity/humanity?”
It’s problematic - yet I think it’s better than the alternative.
Kamq 2 days ago [-]
You're not wrong, but something can work well enough to still be useful despite falling short of the idea of a proof or any formal definition.
Let's say that 95% of individual humans can pass it and only 2% of bots. For someone maintaining a website, who has to decide between using this system and shutting down their site because of the increased costs, that may very well be good enough
Arubis 2 days ago [-]
That’s a pragmatic and understandable argument. And for an individual hobbyist site owner, that’s fine. Are we okay with excluding 1 person in 20 from the services of a midsized organization? What if they’re integral to the workplace? Or a major transport provider without differentiated competitors? What if the organization is a state government?
inigyou 2 days ago [-]
I'm one of the 5% apparently, cloudflare thinks I'm a bot. I suppose you'll exempt me in exchange for all my ID documents and bank statements?
gib444 2 days ago [-]
Running Firefox with Temporary Containers Plus makes challenges 10x more annoying :D
(Each new tab is isolated, unless opening a link in a new tab. Same as Safari in private mode)
kgeist 2 days ago [-]
How about: "Type the seahorse emoji to solve the CAPTCHA" :) Something that triggers infinite loops in LLMs or trips the guardrails.
I wonder if we're back to peer to peer networks with proof of useful work (e.g. serving read requests) vs proof of wasted work.
6d6b73 2 days ago [-]
Add a lot of random text to the html pages, preferably hidden to regular users, have the bots use lots of tokens to process it all.
gruntled-worker 2 days ago [-]
There's something missing from the picture. The bots are:
- Using a terribly inefficient way to redownload the same commits as e.g. HTML diffs, possibly the most inefficient.
- Putting in tons of CPU cycles to surpass the Anubis PoC.
- Putting in other kinds of active effort like reworking access methods and buying "residential proxies" that are probably illegal in most jurisdictions.
This sounds more like escalating DDoS than AI scraping.
zbentley 2 days ago [-]
Possible, but I doubt it. The sheer number of other free-to-read content sites dealing with the exact same problem the last few years (many of whom there's no plausible reason to DDoS) tells us that this is content harvesting, not an attack.
gruntled-worker 2 days ago [-]
I'm curious and would like to see those reports. There's AI scraping for sure, but intentionally resource-consuming, increasingly-insidious AI scraping I've never actually read about.
AI scraping might be bad, but if a particular case that's actually a DDoS becomes the cause celebre against AI scraping, it will weaken the argument, not strengthen it.
talkingtab 2 days ago [-]
Time to F*$k the internet. The whole concept of anonymous IP addresses was broken but worked for a long time. Now it is just stupid. Just like domain names. (Are more names used by squatters than real?). And email as identity? Time to engineer solutions and create a new protocol layer.
This is not a hard problem. It just requires that someone build a certification wall. The IETF should have done this long ago, right?
desterothx 1 days ago [-]
It's not a hard problem, the hard part is doing it while keeping a similar level of privacy/anonimity
boredatoms 2 days ago [-]
Can I suggest putting some text in the page that tells the bot what the more efficient download method is?
zbentley 2 days ago [-]
What makes you think that would accomplish anything?
These scrapers aren't LLM agents. They're distributed classical programs that harvest data which is later used to train an LLM. The LLM doesn't write the scraper or respond to individual scrape events. The entity training the LLM contracts someone, who contracts someone, who contracts someone to run a web scraper and send them the data.
singpolyma3 2 days ago [-]
Why is no one filing lawsuits over this yet?
mmooss 2 days ago [-]
Indeed. The solution isn't technical but legal. It's clearly abusive of - really stealing - other people's resources; there's no question about it.
For some reason, like with fraud via email, text, and phone, we don't do anything about it. All this brazen crime and government does nothing; we don't even imagine government doing anything.
akersten 2 days ago [-]
The brazen crime of asking for a resource and receiving it.
You have the power, and should exercise it, to rate limit bad actors
mmooss 2 days ago [-]
> You have the power, and should exercise it, to rate limit bad actors
It's well known, and it was the point of the OP, that we don't have that power. If you know a solution, please share it - the world would love to hear it.
inigyou 2 days ago [-]
How do you know who the bad actors are, if you don't use the legal process to unmask them?
Symbiote 2 days ago [-]
Against what person or entity?
inigyou 2 days ago [-]
Sometimes the first step of a lawsuit is discovering who you're suing. It's not unusual and there are processes for it. You could bring something like an access log to a court and receive an order for all ISPs involved to unmask the corresponding users.
singpolyma3 2 days ago [-]
Whoever is doing the abuse. If we don't know who that is we should find out
johneth 2 days ago [-]
Bright Data et al.
inigyou 2 days ago [-]
Bright Data's business is legal, but they could be subpoenaed to find out which customer is making these requests, but first you would have to prove they were actually involved, because there are many residential proxy providers.
dunlin 2 days ago [-]
Reminds me of debugging production issues at 3 AM. Both can make you jump out of your skin.
vist_orn 2 days ago [-]
Real creepy crawlies in the server rack are always a bigger surprise than any code bug.
Velocifyer 2 days ago [-]
But why don't they just git clone?
rcxdude 2 days ago [-]
These crawlers (in contrast to e.g. googlebot and similar better behaved crawlers) are not very smart: they seem to make very little effort to avoiding crawling useless deep trees of generated pages. About the only thing they seem to put a lot of effort into is avoiding blocking.
(I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually obvious who's running these bots. The big players all have crawlers that identify themselves and are reasonably well behaved, but I don't know if anyone has managed to positively attribute these other ones to any particular group)
ipdashc 2 days ago [-]
> it's not actually obvious who's running these bots
This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!
Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option
tmp10423288442 2 days ago [-]
ChatGPT uses a distinct bot type for ChatGPT-initiated web search versus model training:
GPTBot → model improvement / training crawl
OAI-SearchBot → search index / ChatGPT Search
ChatGPT-User → fetch initiated by a user's ChatGPT interaction
OAI-AdsBot → advertising landing-page crawl
inigyou 2 days ago [-]
Especially since if any crawler IP is in your country, you can get a subpoena to find out who it is.
Symbiote 2 days ago [-]
Meta is an exception. They intentionally ignore robots.txt rules.
(No second chances, I blocked their whole IP range.)
inigyou 2 days ago [-]
Everyone ignores robots.txt because every robots.txt blocks everyone.
nubinetwork 2 days ago [-]
I allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.
inigyou 2 days ago [-]
Think about incentives, what benefit does Anthropic obtain by going to hell?
voakbasda 2 days ago [-]
Why do we think that only “good guys” are training LLMs? I imagine organized crime is getting in on the game too.
lkbm 2 days ago [-]
Because they're crawling a billion webpages, only a tiny fraction of which can be git cloned, and configuring a special case just for that tiny fraction isn't worth the effort (of the crawlers).
acedTrex 2 days ago [-]
Because the crawlers dont care, they are the internets parasites. Their creators care nothing for people or systems downstream of their greed.
DarmokTanagra 2 days ago [-]
vibe coded crawlers run by morally bankrupt trend chasers aren't going to be the most well engineered systems you come across.
iLoveOncall 2 days ago [-]
> Where does that leave us?
Honestly, the answer is simple: sue.
It'd be hard to argue that it's not a DDOS.
throwawayffffas 2 days ago [-]
On shallow clones, can't you disallow shallow clones?
dTP90pN 2 days ago [-]
Does cgit not support a .patch or .diff URL suffix? How hard could it be for AI companies to add a preference for such URLs into their models or system prompts? Wouldn't that be obvious improvement for any model?
Velocifyer 2 days ago [-]
I would add cloudflare, but set it to cache only mode *without* the bot blocking features.
inigyou 2 days ago [-]
No point, they are all unique requests.
wingworks 2 days ago [-]
I think he means, get cloudflare to cache your content, so the traffic never reaches your servers to begin with. Assuming your sites content is cacheable by cloudflare. I agree it's a sad state of affairs if you have to rely on a 3rd party..
Maybe I'm not understanding how many requests at a time bots are sending to kernel.org (or how larger kernel is), but couldn't they have a local cache system too, where all it has to do it serve up dumb .html pages, needing next to no compute cycles.
tliltocatl 2 days ago [-]
It is server-rendered cgit pages, there are potentially quadrillions of unique. They are not cacheable.
inigyou 2 days ago [-]
Maybe you're a little hard of hearing. NO POINT CACHING, THEY ARE ALL UNIQUE REQUESTS.
danieltanfh95 2 days ago [-]
> the bots started solving difficulty 5
turning it into a problem of ROI is one of the stupidest things you can do to prevent bots.
But when they do… boy do I have a trap waiting for them.
My wife calls me The Genius. I’m the guy she calls when her battery dies or when her instagram breaks like when it shows that random guy in her DMs, stupid bugs LOL
I digress. Alas, when a user lands on my page. My page wants to know exactly 2 things:
1. Why are you here and who are you
And 2. Can you produce a working solution to Pharoah’s Fortune
…those of you aren’t familiar Pharoah’s Fortune is an old chestnut little poem, a riddle if you will I like to ask candidates and so far nobody’s solved it
And the reason nobody has solved it is Pharoah’s Fortune is a very tricky problem. It’s not something you can “solve” per se it’s more like you arrive there.
So far no one has solved it. They all fall for the same trick! It is of course what separates those who write elegant C versus those write poor quality JavaScript.
So I always say to my students to keep an open mind because you never know who - or should I say where you’re talking to.
I’m bookish.
phyzome 2 days ago [-]
I don't see any relevant reference online to "Pharoah’s Fortune".
Wowfunhappy 2 days ago [-]
Who exactly is running all these scrapers? There are, what, maybe 15 major AI labs, if that?
And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
micah_chatt 2 days ago [-]
If you do the math (also a common system design interview question for an AI lab), its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes. At that price point, its actually very affordable to many thousands of organizations to get their own copy. I would expect the major labs to special case kernel.org similarly to other sites like Wikipedia, but not the majority of scrapers
Wowfunhappy 2 days ago [-]
> its actually only ~3PB (compress to 1PB, ~$22,000/mo in S3) and a few thousand/mo in compute over less than 4 months to index the entire internet for pretraining purposes.
That is super interesting, thank you!
> At that price point, its actually very affordable to many thousands of organizations to get their own copy.
I'm still confused as to who is actually doing it though! Maybe it's affordable to scrape and store, but training a competitive AI model is going to cost much more, right?
GoblinSlayer 1 days ago [-]
I suppose proxies provide only http request interface.
strix_varius 2 days ago [-]
There are many more labs than that, and humans aren't designing unique scraping processes per domain.
DarmokTanagra 1 days ago [-]
three guesses what geographic regions they are based in.
ChocolateGod 2 days ago [-]
> phone gets uncomfortably warm as it's doing the number crunching
IMHO if I visit your website and it intentionally starts wasting my electricity for no other reason than to cost me money, with no opt in, it's hostile and malicious.
inigyou 2 days ago [-]
You try hosting gitea in 2026. The only other option is taking the site offline.
asah 2 days ago [-]
Just slow unauthenticated traffic to non-essential stuff...
lmz 2 days ago [-]
It's funny how some people say "AI bad, datacenters waste energy" then other people say "AI bad, going to make humans and their phones waste energy".
phyzome 2 days ago [-]
This is a silly take. AI is why they're having to waste energy.
lmz 2 days ago [-]
Sure, and the humans and their phones are fair collateral damage in that worldview.
arlattimore 2 days ago [-]
In the case of kernel.org, why not make the unauthenticated version return only the latest kernel repo with no history (tiny number of URLs relatively speaking). If you want full kernel.org features, login.
kuschkufan 2 days ago [-]
because they do not want to be twitter, reddit, facebook, ...
arlattimore 2 days ago [-]
I'm not suggesting 'login' because facebook/twitter/etc, just as a mechanism to make the bot problem go away. They clearly want it to stop, they tried obvious methods but the AI platforms are circumventing it (deliberately) which is poor form.
kuschkufan 2 days ago [-]
login or you will not get the full service is exactly what facebook and co are doing and what your suggested would amount to. the kernel.org stated quite clearly they want to remain a public service. which i applaud.
bjourne 2 days ago [-]
> They still do that — welcome to the wonderful world of “proxy SDK monetization.” It's big business, and your TV is probably doing it.
I must be missing something. How can using peoples' TVs as bot farms be even remotely legal? Especially when the purpose is to avoid IP blocks?
sgsjchs 2 days ago [-]
The people "consented" to this when they clicked OK on the user agreement.
Kuinox 2 days ago [-]
1.4 billions requests, 258 160 cpu hours.
That's 1.5 requests per second ?
I'm starting to believe, the issue is more that their software is not well optimized.
Skunkleton 2 days ago [-]
You are forgetting that there are multiple multi-core servers.
Kuinox 2 days ago [-]
It's incredibly slow for a single core, that's my reference point.
2 days ago [-]
akoboldfrying 2 days ago [-]
Are you prepared to randomly serve data that is incorrect -- but in such a way that real people can easily detect it -- some very small fraction of the time?
If so, you could serve iocaine-style bogus pages 1% (say) of the time that:
1. "Look like" real pages to an LLM-less computer (if you get to the point where you have pushed crawlers to use LLMs to detect nonsense, that already increases the cost a lot)
2. Look "obviously wrong" to a human (E.g., you could take some regular text and swap the order of each adjacent pair of words)
3. Are cheap to generate
4. Important: Contain more links than regular pages, on average, and each to an always-bogus page
The idea is that, due to the large number of pages fetched by crawlers, even with a very low "random bogus page rate", like 1%, they will soon unwittingly hit a bogus page, from which point the fraction of their time spent accessing expensive genuine pages will fall exponentially due to the compounding effect of the higher outbound link count on bogus pages. Humans seeing a bogus page will be confused and annoyed, but simply refreshing the page in the browser will solve the problem 99% of the time (and of course the possibility of this happening can be documented, even on the page itself).
The main advantage is that this does not require any IP-based tracking. You could of course decide to apply this only to pages that are already slightly suspicious (e.g., very old commits).
monegator 2 days ago [-]
The thing that bothers me is why the fuck are they still scraping git.kernel.org or any other site that has already been scraped a million times before. Who would pay for that data?
Then again there is the conspiracy theory about cloudflare sponsoring the scrapers
hubraumhugo 2 days ago [-]
There is a HN article on abusive AI crawlers on the front page almost every week, but we rarely talk about the path forward. Web scraping has been around for as long as the internet, and it was fine because we had established best practices (rate limiting, self-identification, robots.txt, etc.) that the industry agreed upon. Now we have AI labs and their crawlers that don't care about any of this gentlemen's agreement.
So how do we go from here? Is adding more difficult Anubis and Cloudflare bot protection really the solution? How many millions of human hours and billions in infra costs are we willing to spend on this arms race?
Some approaches that I think are promising:
- A robots.txt V2[0] as a standard way for website owners to state how bots and AI crawlers can use their online content and where to go (e.g. distinguish search from AI training use cases, point to a downloadable file instead of crawling everything, etc.).
- Something like Web Bot Auth[1] as a non-centralized standard for self-identifying bots and agents cryptographically. This would allow websites to allow or deny bots very precisely.
Instead of trying to block why not monetize? So the proof of work can be directed at something you can be paid for (bitcoin mining)?
desterothx 1 days ago [-]
If everyone had devices that could do some compute worth paying for, people would be doing the work all of the time. The problem is actually useful POW is a lot more expensive compute-wise, making it non viable for normal users
inigyou 2 days ago [-]
Monero would work better.
JakeSc 2 days ago [-]
> after all, it's easy to figure out that an IP that is trying to grab every possible commit in a 8-year-old abandoned fork of linux is not really some lone Chrome on Windows user who is just furiously clicking every link that comes across their screen.
Ah yes, the stereotypical Linux kernel developer.
greatgib 2 days ago [-]
I have the feeling that the hate might be misplaced.
For a shopping website or user generated content website, I might understand the terrible load of crawlers that are trying to "steal" the data.
But for the kernel, what's the purpose? Are you that "no human" are seeing your page or its content? Maybe we should investigate more the usage being this "bots".
I don't buy the explanation that there are millions LLM that are constantly trained on redownloaded data from kernel.org.
What would be my better guess is that it is not training, but users are actually accessing this content through chatbot and co.
Like when you ask why your sound is suddenly not working anymore after an update or why your wifi driver is constantly disconnected after leaving sleep, it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them.
Is it a bad thing if it helps users?
But actually, regarding this article, I'm quite amazed that with all the advances of the linux kernel, and server softwares, and that the C10k challenge is solved since a long time, still such a basic traffic is such an issue.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
14 cpu looks nothing to me. It's like you have 1 iphone and 1 raspberry pi active in a corner of a room.
Counting in "seconds" of activities, easily shows meaningless huge numbers. Do you want to know how many breaths I take per year? 8 to 9 millions!
Most certainly, the usage of this shitty Anubis has ruined the climate million times more only with the wasted cpu resources of legit users...
But moreover, by definition the git commits are not supposed to change, ever, so can someone explain to me why the fuck do kernel.org "re-render" the commit to html each time someone is accessing it instead of using a cache or a static version of the html of this commit?
> oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits
Again, reading that, my immediate thinking is that it is a shame that such talented people would not be able to have a proper optimization, so that getting the 922 duplicates are just costing a fraction millisecond more after the first person retrieve the first page.
zbentley 2 days ago [-]
> the C10k challenge is solved since a long time
This has nothing to do with that.
Any Node.JS application will happily accept 100K connections. They'll all wait for the under-resourced database behind it. That application "solved" the C10K challenge, but it's still overwhelmed.
> it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
The article describes random algorithmically-generated traffic arriving in batched waves from laundered residential proxy IP addresses, a few unrelated hits in a group then gone. That's not the pattern you'd see if end users were asking their agents for help.
> it is a shame that such talented people would not be able to have a proper optimization
It's mostly not static content in the sense that you're implying.
Routes that access a single commit can be cached. But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
I'm sure they're already caching their useful-to-real-humans data. As the article said, the vast majority of their traffic is bots hitting those arbitrary, permuted URLs. So whatever cache they're using is probably a) missed almost every time, and b) constantly getting evicted to make room for data served to bots (unless they eschew caching to avoid this--fair--and are thus back to the original issue regardless).
There is no "proper optimization" here. It's not slow to go compute the diff between a random pair of refs, render that into pretty HTML, and serve it. But it costs something more than a cache hit, and doing that dozens-to-hundreds of times a second constantly consumes resources.
greatgib 2 days ago [-]
You raise a lot of wrong points to defend the article author:
> But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
Looks like to be wrong based on the article:
Today, git.kernel.org receives about 6M daily requests demanding to see random commits.
Also, in "how bad is it", there is a bar graph of the tasks requested. And only diff can't be cached, and it is clearly not the most important one.
> batched waves from laundered residentail proxy IP addresses
Again, I don't see anywhere anything stating that there was "grouped" requests that are directly related, except in something not related to AI:
The thing that usually takes us down are not scraper bots, but poorly designed CI systems that try to do something stupid like shallow-clone stable.git from 20 different nodes, all at the same time. (Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.)
Just "waves" of requests coming from the same "residential proxy IP". But that makes sense because they might come from a same service without being a single "source" trying to scrap for training.
Let's say that I'm chatgpt, or Claude, and I have a lot of user related requests to do things based on the commits there, each one will be a different task, but they will all go through the same pipe, that would be a proxy with the residential IP if the provider notice major websites like this one blocking requests for bots.
hnisjafx40 2 days ago [-]
Learned this the expensive way
bluedino 2 days ago [-]
> But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.
I feel like I'm at work.
We had some web crawler using Selenium to make queries and scrape the data instead of just downloading the whole file.
Every day it seems like we have some people that know just enough to be dangerous creating things like that. And then of course it's our fault that things are slow, or we won't give them infinite system resources, etc
emsign 2 days ago [-]
What a waste of energy LLM training is. Meanwhile Himalayan mountains are crashing down. I love this world. It's so idiotic.
sunaookami 2 days ago [-]
Have the exact same problem with my MediaWiki and Gitea instances. Doing a managed challenge through Cloudflare (sigh) on "expensive URLs" helped and minimizes the impact on real users. These are also all over a million of residential IPs, very annoying.
oowa 2 days ago [-]
have a hackathon to solve for this. OP says it's not a problem for him right now but if we extrapolate what he's talking about it's definitely a problem aaaaaaand It's totally solvable, Even with all of the crazy combinations he's talking about it's still solvable. And it's already been solved using patterns we see in streaming services. This is completely hackathonable. but why do we even need to bother with this? The slurpers are the cause of this, and they can cause this problem because of Murphy's Law. well you can only account for Murphy's Law with good architecture or something like that or whatever. Ha ha hackathon.
brador 2 days ago [-]
Send an invoice or your tears are worthless.
0xdeadbeefbabe 2 days ago [-]
> permanently tying up a chunk of capacity spent on producing output that is only useful for a single purpose — feeding a learning model.
The horror.
UltraSane 2 days ago [-]
$1 dollar a year subscriptions would help.
DarmokTanagra 1 days ago [-]
you do realize crypto has faced this problem, presented the exact same solution, and utterly failed right?
UltraSane 20 hours ago [-]
requiring payment requires a payment method and burning those is much more painful for whoever is running a scraper.
nicman23 2 days ago [-]
couldn't you have anubis on a dynamic difficulty? ie if a ip requests more than 1k pages per day +1 the difficulty ?
dunder_cat 2 days ago [-]
Yes, but the article (not to call you out - I just think it's a very important point!) points out that this type of throttling would not be effective:
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back.
Without something like cookies (which are almost certainly tossed after the IP is rotated) or some other persistent identifier, you are stuck have to apply mitigations that scale with the load you're encountering, which means longer challenges for everyone or degraded functionality, like removing some of the fancier cgit features.
marginalia_nu 2 days ago [-]
I've had a fair bit of success with increasing the bot mitigation based on a global rate limit. During periods of high request rates, I throw progressively more hurdles at the bots, and during periods of low request rates I disable them all.
nicman23 2 days ago [-]
yeah i my head i thought they meant 4-5 _K_ requests
PinkaDunka 2 days ago [-]
Maybe anubis difficulty should depend on the age of commit. This year - 4, everything older 8
nicman23 2 days ago [-]
or on cache hit /miss
2 days ago [-]
adverbly 2 days ago [-]
Is it really stupid if it means more data centers need to be built and it keeps the AI bubble going and GDP number go up?
stcg 2 days ago [-]
> proxy SDK monetization
Wait what? I never heard of that. I call that a botnet
GoblinSlayer 1 days ago [-]
It's called capitalism.
aimen2 2 days ago [-]
Now hosting will cost more for everyone...
dzhiurgis 2 days ago [-]
These are most likely not training scrapers, but people looking for concrete pieces of information (i.e. commit, comment, etc).
dzhiurgis 1 days ago [-]
Clarification - these are crawlers that llm’s use when you ask them to check this site for foo.
stefantalpalaru 2 days ago [-]
[dead]
mleonhard 1 days ago [-]
[dead]
runtime_lens 2 days ago [-]
[dead]
oowa 2 days ago [-]
TLDR basically old tech is not optimized for scrapers / slurpers / etc. to the point it would take 42^n to solve all possible combinations. Why? Murphy's law. Solution for OP is to ignore for now. Otherwise Use or invent something else. Easy enough. other notes... Anubis and other gatekeepers dont work perfectly, but ok for now.
charcircuit 2 days ago [-]
How about making cgit more efficient at serving these pages. There's no excuse for burning a ton of CPU power on purely static pages when you have generous resources available to you.
userbinator 2 days ago [-]
So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right?
I'm strongly convinced that these aren't "AI crawlers"; they're just plain DDoSes done by those who have interests in turning the Internet into a dystopian walled garden with "security", and now they have a convenient scapegoat to blame. Don't you find it too coincidental with the rise in identity/age verification and other attempts at silencing free speech on the Internet?
It's widely known there are questions that LLMs can't solve, and once in a while an obvious example appears, so a simple CAPTCHA-like challenge with an HTML-only form would be the logical "defense". Instead there's a huge interest in pushing JS-required proof-of-work (as others have pointed out, these attackers have far more compute than the average user) and remote attestation (there are already providers with huge farms of mobile devices that can defeat this easily).
Things just don't add up.
desterothx 1 days ago [-]
The questions the LLMs can't solve puts you in a cat and mouse game. They sure can solve most of those with the correct tooling, and i would argue its easier to create this tooling than it is to keep innovating with new questions LLMs cannot answer
leoqa 2 days ago [-]
It seems clear to me we are moving towards a world where you will have to perform device attestation to access the internet. The spam/abuse is too great and accelerating.
okanat 2 days ago [-]
What prevents people from obtaining or buying such devices and automating them? Using TVs as proxies is just one example of that. People will be willing to give their ID cards away too, if you pay them or beat them enough.
inigyou 2 days ago [-]
In fact they already do this. Buying 100 android phones and chargers is cheaper than reverse engineering whatever you're trying to automate - or was, before AI.
leoqa 2 days ago [-]
.. and we ban those device ids and move on. Your capital is lost.
inigyou 2 days ago [-]
Evidence shows otherwise. There are people making lots of money from these device farms. Right now. This isn't hypothetical.
leoqa 7 hours ago [-]
I literally work on this. The problem is that enforcing device attestation impacts DAU so platforms don’t want to do it but it’s effective. Once the internet is unusable, device attestation will be required to access services. Ad driven businesses will start to see demand from ad buyers for non-bot traffic etc.
adangert 2 days ago [-]
Curious, if serving bots (and traffic) is the main concern here, why is a distributed git solution like radicle not considered?
Training on large quantities of LLM-generated synthetic data is an important part of training LLMs.
Lerc 2 days ago [-]
As the article states, this phenomenon may be documented, but there is no consensus that it describes any practical reality.
The predicted consequences have now had time to manifest, and have not done so. This makes the claim either false or overstated. Perhaps there will be issues in the future, but to date there have been many claims that AI development will stall (for a variety of reasons). If they were the critical weaknesses they have been portrayed as, models would not have advanced to the level they are today.
If you have a hypothesis, make a clear prediction based upon it. If you start pushing the date forward after each failed prediction, you end up looking like a hapless doomsday cult. If your hypothesis is correct however, your prediction should actually happen. Then provided you have not made so many predictions to get one right by chance, people will take what you have to say seriously.
desterothx 1 days ago [-]
You state it as if its only the quality of the hypothesis that matters, but you are ignoring an important part of it, timing. During the 08 financial crisis Burry had a hypothesis that was correct, however he almost went bankrupt still because he thought it would happen earlier than it did because of the government bailouts. He was pushing the day forward, and was looking like a "hapless doomsday cult". His hypothesis still turned out correct
Lerc 15 hours ago [-]
If you can't say when something will happen you are just playing with statistics on another domain.
Seems like this chaps problem was that the actions were based upon a different hypothesis than the one he stated. X will cause Y by the end of the year is considerably different to X will eventually cause Y.
andruby 2 days ago [-]
A creepy crawly is a South African invention to clean your swimming pool. The company that introduced them in the 70ies is called Kreepy Krauly. Also popular in Australia.
I love to see the 'leet kernel hackers and maintainers' struggling with basic volume. Each page load should cost you near nil. Us lowly PHP developers have been caching shit for close to twenty years.
Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.
riknos314 2 days ago [-]
The amount of memory required to cache all possible diffs (defined as an ordered pair of commits) would likely be in the exobytes. At current ram prices that's easily a trillion dollars of ram to run that cache lol. Git focuses on making diff calculations efficient largely because the space of possible diffs is very expensive to enumerate.
The following from Claude:
"""
A diff between two randomly chosen commits usually spans years of history, so it's not a few KB — the tree itself is ~1.5 GB of text, and a multi-year span rewrites a large slice of it. Call it 100–200 MB per pair on average:
8.5×10¹¹ pairs × ~2×10⁸ bytes ≈ 10²⁰ bytes, or ~150 exabytes
"""
riknos314 2 days ago [-]
I'm not sure if the website allows for diffs against arbitrary tree states, but if it does than the diff space is completely unbounded, and the ram demand is theoretically infinite.
calvinmorrison 2 days ago [-]
cache all of them? no... but you could cache some of them, likely the newer are accessed more frequently.
zbentley 2 days ago [-]
Per the article, the vast majority of traffic is scrapers that are generating effectively random, permuted URLs. Even if they're already caching their popular-with-real-humans most recent data, that efficiency gain is a tiny drop in a very large bucket.
desterothx 1 days ago [-]
Yes, and it still wouldnt help you since they are trying to fetch everything for each of the 921 forks, as stated in the article (same commits need to be cached again on a forked branch)
inigyou 2 days ago [-]
It's running a diff between two arbitrary blobs of text. Do you actually have a solution or are you just saying to remove the feature from the site entirely?
> Us PHP developers
I can tell.
calvinmorrison 2 days ago [-]
it's not an insult. yes. cache heavily. shitty php stacks serving trillions of dollars of ecommerce sales have managed to do it for a long time.
zbentley 2 days ago [-]
Those ecommerce stacks serve a large fixed number of pages. cgit does not.
Imagine if WooCommerce had a route "/product/<sku1>/compare/<sku2>" which displayed an auto-generated comparison between any two product pages.
Now imagine running a million-SKU WooCommerce site, where each product page was 100kb of text.
Now imagine scrapers are permuting those URLs.
How would you cache that?
That's what cgit/kernel.org and many other Git forges are dealing with. These aren't static websites, even if the underlying Git repo is largely static; they're rendering arbitrary diffs and other generated-on-the-fly views into Git history. The ability to do that is a large part of the value of a Git UI.
inigyou 2 days ago [-]
Can you explain how a cache lets you avoid serving a request for the first time?
znnajdla 2 days ago [-]
Put a CDN in front and let them absorb the load? Seriously, this is static content, which is so cheap to serve it should be free.
ninglor 2 days ago [-]
This is not meaningfully static content. Look at the charts in TFA. There is a combinatorial explosion of distinct URLs which the crawlers can and do request.
jopsen 2 days ago [-]
They allow you to diff commits, which is an awesome feature.
But if bots a crawling diffs between all possible commits it's crazy.
CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.
inigyou 2 days ago [-]
No CDN blocks bots. Some claim to. None do.
grep_it 2 days ago [-]
Did you read the article?
“[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
inigyou 2 days ago [-]
What if it's really obvious they didn't read the article?
znnajdla 2 days ago [-]
I did read the article. It just didn't occur to me that their combinatorial explosion of diffs was scrape-able. To be honest that sounds like an scrapers tarpit / honeypot now, because there is no value in scraping trillions of diffs. Sounds like the issue could be fixed by putting the diffs in a frontend app, not scrapable by URL, only by clicking around the app.
Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.
Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.
You can't tease us like that without showing screenshots.
https://ibb.co/mCgXPZPg https://ibb.co/qFxVRQGW https://ibb.co/9m6xV1xg
I only just added the token wasting measure so I'm really curious to see what happens if another bot loads over 400 pages.
No hot toaster pics though :( on second thought perhaps better in our imaginations
Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down.
We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got complaints on Reddit from day one, pretty much. At the same time, it got a bit expensive having a WAF.
In the end the only solution that worked was blocking every single Chinese IP.
Traffic instantly got down to 10x less, same as it was before. Same conversion percentage as before.
Meta bots are also problematic, as they seem to be quite buggy. But the volume was nowhere near the hundreds of bots from China :/
https://firesphere.dev/articles/iocaine-the-deadliest-poison...
---
Gymnastics trophy two years late, not two years wrote down on the other side of the two thousand millions. Stop all the time was now a.
Anything of this real London, these actual civilized men and women; in uniform batches. The whole incident could not control the beating of a telescreen. Folly, folly, folly! He.
---
The only punishment would be unique text that trains their models to be degenerate. And even then you'd probably have to serve across many domains.
The altered versions would be the page or paragraphs of it, with sentences and words swapped arbitrarily so that training on the data adds as many iffy token sequences as correct ones.
It'll just punish the bad actors running the scrapers. As the original poster mentions that they are using TVs as proxies to get residential IPs, one really can't think of these bots as criminal enterprises.
Sadly, if the bad actors has two cents for brain, they'll limit how much importance each domain name can have on training data. To mitigate impact of bad data like this.
(Note I'd suggest only linking to them from robot.txt as pages to not be indexed, that way no human or well behaved not ever will see them, which is kind of the point).
Exactly. I can't hope to keep them all at bay, but I can at least have the petty little victory of making their visit less convenient than it might otherwise be.
> if the bad actors has two cents for brain
I suspect that a majority of them are little better than the script kiddies of yore, running tools with minimal understanding of what is actually going on.
> I'd suggest only linking to them from robot.txt as pages to not be indexed
Agreed. Blocking all bots from all pages, well those that bother to listen to robots.txt. All bots because pretty much all of them are scraping for AI and similar these days, even googlebot. If I want people to see my stuff they'll get a link, and maybe they'll pass it on further, but all indexers/trainers can get stuffed. I'll likely make an exception for archive.org and similar.
I’ve also toyed with this approach and have a similar labyrinth on my website, but it just serves back huge amounts of meaningless text one word at a time.
I do want to add some more gibberish generation to mine. Once a visitor hits the honeypot route it does replace the rest of the site content with nonsense but I think it'd be fun to have it spewing really daft, really long LLM-generated essays on toasters.
However the article mentions that now they are being scraped 4-5 requests at a time from random IPs that are then never reused. It still probably possible to detect scrapers if they start by requesting obscure pages, but in general it's a much more difficult case.
Elixir's really fun to do this in because the BEAM will let you have hundreds of thousands of processes sat around doing nothing quite happily, so slow IO and such is something it's weirdly well-suited to.
This is on my photography site so I'm less fussed about them harvesting my writing, and more about causing problems for image harvesters. I'm sure they'll get some stuff anyway but at least one bot got stuck in a 400 page hellhole earlier today so I'm overall very pleased :D
I also explicitly tell all crawlers not to index the images on my site, actually, so I feel even better about serving the ones that do complete junk.
Not sure why I get more crawler traffic than I do legitimate traffic tbh - it's a relatively new website that I've been trying to sort out the SEO for as I'm trying to get photography work, so it's probably started cropping up on the radar of scrapers looking for training data.
I think the post underestimates just how little thought and effort is put into these bots. I also run a cgit instance with far less interesting projects, and am not spared from the deluge of HTTP requests.
The explanation I could come up with is that they try to crawl all links regardless of how much sense it makes or how much load it causes. cgit being cgit, this means billions of links for all combinations of parameters and hashes. That, or it's a deliberate DDoS attack.
- Respects robots.txt
- Detects if something is likely a git forge and as a result
- Clone the git repository instead of trying to crawl the forge
None of this needs to be happening, and yet it does at an unprecedented scale.
The only thing that ended up helping was disabling access for anonymous users.
I noticed the other day that lists.ffmpeg.org had moved to Anubis difficulty level 6, which takes ~180sec for my iPhone 17 to solve at ~100KH/s, making the site unusable. So I spent ~10 minutes vibe coding a safari extension with a native bridge to an optimized C kernel using ARM SHA256H* instructions that can do 200+ MH/s on the same device. This solves Anubis difficulty level 6 in a handful of milliseconds.
Given the numbers and capabilities involved (a single $5K ASIC miner yields 200TH/s, a million times more hash rate than my optimized kernel running on an iPhone), I don’t see how proof of work could possibly be a sustainable strategy to keep bots out without ruining human user experience. It’s an arms race that can’t be won.
Edit: I encourage you to try this yourself. Here's a sample prompt that ought to one-shot the task:
For reference, the challenge on lists.ffmpeg.org takes 8 seconds on Firefox on my three year old laptop CPU that has worse benchmark scores than the iPhone 17 (tbf, the laptop also cost less than an iPhone 17). 8 seconds doesn't run against thermal limitations, so I really don't see why Safari on a modern iPhone should be so slow at this
They're not trying to engage in an arms race, they're trying to channel a racing river into its natural course.
I think the path forward is that websites offer one path for humans, and another for scrapers. But the huge catch is the path for scrapers must be _genuinely_ and _reliably_ the more economical and scalable path (either through something like PoW arms races, or through fear of litigation). Otherwise they will continue to ignore instructions and intrude on the human path.
This would likely work as "Cloudflare SideChannel", a (hypothetical) Cloudflare product that would let scrapers download the pages that humans actually visit, as they are added to the CF cache. It wouldn't work for the non-Cloudflare part of the internet where humans connect directly to the servers that have their content.
(Also, blocking a whole country is likely not what you do, but you probably know that).
Oh, and it also only takes a few minutes to reconfigure everything and invalidate those years of legal and investigatory work.
Developers and compute are interchangeable now.
It's more likely that they're trying to scrape the entire internet, and cgit is just unfortunate enough to have autogenerated pages, where each page seems to contain text entirely distinct from its neighbors. This probably bypasses the scrapers' heuristics, which normally prevent them from getting trapped in a maze of twisty little autogenerated pages, all alike.
The way you normally deal with datasets at that scale is to move postprocessing (particularly expensive postprocessing) as late in the pipeline as possible.
If someone adds an Anubis solver to their spider, but then also makes it respect cache tags and stops crawling random old commits through a git-web interface, that's probably still mission accomplished.
Depending on the configuration, Anubis will supply a token after the challenge that bypasses the challenge for a time.
So any scraper that retains basic cookies will be able to bypass the challenge for a number of page views.
A user who needs to load a single page and a bot that wants to scrape a number of pages may pay the same cost.
The amortized per-view cost is highest for the real user.
A legitimate user solves it once, shows legitimate-use behavior, and retains their access for thousands of requests. A crawler solves a challenge, is determined to use bot-like behavior after a dozen requests, gets an IP ban, and needs to recycle their IP and start from scratch.
It is indeed worse for fly-by users, but with a website like Linux cgit those basically don't exist. It would be a completely different story for, say, someone's personal blog.
Yes this needs to be tailored for Anubis, but that was the point: the moment it will become popular enough, scrapers will have no problems to bypass it.
> so I really don't see why Safari on a modern iPhone should be so slow at this
FWIW, my iPhone 12 Mini also does ~110KH/s with Anubis on lists.ffmpeg.org, so seems fairly likely that Safari somehow here isn't working as expected.
The GPU cost to pretrain on that page once will dwarf by 2-3 OOMs the CPU cost to compute Anubis, scrape and post process it. And you’re not going to just train on it once!
At best you’re creating a speed bump for wannabe players scraping with no real plan. The folks training models people use just do not care.
It's like trying to keep billionaires out of a restaurant by charging high prices, but still keeping it affordable for the middle class, it doesn't work on a fundamental level.
The people trying to do the scrapping will always have more compute than your typical user.
Exactly, and that's even before accounting for more sophisticated PoW algorithms such as the one used by XMR that almost entirely eliminate any benefit from offloading to an accelerator.
Lots of work has been done to do just that by monero’s proof of work algorithm. They achieved ASIC and GPU resistance by making you generate entire programs (with quite a bit of RAM usage) and finding one that produces a desired output. Then you submit the seed used for the program (a series of programs actually but let’s not get into the details)
Anubis wants "slow for bots but fast for humans" which is not possible if somebody has a farm of devices that costs them basically zero.
Combine with heuristics like in the article, so you can crank up the difficulty only for likely bot traffic, and you've got a bot supported site.
[1] https://krebsonsecurity.com/2018/03/who-and-what-is-coinhive...
That's not true. The goal is to raise the cost for bulk scrapers to the extent possible while minimizing any inconvenience to regular users. It's not about being slow for bots but rather reducing the asymmetry of the problem (from the perspective of the server) by imposing a (minor) cost on anyone requesting a page.
I know having the Anubis javascript at all gets in the way of some bots. But is the calculation helping enough?
It's complicated. I think anubis currently only collects low hanging fruit so to speak but in practice that's the bulk of what was causing problems so it's good enough.
PoW in general absolutely yes - I think monero clearly demonstrates this. Last I checked it remained viable to mine monero on at least somewhat low end and at least somewhat outdated consumer CPUs. It seems to me that fact implies that a well constructed PoW scheme can in practice squash the impact of any performance differences to within at most a couple orders of magnitude.
And even if you can guarantee the same CPU time on bots and legitimate requests, you still have a bunch of asymmetry in favor of the bots.
But it's also true that charging anything is going to be expected to reduce server load, that the cheaper something is the more carelessly people will tend to make use of it, and that at least in general bot operators are expected to be more cost sensitive than regular visitors on account of scale.
To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.
I keep seeing comments along the lines of anubis only working because it bounces bots running lightweight setups without js but stop and ask yourself why are they running such lightweight setups to begin with?
It's easier and still gives access to almost everything. I don't think CPU limitations are the main issue there. As far as computer resources, I'd bet RAM is the bottleneck to running the script, not cycles. That suggests a RAM-heavy hash might do better than SHA256, but still agrees with the idea that the PoW in Anubis is not the important part.
> To that last point consider that a regular visitor will generally have an idle CPU and multiple gigabytes of free RAM. Meanwhile a competent scraper will generally be optimizing request parallelism per cheap VPS while an incompetent one will just point and shoot (and proceed to blindly visit every cgit url). The latter is by far the most troublesome and conveniently also the most likely to be impacted by even a poorly implemented PoW scheme.
I considered that. But while a visitor is idle they need the result quickly and they face the full brunt of the challenge all at once. A scraper can be working on a thousand sites at once so the per-request impact of the challenges is a lot lower, and patiently working through challenges in the background usually reduces the real cost, potentially to almost zero.
And you'd better not need multiple gigabytes of RAM to visit your site, that'll freeze my computer.
Opportunity costs are real.
I might try to tackle it again when I can give it some more attention.
At this point isn't it basically auth-only?
Rant: (genuinely wondering too, and RFC, request for conversation) at this point don't we have Google, etc. basically doing Real World ID Verification, but without an open protocol backing, using it to corral users into their ecosystem and gather data, and leaving us without some open and distributed alternative like the (fairly well working it seems to me) Public Key infrastructure? (PKI)
I know Sam Altman (with Proof of Human or w/e) and others have suggested and also have their own alternative (which I think is basically what Google already does), but I think this should simply be built based on an open protocol and then hopefully not-too-large (and with a healthy competition) local companies could do it for you.
Fixing a malware infection should be like fixing up anything else broken around your house (clean things up, re-auth yourself with the ID verifier, done).
Also, bots cannot simply automate renewing the credentials of course, after expiry you should be required to physically (or in a trustworthy digital medium) re-authenticate yourself preventing any kind of mass attack, you'd need millions of people to perform authentication multiple times a day (by that point it seems safe to assume they'd rather fix the malware source in their personal devices).
[text field][submit]
(type qwerty and submit)
Do you have an underpowered TV box that would need hours to solve difficulty 6? Do you also have a Windows box where somebody installed gta6_leak_full.iso? Great, have the TV send their challenges to the Windows box for solving.
If you're operating at scale, you probably have hyper-optimized Anubis kernels, E.G. compiled for the particular set of SIMD extensions that a given machine has. At that scale, getting 1% more juice out of your machine pool becomes worth the squeeze. In the worst case scenario, you can probably get some cloud boxes on AWS to do the challenge solving for you.
I don't see much connection between that problem and these controls.
But also I didn't say it should hack when I prompt it. For the purpose of this discussion that could go either way. To avoid the griping it just needs to write the code I want when I prompt it.
All in the name of safety, of course.
Sure, the cheap lock I added to the outside shed's door-latch is terrible and can be shimmed open in two seconds with a bit of flat metal... but the important thing right now is that it keeps the raccoons out. If there's a new threat to the gardening tools or the bikes, then I have a different problem and I'll consider something else.
When the reward to abuse crosses a certain threshold, a determined party will come along and completely dominate it.
Anubis is just "solve hashes to get access to a valuable resources", and if the resource is at least valuable to whoever you're trying to deter as it is to legitimate users, you're not deterring anyone (and by extension, if it's more valuable to them, you're deterring your legitimate users instead).
the only feasible CAPTCHA at this moment is probably hCaptcha. they began to offer animated challenges. it's probably the best you can do, using an LLM to solve them would be a bit expensive for now.
As the article mentions: Anubis blocked virtually all bots for a couple of months, and even now it is still blocking 67% of requests. Without Anubis they'd have to double the number of CPU cores, solely to serve crawler traffic!
It's an arms race. The best solution is the one you can plug in right now - even if it is stupid. hCaptcha might have a better blocking rate at the moment, but it's also $0.99/1K evals, and that "animated challenge" will be impossible to solve for visually-impaired users: for many people that cost is significantly higher than spinning up an extra server or two to feed the crawlers.
You can see the implementation here: https://gitlab.com/cryptsetup/cryptsetup/-/blob/main/lib/cry...
In any case, it is GRUB which behaves slowly during boot. While I haven't tested Argon2, I believe it is affected in the same way as PBKDF2.
Or… the site will serve a random seed and the device must compute 4gb of pseudo-random data, then supply a value at a random server-demanded offset.
Edit: this is already in discussion for Anubis: https://github.com/TecharoHQ/anubis/issues/94
But it would just make Anubis suck a little bit less
Anubis "protection" model is wrong/broken on several levels. I suppose it does not even have ability to bypass hashing screen when server load is low.
Anubis is not a load optimizer. It's sole purpose is to deny the bots you don't want on your site.
It's on the same category as Iocaine and Go-Away, even if what they do is different in some cases.
This is fundamentally impossible.
In the end only pay per crawl will work. Like x402 but maybe something similar.
In other words: not only client has to spend some kind of resource for access, but it has to be a transaction, server must get something from serving the request.
No, this very article will tell you that the purpose of Anubis is to reduce load on the site. The purpose of denying bot requests is that you're strained by the load they impose.
Conversely, bots that pose no problems to you aren't "the bots you don't want on your site".
That's exactly not what happens with scrapers. Scrapers make millions of requests, yes. But each one of those requests produces an average (token-denominated) reward V'. There's no asymmetry at all: you're just adding a fixed cost to each reward, for both users and abusers.
That's before you even get to questions like "does the system actually work, or does a single solve get practically unbounded rewards".
There is: users solve the PoW once, scrappers solve it every time they rotate IP addresses. Some of them change their IP address on every request, so they keep solving the PoW over and over.
Anubis doesn't really help against this.
Which: I get! If it's your site, perform all the anger you want.
Even if you had to solve a L6 challenge for every request it's faster than the total RTT time of most servers. In other words not a meaningful barrier. And L6 is already a level which severely interferes with human usage of a site.
a $5000 ASIC-based device can hash SHA256 at 200TH/s or more.
If the source IP was an issue, you could do it other ways: for example, make the cookie rotate on every access, and insist there is a single stream of accesses.
You say “they” as if all scrapers are a monolithic group with the same constraints and goals. Part of the problem is the massive diversity.
And yet many report it works, at least for now, and the excess load due to scraping activity falls precipitously when Anubis or similar solutions are used. Maybe once this sort of challenge is used almost everywhere we'll see concerted attempts to get around it, but for now it is easier for them to just move on to another target.
> a $5000 ASIC-based device can hash SHA256 at 200TH/s or more
Peanuts for the big players, but many (almost all?) running smaller scale scraping operations are going to find $5000 rather prohibitive, and they are unlikely to be able to integrate it as they are probably running a “stock” scraper that they didn't write themselves.
Not sure why anyone would characterize these scrapers im general as all being fly-by-night operations that don’t have two cents to scrape together.
If they are so stupid an idea that they could never work, why do they seem to having the desired effect?
If you've seen anyone post a comparison of crawl rate versus difficulty, I'd love to see it. There's probably some difference but I want to know how much of the overall effect it is.
An even stupider approach would work exactly as well or better, like a form saying “type the letter y in this box to continue”. The only benefit is as a road bump that makes the site in any way custom. The moment anyone you are defending against so much as looks at the mechanics of solving the challenge it completely falls apart.
In that case I have read too deeply between the lines/tone. Sorry for the somewhat accusatory assumption of your opinion's details.
I wonder if AI might be able to come up with new PoW algorithms in a nightly CI job so every day is a different puzzle...
My earlier idea was to imagine that each day Anubis picks an entirely different problem-class. Ex: one day it is Sha256, the next it is prime factorization, the next it is twin-prime-finding, the next it is cracking elliptic curves, the next it is some kind of sorting / information theory problem...
All with the goal of adapting constantly so that scrapers have a harder time optimizing for the PoW problem (i.e. with Sha256 ASICs)
Nobody is using ASICs on Anubis.
This is something I'm working on in a prototype. Stay tuned!
You yourself describe the solution.
Extensions or browser integrations of the Anubis task.
The problem here is not merely that Anubis is inconvenient to humans. It’s that and also that it’s not very effective for blocking bots. Anything that makes it easier for humans to get past will also make it easier for bots to get past.
Edit: after some more analysis; Average with extension disabled is 90 seconds, with extension enabled it's .3 seconds
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
Why not? At this point, any attempt to try to meet the bots halfway is doomed.
More over, I actually pay for three different AI subscriptions, and use them a lot. I also think it’s fair that I pay for their services. I just want them to also pay for what they use.
But I’m seeing requests coming in from random IP addresses from all over the world using randomised user-agents. But somehow these are trying URLs in sequential order. A while ago this would be classed as a DDoS cyber attack.
Not as in “1 out of 1000 legit requests get blocked” but “you’re going to block 100% of some person’s requests.”
I feel like it's the same issue that public resources are facing.
Step 1: public resource is freely available
Step 2: asshole shows up, drains the resource completely just because they can
Step 3: either resource is taken away, or it has massive security behind it
We don't even need a new law. What they're doing is already a felony, and there's a process already in place to go to a court and request unmasking of 10000 IP addresses.
I would promote this idea to all of my competitors. Nah mate, you don't have to ask her out right now. You can wait until next week ;)
Anyway, silver linings, looks like we're finally going to get widely adopted infra for microtransactions.
https://web.archive.org/web/20030202042510/http://www.openp2...
Are the AI companies worried about anything but their valuations?
I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large.
Micro transactions [in this case] are a great idea, these crawlers need to be taxed and made to pay for the unaccounted external costs. Furthermore, we need to fund any risk mitigations, security coordination, and other safety work. If these things are not imposed on the market, the most ruthless player will create the biggest mess they can get away with.
I don’t see a better way to do this, right now, than to allow Strips, or CF, to create some way to charge the crawlers. Imagine I have some really expensive content, I can put it behind the paywall, or subscription model, but that’s very specific to the human consumer. What if I do want to search engine or content aggregator to consume some of the content? It would be quite hard for me to sign an exclusive agreement with OpenAI, like what Reddit or NYT are doing. It might be quite beneficial to create some infrastructure to let bots automatically to pay for access. It might even be crypto based, I don’t think it really matters. It just has to be easy for me to integrate it, and the bots need to find it easy to use.
If we all start doing this, I think this could be of benefit to all content creators, such as musicians, movie makers, game developers, and even independent journalists.
We used to pay for software and things, but what has happened? Walled gardens and production monopolies are taking over. This has largely reduced the quality of all the content we consume.
>I don’t care about their valuations, but I do care about the risks that they are creating for the economy, society, and the technological advancement, at large.
Doesn't the same thing apply to every company? And also every government? And also individual?
We're all doing the "locally rational" thing, and globally doing... well...
I think the main factor is time horizon.
Locally rational behavior begins to look globally rational when you add time horizon.
For example, if I'm doing what's best for my family for the next five generations, that starts to look a lot like what is good for the whole country and the whole world.
Companies, individuals, and governments have different roles. Rationality from their perspective is specific to their perspective, and I don’t believe just because evolution or physics are rational we should die off from a new virus or asteroid impact. That would be perfectly rational, but I don’t care about the global rationality of the universe.
What is best for my family and other families is not what is best for Sam Altman or Sataya, or Elon Mask.
We’re just talking about market players. What I’m saying is that these market forces need an outlet via some new infrastructure and regulation. Maybe it’s not micro-payments, maybe it’s some other new ideas.
I just don’t want the next century to be the century of East India AI sticking me in a Matrix pod, rationally.
What is its objective function?
Anyway, what’s wrong with making 2 cents? :)
I don’t think this will make the internet less open, I think the internet is already pretty much less open. Who blogs these days for free? They rely on medium or substack, and have no choice, they submit to these platforms rules about the articles they write, even though they’re pretty generous.
Now, if the reward system was as open as say, Bitcoin, I think there would be a lot more independent and different bloggers and journalists, and independent platforms.
they have one of the worst bot detection in the industry. but because everyone uses them, their huge false positive numbers won't show up anywhere.
And I only stand to gain from pay-to-crawl, so I don't really mind that play.
It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tuned blocklists. Don't fall for the marketing!
Try it yourself. Block those fingerprinting tech and try to access their own user forum. you get blocked. Now allow fingerprinting (even if emulated in a headless chrome!) and access by the most toxic vpn you can think off, access granted.
All cloudflare does is act like a gatekeep for monetizable-by-creepy-advertising-networks traffic, not bots. If the bots pay the small headless chrome tax, you get bombarded with their requests just fine.
So you can definitely block by IP, by fingerprint, both or neither
why do you feel like discussing features was relevant?
And Cloudflare is no exception, the "hello world" of bots is Cloudflare bypass.
https://news.ycombinator.com/item?id=44962529
It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.
However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense as a theoretical barrier against bots. Who cares as long as it works empirically, even briefly? Are you suggesting people let their sites continue to be degraded or unusable until they find a better solution, even though PoW challenges can improve the situation today?
Not everyone has the luxury of writing their own web app designed to handle massive bot traffic at minimal cost, gutting functionality to meet that requirement if necessary.
I noticed a couple of weeks ago that 1-2 orders of magnitude more bots were getting past anubis challenges. However, it would be incorrect to conclude that it's doing nothing and now worthless aside from heating up visitors' cpus. If I turn it off, I see >2x bots.
The next salvo in the war of attrition will probably be when anubis switches to wasm hashx, which the author suggested, in previous HN threads, is on the roadmap.
If bots quickly adapt (Claude vibe-coded JIT wasm-to-native-transpiler for hashx challenge algorithms?), that may be the end of anubis and efforts like it. If they don't, I expect more HN threads with people complaining that such compiler efforts are possible and therefore PoW-gated sites are doing it wrong... even if those sites stay usable.
I'm just saying, it's bad computer science.
It's like saying we should just completely give up on all NP-hard problems because we'll never discover a way to find the optimal solution in all cases. Meanwhile in the real world, people are more than happy with, say, a route planner which virtually all of the time gives them a solution which is within a rounding error of optimal. Nobody is going to care that it is technically bad science.
Same with Anubis: Sure, it's a dead end in an ongoing arms race. It won't last forever. But right now it works pretty well, so let's hope it lasts until someone can figure out a better approach.
A better, perhaps the best, argument is that a worse bot mix might appear tomorrow, so anubis should be used as a runway for a better solution, and not as a solution in itself. If only everyone with a website had the desire (or AI tokens) to implement their own better solution. In some cases even desire or a lot of tokens isn't enough. I mean, you think kernel.org folks haven't put some serious thought into it? And yet, as HN argues about it, there's kernel.org running anubis at difficulty 5.
It shows they really don't care and they are just rage-baiting readers in their weekly grumpiness without any suggestions.
Unless they come up with said "solution", they can completely ignored in their complaints.
So could you give a short elaboration on what was this "better computer science" solution that you came up with in your discussion?
(Nothing against it, I'm always happy to hear what Thomas has to say)
If you were dealing with a DoS like this and had an option that cut the annoying traffic in half in a fairly deterministic way, that sounds like a decent thing to do operationally in the moment right?
- It's the SOTA companies doing it?
- Scrapers are doing it for training data?
Those are two assumptions I see in posts and threads around Anubis, that are taken at faith, and never once substantiated.
[0] https://www.ft.com/content/07611b74-3d69-4579-9089-f2fc2af61...
There are more than just the American top dogs (OAI, Anthropic, SpaceX, Facebook)... especially the Chinese government with all its infinite cash resources and next to zero ethical constraints.
I don't trust the US top dogs at all, but I think the fear of discovery alone would lead them to not use "residential proxy" services. Non-US/EU entities however... who cares?
The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site.
If each new site they hit forces them to solve a proof-of-work, that’s a meaningful dent in their scraping performance. Many of these boxes may not even have the spare CPU power to efficiently solve so many proofs of work - and anything that makes an owner notice their device is running slow is something that could meaningfully impede adoption of these SDKs, or force the operators to choose between minimizing performance impact or scraping more sites.
I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing.
I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
An author of a scraper would need to trade off how much arbitrary compute they'll run against how much risk they take that someone will use them for free compute (inverting their role from exploiter to victim). They can work around that somewhat by analysing common enough code (to justify the LLM or human expense) and deciding to run it.
PoW challenges work when they have low enough adoption that it isn't worth scrapers' while to have a custom countermeasure for. To scale them once adoption is big enough, it probably takes some bespoke engineering per group of sites (coupled perhaps with some scraper traps that take advantage of scraper CPU for something useful, and aren't easily distinguished from the normal challenge).
I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point.
Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol.
If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.
See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha
All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.
Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users.
The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while.
[1] https://news.ycombinator.com/item?id=49500040
Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good.
The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.
And I really wasn't trying to be snarky; people had "what is i + j" type of captchas for a long time and I still see them every once in a while!
And the bot progression from "alter user agent" to "change IP addresses" to providers having to ban whole subnets, whole ASNs, and realizing "proxy SDK monetization" is a thing mirrors threat actor progression from the time before LLMs.
The AI crawler traffic I’ve seen sends one request per ip and seemingly has an infinite pool of residential IPs. You can’t block the ASNs becuase you also block honest clients. IP blocks are the wrong solution.
And because I’m being negative I’ll also be constructive, IMHO the correct solution for fighting residential proxy crawlers is using RTT diffs this is one example https://github.com/Sakura-sx/Aroma
It's not the most sophisticated thing.
Fork Anubis. Slightly modify the hash function it computes. Deploy. Do not try to make your fork widely adopted. Do not even publish it.
You've just defeated ASICs and any craweler that's special-cased Anubis (currently all of them). If enough people do this, the only recourse they will have is either genuinely executing served js code like a real user or building some unholy pipeline that uses ai agents to compile it to a GPU kernel for every host.
A more robust solution would be to keep a few patches handy with different versions of the algorithm, and rotate which one is in use. This would keep the crawlers on their toes if they wise up to the changed algorithm. One could even imagine automatically rotating witch algorithm to use on a weekly basis.
I do wonder about the premise as well: are people special casing for Anubis?
[0]: https://github.com/Homebrew/brew/pull/9383
[1]: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm...
If I'm doing a shallow clone it isn't because I only want to receive a specific commit, it's because I don't want to burn a giant amount of disk space and network traffic on a full history.
In most use cases it would be perfectly acceptable for the server to send additional data. The client doesn't care about it because it is meaningless to them, but if it results in a significant load reduction on the server's side they don't really mind receiving it either. A 100MB shallow checkout coming with 400MB of garbage still beats cloning an entire 5GB history!
As a tech person who works with tech people, I have become extremely sensitive to this kind of bias. Is this solution actually better? Is the CPU cost actually worse than mildly annoying everyone, or is it a problem being solved because it “offends the senses?”
I’m not leaning towards yes or no for this instance. But I regularly see people jumping to conclusions without measuring. What is the cost of 20% and is that cost worth “mildly annoying” everyone?
The problem is the cost here is paid by a group of volunteers, no? It's objectively reducing capacity for a really important public project by 20%. If you're responsible for keeping a public good like this available, it seems obvious to me to want to prevent this overuse of resources. Only one of the 3 parties involved here is not willing to engage in good faith right now and causing harm. I don't understand a need to try and tolerate them.
> What is the cost of 20% and is that cost worth “mildly annoying” everyone?
from the articled:
> With a bunch of generous assumptions, legitimate requests are only about 2% of git.kernel.org traffic — everything else are scrapers.
this is not some "CPU use is 20% higher than baseline" situation. it seems that people still do not understand the scale of these bad actors.
Nevertheless, without knowing what it actually is in cost, it’s impossible to have an objective sense of the severity of this problem.
Is it, “everyone’s mildly annoyed so we can save $10k per year” or is it “everyone’s mildly annoyed so we can save $100k per year?” And has someone attempted to reason about what the $/year cost is of that mild annoyance?
I’ve seen this a lot in my career. Developers coming to conclusions that are ultimately based on what feels expensive or problematic.
However day after day it just gets blasted with requests for deep pages. I was quite alarmed when I saw a 100x increase in the daily active user numbers which relied on session length, only to realize they were all bots.
Naively I too initially resorted to blocking user agents(Meta is thankfully nice enough to identify themselves, not nice enough to stop blasting 50k requests a day however), IP ranges from cloud providers and various browser fingerprints that I found connected to suspicious traffic.
However the battle seems unwinnable at the moment, outside of gating all content behind auth which I don't want to do. We have around 500k user generated content pages and I want those to remain publicly available.
I would be happy to provide our data to any one of these scrapers and I even added a message asking them to contact us if they want access to our data whenever I return a 403 response, however nobody has reached out.
Another campaign that someone is constantly running is daily checks for 100s of possible secret/config paths in hopes of finding an exposed private variable, these i've just blocked even though they would return a 404.
I still haven't found a way to deal with rotating residential IPs however, and most likely never will.
My current approach is to just run a 24 hour scan of all requests with codex and update my next.js proxy with more IP ranges, browser fingerprints and anything else that won't affect a real person.
Has anyone managed to come up with a way to stop this onslaught of crawlers and scrapers?
which is called l402/x402. micropayments' day has come
The author of Anubis hates this fact and will ban you if you mention it, so don't. He insists it's the PoW.
>will ban you if you mention it
Even if mentioned really politely?
[1] would hope anyone trying this makes it accessible to visitors with disabilities
Anyone who responds with something like:
> I don’t have subjective feelings, consciousness, or an inner point of view, so there isn’t a sensation of “being me.”
is definitely an LLM.
Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.
The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.
Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.
I'm certainly not saying you should use this code, but it's a proof of concept for avoiding the CPU overhead of cgit rendering by loading the data on the client.
It cost me £8.27 of Fable use (from the free credits I've been given) and 56% of my five hour quota on a $20/month Pro plan. There's no server logic, it's 1.3MB of minified JS and CSS and (while I'm absolutely not suggesting anyone try to use it) it basically works: https://github.com/andrewaylett/rgitweb
This is a one-shot, my prompt set the expectation that I'd be able to load resources using CORS but (not entirely unreasonably) the Git hosts I've tried don't set CORS headers.
Shared more because I was pleasantly surprised at how cheap and easy this was -- and with a repo link because talking about it without sharing the link would be a bit crass.
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
Thats a good enough explanation in my opinion.
Source: someone who reviews AI generated PR's on a daily basis
This ends up tacitly turning a very legitimate complaint (ill-behaved crawlers) into a justification for head-in-the-sand AI denialism.
That's head-in-the-sand stuff. AI is certainly very capable of being dumb (as are humans). But:
> “The problem was in need of a new real idea, which this new result seems to provide,” says James Maynard, a mathematician at the University of Oxford. “It seems that the AI has made a genuinely interesting mathematical contribution.”
https://www.scientificamerican.com/article/no-ai-didnt-just-...
Nobody a decade ago would have said "oh yeah solving a bunch of open problems in research mathematics, and finding a bunch of zero days in Chrome and Firefox, and winning literature prizes, are things that don't require intelligence."
I wonder how exactly the average scraper got to be so inefficient on kernel.org.
Did someone prompt a SotA model to write the most generic scraper possible?
Did someone prompt an old local model on their laptop to write a kernel.org scraper?
Perhaps no LLMs were involved in the first place. Seems to me there isn't much relation between how good a random scraper is and how usable/effective Mythos/Sol's outputs can be.
> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pickaxes.
It's not really going to help though because the scrapers using residential proxies aren't burning their own compute.
I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.
I just close the website if I see Anubis. Some have it set at reasonable difficulties (like 2)… others have it where I need to wait for like 30 seconds, I'm not wasting 30 seconds of my life for that.
when a system was designed for 100 req/s and bots hit it with 5000 req/s, nobody entering that queue is having a good time. Anubis is the trade those operators make just to ensure your request gets serviced at all.
30s load time is already a sign that the Anubis approach is breaking down. if there's nothing else ready by the next order-of-magnitude increase in crawler load, those sites quite likely will just disappear from the public internet. hate Anubis all you want: for most of us, the realistic alternative is strictly worse.
It feels like turning these frontends into JS-only viewers would resolve this, for the most part. The JS clones the repo in memory and renders whatever lens the requestor wants, and the server becomes a dumb object storage that uses less resources. The anti-JS folks are free to clone the repo still, and view whatever lens they want, so that minuscule slice of the legitimate requests is still served, albeit with a degraded experience.
First, scrapers would start running JS. Whether they're running chromium-in-a-box or something more clever doesn't matter. Compute is effectively free for them, and even shitty WebOS set-top boxes can probably run a stripped-down headless browser with a JavaScript engine.
Second, a full clone in browser memory is massive for something like the Linux kernel. Lots of browsers and pro-JS folks wouldn't have the resources to run that.
Third, assuming that scrapers are willing to play the JS game, suddenly you have a massively increased rate of full clones happening. Even if serving raw git is cheap for your backend, the bandwidth the elevated clone count drives is not.
2. Why would you need a full clone to access /blob/cee9395acd8043be0644b25c34bfa86623f2b935/block/badblocks.c?
Rather, the article describes the routes that compute views (e.g. show the change history across several commits) into history as being the issue. If we take that example, a JS client would have to fetch N individual commits/routes/objects and compute the requested view, but first it'd have to fetch the indexes/logs to determine what commits exist within e.g. a specified range.
I suspect that'd require more work on the frontend than "just run WASM-built git" ... unless the proposal is for it to fetch all requested objects lazily, in which case I think you'd be surprised how many files are read by git when answering a question like "show me the diff by user XYZ in file ABC on branch QRS between date 1 and date 2". That starts to get expensive to pull in the browser, and the bandwidth costs might start to hurt even if the backend now only had to serve cacheable dumb blobs.
But this begs the question, why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes? (Making a space-time tradeoff that doesn't make sense on a public-access website in a post-stupid-AI-scraper world?)
Thanks! FWIW, I engaged because I think there's the basis for a good approach here and I want to see if we can expose what works through discussion.
> why is cgit scanning 340 MB to answer this question? Is cgit not using good indexes?
That's the critical question (ugh, as I write this I realize I'm starting to sound like an AI, sorry). It comes down (I think, I don't work on kernel.org) to a few things:
- cgit is ubiquitous and easy to deploy.
- Cgit doesn't have a database, or indexes. It's a fairly thin wrapper over git itself. Git itself is optimized for performance of local filesystem operations, which is nice, but means it can be written with e.g. the page cache in mind for performance, and doesn't need to spend a ton of time thinking about the number of files it accesses--after all, they're all small and locally-available, and thus probably cached.
- Things like cgit often have to combine (the C equivalents of) multiple chained git commands to render certain views.
- Cgit has caching, but it's simultaneously too naïve and too specific to handle use-cases like this. It's too naive in that it's caching content blobs via the filesystem (and using sendfile to serve them) without awareness of e.g. shared cache blocks for deduplication or anything, so it can't cache intermediate states used for rendering e.g. diffs. It's too specific in that it's only caching the results of certain underlying git operations and specified renders, so it can't prioritize e.g. cross-application LRU/frecency patterns when deciding what to cache and what to evict.
In short, what you're proposing is probably best implemented server-side (and hopefully already exists): it's an indexed database of Git's objects that has behavior parity with Git's rendering of the various views/questions people can make of the server. That's a nontrivial undertaking: git doesn't have a VFS layer you could plug a database into the way something like SQLite does; it very strongly prefers procedural file-based accesses to its data. If you could use something like that, you could engineer it like a typical webapp: content-unaware caching layers and indexes to make certain operations fast. Not easy to build, and not easy to swap in in place of cgit (which you also probably have to keep a lot of behavior parity with), though.
If someone else wants to run a public HTML service, let them deal with the bots.
If you really want to offer a web interface, put it behind login. You can apply enough restrictions (captcha, super slow rate limit for new accounts) that it isn't cost effective to generate zillions of logins, and you can monitor logins for bot behavior.
Sucks, but here we are.
You can fairly arbitrarily increase the difficulty of user registration, far beyond what users will tolerate for viewing an individual page. You can exploit this; a bot needs to make many accounts for the activity desired, and it's not hard to make "generate account" more expensive than it's worth for the amount of activity they get from each account.
If it costs your attacker a penny to solve the captcha to make an account, and they can only get 100 pages out of an account, you win.
For many sites, your hypothetical penny to create an account is a cost an attacker would gladly pay.
Reason they interact so poorly is that is that git hosts generate a lot of links. One for each file in each commit, and a diff for each file appearing in a pair of commits. Even a small repo can have millions of viable links, and most of these are stupidly expensive to render for the git host. On top of this crawlers generally don't have a very deep understanding of what they are crawling, and can't meaningfully distinguish computationally expensive requests from cheap ones.
Distributed scrapers are going to generally try and hit their assigned list of pages; it’s a bigger waste of time if they have to go off to visit other pages first to get the cookie challenge.
Anubis has a fairly capable "policy" system. you can place something like this in your policy.json:
``` { "bots": [ { "action": "WEIGH", "expression": "path.startsWith(\"/expensive/endpoint\")", "name": "scrutinize-expensive-endpoints", "weight": { "adjust": 20 } } ] } ```
another thing smaller sites benefit from -- where the load induced by crawlers tends to be bursty (e.g. as they discover new expensive endpoints to crawl) -- is to adjust the difficulty up/down to maintain a steady system load.
``` { "bots": [ { "action": "WEIGH", "expression": "load_15m <= 16.0", "name": "sustained-low-load", "weight": { "adjust": -10 } }, { "action": "WEIGH", "expression": "load_5m >= 24.0", "name": "intermittent-high-load", "weight": { "adjust": 10 } }, ] } ```
Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.
1) ban TV-proxy-as-a-service - straight up go to every representative there is and start pushing and lobbying and everything to stop spammers from distributing over non-computer devices, especially legally
2) make old commits more expensive to access than new ones. Legit users are not going to access those much, so they can pay the time. I assume diverse (unpredictable?) difficulty can also make spam pulling harder
In every country of the world? I think you might have a significantly easier time convincing Samsung and LG to ban this via their T&C.
Another solution I came up with while reading HN comments: whitelist IPs instead blacklisting. Give access to well-behaving hosts/groups. It can even be shared across different sites. Although this would create a market for selling "good IP" proxies.
Obviously though none of us want to give that up, so the alternative is that we can almost never 'prove' we are human especially with bots getting as smart or smarter than the average redditor.
Reminded of the mules renting Airbnbs to use as USA-based delivery locations (tricking grandma into FedExing cash for one scam or another) - https://getrichslowly.org/scambaiters (probably Jim Browning + Mark Rober specifically https://youtube.com/watch?v=Xvjjpzyiig4 )
But! Using a network of real ID-checked humans to scrape the web, what would that be--half a billion times harder than Firecrawl or whatever they use today?
Too bad it's dead in the water today because so many (like me) hate the idea so much. Perhaps a biometric dongle (retinal-scanning orb :-/ ) that the staunchest privacy hawks stamp with their seals of approval because it's somehow nearly impossible to go horribly horribly... horribly... wrong... Yeah, anybody who can crack this issue, hope you have the free time or find the funding to try it, we need ya.
I think I agree with this. There will be difficulties though:
- Will likely have to be pay-to-play. No point in whitelisting IPs if you are bombarded with whitelist requests by a bot.
- Will come with the assurance that any bot traffic will result in an IP ban.
- Will rule out people with dynamic IP addresses.
- Will be difficult to gain new users.
Why? We are not so special or magical.
If we trusted digital signatures on content instead of server names, we could have a model where a single bit of server load propagates to millions of interested parties. As it is the server must do something distinct for each interested party.
CDNs mitigate this only partially, because mutable data means they have no good cache invalidation strategy.
There's got to be a solution that doesn't involve heaping even more burdensome requirements on those who would dare to publish.
What irony that this article is about crawling content-addressed data.
I'm not sure where IPFS went wrong, but I think our web will continue to degrade until we figure it out. (It'll continue to degrade after that also, but then we can let it burn since we'll have a replacement to switch to).
We hardly even use the globally consistent names we have. Between humans its always "hey did you check out that website I sent you?" And never "hey did you check out https://foo.bar.biz"
Sharing and endorsement can just be part of the protocol. My reference to the site may as well be in terms of which of my friends think its trustworthy so I'm better enabled to make that determination myself (some of my friends have better taste than others).
Knowing that Alice.brownierecipe.2 and Bob.favoritebrownies refer to the same underlying document with an unfriendly global name is not so gnarly of a UX problem that we need to tolerate single points of failure to avoid it.
https://bsky.app/profile/xeiaso.net/post/3muasair5uk2j
Rather than using Anubis to give the client fake work, maybe the way to go would be to get the client to do the real work in JavaScript? Then the server doesn’t have to do it.
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
[0] https://git.zx2c4.com/cgit/about/
But given that it's only a few million actual commits it may be possible to come up with an intelligent caching system that at full capacity require only a few TB.
It's ridiculous also how you ban an IP then 1 second later another one picks up from where the first one left on.
Maybe we should serve the bots a problem worth solving and benefit both parties. They spend some energy/tokens on a problem and we pay them with content.
If only I had a bot problem, this would be interesting to explore.
The people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.
Extensions like ModHeader, BrowserMask do this: https://github.com/apify/crawlee-python
https://github.com/mthcht/Masquerade-Spoofer
Great for AI scrapers, bad for hosters and everyone else.
It's like anubis but instead of doing useless math, you will be moving the legitimate cpu work to their side.
This seems fine to me. It would be a better world if we could have anonymous bulk data access. But if aggressive scrapers are bloating host costs, I’m fine with logging in.
Now, the flip side is that ONCE logged in, I want my bulk access. The worst of all worlds with when you demand authentication and then STILL block bulk access.
Case in point, I want to automatically download my Amazon and Target order records. This is easy to automate with playwright or whatever, but authentication stays annoying. My sessions expire quickly and I have to re-auth all the time. There should be an API to pull this data down.
One persistent thought is what are people doing with this data? I get that people want to train models, but I also have a hard time believing there are more than a hundred companies with the resources to spider the web like this and actually do anything meaningful with all that data. Academics, researchers, and people working on lower level innovation are probably well off with CommonCrawl.. only people trying to make frontier models really need fresh and endless data right?
TIL shallow clones are expensive. That's wild to me. It's supposed to be cheaper!
High bot likely hood if: If a session jumps to a different ip. If the session jumps IP after just a few requests. If a new blank session starts with a deep link. Maybe those are cases where some POW is better justified?
Assumption: the rendered HTML might be viewed by a legitimate developer, even via a deep link from outside. But rarely from a wget script without a session cookie .
The other nice idea from the comments - is this rendering effort something that can be pushed to the user? Instead of pointless POW work, can you offload the expensive rendering to the user side?
But certainly, this is just an armchair comment and the kernel guys certainly have tried everything in this arms race...
If you do scraping on a large scale you emulate a human very well to bypass bot protection. Setting dynamic but accurate user agents, setting proper sessions and persisting it, emulating the TLS handshake (https://fingerprint.com/blog/what-is-tls-fingerprinting-tran...) and emulating mouse movements to be more "human like" (https://github.com/oxylabs/OxyMouse) are table stakes.
The article mentions that some requested pages are less likely to be legit traffic (old commits) and more likely to have bot activity. Perhaps they could increase difficulty on those pages for the proof of work. Keeping the "current" at a lower difficulty allows most normal users to use the pages as normal, while penalizing the bots.
One other way I have been thinking of is just delay the delivery of the pages, either limit bandwidth or just wait for a bit until you deliver the page. For one user a (lets say max)3s delay on some pages is not a huge deal, however at scale that adds up and means the client can't gather other pages in the meantime.
Another option is to lock the out of date html renderings behind a login page while keeping the git openly available.
Or some kind of vintage “set up a request in a form and press a submit form, and we’ll pretend to take a while to put things together.”
Allow git clone for free/unrestricted still.
Require the user to sign in to view html views. Sign in require a valid email or phone where a validation link is sent.
Or: Users signed in won’t see the Anubis. Users not signed in can still see the html views but have to use a very high work level?
Or: Limit unauthenticated requests from an IP to 5/minute. Authenticated requests can do a lot more before hitting the limit.
Want to render than seven-year old commit via HTML? Sign in. (But I have no idea.)
Why cannot we turn this whole proof of work thing into an official "help mining $SHITCOIN"? I mean, if they really want the data that badly, at least have them pay the hosting with their CPU/GPU/ASIC cycles.
dont click next link because he will show a nutsack image if the referrer contains hackernews. love the guy.
www.jwz.org/blog/2025/01/exterminate-all-rational-ai-scrapers/
basically, instead of blocking, he just poison it. and if a human sees it, it takes less effort to ignore the nonsense than it takes your pocket computer to deal with proof of work.
First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider.
So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.
So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks.
I don't have a solution, but "break how the web currently works" is not one I would accept all willy nilly.
EDIT: yes, I realize we already broke the web (with Anubis, cloudflare, recaptcha etc) but I think we should resist slowly breaking it further.
First of all - I suppose it should be very cheap. So, real humans will not pay much.
Second - why do u think that websites are meant to be free? They provide some service, so its a rather strange that the internet is so free (in both senses). I think, this freeiness is allowed to greatly speed up popularization. But for me is obvious that it can demand payment for service.
And third - service owner really meant it to be free, I don't see any problems with this in my idea. It can still provide free service.
> [four phases, each of which has historically been hard-to-impossible for highly organized international coalitions]
Never going to happen.
If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?
Barring that, I think the solution is to charge money for access. Require users to sign up to render HTML, and provide a form of payment (any form you want). The cost is, say, $0.1 per GB. Rate limit all requests to reduce CPU. For the average user this will cost a few cents. For the bots you'll cover your costs and have a rate limiter to keep your system from being overwhelmed. Or they can git clone for free with no limit.
Also, many plotting libraries include an xkcd style these days:
https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot....
And libraries for various languages:
https://github.com/timqian/chart.xkcd
So if you have a preferred dev environment there's probably a way to set it to xkcd style.
Except the majority of LLM training content nowadays is synthetically generated by LLMs. I wish people would stop making this statement, I don't know why this claim persists to this day. It wasn't true two years ago and it sure isn't true now
I think any service that is trying to sell a way to solve captchas must be banned by government. At least these things shouldn't be done so openly.
Not sure how difficult Anubis is, but I would not be surprised if parsing the request through the LLM costs multiple orders of magnitude more compute than the Anubis challenge.
And that's just known AI, I can't imagine what requests are pretending to a real person when they aren't.
I think it's a few bad actors really. Because nobody serious about indexing content will do what these crawlers are doing..
They are consume lots of content that is unoriginal or duplicate or duplicate with minor modifications.
Not sure how to block, but maybe a little bit of law enforcement could dramatically reduce the number of TVs being used a proxies.
That's bad for static content.
Try adding search, with pagination and 16 filters that can toggled on off. And 1000 tags, give each page 5.
I've handled every type of attack and malware that came their way but these past few years, ai scrapers are a large por or their traffic, eating into the budget and now interfering with sales.
and I don't know if anyone else is noticing or watching these ASNs but it sure looks like a few well know and big name AI companies are using *residential proxies* to so their scraping.
That’s not an option for the kernel. It’s hard to read the cat-and-mouse account with any hope today. I think the flood abates someday but not sure how it happens.
It's hugely inconvenient for users not having an account and want to do more than clone an entire repo via an API of course.
If you have a thousands of IP addresses, and you know all the urls you want to request in advance, you can just request them all simultaneously I guess.
The next more advanced version is that URLs are unique to your cookie.
Users can't share urls anymore, but it might be a tradeoff worth making. Unique urls for each user.
You could probably still make this work, if you share your url with another user, they get the page, but heavily rate limited like a regular no cookie request. (a cookie url mismatch gets the rate limited version of the page)
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you.
Some ISPs ban customers based on a single report there - have fun!
You are, in reality, only hurting the actual owners, the subscribers of those ISPs who are behind those addresses. We call that "collateral damage".
If any of those actual residential users try to use a website, their ability to freely access the Internet may be harmed by a bad reputation that they do not deserve. They may be totally unaware and non-consenting to residential proxy use.
You are not, in fact, hurting the residential proxy-ers at all. Not one bit. They will move on to another IP and another compromised LAN, and they will continue to move on and on and on. They will not be harmed or impeded; they will simply keep turning up fresh, new, high-reputation IPv4 and IPv6 sources. This is a sheer numbers game, where the numbers are always in favor of the attackers.
Also if network admins keep blocking/filtering abusive residential proxies, they will balloon their firewall rules and cause actual performance issues at the network level. You will turn into your own DDOS without any actual benefit. You're on the losing side of the numbers game, and in the immortal words of W.O.P.R., "The Only Winning Move Is... Not to Play."
And similar to how most people running mail daemons are using blackhole lists nowadays and are keen to not end up on there, maybe ISPs and web hosters can use the AbuseIPDB to sort out their customers.
Just doing nothing doesn't appear to stop the scans hammering my poor Raspberry Pi serving my few Git repositories.
Now in 2026, running a "public proxy" doesn't take an administrator. You don't even need to be aware. Most victims are unknowing victims. They simply subscribe to an ISP and they have their own devices. They are being exploited for that innocence and ignorance. Most victims have no visibility to even detect that they're being used as a proxy. Most victims couldn't stop it, even if they wanted to.
I challenge anyone with a home router to list the processes running on that router, and list all current open connections on that router, and list all open, listening sockets on that router. I bet you can't do it. There are no consumer router OS that lend themselves to being secured, or even diagnosed. Malware can easily be planted on any of them and run, completely invisibly.
A residential proxy server could run on routers, could run on a switch, could run on your "Smart TV" or a smartphone, or a notebook computer. It could be anywhere in any form. Perhaps you consented to it, perhaps you didn't notice.
In no way is this the same as an SMTP open relay situation. If you wanna play "whack-a-mole" with a "blackhole list" you're simply going to overwhelm those lists with false positives and collateral damage. The residential proxies have long since moved on. You won't even find the culprits using those addresses you just blocked. You're just clogging up your own machines. It's a total self-own.
How many times do I have to hurt them before they decide to buy a different smart TV?
Seriously, that's like saying "if you try to stop your neighborhood rodent problem by getting citations sent to people with cat food on their porch, you're just hurting the innocent outdoor cat owners". They're participating, whether they know it or not. We can and should PSA and shame and regulate away residential proxies on the supplier side, but we can and should also simultaneously discourage them on the end-user side as well.
Using shared ip banlists is the best solution so far, like cloudflare. Sure maybe they hit your server for 5 seconds and then desist, but they'll attack someone else, and they'll eventually rotate.
I'm not sure if Anubis has a feature for centralized banlists, but I'm assuming since it's OS and privacy oriented, there isn't.
There's a tradeoff between privacy and abuse, you want privacy? You get abuse, you want to battle abuse? Gotta sacrifice privacy. Worth noting that unmarked vpn users (residential proxy or residential vpn users) use these proxies for privacy, and therefore give a reasonable alibi to abusers.
If you know CloudFlare as anything else, it speaks to how successfully it has grown and marketed itself into other areas.
The people that Anubis tries to block have tons of compute to spare.
For example at runpod, if you rent a container with just one B300, you get 32 cpu cores and 250 gbs or ram that are essentially just sitting there while the gpu does all the work. If you think you can impose a compute cost on these people without blocking everyone else, well think again.
Or in my case, I am running qwen3.8 at home on a couple of gpus, these are attached on 32 core epyc server with 128gb of ram I am pretty sure i have more compute than the typical dev laptop.
I am of course nice, and don't aggressively scrape peoples services.
Is that really the case? It was my understanding that LLM-based agents were explicitly allowed as long as their users follow certain guidelines [1]?
And more generally: Somehow the theory of "essentially all bot traffic is AI labs crawling the Internet for LLM training data" doesn't make sense to me at all. There are at best dozens of labs capable of running their own crawl at Internet scale, but hundreds of millions of people using LLMs to answer their questions. (If my personal LLM usage is any indication, firing off dozens or hundreds of web fetches to answer a single question is not unusual.)
While I understand that many existing projects have been resourced only for human readers and might as a result be struggling due to this, this characterization sounds a bit dishonest to me.
And unfortunately, for this use case (i.e. ephemeral queries in a context possibly lacking storage or git access), forking the individual repo to answer a handful of string match queries against it might just be more expensive than to run that query against a web search index and then just fetch those results over HTTP.
The solution would accordingly also look very different, as caching at the inference layer is significantly harder than at the training one (where it's most likely already widely done as that seems like a no-brainer).
[1] https://docs.kernel.org/process/coding-assistants.html
If you're reading this - go ahead and see HOW Linux Foundation spends their money.
We’re talking about privacy-preserving proof of age, but as we see here the real utility of such a system will be proof of humanity.
How do you ensure it includes every human?
How do you ensure it doesn’t include every non-human?
I’m not even asking about computation or algorithms. I straight up don’t think you can make a definition that isn’t a tautology or an approximation. Both of which are useful, but neither of which can fit a _proof_.
I want to take my existing national digital ID and use it online basically. BUT I don’t want the websites to know it’s me. Just that I’m human (or perhaps over a certain age). And I don’t want the ID issuer to know what site/service asked whether I’m a human or I’m 18 etc.
I actually doubt that! And in places that do I doubt it’s perfection. Citizenship is mostly defined legally. Residency has a legal definition that may or may not track with reality. Most legal regimes I’m aware of (IANAL) define personhood; whether humanness is defined I’m less sure of. Things that are more nebulous and subjective…sometimes they’re defined, but necessarily subjectively so.
And that gets you into dangerous territory if you’re at the margins.
A fun example: how do you define “alive”? Does the person have to have been born? Are there conditions on that? Would they exclude some people we think of as alive? Does the person’s body need to be self-sufficient or can it rely on mechanical or other intervention? Is there a line? Is the person alive if their body is functioning but their brain is not? What about vice-versa? And even if the answers to all these questions are currently perfect, what happens if and when the definition shifts?
Being alive and human is messy business. And this is setting aside the discussion of making government issued records safely usable without leaking, which they already aren’t.
Of course the debate would revolve around ”is it fine to have a national government issued ID be the ultimate proof of your identity/humanity?”
It’s problematic - yet I think it’s better than the alternative.
Let's say that 95% of individual humans can pass it and only 2% of bots. For someone maintaining a website, who has to decide between using this system and shutting down their site because of the increased costs, that may very well be good enough
(Each new tab is isolated, unless opening a link in a new tab. Same as Safari in private mode)
- Using a terribly inefficient way to redownload the same commits as e.g. HTML diffs, possibly the most inefficient.
- Putting in tons of CPU cycles to surpass the Anubis PoC.
- Putting in other kinds of active effort like reworking access methods and buying "residential proxies" that are probably illegal in most jurisdictions.
This sounds more like escalating DDoS than AI scraping.
AI scraping might be bad, but if a particular case that's actually a DDoS becomes the cause celebre against AI scraping, it will weaken the argument, not strengthen it.
This is not a hard problem. It just requires that someone build a certification wall. The IETF should have done this long ago, right?
These scrapers aren't LLM agents. They're distributed classical programs that harvest data which is later used to train an LLM. The LLM doesn't write the scraper or respond to individual scrape events. The entity training the LLM contracts someone, who contracts someone, who contracts someone to run a web scraper and send them the data.
For some reason, like with fraud via email, text, and phone, we don't do anything about it. All this brazen crime and government does nothing; we don't even imagine government doing anything.
You have the power, and should exercise it, to rate limit bad actors
It's well known, and it was the point of the OP, that we don't have that power. If you know a solution, please share it - the world would love to hear it.
(I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually obvious who's running these bots. The big players all have crawlers that identify themselves and are reasonably well behaved, but I don't know if anyone has managed to positively attribute these other ones to any particular group)
This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!
Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option
GPTBot → model improvement / training crawl
OAI-SearchBot → search index / ChatGPT Search
ChatGPT-User → fetch initiated by a user's ChatGPT interaction
OAI-AdsBot → advertising landing-page crawl
(No second chances, I blocked their whole IP range.)
Honestly, the answer is simple: sue.
It'd be hard to argue that it's not a DDOS.
Maybe I'm not understanding how many requests at a time bots are sending to kernel.org (or how larger kernel is), but couldn't they have a local cache system too, where all it has to do it serve up dumb .html pages, needing next to no compute cycles.
turning it into a problem of ROI is one of the stupidest things you can do to prevent bots.
use https://github.com/danieltanfh95/continuity-auth instead
muahahahahah
Sorry, just the thought of it
But when they do… boy do I have a trap waiting for them.
My wife calls me The Genius. I’m the guy she calls when her battery dies or when her instagram breaks like when it shows that random guy in her DMs, stupid bugs LOL
I digress. Alas, when a user lands on my page. My page wants to know exactly 2 things:
1. Why are you here and who are you
And 2. Can you produce a working solution to Pharoah’s Fortune
…those of you aren’t familiar Pharoah’s Fortune is an old chestnut little poem, a riddle if you will I like to ask candidates and so far nobody’s solved it
And the reason nobody has solved it is Pharoah’s Fortune is a very tricky problem. It’s not something you can “solve” per se it’s more like you arrive there.
So far no one has solved it. They all fall for the same trick! It is of course what separates those who write elegant C versus those write poor quality JavaScript.
So I always say to my students to keep an open mind because you never know who - or should I say where you’re talking to.
I’m bookish.
And none of them are smart enough to realize they could just `git clone` all the content and use it offline?
That is super interesting, thank you!
> At that price point, its actually very affordable to many thousands of organizations to get their own copy.
I'm still confused as to who is actually doing it though! Maybe it's affordable to scrape and store, but training a competitive AI model is going to cost much more, right?
IMHO if I visit your website and it intentionally starts wasting my electricity for no other reason than to cost me money, with no opt in, it's hostile and malicious.
I must be missing something. How can using peoples' TVs as bot farms be even remotely legal? Especially when the purpose is to avoid IP blocks?
If so, you could serve iocaine-style bogus pages 1% (say) of the time that:
1. "Look like" real pages to an LLM-less computer (if you get to the point where you have pushed crawlers to use LLMs to detect nonsense, that already increases the cost a lot)
2. Look "obviously wrong" to a human (E.g., you could take some regular text and swap the order of each adjacent pair of words)
3. Are cheap to generate
4. Important: Contain more links than regular pages, on average, and each to an always-bogus page
The idea is that, due to the large number of pages fetched by crawlers, even with a very low "random bogus page rate", like 1%, they will soon unwittingly hit a bogus page, from which point the fraction of their time spent accessing expensive genuine pages will fall exponentially due to the compounding effect of the higher outbound link count on bogus pages. Humans seeing a bogus page will be confused and annoyed, but simply refreshing the page in the browser will solve the problem 99% of the time (and of course the possibility of this happening can be documented, even on the page itself).
The main advantage is that this does not require any IP-based tracking. You could of course decide to apply this only to pages that are already slightly suspicious (e.g., very old commits).
Then again there is the conspiracy theory about cloudflare sponsoring the scrapers
So how do we go from here? Is adding more difficult Anubis and Cloudflare bot protection really the solution? How many millions of human hours and billions in infra costs are we willing to spend on this arms race?
Some approaches that I think are promising:
- A robots.txt V2[0] as a standard way for website owners to state how bots and AI crawlers can use their online content and where to go (e.g. distinguish search from AI training use cases, point to a downloadable file instead of crawling everything, etc.).
- Something like Web Bot Auth[1] as a non-centralized standard for self-identifying bots and agents cryptographically. This would allow websites to allow or deny bots very precisely.
- what else?
[0] https://datatracker.ietf.org/doc/draft-vaughan-machine-reada...
[1] https://datatracker.ietf.org/doc/html/draft-meunier-http-mes...
Ah yes, the stereotypical Linux kernel developer.
For a shopping website or user generated content website, I might understand the terrible load of crawlers that are trying to "steal" the data.
But for the kernel, what's the purpose? Are you that "no human" are seeing your page or its content? Maybe we should investigate more the usage being this "bots".
I don't buy the explanation that there are millions LLM that are constantly trained on redownloaded data from kernel.org. What would be my better guess is that it is not training, but users are actually accessing this content through chatbot and co. Like when you ask why your sound is suddenly not working anymore after an update or why your wifi driver is constantly disconnected after leaving sleep, it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
But actually, regarding this article, I'm quite amazed that with all the advances of the linux kernel, and server softwares, and that the C10k challenge is solved since a long time, still such a basic traffic is such an issue.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html. 14 cpu looks nothing to me. It's like you have 1 iphone and 1 raspberry pi active in a corner of a room. Counting in "seconds" of activities, easily shows meaningless huge numbers. Do you want to know how many breaths I take per year? 8 to 9 millions!
Most certainly, the usage of this shitty Anubis has ruined the climate million times more only with the wasted cpu resources of legit users...
But moreover, by definition the git commits are not supposed to change, ever, so can someone explain to me why the fuck do kernel.org "re-render" the commit to html each time someone is accessing it instead of using a cache or a static version of the html of this commit?
> oh, several BILLION valid URLs you can scrape, only to get 922 duplicates of the same 1.48 million commits Again, reading that, my immediate thinking is that it is a shame that such talented people would not be able to have a proper optimization, so that getting the 922 duplicates are just costing a fraction millisecond more after the first person retrieve the first page.
This has nothing to do with that.
Any Node.JS application will happily accept 100K connections. They'll all wait for the under-resourced database behind it. That application "solved" the C10K challenge, but it's still overwhelmed.
> it might be possible that the "LLM agent" is requesting the commit contents to "understand" or refer or explain them. Is it a bad thing if it helps users?
The article describes random algorithmically-generated traffic arriving in batched waves from laundered residential proxy IP addresses, a few unrelated hits in a group then gone. That's not the pattern you'd see if end users were asking their agents for help.
> it is a shame that such talented people would not be able to have a proper optimization
It's mostly not static content in the sense that you're implying.
Routes that access a single commit can be cached. But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
I'm sure they're already caching their useful-to-real-humans data. As the article said, the vast majority of their traffic is bots hitting those arbitrary, permuted URLs. So whatever cache they're using is probably a) missed almost every time, and b) constantly getting evicted to make room for data served to bots (unless they eschew caching to avoid this--fair--and are thus back to the original issue regardless).
There is no "proper optimization" here. It's not slow to go compute the diff between a random pair of refs, render that into pretty HTML, and serve it. But it costs something more than a cache hit, and doing that dozens-to-hundreds of times a second constantly consumes resources.
> But most of the routes scrapers are hitting are e.g. computing diffs between arbitrary pairs of commits, or other computed-on-the-fly views into history.
Looks like to be wrong based on the article: Today, git.kernel.org receives about 6M daily requests demanding to see random commits.
Also, in "how bad is it", there is a bar graph of the tasks requested. And only diff can't be cached, and it is clearly not the most important one.
> batched waves from laundered residentail proxy IP addresses
Again, I don't see anywhere anything stating that there was "grouped" requests that are directly related, except in something not related to AI: The thing that usually takes us down are not scraper bots, but poorly designed CI systems that try to do something stupid like shallow-clone stable.git from 20 different nodes, all at the same time. (Shallow clones are awful. Run your own damn mirror if you're going to do something nasty like that.)
Just "waves" of requests coming from the same "residential proxy IP". But that makes sense because they might come from a same service without being a single "source" trying to scrap for training.
Let's say that I'm chatgpt, or Claude, and I have a lot of user related requests to do things based on the commits there, each one will be a different task, but they will all go through the same pipe, that would be a proxy with the residential IP if the provider notice major websites like this one blocking requests for bots.
I feel like I'm at work.
We had some web crawler using Selenium to make queries and scrape the data instead of just downloading the whole file.
Every day it seems like we have some people that know just enough to be dangerous creating things like that. And then of course it's our fault that things are slow, or we won't give them infinite system resources, etc
The horror.
> Suddenly, the crawlers were coming from millions of random residential or mobile IPs, all pretending to be random modern browsers. An IP like that would make 4-5 requests and then never show up in the logs again. There was no point in banning them, because by the time you figured out that they were bots, they were already done with you. You just needlessly ballooned your firewall ruleset by adding IPs that would never be back.
Without something like cookies (which are almost certainly tossed after the IP is rotated) or some other persistent identifier, you are stuck have to apply mitigations that scale with the load you're encountering, which means longer challenges for everyone or degraded functionality, like removing some of the fancier cgit features.
Wait what? I never heard of that. I call that a botnet
I'm strongly convinced that these aren't "AI crawlers"; they're just plain DDoSes done by those who have interests in turning the Internet into a dystopian walled garden with "security", and now they have a convenient scapegoat to blame. Don't you find it too coincidental with the rise in identity/age verification and other attempts at silencing free speech on the Internet?
It's widely known there are questions that LLMs can't solve, and once in a while an obvious example appears, so a simple CAPTCHA-like challenge with an HTML-only form would be the logical "defense". Instead there's a huge interest in pushing JS-required proof-of-work (as others have pointed out, these attackers have far more compute than the average user) and remote attestation (there are already providers with huge farms of mobile devices that can defeat this easily).
Things just don't add up.
https://radicle.dev/
Is it foolish of me to have expected more from a blog post on kernel.org?
The predicted consequences have now had time to manifest, and have not done so. This makes the claim either false or overstated. Perhaps there will be issues in the future, but to date there have been many claims that AI development will stall (for a variety of reasons). If they were the critical weaknesses they have been portrayed as, models would not have advanced to the level they are today.
If you have a hypothesis, make a clear prediction based upon it. If you start pushing the date forward after each failed prediction, you end up looking like a hapless doomsday cult. If your hypothesis is correct however, your prediction should actually happen. Then provided you have not made so many predictions to get one right by chance, people will take what you have to say seriously.
Seems like this chaps problem was that the actions were based upon a different hypothesis than the one he stated. X will cause Y by the end of the year is considerably different to X will eventually cause Y.
https://kreepykrauly.co.za/about-us/
Learn how to cache your application and your cpu usage should be almost zero. In fact basically any read should cost nothing in comparison to writes.
The following from Claude: """ A diff between two randomly chosen commits usually spans years of history, so it's not a few KB — the tree itself is ~1.5 GB of text, and a multi-year span rewrites a large slice of it. Call it 100–200 MB per pair on average: 8.5×10¹¹ pairs × ~2×10⁸ bytes ≈ 10²⁰ bytes, or ~150 exabytes """
> Us PHP developers
I can tell.
Imagine if WooCommerce had a route "/product/<sku1>/compare/<sku2>" which displayed an auto-generated comparison between any two product pages.
Now imagine running a million-SKU WooCommerce site, where each product page was 100kb of text.
Now imagine scrapers are permuting those URLs.
How would you cache that?
That's what cgit/kernel.org and many other Git forges are dealing with. These aren't static websites, even if the underlying Git repo is largely static; they're rendering arbitrary diffs and other generated-on-the-fly views into Git history. The ability to do that is a large part of the value of a Git UI.
But if bots a crawling diffs between all possible commits it's crazy.
CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.
“[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".