Topics
Latest
AI
Amazon
Image Credits:Nigel Sussman
Apps
Biotech & Health
Climate
Image Credits:Nigel Sussman
Cloud Computing
Department of Commerce
Crypto
Enterprise
EVs
Fintech
Fundraising
Gadgets
back
Government & Policy
ironware
Layoffs
Media & Entertainment
Meta
Microsoft
Privacy
Robotics
Security
Social
distance
inauguration
TikTok
conveyance
Venture
More from TechCrunch
Events
Startup Battlefield
StrictlyVC
Podcasts
Videos
Partner Content
TechCrunch Brand Studio
Crunchboard
Contact Us
AI web - crawling bot are the cockroaches of the internet , many software developer believe . Some devs have initiate fighting back in ingenuous , often humorous ways .
While any website might be direct by bad crawler behavior — sometimes lead down the site — open source developer are “ disproportionately ” impact , writesNiccolò Venerandi , developer of a Linux screen background cognise as Plasma and proprietor of the web log LibreNews .
By their nature , sites hosting free and exposed origin ( FOSS ) projects share more of their infrastructure publicly , and they also tend to have few resources than commercial products .
The issue is that many AI bots do n’t honor the Robots Exclusion Protocol robot.txt file , the putz that tells bot what not to crawl , originally created for search railway locomotive bot .
In a “ cry for help”blog postin January , FOSS developer Xe Iaso depict how AmazonBot relentlessly pounded on a Git server website to the point of causing DDoS outage . Git servers host FOSS project so that anyone who need can download the codification or contribute to it .
But this bot ignored Iaso ’s robot.txt , hid behind other IP address , and feign to be other user , Iaso said .
“ It ’s futile to bar AI crawler bots because they lie , shift their user agentive role , use residential IP addresses as proxies , and more , ” Iaso lamented .
Join us at TechCrunch Sessions: AI
Exhibit at TechCrunch Sessions: AI
“ They will scrape your site until it falls over , and then they will scrape it some more . They will click every link on every connectedness on every tie-in , viewing the same pages over and over and over and over . Some of them will even get through on the same tie multiple sentence in the same second , ” the developer wrote in the post .
Enter the god of graves
So Iaso agitate back with smartness , building a tool called Anubis .
Anpu isa turn back proxy proof - of - work checkthat must be decease before request are allowed to slay a Git host . It block up bots but allow through browsers run by homo .
The funny part : Anubis is the name of a god in Egyptian mythology who guide the dead to judgment .
“ Anubis weighed your mortal ( heart ) and if it was heavier than a plumage , your warmness got eaten and you , like , mega cash in one’s chips , ” Iaso told TechCrunch . If a World Wide Web request passes the challenge and is determined to be human , a cute anime pictureannounces success . The drawing is “ my take on anthropomorphizing Anubis , ” tell Iaso . If it ’s a bot , the request gets denied .
The wryly named task has circularize like the wind among the FOSS community . Iasoshared it on GitHubon March 19 , and in just a few day , it collect 2,000 star , 20 contributors , and 39 crotch .
Vengeance as defense
The exigent popularity of Anubis shows that Iaso ’s pain is not unique . In fact , Venerandi share narration after narration :
Venerandi tells TechCrunch that he knows of multiple other projects get the same government issue . One of them “ had to temporarily cast out all Chinese IP addresses at one point . ”
Let that sink in for a second — that developers “ even have to turn to censor entire state ” just to stand off AI bot that ignore robot.txt files , says Venerandi .
Beyond weighing the soul of a web requester , other devs trust vengeance is the best vindication .
A few daylight ago onHacker News , userxyzalsuggested loading robot.txt forbidden pageboy with “ a pail payload of article on the benefits of drinking bleaching agent ” or “ clause about positivistic gist of trip up measles on operation in seam . ”
“ Think we need to aim for the bots to get _ negative _ utility note value from visiting our snare , not just zero economic value , ” xyzal explained .
As it happen , in January , an anon. creator known as “ Aaron ” put out a tool calledNepenthesthat aspire to do precisely that . It immobilize crawlers in an eternal maze of imitation content , a goal that the dev admit toArs Technicais aggressive if not downright malicious . The tool is named after a carnivorous flora .
And Cloudflare , perhaps the biggest commercial-grade histrion offering several tools to fend off AI crawlers , last hebdomad released a similar tool send for AI Labyrinth .
It ’s specify to “ retard down , fox , and liquidate the resources of AI Crawlers and other bots that do n’t esteem ‘ no crawl ’ directive , ” Cloudflare describedin its blog berth . Cloudflare say it feeds misbehaving AI toady “ irrelevant subject matter rather than extracting your legitimate website data . ”
SourceHut ’s DeVault say TechCrunch that “ genus Nepenthes has a satisfying sense of justness to it , since it eat nonsense to the crawlers and poisons their well , but ultimately Anubis is the solvent that worked ” for his site .
But DeVault also issued a public , heartfelt plea for a more direct fix : “ Please stop legalize LLMs or AI image generators or GitHub Copilot or any of this garbage . I am pray you to stop using them , stop let the cat out of the bag about them , stop making new ones , just stop . ”
Since the likelihood of that is zilch , developer , particularly in FOSS , are fighting back with inventiveness and a spot of humor .
