Now that we know AI bots will ignore robots.txt and churn residential IP addresses to scrape websites, does anyone know of a method to block them that doesn’t entail handing over your website to Cloudflare?
Now that we know AI bots will ignore robots.txt and churn residential IP addresses to scrape websites, does anyone know of a method to block them that doesn’t entail handing over your website to Cloudflare?
The only way I can think of is blacklisting everything by default, directing to a challanging proper captcha (can be selfhosted) and temporarily whitelisting proven human IPs.
When you try to “enumerate badness” and block all AI useragents and IP ranges, you’ll always leave some new ones through and you’ll never be done with adding them.
Only allow proven humans.
A captcha will inconvenience the users. If you just want to make it worse for the crawlers, let them spend compute ressources through something like https://altcha.org/ (which would still allow them to crawl your site, but make DDoSing very expensive) or AI honeypots.
Any reason you prefer this to mCAPTCHA?
I didn’t know about mCaptcha. Thanks for sharing.
I hadn’t heard of that before, thanks for the link.
I haven’t read through the docs yet… But PoW makes me wonder what the work is and if it’s cryptocurrency related.
Edit: Found it: https://altcha.org/docs/proof-of-work/
Hashcash predates crypto currencies