I’m trying to work out if any AI bots have been crawling my site. I find it remarkable that thirty years of responsible crawling has been discarded by AI companies.

As Dries Buytaert found, not even following their recommendations help, it doubles the number of pages they crawl.