Martin Brennan c791346088
FEATURE: Rate limit common AI bots crawlers by default (#26731)
This commit adds the most common AI bot crawlers seen
on our hosting (claudebot, gptbot, anthropic-ai, brightbot)
to our `slow_down_crawler_user_agents` and `crawler_user_agents`
site settings by default.

This means these AI bots will be rate limited by default instead
of site admins having to remember to do it for themselves.
2024-04-24 12:01:42 +10:00
..
2024-04-23 16:22:27 +02:00