Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

FEATURE: Rate limit common AI bots crawlers by default (#26731) 

This commit adds the most common AI bot crawlers seen
on our hosting (claudebot, gptbot, anthropic-ai, brightbot)
to our `slow_down_crawler_user_agents` and `crawler_user_agents`
site settings by default.

This means these AI bots will be rate limited by default instead
of site admins having to remember to do it for themselves.
This commit is contained in:
Martin Brennan 2024-04-24 12:01:42 +10:00 committed by GitHub
parent d2a52c3c35
commit c791346088
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
config/site_settings.yml
View File

@ -1958,7 +1958,7 @@ security:
list_type: compact
crawler_user_agents:
hidden: true
default: "rss|bot|spider|crawler|facebook|archive|wayback|ping|monitor|lighthouse|google-inspectiontool"
default: "rss|bot|spider|crawler|facebook|archive|wayback|ping|monitor|lighthouse|google-inspectiontool|gptbot|claudebot|anthropic-ai|brightbot"
type: list
list_type: compact
browser_update_user_agents:
@ -2006,7 +2006,7 @@ security:
list_type: compact
slow_down_crawler_user_agents:
type: list
default: ""
default: "gptbot|claudebot|anthropic-ai|brightbot"
list_type: compact
slow_down_crawler_rate: 60
content_security_policy: