Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

FEATURE: block screaming frog SEO spider by default 

This crawler is not respectful and often can make 1000s of reqs a minute.

Block by default, site operators can amend this default if they want to.
This commit is contained in:
Sam Saffron 2019-11-18 16:47:47 +11:00
parent a9704da34c
commit d28de23ac8
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
config/site_settings.yml
View File

@ -1371,7 +1371,7 @@ security:
list_type: compact
blacklisted_crawler_user_agents:
type: list
default: "mauibot|semrushbot|ahrefsbot|blexbot"
default: "mauibot|semrushbot|ahrefsbot|blexbot|seo spider"
list_type: compact
slow_down_crawler_user_agents:
type: list