Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

add alt 'archive.org_bot' user-agent (#5273) 

add `archive.org_bot' – another user-agent used by Internet Archive when crawling for Wayback Machine
This commit is contained in:
Gordon Mohr 2017-10-29 02:30:29 -07:00 committed by Régis Hanol
parent bf00ab5d4a
commit da72ecc3fc
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
config/site_settings.yml
View File

@ -929,7 +929,7 @@ security:
allow_index_in_robots_txt: true
allow_moderators_to_create_categories: false
crawler_user_agents:
default: 'Googlebot|Mediapartners|AdsBot|curl|HTTrack|Twitterbot|facebookexternalhit|bingbot|Baiduspider|ia_archiver|Wayback Save Page|360Spider|Swiftbot|YandexBot'
default: 'Googlebot|Mediapartners|AdsBot|curl|HTTrack|Twitterbot|facebookexternalhit|bingbot|Baiduspider|ia_archiver|archive.org_bot|Wayback Save Page|360Spider|Swiftbot|YandexBot'
type: list
cors_origins:
default: ''