Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add GoogleOther crawler user agent (#27000) 

Per https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers

> GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.

This commit will ensure it's served the crawler view, and included in crawler metrics
This commit is contained in:
David Taylor 2024-05-13 13:13:04 +01:00 committed by GitHub
parent ebc1763aa5
commit 742a3b138d
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
config/site_settings.yml
View File

@ -1958,7 +1958,7 @@ security:
list_type: compact
crawler_user_agents:
hidden: true
default: "rss|bot|spider|crawler|facebook|archive|wayback|ping|monitor|lighthouse|google-inspectiontool|gptbot|claudebot|anthropic-ai|brightbot"
default: "rss|bot|spider|crawler|facebook|archive|wayback|ping|monitor|lighthouse|google-inspectiontool|gptbot|claudebot|anthropic-ai|brightbot|googleother"
type: list
list_type: compact
browser_update_user_agents: