discourse/lib/crawler_detection.rb

module CrawlerDetection
  # added 'ia_archiver' based on https://meta.discourse.org/t/unable-to-archive-discourse-pages-with-the-internet-archive/21232
  # added 'Wayback Save Page' based on https://meta.discourse.org/t/unable-to-archive-discourse-with-the-internet-archive-save-page-now-button/22875
  # added 'Swiftbot' based on https://meta.discourse.org/t/how-to-add-html-markup-or-meta-tags-for-external-search-engine/28220

  def self.crawler?(user_agent)
    !/Googlebot|Mediapartners|AdsBot|curl|Twitterbot|facebookexternalhit|bingbot|Baiduspider|ia_archiver|Wayback Save Page|360Spider|Swiftbot/.match(user_agent).nil?
  end
end
REFACTOR: Rename `GooglebotDetection` to `CrawlerDetection` because we will likely whitelist more crawlers in the future. 2014-02-20 16:07:02 -05:00			`module CrawlerDetection`
add support for "Save Page Now" archive.org/web 2015-01-06 04:05:45 -05:00			`# added 'ia_archiver' based on https://meta.discourse.org/t/unable-to-archive-discourse-pages-with-the-internet-archive/21232`
			`# added 'Wayback Save Page' based on https://meta.discourse.org/t/unable-to-archive-discourse-with-the-internet-archive-save-page-now-button/22875`
add Swiftbot to crawler regex 2015-05-02 06:18:58 -04:00			`# added 'Swiftbot' based on https://meta.discourse.org/t/how-to-add-html-markup-or-meta-tags-for-external-search-engine/28220`
FEATURE: add 360Spider UA to allow 360 crawl Discourse sites 2015-02-14 09:24:51 -05:00
REFACTOR: Rename `GooglebotDetection` to `CrawlerDetection` because we will likely whitelist more crawlers in the future. 2014-02-20 16:07:02 -05:00			`def self.crawler?(user_agent)`
add Swiftbot to crawler regex 2015-05-02 06:18:58 -04:00			`!/Googlebot\|Mediapartners\|AdsBot\|curl\|Twitterbot\|facebookexternalhit\|bingbot\|Baiduspider\|ia_archiver\|Wayback Save Page\|360Spider\|Swiftbot/.match(user_agent).nil?`
Detect Googlebot from user agent and use a different layout that doesn't load javascript 2014-02-14 17:10:08 -05:00			`end`
			`end`