Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
discourse/lib/crawler_detection.rb

10 lines
648 B
Ruby
Raw Normal View History

REFACTOR: Rename `GooglebotDetection` to `CrawlerDetection` because we will likely whitelist more crawlers in the future.
2014-02-20 16:07:02 -05:00
module CrawlerDetection
add support for "Save Page Now" archive.org/web
2015-01-06 04:05:45 -05:00
# added 'ia_archiver' based on https://meta.discourse.org/t/unable-to-archive-discourse-pages-with-the-internet-archive/21232
# added 'Wayback Save Page' based on https://meta.discourse.org/t/unable-to-archive-discourse-with-the-internet-archive-save-page-now-button/22875
add Swiftbot to crawler regex
2015-05-02 06:18:58 -04:00
# added 'Swiftbot' based on https://meta.discourse.org/t/how-to-add-html-markup-or-meta-tags-for-external-search-engine/28220
FEATURE: add 360Spider UA to allow 360 crawl Discourse sites
2015-02-14 09:24:51 -05:00
REFACTOR: Rename `GooglebotDetection` to `CrawlerDetection` because we will likely whitelist more crawlers in the future.
2014-02-20 16:07:02 -05:00
def self.crawler?(user_agent)
Add `YandexBot` to our list of crawlers
2016-07-26 13:21:13 -04:00
!/Googlebot|Mediapartners|AdsBot|curl|Twitterbot|facebookexternalhit|bingbot|Baiduspider|ia_archiver|Wayback Save Page|360Spider|Swiftbot|YandexBot/.match(user_agent).nil?
Detect Googlebot from user agent and use a different layout that doesn't load javascript
2014-02-14 17:10:08 -05:00
end
end