Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
discourse/spec/components/crawler_detection_spec.rb

50 lines
3.8 KiB
Ruby
Raw Normal View History

Prepare for separation of RSpec helper files Since rspec-rails 3, the default installation creates two helper files: * `spec_helper.rb` * `rails_helper.rb` `spec_helper.rb` is intended as a way of running specs that do not require Rails, whereas `rails_helper.rb` loads Rails (as Discourse's current `spec_helper.rb` does). For more information: https://www.relishapp.com/rspec/rspec-rails/docs/upgrade#default-helper-files In this commit, I've simply replaced all instances of `spec_helper` with `rails_helper`, and renamed the original `spec_helper.rb`. This brings the Discourse project closer to the standard usage of RSpec in a Rails app. At present, every spec relies on loading Rails, but there are likely many that don't need to. In a future pull request, I hope to introduce a separate, minimal `spec_helper.rb` which can be used in tests which don't rely on Rails.
2015-10-11 05:41:23 -04:00
require 'rails_helper'
REFACTOR: Rename `GooglebotDetection` to `CrawlerDetection` because we will likely whitelist more crawlers in the future.
2014-02-20 16:07:02 -05:00
require_dependency 'crawler_detection'
describe CrawlerDetection do
describe "crawler?" do
FEATURE: flexible crawler detection You can use the crawler user agents site setting to amend what user agents are considered crawlers based on a string match in the user agent Also improves performance of crawler detection slightly
2017-09-28 22:31:50 -04:00
it "can be amended via site settings" do
SiteSetting.crawler_user_agents = 'Mooble|Kaboodle+*'
FEATURE: much improved and simplified crawler detection - phase one does it match 'trident|webkit|gecko|chrome|safari|msie|opera' yes- well it is possibly a browser - phase two does it match 'rss|bot|spider|crawler|facebook|archive|wayback|ping|monitor' probably a crawler then Based off: https://gist.github.com/SamSaffron/6cfad7ea3e6df321ffb7a84f93720a53
2018-01-15 23:41:13 -05:00
expect(CrawlerDetection.crawler?("Mozilla/5.0 Safari (compatible; Kaboodle+*/2.1; +http://www.google.com/bot.html)")).to eq(true)
expect(CrawlerDetection.crawler?("Mozilla/5.0 Safari (compatible; Mooble+*/2.1; +http://www.google.com/bot.html)")).to eq(true)
expect(CrawlerDetection.crawler?("Mozilla/5.0 Safari (compatible; Gooble+*/2.1; +http://www.google.com/bot.html)")).to eq(false)
FEATURE: flexible crawler detection You can use the crawler user agents site setting to amend what user agents are considered crawlers based on a string match in the user agent Also improves performance of crawler detection slightly
2017-09-28 22:31:50 -04:00
end
REFACTOR: Rename `GooglebotDetection` to `CrawlerDetection` because we will likely whitelist more crawlers in the future.
2014-02-20 16:07:02 -05:00
it "returns true for crawler user agents" do
# https://support.google.com/webmasters/answer/1061943?hl=en
few components with rspec3 syntax
2015-01-09 11:34:37 -05:00
expect(described_class.crawler?("Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")).to eq(true)
expect(described_class.crawler?("Googlebot/2.1 (+http://www.google.com/bot.html)")).to eq(true)
expect(described_class.crawler?("Googlebot-News")).to eq(true)
expect(described_class.crawler?("Googlebot-Image/1.0")).to eq(true)
expect(described_class.crawler?("Googlebot-Video/1.0")).to eq(true)
expect(described_class.crawler?("(compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)")).to eq(true)
expect(described_class.crawler?("Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)")).to eq(true)
expect(described_class.crawler?("(compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)")).to eq(true)
expect(described_class.crawler?("Mediapartners-Google")).to eq(true)
expect(described_class.crawler?("AdsBot-Google (+http://www.google.com/adsbot.html)")).to eq(true)
expect(described_class.crawler?("Twitterbot")).to eq(true)
expect(described_class.crawler?("facebookexternalhit/1.1 (+http(s)://www.facebook.com/externalhit_uatext.php)")).to eq(true)
expect(described_class.crawler?("Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)")).to eq(true)
expect(described_class.crawler?("Baiduspider+(+http://www.baidu.com/search/spider.htm)")).to eq(true)
Add `YandexBot` to our list of crawlers
2016-07-26 13:21:13 -04:00
expect(described_class.crawler?("Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)")).to eq(true)
REFACTOR: Rename `GooglebotDetection` to `CrawlerDetection` because we will likely whitelist more crawlers in the future.
2014-02-20 16:07:02 -05:00
end
it "returns false for non-crawler user agents" do
few components with rspec3 syntax
2015-01-09 11:34:37 -05:00
expect(described_class.crawler?("Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36")).to eq(false)
expect(described_class.crawler?("Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko")).to eq(false)
expect(described_class.crawler?("Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0)")).to eq(false)
expect(described_class.crawler?("Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25")).to eq(false)
expect(described_class.crawler?("Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0")).to eq(false)
expect(described_class.crawler?("Mozilla/5.0 (Linux; U; Android 4.0.3; ko-kr; LG-L160L Build/IML74K) AppleWebkit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30")).to eq(false)
FEATURE: much improved and simplified crawler detection - phase one does it match 'trident|webkit|gecko|chrome|safari|msie|opera' yes- well it is possibly a browser - phase two does it match 'rss|bot|spider|crawler|facebook|archive|wayback|ping|monitor' probably a crawler then Based off: https://gist.github.com/SamSaffron/6cfad7ea3e6df321ffb7a84f93720a53
2018-01-15 23:41:13 -05:00
expect(described_class.crawler?("DiscourseAPI Ruby Gem 0.19.0")).to eq(true)
expect(described_class.crawler?("Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)")).to eq(true)
expect(described_class.crawler?("LogicMonitor SiteMonitor/1.0")).to eq(true)
expect(described_class.crawler?("Java/1.8.0_151")).to eq(true)
expect(described_class.crawler?("Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)")).to eq(true)
REFACTOR: Rename `GooglebotDetection` to `CrawlerDetection` because we will likely whitelist more crawlers in the future.
2014-02-20 16:07:02 -05:00
end
end
end