Sam 3a7b696703 FEATURE: allow for setting crawl delay per user agent ... Also moved to default crawl delay bing so no more than a req every 5 seconds is allowed New site settings: "slow_down_crawler_user_agents" - list of crawlers that will be slowed down "slow_down_crawler_rate" - how many seconds to wait between requests Not enforced server side yet		2018-04-06 10:15:23 +10:00
..
index.erb	FEATURE: allow for setting crawl delay per user agent	2018-04-06 10:15:23 +10:00
no_index.erb	Fix all the errors to get our tests green on Rails 5.1.	2017-09-25 13:48:58 +08:00