4 Commits

Author	SHA1	Message	Date
Sam Saffron	bb4e8899c4	FEATURE: let Google index pages so it can remove them ... Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag	2020-05-11 12:15:18 +10:00
Régis Hanol	df7970a6f6	prefix the robots.txt rules with the directory when using subfolder	2018-04-11 22:05:02 +02:00
Guo Xiang Tan	77d4c4d8dc	Fix all the errors to get our tests green on Rails 5.1.	2017-09-25 13:48:58 +08:00
Sam Saffron	c50a9e4d01	added support for disabling indexing by google using SiteSetting.allow_index_in_robots_txt = false	2013-02-11 11:02:57 +11:00