Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
discourse/app/views/robots_txt/no_index.erb

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

14 lines
488 B
Plaintext
Raw Normal View History

Initial release of Discourse
2013-02-05 14:16:51 -05:00
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
FEATURE: let Google index pages so it can remove them Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag
2020-05-10 22:14:21 -04:00
# Googlebot must be allowed to index so it can remove items from the index
# we return the X-Robots-Tag with noindex, nofollow which will ensure
# indexing is minimized and nothing shows up in Google search results
User-agent: googlebot
Allow: <%= Discourse.base_uri + "/" %>
Disallow: <%= Discourse.base_uri + "/uploads/*" %>
Fix all the errors to get our tests green on Rails 5.1.
2017-08-31 00:06:56 -04:00
User-agent: *
prefix the robots.txt rules with the directory when using subfolder
2018-04-11 16:05:02 -04:00
Disallow: <%= Discourse.base_uri + "/" %>
FEATURE: let Google index pages so it can remove them Google insists on indexing pages so it can figure out if they can be removed from the index. see: https://support.google.com/webmasters/answer/6332384?hl=en This change ensures the we have special behavior for Googlebot where we allow indexing, but block the actual indexing via X-Robots-Tag
2020-05-10 22:14:21 -04:00