discourse/app/views/robots_txt/index.erb

<%= @robots_info[:header] %>
<% if Discourse.base_uri.present? %>
# This robots.txt file is not used. Please append the content below in the robots.txt file located at the root
<% end %>
#
<% @robots_info[:agents].each do |agent| %>
User-agent: <%= agent[:name] %>
<%- if agent[:delay] -%>
Crawl-delay: <%= agent[:delay] %>
<%- end -%>
<% agent[:disallow].each do |path| %>
Disallow: <%= path %>
Noindex: <%= path %>
<% end %>


<% end %>

<%= server_plugin_outlet "robots_txt_index" %>
FEATURE: An API to help sites build robots.txt files programatically This is mainly useful for subfolder sites, who need to expose their robots.txt contents to a parent site. 2018-04-16 15:43:20 -04:00			`<%= @robots_info[:header] %>`
add a warning in robots.txt when using subfolder 2018-04-11 18:00:15 -04:00			`<% if Discourse.base_uri.present? %>`
			`# This robots.txt file is not used. Please append the content below in the robots.txt file located at the root`
			`<% end %>`
Initial release of Discourse 2013-02-05 14:16:51 -05:00			`#`
FEATURE: An API to help sites build robots.txt files programatically This is mainly useful for subfolder sites, who need to expose their robots.txt contents to a parent site. 2018-04-16 15:43:20 -04:00			`<% @robots_info[:agents].each do \|agent\| %>`
			`User-agent: <%= agent[:name] %>`
			`<%- if agent[:delay] -%>`
			`Crawl-delay: <%= agent[:delay] %>`
			`<%- end -%>`
			`<% agent[:disallow].each do \|path\| %>`
			`Disallow: <%= path %>`
FEATURE: add Noindex to robots.txt for disallowed routes This strips pages out of indexes that should not exist see: https://meta.discourse.org/t/pages-listed-in-the-robots-txt-are-crawled-and-indexed-by-google/100309/11?u=sam 2018-11-02 01:39:47 -04:00			`Noindex: <%= path %>`
prefix the robots.txt rules with the directory when using subfolder 2018-04-11 16:05:02 -04:00			`<% end %>`
new: server plugin outlet for indexable robots.txt 2017-02-13 07:01:10 -05:00
FEATURE: control which web crawlers can access using a whitelist or blacklist 2018-03-15 17:10:45 -04:00
			`<% end %>`

new: server plugin outlet for indexable robots.txt 2017-02-13 07:01:10 -05:00			`<%= server_plugin_outlet "robots_txt_index" %>`