discourse/app/views/robots_txt/index.erb

<%= @robots_info[:header] %>
<% if Discourse.base_path.present? %>
# This robots.txt file is not used. Please append the content below in the robots.txt file located at the root
<% end %>
#
<% @robots_info[:agents].each do |agent| %>
User-agent: <%= agent[:name] %>
<%- if agent[:delay] -%>
Crawl-delay: <%= agent[:delay] %>
<%- end -%>
<% agent[:disallow].each do |path| %>
Disallow: <%= path %>
<% end %>


<% end %>

<%= server_plugin_outlet "robots_txt_index" %>
FEATURE: An API to help sites build robots.txt files programatically This is mainly useful for subfolder sites, who need to expose their robots.txt contents to a parent site. 2018-04-16 15:43:20 -04:00			`<%= @robots_info[:header] %>`
Replace `base_uri` with `base_path` (#10879) DEV: Replace instances of Discourse.base_uri with Discourse.base_path This is clearer because the base_uri is actually just a path prefix. This continues the work started in 555f467. 2020-10-09 07:51:24 -04:00			`<% if Discourse.base_path.present? %>`
add a warning in robots.txt when using subfolder 2018-04-11 18:00:15 -04:00			`# This robots.txt file is not used. Please append the content below in the robots.txt file located at the root`
			`<% end %>`
Initial release of Discourse 2013-02-05 14:16:51 -05:00			`#`
FEATURE: An API to help sites build robots.txt files programatically This is mainly useful for subfolder sites, who need to expose their robots.txt contents to a parent site. 2018-04-16 15:43:20 -04:00			`<% @robots_info[:agents].each do \|agent\| %>`
			`User-agent: <%= agent[:name] %>`
			`<%- if agent[:delay] -%>`
			`Crawl-delay: <%= agent[:delay] %>`
			`<%- end -%>`
			`<% agent[:disallow].each do \|path\| %>`
			`Disallow: <%= path %>`
prefix the robots.txt rules with the directory when using subfolder 2018-04-11 16:05:02 -04:00			`<% end %>`
new: server plugin outlet for indexable robots.txt 2017-02-13 07:01:10 -05:00
FEATURE: control which web crawlers can access using a whitelist or blacklist 2018-03-15 17:10:45 -04:00
			`<% end %>`

new: server plugin outlet for indexable robots.txt 2017-02-13 07:01:10 -05:00			`<%= server_plugin_outlet "robots_txt_index" %>`