Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

per spec we need to repeat disallow paths per agent

This commit is contained in:
Sam 2018-04-16 15:38:10 +10:00
parent a1ef455c78
commit 223379e21a
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
app/views/robots_txt/index.erb
View File

@ -23,4 +23,8 @@ Disallow: <%= Discourse.base_uri + "/" %>
<% @crawler_delayed_agents.each do |agent, delay| %> <% @crawler_delayed_agents.each do |agent, delay| %>
User-agent: <%= agent %> User-agent: <%= agent %>
Crawl-delay: <%= delay %> Crawl-delay: <%= delay %>
<% RobotsTxtController::DISALLOWED_PATHS.each do |path| %>
Disallow: <%= Discourse.base_uri + path %>
<% end %>
<% end %> <% end %>