FEATURE: Disallow tags and categories rss feeds

This stops crawlers from hitting tags and category rss feeds to discover
new content, instead they should focus on latest/posts if they need to
consume something regular
This commit is contained in:
Sam 2018-04-11 14:35:54 +10:00
parent 284c946128
commit 489c22d93c
1 changed files with 2 additions and 0 deletions

View File

@ -30,6 +30,8 @@ Disallow: /*?*api_key*
Disallow: /groups
Disallow: /groups/
Disallow: /t/*/*.rss
Disallow: /tags/*.rss
Disallow: /c/*.rss
<% if @disallowed_user_agents %>
<% @disallowed_user_agents.each do |user_agent| %>