FEATURE: Disallow tags and categories rss feeds
This stops crawlers from hitting tags and category rss feeds to discover new content, instead they should focus on latest/posts if they need to consume something regular
This commit is contained in:
parent
284c946128
commit
489c22d93c
|
@ -30,6 +30,8 @@ Disallow: /*?*api_key*
|
|||
Disallow: /groups
|
||||
Disallow: /groups/
|
||||
Disallow: /t/*/*.rss
|
||||
Disallow: /tags/*.rss
|
||||
Disallow: /c/*.rss
|
||||
|
||||
<% if @disallowed_user_agents %>
|
||||
<% @disallowed_user_agents.each do |user_agent| %>
|
||||
|
|
Loading…
Reference in New Issue