FEATURE: remove topic rss from robots
Crawlers love hitting the rss feeds (confirmed that both Google and Bing do) Experimenting with the impact of blocking these feeds and forcing Crawlers to hit the content direct. It is better if they hit the actual page to start with as opposed to 1. Hit RSS feed 2. Find new content 3. Hit post link 4. Get canonical 5. Hit canonical Lots of pointless work. We do not know for sure what impact this will have on newsreader apps, we will listen for feedback.
This commit is contained in:
parent
abf0b1c5bd
commit
f40f10240c
|
@ -29,6 +29,7 @@ Disallow: /*?api_key*
|
|||
Disallow: /*?*api_key*
|
||||
Disallow: /groups
|
||||
Disallow: /groups/
|
||||
Disallow: /t/*/*.rss
|
||||
|
||||
<% if @disallowed_user_agents %>
|
||||
<% @disallowed_user_agents.each do |user_agent| %>
|
||||
|
|
Loading…
Reference in New Issue