FEATURE: remove topic rss from robots

Crawlers love hitting the rss feeds (confirmed that both Google and Bing do)

Experimenting with the impact of blocking these feeds and forcing Crawlers to hit
the content direct. It is better if they hit the actual page to start with as opposed to

1. Hit RSS feed
2. Find new content
3. Hit post link
4. Get canonical
5. Hit canonical

Lots of pointless work.

We do not know for sure what impact this will have on newsreader apps,
we will listen for feedback.
This commit is contained in:
Sam 2018-04-11 11:57:52 +10:00
parent abf0b1c5bd
commit f40f10240c
1 changed files with 1 additions and 0 deletions

View File

@ -29,6 +29,7 @@ Disallow: /*?api_key*
Disallow: /*?*api_key*
Disallow: /groups
Disallow: /groups/
Disallow: /t/*/*.rss
<% if @disallowed_user_agents %>
<% @disallowed_user_agents.each do |user_agent| %>