FIX: add X-Robots-Tag header for check_xhr-covered GET actions, too (#9868)

* FIX: add X-Robots-Tag header for check_xhr-covered GET actions, too

see https://meta.discourse.org/t/missing-x-robots-tag/152593/3 for context

* test: a spec making sure X-Robots-Tag header is present when needed

/groups path responds to anonymous requests and doesn't skip `check_xhr` method, so we can use it here.

app/controllers/application_controller.rb

@@ -39,12 +39,12 @@ class ApplicationController < ActionController::Base
   before_action :redirect_to_login_if_required
   before_action :block_if_requires_login
   before_action :preload_json
+  before_action :add_noindex_header, if: -> { is_feed_request? || !SiteSetting.allow_index_in_robots_txt }
   before_action :check_xhr
   after_action  :add_readonly_header
   after_action  :perform_refresh_session
   after_action  :dont_cache_page
   after_action  :conditionally_allow_site_embedding
-  after_action  :add_noindex_header, if: -> { is_feed_request? || !SiteSetting.allow_index_in_robots_txt }
 
   layout :set_layout
 

spec/requests/groups_controller_spec.rb

@@ -176,6 +176,12 @@ describe GroupsController do
       )
     end
 
+    it 'should return correct X-Robots-Tag header when allow_index_in_robots_txt is set to false' do
+      SiteSetting.allow_index_in_robots_txt = false
+      get "/groups"
+      expect(response.headers['X-Robots-Tag']).to eq('noindex, nofollow')
+    end
+
     context 'viewing groups of another user' do
       describe 'when an invalid username is given' do
         it 'should return the right response' do