FIX: Include HTML content for authenticated users with crawler UA (#26757)

e05628c0 introduced an optimization to remove basic-HTML content for authenticated users. The assumption is that, if they were able to log in, they must have a JS capable browser and do not need the basic HTML.

However, there are use-cases where an API-key is used to crawl a private site, or private categories of a public site. This commit re-enables those use cases by keeping the basic-html in place for crawler/bot user agents.

app/helpers/application_helper.rb

@@ -431,7 +431,7 @@ module ApplicationHelper
   end
 
   def include_crawler_content?
-    if current_user
+    if current_user && !crawler_layout?
       params.key?(:print)
     else
       crawler_layout? || !mobile_view? || !modern_mobile_device?

spec/helpers/application_helper_spec.rb

@@ -19,11 +19,19 @@ RSpec.describe ApplicationHelper do
 
     it "sends crawler content to logged on users who wants to print" do
       helper.stubs(:current_user).returns(Fabricate(:user))
+      controller.stubs(:use_crawler_layout?).returns(false)
       helper.stubs(:params).returns(print: true)
 
       expect(helper.include_crawler_content?).to eq(true)
     end
 
+    it "sends crawler content to logged on users with a crawler user agent" do
+      helper.stubs(:current_user).returns(Fabricate(:user))
+      controller.stubs(:use_crawler_layout?).returns(true)
+
+      expect(helper.include_crawler_content?).to eq(true)
+    end
+
     it "sends crawler content to old mobiles" do
       controller.stubs(:use_crawler_layout?).returns(false)