PERF: Run multiple threads for regular job schedules

Under extreme load on large databases certain regular jobs can take quite
a while to run. We need to ensure we never starve a sidekiq from running
mini scheduler, cause without it we are unable to queue stuff such as
heartbeat jobs.

Gemfile.lock

@@ -186,7 +186,7 @@ GEM
     mini_portile2 (2.4.0)
     mini_racer (0.2.6)
       libv8 (>= 6.9.411)
-    mini_scheduler (0.11.0)
+    mini_scheduler (0.12.0)
       sidekiq
     mini_sql (0.2.2)
     mini_suffix (0.3.0)
@@ -274,7 +274,7 @@ GEM
     rack-openid (1.3.1)
       rack (>= 1.1.0)
       ruby-openid (>= 2.1.8)
-    rack-protection (2.0.5)
+    rack-protection (2.0.7)
       rack
     rack-test (1.1.0)
       rack (>= 1.0, < 3)

config/discourse_defaults.conf

@@ -259,3 +259,10 @@ enable_performance_http_headers = false
 # gather JavaScript errors from clients (rate limited to 1 error per IP per minute)
 enable_js_error_reporting = true
 
+# This is probably not a number you want to touch, it controls the number of workers
+# we allow mini scheduler to run. Prior to 2019 we ran a single worker.
+# On extremely busy setups this could lead to situations where regular jobs would
+# starve. Specifically jobs such as "run heartbeat" which keeps sidekiq running.
+# Having a high number here is very low risk. Regular jobs are limited in scope and scale.
+mini_scheduler_workers = 5
+

config/initializers/100-sidekiq.rb

@@ -62,7 +62,7 @@ if Sidekiq.server?
     scheduler_hostname = ENV["UNICORN_SCHEDULER_HOSTNAME"]
 
     if !scheduler_hostname || scheduler_hostname.split(',').include?(`hostname`.strip)
-      MiniScheduler.start
+      MiniScheduler.start(workers: GlobalSetting.mini_scheduler_workers)
     end
   end
 end