Commit Graph

586 Commits

Author SHA1 Message Date
Krzysztof Kotlarek f84ff26aa9
FIX: use Redis replica host and port (#11566)
Introduce Redis `replica` config and deprecate `slave`
2020-12-23 13:14:19 +11:00
Krzysztof Kotlarek 07bf7a91f4
Revert "FIX: use Redis replica host and port (#11543)" (#11552)
This reverts commit b0e1210b0c.
2020-12-22 16:16:50 +11:00
Krzysztof Kotlarek b0e1210b0c
FIX: use Redis replica host and port (#11543)
* FIX: use Redis replica host and port

Introduce Redis `replica` config and deprecate `slave`

* FIX: move deprecations to separate file
2020-12-22 15:52:00 +11:00
David Taylor 5289fc7886
FIX: Improve failover for multisite clusters (#11150)
- Bump rails_failover for new per-backend callback feature
- If the master backend fails over, make all sites readonly. And vice-versa for fallback
- If a single backend fails over, make that individual site readonly. And vice-versa for fallback
- When a single backend fails, also check connection to the master backend
2020-11-11 10:27:24 +00:00
Osama Sayegh 84a1f15791
DEV: Track redis calls count in mini profiler (#11088) 2020-10-30 19:11:22 +03:00
Vinoth Kannan af4938baf1
Revert "DEV: enable cors to all cdn get requests from workbox. (#10684)" (#11076)
This reverts commit e3de45359f.

We need to improve out strategy by adding a cache breaker with this change ... some assets on CDNs and clients may have incorrect CORS headers which can cause stuff to break.
2020-10-30 16:05:35 +11:00
Vinoth Kannan 72810853ea
FIX: strip the trailing slash (/) of cors origins. (#10996)
Strips trailing `/` from global settings
Provides a validation for site settings to ensure a trailing `/` is not added
2020-10-29 13:01:06 +11:00
Vinoth Kannan e3de45359f
DEV: enable cors to all cdn get requests from workbox. (#10685)
Now all external requests from the service worker will be in CORS mode without credentials.
2020-10-28 23:36:19 +05:30
Osama Sayegh a04c300495
DEV: Add optional ENV variables for MiniProfiler snapshots transporter (#10985) 2020-10-21 19:37:28 +03:00
Guo Xiang Tan cabbc32024 FIX: Scheduled jobs not showing up in /sidekiq/schedule in dev. 2020-10-08 10:24:51 +08:00
Penar Musaraj a4356b99af
FEATURE: Separate base and heading font site_settings (#10807)
Allows site administrators to pick different fonts for headings in the wizard and in their site settings. Also correctly displays the header logos in wizard previews.
2020-10-05 13:40:41 -04:00
David Taylor ce686a008f
DEV: Refresh site settings on Redis fallback (#10700)
This ensures that all app instances have up-to-date data, even if they missed a message-bus update during the failover
2020-09-18 11:44:25 +01:00
Guo Xiang Tan d83d9ec408
DEV: Clear all `DistributedCache` on Redis fallback. 2020-09-18 11:31:49 +08:00
Osama Sayegh a92d88747e
DEV: Add ENV variable for enabling MiniProfiler snapshots (#10690)
* DEV: Add ENV variable for enabling MiniProfiler snapshots

* MiniProfiler is not loaded in test env
2020-09-17 18:18:35 +03:00
Daniel Waterworth 6e69c25134 FIX: Use asset_hostnames in rails_multisite
We want to respond to the x-forwarded-for host from the CDN too.
2020-09-17 10:32:43 +01:00
Penar Musaraj 6610576deb
FIX: Clear stylesheet cache on base font change
Previously, we were not clearing the cache on themes using the default
color scheme.
2020-09-15 13:13:09 -04:00
Daniel Waterworth 19b24e178c SECURITY: __ws shouldn't be able to override every domain in multisite 2020-09-15 17:35:57 +01:00
Bianca Nenciu 58b97ace23
DEV: Use a special import to declare font faces (#10583)
Update discourse-fonts to v0.0.3.

Follow-up to 7b7357147e.
2020-09-04 16:25:50 +03:00
Guo Xiang Tan 069a109cbb
DEV: Require scheduled job in development to avoid loading file twice.
This removes the need to memoize constant in order to avoid the "warning: already initialized constant".
2020-09-01 10:14:40 +08:00
Guo Xiang Tan 09a89cff1a
DEV: Increase log levels for more assets/static paths. 2020-09-01 09:52:50 +08:00
Guo Xiang Tan 20bd6d9797
DEV: Only eagerload scheduled jobs in a Sidekiq process. 2020-08-31 14:28:38 +08:00
Guo Xiang Tan 23778f4bfd
DEV: Avoid logging Redis cannot connect error during failover.
Reduce amount of noise in our logs.
2020-07-21 14:53:46 +08:00
Guo Xiang Tan df66a559c9
DEV: Pause Sidekiq before readonly mode during PG failover. 2020-07-21 13:51:37 +08:00
Guo Xiang Tan 031a6616a3
DEV: Allow writes to PG even if Redis is readonly mode.
Previously, I thought it was better to drop the site into reading mode
when Redis has failed over to the replica but it created more errors
while Redis is in readonly mode since ActiveRecord would prevent us from
writing to PG even though PG is up.
2020-07-21 12:34:07 +08:00
Guo Xiang Tan 383537a2fa
DEV: Force ActiveRecord reading role on Redis failover take 2. 2020-07-20 16:11:20 +08:00
Guo Xiang Tan 5adf2ccaca
DEV: Force ActiveRecord reading role on Redis failover. 2020-07-20 15:10:53 +08:00
Guo Xiang Tan 54d002f7db
DEV: Fix error introduced in d5c56a846. 2020-07-13 10:13:37 +08:00
Guo Xiang Tan d5c56a846a
DEV: Only failover the entire cluster when the default db goes down. 2020-07-09 11:49:03 +08:00
Guo Xiang Tan fd38c2fac3
FIX: Force ActiveRecord reading role if Redis is down take 2.
follow-up f03c7a1ba1
2020-07-09 11:14:19 +08:00
Guo Xiang Tan f03c7a1ba1
FIX: Force ActiveRecord reading role if Redis is down. 2020-07-09 11:13:02 +08:00
Guo Xiang Tan 82964265cc
DEV: Remove logster current context config.
Multisite middleware sits at the top of the middleware stack.
2020-07-01 11:44:22 +08:00
Guo Xiang Tan 2c4c953bf8
DEV: Avoid logging errors on bad Redis connection during PG failover. 2020-06-29 11:54:55 +08:00
Guo Xiang Tan 42a6c8a85f
DEV: Rescue from ActiveRecord::Readonly error in lograge. 2020-06-25 10:25:28 +08:00
Guo Xiang Tan 27b2e335ef
DEV: Retry on distributed mutex timeout error when starting sidekiq.
We need Sidekiq to start `mini_scheduler` no matter what. Timeouts
happen when trying to boot an app with Redis in readonly mode.
2020-06-23 15:43:28 +08:00
Guo Xiang Tan def4cd33ea
DEV: Disable Redis warnings. 2020-06-16 12:53:04 +08:00
Guo Xiang Tan c611f3703c
DEV: Don't use logster when logging in Redis failover. 2020-06-16 11:53:52 +08:00
Guo Xiang Tan b08a0d15c4
DEV: Fix undefined method due to rails_failover. 2020-06-16 11:03:57 +08:00
Guo Xiang Tan 092ae858af
DEV: Bump rails_failover.
Avoid configuring AR stuff if `replica_host` and `replica_port` hasn't
been provided.
2020-06-16 10:51:21 +08:00
Guo Xiang Tan 402b80f306
DEV: Make rails_failover compatible with `SKIP_DB_AND_REDIS` env. 2020-06-15 16:23:24 +08:00
Guo Xiang Tan e0d798c06c
DEV: Fix undefined method. 2020-06-15 16:04:41 +08:00
Guo Xiang Tan e0fdf41537
DEV: Don't configure rails_failover is db and redis is skipped. 2020-06-15 15:56:57 +08:00
Guo Xiang Tan 58e52c0e4f
DEV: Use rails_failover gem for ActiveRecord and Redis failover handling 2020-06-15 15:47:07 +08:00
Guo Xiang Tan 0ff86b00cb
DEV: Upgrade Redis to 4.2.1. 2020-06-15 10:05:22 +08:00
Guo Xiang Tan c9964b95ce
DEV: Increase log level for `/srv/status` route.
This reduces the amount of noise in our logs.
2020-06-12 12:17:28 +08:00
Guo Xiang Tan 78b5ab746c
DEV: No longer need to clear anon cache when toggling readonly mode. 2020-06-12 09:58:17 +08:00
Guo Xiang Tan dc4071dfef
DEV: Use `Rails.logger` instead of logster for rails_failover callbacks
`Discourse.warn_exception` logs to logger by default but it means we
lose all the backtrace when the logs are written to the log file.
2020-06-11 17:24:32 +08:00
Guo Xiang Tan 1411b095eb
DEV: Rescue errors when runngin AR failover/fallback callbacks. 2020-06-11 13:02:42 +08:00
Guo Xiang Tan 34ee1f2e71
DEV: Fix undefined method in rails_failover initializer. 2020-06-11 12:30:01 +08:00
Guo Xiang Tan cbb4ea1ea8
DEV: Don't blow up request redis can't be reached during failover. 2020-06-11 11:41:06 +08:00
Guo Xiang Tan a3dfd553a1
Revert "Bump redis to 4.2.0."
This reverts commit 98bc28cea2.
2020-06-10 14:52:05 +08:00
Guo Xiang Tan 98bc28cea2
Bump redis to 4.2.0. 2020-06-10 14:28:56 +08:00
Guo Xiang Tan 2ce829cc01
DEV: Allow multisite apps to boot with reading connection handler. 2020-06-10 14:18:29 +08:00
Guo Xiang Tan a1c13eb3c6
DEV: Redis failover should only clear redis recently readonly. 2020-06-09 16:36:31 +08:00
Guo Xiang Tan f6628e4f43
DEV: Disable messageBus keepalive when Redis fails over. 2020-06-08 12:33:08 +08:00
Guo Xiang Tan e82d4d8a75
DEV: Update rails_failover to avoid monkey patching Rails config. 2020-06-05 09:05:19 +08:00
Guo Xiang Tan 87673e6571
DEV: Fix moving of rails_failover middleware. 2020-06-04 20:29:47 +08:00
Guo Xiang Tan aaece34e8b DEV: Update rails_failover so that we can move middleware up the stack. 2020-06-04 17:14:13 +08:00
Guo Xiang Tan 54f79ea3ba DEV: Pausing Sidekiq is multisite aware. 2020-06-04 15:46:30 +08:00
Guo Xiang Tan 8e1681d356 Bump rails_failover. 2020-06-04 15:22:35 +08:00
Guo Xiang Tan 439db7ca1e
DEV: Add `REDIS_RAILS_FAILOVER` env to test our new redis failover. 2020-06-02 17:24:14 +08:00
Guo Xiang Tan ade60b0cbc
DEV: Enable readonly mode for all multisite sites when PG goes down.
The risk here is that the database for one site goes down in the multisite setup and we drop everything to readonly mode. However, I discussed this with Sam and we agree that one database having problem is very rare. Most of the time, it is the entire DB cluster that goes down.
2020-06-02 11:32:07 +08:00
Guo Xiang Tan 326d6d5b0f
DEV: Pause Sidekiq when forcing pg readonly mode. 2020-06-02 09:20:03 +08:00
Guo Xiang Tan 32735be5bd
DEV: Fix publish to message_bus when forcing pg readonly. 2020-06-01 13:26:12 +08:00
Guo Xiang Tan 2c3ff3e524
DEV: Missing readonly mode banner when forcing PG readonly. 2020-06-01 12:25:27 +08:00
Guo Xiang Tan b0b37bf5a3
DEV: Add force rails_over switch via Redis. 2020-06-01 11:23:58 +08:00
Guo Xiang Tan 8c86a109bb
DEV: Add ENV flag to test out `ActiveRecord::Failover`. 2020-05-28 16:24:22 +08:00
Guo Xiang Tan 878f06f1fe DEV: Remove custom connection reaper.
Rails 6 fixed the reaper to use one thread to reap all the connection pools.
2020-05-26 09:09:46 +08:00
Michael Brown d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood 20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Guo Xiang Tan 2f03a879f9
DEV: Require `rails_failover` before global settings. 2020-05-20 16:06:40 +08:00
Guo Xiang Tan f7f436e536
DEV: Install `rails_failover` gem to test our Redis changes. 2020-05-20 15:40:27 +08:00
Guo Xiang Tan 96c02caba7
DEV: Change use of Redis `flushall` to `flushdb`.
FLUSHALL removes all keys from all databases. Instead we only want to
remove keys from the current Redis database.
2020-05-19 10:20:00 +08:00
Sam Saffron 609e929186
Revert "Revert "DEV: upgrade to Rails 6.0.3""
This reverts commit 2ff8b4f5d9.

Attempt #2 at a Rails update this time we also update the
rails_multisite gem to allow for cleaner reordering
2020-05-08 11:49:22 +10:00
David Taylor 2ff8b4f5d9
Revert "DEV: upgrade to Rails 6.0.3"
This was causing issues during multisite:migrate

https://meta.discourse.org/t/multisite-migrate-broken-since-rails-6-0-3-update/150691

This reverts commit 136a545653.
2020-05-07 11:44:39 +01:00
Sam Saffron 136a545653
DEV: upgrade to Rails 6.0.3
Upgrades Rails to latest, this version has better compatibility
with Ruby 2.7

During the upgrade we needed a new cleaner mechanism for configuring
message bus.

All tests are green.

If anything weird pops up please revert.
2020-05-07 15:53:40 +10:00
Sam Saffron d0d5a138c3
DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
Jarek Radosz 17cf300b71
DEV: Use more specific error responses (#9472)
* DEV: Use `render_json_error` (Adds specs for Admin::GroupsController)
* DEV: Use a specific error on blank category slug (Fixes a `render_json_error` warning)
* DEV: Use a specific error on reviewable claim conflict (Fixes a `render_json_error` warning)
* DEV: Use specific errors in Admin::UsersController (Fixes `render_json_error` warnings)
* FIX: PublishedPages error responses
* FIX: TopicsController error responses (There was an issue of two separate `Topic` instances for the same record. This makes sure there's only one up-to-date instance.)
2020-04-21 03:50:20 +02:00
Dan Ungureanu 3d9c320aab
PERF: Cache Category.subcategory_ids (#9350)
Also reset category cache after backup restore.
2020-04-09 15:42:24 +03:00
Kane York 138d4aebde FEATURE: Webhooks and Event for user being granted a badge
Adding a webhook for badge revocation is left for future work as it's relatively rare.
2020-04-08 14:38:30 -07:00
Daniel Waterworth bca126f3f5 REFACTOR: Move the multisite middleware to the front
Both request tracking and message bus rely on multisite before the
middleware has run which is not ideal.

Follow-up-to: ca1208a636
2020-04-02 16:44:44 +01:00
Daniel Waterworth ca1208a636 Revert "REFACTOR: Move the multisite middleware to the front"
Looks like this is causing problems.

Follow-up-to: a91843f0dc
2020-04-02 15:20:28 +01:00
Daniel Waterworth a91843f0dc REFACTOR: Move the multisite middleware to the front
Both request tracking and message bus rely on multisite before the
middleware has run which is not ideal.
2020-04-02 10:15:38 +01:00
Sam Saffron 25f1f23288
FEATURE: Stricter rules for user presence
Previously we would consider a user "present" and "last seen" if the
browser window was visible.

This has many edge cases, you could be considered present and around for
days just by having a window open and no screensaver on.

Instead we now also check that you either clicked, transitioned around app
or scrolled the page in the last minute in combination with window
visibility

This will lead to more reliable notifications via email and reduce load of
message bus for cases where a user walks away from the terminal
2020-03-26 17:36:52 +11:00
Martin Brennan 097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00
David Taylor fb8b0ca197
DEV: Remove unused middleware (#9203)
This has not been used since January 2014, per 166a8d29
2020-03-16 12:37:43 +00:00
Gerhard Schlager 8022e51179 FIX: Failed to restore backups from versions without translation overrides
Rails calls I18n.translate during initialization and by default translation overrides are used. Database migrations would fail if the system tried to migrate from an old version that didn't have the `translation_overrides` table with all its columns yet.

This makes restoring really old backups work again. Running `DISABLE_TRANSLATION_OVERRIDES=1 rake db:migrate` will allow you to upgrade such an old database as well.
2020-03-14 00:00:22 +01:00
Robin Ward a3f0543f99
Support for transpiling `.js` files (#9160)
* Remove some `.es6` from comments where it does not matter

* Use a post processor for transpilation

This will allow us to eventually use the directory structure to
transpile rather than the extension.

* FIX: Some errors and clean up in confirm-new-email

It would throw an error if the webauthn element wasn't present.
Also I changed things so that no-module is not explicitly
referenced.

* Remove `no-module`

Instead we allow a magic comment: `// discourse-skip-module` to prevent
the asset pipeline from creating a module.

* DEV: Enable babel transpilation based on directory

If it's in `app/assets/javascripts/dicourse` it will be transpiled
even without the `.es6` extension.

* REFACTOR: Remove Tilt/ES6ModuleTranspiler
2020-03-11 09:43:55 -04:00
Roman Rizzi 537f87562e
FIX: We need to skip users with associated reviewables when auto-approving (#9080)
* FIX: We need to skip users with associated reviewables when auto-approving them

* Update spec/initializers/track_setting_changes_spec.rb

* Update spec/initializers/track_setting_changes_spec.rb

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2020-03-02 14:33:52 -05:00
Martin Brennan 14adddd18d
FIX: Ignore secure-media-uploads for miniprofiler (#9070) 2020-02-28 12:11:30 +10:00
Matt Palmer 377d2d3fad DEV: Silence spurious rubocop lint warning 2020-02-19 13:10:30 +11:00
Matt Palmer a14a7f1cb8 DEV: Add optional support for running byebug when a PG Clash happens
Tracking down concurrency issues from backtraces and manual repros is a fraught process.
Sometimes you've just got to get your hands dirty and do a live debug.
2020-02-19 12:50:37 +11:00
David Taylor 0b09f5299d
DEV: Improve pg connection access logging
`ensure` that the accessing thread is set to nil after an action
2020-02-18 16:58:47 +00:00
David Taylor ea49ca7ef5
DEV: Handle nil backtraces in pg access logs 2020-02-18 15:45:44 +00:00
David Taylor 2bdd1275ce
DEV: Initialize pg access log mutex in non-sidekiq processes
Followup to be3e4ab3f5
2020-02-18 14:20:28 +00:00
David Taylor be3e4ab3f5
DEV: Report simultaneous use of PG::Connection objects 2020-02-18 13:50:15 +00:00
Sam Saffron 28292d2759
PERF: avoid shelling to get hostname aggressively
Previously we had many places in the app that called `hostname` to get
hostname of a server. This commit replaces the pattern in 2 ways

1. We cache the result in `Discourse.os_hostname` so it is only ever called once

2. We prefer to use Socket.gethostname which avoids making a shell command

This improves performance as we are not spawning hostname processes throughout
the app lifetime
2020-02-18 15:13:19 +11:00
David Taylor cd3fab9ccc
DEV: Allow raw PG tracing to be enabled only for sidekiq processes 2020-02-17 18:14:14 +00:00
David Taylor 0c6f2892c6
DEV: Add raw PG connection tracing behind an environment variable
This should be useful for debugging connection problems. Warning: this will generate some large files, and will likely impact performance
2020-02-17 16:21:26 +00:00
David Taylor 5919618a87
DEV: Drop legacy OpenID 2.0 support (#8894)
This is not used in core or official plugins, and has been printing a deprecation notice since v2.3.0beta4. All OpenID 2.0 code and dependencies have been dropped. The user_open_ids table remains for now, in case anyone has missed the deprecation notice, and needs to migrate their data.

Context at https://meta.discourse.org/t/-/113249
2020-02-07 17:32:35 +00:00
OsamaSayegh a516c5df82 DEV: Bump logster version to 2.6.1 and enable new logster feature
Logster 2.6.1 includes a few new features and fixes. More details here: 58bb5c5368/CHANGELOG.md
2020-02-07 13:35:26 +00:00