Commit Graph

177 Commits

Author SHA1 Message Date
Guo Xiang Tan 84c12d8f3d
DEV: Allow multisite apps to boot with readonly db. 2020-06-23 16:34:25 +08:00
Guo Xiang Tan 58e52c0e4f
DEV: Use rails_failover gem for ActiveRecord and Redis failover handling 2020-06-15 15:47:07 +08:00
Guo Xiang Tan d8cd912769
DEV: Switch to db config to disable advisory locks. 2020-06-15 14:33:41 +08:00
Guo Xiang Tan e82d4d8a75
DEV: Update rails_failover to avoid monkey patching Rails config. 2020-06-05 09:05:19 +08:00
Guo Xiang Tan 248db310d3 DEV: Fix lint. 2020-06-04 17:25:00 +08:00
Guo Xiang Tan aaece34e8b DEV: Update rails_failover so that we can move middleware up the stack. 2020-06-04 17:14:13 +08:00
Gerhard Schlager 69ee94b526 FIX: XML files could be detected as SVG files 2020-05-26 18:18:20 +02:00
Guo Xiang Tan 878f06f1fe DEV: Remove custom connection reaper.
Rails 6 fixed the reaper to use one thread to reap all the connection pools.
2020-05-26 09:09:46 +08:00
Michael Brown d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood 20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Sam Saffron b031e3220a
FIX: migrations should not fail when db is part migrated
When part migrated the translate overrides may have missing columns
in that case just silently ignore overrides
2020-05-20 15:28:01 +10:00
Robin Ward db8e872bda
DEV: Move select kit to an addon (#9797) 2020-05-15 16:07:35 -04:00
Robin Ward d615de9139
DEV: Support for `import Handlebars from 'handlebars'`; (#9600)
* Remove Handlebars.SafeString usage

* DEV: Support for `import Handlebars from 'handlebars'`;

* FIX: Sprockets was broken when `node_modules` was present

By default the old version of sprockets looks for application.js
anywhere, including in a node_modules folder if this exists
(which it will when we move to Ember CLI.)
2020-04-30 16:41:02 -04:00
Robin Ward cbb27241c4
DEV: Make `discourse-common` an Ember addon. (#9578)
This is to help with the migration to Ember CLI. In the current running
version of Discourse everything should be the same as before, just with
a few extra files that are not used. However, using Ember CLI this can
be installed as an Ember addon.

Co-Authored-By: Jarek Radosz <jradosz@gmail.com>
2020-04-29 12:18:21 -04:00
Roman Rizzi 7a2e8d3ead
DEV: Add the missing `app` subdirectory (#9499)
* DEV: Add missing  directory to the Discourse ember app

* DEV: Resolve imports correctly
2020-04-23 10:07:54 -03:00
Sam Saffron e2284cf739
Revert "We have had errors reported due to migrations breaking and are reverting"
This reverts commit 8b46f14744.

It corrects the reason for the revert:

We rely on SafeMigrate existing cause we call it from migrations,
Zeitwerk will autoload it.

Instead of previous pattern we explicitly bypass all the hacks in
production mode.

We need to disable SafeMigrate cause it is not thread safe.

A thread safe implementation is possible but not worth the effort,
we catch the issues in dev and test.
2020-04-14 11:31:07 +10:00
Robin Ward 8b46f14744 We have had errors reported due to migrations breaking and are reverting
these series of commits.

See:
https://meta.discourse.org/t/new-installation-fails-on-migration/147425

This reverts commit 80e832662b.
This reverts commit de5f2d3308.
This reverts commit 6b192d29fa.
This reverts commit a5b582f686.
This reverts commit 708dd97dfd.
2020-04-09 14:18:56 -04:00
Sam Saffron de5f2d3308
DEV: avoid loading module in production 2020-04-09 13:54:18 +10:00
Sam Saffron 6b192d29fa
DEV: only enable SafeMigrate in development and test
SafeMigrate outputs text when we detect attempts to unsafely drop tables
and columns

It is unfortunately not thread safe

This is not needed in production as we would have already caught it by then
in our test suite.
2020-04-09 13:41:03 +10:00
Daniel Waterworth bca126f3f5 REFACTOR: Move the multisite middleware to the front
Both request tracking and message bus rely on multisite before the
middleware has run which is not ideal.

Follow-up-to: ca1208a636
2020-04-02 16:44:44 +01:00
Daniel Waterworth ca1208a636 Revert "REFACTOR: Move the multisite middleware to the front"
Looks like this is causing problems.

Follow-up-to: a91843f0dc
2020-04-02 15:20:28 +01:00
Daniel Waterworth a91843f0dc REFACTOR: Move the multisite middleware to the front
Both request tracking and message bus rely on multisite before the
middleware has run which is not ideal.
2020-04-02 10:15:38 +01:00
Robin Ward 27641f21e4 Migrate `discourse-common` from es6 -> js 2020-03-23 15:05:58 -04:00
Gerhard Schlager 8022e51179 FIX: Failed to restore backups from versions without translation overrides
Rails calls I18n.translate during initialization and by default translation overrides are used. Database migrations would fail if the system tried to migrate from an old version that didn't have the `translation_overrides` table with all its columns yet.

This makes restoring really old backups work again. Running `DISABLE_TRANSLATION_OVERRIDES=1 rake db:migrate` will allow you to upgrade such an old database as well.
2020-03-14 00:00:22 +01:00
Robin Ward a3f0543f99
Support for transpiling `.js` files (#9160)
* Remove some `.es6` from comments where it does not matter

* Use a post processor for transpilation

This will allow us to eventually use the directory structure to
transpile rather than the extension.

* FIX: Some errors and clean up in confirm-new-email

It would throw an error if the webauthn element wasn't present.
Also I changed things so that no-module is not explicitly
referenced.

* Remove `no-module`

Instead we allow a magic comment: `// discourse-skip-module` to prevent
the asset pipeline from creating a module.

* DEV: Enable babel transpilation based on directory

If it's in `app/assets/javascripts/dicourse` it will be transpiled
even without the `.es6` extension.

* REFACTOR: Remove Tilt/ES6ModuleTranspiler
2020-03-11 09:43:55 -04:00
David Taylor 836ab73d59
FIX: Patch ActiveRecord SchemaCache for safe concurrency support
A single SchemaCache instance is maintained by the connection pool, and made available via a schema_cache method on each connection. When the SchemaCache instance is fetched from the pool, its internal connection reference is updated to equal the requesting connection. However, since there is only one instance of SchemaCache, this internal connection reference is updated everywhere, and can ultimately result in multiple threads accessing the same database connection. In Discourse, this could result in Sidekiq jobs getting 'stuck' in database connections.

This patch modifies SchemaCache so that it caches the internal connection on a per-thread basis

Co-authored-by: Sam Saffron <sam.saffron@gmail.com>
Co-authored-by: Matt Palmer <mpalmer@hezmatt.org>
2020-02-19 16:26:02 +00:00
Sam Saffron 28292d2759
PERF: avoid shelling to get hostname aggressively
Previously we had many places in the app that called `hostname` to get
hostname of a server. This commit replaces the pattern in 2 ways

1. We cache the result in `Discourse.os_hostname` so it is only ever called once

2. We prefer to use Socket.gethostname which avoids making a shell command

This improves performance as we are not spawning hostname processes throughout
the app lifetime
2020-02-18 15:13:19 +11:00
Mark VanLandingham 3e89774908
DEV: Use .hbr for raw template file extension (#8883) 2020-02-11 13:38:12 -06:00
Penar Musaraj 067696df8f DEV: Apply Rubocop redundant return style 2019-11-14 15:10:51 -05:00
Sam Saffron d4c9ed68a9 DEV: properly pass keyword args
Ruby 2.7 is going to deprecate our old and incorrect usage, use the correct
form for passing kwargs
2019-11-08 17:30:42 +11:00
Sam Saffron 26c0199c01 DEV: update Rails to version 6.0.1
This version of Rails eliminates a monkey patch that is no longer needed!

Additionally it preps us for Ruby 2.7 support.
2019-11-08 16:56:30 +11:00
Gerhard Schlager 61b1f9c36b FEATURE: Load translation overrides without JS `eval` 2019-11-05 19:16:38 +01:00
Robin Ward 785ebb674d REFACTOR: Split off raw handlebars helpers from compiler
This allows us to compile without Ember being present
2019-11-01 13:50:15 -04:00
Daniel Waterworth 55a1394342 DEV: pluck_first
Doing .pluck(:column).first is a very common pattern in Discourse and in
most cases, a limit cause isn't being added. Instead of adding a limit
clause to all these callsites, this commit adds two new methods to
ActiveRecord::Relation:

pluck_first, equivalent to limit(1).pluck(*columns).first

and pluck_first! which, like other finder methods, raises an exception
when no record is found
2019-10-21 12:08:20 +01:00
Krzysztof Kotlarek 858cf5836c
FIX: update Redis gem to version 4.1.3
I run our benchmark on commit with hiredis and redis-4.1.3

Results:
type | hidredis | redis 4.1.3 | percent
--- | --- | --- | ---
Categories-50 | 49 | 50 | 102.04%
Categories-75 | 51 | 51 | 100.00%
Categories-90 | 63 | 64 | 101.59%
Categories-99 | 86 | 85 | 98.84%
Home-50 | 55 | 55 | 100.00%
Home-75 | 56 | 57 | 101.79%
Home-90 | 68 | 69 | 101.47%
Home-99 | 102 | 104 | 101.96%
Topic-50 | 36 | 37 | 102.78%
Topic-75 | 37 | 37 | 100.00%
Topic-90 | 47 | 48 | 102.13%
Topic-99 | 60 | 61 | 101.67%
Categories-admin-50 | 124 | 117 | 94.35%
Categories-admin-75 | 130 | 129 | 99.23%
Categories-admin-90 | 147 | 143 | 97.28%
Categories-admin-99 | 204 | 199 | 97.55%
Home-admin-50 | 146 | 148 | 101.37%
Home-admin-75 | 150 | 152 | 101.33%
Home-admin-90 | 169 | 168 | 99.41%
Home-admin-99 | 232 | 223 | 96.12%
Topic-admin-50 | 60 | 61 | 101.67%
Topic-admin-75 | 64 | 63 | 98.44%
Topic-admin-90 | 76 | 73 | 96.05%
Topic-admin-99 | 124 | 94 | 75.81%
Load rails | 2412 | 2360 | 97.84%
rss | 290204 | 295828 | 101.94%
pss | 277948 | 283624 | 102.04%

Redis gem is manipulating Redis config https://github.com/redis/redis-rb/blob/master/lib/redis/client.rb#L95
therefore we cannot pass the frozen config object.

Pass of the copy of the object is protecting original config
2019-10-21 09:59:24 +11:00
Sam Saffron ae2a56999e Revert "FIX: update Redis gem to version 4.1.3 (#8197)"
This reverts commit ab74a50d85.

We really want to upgrade redis, but discovered some edge cases
around failover we need to test.

Holding off on the upgrade till a bit more testing happens
2019-10-17 11:41:46 +11:00
Krzysztof Kotlarek ab74a50d85 FIX: update Redis gem to version 4.1.3 (#8197)
* FIX: update Redis gem to version 4.1.3

I run our benchmark on commit with hiredis and redis-4.1.3

Results:
type | hidredis | redis 4.1.3 | percent
--- | --- | --- | ---
Categories-50 | 49 | 50 | 102.04%
Categories-75 | 51 | 51 | 100.00%
Categories-90 | 63 | 64 | 101.59%
Categories-99 | 86 | 85 | 98.84%
Home-50 | 55 | 55 | 100.00%
Home-75 | 56 | 57 | 101.79%
Home-90 | 68 | 69 | 101.47%
Home-99 | 102 | 104 | 101.96%
Topic-50 | 36 | 37 | 102.78%
Topic-75 | 37 | 37 | 100.00%
Topic-90 | 47 | 48 | 102.13%
Topic-99 | 60 | 61 | 101.67%
Categories-admin-50 | 124 | 117 | 94.35%
Categories-admin-75 | 130 | 129 | 99.23%
Categories-admin-90 | 147 | 143 | 97.28%
Categories-admin-99 | 204 | 199 | 97.55%
Home-admin-50 | 146 | 148 | 101.37%
Home-admin-75 | 150 | 152 | 101.33%
Home-admin-90 | 169 | 168 | 99.41%
Home-admin-99 | 232 | 223 | 96.12%
Topic-admin-50 | 60 | 61 | 101.67%
Topic-admin-75 | 64 | 63 | 98.44%
Topic-admin-90 | 76 | 73 | 96.05%
Topic-admin-99 | 124 | 94 | 75.81%
Load rails | 2412 | 2360 | 97.84%
rss | 290204 | 295828 | 101.94%
pss | 277948 | 283624 | 102.04%

* FIX: get rid of redis freedom patch
2019-10-17 08:49:23 +11:00
Sam Saffron 586dfcc795 PERF: avoid regex for hot path
Camelize is called quite a lot in zeitwerk, avoid using a regex here which
is far slower than using ends_with?
2019-10-08 13:43:54 +11:00
Krzysztof Kotlarek 35b1185a08 FIX: Revert Demon::DemonBase back to Demon::Base (#8132)
I introduced DemonBase because I had got some conflict between `demon/base.rb` and `jobs/base.rb`, however, to not rename base class, it is possible to use regex on absolute path in Zeitwerk custom inflector.
2019-10-02 14:54:08 +10:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Sam Saffron 445d305154 DEV: initial migration can fail
db:migrate can issue translations due to module loading localizing
end user messages

This allows db:migrate to work even when db is blank
2019-09-17 13:38:01 +10:00
Sam Saffron 082f59842d DEV: bind for thread local vars should yield block
followup on #015051ec without this improvement monkey patch does not work.

bind should yield the block it is passed for it to work.
2019-09-12 17:57:57 +10:00
Sam Saffron 015051ecaf PERF: avoid spinning a thread each time we close a connection
This is a temporary workaround for the issue in https://github.com/rails/rails/pull/36949

Discussing a proper fix in Rails with the Rails team.

Prior to this fix we were spinning up a thread every time we closed a connection
to the db.
2019-09-12 17:34:04 +10:00
Krzysztof Kotlarek 32b8a2ccff DEV: Upgrade Discourse to Rails 6 (#8083)
* Adjustments to pass specs on Rails 6.0.0
* Use classic autoloader instead of Zeitwerk
* Update Rails 6.0.0 deprecated methods
* Rails 6.0.0 not allowing column with integer name
* Drop freedom_patches/rails6.rb
* Default value for trigger_transactional_callbacks? is true
* Bump rspec-rails version to 4.0.0.beta2
2019-09-12 10:41:50 +10:00
Gerhard Schlager 24877a7b8c FIX: Correctly encode non-ASCII filenames in HTTP header
Backport of fix from Rails 6: 890485cfce
2019-08-07 19:10:50 +02:00
Sam fa2a5f6f56
FEATURE: SKIP_DB_AND_REDIS env var (#7756)
Sometimes we would like to create a base image without any DB access, this
assists in creating custom base images with custom plugins that already
includes `public/assets`

Following this change set you can run:

```
SPROCKETS_CONCURRENT=1 DONT_PRECOMPILE_CSS=1 SKIP_DB_AND_REDIS=1 RAILS_ENV=production bin/rake assets:precompile
```

Then it is straight forward to create a base image without needing a DB or
Redis.
2019-06-13 12:58:27 +10:00
Gerhard Schlager a77d8bee68 FIX: Use correct locale when translating without cache
follow-up to 6d8eb9c1
2019-06-05 14:19:56 +02:00
Sam Saffron 6d8eb9c144 PERF: bypass slow locale lookups in various cases
Previously as soon as any override was defined we would regress to the slow
path for locale lookups. Additionally if `raise: true` was specified which
rails likes to add in views we would bypass the cache

The new design manages to use the fast path for many more cases
2019-06-05 14:30:25 +10:00
Gerhard Schlager a3e4f43a4d DEV: Remove patch for I18n.exists?
It was merged upstream a long time ago
2019-05-13 09:36:05 +02:00
Sam Saffron 982c4496ee Correct some missing spots for frozen_string_literal 2019-05-13 09:31:32 +08:00