* Revert "DEV: increase lock timeout for multisite migration (#14831)"
This partially reverts commit 337ef60303.
We need to revert the mutex around `db:status:json` because the mutex is not available unless the rails environment is loaded which the `db:status:json` doesn't load before the mutex. We can't load the environment before entering the mutex because the mutex is meant to prevent other instances of the task from loading a rails environment while the database is migrating.
Co-authored-by: David Taylor <david@taylorhq.com>
Co-authored-by: David Taylor <david@taylorhq.com>
In development we regularly restart/reload Rails, which wipes out the schema cache. This then has to be regenerated using DDL queries on the database.
Instead, we can make use of the `rake db:schema:cache:dump` command. This will dump the schema cache to a YAML file, and then load it when needed. This is significantly faster than rebuilding the cache from DDL queries every time.
This commit allows site admins to run theme tests in production via a new `/theme-qunit` route. When you visit `/theme-qunit`, you'll see a list of the themes/components installed on your site that have tests, and from there you can select a theme or component that you run its tests.
We also have a new rake task `themes:install_and_test` that can be used to install a list of themes/components on a temporary database and run the tests of the themes/components that are installed. This rake task can be useful when upgrading/deploying a Discourse instance to make sure that the installed themes/components are compatible with the new Discourse version being deployed, and if the tests fail you can abort the build/deploy process so you don't end up with a broken site.
This commit allows site admins to run theme tests in production via a new `/theme-qunit` route. When you visit `/theme-qunit`, you'll see a list of the themes/components installed on your site that have tests, and from there you can select a theme or component that you run its tests.
We also have a new rake task `themes:install_and_test` that can be used to install a list of themes/components on a temporary database and run the tests of the themes/components that are installed. This rake task can be useful when upgrading/deploying a Discourse instance to make sure that the installed themes/components are compatible with the new Discourse version being deployed, and if the tests fail you can abort the build/deploy process so you don't end up with a broken site.
Added a new task to test if indexes are coherent with a blank database
This allows us to detect for cases where somehow indexes are out of sync
FIX_INDEXES=1 or `rake db:validate_indexes[fix]` to correct the issues it finds.
Detects:
- Badly named indexes that need to be renamed
- Missing indexes
- Extra indexes
Can correct all 3 with the fix option
Added a small helper class to for seed data because we need to add the
same filter to multisite:migrate as we have in db:migrate. Having this
filter in both places means we can get rid of the SKIP_SEED flag.
A future-dated migration was accidently introduced by me in 45c399f0. This was removed in b9762afc, but other migrations had already been generated based on its incorrect date. This commit removes the offending data in the schema_migrations table, and corrects the version in the published_pages migration.
This commit also adds a check to db:migrate which raises an error when invalid migration timestamps are used.
This reverts commit 8b46f14744.
It corrects the reason for the revert:
We rely on SafeMigrate existing cause we call it from migrations,
Zeitwerk will autoload it.
Instead of previous pattern we explicitly bypass all the hacks in
production mode.
We need to disable SafeMigrate cause it is not thread safe.
A thread safe implementation is possible but not worth the effort,
we catch the issues in dev and test.
Previously we were migrating multisites serially, this is extremely slow
especially when 200 dbs are involved.
The new implementation defaults to running 20 migrations concurrently, leading
to a 20x speedup.
We also amended it so errors are printed out last, something that makes
debugging failures easier.
This is code specific to Discourse cause we integrate SeedFu with our
migrations and can not include this in the multisite gem.
If a migration performs no changes it should not output stuff.
Previously we would output information about seeds which was very noisy.
On multisite this was particularly bad
This used to work due to side effects.
`rake parallel:migrate` used to work very inconsistently and would only migrate
some of the databases.
This introduces the recommended change to db.yml so the correct database is
found based off TEST_ENV_NUMBER if for some reason we did not set it using
RAILS_DB
Also avoids a bunch of schema dumping which is not needed when migrating
parallel specs
DB number 1 is very odd cause for whatever reason parallel spec is not
setting it.
For this to work we need to overwrite `db:rollback` in our Rakefile like
we do for migrate, so that it removes the load_config dependency. This
allows our custom migration paths to work.
In Rails 6 due to internal changes, the following sequence no longer works:
```
RAILS_ENV=test bin/rake db:migrate
RAILS_ENV=test bin/rake db:schema:dump
dropdb discourse_test
createdb discourse_test
RAILS_ENV=test bin/rake db:schema:load
RAILS_ENV=test bin/rake db:migrate
```
What appears to be happening is that our tracking of plugin migrations is
being missed on schema:dump or load.
A more comprehensive fix restoring schema:dump / load support will be
investigated.
Prior to this change plugin migrations were not working and multisite
migrations not working.
Rails internals changed so we need to account for it.
Specifically semantics of `db:migrate` in rails changed so it is sort of
a "multisite:migrate".
This fixes a condition where an intermittent db connection could cause
invalid site settings to be stored
It also removes a catch all we had.
Somewhere around Rails 5 `db:create` started wanting full environment
this is a problem for Discourse since it needs to boot up data from the
db.
This removes the catch all and surgically adds a db / redis bypass to
db:create task.
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
This change automatically resizes icons for various purposes. Admins can now upload `logo` and `logo_small`, and everything else will be auto-generated. Specific icons can still be uploaded separately if required.
## Core
- Adds an SiteIconManager module which manages automatic resizing and fallback
- Icons are looked up in the OptimizedImage table at runtime, and then cached in Redis. If the resized version is missing for some reason, then most icons will fall back to the original files. Some icons (e.g. PWA Manifest) will return `nil` (because an incorrectly sized icon is worse than a missing icon).
- `SiteSetting.site_large_icon_url` will return the optimized version, including any fallback. `SiteSetting.large_icon` continues to return the upload object. This means that (almost) no changes are required in core/plugins to support this new system.
- Icons are resized whenever a relevant site setting is changed, and during post-deploy migrations
## Wizard
- Allows `requiresRefresh` wizard steps to reload data via AJAX instead of a full page reload
- Add placeholders to the **icons** step of the wizard, which automatically update from the "Square Logo"
- Various copy updates to support the changes
- Remove the "upload-time" resizing for `large_icon`. This is no longer required.
## Site Settings UX
- Move logo/icon settings under a new "Branding" tab
- Various copy changes to support the changes
- Adds placeholder support to the `image-uploader` component
- Automatically reloads site settings after saving. This allows setting placeholders to change based on changes to other settings
- Upload site settings will be assigned a placeholder if SiteIconManager `responds_to?` an icon of the same name
## Dashboard Warnings
- Remove PWA icon and PWA title warnings. Both are now handled automatically.
## Bonus
- Updated the sketch logos to use @awesomerobot's new high-res designs
Introduce new patterns for direct sql that are safe and fast.
MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API
- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder
See more at: https://github.com/discourse/mini_sql
Osama Sayegh
Sam
Daniel Waterworth
David Taylor
Guo Xiang Tan
Blake Erickson
Dan Ungureanu
Robin Ward
Gerhard Schlager
Guo Xiang Tan