Commit Graph

49 Commits

Author SHA1 Message Date
David Taylor 72ad701df0
DEV: Stub #flush in StdOutDemux for multisite:migrate
https://meta.discourse.org/t/multisite-migrate-error/150579/2
2020-05-06 11:58:35 +01:00
Sam Saffron e2284cf739
Revert "We have had errors reported due to migrations breaking and are reverting"
This reverts commit 8b46f14744.

It corrects the reason for the revert:

We rely on SafeMigrate existing cause we call it from migrations,
Zeitwerk will autoload it.

Instead of previous pattern we explicitly bypass all the hacks in
production mode.

We need to disable SafeMigrate cause it is not thread safe.

A thread safe implementation is possible but not worth the effort,
we catch the issues in dev and test.
2020-04-14 11:31:07 +10:00
Robin Ward 8b46f14744 We have had errors reported due to migrations breaking and are reverting
these series of commits.

See:
https://meta.discourse.org/t/new-installation-fails-on-migration/147425

This reverts commit 80e832662b.
This reverts commit de5f2d3308.
This reverts commit 6b192d29fa.
This reverts commit a5b582f686.
This reverts commit 708dd97dfd.
2020-04-09 14:18:56 -04:00
Sam Saffron 80e832662b
DEV: fake stdout must respond to close
If it does not respond to close Logger will refuse to use it
2020-04-09 14:36:01 +10:00
Sam Saffron a5b582f686
DEV: demux stdout when running multisite migrate
This avoids mixing up output in such a way that we can not tell which
site ran which migrations

Avoids threads all fighting for output
2020-04-09 12:50:14 +10:00
Sam Saffron 708dd97dfd
PERF: speed up migrations on multisite
Previously we were migrating multisites serially, this is extremely slow
especially when 200 dbs are involved.

The new implementation defaults to running 20 migrations concurrently, leading
to a 20x speedup.

We also amended it so errors are printed out last, something that makes
debugging failures easier.

This is code specific to Discourse cause we integrate SeedFu with our
migrations and can not include this in the multisite gem.
2020-04-09 11:58:20 +10:00
Sam Saffron f062ebf274
DEV: reduce noise in logs regarding migration
If a migration performs no changes it should not output stuff.

Previously we would output information about seeds which was very noisy.

On multisite this was particularly bad
2020-04-09 10:05:22 +10:00
Gerhard Schlager 8fa8bab9ff FIX: Don't optimize icons during db:migrate when restoring backup
Uploads are extracted after the DB migration, so this could lead to a failure during the restore. Site icons get optimized after extracting uploads.
2020-03-04 16:59:49 +01:00
Sam Saffron 412e1ebbe2 DEV: correct parallel specs rake tasks
This used to work due to side effects.

`rake parallel:migrate` used to work very inconsistently and would only migrate
some of the databases.

This introduces the recommended change to db.yml so the correct database is
found based off TEST_ENV_NUMBER if for some reason we did not set it using
RAILS_DB

Also avoids a bunch of schema dumping which is not needed when migrating
parallel specs



DB number 1 is very odd cause for whatever reason parallel spec is not
setting it.
2019-12-31 14:07:55 +11:00
Robin Ward 9fc1a754fe DEV: Error message if multisite DB can't be created 2019-12-19 11:52:27 -05:00
Robin Ward 480e2ec523 FIX: If we run `db:migrate` on its own, it should load the environment 2019-12-16 14:46:33 -05:00
Robin Ward 397852a9a6 FIX: Support for `rake db:rollback` with plugins
For this to work we need to overwrite `db:rollback` in our Rakefile like
we do for migrate, so that it removes the load_config dependency. This
allows our custom migration paths to work.
2019-11-15 16:38:46 -05:00
Sam Saffron 1ca257be79 DEV: db:migrate no longer works after db:schema:load
In Rails 6 due to internal changes, the following sequence no longer works:

```
RAILS_ENV=test bin/rake db:migrate
RAILS_ENV=test bin/rake db:schema:dump
dropdb discourse_test
createdb discourse_test
RAILS_ENV=test bin/rake db:schema:load
RAILS_ENV=test bin/rake db:migrate
```

What appears to be happening is that our tracking of plugin migrations is
being missed on schema:dump or load.

A more comprehensive fix restoring schema:dump / load support will be
investigated.
2019-09-18 13:17:49 +10:00
Sam Saffron 025d4ee91f FIX: Rails 6 multisite migrations and plugin migrations
Prior to this change plugin migrations were not working and multisite
migrations not working.

Rails internals changed so we need to account for it.

Specifically semantics of `db:migrate` in rails changed so it is sort of
a "multisite:migrate".
2019-09-13 09:17:54 +10:00
Sam Saffron 4c6a0313f2 FEATURE: improve stats provided by rake db:stats
New stats include index size and sort by index + table size

Eg:

```
table_name                  | row_estimate | table_size | index_size | total_size
--------------------------------------------------------------------------
topic_views                 | 18260776     | 796 MB     | 2015 MB    | 2811 MB
post_timings                | 15813837     | 668 MB     | 1153 MB    | 1821 MB
posts                       | 386890       | 1532 MB    | 215 MB     | 1747 MB
post_search_data            | 317575       | 580 MB     | 162 MB     | 742 MB
incoming_emails             | 34412        | 734 MB     | 7160 kB    | 741 MB
incoming_links              | 5169302      | 306 MB     | 377 MB     | 683 MB
email_logs                  | 5            | 32 kB      | 418 MB     | 418 MB
topic_users                 | 2410166      | 217 MB     | 155 MB     | 372 MB
user_actions                | 1361253      | 101 MB     | 266 MB     | 366 MB
...

```
2019-08-02 17:25:42 +10:00
Sam Saffron ecb2fd8222 DEV: improve db:create to catch all cases
- No RAILS_ENV - create multisite / dev / test
- `test` RAILS_ENV - create multisite / test
2019-06-14 15:47:05 +10:00
Sam Saffron a01488ae67 DEV: improve on rake db:create
Followup on 3af00a65 which broke build
2019-06-14 15:06:07 +10:00
Sam Saffron 3af00a65e6 FIX: site settings loading default values when no db
This fixes a condition where an intermittent db connection could cause
invalid site settings to be stored

It also removes a catch all we had.

Somewhere around Rails 5 `db:create` started wanting full environment
this is a problem for Discourse since it needs to boot up data from the
db.

This removes the catch all and surgically adds a db / redis bypass to
db:create task.
2019-06-14 14:21:07 +10:00
David Taylor 81dcadf788
DEV: Stop compiling themes during DB migration. Recompile on cdn change. (#7676)
This is an improved implementation for bc8b7b13
2019-06-03 16:38:02 +01:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
David Taylor 0e303c7f5d
FEATURE: Automatically generate optimized site metadata icons (#7372)
This change automatically resizes icons for various purposes. Admins can now upload `logo` and `logo_small`, and everything else will be auto-generated. Specific icons can still be uploaded separately if required.

## Core

- Adds an SiteIconManager module which manages automatic resizing and fallback

- Icons are looked up in the OptimizedImage table at runtime, and then cached in Redis. If the resized version is missing for some reason, then most icons will fall back to the original files. Some icons (e.g. PWA Manifest) will return `nil` (because an incorrectly sized icon is worse than a missing icon). 

- `SiteSetting.site_large_icon_url` will return the optimized version, including any fallback. `SiteSetting.large_icon` continues to return the upload object. This means that (almost) no changes are required in core/plugins to support this new system.

- Icons are resized whenever a relevant site setting is changed, and during post-deploy migrations

## Wizard

- Allows `requiresRefresh` wizard steps to reload data via AJAX instead of a full page reload

- Add placeholders to the **icons** step of the wizard, which automatically update from the "Square Logo"

- Various copy updates to support the changes

- Remove the "upload-time" resizing for `large_icon`. This is no longer required.

## Site Settings UX

- Move logo/icon settings under a new "Branding" tab

- Various copy changes to support the changes

- Adds placeholder support to the `image-uploader` component

- Automatically reloads site settings after saving. This allows setting placeholders to change based on changes to other settings

- Upload site settings will be assigned a placeholder if SiteIconManager `responds_to?` an icon of the same name

## Dashboard Warnings

- Remove PWA icon and PWA title warnings. Both are now handled automatically.

## Bonus

- Updated the sketch logos to use @awesomerobot's new high-res designs
2019-05-01 14:44:45 +01:00
David Taylor bc8b7b13ef DEV: Recompile theme fields after db:migrate 2019-04-30 11:52:30 +01:00
Guo Xiang Tan 8e5e5d7d35 DEV: Make setting up of multisite DB in test env clearer. 2019-03-21 09:58:07 +08:00
Sam Saffron 600313373b DEV: skip creating multisite db when running JS tests only 2019-03-21 12:19:12 +11:00
Sam 5f64fd0a21 DEV: remove exec_sql and replace with mini_sql
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
Guo Xiang Tan 26d14dbcbb Remove onceoff job in migrate rake task. 2017-11-22 12:50:01 +08:00
Guo Xiang Tan edfa2c3af9 FIX: Don't enqueue onceoff job in migration.
* Deployment of app servers may happen minutes after migrations.
2017-11-22 12:07:48 +08:00
Robin Ward 966c7e7f07 FEATURE: Allow plugins to dynamically add seed fixture paths
This is useful if your plugin wants different seed data for different
locales for example.
2017-11-16 14:43:17 -05:00
Robin Ward 8cd0026cde FIX: If there is an invalid locale in the site setting, default to en 2017-11-16 10:58:29 -05:00
Guo Xiang Tan 77d4c4d8dc Fix all the errors to get our tests green on Rails 5.1. 2017-09-25 13:48:58 +08:00
Sam fef08c6fee FEATURE: rake db:stats
Basic rake task to provide db stats like largest tables, row count and size
2017-08-25 10:29:04 -04:00
Guo Xiang Tan a1f60cfcb8 Fix incorrect test migration. 2017-08-10 00:20:07 +09:00
Guo Xiang Tan 2c39743d5d Introduce multisite tests for better coverage. 2017-08-08 12:58:22 +09:00
Guo Xiang Tan 1a7e954e09 FIX: Store custom emojis as uploads.
* Depending on a hardcoded directory was a flawed design
  which made it impossible to debug when custom emojis go
  missing.
2017-03-14 13:07:18 +08:00
Guo Xiang Tan 9baf89a901 Remove database vacuum task from Discourse. 2017-02-20 09:02:38 +08:00
Robin Ward 3f7ced9236 Try running once off jobs outside of db:migrate 2016-04-07 15:07:24 -04:00
Robin Ward 078b3bc87e Log once off jobs and enqueue on `db:migrate` 2016-04-07 14:32:31 -04:00
Sam 8d49091572 PERF: stop running VACUUM FULL only run VACUUM 2015-12-07 20:14:26 +11:00
Robin Ward 57fc1e5e0c FIX: Don't run a `FULL ANALYZE` on first migration.
This seems to block databases unecessarily. We should only really be
vacuuming when there is a lot of deleted data to recover.
2015-10-22 12:34:09 -04:00
Erick Guan 29840888e5 FIX: respect default locale settings when seed
We seed special categories during migration so we have to set the
locale before the migration happens.
2015-02-14 23:51:53 +08:00
Blake Erickson 5f124df251 add rake task and seed data for discourse_api 2014-09-10 20:53:21 -06:00
Sam fe6e4d7da1 FIX: fails on initial migration if default_locale is nil 2014-08-29 12:01:28 +10:00
Sam ae2d80501a PERF: run full vacuum every 90 days (during migration)
Will reclaim space and improve perf
2014-08-07 11:07:30 +10:00
Neil Lalonde b942436d7b FIX: rescue from I18n.locale = nil when trying to seed db 2014-07-28 12:40:00 -04:00
Neil Lalonde 5a33e6f00c Move FAQ, Terms of Service, and Privacy Policy into topics in the Staff category. First post of those topics will be rendered on their respective pages. Site settings and content are not used for these documents anymore. Translations of the default text is moved into the standard YML files. 2014-07-25 14:41:20 -04:00
Régis Hanol 269f52b8f7 fix db rake tasks 2014-02-13 13:31:13 -08:00
Neil Lalonde 3a6c3ee65d Add two rake tasks: db:rebuild_indexes and import:remove_backup 2014-01-31 15:30:47 -05:00
Sam 270fde7dbd correct rake test:prepare so it seeds the db correctly
move category creation into seeds as well, so db can be seeded from structure.sql
2013-10-25 10:31:33 +11:00
Sam 3d3b589b4d for now, run seed fu automatically after migrate ... still think there should be a cleaner way 2013-04-08 09:56:42 +10:00