Commit Graph

1175 Commits

Author SHA1 Message Date
David Taylor 1121062aa9
DEV: Run prettier and rubocop in parallel for CI (#17357) 2022-07-07 13:29:14 +08:00
tshenry d1a15d4f8d
FIX: Should be UploadReference instead of UploadReferences (#17361)
This fixes a couple of typos that were introduced in 9db8f00
2022-07-06 11:40:54 -07:00
Jarek Radosz 2e4056d185
DEV: Improve multisite db scripts in dev (#17337)
## Without multisite.yml config

No change. `bin/rails db:create` / `db:migrate` / `db:drop` should work the same.

## With multisite.yml config

### db:create

`bin/rails db:create` creates development, test, and all databases from the multisite config

`RAILS_DB=[site] bin/rails db:create` creates the database for the specified site from the multisite config

### db:migrate

`bin/rails db:migrate` migrates the development database and all databases from the multisite config

`RAILS_ENV=test bin/rails db:migrate` migrates the test database and `discourse_test_multisite`

`RAILS_DB=[site] bin/rails db:migrate` migrates the database for the specified site from the multisite config

### db:drop

`bin/rails db:drop` drops development, test, and all databases from the multisite config

`RAILS_DB=[site] bin/rails db:create` drops the database for the specified site from the multisite config
2022-07-06 10:39:03 +02:00
Jarek Radosz c5f0aa2f32
DEV: Remove handlebars from the old package.json (#17319) 2022-07-04 15:05:03 +02:00
Jarek Radosz aa7792cf93
DEV: Use npm bootstrap (#17315) 2022-07-04 11:36:51 +02:00
Jarek Radosz a1170970db
FIX: Add wizard.js to ember-cli assets (#17153) 2022-06-20 17:18:35 +02:00
Jarek Radosz 2c1fc28d00
DEV: Remove ember-cli flags from the backend (#17147)
…and other auxiliary code

* Restore `QUNIT_EMBER_CLI` flag warning
* Add `ALLOW_EMBER_CLI_PROXY_BYPASS`
2022-06-20 16:33:05 +02:00
David Taylor 1b4692039e DEV: Remove legacy JS manifests and vendored scripts
Now that we've switched to Ember CLI, these things are no longer used.

- These sprockets manifests are superceded by the assets generated by ember cli
- These vendored scripts are now fetched by ember-auto-import at compile time
2022-06-20 15:01:06 +01:00
David Taylor 12980418ae
DEV: Disable the use of 'legacy' Ember assets (#17127)
Anyone still using `EMBER_CLI_PROD_ASSETS=0` in development or production will be gracefully switched to Ember CLI. In development, a repeated message will be logged to STDERR.

Similarly, passing `QUNIT_EMBER_CLI=0` to the qunit rake task will now do nothing. A warning will be printed, and ember-cli mode will be used. Note that we've chosen not to fail the task, so that existing plugin/theme CI jobs don't immediately start failing. We may switch to a hard fail in the coming days/weeks.
2022-06-17 16:51:28 +01:00
Jarek Radosz 91bc4442d1
DEV: Remove wizard step from docker tests (#17125)
Wizard tests are now in core.
2022-06-17 15:35:18 +02:00
Jarek Radosz fcb4e5a1a1
DEV: Make wizard an ember addon (#17027)
Co-authored-by: David Taylor <david@taylorhq.com>
2022-06-17 14:50:21 +02:00
Gerhard Schlager 888f50543d
DEV: Fix `javascript:update` rake task (#17098)
* The `javascript:update` rake task failed because recent versions of chart.js use a lowercase filename (`chart.min.js` instead of `Chart.min.js`)

* Changed `loadScript()` to use lowercase keys to lookup scripts

* `svg-arrow.css` seems to have changed slightly (linebreak at the end of file)
2022-06-15 19:49:04 +02:00
Gerhard Schlager 96e87af605
FIX: Rake tasks related to uploads were broken (#17085) 2022-06-14 11:05:03 +02:00
Bianca Nenciu 9db8f00b3d
FEATURE: Create upload_references table (#16146)
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.

* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
2022-06-09 09:24:30 +10:00
Daniel Waterworth d7bc395182
FIX: Seed multisite dbs after migrating in development (#17017)
Dumping the schema cache reset the current_db and we only need to do
this once.
2022-06-06 14:43:43 -05:00
Alan Guo Xiang Tan 8471eabf62 DEV: Add env to skip wizard tests 2022-05-25 13:48:09 +08:00
David Taylor 115fe90bff
DEV: Only demux migration stdout if running concurrently (#16895)
This feature only was only demuxing stdout, not stderr. That means that stdout and stderr output appears out-of-order, and makes debugging migrations very confusing.

In future we may want to add stderr support to the demuxing. But right now, the concurrency variable is hard-coded to 1. Therefore the easiest fix is to bypass the demuxing.
2022-05-23 14:26:13 +01:00
Martin Brennan faf5b4d3e9
PERF: Speed up secure media and ACL sync rake tasks (#16849)
Incorporates learnings from /t/64227:

* Changes the code to set access control posts in the rake
  task to be an efficient UPDATE SQL query.
  The original version was timing out with 312017 post uploads,
  the new query took ~3s to run.
* Changes the code to mark uploads as secure/not secure in
  the rake task to be an efficient UPDATE SQL query rather than
  using UploadSecurity. This took a very long time previously,
  and now takes only a few seconds.
* Spread out ACL syncing for uploads into jobs with batches of
  100 uploads at a time, so they can be parallelized instead
  of having to wait ~1.25 seconds for each ACL to be changed
  in S3 serially.

One issue that still remains is post rebaking. Doing this serially
is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates
but this is limited by max_old_rebakes_per_15_minutes. It would
be better to fan this rebaking out into jobs like we did for the
ACL sync, but that should be done in another PR.
2022-05-23 13:14:11 +10:00
Penar Musaraj 18ebe2fc28
DEV: Add ability to populate private messages (#16705) 2022-05-12 14:29:49 -04:00
Bianca Nenciu 5165fb638e
FIX: Show error message if extensions cannot be created (#16719)
It used to stop the db:migrate task and broke sites that are deployed
in non-standard environments (using external database server or older
versions).
2022-05-11 15:46:49 +03:00
David Taylor 476bd1d237
DEV: Fix production sourcemaps with Ember CLI (#16707)
22a7905f restructured how we load Ember CLI assets in production. Unfortunately, it also broke sourcemaps for those assets. This commit fixes that regression via a couple of changes:

- It adds the necessary `.map` paths to `config.assets.precompile`
- It swaps Sprockets' default `SourcemappingUrlProcessor` with an extended version which maintains relative URLs of maps
2022-05-11 10:23:32 +01:00
Bianca Nenciu 631bfb2a23
FIX: Create PostgreSQL extensions before migrating (#16158)
Enabled extensions are not saved in backups and thus not created when
restoring a newer backup (that has a new extension) to an old site
(that does not have the migration).
2022-05-11 09:48:03 +03:00
Sam 2df3c65ba9
FIX: add support for pipelined and multi redis commands (#16682)
Latest redis interoduces a block form of multi / pipelined, this was incorrectly
passed through and not namespaced.

Fix also updates logster, we held off on upgrading it due to missing functions
2022-05-10 08:19:02 +10:00
Jarek Radosz 68ccaa3acb
DEV: Fix typos and outdated comments (#16614) 2022-05-04 14:12:18 +08:00
Joffrey JAFFEUX fad94160c7
FIX: uses tippy for popover (#15409)
Note this commit also introduce a new {{d-popover}} component, example usage:

```hbs
{{#d-popover |state|}}
  {{d-button label="foo.things" class="d-popover-trigger"}}

  <div class="d-popover-content">
    Some content
  <div>
{{/d-popover}}
```
2022-05-02 17:10:26 +02:00
Loïc Guitaut 008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
David Taylor 127ba698a7
DEV: Allow running theme-qunit tests via testem (#16540)
This allows `QUNIT_EMBER_CLI=1 bin/rake theme:qunit[...]` to test themes using `testem` with Ember-CLI-generated assets
2022-04-22 15:04:01 +01:00
Jarek Radosz 1209d51a7f
DEV: Don't use `Dir.chdir()` (#16535)
`system()` provides `chdir:` option for that.
2022-04-22 11:17:10 +02:00
David Taylor 22a7905f2d
DEV: Allow Ember CLI assets to be used by development Rails app (#16511)
Previously, accessing the Rails app directly in development mode would give you assets from our 'legacy' Ember asset pipeline. The only way to run with Ember CLI assets was to run ember-cli as a proxy. This was quite limiting when working on things which are bypassed when using the ember-cli proxy (e.g. changes to `application.html.erb`). Also, since `ember-auto-import` introduced chunking, visiting `/theme-qunit` under Ember CLI was failing to include all necessary chunks.

This commit teaches Sprockets about our Ember CLI assets so that they can be used in development mode, and are automatically collected up under `/public/assets` during `assets:precompile`. As a bonus, this allows us to remove all the custom manifest modification from `assets:precompile`.

The key changes are:
- Introduce a shared `EmberCli.enabled?` helper
- When ember-cli is enabled, add ember-cli `/dist/assets` as the top-priority Rails asset directory
- Have ember-cli output a `chunks.json` manifest, and teach `preload_script` to read it and append the correct chunks to their associated `afterFile`
- Remove most custom ember-cli logic from the `assets:precompile` step. Instead, rely on Rails to take care of pulling the 'precompiled' assets into the `public/assets` directory. Move the 'renaming' logic to runtime, so it can be used in development mode as well.
- Remove fingerprinting from `ember-cli-build`, and allow Rails to take care of things

Long-term, we may want to replace Sprockets with the lighter-weight Propshaft. The changes made in this commit have been made with that long-term goal in mind.

tldr: when you visit the rails app directly, you'll now be served the current ember-cli assets. To keep these up-to-date make sure either `ember serve`, or `ember build --watch` is running. If you really want to load the old non-ember-cli assets, then you should start the server with `EMBER_CLI_PROD_ASSETS=0`. (the legacy asset pipeline will be removed very soon)
2022-04-21 16:26:34 +01:00
Jarek Radosz 69dbc207ab
DEV: Improve `plugin:versions` task (#16391)
1. Sort plugins by name
2. Include plugins that are a symbolic link to a submodule repo (in those cases `.git` isn't a directory but a file that looks like e.g. `gitdir: ../../.git/modules/plugins/name-here`)
2022-04-06 18:29:39 +02:00
Jarek Radosz 9060a045c2
DEV: Minor linting fixes (#16359)
1. `test/run-qunit.js` wasn't eslinted (I'm not adding it to the CI workflow for now, just fixed the issues)
2. "…" utf character isn't rendered correctly in Jenkins, replaced with three dots
3. Don't try to lint `tmp` when doing `eslint .` in the root dir
2022-04-02 21:15:38 +02:00
David Taylor 9df28fe4b3
DEV: Only raise `rake themes:update` errors when flag provided (#16254)
Switching behavior based on multisite/single-site configuration can create some difficult-to-debug situations. The flag is much more obvious.
2022-03-22 17:02:14 +00:00
Jarek Radosz e4a92a8113
DEV: Add `SKIP_DB_CREATE` option to docker:test (#16225)
So that the database creation can be run separately from the specs.
2022-03-21 13:21:06 +01:00
Jarek Radosz bf8dc394bd
DEV: Add chromium to ChromeInstalledChecker (#16224) 2022-03-19 11:00:06 +01:00
David Taylor c58c2f53e8
FIX: Skip ember-cli build memory optimisation for multiple rake tasks (#16198) 2022-03-16 12:02:21 +00:00
Jarek Radosz dec68d780c
DEV: Don't install plugin gems twice (#16192)
Missing plugin gems are installed when the app is being loaded.

That means when you run `bin/rails plugin:install_all_gems` it first installs missing gems and then reinstalls all gems…

Also, the method these rake tasks were using to install gems was very crude, and the regex there was incorrect which resulted in failures in certain cases. Though that didn't matter since those gems were being installed using a correct method just moments before…
2022-03-15 15:08:05 +01:00
David Taylor ea1865cc01
DEV: Ensure yarn dependencies are installed during docker.rake (#16185)
Followup to 349a4acc9f
2022-03-14 17:53:21 +00:00
David Taylor 349a4acc9f
DEV: Use Ember CLI for docker.rake CI testing (#16182) 2022-03-14 17:32:03 +00:00
David Taylor 21cc57674c
FIX: Do not run Ember CLI chunk files through babel twice (#16183) 2022-03-14 16:11:13 +00:00
Jarek Radosz 8d5281115e
DEV: Use the --no-rebase option of `git pull` (#16180)
…to avoid repeatedly printed notes:

```
hint: Pulling without specifying how to reconcile divergent branches is
hint: discouraged. You can squelch this message by running one of the following
hint: commands sometime before your next pull:
hint:
hint:   git config pull.rebase false  # merge (the default strategy)
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
```
2022-03-14 15:13:06 +01:00
Jarek Radosz 3db4b3bc5e
DEV: Drop lodash (#16110) 2022-03-06 18:15:25 +01:00
Jarek Radosz f9e1ba4f8f
DEV: Remove remnants of nginx-perf-report plugin (#16107)
The plugin is no longer official as of Nov 18, 2019 (e2ccb0c608)
2022-03-05 17:22:23 +01:00
David Taylor 9415ec91a1
DEV: Make load order consistent for `rake plugin:spec` (#16076)
`Dir.glob` doesn't guarantee any particular order for results. However, it does appear to be consistent on a given machine. This means that specs can consistently pass on one machine while consistently failing on another. This can lead to some very confusing situations!

This commit sorts the spec files alphabetically so that load order is consistent across environments.

Note that the order in which tests are **run** is not affected by this change. Run order is still randomized by RSpec
2022-03-01 17:46:47 +00:00
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Jarek Radosz 29e601c13a
DEV: Fix the javascript:update rake task (#16057)
The source of the moment tz name package has changed in #15761
2022-02-26 02:06:12 +01:00
Dan Ungureanu ae5eab06ad
FEATURE: Add ENV to control themes:update errors (#16051)
This commit introduces a new environment variable `RAISE_THEME_ERRORS`
that can control what happens when `theme:update` Rake task errors. It
can have three possible values: `0` to always print errors, `1` to
always raise on error, or be absent to use the default behavior which
raises errors only for default sites.
2022-02-25 11:54:42 +02:00
Joffrey JAFFEUX 8244b4b163
DEV: prevents warnings with EMBER_CLI being redefined (#16035)
Example error:

```
/__w/discourse/discourse/lib/tasks/assets.rake:3: warning: already initialized constant EMBER_CLI
/__w/discourse/discourse/lib/tasks/assets.rake:3: warning: previous definition of EMBER_CLI was here
```
2022-02-23 10:51:31 +01:00
David Taylor 6ab4d26d84
PERF: Terminate `rake` process during `ember build` (#15920)
The `assets:precompile` rake task loads the full Ruby app, which can consume around 500mb of RAM by itself. Using `exec` to run `ember build` allows us to free up the Ruby memory and make more space for `ember build`
2022-02-14 11:49:46 +00:00
David Taylor a01b1dd648
PERF: Update ember-auto-import and webpack (#15919)
This makes a small improvement to 'cold cache' ember-cli build times, and a large improvement to 'warm cache' build times

The ember-auto-import update means that vendor is now split into multiple files for efficiency. These are named `chunk.*`, and should be included immediately after the `vendor.js` file. This commit also updates the rails app to render script tags for these chunks.

This change was previously merged, and caused memory-related errors on RAM-constrained machines. This was because Webpack 5 switches from multiple worker processes to a single multi-threaded process. This meant that it was hitting node's default heap size limit (~500mb on a 1GB RAM server). Discourse's standard install procedure recommends adding 2GB swap to 1GB-RAM machines, so we can afford to override's Node's default via the `--max-old-space-size` flag.
2022-02-14 11:21:39 +00:00
David Taylor 8025768fd2
DEV: Embed source content in JS sourcemaps (#15914)
This removes the need for a further request for the 'original' source file, and brings our custom terser arguments in line with the ember-cli defaults
2022-02-11 20:38:53 +00:00