Commit Graph

1207 Commits

Author SHA1 Message Date
David Taylor ca72d8d030
PERF: Adjust node memory threshold for assets:precompile (#19040)
Previously we were forcing node's max-old-space-size to be 2GB. This override was added in a01b1dd6 to avoid issues caused by a lower default node heap_size_limit on machines with less memory.

This commit makes that `max-old-space-size` override more specific so that it only applies to machines with less memory. Other machines will go use Node's defaults.

The override is also lowered to 1GB. This is still high enough for the build to complete, while reducing memory usage.

https://meta.discourse.org/t/245547
2022-11-15 22:02:13 +00:00
Jarek Radosz e54a3d5ea9
DEV: Add `START_ID` to `uploads:downsize` task (#18992) 2022-11-11 22:51:48 +01:00
Jarek Radosz dc8a7e74f4
FIX: Allow attr updates of over-size-limit uploads (#18986) 2022-11-11 17:56:11 +01:00
Jarek Radosz bc22fe4fdf
DEV: Convert the downsizing script to a rake task (#18976)
…to make it testable!
2022-11-11 13:00:44 +01:00
David Taylor 8700c5ee6b
PERF: Make stylesheet hashes consistent between deploys (#18909)
Previously the stylesheet cachebusting hash was based on the maximum mtime of files. This works well in development and during in-container updates (e.g. via docker_manager). However, when a fresh docker image is created for each deploy, the file mtimes will change even if the contents has not.

This commit changes the production logic to calculate the cachebuster from the filenames and contents of the relevant assets. This should be consistent across deploys, thereby improving cache hits and improving page load times.
2022-11-07 16:13:35 +00:00
David Taylor f30f9ec5d9
PERF: Update `s3:expire_missing_assets` to delete in batches (#18908)
Some sites may have thousands of stale assets - deleting them one-by-one is very slow.

Followup to e8570b5cc9
2022-11-07 12:53:14 +00:00
David Taylor e8570b5cc9
Fix and improve `s3:expire_missing_assets` task (#18863)
- Ensure it works with prefixed S3 buckets
- Perform a sanity check that all current assets are present on S3 before starting deletion
- Remove the lifecycle rule configuration and delete expired assets immediately. This task should be run post-deploy anyway, so adding a 10-day window is not required
2022-11-07 10:44:45 +00:00
Vinoth Kannan dea44ec923
FEATURE: new site setting to hide user profiles by default. (#18864)
Previously, we didn't have a site-wide setting to set the default behavior for user profile visibility and user presence features. But we already have a user preference for that.
2022-11-06 16:44:17 +05:30
Daniel Waterworth 6f48502dd2
DEV: Add rake task to install a theme from an archive (#18890) 2022-11-04 16:42:07 -05:00
David Taylor b18ebc9648
PERF: Correct should_skip? logic in `s3:upload` (#18862)
This task is supposed to skip uploading if the asset is already present in S3. However, when a bucket 'folder path' was configured, this logic was broken and so the assets would be re-uploaded every time.

This commit fixes that logic to include the bucket 'folder path' in the check
2022-11-04 17:50:46 +00:00
Joffrey JAFFEUX 11f3618b80
DEV: initial system tests for chat and plugins (#18881)
This is a very basic to ensure it's working and open future possible work
2022-11-04 15:06:24 +01:00
Daniel Waterworth 167181f4b7
DEV: Quote values when constructing SQL (#18827)
All of these cases should already be safe, but still good to quote for
"defense in depth".
2022-11-01 14:05:13 -05:00
David Taylor 1da5aa9027
DEV: Parallelize core JS tests in docker.rake (#18756) 2022-10-26 12:00:35 +01:00
Jan Cernik 08476f17ff
FEATURE: Add dark mode option for category logos (#18460)
Adds a new upload field for a second dark mode category logo. 
This alternative will be used when the browser is in dark mode (similar to the global site setting for a dark logo).
2022-10-07 11:00:44 -04:00
Penar Musaraj 4d8011032e
DEV: Add a rake task to export/import translation overrides (#18487)
Use `bin/rake export:translation_overrides` to export to a file. Then,
copy that file to a new site and run `bin/rake import:file["filename"].
2022-10-05 15:22:16 -04:00
Dan Gebhardt 03b7b7d1bc
DEV: Remove usage of {{action}} modifiers - Take 2 (#18476)
This PR enables the [`no-action-modifiers`](https://github.com/ember-template-lint/ember-template-lint/blob/master/docs/rule/no-action-modifiers.md) template lint rule and removes all usages of the `{{action}}` modifier in core.

In general, instances of `{{action "x"}}` have been replaced with `{{on "click" (action "x")}}`. 

In many cases, such as for `a` elements, we also need to prevent default event handling to avoid unwanted side effects. While the `{{action}}` modifier internally calls `event.preventDefault()`, we need to handle these cases more explicitly. For this purpose, this PR also adds the [ember-event-helpers](https://github.com/buschtoens/ember-event-helpers) dependency so we can use the `prevent-default` handler. For instance:

```
<a href {{on "click" (prevent-default (action "x"))}}>Do X</a>
```

Note that `action` has not in general been refactored away as a helper yet. In general, all event handlers should be methods on the corresponding component and referenced directly (e.g. `{{on "click" this.doSomething}}`). However, the `action` helper is used extensively throughout the codebase and often references methods in the `actions` hash on controllers or routes. Thus this refactor will also be extensive and probably deserves a separate PR.

Note: This work was done to complement #17767 by minimizing the potential impact of the `action` modifier override, which uses private API and arguably should be replaced with an AST transform.

This is a followup to #18333, which had to be reverted because it did not account for the default treatment of modifier keys by the {{action}} modifier.

Commits:
* Enable `no-action-modifiers` template lint rule
* Replace {{action "x"}} with {{on "click" (action "x")}}
* Remove unnecessary action helper usage
* Remove ctl+click tests for user-menu
   These tests now break in Chrome when used with addEventListener. As per the comment, they can probably be safely removed.
* Prevent default event handlers to avoid unwanted side effects
   Uses `event.preventDefault()` in event handlers to prevent default event handling. This had been done automatically by the `action` modifier, but is not always desirable or necessary.
* Restore UserCardContents#showUser action to avoid regression
   By keeping the `showUser` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showUser` argument that's been passed.
* Revert EditCategoryTab#selectTab -> EditCategoryTab#select
   Avoid potential breaking change in themes / plugins
* Restore GroupCardContents#showGroup action to avoid regression
   By keeping the `showGroup` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showGroup` argument that's been passed.
* Restore SecondFactorAddTotp#showSecondFactorKey action to avoid regression
   By keeping the `showSecondFactorKey` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showSecondFactorKey` property that's maintained on the controller.
* Refactor away from `actions` hash in ChooseMessage component
* Modernize EmojiPicker#onCategorySelection usage
* Modernize SearchResultEntry#logClick usage
* Modernize Discovery::Categories#showInserted usage
* Modernize Preferences::Account#resendConfirmationEmail usage
* Modernize MultiSelect::SelectedCategory#onSelectedNameClick usage
* Favor fn over action in SelectedChoice component
* Modernize WizardStep event handlers
* Favor fn over action usage in buttons
* Restore Login#forgotPassword action to avoid possible regression
* Introduce modKeysPressed utility
   Returns an array of modifier keys that are pressed during a given `MouseEvent` or `KeyboardEvent`.
* Don't interfere with click events on links with `href` values when modifier keys are pressed
2022-10-05 13:08:54 +01:00
David Taylor 585c584fdb
Revert "DEV: Remove usage of `{{action}}` modifiers (#18333)" (#18469)
This reverts commit ba27ee1637.

We found some issues with handling of cmd/ctrl/shift + click on `<a` elements
2022-10-04 12:27:26 +01:00
Dan Gebhardt ba27ee1637
DEV: Remove usage of `{{action}}` modifiers (#18333)
This PR enables the [`no-action-modifiers`](https://github.com/ember-template-lint/ember-template-lint/blob/master/docs/rule/no-action-modifiers.md) template lint rule and removes all usages of the `{{action}}` modifier in core.

In general, instances of `{{action "x"}}` have been replaced with `{{on "click" (action "x")}}`. 

In many cases, such as for `a` elements, we also need to prevent default event handling to avoid unwanted side effects. While the `{{action}}` modifier internally calls `event.preventDefault()`, we need to handle these cases more explicitly. For this purpose, this PR also adds the [ember-event-helpers](https://github.com/buschtoens/ember-event-helpers) dependency so we can use the `prevent-default` handler. For instance:

```
<a href {{on "click" (prevent-default (action "x"))}}>Do X</a>
```

Note that `action` has not in general been refactored away as a helper yet. In general, all event handlers should be methods on the corresponding component and referenced directly (e.g. `{{on "click" this.doSomething}}`). However, the `action` helper is used extensively throughout the codebase and often references methods in the `actions` hash on controllers or routes. Thus this refactor will also be extensive and probably deserves a separate PR.

Note: This work was done to complement #17767 by minimizing the potential impact of the `action` modifier override, which uses private API and arguably should be replaced with an AST transform.

Commits:
* Enable `no-action-modifiers` template lint rule
* Replace {{action "x"}} with {{on "click" (action "x")}}
* Remove unnecessary action helper usage
* Remove ctl+click tests for user-menu
   These tests now break in Chrome when used with addEventListener. As per the comment, they can probably be safely removed.
* Prevent default event handlers to avoid unwanted side effects
   Uses `event.preventDefault()` in event handlers to prevent default event handling. This had been done automatically by the `action` modifier, but is not always desirable or necessary.
* Restore UserCardContents#showUser action to avoid regression
   By keeping the `showUser` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showUser` argument that's been passed.
* Revert EditCategoryTab#selectTab -> EditCategoryTab#select
   Avoid potential breaking change in themes / plugins
* Restore GroupCardContents#showGroup action to avoid regression
   By keeping the `showGroup` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showGroup` argument that's been passed.
* Restore SecondFactorAddTotp#showSecondFactorKey action to avoid regression
   By keeping the `showSecondFactorKey` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showSecondFactorKey` property that's maintained on the controller.
* Refactor away from `actions` hash in ChooseMessage component
* Modernize EmojiPicker#onCategorySelection usage
* Modernize SearchResultEntry#logClick usage
* Modernize Discovery::Categories#showInserted usage
* Modernize Preferences::Account#resendConfirmationEmail usage
* Modernize MultiSelect::SelectedCategory#onSelectedNameClick usage
* Favor fn over action in SelectedChoice component
* Modernize WizardStep event handlers
* Favor fn over action usage in buttons
* Restore Login#forgotPassword action to avoid possible regression
2022-10-04 10:42:46 +02:00
Martin Brennan 8ebd5edd1e
DEV: Rename secure_media to secure_uploads (#18376)
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.

This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.

Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.

This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.

Deprecated settings:

* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
2022-09-29 09:24:33 +10:00
Joshua Rosenfeld 9662ca7ee4
DEV: Print plugin changes in details blocks (#18345)
Update formatting of the `release_note` rake task. Use `[details]` blocks for each change category so more plugins fit on one screen.
2022-09-23 10:29:00 -04:00
Bianca Nenciu 7d3e763f01
FIX: Query correct model in posts:inline_uploads (#18330)
Follow up to commit 9db8f00b3d.
2022-09-23 15:00:02 +03:00
Keegan George a23d19fab0
DEV: Emoji picker keyboard accessibility updates (#18331)
This PR makes some updates to the prior keyboard accessibility commit (eb98746):
- Makes `tabindex` attribute only appear on emoji markup in the emoji picker.
- After pressing the Esc key, focus returns to the <textarea/> input (composer editor or chat input)
2022-09-22 15:21:34 -07:00
David Taylor 0c743a591f
DEV: Allow `rake qunit` filter to be used alongside parallel option (#18326)
Followup to 61f5c8716d
2022-09-22 10:28:02 +01:00
Sergei Mellow 61f5c8716d
DEV: Update `rake qunit:test` to support filtering (#18249) 2022-09-21 14:00:50 -04:00
David Taylor e06b9d4a52
DEV: Remove support for legacy plugin JS compilation pipeline (#18293)
This became the default in b1755137
2022-09-21 12:38:02 +01:00
David Taylor ce21205d39
DEV: Add support for running plugin qunit in parallel (#18291)
For example, to run in three concurrent browsers and assemble the results:

```
QUNIT_PARALLEL=3 bin/rake "plugin:qunit[discourse-chat]"
```
2022-09-20 18:11:26 +01:00
David Taylor ef39193a06
DEV: Add `rake plugins:turbo_spec` task (#18289)
This leans on our existing `turbo_rspec` implementation to run plugin specs in parallel on all available cores
2022-09-20 15:42:54 +01:00
David Taylor 1bd1664ae0
DEV: Compile markdown-it-bundle with ember-cli (#18104)
We were already compiling the markdown bundle via ember-cli, but that version was only being used in the test environment. This commit improves the implementation, and updates the filename so it's also used in production.

This commit also
- Removes the vendored copy of `markdown-it.js` and fetches from node_modules instead
- Updates `pretty_text.rb` to remove the custom sprockets-manifest-parsing
- Removes `pretty-text-bundle.js`, which was only being used by `pretty_text.rb`
2022-08-29 19:11:59 +01:00
David Taylor 23ec819e70
DEV: Introduce debugging rake task for PostRevision issues (#17998)
See https://meta.discourse.org/t/232885 for context. This rake task enumerates problematic records and prints out a sample of each failure.
2022-08-19 11:30:46 +01:00
David Taylor 14cc67c9b2
DEV: Run `yarn install` for all `rake docker:test` invocations (#17844)
Previously we were only `yarn install`ing for linting and qunit runs. The Rails app now relies on a number of `node_modules` dependencies (e.g. for pretty_text, and discourse_js_processor), so we need to make sure they're available.
2022-08-09 12:28:52 +01:00
Daniel Waterworth e0ece3a77e
Revert "DEV: Improve multisite db scripts in dev (#17337)" (#17801)
This reverts commit 2e4056d185.

Unfortunately, this broke db:create and db:drop when multisite config
wasn't present.
2022-08-04 16:15:06 -05:00
Sérgio Saquetim c99f658a9e
DEV: Allow to specify seed on rake task plugin:spec (#17770)
Allow users to specify the seed of the tests using the env variable RSPEC_SEED

Example:

bundle exec rake "plugin:spec[plugin-name]" RSPEC_SEED=65536
This is useful while fixing flaky tests.
2022-08-02 23:52:53 -03:00
David Taylor 1121062aa9
DEV: Run prettier and rubocop in parallel for CI (#17357) 2022-07-07 13:29:14 +08:00
tshenry d1a15d4f8d
FIX: Should be UploadReference instead of UploadReferences (#17361)
This fixes a couple of typos that were introduced in 9db8f00
2022-07-06 11:40:54 -07:00
Jarek Radosz 2e4056d185
DEV: Improve multisite db scripts in dev (#17337)
## Without multisite.yml config

No change. `bin/rails db:create` / `db:migrate` / `db:drop` should work the same.

## With multisite.yml config

### db:create

`bin/rails db:create` creates development, test, and all databases from the multisite config

`RAILS_DB=[site] bin/rails db:create` creates the database for the specified site from the multisite config

### db:migrate

`bin/rails db:migrate` migrates the development database and all databases from the multisite config

`RAILS_ENV=test bin/rails db:migrate` migrates the test database and `discourse_test_multisite`

`RAILS_DB=[site] bin/rails db:migrate` migrates the database for the specified site from the multisite config

### db:drop

`bin/rails db:drop` drops development, test, and all databases from the multisite config

`RAILS_DB=[site] bin/rails db:create` drops the database for the specified site from the multisite config
2022-07-06 10:39:03 +02:00
Jarek Radosz c5f0aa2f32
DEV: Remove handlebars from the old package.json (#17319) 2022-07-04 15:05:03 +02:00
Jarek Radosz aa7792cf93
DEV: Use npm bootstrap (#17315) 2022-07-04 11:36:51 +02:00
Jarek Radosz a1170970db
FIX: Add wizard.js to ember-cli assets (#17153) 2022-06-20 17:18:35 +02:00
Jarek Radosz 2c1fc28d00
DEV: Remove ember-cli flags from the backend (#17147)
…and other auxiliary code

* Restore `QUNIT_EMBER_CLI` flag warning
* Add `ALLOW_EMBER_CLI_PROXY_BYPASS`
2022-06-20 16:33:05 +02:00
David Taylor 1b4692039e DEV: Remove legacy JS manifests and vendored scripts
Now that we've switched to Ember CLI, these things are no longer used.

- These sprockets manifests are superceded by the assets generated by ember cli
- These vendored scripts are now fetched by ember-auto-import at compile time
2022-06-20 15:01:06 +01:00
David Taylor 12980418ae
DEV: Disable the use of 'legacy' Ember assets (#17127)
Anyone still using `EMBER_CLI_PROD_ASSETS=0` in development or production will be gracefully switched to Ember CLI. In development, a repeated message will be logged to STDERR.

Similarly, passing `QUNIT_EMBER_CLI=0` to the qunit rake task will now do nothing. A warning will be printed, and ember-cli mode will be used. Note that we've chosen not to fail the task, so that existing plugin/theme CI jobs don't immediately start failing. We may switch to a hard fail in the coming days/weeks.
2022-06-17 16:51:28 +01:00
Jarek Radosz 91bc4442d1
DEV: Remove wizard step from docker tests (#17125)
Wizard tests are now in core.
2022-06-17 15:35:18 +02:00
Jarek Radosz fcb4e5a1a1
DEV: Make wizard an ember addon (#17027)
Co-authored-by: David Taylor <david@taylorhq.com>
2022-06-17 14:50:21 +02:00
Gerhard Schlager 888f50543d
DEV: Fix `javascript:update` rake task (#17098)
* The `javascript:update` rake task failed because recent versions of chart.js use a lowercase filename (`chart.min.js` instead of `Chart.min.js`)

* Changed `loadScript()` to use lowercase keys to lookup scripts

* `svg-arrow.css` seems to have changed slightly (linebreak at the end of file)
2022-06-15 19:49:04 +02:00
Gerhard Schlager 96e87af605
FIX: Rake tasks related to uploads were broken (#17085) 2022-06-14 11:05:03 +02:00
Bianca Nenciu 9db8f00b3d
FEATURE: Create upload_references table (#16146)
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.

* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
2022-06-09 09:24:30 +10:00
Daniel Waterworth d7bc395182
FIX: Seed multisite dbs after migrating in development (#17017)
Dumping the schema cache reset the current_db and we only need to do
this once.
2022-06-06 14:43:43 -05:00
Alan Guo Xiang Tan 8471eabf62 DEV: Add env to skip wizard tests 2022-05-25 13:48:09 +08:00
David Taylor 115fe90bff
DEV: Only demux migration stdout if running concurrently (#16895)
This feature only was only demuxing stdout, not stderr. That means that stdout and stderr output appears out-of-order, and makes debugging migrations very confusing.

In future we may want to add stderr support to the demuxing. But right now, the concurrency variable is hard-coded to 1. Therefore the easiest fix is to bypass the demuxing.
2022-05-23 14:26:13 +01:00
Martin Brennan faf5b4d3e9
PERF: Speed up secure media and ACL sync rake tasks (#16849)
Incorporates learnings from /t/64227:

* Changes the code to set access control posts in the rake
  task to be an efficient UPDATE SQL query.
  The original version was timing out with 312017 post uploads,
  the new query took ~3s to run.
* Changes the code to mark uploads as secure/not secure in
  the rake task to be an efficient UPDATE SQL query rather than
  using UploadSecurity. This took a very long time previously,
  and now takes only a few seconds.
* Spread out ACL syncing for uploads into jobs with batches of
  100 uploads at a time, so they can be parallelized instead
  of having to wait ~1.25 seconds for each ACL to be changed
  in S3 serially.

One issue that still remains is post rebaking. Doing this serially
is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates
but this is limited by max_old_rebakes_per_15_minutes. It would
be better to fan this rebaking out into jobs like we did for the
ACL sync, but that should be done in another PR.
2022-05-23 13:14:11 +10:00