Commit Graph

1271 Commits

Author SHA1 Message Date
Martin Brennan 4ab1f76499
DEV: Fix bin/turbo_rspec runtime recording (#20407)
This commit 57caf08e13 broke
`bin/turbo_rspec` timing recording via `TurboTests::Runner`,
because we changed to using all `spec/*` folders except
`spec/system` as default for the runner, rather than
the old `['spec']` array, which is what `TurboTests::Runner`
was relying on to determine whether to record test run
time with `ParallelTests::RSpec::RuntimeLogger`.

Instead, we can just pass a new `use_runtime_info` boolean to the
runner class and use it when running against the default set of
spec files using `bin/turbo_rspec` and the turbo rspec rake task.
2023-02-23 07:47:11 +10:00
Loïc Guitaut f7c57fbc19 DEV: Enable `unless` cops
We discussed the use of `unless` internally and decided to enforce
available rules from rubocop to restrict its most problematic uses.
2023-02-21 10:30:48 +01:00
Osama Sayegh f91631b625
DEV: Update javascript:update_constants rake task following template colocation (#20365)
As of ba3f62f576, handlebars templates are colocated with js files so the path to hbs templates referenced by this rake task is no longer valid. This commit fixes the path to hbs templates and updates a couple of files that are generated by the rake task.
2023-02-20 06:20:47 +03:00
Martin Brennan 60ad836313
DEV: Chat service object initial implementation (#19814)
This is a combined work of Martin Brennan, Loïc Guitaut, and Joffrey Jaffeux.

---

This commit implements a base service object when working in chat. The documentation is available at https://discourse.github.io/discourse/chat/backend/Chat/Service.html

Generating documentation has been made as part of this commit with a bigger goal in mind of generally making it easier to dive into the chat project.

Working with services generally involves 3 parts:

- The service object itself, which is a series of steps where few of them are specialized (model, transaction, policy)

```ruby
class UpdateAge
  include Chat::Service::Base

  model :user, :fetch_user
  policy :can_see_user
  contract
  step :update_age

  class Contract
    attribute :age, :integer
  end

  def fetch_user(user_id:, **)
    User.find_by(id: user_id)
  end

  def can_see_user(guardian:, **)
    guardian.can_see_user(user)
  end

  def update_age(age:, **)
    user.update!(age: age)
  end
end
```

- The `with_service` controller helper, handling success and failure of the service within a service and making easy to return proper response to it from the controller

```ruby
def update
  with_service(UpdateAge) do
    on_success { render_serialized(result.user, BasicUserSerializer, root: "user") }
  end
end
```

- Rspec matchers and steps inspector, improving the dev experience while creating specs for a service

```ruby
RSpec.describe(UpdateAge) do
  subject(:result) do
    described_class.call(guardian: guardian, user_id: user.id, age: age)
  end

  fab!(:user) { Fabricate(:user) }
  fab!(:current_user) { Fabricate(:admin) }

  let(:guardian) { Guardian.new(current_user) }
  let(:age) { 1 }

   it { expect(user.reload.age).to eq(age) }
end
```

Note in case of unexpected failure in your spec, the output will give all the relevant information:

```
  1) UpdateAge when no channel_id is given is expected to fail to find a model named 'user'
     Failure/Error: it { is_expected.to fail_to_find_a_model(:user) }

       Expected model 'foo' (key: 'result.model.user') was not found in the result object.

       [1/4] [model] 'user' 
       [2/4] [policy] 'can_see_user'
       [3/4] [contract] 'default'
       [4/4] [step] 'update_age'

       /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/update_age.rb:32:in `fetch_user': missing keyword: :user_id (ArgumentError)
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:202:in `instance_exec'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:202:in `call'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:219:in `call'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:417:in `block in run!'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:417:in `each'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:417:in `run!'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:411:in `run'
       	from <internal:kernel>:90:in `tap'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/app/services/base.rb:302:in `call'
       	from /Users/joffreyjaffeux/Code/pr-discourse/plugins/chat/spec/services/update_age_spec.rb:15:in `block (3 levels) in <main>'
```
2023-02-13 13:09:57 +01:00
Ted Johansson 25a226279a
DEV: Replace #pluck_first freedom patch with AR #pick in core (#19893)
The #pluck_first freedom patch, first introduced by @danielwaterworth has served us well, and is used widely throughout both core and plugins. It seems to have been a common enough use case that Rails 6 introduced it's own method #pick with the exact same implementation. This allows us to retire the freedom patch and switch over to the built-in ActiveRecord method.

There is no replacement for #pluck_first!, but a quick search shows we are using this in a very limited capacity, and in some cases incorrectly (by assuming a nil return rather than an exception), which can quite easily be replaced with #pick plus some extra handling.
2023-02-13 12:39:45 +08:00
Keegan George b677bb6f24
DEV: Add `yaml` support to `<AceEditor />` (#20198) 2023-02-07 12:49:12 -08:00
Daniel Waterworth 666536cbd1
DEV: Prefer \A and \z over ^ and $ in regexes (#19936) 2023-01-20 12:52:49 -06:00
Martin Brennan 4d2a95ffe6
FIX: Query UploadReference in UploadSecurity for existing uploads (#19917)
This fixes a longstanding issue for sites with the
secure_uploads setting enabled. What would happen is a scenario
like this, since we did not check all places an upload could be
linked to whenever we used UploadSecurity to check whether an
upload should be secure:

* Upload is created and used for site setting, set to secure: false
  since site setting uploads should not be secure. Let's say favicon
* Favicon for the site is used inside a post in a private category,
  e.g. via a Onebox
* We changed the secure status for the upload to true, since it's been
  used in a private category and we don't check if it's originator
  was a public place
* The site favicon breaks :'(

This was a source of constant consternation. Now, when an upload is _not_
being created, and we are checking if an existing upload should be
secure, we now check to see what the first record in the UploadReference
table is for that upload. If it's something public like a site setting,
then we will never change the upload to `secure`.
2023-01-20 10:24:52 +10:00
Martin Brennan 56a93f7532
FEATURE: Add rake task to mark old hashtag format for rebake (#19876)
Since the new hashtag format has been added, we want site
admins to be able to rebake old posts with the old hashtag
format. This can now be done with `rake hashtags:mark_old_format_for_rebake`
which goes and marks posts with the old cooked version of hashtags
in this format for rebake:

```
<a class=\"hashtag\" href=\"/c/ux/14\">#<span>ux</span></a>
```

c.f. https://meta.discourse.org/t/what-rebake-is-required-for-the-new-autocomplete-styling/249642/12
2023-01-18 10:16:05 +10:00
David Taylor 6417173082
DEV: Apply syntax_tree formatting to `lib/*` 2023-01-09 12:10:19 +00:00
David Taylor eebe2f256c
DEV: Skip s3 asset deletion when in readonly mode (#19611)
In some situations (e.g. disaster recovery), it may make sense to spin up a temporary readonly version of a cluster. In that situation, the s3 `expire_missing_assets` job would delete assets which are still in use by the canonical read-write version of the cluster.

To avoid that, this commit will skip deletion if the site is currently in readonly mode.
2022-12-30 11:35:40 +00:00
Alan Guo Xiang Tan 0da79561c3
DEV: Improve/Fix script/bench.rb (#19646)
1. Fix bug where we were not waiting for all unicorn workers to start up
before running benchmarks.

2. Fix a bug where headers were not used when benchmarking. Admin
benchmarks were basically running as anon user.

3. Disable rate limits when in profile env. We're pretty much going to
hit the rate limit every time as a normal user.

4. Benchmark against topic with a fixed posts count of 100. Previously profiling script was just randomly creating posts
and we would benchmark against a topic with a fixed posts count of 30.
Sometimes, the script fails because no topics with a posts count of 30
exists.

5. Benchmarks are not run against a normal user on top of anon and
admin.

6. Add script option to select tests that should be run.
2022-12-30 07:25:11 +08:00
David Taylor 104a16610a DEV: Configure prettier for hbs templates 2022-12-28 13:11:12 +00:00
David Taylor a6af981e1b
DEV: Add retry logic to `plugin:install_all_official` (#19551)
This task sometimes fails in CI due to temporary network issues. Retrying twice should help resolve those situations without needing to manually restart the job.
2022-12-21 15:31:03 +00:00
David Taylor ca72d8d030
PERF: Adjust node memory threshold for assets:precompile (#19040)
Previously we were forcing node's max-old-space-size to be 2GB. This override was added in a01b1dd6 to avoid issues caused by a lower default node heap_size_limit on machines with less memory.

This commit makes that `max-old-space-size` override more specific so that it only applies to machines with less memory. Other machines will go use Node's defaults.

The override is also lowered to 1GB. This is still high enough for the build to complete, while reducing memory usage.

https://meta.discourse.org/t/245547
2022-11-15 22:02:13 +00:00
Jarek Radosz e54a3d5ea9
DEV: Add `START_ID` to `uploads:downsize` task (#18992) 2022-11-11 22:51:48 +01:00
Jarek Radosz dc8a7e74f4
FIX: Allow attr updates of over-size-limit uploads (#18986) 2022-11-11 17:56:11 +01:00
Jarek Radosz bc22fe4fdf
DEV: Convert the downsizing script to a rake task (#18976)
…to make it testable!
2022-11-11 13:00:44 +01:00
David Taylor 8700c5ee6b
PERF: Make stylesheet hashes consistent between deploys (#18909)
Previously the stylesheet cachebusting hash was based on the maximum mtime of files. This works well in development and during in-container updates (e.g. via docker_manager). However, when a fresh docker image is created for each deploy, the file mtimes will change even if the contents has not.

This commit changes the production logic to calculate the cachebuster from the filenames and contents of the relevant assets. This should be consistent across deploys, thereby improving cache hits and improving page load times.
2022-11-07 16:13:35 +00:00
David Taylor f30f9ec5d9
PERF: Update `s3:expire_missing_assets` to delete in batches (#18908)
Some sites may have thousands of stale assets - deleting them one-by-one is very slow.

Followup to e8570b5cc9
2022-11-07 12:53:14 +00:00
David Taylor e8570b5cc9
Fix and improve `s3:expire_missing_assets` task (#18863)
- Ensure it works with prefixed S3 buckets
- Perform a sanity check that all current assets are present on S3 before starting deletion
- Remove the lifecycle rule configuration and delete expired assets immediately. This task should be run post-deploy anyway, so adding a 10-day window is not required
2022-11-07 10:44:45 +00:00
Vinoth Kannan dea44ec923
FEATURE: new site setting to hide user profiles by default. (#18864)
Previously, we didn't have a site-wide setting to set the default behavior for user profile visibility and user presence features. But we already have a user preference for that.
2022-11-06 16:44:17 +05:30
Daniel Waterworth 6f48502dd2
DEV: Add rake task to install a theme from an archive (#18890) 2022-11-04 16:42:07 -05:00
David Taylor b18ebc9648
PERF: Correct should_skip? logic in `s3:upload` (#18862)
This task is supposed to skip uploading if the asset is already present in S3. However, when a bucket 'folder path' was configured, this logic was broken and so the assets would be re-uploaded every time.

This commit fixes that logic to include the bucket 'folder path' in the check
2022-11-04 17:50:46 +00:00
Joffrey JAFFEUX 11f3618b80
DEV: initial system tests for chat and plugins (#18881)
This is a very basic to ensure it's working and open future possible work
2022-11-04 15:06:24 +01:00
Daniel Waterworth 167181f4b7
DEV: Quote values when constructing SQL (#18827)
All of these cases should already be safe, but still good to quote for
"defense in depth".
2022-11-01 14:05:13 -05:00
David Taylor 1da5aa9027
DEV: Parallelize core JS tests in docker.rake (#18756) 2022-10-26 12:00:35 +01:00
Jan Cernik 08476f17ff
FEATURE: Add dark mode option for category logos (#18460)
Adds a new upload field for a second dark mode category logo. 
This alternative will be used when the browser is in dark mode (similar to the global site setting for a dark logo).
2022-10-07 11:00:44 -04:00
Penar Musaraj 4d8011032e
DEV: Add a rake task to export/import translation overrides (#18487)
Use `bin/rake export:translation_overrides` to export to a file. Then,
copy that file to a new site and run `bin/rake import:file["filename"].
2022-10-05 15:22:16 -04:00
Dan Gebhardt 03b7b7d1bc
DEV: Remove usage of {{action}} modifiers - Take 2 (#18476)
This PR enables the [`no-action-modifiers`](https://github.com/ember-template-lint/ember-template-lint/blob/master/docs/rule/no-action-modifiers.md) template lint rule and removes all usages of the `{{action}}` modifier in core.

In general, instances of `{{action "x"}}` have been replaced with `{{on "click" (action "x")}}`. 

In many cases, such as for `a` elements, we also need to prevent default event handling to avoid unwanted side effects. While the `{{action}}` modifier internally calls `event.preventDefault()`, we need to handle these cases more explicitly. For this purpose, this PR also adds the [ember-event-helpers](https://github.com/buschtoens/ember-event-helpers) dependency so we can use the `prevent-default` handler. For instance:

```
<a href {{on "click" (prevent-default (action "x"))}}>Do X</a>
```

Note that `action` has not in general been refactored away as a helper yet. In general, all event handlers should be methods on the corresponding component and referenced directly (e.g. `{{on "click" this.doSomething}}`). However, the `action` helper is used extensively throughout the codebase and often references methods in the `actions` hash on controllers or routes. Thus this refactor will also be extensive and probably deserves a separate PR.

Note: This work was done to complement #17767 by minimizing the potential impact of the `action` modifier override, which uses private API and arguably should be replaced with an AST transform.

This is a followup to #18333, which had to be reverted because it did not account for the default treatment of modifier keys by the {{action}} modifier.

Commits:
* Enable `no-action-modifiers` template lint rule
* Replace {{action "x"}} with {{on "click" (action "x")}}
* Remove unnecessary action helper usage
* Remove ctl+click tests for user-menu
   These tests now break in Chrome when used with addEventListener. As per the comment, they can probably be safely removed.
* Prevent default event handlers to avoid unwanted side effects
   Uses `event.preventDefault()` in event handlers to prevent default event handling. This had been done automatically by the `action` modifier, but is not always desirable or necessary.
* Restore UserCardContents#showUser action to avoid regression
   By keeping the `showUser` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showUser` argument that's been passed.
* Revert EditCategoryTab#selectTab -> EditCategoryTab#select
   Avoid potential breaking change in themes / plugins
* Restore GroupCardContents#showGroup action to avoid regression
   By keeping the `showGroup` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showGroup` argument that's been passed.
* Restore SecondFactorAddTotp#showSecondFactorKey action to avoid regression
   By keeping the `showSecondFactorKey` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showSecondFactorKey` property that's maintained on the controller.
* Refactor away from `actions` hash in ChooseMessage component
* Modernize EmojiPicker#onCategorySelection usage
* Modernize SearchResultEntry#logClick usage
* Modernize Discovery::Categories#showInserted usage
* Modernize Preferences::Account#resendConfirmationEmail usage
* Modernize MultiSelect::SelectedCategory#onSelectedNameClick usage
* Favor fn over action in SelectedChoice component
* Modernize WizardStep event handlers
* Favor fn over action usage in buttons
* Restore Login#forgotPassword action to avoid possible regression
* Introduce modKeysPressed utility
   Returns an array of modifier keys that are pressed during a given `MouseEvent` or `KeyboardEvent`.
* Don't interfere with click events on links with `href` values when modifier keys are pressed
2022-10-05 13:08:54 +01:00
David Taylor 585c584fdb
Revert "DEV: Remove usage of `{{action}}` modifiers (#18333)" (#18469)
This reverts commit ba27ee1637.

We found some issues with handling of cmd/ctrl/shift + click on `<a` elements
2022-10-04 12:27:26 +01:00
Dan Gebhardt ba27ee1637
DEV: Remove usage of `{{action}}` modifiers (#18333)
This PR enables the [`no-action-modifiers`](https://github.com/ember-template-lint/ember-template-lint/blob/master/docs/rule/no-action-modifiers.md) template lint rule and removes all usages of the `{{action}}` modifier in core.

In general, instances of `{{action "x"}}` have been replaced with `{{on "click" (action "x")}}`. 

In many cases, such as for `a` elements, we also need to prevent default event handling to avoid unwanted side effects. While the `{{action}}` modifier internally calls `event.preventDefault()`, we need to handle these cases more explicitly. For this purpose, this PR also adds the [ember-event-helpers](https://github.com/buschtoens/ember-event-helpers) dependency so we can use the `prevent-default` handler. For instance:

```
<a href {{on "click" (prevent-default (action "x"))}}>Do X</a>
```

Note that `action` has not in general been refactored away as a helper yet. In general, all event handlers should be methods on the corresponding component and referenced directly (e.g. `{{on "click" this.doSomething}}`). However, the `action` helper is used extensively throughout the codebase and often references methods in the `actions` hash on controllers or routes. Thus this refactor will also be extensive and probably deserves a separate PR.

Note: This work was done to complement #17767 by minimizing the potential impact of the `action` modifier override, which uses private API and arguably should be replaced with an AST transform.

Commits:
* Enable `no-action-modifiers` template lint rule
* Replace {{action "x"}} with {{on "click" (action "x")}}
* Remove unnecessary action helper usage
* Remove ctl+click tests for user-menu
   These tests now break in Chrome when used with addEventListener. As per the comment, they can probably be safely removed.
* Prevent default event handlers to avoid unwanted side effects
   Uses `event.preventDefault()` in event handlers to prevent default event handling. This had been done automatically by the `action` modifier, but is not always desirable or necessary.
* Restore UserCardContents#showUser action to avoid regression
   By keeping the `showUser` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showUser` argument that's been passed.
* Revert EditCategoryTab#selectTab -> EditCategoryTab#select
   Avoid potential breaking change in themes / plugins
* Restore GroupCardContents#showGroup action to avoid regression
   By keeping the `showGroup` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showGroup` argument that's been passed.
* Restore SecondFactorAddTotp#showSecondFactorKey action to avoid regression
   By keeping the `showSecondFactorKey` action, we can avoid a breaking change for plugins that rely upon it, while not interfering with the `showSecondFactorKey` property that's maintained on the controller.
* Refactor away from `actions` hash in ChooseMessage component
* Modernize EmojiPicker#onCategorySelection usage
* Modernize SearchResultEntry#logClick usage
* Modernize Discovery::Categories#showInserted usage
* Modernize Preferences::Account#resendConfirmationEmail usage
* Modernize MultiSelect::SelectedCategory#onSelectedNameClick usage
* Favor fn over action in SelectedChoice component
* Modernize WizardStep event handlers
* Favor fn over action usage in buttons
* Restore Login#forgotPassword action to avoid possible regression
2022-10-04 10:42:46 +02:00
Martin Brennan 8ebd5edd1e
DEV: Rename secure_media to secure_uploads (#18376)
This commit renames all secure_media related settings to secure_uploads_* along with the associated functionality.

This is being done because "media" does not really cover it, we aren't just doing this for images and videos etc. but for all uploads in the site.

Additionally, in future we want to secure more types of uploads, and enable a kind of "mixed mode" where some uploads are secure and some are not, so keeping media in the name is just confusing.

This also keeps compatibility with the `secure-media-uploads` path, and changes new
secure URLs to be `secure-uploads`.

Deprecated settings:

* secure_media -> secure_uploads
* secure_media_allow_embed_images_in_emails -> secure_uploads_allow_embed_images_in_emails
* secure_media_max_email_embed_image_size_kb -> secure_uploads_max_email_embed_image_size_kb
2022-09-29 09:24:33 +10:00
Joshua Rosenfeld 9662ca7ee4
DEV: Print plugin changes in details blocks (#18345)
Update formatting of the `release_note` rake task. Use `[details]` blocks for each change category so more plugins fit on one screen.
2022-09-23 10:29:00 -04:00
Bianca Nenciu 7d3e763f01
FIX: Query correct model in posts:inline_uploads (#18330)
Follow up to commit 9db8f00b3d.
2022-09-23 15:00:02 +03:00
Keegan George a23d19fab0
DEV: Emoji picker keyboard accessibility updates (#18331)
This PR makes some updates to the prior keyboard accessibility commit (eb98746):
- Makes `tabindex` attribute only appear on emoji markup in the emoji picker.
- After pressing the Esc key, focus returns to the <textarea/> input (composer editor or chat input)
2022-09-22 15:21:34 -07:00
David Taylor 0c743a591f
DEV: Allow `rake qunit` filter to be used alongside parallel option (#18326)
Followup to 61f5c8716d
2022-09-22 10:28:02 +01:00
Sergei Mellow 61f5c8716d
DEV: Update `rake qunit:test` to support filtering (#18249) 2022-09-21 14:00:50 -04:00
David Taylor e06b9d4a52
DEV: Remove support for legacy plugin JS compilation pipeline (#18293)
This became the default in b1755137
2022-09-21 12:38:02 +01:00
David Taylor ce21205d39
DEV: Add support for running plugin qunit in parallel (#18291)
For example, to run in three concurrent browsers and assemble the results:

```
QUNIT_PARALLEL=3 bin/rake "plugin:qunit[discourse-chat]"
```
2022-09-20 18:11:26 +01:00
David Taylor ef39193a06
DEV: Add `rake plugins:turbo_spec` task (#18289)
This leans on our existing `turbo_rspec` implementation to run plugin specs in parallel on all available cores
2022-09-20 15:42:54 +01:00
David Taylor 1bd1664ae0
DEV: Compile markdown-it-bundle with ember-cli (#18104)
We were already compiling the markdown bundle via ember-cli, but that version was only being used in the test environment. This commit improves the implementation, and updates the filename so it's also used in production.

This commit also
- Removes the vendored copy of `markdown-it.js` and fetches from node_modules instead
- Updates `pretty_text.rb` to remove the custom sprockets-manifest-parsing
- Removes `pretty-text-bundle.js`, which was only being used by `pretty_text.rb`
2022-08-29 19:11:59 +01:00
David Taylor 23ec819e70
DEV: Introduce debugging rake task for PostRevision issues (#17998)
See https://meta.discourse.org/t/232885 for context. This rake task enumerates problematic records and prints out a sample of each failure.
2022-08-19 11:30:46 +01:00
David Taylor 14cc67c9b2
DEV: Run `yarn install` for all `rake docker:test` invocations (#17844)
Previously we were only `yarn install`ing for linting and qunit runs. The Rails app now relies on a number of `node_modules` dependencies (e.g. for pretty_text, and discourse_js_processor), so we need to make sure they're available.
2022-08-09 12:28:52 +01:00
Daniel Waterworth e0ece3a77e
Revert "DEV: Improve multisite db scripts in dev (#17337)" (#17801)
This reverts commit 2e4056d185.

Unfortunately, this broke db:create and db:drop when multisite config
wasn't present.
2022-08-04 16:15:06 -05:00
Sérgio Saquetim c99f658a9e
DEV: Allow to specify seed on rake task plugin:spec (#17770)
Allow users to specify the seed of the tests using the env variable RSPEC_SEED

Example:

bundle exec rake "plugin:spec[plugin-name]" RSPEC_SEED=65536
This is useful while fixing flaky tests.
2022-08-02 23:52:53 -03:00
David Taylor 1121062aa9
DEV: Run prettier and rubocop in parallel for CI (#17357) 2022-07-07 13:29:14 +08:00
tshenry d1a15d4f8d
FIX: Should be UploadReference instead of UploadReferences (#17361)
This fixes a couple of typos that were introduced in 9db8f00
2022-07-06 11:40:54 -07:00
Jarek Radosz 2e4056d185
DEV: Improve multisite db scripts in dev (#17337)
## Without multisite.yml config

No change. `bin/rails db:create` / `db:migrate` / `db:drop` should work the same.

## With multisite.yml config

### db:create

`bin/rails db:create` creates development, test, and all databases from the multisite config

`RAILS_DB=[site] bin/rails db:create` creates the database for the specified site from the multisite config

### db:migrate

`bin/rails db:migrate` migrates the development database and all databases from the multisite config

`RAILS_ENV=test bin/rails db:migrate` migrates the test database and `discourse_test_multisite`

`RAILS_DB=[site] bin/rails db:migrate` migrates the database for the specified site from the multisite config

### db:drop

`bin/rails db:drop` drops development, test, and all databases from the multisite config

`RAILS_DB=[site] bin/rails db:create` drops the database for the specified site from the multisite config
2022-07-06 10:39:03 +02:00
Jarek Radosz c5f0aa2f32
DEV: Remove handlebars from the old package.json (#17319) 2022-07-04 15:05:03 +02:00
Jarek Radosz aa7792cf93
DEV: Use npm bootstrap (#17315) 2022-07-04 11:36:51 +02:00
Jarek Radosz a1170970db
FIX: Add wizard.js to ember-cli assets (#17153) 2022-06-20 17:18:35 +02:00
Jarek Radosz 2c1fc28d00
DEV: Remove ember-cli flags from the backend (#17147)
…and other auxiliary code

* Restore `QUNIT_EMBER_CLI` flag warning
* Add `ALLOW_EMBER_CLI_PROXY_BYPASS`
2022-06-20 16:33:05 +02:00
David Taylor 1b4692039e DEV: Remove legacy JS manifests and vendored scripts
Now that we've switched to Ember CLI, these things are no longer used.

- These sprockets manifests are superceded by the assets generated by ember cli
- These vendored scripts are now fetched by ember-auto-import at compile time
2022-06-20 15:01:06 +01:00
David Taylor 12980418ae
DEV: Disable the use of 'legacy' Ember assets (#17127)
Anyone still using `EMBER_CLI_PROD_ASSETS=0` in development or production will be gracefully switched to Ember CLI. In development, a repeated message will be logged to STDERR.

Similarly, passing `QUNIT_EMBER_CLI=0` to the qunit rake task will now do nothing. A warning will be printed, and ember-cli mode will be used. Note that we've chosen not to fail the task, so that existing plugin/theme CI jobs don't immediately start failing. We may switch to a hard fail in the coming days/weeks.
2022-06-17 16:51:28 +01:00
Jarek Radosz 91bc4442d1
DEV: Remove wizard step from docker tests (#17125)
Wizard tests are now in core.
2022-06-17 15:35:18 +02:00
Jarek Radosz fcb4e5a1a1
DEV: Make wizard an ember addon (#17027)
Co-authored-by: David Taylor <david@taylorhq.com>
2022-06-17 14:50:21 +02:00
Gerhard Schlager 888f50543d
DEV: Fix `javascript:update` rake task (#17098)
* The `javascript:update` rake task failed because recent versions of chart.js use a lowercase filename (`chart.min.js` instead of `Chart.min.js`)

* Changed `loadScript()` to use lowercase keys to lookup scripts

* `svg-arrow.css` seems to have changed slightly (linebreak at the end of file)
2022-06-15 19:49:04 +02:00
Gerhard Schlager 96e87af605
FIX: Rake tasks related to uploads were broken (#17085) 2022-06-14 11:05:03 +02:00
Bianca Nenciu 9db8f00b3d
FEATURE: Create upload_references table (#16146)
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.

* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
2022-06-09 09:24:30 +10:00
Daniel Waterworth d7bc395182
FIX: Seed multisite dbs after migrating in development (#17017)
Dumping the schema cache reset the current_db and we only need to do
this once.
2022-06-06 14:43:43 -05:00
Alan Guo Xiang Tan 8471eabf62 DEV: Add env to skip wizard tests 2022-05-25 13:48:09 +08:00
David Taylor 115fe90bff
DEV: Only demux migration stdout if running concurrently (#16895)
This feature only was only demuxing stdout, not stderr. That means that stdout and stderr output appears out-of-order, and makes debugging migrations very confusing.

In future we may want to add stderr support to the demuxing. But right now, the concurrency variable is hard-coded to 1. Therefore the easiest fix is to bypass the demuxing.
2022-05-23 14:26:13 +01:00
Martin Brennan faf5b4d3e9
PERF: Speed up secure media and ACL sync rake tasks (#16849)
Incorporates learnings from /t/64227:

* Changes the code to set access control posts in the rake
  task to be an efficient UPDATE SQL query.
  The original version was timing out with 312017 post uploads,
  the new query took ~3s to run.
* Changes the code to mark uploads as secure/not secure in
  the rake task to be an efficient UPDATE SQL query rather than
  using UploadSecurity. This took a very long time previously,
  and now takes only a few seconds.
* Spread out ACL syncing for uploads into jobs with batches of
  100 uploads at a time, so they can be parallelized instead
  of having to wait ~1.25 seconds for each ACL to be changed
  in S3 serially.

One issue that still remains is post rebaking. Doing this serially
is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates
but this is limited by max_old_rebakes_per_15_minutes. It would
be better to fan this rebaking out into jobs like we did for the
ACL sync, but that should be done in another PR.
2022-05-23 13:14:11 +10:00
Penar Musaraj 18ebe2fc28
DEV: Add ability to populate private messages (#16705) 2022-05-12 14:29:49 -04:00
Bianca Nenciu 5165fb638e
FIX: Show error message if extensions cannot be created (#16719)
It used to stop the db:migrate task and broke sites that are deployed
in non-standard environments (using external database server or older
versions).
2022-05-11 15:46:49 +03:00
David Taylor 476bd1d237
DEV: Fix production sourcemaps with Ember CLI (#16707)
22a7905f restructured how we load Ember CLI assets in production. Unfortunately, it also broke sourcemaps for those assets. This commit fixes that regression via a couple of changes:

- It adds the necessary `.map` paths to `config.assets.precompile`
- It swaps Sprockets' default `SourcemappingUrlProcessor` with an extended version which maintains relative URLs of maps
2022-05-11 10:23:32 +01:00
Bianca Nenciu 631bfb2a23
FIX: Create PostgreSQL extensions before migrating (#16158)
Enabled extensions are not saved in backups and thus not created when
restoring a newer backup (that has a new extension) to an old site
(that does not have the migration).
2022-05-11 09:48:03 +03:00
Sam 2df3c65ba9
FIX: add support for pipelined and multi redis commands (#16682)
Latest redis interoduces a block form of multi / pipelined, this was incorrectly
passed through and not namespaced.

Fix also updates logster, we held off on upgrading it due to missing functions
2022-05-10 08:19:02 +10:00
Jarek Radosz 68ccaa3acb
DEV: Fix typos and outdated comments (#16614) 2022-05-04 14:12:18 +08:00
Joffrey JAFFEUX fad94160c7
FIX: uses tippy for popover (#15409)
Note this commit also introduce a new {{d-popover}} component, example usage:

```hbs
{{#d-popover |state|}}
  {{d-button label="foo.things" class="d-popover-trigger"}}

  <div class="d-popover-content">
    Some content
  <div>
{{/d-popover}}
```
2022-05-02 17:10:26 +02:00
Loïc Guitaut 008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
David Taylor 127ba698a7
DEV: Allow running theme-qunit tests via testem (#16540)
This allows `QUNIT_EMBER_CLI=1 bin/rake theme:qunit[...]` to test themes using `testem` with Ember-CLI-generated assets
2022-04-22 15:04:01 +01:00
Jarek Radosz 1209d51a7f
DEV: Don't use `Dir.chdir()` (#16535)
`system()` provides `chdir:` option for that.
2022-04-22 11:17:10 +02:00
David Taylor 22a7905f2d
DEV: Allow Ember CLI assets to be used by development Rails app (#16511)
Previously, accessing the Rails app directly in development mode would give you assets from our 'legacy' Ember asset pipeline. The only way to run with Ember CLI assets was to run ember-cli as a proxy. This was quite limiting when working on things which are bypassed when using the ember-cli proxy (e.g. changes to `application.html.erb`). Also, since `ember-auto-import` introduced chunking, visiting `/theme-qunit` under Ember CLI was failing to include all necessary chunks.

This commit teaches Sprockets about our Ember CLI assets so that they can be used in development mode, and are automatically collected up under `/public/assets` during `assets:precompile`. As a bonus, this allows us to remove all the custom manifest modification from `assets:precompile`.

The key changes are:
- Introduce a shared `EmberCli.enabled?` helper
- When ember-cli is enabled, add ember-cli `/dist/assets` as the top-priority Rails asset directory
- Have ember-cli output a `chunks.json` manifest, and teach `preload_script` to read it and append the correct chunks to their associated `afterFile`
- Remove most custom ember-cli logic from the `assets:precompile` step. Instead, rely on Rails to take care of pulling the 'precompiled' assets into the `public/assets` directory. Move the 'renaming' logic to runtime, so it can be used in development mode as well.
- Remove fingerprinting from `ember-cli-build`, and allow Rails to take care of things

Long-term, we may want to replace Sprockets with the lighter-weight Propshaft. The changes made in this commit have been made with that long-term goal in mind.

tldr: when you visit the rails app directly, you'll now be served the current ember-cli assets. To keep these up-to-date make sure either `ember serve`, or `ember build --watch` is running. If you really want to load the old non-ember-cli assets, then you should start the server with `EMBER_CLI_PROD_ASSETS=0`. (the legacy asset pipeline will be removed very soon)
2022-04-21 16:26:34 +01:00
Jarek Radosz 69dbc207ab
DEV: Improve `plugin:versions` task (#16391)
1. Sort plugins by name
2. Include plugins that are a symbolic link to a submodule repo (in those cases `.git` isn't a directory but a file that looks like e.g. `gitdir: ../../.git/modules/plugins/name-here`)
2022-04-06 18:29:39 +02:00
Jarek Radosz 9060a045c2
DEV: Minor linting fixes (#16359)
1. `test/run-qunit.js` wasn't eslinted (I'm not adding it to the CI workflow for now, just fixed the issues)
2. "…" utf character isn't rendered correctly in Jenkins, replaced with three dots
3. Don't try to lint `tmp` when doing `eslint .` in the root dir
2022-04-02 21:15:38 +02:00
David Taylor 9df28fe4b3
DEV: Only raise `rake themes:update` errors when flag provided (#16254)
Switching behavior based on multisite/single-site configuration can create some difficult-to-debug situations. The flag is much more obvious.
2022-03-22 17:02:14 +00:00
Jarek Radosz e4a92a8113
DEV: Add `SKIP_DB_CREATE` option to docker:test (#16225)
So that the database creation can be run separately from the specs.
2022-03-21 13:21:06 +01:00
Jarek Radosz bf8dc394bd
DEV: Add chromium to ChromeInstalledChecker (#16224) 2022-03-19 11:00:06 +01:00
David Taylor c58c2f53e8
FIX: Skip ember-cli build memory optimisation for multiple rake tasks (#16198) 2022-03-16 12:02:21 +00:00
Jarek Radosz dec68d780c
DEV: Don't install plugin gems twice (#16192)
Missing plugin gems are installed when the app is being loaded.

That means when you run `bin/rails plugin:install_all_gems` it first installs missing gems and then reinstalls all gems…

Also, the method these rake tasks were using to install gems was very crude, and the regex there was incorrect which resulted in failures in certain cases. Though that didn't matter since those gems were being installed using a correct method just moments before…
2022-03-15 15:08:05 +01:00
David Taylor ea1865cc01
DEV: Ensure yarn dependencies are installed during docker.rake (#16185)
Followup to 349a4acc9f
2022-03-14 17:53:21 +00:00
David Taylor 349a4acc9f
DEV: Use Ember CLI for docker.rake CI testing (#16182) 2022-03-14 17:32:03 +00:00
David Taylor 21cc57674c
FIX: Do not run Ember CLI chunk files through babel twice (#16183) 2022-03-14 16:11:13 +00:00
Jarek Radosz 8d5281115e
DEV: Use the --no-rebase option of `git pull` (#16180)
…to avoid repeatedly printed notes:

```
hint: Pulling without specifying how to reconcile divergent branches is
hint: discouraged. You can squelch this message by running one of the following
hint: commands sometime before your next pull:
hint:
hint:   git config pull.rebase false  # merge (the default strategy)
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
```
2022-03-14 15:13:06 +01:00
Jarek Radosz 3db4b3bc5e
DEV: Drop lodash (#16110) 2022-03-06 18:15:25 +01:00
Jarek Radosz f9e1ba4f8f
DEV: Remove remnants of nginx-perf-report plugin (#16107)
The plugin is no longer official as of Nov 18, 2019 (e2ccb0c608)
2022-03-05 17:22:23 +01:00
David Taylor 9415ec91a1
DEV: Make load order consistent for `rake plugin:spec` (#16076)
`Dir.glob` doesn't guarantee any particular order for results. However, it does appear to be consistent on a given machine. This means that specs can consistently pass on one machine while consistently failing on another. This can lead to some very confusing situations!

This commit sorts the spec files alphabetically so that load order is consistent across environments.

Note that the order in which tests are **run** is not affected by this change. Run order is still randomized by RSpec
2022-03-01 17:46:47 +00:00
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Jarek Radosz 29e601c13a
DEV: Fix the javascript:update rake task (#16057)
The source of the moment tz name package has changed in #15761
2022-02-26 02:06:12 +01:00
Dan Ungureanu ae5eab06ad
FEATURE: Add ENV to control themes:update errors (#16051)
This commit introduces a new environment variable `RAISE_THEME_ERRORS`
that can control what happens when `theme:update` Rake task errors. It
can have three possible values: `0` to always print errors, `1` to
always raise on error, or be absent to use the default behavior which
raises errors only for default sites.
2022-02-25 11:54:42 +02:00
Joffrey JAFFEUX 8244b4b163
DEV: prevents warnings with EMBER_CLI being redefined (#16035)
Example error:

```
/__w/discourse/discourse/lib/tasks/assets.rake:3: warning: already initialized constant EMBER_CLI
/__w/discourse/discourse/lib/tasks/assets.rake:3: warning: previous definition of EMBER_CLI was here
```
2022-02-23 10:51:31 +01:00
David Taylor 6ab4d26d84
PERF: Terminate `rake` process during `ember build` (#15920)
The `assets:precompile` rake task loads the full Ruby app, which can consume around 500mb of RAM by itself. Using `exec` to run `ember build` allows us to free up the Ruby memory and make more space for `ember build`
2022-02-14 11:49:46 +00:00
David Taylor a01b1dd648
PERF: Update ember-auto-import and webpack (#15919)
This makes a small improvement to 'cold cache' ember-cli build times, and a large improvement to 'warm cache' build times

The ember-auto-import update means that vendor is now split into multiple files for efficiency. These are named `chunk.*`, and should be included immediately after the `vendor.js` file. This commit also updates the rails app to render script tags for these chunks.

This change was previously merged, and caused memory-related errors on RAM-constrained machines. This was because Webpack 5 switches from multiple worker processes to a single multi-threaded process. This meant that it was hitting node's default heap size limit (~500mb on a 1GB RAM server). Discourse's standard install procedure recommends adding 2GB swap to 1GB-RAM machines, so we can afford to override's Node's default via the `--max-old-space-size` flag.
2022-02-14 11:21:39 +00:00
David Taylor 8025768fd2
DEV: Embed source content in JS sourcemaps (#15914)
This removes the need for a further request for the 'original' source file, and brings our custom terser arguments in line with the ember-cli defaults
2022-02-11 20:38:53 +00:00
David Taylor 51b88c0e2c
DEV: Ensure Ember CLI sourcemaps are uploaded to S3 (#15893)
Ember CLI gives sourcemaps their own digest. Our `s3.rake` logic assumes that the digest portion of sourcemap filenames remains the same.

The Ember CLI sourcemaps are included in the manifest file, so we can ensure they are uploaded by letting them past the MiniMime check.

Followup to abefb1beff
2022-02-10 19:00:47 +00:00
David Taylor fcae4a8faf
DEV: Do not include CDN domain in the sourceMappingURL (#15887)
If no path is supplied, browsers will look for the map on the same path as the JS file itself. This fixes two problems that we see in production:

1. When compiling assets against one CDN, and then re-using them on a site with a different CDN, the sourceMappingUrls would be incorrect and print warnings in the console

2. If both an S3 CDN and an app CDN are configured, we were using the S3 CDN for the JS and the app CDN for the map. This commit will make sure we use the S3 CDN for both.
2022-02-10 15:37:44 +00:00
David Taylor abefb1beff
DEV: Do not run ember-cli output through terser again (#15889)
ember-cli already runs terser on its output. Running it through again provides no benefit, takes longer, and also breaks source mapping of these assets in production.
2022-02-10 15:35:54 +00:00
David Taylor ecc07fd8dc
DEV: Make Ember CLI assets the default in production (#15861)
This was reverted in e92f57255d due to memory usage concerns. This memory issue was resolved by 4cceb55621.
2022-02-08 10:03:53 +00:00