Commit Graph

1231 Commits

Author SHA1 Message Date
Jarek Radosz 29e601c13a
DEV: Fix the javascript:update rake task (#16057)
The source of the moment tz name package has changed in #15761
2022-02-26 02:06:12 +01:00
Dan Ungureanu ae5eab06ad
FEATURE: Add ENV to control themes:update errors (#16051)
This commit introduces a new environment variable `RAISE_THEME_ERRORS`
that can control what happens when `theme:update` Rake task errors. It
can have three possible values: `0` to always print errors, `1` to
always raise on error, or be absent to use the default behavior which
raises errors only for default sites.
2022-02-25 11:54:42 +02:00
Joffrey JAFFEUX 8244b4b163
DEV: prevents warnings with EMBER_CLI being redefined (#16035)
Example error:

```
/__w/discourse/discourse/lib/tasks/assets.rake:3: warning: already initialized constant EMBER_CLI
/__w/discourse/discourse/lib/tasks/assets.rake:3: warning: previous definition of EMBER_CLI was here
```
2022-02-23 10:51:31 +01:00
David Taylor 6ab4d26d84
PERF: Terminate `rake` process during `ember build` (#15920)
The `assets:precompile` rake task loads the full Ruby app, which can consume around 500mb of RAM by itself. Using `exec` to run `ember build` allows us to free up the Ruby memory and make more space for `ember build`
2022-02-14 11:49:46 +00:00
David Taylor a01b1dd648
PERF: Update ember-auto-import and webpack (#15919)
This makes a small improvement to 'cold cache' ember-cli build times, and a large improvement to 'warm cache' build times

The ember-auto-import update means that vendor is now split into multiple files for efficiency. These are named `chunk.*`, and should be included immediately after the `vendor.js` file. This commit also updates the rails app to render script tags for these chunks.

This change was previously merged, and caused memory-related errors on RAM-constrained machines. This was because Webpack 5 switches from multiple worker processes to a single multi-threaded process. This meant that it was hitting node's default heap size limit (~500mb on a 1GB RAM server). Discourse's standard install procedure recommends adding 2GB swap to 1GB-RAM machines, so we can afford to override's Node's default via the `--max-old-space-size` flag.
2022-02-14 11:21:39 +00:00
David Taylor 8025768fd2
DEV: Embed source content in JS sourcemaps (#15914)
This removes the need for a further request for the 'original' source file, and brings our custom terser arguments in line with the ember-cli defaults
2022-02-11 20:38:53 +00:00
David Taylor 51b88c0e2c
DEV: Ensure Ember CLI sourcemaps are uploaded to S3 (#15893)
Ember CLI gives sourcemaps their own digest. Our `s3.rake` logic assumes that the digest portion of sourcemap filenames remains the same.

The Ember CLI sourcemaps are included in the manifest file, so we can ensure they are uploaded by letting them past the MiniMime check.

Followup to abefb1beff
2022-02-10 19:00:47 +00:00
David Taylor fcae4a8faf
DEV: Do not include CDN domain in the sourceMappingURL (#15887)
If no path is supplied, browsers will look for the map on the same path as the JS file itself. This fixes two problems that we see in production:

1. When compiling assets against one CDN, and then re-using them on a site with a different CDN, the sourceMappingUrls would be incorrect and print warnings in the console

2. If both an S3 CDN and an app CDN are configured, we were using the S3 CDN for the JS and the app CDN for the map. This commit will make sure we use the S3 CDN for both.
2022-02-10 15:37:44 +00:00
David Taylor abefb1beff
DEV: Do not run ember-cli output through terser again (#15889)
ember-cli already runs terser on its output. Running it through again provides no benefit, takes longer, and also breaks source mapping of these assets in production.
2022-02-10 15:35:54 +00:00
David Taylor ecc07fd8dc
DEV: Make Ember CLI assets the default in production (#15861)
This was reverted in e92f57255d due to memory usage concerns. This memory issue was resolved by 4cceb55621.
2022-02-08 10:03:53 +00:00
David Taylor e92f57255d
Revert "DEV: Make Ember CLI assets the default in production (#15843)" (#15852)
This reverts 1b622667bc

We have had reports of issues rebuilding under memory-constrained environments. Reverting while we investigate further.
2022-02-07 20:31:10 +00:00
David Taylor 1b622667bc
DEV: Make Ember CLI assets the default in production (#15843)
This can be disabled by setting `EMBER_CLI_PROD_ASSETS=0`, but this option will not be available for long. If your theme/plugin/site has issues under Ember CLI, please open a topic on https://meta.discourse.org
2022-02-07 15:25:57 +00:00
David Taylor c3a54eb30f DEV: Use strings for :user_email job type argument
Job arguments go via JSON, and so symbols will appear as strings in the Job's `#execute` method. The latest version of Sidekiq has started warning about this to reduce developer confusion.
2022-02-04 18:28:18 +00:00
Rafael dos Santos Silva 3f694e4ab5
FEATURE: Use native color-picker (#15748) 2022-02-01 11:18:13 -03:00
David Taylor 6c3df84a93
DEV: In themes:update, only update themes which are out-of-date (#15676)
Running `update_from_remote` and `save!` cause a number of side-effects, including instructing all clients to reload CSS files. If there are no changes, then this is wasteful, and can even cause a 'flicker' effect on clients as they reload CSS.

This commit checks if any updates are available before triggering `update_from_remote` / `save!`. This should be much faster, and stop the 'flickering' UX from happening on every themes:update run.

It also improves the output of the command to include the from/to commit hashes, which may be useful for debugging issues. For example:

```
Checking 'Alien Night | A Dark Discourse Theme' for 'default'... already up to date
Checking 'Star Wars' for 'default'... updating from d8a170dd to 66b9756f
Checking 'Media Overlay' for 'default'... already up to date
```
2022-01-21 18:23:26 +00:00
Dan Ungureanu f0c1a4fab8
FIX: Create email token with correct scope (#15658)
`account_created` email contains a URL to `/u/password-reset/TOKEN`
which means that the correct scope for the email token is
`password_reset`, not `signup`.
2022-01-20 16:38:56 +02:00
Gerhard Schlager 27f1630b01
DEV: Try to download missing uploads from origin URL (#15629) 2022-01-19 11:05:58 +01:00
David Taylor 5c23c6cdab DEV: Update `rake qunit:test` and `rake plugin:qunit` to use `testem`
For now this is still gated behind a `QUNIT_EMBER_CLI=1` environment variable, but will eventually become the default so that we can remove `run-qunit.js`.
2022-01-18 10:16:29 +00:00
David Taylor ed2f700440
DEV: Wait for initdb to complete in docker.rake (#15614)
On slower hardware it can take a while to init the database. If we don't wait, the `rake db:create` step will fail.
2022-01-17 17:45:39 +00:00
Rafael dos Santos Silva 8cfd791e0f
FEATURE: Add missing emojis (#15582)
* FEATURE: Add missing emojis

* More missing emojis
2022-01-14 17:51:13 -03:00
Robin Ward 6272edd121 DEV: Support for running theme test with Ember CLI (third attempt)
The second attempt fixed issues with smoke test.

This one makes sure minification only happens in production mode.
2022-01-13 16:02:07 -05:00
Martin Brennan 107239a442
Revert "DEV: Support for running theme test with Ember CLI (second attempt)" (#15559)
This reverts commit 2c7906999a.

The changes break some things in local development (putting JS files
into minified files, not allowing debugger, and others)
2022-01-13 10:05:35 +10:00
Robin Ward 2c7906999a DEV: Support for running theme test with Ember CLI (second attempt)
This PR includes support for running theme tests in legacy ember
production envrionments.
2022-01-12 15:43:29 -05:00
David Taylor 252bb87ab3
Revert "DEV: Support for running theme test with Ember CLI" (#15547)
This reverts commit ea84a82f77.

This is causing problems with `/theme-qunit` on legacy, non-ember-cli production sites. Reverting while we work on a fix
2022-01-11 23:38:59 +00:00
Robin Ward ea84a82f77 DEV: Support for running theme test with Ember CLI
This is quite complex as it means that in production we have to build
Ember CLI test files and allow them to be used by our Rails application.

There is a fair bit of glue we can remove in the future once we move to
Ember CLI completely.
2022-01-11 15:42:13 -05:00
David Taylor b3e52f99e6
FIX: LOAD_PLUGINS=0 in dev/prod, warn in plugin:pull_compatible_all (#15537)
The `plugin:pull_compatible_all` task is intended to take incompatible plugins and downgrade them to an earlier version. Problem is, when running the rake task in development/production environments, the plugins have already been activated. If an incompatible plugin raises an error in `plugin.rb` then the rake task will be unable to start.

This commit centralises our LOAD_PLUGINS detection, adds support for LOAD_PLUGINS=0 in dev/prod, and adds a warning to `plugin:pull_compatible_all` if it's run with plugins enabled.
2022-01-11 12:30:22 +00:00
Rafael dos Santos Silva 06bba76719
DEV: Update emojis constants (#15506)
Follow up to d1cfabd so the new emojis are usable by the app
2022-01-10 14:53:52 -03:00
Jarek Radosz 5a50f18c0c
DEV: Avoid `$` globals (#15453)
Also:
* Remove an unused method (#fill_email)
* Replace a method that was used just once (#generate_username) with `SecureRandom.alphanumeric`
* Remove an obsolete dev puma `tmp/restart` file logic
2022-01-08 23:39:46 +01:00
Jarek Radosz 793c3ae7d4
FIX: Actually save fields on themes:update (#15497)
`RemoteTheme#update_from_remote` does not save theme fields on its own.

Fixes theme/theme component autoupdates working only partially.
2022-01-08 05:41:20 +01:00
Rafael dos Santos Silva 0a0e06fcf8
FEATURE: Update emojis (#15468)
* FEATURE: Update emojis

* Add extra aliases for backwards compatibility
2022-01-07 15:49:56 -03:00
Peter Zhu c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
Martin Brennan 099b679fc5
DEV: Clean up old bookmark code (#15455)
The rake task deleted here was added back in Feb 2020
when bookmarks were first converted from PostAction
records, it is no longer needed. The ignored columns
were removed in ed83d7573e.
2022-01-05 10:02:02 +10:00
Martin Brennan 8a26ea23f6
DEV: Use EmailSettingsValidator in more places (#15404)
Clears out a long-standing TODO I added back in
3d2cace94f, this way we are only
validating these settings using one central class.
2022-01-04 08:30:48 +10:00
Penar Musaraj 108c8302fb
FEATURE: Automatic admin editor dark mode (#15419) 2021-12-29 11:02:37 -05:00
Martin Brennan 19089f21d2
DEV: Add API docs for uploads and API doc watcher (#15387)
This commit adds API documentation for the new upload
endpoints related to direct + multipart external uploads.

Also included is a rake task which watches the files in
the spec/requests/api directory and calls a script file
(spec/regenerate_swagger_docs) whenever one changes. This
script runs rake rswag:specs:swaggerize and then copies
the openapi.yml file over to the discourse_api_docs repo
directory, and hits a script there to convert the YML to
JSON so the API docs are refreshed while the server is
still running. This makes the loop of making a doc change
and seeing it in the local server much faster.

The rake task is rake autospec:swagger
2021-12-23 08:40:15 +10:00
Martin Brennan 667a8a63b3
DEV: Drop jQuery file uploader and old upload components (#15376)
This commit removes jQuery file uploader from Discourse,
completing the transition to Uppy. The image-uploader
and UploadMixin components are also removed in this commit
as they have already been replaced and are the only things
using jQuery file upload.

            .-'~~~`-.
          .'         `.
          |  R  I  P  |
          |   jquery  |
          |    file   |
          |   upload  |
          |           |
        \\| 2013-2021 |//
        -----------------
2021-12-22 08:59:44 +10:00
Martin Brennan d330a5447d
DEV: Remove old backup uploader and resumable.js (#15365)
Now that d5e380e5c1 has been
committed there is nothing in the codebase that uses either
resumable.js or the old backup-uploader component.

R.I.P resumable.js
2021-12-21 15:02:10 +10:00
Penar Musaraj cebf55f590
DEV: Remove `jquery-tags-input` dependency (#15344) 2021-12-17 14:53:52 -05:00
Penar Musaraj 178acd4d46
DEV: Remove jQuery color dependency (#15340) 2021-12-17 14:26:16 -05:00
Penar Musaraj 4ee5d52ac9
DEV: Remove jQuery autoellipsis dependency (#15336) 2021-12-17 11:45:12 -05:00
David Taylor 0e87f882a7
DEV: Use discourse image for postgres in GitHub Actions (#15291)
The discourse base image already contains a postgres installation, so pulling a separate postgres image is a little wasteful. Using the copy of Postgres in the discourse image saves about 20 seconds on every GitHub actions run.

This commit sets up Postgres with a few performance-improving flags, which we were already using for the `rake docker:test` task (used on our internal CI system).
2021-12-14 17:20:06 +00:00
Jarek Radosz df6e8b924e
DEV: Make legacy ember tests less likely to fail (#15147)
…on launch
2021-12-01 19:30:33 +01:00
Osama Sayegh 20f736aa11
FIX: Skip themes that have blank URL in the `themes:update` rake task (#15156)
Themes that are imported via a ZIP file do have a `remote_theme` record in the database but the record has a blank value for the `remote_url` field which means attempting to do an update git via will result in an error.
2021-12-01 19:57:36 +03:00
Jarek Radosz c9e0ad5ec4
DEV: Remove build:stamp task and related code (#15069)
I don't think we use `build:stamp` task anymore, nor we rely on config/version.rb
2021-11-29 19:09:05 +01:00
Dan Ungureanu b1844c45c3
DEV: Stop if theme:update fails for default site (#15090)
The error handling of the theme:update Rake task has been improved. If
an error occurs while updating the default site, then the exception will
be propagated and the process will exit with non-zero status.

This is a follow-up to commit 3f97f884fe.
2021-11-25 16:28:28 +02:00
Dan Ungureanu fa8cd629f1
DEV: Hash tokens stored from email_tokens (#14493)
This commit adds token_hash and scopes columns to email_tokens table.
token_hash is a replacement for the token column to avoid storing email
tokens in plaintext as it can pose a security risk. The new scope column
ensures that email tokens cannot be used to perform a different action
than the one intended.

To sum up, this commit:

* Adds token_hash and scope to email_tokens

* Reuses code that schedules critical_user_email

* Refactors EmailToken.confirm and EmailToken.atomic_confirm methods

* Periodically cleans old, unconfirmed or expired email tokens
2021-11-25 09:34:39 +02:00
Dan Ungureanu 3f97f884fe
DEV: Fail stop if theme update fails (#15074)
This applies only when a single site exists. If a theme update fails
when there are multiple sites, then it will continue updating the
remaining themes.
2021-11-24 19:12:49 +02:00
Jarek Radosz 1c0dcbfd47
DEV: Remove `build_test_topic` task (#15068)
I don't think anyone is using it anymore
2021-11-23 23:41:49 +01:00
Dan Ungureanu fa2fd7fff8
FIX: Do not fail-stop if theme update fails (#15063) 2021-11-23 13:55:09 +02:00
Jarek Radosz 05423e9dfd
DEV: `I18n` global is no longer supported (#14993)
Don't allow it when linting.
2021-11-17 20:52:22 +01:00
Vinoth Kannan a3a4f16e88
UX: add more search aliases for emojis. (#14768) 2021-11-17 11:54:43 +01:00
Jarek Radosz 4ad77f3382
DEV: Remove `.es6` extensions from core (#14912)
Still supported in plugins though.
2021-11-13 12:51:53 +01:00
Osama Sayegh 69ec6899f9
Revert "DEV: increase lock timeout for multisite migration (#14831)" (#14883)
* Revert "DEV: increase lock timeout for multisite migration (#14831)"

This partially reverts commit 337ef60303.

We need to revert the mutex around `db:status:json` because the mutex is not available unless the rails environment is loaded which the `db:status:json` doesn't load before the mutex. We can't load the environment before entering the mutex because the mutex is meant to prevent other instances of the task from loading a rails environment while the database is migrating.

Co-authored-by: David Taylor <david@taylorhq.com>

Co-authored-by: David Taylor <david@taylorhq.com>
2021-11-11 16:16:53 +03:00
Jarek Radosz b72688340f
FIX: `LocalJumpError` in rake tasks (#14874)
`return` isn't allowed in a block context. A followup to #14865
2021-11-10 16:53:55 +01:00
Jarek Radosz 25ef395af8
FIX: Don't fail if `s3_install_cors_rule` is off (#14865)
Regressed in #14802
2021-11-10 08:00:30 +10:00
Sam 337ef60303
DEV: increase lock timeout for multisite migration (#14831)
- Increase lock timeout - given multisites may take a while to migrate
- Ensure we do not check for status while db is migrating
2021-11-09 12:06:06 +11:00
Martin Brennan fc98d1edfa
DEV: Improve s3:ensure_cors_rules logging (#14832) 2021-11-08 11:44:12 +10:00
Jarek Radosz e6245bd603
DEV: Remove hints of jshint (#14828) 2021-11-08 09:12:37 +08:00
Martin Brennan 9a72a0945f
FIX: Ensure CORS rules exist for S3 using rake task (#14802)
This commit introduces a new s3:ensure_cors_rules rake task
that is run as a prerequisite to s3:upload_assets. This rake
task calls out to the S3CorsRulesets class to ensure that
the 3 relevant sets of CORS rules are applied, depending on
site settings:

* assets
* direct S3 backups
* direct S3 uploads

This works for both Global S3 settings and Database S3 settings
(the latter set directly via SiteSetting).

As it is, only one rule can be applied, which is generally
the assets rule as it is called first. This commit changes
the ensure_cors! method to be able to apply new rules as
well as the existing ones.

This commit also slightly changes the existing rules to cover
direct S3 uploads via uppy, especially multipart, which requires
some more headers.
2021-11-08 09:16:38 +10:00
Neil Lalonde 3254d35078
FIX: rake categories:move_topics shouldn't move category description topic (#14797) 2021-11-03 11:27:51 +08:00
Dan Ungureanu 69f0f48dc0
DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
Roman Rizzi dfb7924105
DEV: Specify target browsers when running Ember CLI tests from the docker rake task (#14720) 2021-10-26 11:10:22 -03:00
Yasuo Honda dbbfad7ed0 FIX: Support Ruby 3 keyword arguments 2021-10-05 11:25:00 -04:00
David Taylor 89994cff40 DEV: Allow Ember CLI for `rake qunit:test` and `rake plugin:qunit`
To use Ember CLI, set QUNIT_EMBER_CLI=1
2021-09-21 18:10:04 +01:00
Martin Brennan 22208836c5
DEV: Ignore bookmarks.topic_id column and remove references to it in code (#14289)
We don't need no stinkin' denormalization! This commit ignores
the topic_id column on bookmarks, to be deleted at a later date.
We don't really need this column and it's better to rely on the
post.topic_id as the canonical topic_id for bookmarks, then we
don't need to remember to update both columns if the bookmarked
post moves to another topic.
2021-09-15 10:16:54 +10:00
Daniel Waterworth 05c356f7c6
DEV: Add rake task to check that the DB can be accessed (#14300) 2021-09-09 14:42:10 -05:00
Daniel Waterworth 6262396d8a
FIX: Don't attempt to migrate multisite test db while holding the mutex (#14298)
Since you'll have to wait for the mutex to timeout before it can
continue.
2021-09-09 13:31:52 -05:00
Joffrey JAFFEUX 95b15acb1e
DEV: uses forked Mousetrap to avoid leaking listeners (#14198) 2021-09-08 14:48:13 +02:00
Daniel Waterworth d7873dd823
FIX: Don't attempt to migrate concurrently with other migrations (#14231) 2021-09-03 10:22:25 -05:00
Joshua Rosenfeld 823354d177
DEV: Update Plugin Release Notes heading format (#14218) 2021-09-02 11:08:31 +08:00
David Taylor 0aa8798046
DEV: Remove `yarn install` during `assets:precompile` (#14166)
Instead, we'll do it alongside `bundle install` in `web.template.yml`: https://github.com/discourse/discourse_docker/pull/565
2021-08-26 19:14:50 +01:00
Dan Ungureanu c0ec1e931e
FIX: Make themes:update work with multisites (#14082)
Running this Rake task in a multisite cluster updated only the default
site.
2021-08-19 09:41:58 +03:00
Bianca Nenciu 4380ba34d5
FIX: Make site tasks work with duplicated uploads (#13972)
Uploads can be reused between site settings. This change allows the same
upload to be exported only once and then the same file is reused. The
same applies to import.
2021-08-18 15:57:31 +03:00
Rafael dos Santos Silva b2e4c91818
DEV: Allow removing local assets after s3 sync (#14013) 2021-08-12 14:20:05 -03:00
Rafael dos Santos Silva 8e45fdfbb1
DEV: Log duration of ember-cli asset build (#13980) 2021-08-10 23:43:08 -03:00
Robin Ward 3fdb64184b FIX: Don't swallow an error if we can't run `yarn ember build` 2021-08-06 15:13:18 -04:00
Robin Ward 18c5e9338f DEV: Allow us to use Ember CLI assets in production
This adds an optional ENV variable, `EMBER_CLI_PROD_ASSETS`. If truthy,
compiling production assets will be done via Ember CLI and will replace
the assets Rails would otherwise use.
2021-08-05 08:32:33 -04:00
Martin Brennan d8a0d2262c
DEV: Update pretender and fake-xml-http-request (#13937)
We are still on a version of pretender since 2017
https://github.com/pretenderjs/pretender/releases/tag/v1.6.1

Since then many changes have been made, including adding support
for xhr.upload. Upgrading will let us write proper acceptance
tests for uppy, which uses XmlHTTPRequest internally including
xhr.upload.

Updates pretender to 3.4.7 and fake-xml-http-request to 2.1.2.

Note: There have been no breaking changes in the releases that would
affect us, mainly dropping support for old node versions.
2021-08-05 08:23:01 +10:00
Bianca Nenciu e2af2a2219
FIX: Make rake site:export_structure export uploads (#13938)
It failed because of an undefined variable.
2021-08-04 11:14:30 +03:00
Jean e7b8e75583
FEATURE: Add post edits count to user activity (#13495) 2021-08-02 10:15:53 -04:00
Jarek Radosz 4223541bc5
DEV: Fix release_note task warnings (#13794)
There was a bunch of warnings repeated over and over during spec runs:

```
/var/www/discourse/lib/tasks/release_note.rake:3: warning: already initialized constant DATE_REGEX
/var/www/discourse/lib/tasks/release_note.rake:3: warning: previous definition of DATE_REGEX was here
/var/www/discourse/lib/tasks/release_note.rake:5: warning: already initialized constant CHANGE_TYPES
/var/www/discourse/lib/tasks/release_note.rake:5: warning: previous definition of CHANGE_TYPES was here
```
2021-07-20 20:25:48 +02:00
David Taylor 4ce58fbf0b
DEV: Improve rake `release_note:generate` date handling (#13726)
* DEV: Improve rake `release_note:generate` date handling

A commit-ish value like HEAD@{2021-01-01} is based on the **local state** of HEAD on that date. It does not use dates attached to commits.

Instead, the rake task now detects date-like strings and supplies them to `git log` via the `--after` and `--before` flags

* Skip printing plugin when there are no changes found

A list of skipped plugins is printed on a single line at the end of the output
2021-07-15 16:15:57 +10:00
Gerhard Schlager 2a95f892af
DEV: Update `i18n:check` rake task to detect invalid Markdown links (#13728)
In addition to that it fixes a problem where the check failed on empty locale files and allows calling the rake task with multiple locales.
2021-07-14 13:26:12 +02:00
Kim Lindberger 7e52eada20
FIX: Use Terser for minification even if uglify-js is not available (#13683) 2021-07-14 14:20:57 +08:00
David Taylor 76f279dfb4
DEV: Update release notes output format, and add plugin task (#13709) 2021-07-12 21:23:50 +01:00
Kane York 81d1a61d17
DEV: Report message-id, skip reason in rake email:test (#13674)
meta/t/185109

Additionally, print an extra warning if the `disable_emails` site setting is set.
2021-07-09 11:14:18 -07:00
Arpit Jalan dec7e19da3
FIX: fix error message for fix_missing_s3 rake task (#13661) 2021-07-07 19:59:03 +05:30
Arpit Jalan 236d6d91b2
FIX: `fix_missing_s3` task fails on failed upload (take 2) (#13660)
ref: 935aadbfdd
2021-07-07 18:53:43 +05:30
Arpit Jalan 935aadbfdd
FIX: do not stop `fix_missing_s3` task if saving an upload failed (#13658)
This commit logs an error and moves to next upload when saving a single
upload record fails when running `uploads:fix_missing_s3` task.
2021-07-07 16:57:24 +05:30
Martin Brennan 35f6441938
DEV: Add uppy.js to build and project (#13645)
This PR adds uppy to the project with a custom JS build and the shims needed to import it into our JS code. We need a custom build of Uppy because we do not use webpack for our JS modules/build. The only way to get what you want from Uppy is to use the webpack modules or to include the entire Uppy project including all plugins in a single JS file. This way we can just use the plugins we actually want. Future PRs will actually use Uppy!
2021-07-07 10:39:33 +10:00
Penar Musaraj 95b5794331
DEV: Compile core and plugin stylesheets independently of themes (#13638)
Take 2 of https://github.com/discourse/discourse/pull/13466. 

Fixes a few issues with the original PR: 

- color definition stylesheet target now includes the theme id, to avoid themes set to use the default color scheme loading the same stylesheet 
- changes the internal cache key for color definition stylesheet to reset the pre-existing cache
2021-07-06 13:11:10 -04:00
David Taylor 8c370c3fe3 DEV: Add `annotate` rake tasks, and enforce via GitHub actions
`bin/rake annotate` is an alias of `bin/annotate --models`
`bin/rake annotate:clean` generates annotations by using a temporary, freshly migrated database. This should help us to produce more consistent annotations, even if development databases have been polluted by plugin migrations.

A GitHub actions task is also added which generates annotations on a clean database, and raises an error if they differ from the committed annotations.
2021-07-06 10:11:06 +01:00
David Taylor 2955d64703
DEV: Allow annotations to work in symlinked plugins, add binstub (#13635) 2021-07-05 15:43:10 +01:00
Alan Guo Xiang Tan 37b8ce79c9
FEATURE: Add last visit indication to topic view page. (#13471)
This PR also removes grey old unread bubble from the topic badges by
dropping `TopicUser#highest_seen_post_number`.
2021-07-05 14:17:31 +08:00
Penar Musaraj 02999f5eb2
Skip compressing theme test files (#13619) 2021-07-02 12:16:16 -04:00
Penar Musaraj 128fdf9d9c
DEV: Revert stylesheet refactors (#13584)
* Revert "FIX: Clear appropriate cache when updating font settings (#13582)"

This reverts commit de6cc7a924.

* Revert "DEV: Improve output of `Stylesheet::Mananger.precompile_theme_css`."

This reverts commit 95038856c9.

* Revert "FIX: Child themes being precompiled multiple times."

This reverts commit 6986b36985.

* Revert "Update spec/components/stylesheet/manager_spec.rb"

This reverts commit ddaa7cc7ea.

* Revert "Refactor scss live refreshing"

This reverts commit a838293aaf.

* Revert "Precompile core stylesheets independently of themes"

This reverts commit 99d259d39b.

* Revert "DEV: Add simple digest for core stylesheets"

This reverts commit d82c58e6cc.
2021-06-30 21:33:15 +08:00
Penar Musaraj 99d259d39b Precompile core stylesheets independently of themes 2021-06-30 13:42:40 +08:00
Rafael dos Santos Silva fa4a462517
FEATURE: Optimize images before upload (#13432)
Integrates [mozJPEG](https://github.com/mozilla/mozjpeg) and [Resize](https://github.com/PistonDevelopers/resize) using WebAssembly to optimize user uploads in the composer on the client-side.

NPM libraries are sourced from our [Squoosh fork](https://github.com/discourse/squoosh/tree/discourse), which was needed because we have an older asset pipeline.
2021-06-23 12:31:12 -03:00
Osama Sayegh fa62b5e83b
DEV: Add a way to exclude ENV vars from getting unset in `themes:isolated_test` (#13494) 2021-06-23 14:50:54 +03:00
Osama Sayegh d3a3d1b94c
DEV: Introduce `TemporaryRedis` and unset `DISCOURSE_*` env vars in the `themes:isolated_test` rake task (#13401)
The `themes:isolated_test` rake task will now unset all `DISCOURSE_*` env variables if `UNSET_DISCOURSE_ENV_VARS` env var is set and will also spin up a temporary redis server so the unicorn web server that's spun up for the tests doesn't leak into the "main" redis server.
2021-06-23 07:38:43 +03:00
Gerhard Schlager 820068ddaf
FIX: `fix_missing_s3` rake task could fail due to missing upload (#13479) 2021-06-22 17:00:55 +02:00
Penar Musaraj f343cfd92e
DEV: Remove IntersectionObserver polyfill (#13445) 2021-06-22 09:30:44 -04:00
Martin Brennan 7b31d8a11b
DEV: Move chrome binary check into a shared lib (#13451)
We had checks for the chrome binary in 3 different places
for tests and only one of them checked for google-chrome-stable,
which is problematic for Arch linux users (there are dozens of us!)

This PR moves all the code to one place and references it instead
of copying and pasting.
2021-06-21 13:28:48 +10:00
Osama Sayegh c47f55253f
DEV: Add optional theme test step to the `smoke:test` rake task (#13418)
The purpose of this is to allow us to catch regressions for a feature we've built recently that allows theme tests to run in production. We recently had a regression that we didn't notice for days, so to prevent that from happening again we'll use this in our internal CI pipelines.
2021-06-17 20:09:29 +03:00
Osama Sayegh 503017474c
DEV: Skip CSS watcher when running QUnit tests and expose more Chrome logs (#13390)
There are 2 changes in this PR:

1) Add a new environment variable called `DISCOURSE_SKIP_CSS_WATCHER` to disable our stylesheet watcher, and make the `qunit:test` rake task set this variable on the Unicorn/Rails server it spins up to disable our stylesheet watcher when running the tests because it doesn't really need it.

2) Print more Chrome logs (such as network/security errors) to the console.
2021-06-15 18:27:15 +03:00
Vinoth Kannan 6abc45e57b
DEV: move `discourse_dev` gem to the core. (#13360)
And get avatar images from `discourse_dev_assets` gem.
2021-06-14 20:34:44 +05:30
Alan Guo Xiang Tan 58cb120aa2
DEV: Minor code clean up in assets.rake. (#13245) 2021-06-03 11:37:06 +10:00
Jarek Radosz 3bb765ac92
DEV: Remove the remaining Travis code (#13255)
The second attempt at #10041 now that all our plugins use GitHub Actions CI instead.
2021-06-02 20:29:47 +02:00
Penar Musaraj 9d6780f03d
DEV: Remove emoji_one files (#13236)
And add a symlink so requests to those assets fallback to the default
emoji set (Twitter).
2021-06-02 09:18:25 -04:00
Rafael dos Santos Silva 9118bb2076
FEATURE: Normalize the service worker route (#12343)
Re-lands the change initially proposed on #8359 but without a new nginx
location block, so it has less change surface.

Co-authored-by: Jeff Wong <awole20@gmail.com>

Co-authored-by: Jeff Wong <awole20@gmail.com>
2021-05-25 19:39:31 -03:00
David Taylor 93ae1b2b1c
DEV: Default to `development` RAILS_ENV when running theme tests (#13106)
When testing theme components in development, it doesn't make sense to use the `test` environment. The `test` environment almost certainly has 0 themes installed.

This change still works fine when using the `themes:install_and_test` rake task, because that rake task explicitly specifies environment/database-config.
2021-05-24 13:35:10 +01:00
Bianca Nenciu 7c29ff3d85
FEATURE: Add tasks to export and import site structure (#12584) 2021-05-24 12:38:42 +10:00
Josh Soref 13d40ead97
DEV: Correct spelling mistakes in comments 2021-05-21 13:37:17 +10:00
Josh Soref 59097b207f
DEV: Correct typos and spelling mistakes (#12812)
Over the years we accrued many spelling mistakes in the code base. 

This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change 

- comments
- test descriptions
- other low risk areas
2021-05-21 11:43:47 +10:00
David Taylor bd8c2b4615
DEV: Correct theme:qunit help text (#13104)
- Task name is themes:qunit, not themes:unit
- Some shells try to expand the square brackets. The whole thing should be enclosed in quotes to avoid this
2021-05-20 22:30:12 +01:00
Arpit Jalan 626b8465ba
FIX: do not validate uploads when running `uploads:fix_missing_s3` task (#13096) 2021-05-20 15:29:09 +10:00
Arpit Jalan 130160537c
FEATURE: add support for "skip_validations" option in UploadCreator (#13094)
FIX: do not validate uploads when running `uploads:fix_missing_s3` task
2021-05-19 20:54:52 +05:30
Martin Brennan 3d2cace94f
DEV: Add service to validate email settings (#13021)
We have a few places in the code where we need to validate various email related settings, and will have another soon with the improved group email settings UI. This PR introduces a class which can validate POP3, IMAP, and SMTP credentials and also provide a friendly error message for issues if they must be presented to an end user.

This PR does not change any existing code to use the new service. I have added a TODO to change POP3 validation and the email test rake task to use the new validator post-release.
2021-05-13 15:11:23 +10:00
Blake Erickson 37812b8e35
Revert "DEV: Drop old IE11 intersection-observer references" (#13017)
This reverts commit 7360a0f70f.

iOS still wants this sometimes. Probably best to revert for now and we can
always remove this again later.

See: https://meta.discourse.org/t/189799/11?u=blake
2021-05-10 17:47:09 -06:00
Blake Erickson 1b02dce594
DEV: Drop old IE11 intersection-observer references (#12942)
It's been awhile since we have supported IE11 so it should be safe to remove
IntersectionObserver now.

From a TODO task in this repo:
> drop when we eventually drop IE11

Announcement of when we removed IE11 support:

https://meta.discourse.org/t/137984/40?u=blake
2021-05-04 17:54:54 -06:00
Roman Rizzi 5794787300
FIX: Make the docker:test task install dependencies before calling yarn ember test. (#12943) 2021-05-04 16:30:35 -03:00
Jeff Wong 656b0ae39e
DEV: Add an option to skip a theme update from the themes:install task. (#12905)
A theme can now specify `skip_update: true` in the yml config for
update allowing for the theme to be installed only if it does not already
exist.
2021-04-30 09:31:41 -07:00
David Taylor ad8c7714c8
DEV: Use filesystem-based SchemaCache in development (#12901)
In development we regularly restart/reload Rails, which wipes out the schema cache. This then has to be regenerated using DDL queries on the database.

Instead, we can make use of the `rake db:schema:cache:dump` command. This will dump the schema cache to a YAML file, and then load it when needed. This is significantly faster than rebuilding the cache from DDL queries every time.
2021-04-30 10:54:49 +01:00
Roman Rizzi 7423943ffe
DEV: The docker:test task can run JS tests using the Ember CLI. (#12879)
Set the EMBER_CLI env var to 1 to include these tests.
2021-04-29 14:51:34 -03:00
Osama Sayegh 4f88f2eb15
FEATURE: Allow theme tests to be run in production (take 2) (#12845)
This commit allows site admins to run theme tests in production via a new `/theme-qunit` route. When you visit `/theme-qunit`, you'll see a list of the themes/components installed on your site that have tests, and from there you can select a theme or component that you run its tests.

We also have a new rake task `themes:install_and_test` that can be used to install a list of themes/components on a temporary database and run the tests of the themes/components that are installed. This rake task can be useful when upgrading/deploying a Discourse instance to make sure that the installed themes/components are compatible with the new Discourse version being deployed, and if the tests fail you can abort the build/deploy process so you don't end up with a broken site.
2021-04-28 23:12:08 +03:00
Osama Sayegh a169dc6832
Revert "FEATURE: Allow theme tests to be run in production (#12815)" (#12840)
This reverts commit 7217dcb67a.

https://meta.discourse.org/t/failed-to-bootstrap-due-to-out-of-memory-killer/188141/18?u=osama

Precompiling test_helper.js is so expensive that it can make bootstrap
fail on servers with limited resources (2GB RAM). We will find another
way that doesn't require much resources.
2021-04-26 23:05:58 +03:00
Osama Sayegh 7217dcb67a
FEATURE: Allow theme tests to be run in production (#12815)
This commit allows site admins to run theme tests in production via a new `/theme-qunit` route. When you visit `/theme-qunit`, you'll see a list of the themes/components installed on your site that have tests, and from there you can select a theme or component that you run its tests.

We also have a new rake task `themes:install_and_test` that can be used to install a list of themes/components on a temporary database and run the tests of the themes/components that are installed. This rake task can be useful when upgrading/deploying a Discourse instance to make sure that the installed themes/components are compatible with the new Discourse version being deployed, and if the tests fail you can abort the build/deploy process so you don't end up with a broken site.
2021-04-26 12:56:45 +03:00
Osama Sayegh cd24eff5d9
FEATURE: Introduce theme/component QUnit tests (take 2) (#12661)
This commit allows themes and theme components to have QUnit tests. To add tests to your theme/component, create a top-level directory in your theme and name it `test`, and Discourse will save all the files in that directory (and its sub-directories) as "tests files" in the database. While tests files/directories are not required to be organized in a specific way, we recommend that you follow Discourse core's tests [structure](https://github.com/discourse/discourse/tree/master/app/assets/javascripts/discourse/tests).

Writing theme tests should be identical to writing plugins or core tests; all the `import` statements and APIs that you see in core (or plugins) to define/setup tests should just work in themes.

You do need a working Discourse install to run theme tests, and you have 2 ways to run theme tests:

* In the browser at the `/qunit` route. `/qunit` will run tests of all active themes/components as well as core and plugins. The `/qunit` now accepts a `theme_name` or `theme_url` params that you can use to run tests of a specific theme/component like so: `/qunit?theme_name=<your_theme_name>`.

* In the command line using the `themes:qunit` rake task. This take is meant to run tests of a single theme/component so you need to provide it with a theme name or URL like so: `bundle exec rake themes:qunit[name=<theme_name>]` or `bundle exec rake themes:qunit[url=<theme_url>]`.

There are some refactors to how Discourse processes JavaScript that comes with themes/components, and these refactors may break your JS customizations; see https://meta.discourse.org/t/upcoming-core-changes-that-may-break-some-themes-components-april-12/186252?u=osama for details on how you can check if your themes/components are affected and what you need to do to fix them.

This commit also improves theme error handling in Discourse. We will now be able to catch errors that occur when theme initializers are run and prevent them from breaking the site and other themes/components.
2021-04-12 15:02:58 +03:00
Penar Musaraj e77b9dfd45
DEV: Use Terser for JS minification/compression if available (#12656) 2021-04-09 08:51:49 -04:00
Osama Sayegh 2b9ab3a0d9
Revert "FEATURE: Introduce theme/component QUnit tests (#12517)" (#12632)
This reverts commit a53d8d3e61 and 105634435f.

Reverted because the change broke some components. Will be added back in a few days.
2021-04-07 17:45:49 +03:00
Osama Sayegh a53d8d3e61
FEATURE: Introduce theme/component QUnit tests (#12517)
This commit allows themes and theme components to have QUnit tests. To add tests to your theme/component, create a top-level directory in your theme and name it `test`, and Discourse will save all the files in that directory (and its sub-directories) as "tests files" in the database. While tests files/directories are not required to be organized in a specific way, we recommend that you follow Discourse core's tests [structure](https://github.com/discourse/discourse/tree/master/app/assets/javascripts/discourse/tests).

Writing theme tests should be identical to writing plugins or core tests; all the `import` statements and APIs that you see in core (or plugins) to define/setup tests should just work in themes.

You do need a working Discourse install to run theme tests, and you have 2 ways to run theme tests:

* In the browser at the `/qunit` route. `/qunit` will run tests of all active themes/components as well as core and plugins. The `/qunit` now accepts a `theme_name` or `theme_url` params that you can use to run tests of a specific theme/component like so: `/qunit?theme_name=<your_theme_name>`.

* In the command line using the `themes:qunit` rake task. This take is meant to run tests of a single theme/component so you need to provide it with a theme name or URL like so: `bundle exec rake themes:qunit[name=<theme_name>]` or `bundle exec rake themes:qunit[url=<theme_url>]`.

There are some refactors to internal code that's responsible for processing themes/components in Discourse, most notably:

* `<script type="text/discourse-plugin">` tags are automatically converted to modules.

* The `theme-settings` service is removed in favor of a simple `lib` file responsible for managing theme settings. This was done to allow us to register/lookup theme settings very early in our Ember app lifecycle and because there was no reason for it to be an Ember service.

These refactors should 100% backward compatible and invisible to theme developers.
2021-04-07 10:39:57 +03:00
Jeff Wong 3a22c654e2
FIX: rake themes:update should fail if a theme update fails (#12608) 2021-04-05 15:53:34 -07:00
Gerhard Schlager acc73f2989
FIX: `uploads:fix_missing_s3` rake task used wrong SHA1 (#12495) 2021-03-25 11:35:29 +01:00
Sam dc6b547ed8
FIX: rake db:validate_indexes was broken (#12463)
A file was moved, but zeitwerk can not find it due to custom inflection.

Renamed so it can be properly found.
2021-03-24 08:53:01 +11:00
Penar Musaraj aee7ef0dc9
DEV: Fix build due to highlight.js branch issue (#12441)
Highlight.js changed their default branch from master to main. This switches to the @highlightjs/cdn-assets package, thus sidestepping the problem. It's a slightly cleaner integration though (no need to build locally anymore).
2021-03-18 18:21:23 -04:00
Daniel Waterworth e7ac906f21
DEV: Move TemporaryDB class to its own file (#12383)
rake tasks get repeated reloaded during the tests and it causes
redefinition warnings.
2021-03-12 11:27:13 -06:00
Sam 10001e4e8d
DEV: add db:validate_indexes task (#12273)
Added a new task to test if indexes are coherent with a blank database

This allows us to detect for cases where somehow indexes are out of sync

FIX_INDEXES=1 or `rake db:validate_indexes[fix]` to correct the issues it finds. 

Detects: 

- Badly named indexes that need to be renamed
- Missing indexes
- Extra indexes

Can correct all 3 with the fix option
2021-03-10 10:59:20 +11:00
Penar Musaraj 6809cccd88
DEV: Add experimental json_scheme site setting type (#12226) 2021-03-01 09:15:17 -05:00
Penar Musaraj 45c5fd2172
DEV: Remove JoyPixels emoji option (#12197)
- removes the option from site settings
- deletes the site setting on existing sites that have it
- marks posts using emojis as requiring a rebake

Note that the actual image files are not removed here, the plan is to
remove them in a few weeks/months (when presumably the rebaking of old
posts has been completed).
2021-02-26 07:44:52 -05:00
Rafael dos Santos Silva 01584c5b22
FIX: emails.rake-use domain when sending mail test (#12188)
`rake emails:test` task was always sending `localhost` as the domain name rather than using `smtp[:domain]` (aka `DISCOURSE_SMTP_DOMAIN`. `discourse-setup` has recently been modified to always set `DISCOURSE_SMTP_DOMAIN`, so it's important that this test rake task actually use the value.

I tested this change on a standard production install, and it's working as expected. Hopefully I managed not to bungle the copy/paste of the single line here in the github edit window.

Co-authored-by: Jay Pfaffman <jay@literatecomputing.com>
2021-02-23 14:16:51 -03:00
Joshua Rosenfeld b79c06f3b7
Include UI commits when generating release notes. (#12127)
`UX` is the officially supported prefix per https://meta.discourse.org/t/19392, but sometimes `UI` is used instead. We should still include those commits.
2021-02-18 10:53:23 -05:00
Bianca Nenciu 08acf51be0
FEATURE: Use diffhtml to update composer preview (#11237)
Displaying videos, animated GIFs or any kind of rich content in preview
used to refresh on every keystroke, which could cause performance
problems.
2021-02-18 16:07:26 +02:00
Jeff Wong f17e745fe3
DEV: log rake plugin:update_all plugin_path on error (#12100)
* DEV: log rake plugin:update_all plugin_path on error

Allows a dev to know which plugin was failing to update
2021-02-16 11:50:27 -08:00
Jarek Radosz abe688beb3 DEV: Use discourse/discourse_test in CI workflows
Using our testing Docker image (`discourse/discourse_test:release`) allows us to drop "Update imagemagick" step which shaves ~10 minutes from all runs.
2021-02-11 15:24:15 +01:00
Penar Musaraj a1955b0542
Dev: --dev option does nothing in yarn (#12032) 2021-02-10 19:19:47 -05:00
Jarek Radosz 43948f6a10
DEV: Support for `main` branches in plugin:update (#12027)
Based on https://github.com/discourse/docker_manager/pull/94
2021-02-10 11:34:17 -05:00
T 1a159de36f
tasks/emails.rake: catch exception and puts message (#12024)
* tasks/emails.rake: catch exception and puts message

* add username and auth
2021-02-10 10:42:21 -05:00
Penar Musaraj 3da2e85855
FIX: Save updated remote theme, only update enabled themes/components (#11960) 2021-02-04 14:44:07 +11:00
Vinoth Kannan 9d2eaec88f
DEV: enable CORS to all CDN get requests from workbox. (#11896)
To prevent opaque cache files, now all the CDN files will be requested in 'cors' mode if the cdn_cors_enabled global setting is enabled. Before enabling the setting, should enable the cors in the CDN server by adding the response header `access-control-allow-origin: *` or `access-control-allow-origin: https://discourse.example.com.`

And other external file requests other than CDN will not be cached if the response type is opaque.
2021-02-02 11:38:29 +05:30
Martin Brennan f49e3e5731
DEV: Add security_last_changed_at and security_last_changed_reason to uploads (#11860)
This PR adds security_last_changed_at and security_last_changed_reason to uploads. This has been done to make it easier to track down why an upload's secure column has changed and when. This necessitated a refactor of the UploadSecurity class to provide reasons why the upload security would have changed.

As well as this, a source is now provided from the location which called for the upload's security status to be updated as they are several (e.g. post creator, topic security updater, rake tasks, manual change).
2021-01-29 09:03:44 +10:00