Commit Graph

139 Commits

Author SHA1 Message Date
Daniel Waterworth ae63d9ea09
PERF: Remove unnecessary queries from Theme#set_field (#21281) 2023-04-27 13:04:58 -05:00
Daniel Waterworth 0728fa055b
DEV: Make theme cache work like other caches (#20813) 2023-03-24 12:37:16 -05:00
Daniel Waterworth 84f590ab83
DEV: Store theme sprites in the DB (#20501)
Let's avoid fetching sprites from the CDN during page rendering.
2023-03-14 13:11:45 -05:00
David Taylor 41f3bb8b50
FIX: Restore class-property babel transform for themes (#19883)
This seems to be required for decorators to work on class properties. Followup to 624f4a7de9
2023-01-16 18:06:46 +00:00
David Taylor 624f4a7de9
Drop support for iOS < 15.7 (#19847)
https://meta.discourse.org/t/224747
2023-01-16 17:28:59 +00:00
David Taylor 5a003715d3
DEV: Apply syntax_tree formatting to `app/*` 2023-01-09 14:14:59 +00:00
Rafael dos Santos Silva 0434de6cee
DEV: Compatibility with Ruby 3.2 (#19303) 2022-12-13 16:03:53 -03:00
Kris 07e4b7550d
DEV: relocate `:root` CSS custom properties (#19389) 2022-12-09 14:33:15 -05:00
Isaac Janzen aea492df5e
DEV: Add link rel preload to theme-javascripts (#19231)
* DEV: Add link rel preload to theme-javascripts
2022-11-30 12:43:01 -06:00
David Taylor 695b44269b
DEV: Do not remove `debugger` statements from themes (#18814) 2022-11-01 13:56:33 +00:00
Osama Sayegh ce53152e53
DEV: Include theme_uploads and theme_uploads_local objects in theme tests (#18645)
Our theme system injects a magical `settings` object at the top of themes JS modules to allow theme authors to access the settings as configured by admins in the UI. Within this `settings` object, there are a couple of special objects `theme_uploads` and `theme_uploads_local` that contain URLs for all the assets/uploads that the theme has.

For test modules/files, the theme system also injects a `settings` object at the top of tests modules, but it's not the same object as the object that's injected in non-test files. The difference is that in tests we want the settings to have their default values as opposed to any custom values that may exist in the site's database. This ensures that test results are consistent no matter the site that runs them.

However, the `settings` object in tests files currently doesn't have the special objects `theme_uploads` and `theme_uploads_local` which means that if a theme includes an asset that's lazy-loaded, it's not possible to write tests for anything that depends on the lazy-loaded asset because the theme will not be able to load the asset during the tests since `theme_uploads_local` and `theme_uploads` don't exist. This PR adds these special objects inside the `settings` object for test files.

Internal topic: t/71825/52.
2022-10-20 08:00:29 +03:00
David Taylor 8d3fe3ddc5
DEV: Bump theme compiler version for connector fix (#18665)
This should have been included in c185043590
2022-10-19 10:52:39 +01:00
David Taylor be3d6a56ce
DEV: Introduce minification and source maps for Theme JS (#18646)
Theme javascript is now minified using Terser, just like our core/plugin JS bundles. This reduces the amount of data sent over the network.

This commit also introduces sourcemaps for theme JS. Browser developer tools will now be able show each source file separately when browsing, and also in backtraces.

For theme test JS, the sourcemap is inlined for simplicity. Network load is not a concern for tests.
2022-10-18 18:20:10 +01:00
David Taylor cb87067c77 DEV: Introduce support for template colocation in themes 2022-10-18 10:10:49 +01:00
David Taylor 65a5c84a92 DEV: perform theme extra_js compilation all together
Previously, compiling theme 'extra_js' was done with a number of steps. Each theme_field would be compiled into its own value_baked column, and then the JavascriptCache content would be built by concatenating all of those compiled values.

This commit streamlines things by removing the value_baked step. The raw value of all extra_js theme_fields are passed directly to the ThemeJavascriptCompiler, and then the result is stored in the JavascriptCache.

In itself, this commit should not cause any behavior change. It is designed to open the door to more advanced compilation features which have interdependencies between different source files (e.g. template colocation, sourcemaps).
2022-10-18 10:10:49 +01:00
David Taylor f4e1d0c546
FIX: Ensure `<script>` handlebars templates are namespaced correctly (#18178)
This regressed in 7e74dd0afe
2022-09-04 13:01:10 +02:00
Gerhard Schlager fd6109a6e1
FIX: Theme components should work with empty locale files (#18167)
Theme components didn't work with empty locale files (just the locale key without any translations) that are created by translator-bot.
2022-09-02 18:28:18 +02:00
David Taylor 7e74dd0afe
DEV: Use DiscourseJsProcessor for theme template compilation (#18135)
Previously we were relying on a highly-customized version of the unmaintained Barber gem for theme template compilation. This commit switches us to use our own DiscourseJsProcessor, which makes use of more modern patterns and will be easier to maintain going forward.

In summary:
- Refactors DiscourseJsProcessor to move multiline JS heredocs into a companion `discourse-js-processor.js` file
- Use MiniRacer's `.call` method to avoid manually escaping JS strings
- Move Theme template AST transformers into DiscourseJsProcessor, and formalise interface for extending RawHandlebars AST transformations
- Update Ember template compilation to use a babel-based approach, just like Ember CLI. This gives each template its own ES6 module rather than directly assigning `Ember.TEMPLATES` values
- Improve testing of template compilation (and move some tests from `theme_javascript_compiler_spec.rb` to `discourse_js_processor_spec.rb`
2022-09-01 11:50:46 +01:00
David Taylor 6570fed175
DEV: Fix Ember version lookup (#18136)
The source-of-truth for our ember version is now the installed node_module. The `ember_source` gem carries an old version of Ember and so the constant is no longer useful. We'll be dropping the gem soon.
2022-08-30 16:27:14 +01:00
David Taylor e16c8ea2e7
DEV: Support inline-hbs compilation in themes (#18112)
This commit makes a number of improvements to the DiscourseJsProcessor:

1. Remove dependence on the out-of-date Ember template compiler from the ember-rails gem; switch to modern template compiler
2. Refactor to make use of a proper module system with `define`/`require`
3. Introduce `babel-plugin-ember-template-compilation` to enable inline hbs compilation

The `mini-loader` is upgraded to support relative lookup and `require.has`, so that these new JS packages work correctly.
2022-08-29 19:53:42 +01:00
David Taylor d6bba1ea9d
DEV: Invalidate theme cache when S3 configuration changes (#17872)
Compiled themes often include upload URLs which will need to be re-calculated
2022-08-11 11:03:57 +01:00
David Taylor 47824bc69f
DEV: Bump asset compiler versions (#17843)
This will trigger recompilation of theme/plugin assets following 7f9c2c0b
2022-08-09 12:06:15 +01:00
Osama Sayegh 306dca3a48
DEV: Add a helper method to clear every possible theme cache (#17585)
Our theme system is very complex and it can take a while to figure out how to invalidate the various types of caches that are used throughout the theme system. So, having a single helper method that invalidates everything can be useful in emergency situations where there is no time to read through the code and figure out how to clear the various caches.

Internal ticket: t64732.
2022-07-22 09:46:52 +03:00
David Taylor a4c4d3cb50 DEV: Patch Ruby Ember template compiler to work with recent ember 2022-07-19 10:00:59 +01:00
David Taylor fab1c00c8f
DEV: Drop the deprecated `themeSettings.blah` syntax (#17394)
This syntax has been printing deprecation messages since 880311dd4d
2022-07-18 10:10:23 +01:00
David Taylor 92e8a6d62b
DEV: Key the Theme `DistributedCache` on `BASE_COMPILER_VERSION` (#17495)
All `DistributedCache` instances in Discourse are automatically keyed on the `Discourse.git_version`. Normally the theme compiler version is updated via a commit, and everything is fine. However, in some situations, it's possible for the BASE_COMPILER_VERSION to change without a change to the git_version (e.g. when applying patches directly to the codebase).

This commit adds the `BASE_COMPILER_VERSION` to the DistributedCache key to ensure that content from different compiler versions does not leak into other processes.
2022-07-14 12:43:55 +01:00
Joe 804b8fd9f9
DEV: Defer loading core/plugin/theme JS files (#17063)
This is pre-request work to introduce a splash screen while site assets load.

The only change this commit introduces is that it ensures we add the defer attribute to core/plugin/theme .JS files. This will allow us to insert markup before the browser starts evaluating those scripts later on. It has no visual or functional impact on core.

This will not have any impact on how themes and plugins work. The only exception is themes loading external scripts in the </head> theme field directly via script tags. Everything will work the same but those would need to add the defer attribute if they want to keep the benefits introduced in this PR.
2022-06-20 09:47:37 +08:00
Loïc Guitaut 008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
David Taylor 5e34ce1282
DEV: Ensure theme tests are always loaded in a consistent order (#16569)
If they aren't, then the digest will be different, causing unexpected issues
2022-04-26 17:05:57 +01:00
Sam cedcdb0057
FEATURE: allow for local theme js assets (#16374)
Due to default CSP web workers instantiated from CDN based assets are still
treated as "same-origin" meaning that we had no way of safely instansiating
a web worker from a theme.

This limits the theme system and adds the arbitrary restriction that WASM
based components can not be safely used.

To resolve this limitation all js assets in about.json are also cached on
local domain.

{
  "name": "Header Icons",
  "assets" : {
    "worker" : "assets/worker.js"
  }
}

This can then be referenced in JS via:

settings.theme_uploads_local.worker

local_js_assets are unconditionally served from the site directly and
bypass the entire CDN, using the pre-existing JavascriptCache

Previous to this change this code was completely dormant on sites which
used s3 based uploads, this reuses the very well tested and cached asset
system on s3 based sites.

Note, when creating local_js_assets it is highly recommended to keep the
assets lean and keep all the heavy working in CDN based assets. For example
wasm files can still live on the CDN but the lean worker that loads it can
live on local.

This change unlocks wasm in theme components, so wasm is now also allowed
in `theme_authorized_extensions`

* more usages of upload.content

* add a specific test for upload.content

* Adjust logic to ensure that after upgrades we still get a cached local js
on save
2022-04-07 07:58:10 +10:00
David Taylor 5d6d3fb244
DEV: Remove child theme settings/variables from parent compilation (#16001)
aa1442fdc3 split theme stylesheets so that every component gets its own stylesheet. Therefore, there is now no need for parent themes to collate the settings/variables of its children during scss compilation.

Technically this is a breaking change for any themes which depend on the settings/variables of their child components. That was never a supported/recommended arrangement, so we don't expect this to cause issues.
2022-02-21 11:15:35 +00:00
David Taylor c8d956374d FIX: Use fresh theme setting values when compiling stylesheets
If a theme is updated to introduce a new setting AND immediately make use of it in a stylesheet, then an error was being shown. This is because the stylesheet compilation was using the theme's cached settings, and the cache is only cleared **after** the theme has finished compiling.

This commit updates the SCSS compilation to use uncached values for settings. A similar fix was applied to other parts of theme compilation back in 2020: (a51b8d9c66)
2022-02-21 09:45:14 +00:00
David Taylor 51afa579f7 DEV: Use the theme cache helper for settings
The previous Discourse.cache usage was different to how other theme-related caching is handled, and also requires reaching out to redis every time. The common theme cache is held in memory (as a DistributedCache)
2022-02-21 09:45:14 +00:00
Osama Sayegh 7ed899fed9
FIX: Ensure theme JavaScript cache get consistent SHA1 digest (#15933)
There is a couple of layers of caching for theme JavaScript in Discourse:

The first layer is the `javascript_caches` table in the database. When a theme
with JavaScript files is installed, Discourse stores each one of the JavaScript
files in the `theme_fields` table, and then concatenates the files, compiles
them, computes a SHA1 digest of the compiled JavaScript and store the results
along with the SHA1 digest in the `javascript_caches` table.

Now when a request comes in, we need to render `<script>` tags for the
activated theme(s) of the site. To do this, we retrieve the `javascript_caches`
records of the activated themes and generate a `<script>` tag for each record.
The `src` attribute of these tags is a path to the `/theme-javascripts/:digest`
route which simply responds with the compiled JavaScript that has the requested
digest.

The second layer is a distributed cache whose purpose is to make rendering
`<script>` a lot more efficient. Without this cache, we'd have to query the
`javascript_caches` table to retrieve the SHA1 digests for every single
request. So we use this cache to store the `<script>` tags themselves so that
we only have to retrieve the `javascript_caches` records of the activated
themes for the first request and future requests simply get the cached
`<script>` tags.

What this commit does it ensures that the SHA1 digest in the
`javascript_caches` table stay the same across compilations by adding an order
by id clause to the query that loads the `theme_fields` records. Currently, we
specify no order when retrieving the `theme_fields` records so the order in
which they're retrieved can change across compilations and therefore cause the
SHA1 to change even though the individual records have not changed at all.

An inconsistent SHA1 digest across compilations can cause the database cache
and the distributed cache to have different digests and that causes the
JavaScript to fail to load (and if the theme heavily customizes the site, it
gives the impression that the site is broken) until the cache is cleared.

This can happen in busy sites when 2 concurrent requests recompile the
JavaScript files of a theme at the same time (this can happen when deploying a
new Discourse version) and request A updates the database cache after request B
did, and request B updates the distributed cache after request A did.

Internal ticket: t60783.

Co-authored-by: David Taylor <david@taylorhq.com>
2022-02-14 16:23:06 +03:00
David Taylor af24c10314 DEV: Improve theme error handling UX
- Update UI to improve contrast
- Make it clear that the message is only shown to administrators
- Add theme name and id to the console output
- Parse the error backtrace to identify the theme-id for post-decoration errors
- Improve console output to include the theme name / URL
- Add `?safe_mode=no_custom` to the admin panel link, so that it will work even if the theme is causing the site to break
2022-02-14 10:11:19 +00:00
Alan Guo Xiang Tan 1d6a45d1f9
DEV: Bump Theme::BASE_COMPILER_VERSION. (#15906)
In 8e5b945b0f, we reverted the commit but
at the same time resulted in Theme::BASE_COMPILER_VERSION going
backwards which caused problems with themes caching.

This commit bumps the version to clear all the caches.

Follow-up to 8e5b945b0f
2022-02-11 17:33:04 +11:00
Alan Guo Xiang Tan 8e5b945b0f
Revert "DEV: Improve theme error handling UX" (#15900)
`PrettyText.cook` is breaking on some sites. Revert for now while we
investigate.

This reverts commit c81d369ab6.
2022-02-11 11:30:36 +08:00
David Taylor c81d369ab6 DEV: Improve theme error handling UX
- Update UI to improve contrast
- Make it clear that the message is only shown to administrators
- Add theme name and id to the console output
- Parse the error backtrace to identify the theme-id for post-decoration errors
- Improve console output to include the theme name / URL
- Add `?safe_mode=no_custom` to the admin panel link, so that it will work even if the theme is causing the site to break
2022-02-10 22:56:11 +00:00
Jarek Radosz d91d67a442
FEATURE: Allow theme settings to request refresh (#15037)
Similar to site settings, adds support for `refresh` option to theme settings.

```yaml
super_feature_enabled:
  type: bool
  default: false
  refresh: true
```
2021-11-22 13:16:56 +01:00
Jarek Radosz 61d14a7694
DEV: Fix 3N+1 query in `/admin/customize/themes` (#14876) 2021-11-11 18:11:23 +01:00
jbrw cfc62dbace
FIX: allowed_theme_ids should not be persisted in GlobalSettings (#14756)
* FIX: allowed_theme_ids should not be persisted in GlobalSettings

It was observed that the memoized value of `GlobalSetting.allowed_theme_ids` would be persisted across requests, which could lead to unpredictable/undesired behaviours in a multisite environment.

This change moves that logic out of GlobalSettings so that the returned theme IDs are correct for the current site.

Uses get_set_cache, which ultimately uses DistributedCache, which will take care of multisite issues for us.
2021-10-29 11:46:52 -04:00
Yasuo Honda dbbfad7ed0 FIX: Support Ruby 3 keyword arguments 2021-10-05 11:25:00 -04:00
Penar Musaraj c8f4f7c235
FIX: Ignore missing uploads in theme settings (#13486)
In some rare cases, this could prevent the site from bootstrapping,
because theme settings are invoked early in the application.
2021-06-22 22:34:22 -04:00
Alan Guo Xiang Tan 44aa46ca05 Code review comments. 2021-06-21 11:06:58 +08:00
Alan Guo Xiang Tan 8e3691d537 PERF: Eager load Theme associations in Stylesheet Manager.
Before this change, calling `StyleSheet::Manager.stylesheet_details`
for the first time resulted in multiple queries to the database. This is
because the code was modelled in a way where each `Theme` was loaded
from the database one at a time.

This PR restructures the code such that it allows us to load all the
theme records in a single query. It also allows us to eager load the
required associations upfront. In order to achieve this, I removed the
support of loading multiple themes per request. It was initially added
to support user selectable theme components but the feature was never
completed and abandoned because it wasn't a feature that we thought was
worth building.
2021-06-21 11:06:58 +08:00
Alan Guo Xiang Tan a8667b5454 PERF: Defer setting of distributed cache in more spots.
See follow up commit for rational.

Follow-up to 8cfe203383
2021-06-04 09:13:18 +08:00
Alan Guo Xiang Tan 8cfe203383 PERF: Defer setting of distributed cache in performance critical paths.
Setting a key/value pair in DistributedCache involves waiting on the
write to Redis to finish. In most cases, we don't need to wait on the
setting of the cache to finish. We just need to take our return value
and move on.
2021-06-03 09:30:52 +08:00
David Taylor ea61bcaf13
DEV: Define theme test modules under a `/test` subdirectory (#13158) 2021-05-26 11:49:51 +01:00
Osama Sayegh 4f88f2eb15
FEATURE: Allow theme tests to be run in production (take 2) (#12845)
This commit allows site admins to run theme tests in production via a new `/theme-qunit` route. When you visit `/theme-qunit`, you'll see a list of the themes/components installed on your site that have tests, and from there you can select a theme or component that you run its tests.

We also have a new rake task `themes:install_and_test` that can be used to install a list of themes/components on a temporary database and run the tests of the themes/components that are installed. This rake task can be useful when upgrading/deploying a Discourse instance to make sure that the installed themes/components are compatible with the new Discourse version being deployed, and if the tests fail you can abort the build/deploy process so you don't end up with a broken site.
2021-04-28 23:12:08 +03:00
David Taylor 1fd8f6df5f
PERF: Improve theme stylesheet compilation performance (#12850)
When building the `scss_load_paths`, we were creating a full export of the theme (including uploads), and not cleaning it up. With many uploads, this can be extremely slow (because it downloads every upload from S3), and the lack of cleanup could cause a disk to fill up over time.

This commit updates the ZipExporter to provide a `with_export_dir` API, which takes care of cleanup. It also adds a kwarg which allows exporting only extra_scss fields. This should make things much faster for themes with many uploads.
2021-04-27 14:33:43 +01:00