Commit Graph

279 Commits

Author SHA1 Message Date
David Taylor d0243f741e
UX: Use dominant color as image loading placeholder (#18248)
We previously had a system which would generate a 10x10px preview of images and add their URLs in a data-small-upload attribute. The client would then use that as the background-image of the `<img>` element. This works reasonably well on fast connections, but on slower connections it can take a few seconds for the placeholders to appear. The act of loading the placeholders can also break or delay the loading of the 'real' images.

This commit replaces the placeholder logic with a new approach. Instead of a 10x10px preview, we use imagemagick to calculate the average color of an image and store it in the database. The hex color value then added as a `data-dominant-color` attribute on the `<img>` element, and the client can use this as a `background-color` on the element while the real image is loading. That means no extra HTTP request is required, and so the placeholder color can appear instantly.

Dominant color will be calculated:
1. When a new upload is created
2. During a post rebake, if the dominant color is missing from an upload, it will be calculated and stored
3. Every 15 minutes, 25 old upload records are fetched and their dominant color calculated and stored. (part of the existing PeriodicalUpdates job)

Existing posts will continue to use the old 10x10px placeholder system until they are next rebaked
2022-09-20 10:28:17 +01:00
David Taylor 9ebebfb4cc
FIX: Load admin-specific JS when compiling via ember-cli (#18086)
The previous sprockets implementation was including admin-specific JS in the plugin's main JS file, which would be served to all users regardless of admin status. This commit achieves the same result under the ember-cli plugin asset compiler with one difference: the admin js is compiled into a separate file. That means that in future, we'll be able to make it loaded only for admins. For now though, it's loaded for everyone, just like before.
2022-08-25 11:36:02 +01:00
David Taylor 33a2624f09
DEV: Introduce flag for compiling Plugin JS with Ember CLI (#17965)
When `EMBER_CLI_PLUGIN_ASSETS=1`, plugin application JS will be compiled via Ember CLI. In this mode, the existing `register_asset` API will cause any registered JS files to be made available in `/plugins/{plugin-name}_extra.js`. These 'extra' files will be loaded immediately after the plugin app JS file, so this should not affect functionality.

Plugin compilation in Ember CLI is implemented as an addon, similar to the existing 'admin' addon. We bypass the normal Ember CLI compilation process (which would add the JS to the main app bundle), and reroute the addon Broccoli tree into a separate JS file per-plugin. Previously, Sprockets would add compiled templates directly to `Ember.TEMPLATES`. Under Ember CLI, they are compiled into es6 modules. Some new logic in `discourse-boot.js` takes care of remapping the new module names into the old-style `Ember.TEMPLATES`.

This change has been designed to be a like-for-like replacement of the old plugin compilation system, so we do not expect any breakage. Even so, the environment variable flag will allow us to test this in a range of environments before enabling it by default.

A manual silence implementation is added for the build-time `ember-glimmer.link-to.positional-arguments` deprecation while we work on a better story for plugins.
2022-08-22 09:56:39 +01:00
Sam 4967541275
FIX: properly log all internal job failures (#17805)
Our internal implementation of #perform on jobs performs remapping.

This happens cause we do "exception aggregation".

Scheduled jobs run on every site in the multisite cluster, and we report
one error per site that failed. During this aggregation we reshape the
context from the original object shape returned by mini_scheduler

The new integration test ensures this interface will remain stable even if
decoupled parts of the code change shapes.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2022-08-05 17:40:22 +10:00
Sam 3b42e69174
FIX: avoid usage of dig when looking for job class (#17772)
`{a: "a"}.dig(:a, :b)` will result in an exception, since ruby assumes that `"a"` will be another hash it can look up the `:b` key on.
2022-08-03 14:28:46 +10:00
Sam bfe502012d
FEATURE: track stats around failing scheduled jobs (#17769)
* FEATURE: track stats around failing scheduled jobs

Discourse.job_exception_stats can now be used to gather stats around how
many regular scheduled jobs failed in the current process.

This will be consumed by the Prometheus plugin and potentially other
monitoring plugins.
2022-08-03 12:53:26 +10:00
Osama Sayegh 306dca3a48
DEV: Add a helper method to clear every possible theme cache (#17585)
Our theme system is very complex and it can take a while to figure out how to invalidate the various types of caches that are used throughout the theme system. So, having a single helper method that invalidates everything can be useful in emergency situations where there is no time to read through the code and figure out how to clear the various caches.

Internal ticket: t64732.
2022-07-22 09:46:52 +03:00
Alan Guo Xiang Tan 3562c598ef
DEV: Upgrade Sidekiq to 6.5 (#17142) 2022-06-21 09:23:36 +08:00
Daniel Waterworth 6e53f4d913
DEV: New readonly mode. Only applies to non-staff (#16243) 2022-05-17 13:06:08 -05:00
Loïc Guitaut 008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
David Taylor 22a7905f2d
DEV: Allow Ember CLI assets to be used by development Rails app (#16511)
Previously, accessing the Rails app directly in development mode would give you assets from our 'legacy' Ember asset pipeline. The only way to run with Ember CLI assets was to run ember-cli as a proxy. This was quite limiting when working on things which are bypassed when using the ember-cli proxy (e.g. changes to `application.html.erb`). Also, since `ember-auto-import` introduced chunking, visiting `/theme-qunit` under Ember CLI was failing to include all necessary chunks.

This commit teaches Sprockets about our Ember CLI assets so that they can be used in development mode, and are automatically collected up under `/public/assets` during `assets:precompile`. As a bonus, this allows us to remove all the custom manifest modification from `assets:precompile`.

The key changes are:
- Introduce a shared `EmberCli.enabled?` helper
- When ember-cli is enabled, add ember-cli `/dist/assets` as the top-priority Rails asset directory
- Have ember-cli output a `chunks.json` manifest, and teach `preload_script` to read it and append the correct chunks to their associated `afterFile`
- Remove most custom ember-cli logic from the `assets:precompile` step. Instead, rely on Rails to take care of pulling the 'precompiled' assets into the `public/assets` directory. Move the 'renaming' logic to runtime, so it can be used in development mode as well.
- Remove fingerprinting from `ember-cli-build`, and allow Rails to take care of things

Long-term, we may want to replace Sprockets with the lighter-weight Propshaft. The changes made in this commit have been made with that long-term goal in mind.

tldr: when you visit the rails app directly, you'll now be served the current ember-cli assets. To keep these up-to-date make sure either `ember serve`, or `ember build --watch` is running. If you really want to load the old non-ember-cli assets, then you should start the server with `EMBER_CLI_PROD_ASSETS=0`. (the legacy asset pipeline will be removed very soon)
2022-04-21 16:26:34 +01:00
David Taylor be8c0baa18
DEV: Include exception class in Discourse.warn_exception log (#15822)
Sometimes, the 'message' portion of an exception isn't enough to work out what's happening. In these cases, including the exception class name can help with debugging.
2022-02-04 19:41:08 +00:00
Jarek Radosz 40147ca2ee
DEV: Don't warn on missing git tags (#15507)
Fixes the following output in specs:

```
fatal: No names found, cannot describe anything.
```
2022-01-09 20:25:58 +01:00
Jarek Radosz 5a50f18c0c
DEV: Avoid `$` globals (#15453)
Also:
* Remove an unused method (#fill_email)
* Replace a method that was used just once (#generate_username) with `SecureRandom.alphanumeric`
* Remove an obsolete dev puma `tmp/restart` file logic
2022-01-08 23:39:46 +01:00
David Taylor 78d0ec35a5
DEV: Deprecate OAuth2Authenticator and OAuth2UserInfo (#15427)
These have been superseded by ManagedAuthenticator and UserAssociatedAccount. For more information, see https://meta.discourse.org/t/106695
2022-01-06 16:50:18 +00:00
David Taylor 1fa7a87f86
SECURITY: Remove ember-cli specific response from application routes (#15155)
Under some conditions, these varied responses could lead to cache poisoning, hence the 'security' label.

Previously the Rails application would serve JSON data in place of HTML whenever Ember CLI requested an `application.html.erb`-rendered page. This commit removes that logic, and instead parses the HTML out of the standard response. This means that Rails doesn't need to customize its response for Ember CLI.
2021-12-01 16:10:40 +00:00
Jarek Radosz c9e0ad5ec4
DEV: Remove build:stamp task and related code (#15069)
I don't think we use `build:stamp` task anymore, nor we rely on config/version.rb
2021-11-29 19:09:05 +01:00
Jarek Radosz d40e56272f
DEV: Remove unnecessary requires (#15067)
`auth/*` one was causing warnings (the file was loaded twice)

and `sha1` isn't used anymore here
2021-11-23 23:42:24 +01:00
Andrei Prigorshnev 622859dbe6
FEATURE: add Unseen view (#13977)
This view is the same as Latest except it hides the topics you have fully read. Based on this plugin of @davidtaylorhq https://meta.discourse.org/t/simple-unread-list-plugin-discourse-simple-unread/70013.
2021-08-10 18:30:34 +04:00
Laurent Arnoud 03896a9c43
FIX: Errno::EXDEV when across filesystem boundaries (#13976)
ref https://bugs.ruby-lang.org/issues/12615

```
        W: rake aborted!
        W: Errno::EXDEV: Invalid cross-device link @ rb_file_s_rename - (/app/tmp/adeeee62504de67238341871bda1aaf0, /app/app/assets/javascripts/plugins/discourse-cakeday.js.e
rb)
        W: lib/discourse.rb:65:in `rename'
        W: lib/discourse.rb:65:in `atomic_write_file'
        W: /app/lib/plugin/instance.rb:683:in `activate!'
        W: lib/discourse.rb:246:in `block in activate_plugins!'
        W: lib/discourse.rb:243:in `each'
        W: lib/discourse.rb:243:in `activate_plugins!'
```
2021-08-09 12:20:26 +02:00
Bianca Nenciu e2c415457c
FEATURE: Attach backup log as upload (#13849)
Discourse automatically sends a private message after backup or
restore finished. The private message used to contain the log inline
even when it was very long. A very long log can create issues because
the length of the post will be over the maximum allowed length of a
post. When that happens, Discourse will try to create an upload with
the logs. If that fails, it will trim the log and inline it.
2021-08-03 20:06:50 +03:00
Vinoth Kannan 08c6a7679d
DEV: allow `dev:populate` rake task in production environment. (#13777)
To use this rake task in production `ALLOW_DEV_POPULATE` environment variable should be set to "1".
2021-07-20 12:25:59 +05:30
Alan Guo Xiang Tan 2717615f11
PERF: Preload client settings json while booting app. (#13283)
Generating the client settings json involves santizing all string based
site settings. This is expensive as per our profile in production (~120ms) and one request after each deploy has
to pay this penalty.
2021-06-08 13:15:55 +10:00
Alan Guo Xiang Tan cadf5eafe6 DEV: Move Discourse app specific concern out of unicorn conf. 2021-06-04 09:13:34 +08:00
Alan Guo Xiang Tan 58b30fb510 PERF: Preload settings, groups and badge icons in SvgSprite.
Identified as a hot path in production. Preload it early instead of
executing the queries in a live request.
2021-06-03 16:45:56 +08:00
Alan Guo Xiang Tan 2e4182b4b3 PERF: Warm up caches in `SiteSerializer` while preloading app.
Otherwise, time will be spent warming up the caches in the initial
requests.
2021-06-02 13:33:31 +08:00
Alan Guo Xiang Tan 0700e9382d PERF: Memoize core svgs in memory to avoid expensive XML parsing.
The XML parsing of SVGs is done whenever the cache expires or on the
first load after a reboot. In one of our production instance, parsing
ranges from 30ms to 70ms which is not ideal. Instead, we've decided to
make a small memory trade off here by memoizing the core SVGs once on
boot to avoid parsing of the SVG files during the duration of a request.
The memozied hash will take up 57440 bytes or 0.05744 megabytes in size.
2021-06-02 09:16:37 +08:00
Gerhard Schlager c1c090419b DEV: Fix warning about already initialized constants in `FileStore::BaseStore` 2021-06-01 09:30:37 +08:00
Alan Guo Xiang Tan f05f5fde0f
PERF: Preload `JsLocaleHelper.load_translations` for default locale. (#13041)
In production, each Unicorn child process will currently hold the
default locale in memory on first load. Instead, we should preload it in
the Unicorn master process so that the memory is immediately shared when
forking.

Also, the translations are only memoized on first load now and is
adding considerable overhead to the first few requests after a fresh
boot.
2021-05-13 16:16:01 +10:00
Alan Guo Xiang Tan 4bad1e7e93 PERF: Preload `LetterAvatar.image_magick_version` in master process.
It is wasteful to run the clean up and command to generate the version
digest on every unicorn child process.
2021-05-07 14:00:38 +08:00
David Taylor cac7725e28
DEV: Improve `bin/unicorn` boot time in development environment (#12900)
This commit makes a few changes to improve boot time in development environments. It will have no effect on production boot times.

- Skip the SchemaCache warmup. In development mode, the SchemaCache is refreshed every time there is a code change, so warmup is of limited use.

- Skip warming up PrettyText. This adds ~2s to each web worker's boot time. The vast majority of requests do not use PrettyText, so it is more efficient to defer its warmup until it's needed

- Skip the intentional 1 second pause during Unicorn worker forking. The comment (which also exists in Unicorn's documentation) suggests this works around a Unix signal handling bug, but I haven't been able to locate any more information. Skipping it in dev will significantly speed up boot. If we start to see issues, we can revert this change.

On my machine, this improves `/bin/unicorn` boot time from >10s to ~4s
2021-04-30 11:32:13 +01:00
Robin Ward e3b1d1a718
DEV: Improve Ember CLI's bootstrap logic (#12792)
* DEV: Give a nicer error when `--proxy` argument is missing

* DEV: Improve Ember CLI's bootstrap logic

Instead of having Ember CLI know which URLs to proxy or not, have it try
the URL with a special header `HTTP_X_DISCOURSE_EMBER_CLI`. If present,
and Discourse thinks we should bootstrap the application, it will
instead stop rendering and return a HTTP HEAD with a response header
telling Ember CLI to bootstrap.

In other words, any time Rails would otherwise serve up the HTML for the
Ember app, it stops and says "no, you do it."

* DEV: Support asset filters by path using a new options object

Without this, Ember CLI's bootstrap would not get the assets it wants
because the path it was requesting was different than the browser path.
This adds an optional request header to fix it.

So far this is only used by the styleguide.
2021-04-23 10:24:42 -04:00
David Taylor 0ec5fd5262
DEV: Raise exception when execute_command will spawn a shell (#12716) 2021-04-15 16:29:37 +01:00
David Taylor 65647000a1
DEV: Allow `Discourse::Utils.execute_command` timeout with `env` (#12672)
Followup to 5deda5ef3e

The first argument to `Open3.capture3` can be an environment variable hash. In this case, we need to insert the `timeout` command after the env hash.
2021-04-12 13:53:41 +01:00
Sam 5deda5ef3e
FIX: automatically timeout long running image magick commands (#12670)
Previously certain images may lead to convert / identify to run for unreasonable
amounts of time

This adds a maximum amount of time these commands can run prior to forcing
them to stop
2021-04-12 13:55:54 +10:00
Vinoth Kannan a5923ad603
DEV: apply allow origin response header for CDN requests. (#11893)
Currently, it creates a CORS error while accessing those static files.
2021-01-29 07:44:49 +05:30
Dan Ungureanu 123107c28f
UX: Add group name to error message (#11333)
The group name used to be part of the error message, but was removed
in a past commit.
2020-11-24 13:06:52 +02:00
Dan Ungureanu 9b7525bb03
FIX: Handle uncaught exception (#11263)
After the search term is parsed for advanced search filters, the term may
become empty. Later, the same term will be passed to Discourse.route_for
which will raise an ArgumentError.

> URI(nil)
ArgumentError: bad argument (expected URI object or URI string)
2020-11-20 11:28:14 +02:00
David Taylor 5289fc7886
FIX: Improve failover for multisite clusters (#11150)
- Bump rails_failover for new per-backend callback feature
- If the master backend fails over, make all sites readonly. And vice-versa for fallback
- If a single backend fails over, make that individual site readonly. And vice-versa for fallback
- When a single backend fails, also check connection to the master backend
2020-11-11 10:27:24 +00:00
Vinoth Kannan af4938baf1
Revert "DEV: enable cors to all cdn get requests from workbox. (#10684)" (#11076)
This reverts commit e3de45359f.

We need to improve out strategy by adding a cache breaker with this change ... some assets on CDNs and clients may have incorrect CORS headers which can cause stuff to break.
2020-10-30 16:05:35 +11:00
Vinoth Kannan 347423007a
DEV: remove instagram login site settings and auth classes. (#11073)
Instagram removed the support for login and should use Facebook login instead.
2020-10-30 09:09:56 +05:30
Vinoth Kannan e3de45359f
DEV: enable cors to all cdn get requests from workbox. (#10685)
Now all external requests from the service worker will be in CORS mode without credentials.
2020-10-28 23:36:19 +05:30
Daniel Waterworth 721ee36425
Replace `base_uri` with `base_path` (#10879)
DEV: Replace instances of Discourse.base_uri with Discourse.base_path

This is clearer because the base_uri is actually just a path prefix. This continues the work started in 555f467.
2020-10-09 12:51:24 +01:00
Régis Hanol c5c5bb4fad DEV: prevent already initialized constant warning 2020-08-07 16:28:43 +02:00
Guo Xiang Tan 255b0e9f14
PERF: Replace video and audio links in search blurb while indexing.
In the near future, we will be swtiching to PG headlines to generate the
search blurb. As such, we need to replace audio and video links in the
raw data used for headline generation. This also means that we avoid
replacing links each time we need to generate the blurb.
2020-08-06 12:25:03 +08:00
Vinoth Kannan 0884d570b1
FEATURE: add support for `top` filter in tag page. (#10281)
Currently, tag pages only have the `latest` filter.
2020-07-22 19:26:36 +05:30
Guo Xiang Tan 94a2a70462
DEV: Use a longer TTL for pg readonly mode. 2020-07-14 16:15:58 +08:00
Guo Xiang Tan 58e52c0e4f
DEV: Use rails_failover gem for ActiveRecord and Redis failover handling 2020-06-15 15:47:07 +08:00
Guo Xiang Tan 0ff86b00cb
DEV: Upgrade Redis to 4.2.1. 2020-06-15 10:05:22 +08:00
Guo Xiang Tan 78b5ab746c
DEV: No longer need to clear anon cache when toggling readonly mode. 2020-06-12 09:58:17 +08:00