Commit Graph

303 Commits

Author SHA1 Message Date
Daniel Waterworth b7404373cf
FIX: Always clear caches after committing the current transaction (#22550)
Instead of having to remember every time, just always wait until the
current transaction (if it exists) has committed before clearing any
DistributedCache.

The only exception to this is caches that aren't caching things from
postgres.

This means we have to do the test setup after setting the test
transaction, because doing the test setup involves clearing caches.

Reapplying this - it now doesn't use after_commit if skip_db is set
2023-07-12 09:49:28 -05:00
Daniel Waterworth cb794275a7
Revert "FIX: Always clear caches after committing the current transaction" (#22493)
This reverts commit 8310c7842c.

This was breaking precompilation.
2023-07-07 18:05:38 -05:00
Daniel Waterworth 9dd01ca2ef
FIX: Always clear caches after committing the current transaction (#22491)
Instead of having to remember every time, just always wait until the
current transaction (if it exists) has committed before clearing any
DistributedCache.

The only exception to this is caches that aren't caching things from
postgres.

This means we have to do the test setup after setting the test
transaction, because doing the test setup involves clearing caches.
2023-07-07 14:24:56 -05:00
Vinoth Kannan 52894b9d7c
FEATURE: display commit hash for each plugin on `/admin/plugins` page. (#22176)
It will help to find out the current version of the plugins even without the `docker_manager` plugin.
2023-06-26 10:09:57 +05:30
Bianca Nenciu 5fc1586abf
PERF: Cache ToS and Privacy Policy paths (#21860)
Checking if the topic exists happened often and that can cause
performance issues.
2023-06-07 21:31:20 +03:00
Osama Sayegh c2fcd55a80
FEATURE: Serve RTL versions of admin and plugins CSS bundles for RTL locales (#21876)
Prior to this commit, we didn't have RTL versions of our admin and plugins CSS bundles and we always served LTR versions of those bundles even when users used an RTL locale, causing admin and plugins UI elements to never look as good as when an LTR locale was used. Example of UI issues prior to this commit were: missing margins, borders on the wrong side and buttons too close to each other etc.

This commit creates an RTL version for the admin CSS bundle as well as RTL bundles for all the installed plugins and serves those RTL bundles to users/sites who use RTL locales.
2023-06-01 05:27:11 +03:00
Sam c2332d7505
FEATURE: reduce avatar sizes to 6 from 20 (#21319)
* FEATURE: reduce avatar sizes to 6 from 20

This PR introduces 3 changes:

1. SiteSetting.avatar_sizes, now does what is says on the tin.
previously it would introduce a large number of extra sizes, to allow for
various DPIs. Instead we now trust the admin with the size list.

2. When `avatar_sizes` changes, we ensure consistency and remove resized
avatars that are not longer allowed per site setting. This happens on the
12 hourly job and limited out of the box to 20k cleanups per cycle, given
this may reach out to AWS 20k times to remove things.

3.Our default avatar sizes are now "24|48|72|96|144|288" these sizes were
very specifically picked to limit amount of bluriness introduced by webkit.
Our avatars are already blurry due to 1px border, so this corrects old blur.

This change heavily reduces storage required by forums which simplifies
site moves and more.

Co-authored-by: David Taylor <david@taylorhq.com>
2023-06-01 10:00:01 +10:00
Jarek Radosz bd742de0b7
DEV: Avoid double deprecation warnings in dev (#21747) 2023-05-25 14:51:38 +02:00
Bianca Nenciu d3a5a493fa
DEV: Add configurable? helper to Plugin::Instance (#21472)
This reapplies commit 3073e5cfb0, with
a fix that makes sure that plugins can be looked up both by the name
present in metadata and directory name.
2023-05-10 16:21:48 +03:00
Joffrey JAFFEUX 3727c95f6f
Revert "DEV: Add configurable? helper to Plugin::Instance (#20767)" (#21469)
This reverts commit 3073e5cfb0.
2023-05-10 12:41:55 +02:00
Bianca Nenciu 3073e5cfb0
DEV: Add configurable? helper to Plugin::Instance (#20767)
This can be used to forcibly disable plugins.
2023-05-10 13:16:37 +03:00
David Taylor 26b7f8a63b
DEV: Improve add_to_serializer include_* options (#21220)
- Move the old '`define_include_method`' arg to a `respect_plugin_enabled` kwarg

- Introduce an `include_condition` kwarg which can be passed a lambda with inclusion logic. Lambda will be run via `instance_exec` in the context of the serializer instance

This is backwards compatible - old-style invocations will trigger a deprecation message
2023-04-24 12:17:51 +01:00
Gerhard Schlager 1ee87cbfa3 FEATURE: Detect current git "branch" even when a tag is checked out
It switches to a different command for detecting the current git branch because the old command always returned HEAD as branch when the git repository is on a detached head (e.g. tag). The new command doesn't return a branch when the repository is on a detached head, which allows us to fall back to the `version` variable that is stored in the git config since https://github.com/discourse/discourse_docker/pull/707. It contains the value of the `version` from `app.yml`.

It also includes a small change to specs, because our tests usually run on specific commits instead of a branch or tag, so Discourse.git_branch always returns "unknown". We can use the "unknown" branch for tests, so it makes sense to ignore it only in other envs.
2023-04-19 15:30:12 +02:00
Gerhard Schlager f523dcf9df REFACTOR: Simplify git command execution 2023-04-19 15:30:12 +02:00
Daniel Waterworth c68497159f
FIX: Use string for postgres recently readonly DistributedCache (#21040)
Since DistributedCaches don't marshal timestamps
2023-04-10 13:54:55 -05:00
Jarek Radosz 29e2e3ff3b
DEV: Fix random typos (#20937) 2023-04-03 19:27:32 +02:00
Daniel Waterworth 2df0eca39a
DEV: Make postgres_readonly cache work like other caches (#20879)
We didn't have an authoritative source for this data previously, so now
it's stored in redis.
2023-03-30 09:14:59 -05:00
David Taylor 5406e24acb
FEATURE: Introduce pg_force_readonly_mode GlobalSetting (#19612)
This allows the entire cluster to be forced into pg readonly mode. Equivalent to running `Discourse.enable_pg_force_readonly_mode` on the console.
2023-01-19 13:59:11 +00:00
David Taylor 29f7ec7090
DEV: Prevent defer stats exception when thread aborted (#19863)
When the thread is aborted, an exception is raised before the `start` of a job is set, and therefore raises an exception in the `ensure` block. This commit checks that `start` exists, and also adds `abort_on_exception=true` so that this issue would have caused test failures.
2023-01-16 09:08:44 +11:00
Ted Johansson 5dcb245eac
FIX: Ruby 2 backward compatible plugin logout redirect (#19845)
This is a very subtle one. Setting the redirect URL is done by passing
a hash through a Discourse event. This is broken on Ruby 2 since the
support for keyword arguments in events was added.

In Ruby 2 the last argument is cast to keyword arguments if it is a
hash. The key point here is that creates a new copy of the hash, so
what the plugin is modifying is not the hash that was passed.
2023-01-12 19:12:20 +08:00
Daniel Waterworth dfc449a32e
FIX: Continue if checking themes if updating one fails (#19665) 2023-01-09 11:56:43 -06:00
David Taylor 6417173082
DEV: Apply syntax_tree formatting to `lib/*` 2023-01-09 12:10:19 +00:00
Penar Musaraj 8546c2084a
DEV: Load SVG sprites during system spec runs (#19497)
Co-authored-by: David Taylor <david@taylorhq.com>
2022-12-22 08:13:43 -05:00
Vinoth Kannan 076abe46fa
FEATURE: new site setting to set locale from cookie for anonymous users. (#18377)
This new hidden default-disabled site setting `set_locale_from_cookie` will set locale from anonymous user's cookie value.
2022-09-27 14:26:06 +05:30
David Taylor d0243f741e
UX: Use dominant color as image loading placeholder (#18248)
We previously had a system which would generate a 10x10px preview of images and add their URLs in a data-small-upload attribute. The client would then use that as the background-image of the `<img>` element. This works reasonably well on fast connections, but on slower connections it can take a few seconds for the placeholders to appear. The act of loading the placeholders can also break or delay the loading of the 'real' images.

This commit replaces the placeholder logic with a new approach. Instead of a 10x10px preview, we use imagemagick to calculate the average color of an image and store it in the database. The hex color value then added as a `data-dominant-color` attribute on the `<img>` element, and the client can use this as a `background-color` on the element while the real image is loading. That means no extra HTTP request is required, and so the placeholder color can appear instantly.

Dominant color will be calculated:
1. When a new upload is created
2. During a post rebake, if the dominant color is missing from an upload, it will be calculated and stored
3. Every 15 minutes, 25 old upload records are fetched and their dominant color calculated and stored. (part of the existing PeriodicalUpdates job)

Existing posts will continue to use the old 10x10px placeholder system until they are next rebaked
2022-09-20 10:28:17 +01:00
David Taylor 9ebebfb4cc
FIX: Load admin-specific JS when compiling via ember-cli (#18086)
The previous sprockets implementation was including admin-specific JS in the plugin's main JS file, which would be served to all users regardless of admin status. This commit achieves the same result under the ember-cli plugin asset compiler with one difference: the admin js is compiled into a separate file. That means that in future, we'll be able to make it loaded only for admins. For now though, it's loaded for everyone, just like before.
2022-08-25 11:36:02 +01:00
David Taylor 33a2624f09
DEV: Introduce flag for compiling Plugin JS with Ember CLI (#17965)
When `EMBER_CLI_PLUGIN_ASSETS=1`, plugin application JS will be compiled via Ember CLI. In this mode, the existing `register_asset` API will cause any registered JS files to be made available in `/plugins/{plugin-name}_extra.js`. These 'extra' files will be loaded immediately after the plugin app JS file, so this should not affect functionality.

Plugin compilation in Ember CLI is implemented as an addon, similar to the existing 'admin' addon. We bypass the normal Ember CLI compilation process (which would add the JS to the main app bundle), and reroute the addon Broccoli tree into a separate JS file per-plugin. Previously, Sprockets would add compiled templates directly to `Ember.TEMPLATES`. Under Ember CLI, they are compiled into es6 modules. Some new logic in `discourse-boot.js` takes care of remapping the new module names into the old-style `Ember.TEMPLATES`.

This change has been designed to be a like-for-like replacement of the old plugin compilation system, so we do not expect any breakage. Even so, the environment variable flag will allow us to test this in a range of environments before enabling it by default.

A manual silence implementation is added for the build-time `ember-glimmer.link-to.positional-arguments` deprecation while we work on a better story for plugins.
2022-08-22 09:56:39 +01:00
Sam 4967541275
FIX: properly log all internal job failures (#17805)
Our internal implementation of #perform on jobs performs remapping.

This happens cause we do "exception aggregation".

Scheduled jobs run on every site in the multisite cluster, and we report
one error per site that failed. During this aggregation we reshape the
context from the original object shape returned by mini_scheduler

The new integration test ensures this interface will remain stable even if
decoupled parts of the code change shapes.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2022-08-05 17:40:22 +10:00
Sam 3b42e69174
FIX: avoid usage of dig when looking for job class (#17772)
`{a: "a"}.dig(:a, :b)` will result in an exception, since ruby assumes that `"a"` will be another hash it can look up the `:b` key on.
2022-08-03 14:28:46 +10:00
Sam bfe502012d
FEATURE: track stats around failing scheduled jobs (#17769)
* FEATURE: track stats around failing scheduled jobs

Discourse.job_exception_stats can now be used to gather stats around how
many regular scheduled jobs failed in the current process.

This will be consumed by the Prometheus plugin and potentially other
monitoring plugins.
2022-08-03 12:53:26 +10:00
Osama Sayegh 306dca3a48
DEV: Add a helper method to clear every possible theme cache (#17585)
Our theme system is very complex and it can take a while to figure out how to invalidate the various types of caches that are used throughout the theme system. So, having a single helper method that invalidates everything can be useful in emergency situations where there is no time to read through the code and figure out how to clear the various caches.

Internal ticket: t64732.
2022-07-22 09:46:52 +03:00
Alan Guo Xiang Tan 3562c598ef
DEV: Upgrade Sidekiq to 6.5 (#17142) 2022-06-21 09:23:36 +08:00
Daniel Waterworth 6e53f4d913
DEV: New readonly mode. Only applies to non-staff (#16243) 2022-05-17 13:06:08 -05:00
Loïc Guitaut 008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
David Taylor 22a7905f2d
DEV: Allow Ember CLI assets to be used by development Rails app (#16511)
Previously, accessing the Rails app directly in development mode would give you assets from our 'legacy' Ember asset pipeline. The only way to run with Ember CLI assets was to run ember-cli as a proxy. This was quite limiting when working on things which are bypassed when using the ember-cli proxy (e.g. changes to `application.html.erb`). Also, since `ember-auto-import` introduced chunking, visiting `/theme-qunit` under Ember CLI was failing to include all necessary chunks.

This commit teaches Sprockets about our Ember CLI assets so that they can be used in development mode, and are automatically collected up under `/public/assets` during `assets:precompile`. As a bonus, this allows us to remove all the custom manifest modification from `assets:precompile`.

The key changes are:
- Introduce a shared `EmberCli.enabled?` helper
- When ember-cli is enabled, add ember-cli `/dist/assets` as the top-priority Rails asset directory
- Have ember-cli output a `chunks.json` manifest, and teach `preload_script` to read it and append the correct chunks to their associated `afterFile`
- Remove most custom ember-cli logic from the `assets:precompile` step. Instead, rely on Rails to take care of pulling the 'precompiled' assets into the `public/assets` directory. Move the 'renaming' logic to runtime, so it can be used in development mode as well.
- Remove fingerprinting from `ember-cli-build`, and allow Rails to take care of things

Long-term, we may want to replace Sprockets with the lighter-weight Propshaft. The changes made in this commit have been made with that long-term goal in mind.

tldr: when you visit the rails app directly, you'll now be served the current ember-cli assets. To keep these up-to-date make sure either `ember serve`, or `ember build --watch` is running. If you really want to load the old non-ember-cli assets, then you should start the server with `EMBER_CLI_PROD_ASSETS=0`. (the legacy asset pipeline will be removed very soon)
2022-04-21 16:26:34 +01:00
David Taylor be8c0baa18
DEV: Include exception class in Discourse.warn_exception log (#15822)
Sometimes, the 'message' portion of an exception isn't enough to work out what's happening. In these cases, including the exception class name can help with debugging.
2022-02-04 19:41:08 +00:00
Jarek Radosz 40147ca2ee
DEV: Don't warn on missing git tags (#15507)
Fixes the following output in specs:

```
fatal: No names found, cannot describe anything.
```
2022-01-09 20:25:58 +01:00
Jarek Radosz 5a50f18c0c
DEV: Avoid `$` globals (#15453)
Also:
* Remove an unused method (#fill_email)
* Replace a method that was used just once (#generate_username) with `SecureRandom.alphanumeric`
* Remove an obsolete dev puma `tmp/restart` file logic
2022-01-08 23:39:46 +01:00
David Taylor 78d0ec35a5
DEV: Deprecate OAuth2Authenticator and OAuth2UserInfo (#15427)
These have been superseded by ManagedAuthenticator and UserAssociatedAccount. For more information, see https://meta.discourse.org/t/106695
2022-01-06 16:50:18 +00:00
David Taylor 1fa7a87f86
SECURITY: Remove ember-cli specific response from application routes (#15155)
Under some conditions, these varied responses could lead to cache poisoning, hence the 'security' label.

Previously the Rails application would serve JSON data in place of HTML whenever Ember CLI requested an `application.html.erb`-rendered page. This commit removes that logic, and instead parses the HTML out of the standard response. This means that Rails doesn't need to customize its response for Ember CLI.
2021-12-01 16:10:40 +00:00
Jarek Radosz c9e0ad5ec4
DEV: Remove build:stamp task and related code (#15069)
I don't think we use `build:stamp` task anymore, nor we rely on config/version.rb
2021-11-29 19:09:05 +01:00
Jarek Radosz d40e56272f
DEV: Remove unnecessary requires (#15067)
`auth/*` one was causing warnings (the file was loaded twice)

and `sha1` isn't used anymore here
2021-11-23 23:42:24 +01:00
Andrei Prigorshnev 622859dbe6
FEATURE: add Unseen view (#13977)
This view is the same as Latest except it hides the topics you have fully read. Based on this plugin of @davidtaylorhq https://meta.discourse.org/t/simple-unread-list-plugin-discourse-simple-unread/70013.
2021-08-10 18:30:34 +04:00
Laurent Arnoud 03896a9c43
FIX: Errno::EXDEV when across filesystem boundaries (#13976)
ref https://bugs.ruby-lang.org/issues/12615

```
        W: rake aborted!
        W: Errno::EXDEV: Invalid cross-device link @ rb_file_s_rename - (/app/tmp/adeeee62504de67238341871bda1aaf0, /app/app/assets/javascripts/plugins/discourse-cakeday.js.e
rb)
        W: lib/discourse.rb:65:in `rename'
        W: lib/discourse.rb:65:in `atomic_write_file'
        W: /app/lib/plugin/instance.rb:683:in `activate!'
        W: lib/discourse.rb:246:in `block in activate_plugins!'
        W: lib/discourse.rb:243:in `each'
        W: lib/discourse.rb:243:in `activate_plugins!'
```
2021-08-09 12:20:26 +02:00
Bianca Nenciu e2c415457c
FEATURE: Attach backup log as upload (#13849)
Discourse automatically sends a private message after backup or
restore finished. The private message used to contain the log inline
even when it was very long. A very long log can create issues because
the length of the post will be over the maximum allowed length of a
post. When that happens, Discourse will try to create an upload with
the logs. If that fails, it will trim the log and inline it.
2021-08-03 20:06:50 +03:00
Vinoth Kannan 08c6a7679d
DEV: allow `dev:populate` rake task in production environment. (#13777)
To use this rake task in production `ALLOW_DEV_POPULATE` environment variable should be set to "1".
2021-07-20 12:25:59 +05:30
Alan Guo Xiang Tan 2717615f11
PERF: Preload client settings json while booting app. (#13283)
Generating the client settings json involves santizing all string based
site settings. This is expensive as per our profile in production (~120ms) and one request after each deploy has
to pay this penalty.
2021-06-08 13:15:55 +10:00
Alan Guo Xiang Tan cadf5eafe6 DEV: Move Discourse app specific concern out of unicorn conf. 2021-06-04 09:13:34 +08:00
Alan Guo Xiang Tan 58b30fb510 PERF: Preload settings, groups and badge icons in SvgSprite.
Identified as a hot path in production. Preload it early instead of
executing the queries in a live request.
2021-06-03 16:45:56 +08:00
Alan Guo Xiang Tan 2e4182b4b3 PERF: Warm up caches in `SiteSerializer` while preloading app.
Otherwise, time will be spent warming up the caches in the initial
requests.
2021-06-02 13:33:31 +08:00