Commit Graph

10412 Commits

Author SHA1 Message Date
Krzysztof Kotlarek 95564a3df2 SECURITY: Moderators cannot see user emails.
Unless `moderators_view_emails` SiteSetting is enabled, moderators should not be able to discover users’ emails.
2024-12-19 13:13:18 -03:00
Alan Guo Xiang Tan e4e5db57f0
DEV: Fix undefined method `check_email_sync_heartbeat` in unicorn conf (#30360)
This is a follow-up to 9812407f76
2024-12-19 10:10:11 +08:00
Loïc Guitaut 133a648d9b DEV: Fix policy classes delegating their `#call` method in services
There’s currently a bug when using a dedicated class as a policy in
services: if that class delegates its `#call` method (to an underlying
strategy object for example), then an error will be raised saying steps
aren’t allowed to provide default parameters.

This should not happen, and this patch fixes that issue.
2024-12-18 09:59:40 +01:00
Alan Guo Xiang Tan 9812407f76
FIX: Redo Sidekiq monitoring to restart stuck sidekiq processes (#30198)
This commit reimplements how we monitor Sidekiq processes that are
forked from the Unicorn master process. Prior to this change, we rely on
`Jobs::Heartbeat` to enqueue a `Jobs::RunHeartbeat` job every 3 minutes.
The `Jobs::RunHeartbeat` job then sets a Redis key with a timestamp. In
the Unicorn master process, we then fetch the timestamp that has been set
by the job from Redis every 30 minutes. If the timestamp has not been
updated for more than 30 minutes, we restart the Sidekiq process. The
fundamental flaw with this approach is that it fails to consider
deployments with multiple hosts and multiple Sidekiq processes. A
sidekiq process on a host may be in a bad state but the heartbeat check
will not restart the process because the `Jobs::RunHeartbeat` job is
still being executed by the working Sidekiq processes on other hosts.

In order to properly ensure that stuck Sidekiq processs are restarted,
we now rely on the [Sidekiq::ProcessSet](https://github.com/sidekiq/sidekiq/wiki/API#processes)
API that is supported by Sidekiq. The API provides us with "near real-time (updated every 5 sec)
info about the current set of Sidekiq processes running". The API
provides useful information like the hostname, pid and also when Sidekiq
last did its own heartbeat check. With that information, we can easily
determine if a Sidekiq process needs to be restarted from the Unicorn
master process.
2024-12-18 12:48:50 +08:00
Sam 4437aced91
FIX: use relations for new_in_category (#30313)
`new_in_category` was using `first` instead of `limit`

This meant it gets an array and that means that you can not operate on it easily in a modifier.

This ensures we always give the modifier a relation, with the notable exception of suggested topics.
2024-12-17 16:39:07 +11:00
David Taylor ea9cdf7d47
DEV: Compile theme raw-hbr to modules (#30299)
Previously, theme hbr files were compiled to an IIFE, which would be executed before the app is booted. That is causing silenced deprecations to be printed, because the deprecation-workflow isn't set up when the IIFE is run.

This commit updates the theme compiler so that it matches the ember-cli-based raw-hbs compiler. Templates are output to normal modules, which will then be loaded by the existing `eager-load-raw-templates` initializer. This runs after the app has started booting.
2024-12-16 17:31:49 +00:00
Gerhard Schlager 6b3e28216c
FEATURE: Allow pausing of restore before DB migration and uploads are restored (#30269)
This can be helpful if you need to fix problems in the DB before the DB gets migrated as well as before uploads are restored.
2024-12-16 12:50:08 +01:00
David Taylor ce8c2ef6d9
Revert "DEV: prioritize new email styles over existing, to make customization easier (#30244)" (#30297)
This reverts commit 9694dc6cb0.

Some of our previous email styling depended on this 'incorrect' ordering, so the change caused some text to become illegible. Reverting while we work out a better solution
2024-12-16 11:16:17 +00:00
Loïc Guitaut 9e9abe0a82 DEV: Unify params access in services
Currently, there are two ways (kind of) for accessing `params` inside a
service:
- when there is no contract or it hasn’t been reached yet, `params` is
  just the hash that was provided to the service. To access a key, you
  have to use the bracket notation `params[:my_key]`.
- when there is a contract and it has been executed successfully,
  `params` now references the contract and the attributes are accessible
  using methods (`params.my_key`).

This patch unifies how `params` exposes its attributes. Now, even if
there is no contract at all in a service, `params` will expose its
attributes through methods, that way things are more consistent.

This patch also makes sure there is always a `params` object available
even when no `params` key is provided to the service (this allows a
contract to fail because its attributes are blank instead of having the
service raising an error because it doesn’t find `params` in its context).
2024-12-13 11:13:18 +01:00
Alan Guo Xiang Tan ebfc33b556
DEV: Remove line of code that does not work (#30258)
We can't delete the file from disk as some of the assets are still
served by the app instead of going through the S3 bucket. It is a bug we
need to fix but it also means this ENV is unsafe now. Just drop the env
until we ensure all assets requested by the app are requested from the
S3 bucket directly.
2024-12-13 09:36:51 +08:00
Kris 9694dc6cb0
DEV: prioritize new email styles over existing, to make customization easier (#30244) 2024-12-12 11:42:50 -05:00
Loïc Guitaut a589b48f9a DEV: Display better output when inspecting service steps
This patch aims to improve the steps inspector output:
- The service class name is displayed at the top.
- Next to each step is displayed the time it took to run said step.
- Steps that didn’t run are hidden.
- `#inspect` automatically outputs the error when it is present.
2024-12-12 15:21:10 +01:00
Régis Hanol 44cabc3569
FIX: proper details / summary excerpt (#30229)
It doesn't make much sense to have the content of a `<details>` in an excerpt so I replaced them with "▶ summary" instead.

That way, they can't be (ab)used in user cards for example.

Reference - https://meta.discourse.org/t/335094
2024-12-12 09:09:49 +01:00
Bianca Nenciu a835fd99bd FIX: Truncate bookmarks.name when remapping
The new name may be too long for the bookmarks.name column and raise an
exception. This changes allows the remapper to truncate the new value to
fit (truncates to 100 characters).
2024-12-11 18:53:17 -05:00
Alan Guo Xiang Tan c97d1d7c59
DEV: Remove max compression level for brotli in assets.rake (#30220)
The `max_compress?` logic is totally broken at least when used for
brotli compression because we are only seeing 4 assets subjected to the
max compression level in production. Instead of fixing the broken logic,
we should just drop this unnecessary complexity cause things are easier
to reason about when we only have one compression level to deal with
across all assets.
2024-12-11 14:01:33 +08:00
Alan Guo Xiang Tan 19321a0b86
DEV: Fix `s3:upload_assets` not logging newlines (#30219)
Follow-up to 9a2e31b9af
2024-12-11 12:59:17 +08:00
Alan Guo Xiang Tan 9a2e31b9af
DEV: Use a `Logger` for `s3:upload_assets` (#30218)
Now that we run the `upload` method in different threads, we need to
synchronize writes to `STDOUT` which we can do so by using a `Logger`.

Follow-up to 49e8353959
2024-12-11 11:48:06 +08:00
Alan Guo Xiang Tan 49e8353959
FIX: `s3:upload_assets` was uploaded some source maps twice (#30216)
This is because Sprocket's manifest already contains the source maps.
The easy and safe fix here is to just use a `Set` to prevent
duplications.
2024-12-11 11:19:38 +08:00
Bianca Nenciu b9f8a77d9b
DEV: Upload assets to S3 in parallel (#30210)
In my local setup (with Minio), this uploads the assets to S3 ~40% faster.
2024-12-11 10:51:05 +08:00
Alan Guo Xiang Tan 864b7b6bc8
DEV: Fix flaky test (#30215)
The test was flaky and failing with the following errors:

```
Failure/Error:
  klass
    .connection
    .select_raw(relation.arel) do |result, _|
      result.type_map = DB.type_map
      result.nfields == 1 ? result.column_values(0) : result.values
    end

NoMethodError:
  undefined method `select_raw' for nil

./lib/freedom_patches/fast_pluck.rb:60:in `pluck'
./vendor/bundle/ruby/3.3.0/gems/activerecord-7.2.2.1/lib/active_record/relation/calculations.rb:354:in `pick'
./app/models/web_crawler_request.rb:27:in `request_id'
./app/models/web_crawler_request.rb:31:in `rescue in request_id'
./app/models/web_crawler_request.rb:26:in `request_id'
./app/models/web_crawler_request.rb:19:in `write_cache!'
./app/models/concerns/cached_counting.rb:135:in `block (3 levels) in flush_to_db'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
./app/models/concerns/cached_counting.rb:134:in `block (2 levels) in flush_to_db'
./app/models/concerns/cached_counting.rb:124:in `each'
./app/models/concerns/cached_counting.rb:124:in `block in flush_to_db'
./lib/distributed_mutex.rb:53:in `block in synchronize'
./lib/distributed_mutex.rb:49:in `synchronize'
./lib/distributed_mutex.rb:49:in `synchronize'
./lib/distributed_mutex.rb:34:in `synchronize'
./app/models/concerns/cached_counting.rb:120:in `flush_to_db'
./app/models/concerns/cached_counting.rb:187:in `perform_increment!'
./app/models/web_crawler_request.rb:15:in `increment!'
./lib/middleware/request_tracker.rb:74:in `log_request'
./lib/middleware/request_tracker.rb:409:in `block in log_later'
./lib/scheduler/defer.rb:125:in `block in do_work'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management/null_instance.rb:49:in `with_connection'
./vendor/bundle/ruby/3.3.0/gems/rails_multisite-6.1.0/lib/rails_multisite/connection_management.rb:21:in `with_connection'
./lib/scheduler/defer.rb:119:in `do_work'
./lib/scheduler/defer.rb:105:in `block (2 levels) in start_thread'
```

This was due to running the defer thread in an async manner which is
actually no representative of the production environment. It also
revealed a spot in our code base where writes are happening in a GET
request which can cause requests to fail if ActiveRecord is in readonly
mode.
2024-12-11 10:12:58 +08:00
Alan Guo Xiang Tan eeb01ea0de
DEV: Remove unnecessary thread in `Jobs::Base::JobInstrumenter` take 2 (#30195)
This reverts commit 766ff723f8.

Ensure that we create the sidekiq log file first before opening it for
logging. This avoids any issue of the log file not being present when we
initialize an instance of the `Logger`.
2024-12-10 12:44:56 +08:00
Michael Brown c546111703 DEV: add the notion of a 'crawler identifier' in anonymous_cache
We identify and deny blocked crawlers here in anonymous_cache.

Separating the notion of the crawler identifier here lets plugins perform an
override if they perform more advanced detection.
2024-12-09 13:40:22 -05:00
Osama Sayegh 976aca68f6
FEATURE: Restrict profile visibility of low-trust users (#29981)
We've seen in some communities abuse of user profile where bios and other fields are used in malicious ways, such as malware distribution. A common pattern between all the abuse cases we've seen is that the malicious actors tend to have 0 posts and have a low trust level.

To eliminate this abuse vector, or at least make it much less effective, we're making the following changes to user profiles:

1. Anonymous, TL0 and TL1 users cannot see any user profiles for users with 0 posts except for staff users
2. Anonymous and TL0 users can only see profiles of TL1 users and above

Users can always see their own profile, and they can still hide their profiles via the "Hide my public profile" preference. Staff can always see any user's profile.

Internal topic: t/142853.
2024-12-09 13:07:59 +03:00
Alan Guo Xiang Tan 25ce1f3399
PERF: Don't execute a `git` command each time we log a log line (#30177)
We already have a `GIT_VERSION` constant in `DiscourseLogstashLogger` so
we can just use that.
2024-12-09 11:11:03 +08:00
Martin Brennan 8a89a77248
FIX: Discard empty bundles for reviewables (#30121)
Followup c7e471d35a

It is currently possible to add a bundle (which is a collection
of actions used for a dropdown on the client) for a reviewable
via actions.add_bundle and then never add any actions to it.

This causes the client to explode, as seen in the referenced
commit, because of the way our store expects to resolve objects
referenced by ID that are passed down by the serializer, which
then causes Ember to have an unrecoverable render error.

Fixing this on the serializer level is not really possible because
of all the ActiveModel::Serializer magic that serializes
objects by ID reference when doing things like has_many.
`Reviewable#actions_for` is a better place to do this anyway,
because this is the main location where the bundles and actions
are built for every action via the serializer.
2024-12-05 15:41:13 +10:00
Krzysztof Kotlarek 28b4ff6cb6
FIX: update flag reason message with default value (#30026)
Currently only system flags are translated. When we send message to the user that their post was deleted because of custom flag, we should default to custom flag name.
2024-12-04 14:46:52 +11:00
Kris 60826162b5
A11Y: remove redundant alt text from github oneboxes (#30083) 2024-12-04 12:25:03 +11:00
Gary Pendergast 2513339955
FEATURE: Show when a badge has been granted for a post (#29696)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2024-12-03 13:43:27 +11:00
Kelv 435fbb7408
DEV: unsilence deprecation warning for old Font Awesome icons (#30028)
* DEV: unsilence deprecation warnings for old Font Awesome icon names

* update fa-user to user font awesome icon name
* update pencil-alt to pencil font awesome 6 icon name
2024-12-03 10:28:39 +08:00
David Taylor b47ae6d437
UX: Strip multiline comments in github oneboxes (#30040)
We were already stripping comments from GitHub issue/PR oneboxes, but the regex was not correctly matching multiline comments.
2024-12-02 18:08:55 +00:00
Alan Guo Xiang Tan 44a81069ac
DEV: Avoid creating system message when system user initiates restore (#30027)
There is no point creating a message for the system user since it is a
non-human user.
2024-12-02 16:13:38 +08:00
Régis Hanol 7d58793759
DEV: deduplicate inline styles in emails (#30015)
In order to limit issues with duplicate inline CSS definitions, this will now deduplicate inline CSS styles with the "last-to-be-defined-wins" strategy.

Also removes unecessary whitespaces in inline styles.

Context - https://meta.discourse.org/t/resolve-final-styles-in-email-notifications/310219

Co-authored-by: Thomas Kalka <thomas.kalka@gmail.com>
2024-11-30 16:38:45 +01:00
Régis Hanol 20d46c9583
PERF: only diff HTML / Markdown when needed (#30014)
When serializing the `body_changes` in the `PostRevisionSerializer`, we create two diffs: one for the `cooked` and another one for the `raw` version of the post.

Inside `DiscourseDiff`, we generate both `html` and `markdown` diffs when we only need the `html` diffs for the `cooked` version of the post and the `markdown` diff for the `raw` version of the post.

This solves the issue repored in https://meta.discourse.org/t/server-error-accessing-topic-revisions-on-a-specific-topic/339185 where some revisions would return 500 because of a `ArgumentError : Attributes per element limit exceeded` exception when trying to generate the `html` diff on a very large `raw`.
2024-11-30 16:30:30 +01:00
Jarek Radosz 85ead5ac7a
Revert "FIX: deduplicate css in mails (#30003)" (#30013)
This reverts commit 6e726d436f.

The specs were failing in the original PR but the CI didn't run.
2024-11-30 15:32:32 +01:00
Thomas Kalka 6e726d436f
FIX: deduplicate css in mails (#30003)
Feature: Resolve final styles in email notifications

Context - https://meta.discourse.org/t/resolve-final-styles-in-email-notifications/310219
2024-11-30 14:51:02 +01:00
Hoa Nguyen 607dd2cbd8
DEV: improve the plugin:spec rake task (#29050)
Allow the plugin:spec to receive the test file path, rather than always run all tests of the plugin.
2024-11-29 06:33:14 +11:00
Bianca Nenciu 5abee8ac6b
DEV: Log number of live slots used by requests (#29884) 2024-11-28 18:25:48 +02:00
Loïc Guitaut 88f1b3b195 DEV: Try fixing flaky spec related to Scheduler::Defer
Checking if a connection is available is probably not enough, when the
connection is available, we should still verify it’s not stale.
2024-11-28 15:30:13 +01:00
Loïc Guitaut f69f0211df DEV: Fix flaky spec related to Scheduler::Defer
In some cases in CI env, it seems the AR connection isn’t available and
the `ensure` block is executed. It’s calling `#verify!` on the
connection, so it can fail sometimes. This is probably why
`#clear_active_connections!` was failing too sometimes.

Here, we just check the connection is present before clearing the
connections.
2024-11-28 11:46:52 +01:00
Sam 07813ba83c
DEV: fix hanging spec (#29974) 2024-11-28 11:06:19 +08:00
Sam 72132c35fb
DEV: fix flaky spec (#29972)
Spec was flaky cause work could still be in pipeline after the defer
length is 0. Our length denotes the backlog, not the in progress
count.

This adds a mechanism for gracefully stopping the queue and avoids
wait_for callse
2024-11-28 11:21:35 +11:00
Angus McLeod 6acf673f8d
FIX: topic post counts for webhook post_destroyed event (#29853)
* FIX: topic post counts for webhook post_destroyed event

- Generate webhook data after posts are destroyed
- Don't count user_deleted posts

* Remove unnecessary conditional
2024-11-27 11:36:51 -08:00
Loïc Guitaut fac6147039 DEV: Verify DB connection before trying to clear active connections 2024-11-27 18:12:11 +01:00
Loïc Guitaut d6bec460a8 DEV: Upgrade Rails to version 7.2 2024-11-27 10:48:47 +01:00
Ted Johansson f4d0a77d5f
DEV: Add "delete user" options to illegal flag review (#29956)
We already add the "delete user" and "delete and block user" options to the drop-down for potential spam, but we should do this for potentially illegal posts as well.

This is entirely based on the implementation for the potential spam one, including caching the status on the reviewable record.

Also note that just as for potential spam, the user must be "deletable" for the option to appear.

I also took the liberty to move the options in the drop-down to what I think is a more intuitive place. (Between delete post and suspend/silence user.)
2024-11-27 17:23:57 +08:00
Martin Brennan 2ef9d6ac47
FEATURE: Allow admins to force refresh "What's new?" (#29911)
Sometimes changes to "What's new?" feed items are made or the feed items are
removed altogether, and the polling interval to check for new features is 1 day.

This is quite long, so this commit introduces a "Check for updates"
button for admins to click on the "What's new?" page which will bust
the cache for the feed and check again at the new features endpoint.
This is limited to 5 times per minute to avoid rapid sending of
requests.
2024-11-27 09:40:55 +10:00
Martin Brennan b8a5f95eb6
FIX: Handle multiple In-Reply-To Message-ID in group inbox (#29912)
This fix handles the case where an In-Reply-To mail header
can contain multiple Message-IDs. We use this header to
try look up an EmailLog record to find the post to reply
to in the group email inbox flow.

Since the case where multiple In-Reply-To Message-IDs is
rare (we've only seen a couple of instances of this causing
errors in the wild), we are just going to use the first one
in the array.

Also, Discourse does not support replying to multiple posts
at once, so it doesn't really make sense to use multiple
In-Reply-To Message-IDs anyway.
2024-11-26 11:12:40 +10:00
Selase Krakani a20b7fa83f
DEV: Gracefully handle `regex_replace` max column length violations (#29787)
* DEV: Gracefully handle `regex_replace` violations of column length constraints

This is a follow-up to the `remap` [refactor](9b0cfa99c5).
Similar to `remap`, the entire `regex_replace` operation fails if the new content exceeds the column’s max length.

This change introduces an optional mode, controlled by the new `skip_max_length_violations` param
to skip records eligible for `regex_replace`  where the new content violates the max column length constraint.

It also includes updates to the exception message raised when `regex_replace` fails to include more details

* DEV: Remove string escapes in heredoc text
2024-11-25 11:39:53 +00:00
Osama Sayegh eaa3f813c1
FIX: Don't secure the about banner image (#29889)
Uploads that are linked to site settings shouldn't be flagged as secure in login-required sites that enable secure uploads. However, in order for site setting uploads to not be marked secured, the frontend uploader has to include 2 params in the upload request: `for_site_setting: true` and `type: "site_setting"`.

Since these 2 params are semantically identical, we want the `type: "site_setting"` param alone to make the upload correctly treated as a site setting upload. To achieve that, we need to include the `site_setting` type in the public types list because the `for_site_setting` param has the same effect — it marks the upload as a public type.

b138eaf9e5/lib/upload_security.rb (L128-L131)
2024-11-25 11:12:00 +03:00
Ted Johansson 88af23e1ca
DEV: Modernize admin user fields (#29843)
This PR modernizes the user fields area of the admin UI. It is largely based on the work on the emoji section.
2024-11-25 11:54:43 +08:00
Mark VanLandingham d880db3b7b
DEV: Apply modifier for topic_view link_counts (#29883) 2024-11-22 14:49:39 -06:00
Jarek Radosz 2589545623
DEV: Detect hbr topic list customizations (#29793) 2024-11-21 16:00:49 +01:00
Loïc Guitaut 581fb97bfa DEV: Fix benchmark script
Following a recent commit (cb4b8146a3),
the benchmark script wasn’t working anymore (and the related rake task).

This patch fixes it. It also adds some information about Ruby YJIT being
enabled or not.
2024-11-20 14:36:44 +01:00
David Taylor 32665cf9dd
DEV: Consolidate i18n import paths (#29804)
Enables our new eslint rules which enforce consistent i18n imports. For more info, see 0d58b40cd7
2024-11-19 20:45:18 +00:00
Loïc Guitaut 719457e430 DEV: Add a `try` step to services
This patch adds a new step to services named `try`.

It’s useful to rescue exceptions that some steps could raise. That way,
if an exception is caught, the service will stop its execution and can
be inspected like with any other steps.

Just wrap the steps that can raise with a `try` block:
```ruby
try do
  step :step_that_can_raise
  step :another_step_that_can_raise
end
```
By default, `try` will catch any exception inheriting from
`StandardError`, but we can specify what exceptions to catch:
```ruby
try(ArgumentError, RuntimeError) do
  step :will_raise
end
```

An outcome matcher has been added: `on_exceptions`. By default it will
be executed for any exception caught by the `try` step.
Here also, we can specify what exceptions to catch:
```ruby
on_exceptions(ArgumentError, RuntimeError) do |exception|
  …
end
```

Finally, an RSpec matcher has been added:
```ruby
  it { is_expected.to fail_with_exception }
  # or
  it { is_expected.to fail_with_exception(ArgumentError) }
```
2024-11-19 12:01:07 +01:00
Selase Krakani 9b0cfa99c5
DEV: Gracefully handle remaps which violate DB column constraints (#29501)
* DEV: Gracefully handle remaps which violate DB column constraints

This change implements length constraint enforcement to skip remaps
which exceed column max lengths

* DEV: Only perform skipped column stats lookup when verbose is true

* DEV: Tidy up specs

* DEV: Make skipping violating remap behaviour opt-in

This change introduces a new `skip_max_length_violations` param for
`remap`, set to `false` by default to ensure we still continue to fail
hard when max lenth constraints are violated.

To aid in quick resolution when remaps fail, this change also
adds more context to the exception message to include the offending table
and column information

* Apply suggestions from code review

Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>

* FIX: Various fixes

- Linter errors
- Remap status "logger" early return condition

---------

Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2024-11-15 10:42:25 +00:00
Alan Guo Xiang Tan 6bf0ac730f
FIX: Rescue `ActiveRecord::ReadOnlyError` when baking theme field (#29776)
Firstly, we need to understand that ActiveRecord can be
connected to a role which prevent writes and this happens in Discourse when a
replica database has been setup for failover purposes. When a role
prevent writes from happening, ActiveRecord will raise the
`ActiveRecord::ReadOnlyError` if a write query is attempted.

Secondly, theme fields are baked at runtime within GET requests. The
baking process involves writing the baked value to the
`ThemeField#baked_value` column in the database.

If we combine the two points above, we can see how the writing of the
baked value to the database will trigger a `ActiveRecord::ReadOnlyError`
in a GET requests when the database is connected to a role preventing
writes. However, failing to bake a theme is not the end of the world and
should not cause GET requests to fail. Therefore, this commit adds a rescue
for `ActiveRecord::ReadOnlyError` in the `ThemeField#ensure_baked!`
method.
2024-11-15 10:19:10 +08:00
Alan Guo Xiang Tan 69b552a211
DEV: Fix `uploads:fix_missing_s3` rake task when file is too big (#29735)
If the upload has existed before, we should allow the upload to be
created even if the upload's size is too big.
2024-11-13 15:01:44 +08:00
Jan Cernik 234133bd3b
UX: Split `hide_profile_and_presence` user option (#29632)
It splits the hide_profile_and_presence user option and the default_hide_profile_and_presence site setting for more granular control. It keeps the option to hide the profile under /u/username/preferences/interface and adds the presence toggle in the quick user menu.

Co-authored-by: Régis Hanol <regis@hanol.fr>
2024-11-12 22:22:58 -03:00
Angus McLeod cb4b8146a3
Add dedicated user_api_key_clients table to allow for 1:many use cases (#28119) 2024-11-08 12:05:03 -05:00
Alan Guo Xiang Tan af642d0d69
Revert "FEATURE: Mark bad uploads with :invalid_url (#29640)" (#29657)
This reverts commit 5a00a041f1.

Implementation is currently not correct. Multiple uploads can share the
same etag but have different paths in the S3 bucket.
2024-11-08 13:04:52 +08:00
Natalie Tay 6c36af9f62
DEV: Add verbose logging for google oauth (#29459) 2024-11-08 11:03:17 +08:00
Bianca Nenciu 5a00a041f1
FEATURE: Mark bad uploads with :invalid_url (#29640)
A "bad upload" in this context is a upload with a mismatched URL. This can happen when changing the S3 bucket used for uploads and the upload records in the database have not been remapped correctly.
2024-11-08 08:05:14 +08:00
Martin Brennan 0568d36133
FIX: Use dualstack S3 endpoint for direct uploads (#29611)
When we added direct S3 uploads to Discourse, which use
presigned URLs, we never took into account the dualstack
endpoints for IPv6 on S3.

This commit fixes the issue by using the dualstack endpoints
for presigned URLs and requests, which are used in the
get-presigned-put and batch-presign-urls endpoints used when
directly uploading to S3.

It also makes regular S3 requests for `put` and so on use
dualstack URLs. It doesn't seem like there is a downside to
doing this, but a bunch of specs needed to be updated to reflect this.
2024-11-07 11:06:39 +10:00
Keegan George fad0d2ec62
UX: New experimental features should be marked clearly (#29622)
This PR adds a small visual change to the new feature item on the `/admin/whats-new` page. When features are marked with an experimental site setting, they should show an indication on the feature item that it is "Experimental"
2024-11-06 15:26:30 -08:00
Kris f24c274867
UX: replace broken link SVG with new Font Awesome 6 name (#29623) 2024-11-06 13:35:07 -05:00
Gabriel Grubba 75beb5a84f
FIX: Change create_post_for_category_and_tag_changes setting to use whispers instead of small actions (#29602)
It currently can leak private tags/categories, to address this we are moving to whispers.
2024-11-06 09:28:28 -03:00
Alan Guo Xiang Tan 57f4176b57
DEV: Bump rubocop_discourse (#29608) 2024-11-06 06:27:49 +08:00
Sam 3b0332ef6c
DEV: use Sets for setting lists. (#29584)
This makes membership checks more efficient
2024-11-05 16:04:24 +11:00
Martin Brennan 1a3b9a7352
DEV: Secure upload rake task improvements (#29484)
This commit changes the uploads:secure_upload_analyse_and_update
and uploads:disable_secure_uploads to no longer rebake affected
posts inline. This just took way too long, and if the task stalled
you couldn't be sure if the rest of it completed.

Instead, we can update the baked_version of affected posts and
utilize our PeriodicalUpdates job to gradually rebake them. I added
warnings about increasing the site setting rebake_old_posts_count and
the global setting max_old_rebakes_per_15_minutes before doing this
as well.

For good measure, the affected post IDs are written to a JSON file too.
2024-10-31 13:33:11 +10:00
Martin Brennan d5b328b193
DEV: Site setting keyword changes (#29486)
This commit contains two changes to how our site setting
keyword system works:

1. Crowdin, our translation provider, does not support YAML lists,
   so we are changing site setting keywords in server.en.yml to
   be pipe-separated (|)
2. It's unclear to translators what they are supposed to do with
   aliases of site settings where the name has changed, e.g.
   min_trust_level_for_here_mention. Instead of getting these as
   keywords from the yml file, we can discern these from
   SiteSettings::DeprecatedSettings automatically, and still use
   them for client-side search

These changes should help improve the situation for translators.
2024-10-31 13:18:34 +10:00
Bianca Nenciu 430c42acde
FEATURE: Conditionally force optimized category style (#29473)
The `categories_only_optimized` category page style has been introduced
in commit d37a0d401c. This commit makes
sure that style is enforced for users who can see over 1000 categories
in order to keep `/categories` page functional.
2024-10-30 16:46:56 +02:00
Osama Sayegh 2ffe413b0b
FEATURE: Enable the new /about page for everyone (#29390)
This commit removes the feature flag for the new /about page, enabling it for all sites, and removes the code for old the /about page.

Internal topic: t/140413.
2024-10-29 18:40:11 +03:00
Loïc Guitaut 2f334964f2 DEV: Remove hash-like access from service contracts
We decided to keep only one way to access values from a contract. This
patch thus removes the hash-like access from contracts.
2024-10-29 16:02:51 +01:00
Loïc Guitaut c78211cf8d DEV: Make service contracts immutable
We decided to make contracts immutable once their validations have run.
Indeed, it doesn’t make a lot of sense to modify a contract value
outside the contract itself.

If processing is needed, then it should happen inside the contract
itself.
2024-10-29 12:23:35 +01:00
Krzysztof Kotlarek 0839bce7b6
DEV: allow the plugin to register valid site setting areas (#29432)
In this PR, we defined the ability to group site settings by area - https://github.com/discourse/discourse/pull/28570

Plugins should be able to register in their own areas.
2024-10-29 09:40:31 +11:00
Martin Brennan 27bdfb6437
FEATURE: Add user preference to disable smart lists (#29434)
Followup 30fdd7738e

Adds a new site setting and corresponding user preference
to disable smart lists. By default they are enabled, because
this is a better experience for most users. A small number of
users would prefer to not have this enabled.

Smart lists automatically append new items to each
list started in the composer when enter is pressed. If
enter is pressed on an empty list item, it is cleared.

This setting will be removed when the new composer is complete.
2024-10-28 14:18:15 +10:00
Loïc Guitaut 584424594e DEV: Replace `params` by the contract object in services
This patch replaces the parameters provided to a service through
`params` by the contract object.

That way, it allows better consistency when accessing input params. For
example, if you have a service without a contract, to access a
parameter, you need to use `params[:my_parameter]`. But with a contract,
you do this through `contract.my_parameter`. Now, with this patch,
you’ll be able to access it through `params.my_parameter` or
`params[:my_parameter]`.

Some methods have been added to the contract object to better mimic a
Hash. That way, when accessing/using `params`, you don’t have to think
too much about it:
- `params.my_key` is also accessible through `params[:my_key]`.
- `params.my_key = value` can also be done through `params[:my_key] =
  value`.
- `#slice` and `#merge` are available.
- `#to_hash` has been implemented, so the contract object will be
  automatically cast as a hash by Ruby depending on the context. For
  example, with an AR model, you can do this: `user.update(**params)`.
2024-10-25 14:48:34 +02:00
Loïc Guitaut 41584ab40c DEV: Provide user input to services using `params` key
Currently in services, we don’t make a distinction between input
parameters, options and dependencies.

This can lead to user input modifying the service behavior, whereas it
was not the developer intention.

This patch addresses the issue by changing how data is provided to
services:
- `params` is now used to hold all data coming from outside (typically
  user input from a controller) and a contract will take its values from
  `params`.
- `options` is a new key to provide options to a service. This typically
  allows changing a service behavior at runtime. It is, of course,
  totally optional.
- `dependencies` is actually anything else provided to the service (like
  `guardian`) and available directly from the context object.

The `service_params` helper in controllers has been updated to reflect
those changes, so most of the existing services didn’t need specific
changes.

The options block has the same DSL as contracts, as it’s also based on
`ActiveModel`. There aren’t any validations, though. Here’s an example:
```ruby
options do
  attribute :allow_changing_hidden, :boolean, default: false
end
```
And here’s an example of how to call a service with the new keys:
```ruby
MyService.call(params: { key1: value1, … }, options: { my_option: true }, guardian:, …)
```
2024-10-25 09:57:59 +02:00
Gabriel Grubba 948f5c0140
DEV: add rake task to annotate plugins in clean DB (#29378) 2024-10-24 15:15:59 -03:00
overgrow 95d8ed59db
PERF: Faster disk space calculation on upload heavy instances (#28307) 2024-10-24 12:11:40 +02:00
Krzysztof Kotlarek cd077ef93b
FIX: visual regression for new features (#29359)
Bug introduced in this PR https://github.com/discourse/discourse/pull/29244

When the experiment toggle button was introduced, new features did not look right when the toggle button was not available.

In addition, the plugin name can be an empty string. In that case, information about new features should be displayed.
2024-10-23 16:16:19 +11:00
Loïc Guitaut f79dd5c8b5 DEV: Stop injecting a service result object in the caller object
Currently, when calling a service with its block form, a `#result`
method is automatically created on the caller object. Even if it never
clashed so far, this could happen.

This patch removes that method, and instead use a more classical way of
doing things: the result object is now provided as an argument to the
main block. This means if we need to access the result object in an
outcome block, it will be done like this from now on:
```ruby
MyService.call(params) do |result|
  on_success do
    # do something with the result object
    do_something(result)
  end
end
```

In the same vein, this patch introduces the ability to match keys from
the result object in the outcome blocks, like we already do with step
definitions in a service. For example:
```ruby
on_success do |model:, contract:|
  do_something(model, contract)
end
```
Instead of
```ruby
on_success do
  do_something(result.model, result.contract)
end
```
2024-10-22 16:58:54 +02:00
Gerhard Schlager 07ff21d045
FIX: Restoring backup could fail due to missing `discourse_functions` (#29332)
Database dumps sometimes reference functions in the `discourse_functions` schema. It's possible that some of these functions have been dropped in a newer version of Discourse. In that case, restoring an older backup will fail with a `ERROR:  function discourse_functions.something_something() does not exist` error. The restore functionality contains a workaround for that problem, but it didn't work with functions created in plugin migrations.

This commit adds support for temporarily creating missing `discourse_functions` from plugins. And it adds a simple check if the DB migration file even contains the required `DROPPED_TABLES` or `DROPPED_COLUMNS` constant. We don't need to create an instance of the DB migration class unless one of those constants is used. This makes the restore slightly faster and works around a problem with migrations that execute without `up` or `down` methods (e.g. `BackfillChatChannelAndThreadLastMessageIdsPostMigrate`).
2024-10-22 16:13:01 +02:00
Krzysztof Kotlarek 433fadbd52
FEATURE: allow admins to enable announced experimental features (#29244)
Toggle the button to enable the experimental site setting from "What's new" announcement.

The toggle button is displayed when:
- site setting exists and is boolean;
- potentially required plugin is enabled.
2024-10-22 10:56:58 +11:00
Régis Hanol 88449541a5
FIX: participating users statistics... (#29293)
* FIX: participating users statistics...

... was (mis-)counting

- bots
- anonymous users
- suspended users

There's now a "valid_users" function that holds the AR query for valid users and which is used in all "users", "active_users", and "participating_users" queries.

Internal ref - t/138435
2024-10-21 18:18:42 +02:00
David Taylor b3b0695bb1
PERF: Optimize themes:update task (#29290)
- Add concurrency when running on multisite clusters (default 10, configurable via THEME_UPDATE_CONCURRENCY env)

- Add a version cache for the duration of the rake task. This avoids duplicating work when many sites in the cluster have the same theme installed, and it is already up-to-date

- Updates output to be more concurrent friendly (all `puts`, no `print`)
2024-10-21 12:36:40 +01:00
Régis Hanol 97ba39e60f
FIX: bump the number of svg icons we return to first 500 (#29286)
instead of the first 200 which would "hide" some icons from the list when picking an icon for a badge or a sidebar link.

Internal ref - t/119652
2024-10-18 19:22:13 +02:00
Gabriel Grubba fe4e2a17cb
FEATURE: Create rake for db migration in plugins (#29163)
* FEATURE: Create rake for db migration in plugins

before the dev-xp was clunky, we had to create a migration file in core and
move it to the plugin.

Now we automated this process, we still create the migration file in core
but the rake task will move it to the plugin.

the usage is:

```
rake plugin:generate_migration[plugin_name,migration_name,migration_args]

rake plugin:generate_migration[discourse-automation,add_group_id_to_automation_rule,"group_id:integer"]
```

* DEV: change rake to be a generator for plugin migrations

* DEV: trying to add extra class option to migration generator

* DEV: revert to have only `plugin_migration_generator`

* DEV: remove rake task for plugin migration creation

* DEV: remove migration_generator.rb

* DEV: remove if because options with `plugin_name` will always be true
2024-10-18 11:43:45 -03:00
David Battersby 48308a5ee6
FIX: show lightbox for small images (#29140)
We want to allow lightboxing of smaller images, even if they are below the minimum size for image thumbnail generation.

This change sets a minimum threshold of 100 x 100 pixels for triggering the lightbox.

---------

Co-authored-by: Régis Hanol <regis@hanol.fr>
2024-10-18 09:45:08 +04:00
Loïc Guitaut 23c486799f DEV: Improve `array` type in service contracts
This patch improves the custom `array` type available in contracts.
It’s now able to split strings on `|` on top of `,`, and to be more
consistent, it also tries to cast the resulting items to integers.
2024-10-17 17:02:02 +02:00
Sam c08c40dc23
FEATURE: theme_modifiers can depend on theme settings (plus serialize_post_user_badges) (#29227)
Theme modifiers can now be defined as theme settings, this allows for
site operators to override behavior of theme modifiers.

New syntax is:

```
{
    ...
   "modifiers": {
      "modifier_name": {
         "type": "setting",
         "value": "setting_name"
      }
   }
}
```

This also introduces a new theme modifier for serialize_post_user_badges. Name of badge must match the name of the badge in the badges table. The client-side is updated to load this new data from the post-stream serializer.

Co-authored-by: David Taylor <david@taylorhq.com>
2024-10-17 15:16:16 +01:00
Juan David Martínez Cubillos 789aa2d9de
DEV: Add PluginRegistry modifiers to #review and #recalculate (#29128)
* DEV: Add PluginRegistry modifiers to #review and #recalculate

* added tests

* changed added registry logic
2024-10-16 10:26:10 -05:00
David Taylor 7dc60d0c99
DEV: Remove discourse-encrypt from core CI suite (#29230)
This plugin is prone to causing flaky test runs, and is in the process of being deprecated
2024-10-16 11:47:02 +01:00
Alan Guo Xiang Tan 322a3be2db
DEV: Remove logical OR assignment of constants (#29201)
Constants should always be only assigned once. The logical OR assignment
of a constant is a relic of the past before we used zeitwerk for
autoloading and had bugs where a file could be loaded twice resulting in
constant redefinition warnings.
2024-10-16 10:09:07 +08:00
Penar Musaraj 80ac3275ba
DEV: update rake task to disable 2FA for a user (#29052)
- limits security key deletes to second factor keys
- also deletes backup codes (lingering backup codes break login flow entirely)

* Add spec for rake task to disable 2FA for a user
2024-10-16 09:11:29 +11:00
Loïc Guitaut 281570226b DEV: Output failing MF keys when compilation fails
Currently, when the MessageFormat compiler fails on some translations,
we just have the raw output from the compiler in the logs and that’s not
always very helpful.

Now, when there is an error, we iterate over the translation keys and
try to compile them one by one. When we detect one that is failing, it’s
added to a list that is now outputted in the logs. That way, it’s easier
to know which keys are not properly translated, and the problems can be
addressed quicker.

---

The previous implementation of this patch had a bug: it wasn’t handling
locales with country/region code properly. So instead of iterating over
the problematic keys, it was raising an error.
2024-10-15 12:45:23 +02:00
Jeff Wong d92d65fe73
Revert "DEV: Output failing MF keys when compilation fails" (#29200)
This reverts commit 8eff922ace.
2024-10-14 17:42:36 -07:00
Juan David Martínez Cubillos d3f09f8f61
DEV: Add discourse-hcaptcha plugin to the metadata list of official plugins (#29177) 2024-10-11 15:35:12 -05:00
Martin Brennan 2193667e1f
FIX: Plugin JS failing to load would break admin interface (#29139)
If a plugin's JS fails to load for some reason, most commonly
ad blockers, the entire admin interface would break. This is because
we are adding links to the admin routes for plugins that define
them in the sidebar.

We have a fix for this already in the plugin list which shows a warning
to the admin. This fix just prevents the broken link from rendering
in the sidebar if the route is not valid.
2024-10-11 09:26:10 +10:00