Commit Graph

10265 Commits

Author SHA1 Message Date
Bianca Nenciu 5d7fdfb0a6
DEV: Use recorded runtime log for plugin specs (#28711)
The recorded runtime log was specified when running the plugin specs,
but it failed to match the entries from the runtime log and the passed
as command line arguments because the runtime log entries were not
prefixed with `./`.
2024-09-03 20:36:46 +03:00
Joffrey JAFFEUX e418f7056f
FIX: prevents PM to large groups (#28681)
This commit introduces a new hidden site setting: `group_pm_user_limit`, default to `1000` which will raise an error when attempting to create a PM target a large group.
2024-09-03 12:08:14 +02:00
David Taylor 80b9c280ba
DEV: Switch to pnpm for JS dependencies (#28671)
This will bring significant improvements to install speed & storage requirements. For information on how it may affect you, see https://meta.discourse.org/t/324521

This commit:
- removes the `yarn.lock` and replaces with `pnpm-lock.yaml`
- updates workspaces to pnpm format
- adjusts package dependencies to work with pnpm's stricter resolution strategy
- updates Rails app to load modules from more specific node_modules directories
- adds a `.pnpmfile` which automatically cleans up old yarn-managed `node_modules` directories
- updates various scripts to call `pnpm` instead of `yarn`
- updates patches to use pnpm's native patch system instead of patch-package
- adds a patch for licensee to support pnpm
2024-09-03 10:51:07 +01:00
Loïc Guitaut 9b4b5b5028 FIX: Return proper results when searching for a topic in Japanese
Currently, when the default locale is Japanese, the search for a topic
using its URL, path or ID doesn’t work as expected. It will either
return wrong results or no result at all.

The problem lies with how we process the provided terms in Japanese
mode. For example, if `http://localhost/t/-/55` is provided, currently
this will result in `http localhost t 5 5` to be searched for.

This patch addresses the issue by checking whether the provided term
needs segmenting. If the provided term is a number, or a path or a full
URL, then it doesn’t need segmenting. When that happens we skip the
processing we normally apply for Japanese, making the search return the
expected results.
2024-09-03 09:48:58 +02:00
Daniel Waterworth baf41790dd
PERF: Don't do initialization for every DB if RAILS_DB is set (#28668) 2024-09-03 15:56:46 +10:00
Krzysztof Kotlarek 7577231ba2
DEV: the ability to define setting areas (#28570)
A new setting attribute is used to define the areas (separated by `|`).

In addition, endpoint `/admin/config/site_settings.json` accepts new `filter_area` data.
2024-09-03 09:25:45 +10:00
Joffrey JAFFEUX 153175d676
DEV: removes unused intergation.rake (#28687)
This is dead code which won't even run.
2024-09-02 21:37:49 +02:00
David Taylor a31dc0a84a
DEV: Raise an exception if trying to set a readonly column with default (#27416) 2024-09-02 10:09:40 +01:00
Joffrey JAFFEUX d81fc1448b
DEV: implements register_modifier(:custom_homepage_enabled) (#28657)
This commit will allow plugin developers to enable/disable the custom homepage.

Usage:

```ruby
register_modifier(:custom_homepage_enabled) do |enabled, args|
  true
end 
```

Args might contain request and/or current_user.
2024-08-30 11:06:07 +02:00
Martin Brennan daa06a1c00
DEV: Improve external upload debugging (#28627)
* Do not delete created external upload stubs for 2 days
  instead of 1 hour if enable_upload_debug_mode is true,
  this aids with server-side debugging.
* If using an API call, return the detailed error message
  if enable_upload_debug_mode is true. In this case the user
  is not using the UI, so a more detailed message is appropriate.
* Add a prefix to log messages in ExternalUploadHelpers, to
  make it easier to find these in logster.
2024-08-30 10:25:04 +10:00
David Taylor 07de0db5e2
DEV: Cap qunit concurrency at 8 browsers (#28635)
`ember exam` points all browsers at a single ember-cli server process. With very high parallelism, we've started seeing `ChunkLoadError` exceptions, which may indicate the server being overloaded. Capping to 8 browsers to avoid that.
2024-08-29 19:00:54 +08:00
Gabriel Grubba c8f5445030
Revert "DEV: Removal of create_post_for_category_and_tag_changes setting (#28…" (#28587)
This reverts commit fc33826dc5.
2024-08-27 16:19:51 -03:00
Penar Musaraj f405c021eb
Bump version to v3.4.0.beta2-dev 2024-08-27 10:54:25 -04:00
Penar Musaraj 85abac4230
Bump version to v3.4.0.beta1 2024-08-27 10:54:25 -04:00
Alan Guo Xiang Tan 4a6fc45429
DEV: Migrate `User#seen_notification_id` to `bigint` (#28572)
`Notification#id` was migrated to `bigint` in 799a45a291
2024-08-27 14:32:55 +03:00
Martin Brennan e58e7a49f5
FIX: Bookmark reminder was clearing incorrectly (#28506)
Followup 76c56c8284

The change introduced above made it so the expired
bookmark reminders were cleared when using the bulk
action menu for bookmarks. However this also affected
clearing reminders for bookmarks when sending notifications.

When clearing bookmark reminders after sending notifications,
we take into account the auto delete preference:

* never          - The bookmark `reminder_at` date should not be cleared,
                   and the bookmark is kept.
* clear_reminder - The bookmark `reminder_at` date is cleared and
                   the bookmark is kept

The `never` option made it so "expired" bookmark reminder show
on the user's bookmark list.

This commit fixes the change from the other commit and only
forces clearing of `reminder_at` if using the bookmark bulk
action service.
2024-08-26 09:17:39 +10:00
Alan Guo Xiang Tan 21bb28df91
PERF: Ensure suggested topics is only loaded on last page of topic view (#28507)
This commit improves `TopicsController#show` to not load suggested and
related topics unless it is the last page of the topic's view.
Previously, we avoided loading suggested and related topics by the use
of conditionals in the `TopicViewSerializer` to avoid calling
`TopicView#suggested_topics` and `TopicView#related_topics`. However,
this pattern is not reliable as the methods can still be called from
other spots in the code base. Instead, we ensure that
`TopicView#include_suggested` and `TopicView#include_related` is set
correctly on the instance of `TopicView` which ensures that for the
given instance, `TopicView#suggested_topics` and
`TopicView#related_topics` will be a noop.
2024-08-23 16:10:50 +08:00
锦心 ac365a86a2
FIX: Trigger user_logged_out event when the user logs out (#28482)
What did this fix?
===============

Previously, we only triggered this event in the `user.logged_out` method.
This resulted in the event being triggered only when the user was logged
out by the administrator or the site had strict logout mode enabled.
This bug affected customers who managed user status via webhooks.

meta topic: https://meta.discourse.org/t/user-log-out-event-not-triggered-in-webhooks/249464
2024-08-22 21:48:27 +08:00
David Taylor 819d07469d
DEV: fixup themes:install:archive flag (#28486)
Followup to a69f3a0880
2024-08-22 13:22:37 +01:00
David Taylor 58c4528a1c
DEV: Replace silent theme-install fails with exceptions (#28485)
Previously, we were silently failing when a theme hit SSRF protection, or the `git clone` command failed for some reason. This commit updates them to be exceptions, so they provide more useful error messages
2024-08-22 12:09:56 +01:00
David Taylor a69f3a0880
DEV: Add env to `themes:install:archive` to allow skipping child themes (#28484) 2024-08-22 12:07:27 +01:00
Osama Sayegh 10ae7ef44a
FEATURE: Add estimated number of global and EU visitors to the about page (#28382)
This commit implements 2 new metrics/stats in the /about page for the _estimated_ numbers of unique visitors from the EU and the rest of the world. This new feature is currently off by default, but it can be enabled by turning on the hidden `display_eu_visitor_stats` site settings via the rails console.

There are a number of assumptions that we're making here in order to estimate the number of unique visitors, specifically:

1. we're assuming that the average of page views per anonymous visitor is similar to the average number of page views that a logged-in visitor makes, and
2. we're assuming that the ratio of logged in visitors from the EU is similar to the ratio of anonymous visitors from the EU

Discourse keeps track of the number of both logged-in and anonymous page views, and also the number of unique logged-in visitors and where they're from. So with those numbers and the assumptions above, we can estimate the number of unique anonymous visitors from the EU and the rest of the world.

Internal topic: t/128480.
2024-08-21 00:03:42 +03:00
Loïc Guitaut 0636855706 DEV: Allow using an AR relation as a model in services
This patch allows using an AR relation as a model in services without
fetching associated records. It will just check if the relation is empty
or not. In the former case, the execution will stop at that point, as
expected.
2024-08-20 16:32:46 +02:00
Osama Sayegh db6eff7be9
DEV: Allow custom site activity items in the new /about page (#28400)
This commit introduces a new frontend API to add custom items to the "Site activity" section in the new /about page. The new API is called `addAboutPageActivity` and it works along side the `register_stat` serve-side API which serializes the data that the frontend API consumes. More details of how the two APIs work together is in the JSDoc comment above the API function definition.

Internal topic: t/128545/9.
2024-08-20 16:16:05 +03:00
Joffrey JAFFEUX 5b17e85fe1
FIX: broken mentioned users with capitalized usernames (#28421)
This commit fixes two codepaths which where incorrectly working with capitalized usernames as we were doing a mix of username_lower and non lower username.

Also adds two specs for these cases.
2024-08-19 20:57:45 +02:00
David Taylor 02678b3107
DEV: Use `yarn`-installed terser in assets.rake (#28422)
This was previously relying on a globally-installed terser, which comes from a time before Discourse had proper JS dependency management
2024-08-19 16:45:15 +01:00
Loïc Guitaut 86f1c177d6 DEV: Remove unnecessary freedom patches
This patch removes two freedom patches:

- `mail_disable_starttls.rb`: this has been fixed in the 2.8 release of
  the mail gem, so we don’t need it anymore.
- `rails4.rb`: those methods have been deprecated for a while now and
  should have been dropped with Discourse v3.2.
2024-08-19 12:06:54 +02:00
Alan Guo Xiang Tan e1c4cf2566
DEV: Update `Guardian#is_developer` to only require admin for Developer (#28396)
In development mode and when a developer's email is configured as part
of `Rails.configuration.developer_emails`, the user can be trusted and
should not be required to be an admin user.
2024-08-19 13:46:28 +08:00
Guhyoun Nam 9c1812e071
FEATURE: add `system_user_max_attachment_size_kb` site setting (#28351)
* System user attachment size WIP

* spec check

* controller update

* add max to system_user_max_attachment_size_kb

* DEV: update to use static method for `max_attachment_size_for_user`

add test to use large image.
add check for failure.

* DEV: update `system_user_max_attachment_size_kb` default value to 0

remove unecessary test.
update tests to reflect the new default value of `system_user_max_attachment_size_kb`

* DEV: update maximum_file_size to check when is an attachment made by a system user

Add tests for when `system_user_max_attachment_size_kb` is over and under the limit
Add test for checking interaction with `max_attachment_size_kb`

* DEV: move `max_attachment_size_for_user` to private methods

* DEV: turn `max_attachment_size_for_user` into a static method

* DEV: typo in test case

* DEV: move max_attachment_size_for_user to private class method

* Revert "DEV: move max_attachment_size_for_user to private class method"

This reverts commit 5d5ae0b715.

---------

Co-authored-by: Gabriel Grubba <gabriel@discourse.org>
2024-08-16 11:03:39 -03:00
Gabriel Grubba fc33826dc5
DEV: Removal of create_post_for_category_and_tag_changes setting (#28321)
* DEV: Removal of create_post_for_category_and_tag_changes setting

reverting commit: #65f35e1
and adding a migration to remove the setting
ref: t/132320

* DEV: change checks for zeros to check for nils

* DEV: remove create_post_for_category_and_tag_changes migration file

If anything goes wrong, we can always revert back to the previous state.
2024-08-14 12:12:21 -03:00
Gabriel Grubba f6fadd7129
FEATURE: bring plugin:create to core from `create-discourse-plugin` gem (#28350)
* FEATURE: bring plugin:create to core from `gem exec create-discourse-plugin`

* DEV: remove plugin_rake spec and updated plugin rake creation with `begin/rescue` block
2024-08-14 11:45:46 -03:00
Alan Guo Xiang Tan 10ff0ee0cc
FIX: Ensure we dispose of MiniRacer::Context before forking daemons (#28361)
This commit updates `Demon::Base#start` to call `Discourse.before_fork`
before forking. According to the docs in `mini_racer`, we need to
"Dispose manually of all MiniRacer::Context objects prior to forking".

This commit is motivated by a segmentation fault which we are seeing in
production when killing a daemon process. Backtrace of the core dump
includes traces of `mini_racer` so we think this is the cause. Note that
we are not 100% sure if this will fix the issue.
2024-08-14 12:45:34 +08:00
Krzysztof Kotlarek e82e255531
FIX: serialize Flags instead of PostActionType (#28362)
### Why?
Before, all flags were static. Therefore, they were stored in class variables and serialized by SiteSerializer. Recently, we added an option for admins to add their own flags or disable existing flags. Therefore, the class variable had to be dropped because it was unsafe for a multisite environment. However, it started causing performance problems. 

### Solution
When a new Flag system is used, instead of using PostActionType, we can serialize Flags and use fragment cache for performance reasons. 

At the same time, we are still supporting deprecated `replace_flags` API call. When it is used, we fall back to the old solution and the admin cannot add custom flags. In a couple of months, we will be able to drop that API function and clean that code properly. However, because it may still be used, redis cache was introduced to improve performance.

To test backward compatibility you can add this code to any plugin
```ruby
  replace_flags do |flag_settings|
    flag_settings.add(
      4,
      :inappropriate,
      topic_type: true,
      notify_type: true,
      auto_action_type: true,
    )
    flag_settings.add(1001, :trolling, topic_type: true, notify_type: true, auto_action_type: true)
  end
```
2024-08-14 12:13:46 +10:00
Krzysztof Kotlarek 559c9dfe0a
REVERT: FIX: serialize Flags instead of PostActionType (#28334) 2024-08-13 18:32:11 +10:00
Krzysztof Kotlarek 094052c1ff
FIX: serialize Flags instead of PostActionType (#28259)
### Why?
Before, all flags were static. Therefore, they were stored in class variables and serialized by SiteSerializer. Recently, we added an option for admins to add their own flags or disable existing flags. Therefore, the class variable had to be dropped because it was unsafe for a multisite environment. However, it started causing performance problems. 

### Solution
When a new Flag system is used, instead of using PostActionType, we can serialize Flags and use fragment cache for performance reasons. 

At the same time, we are still supporting deprecated `replace_flags` API call. When it is used, we fall back to the old solution and the admin cannot add custom flags. In a couple of months, we will be able to drop that API function and clean that code properly. However, because it may still be used, redis cache was introduced to improve performance.

To test backward compatibility you can add this code to any plugin
```ruby
  replace_flags do |flag_settings|
    flag_settings.add(
      4,
      :inappropriate,
      topic_type: true,
      notify_type: true,
      auto_action_type: true,
    )
    flag_settings.add(1001, :trolling, topic_type: true, notify_type: true, auto_action_type: true)
  end
```
2024-08-13 11:22:37 +10:00
Régis Hanol d10fd36319
FEATURE: participating users statistics (#28322)
Adds a new statistics (hidden from the UI, but available via the API) that tracks daily participating users.

A user is considered as "participating" if they have

- Reacted to a post
- Replied to a topic
- Created a new topic
- Created a new PM
- Sent a chat message
- Reacted to a chat message

Internal ref - t/131013
2024-08-12 23:47:13 +02:00
Loïc Guitaut 84823550d4 DEV: Enable Rails 7.1 defaults 2024-08-12 10:41:13 +02:00
Isaac Janzen 5b1d9d602f
DEV: Add `guardian` modifier to prevent sending PM (#28282) 2024-08-08 12:57:13 -05:00
Alan Guo Xiang Tan 4c0af24173
DEV: Log a warning message when a MiniScheduler scheduled job is stuck (#28258)
This commit adds a `MiniSchedulerLongRunningJobLogger` class which will
poll every 60 seconds for mini_scheduler jobs which are stuck. When it
detects that a job is stuck, it will log a warning message with the
current backtrace of the thread that is executing the job.

Note that for scheduled jobs which are executed at a frequency of less
than 30 minutes, we will log when the job has been executing for 30
minutes.

For scheduled jobs executed at a frequency of less than 2 hours, we will
log when the job has been executing for a duration greater than its
specified frequency.

For scheduled jobs executed at a frequency greater than 2 hours, we will
log as long as the job has been executing for more than 2 hours.
2024-08-08 12:20:16 +08:00
Jarek Radosz 064332ef6e
DEV: Update ember-cli-deprecation-workflow from 2.2.0 to 3.0.1 (#28226)
From dependabot PR:

<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/ember-cli/ember-cli-deprecation-workflow/releases">ember-cli-deprecation-workflow's releases</a>.</em></p>
<blockquote>
<h2>v3.0.1-ember-cli-deprecation-workflow</h2>
<h2>Release (2024-07-11)</h2>
<p>ember-cli-deprecation-workflow 3.0.1 (patch)</p>
<h4>🏠 Internal</h4>
<ul>
<li><code>ember-cli-deprecation-workflow</code>
<ul>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/192">#192</a> fix repository link in package.json (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/191">#191</a> update release plan workflow (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
</ul>
</li>
</ul>
<h4>Committers: 1</h4>
<ul>
<li>Chris Manson (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
</ul>
<h2>v3.0.0</h2>
<h2>Release (2024-06-25)</h2>
<p>ember-cli-deprecation-workflow 3.0.0 (major)</p>
<h4>💥 Breaking Change</h4>
<ul>
<li><code>ember-cli-deprecation-workflow</code>
<ul>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/159">#159</a> [BREAKING] Convert to a module. Drops support for Ember &lt; 3.28, requires manual initialization (<a href="https://github.com/lolmaus"><code>@​lolmaus</code></a>)</li>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/175">#175</a> Node 16 is the minimum supported version (<a href="https://github.com/mixonic"><code>@​mixonic</code></a>)</li>
</ul>
</li>
</ul>
<h4>🐛 Bug Fix</h4>
<ul>
<li><code>ember-cli-deprecation-workflow</code>
<ul>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/181">#181</a> Remove unused broccoli magic (<a href="https://github.com/simonihmig"><code>@​simonihmig</code></a>)</li>
</ul>
</li>
</ul>
<h4>📝 Documentation</h4>
<ul>
<li><code>ember-cli-deprecation-workflow</code>
<ul>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/184">#184</a> Update configuration paths in documentation (<a href="https://github.com/backspace"><code>@​backspace</code></a>)</li>
</ul>
</li>
</ul>
<h4>🏠 Internal</h4>
<ul>
<li><code>ember-cli-deprecation-workflow</code>
<ul>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/189">#189</a> start using release-plan (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/188">#188</a> start using pnpm (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/178">#178</a> Upgrade Ember CLI to 5.4 (<a href="https://github.com/lolmaus"><code>@​lolmaus</code></a>)</li>
<li><a href="https://redirect.github.com/ember-cli/ember-cli-deprecation-workflow/pull/170">#170</a> Bump Node, swap to npm, update CI pipeline (<a href="https://github.com/mixonic"><code>@​mixonic</code></a>)</li>
</ul>
</li>
</ul>
<h4>Committers: 5</h4>
<ul>
<li>Andrey Mikhaylov (lolmaus) (<a href="https://github.com/lolmaus"><code>@​lolmaus</code></a>)</li>
<li>Buck Doyle (<a href="https://github.com/backspace"><code>@​backspace</code></a>)</li>
<li>Chris Manson (<a href="https://github.com/mansona"><code>@​mansona</code></a>)</li>
<li>Matthew Beale (<a href="https://github.com/mixonic"><code>@​mixonic</code></a>)</li>
<li>Simon Ihmig (<a href="https://github.com/simonihmig"><code>@​simonihmig</code></a>)</li>
</ul>
</blockquote>
</details>
2024-08-07 17:59:42 +02:00
Natalie Tay a49a6941c6
DEV: Extend /filter's ability to order (#28242) 2024-08-07 16:37:00 +08:00
Osama Sayegh 5dbf812d32
DEV: Add site age and site activities section to the redesigned about page (#28214)
This commit continues on work laid out by 6039b513fe to redesign the /about page. In this commit, we add the site age and a section on the right hand side to show site activities/statistics such as topics, posts, sign-ups, likes etc.
2024-08-07 11:11:41 +03:00
David Taylor 6f80ebfc41
Revert "FIX: backup_restore.rb wants db user from user, not username (#28229)" (#28245)
This reverts commit 2e8273dcb4. We're seeing this cause issues in production multisite environments. Reverting while we investigate
2024-08-06 14:11:48 +01:00
Krzysztof Kotlarek fc2259d1c8
FIX: limit the number of custom flags to 50 (#28221)
Admin can create up to 50 custom flags. It is limited for performance reasons.

When the limit is reached "Add button" is disabled and backend is protected by guardian.
2024-08-06 10:50:12 +10:00
Jay Pfaffman 2e8273dcb4
FIX: backup_restore.rb wants db user from user, not username (#28229)
ActiveRecord::Base.connection_pool.db_config.configuration_hash has the username in `user`, not `username`.
2024-08-06 08:49:20 +08:00
Alan Guo Xiang Tan 2492fe7715
FIX: Set sane default for `Net::HTTP` when processing a request (#28141)
This commit patches `Net::HTTP` to reduce the default timeouts of 60
seconds when we are processing a request. There are certain routes in
Discourse which makes external requests and if the proper timeouts are
not set, we risk having the Unicorn master process force restarting the
Unicorn workers once the `30` seconds timeout is reached. This can
potentially become a vector for DoS attacks and this commit is aimed at
reducing the risk here.
2024-08-06 07:12:42 +08:00
David Taylor 1b28b8e169
DEV: Upgrade babel & remove vendored safari-bugfix transformation (#28208)
The Safari 15 bugfix has been rolled into @babel/preset-env in the most recent version, so we no longer need to carry our vendored copy.

This commit updates @babel/preset-env, runs npx yarn-deduplicate yarn.lock, and removes the vendored transform.

This commit also refactors our theme transpiler to use @babel/preset-env, with the same list of target browsers as our ember-cli build uses. This means we no longer need to maintain a separate list of babel transforms for themes.
2024-08-05 10:35:26 +01:00
David Taylor 11369018b6
DEV: Stop storing theme-transpiler on filesystem in development (#28198)
We were writing theme-transpiler JS files to the filesystem on a per-process basis, and then immediately reading them back in. Plus, there was no cleanup mechanism, so the tmp directory would grow indefinitely.

This commit refactors things so that the `build.js` script outputs the theme-transpiler source to stdout. That way, we can read it directly into the process, and then into mini-racer, without needing to go via the filesystem. No cleanup required!

In production, the theme-transpiler is still cached in a file during `assets:precompile`
2024-08-01 17:59:34 +01:00
Blake Erickson 6ee6b1f1d1
DEV: Add validation for allowed iframes setting (#28178)
- Adds a validator for the allowed iframes site setting
- Adds a migration to update any values that don't pass the validator

Follow up to: 188cb58daa
2024-08-01 06:51:02 -06:00
Sérgio Saquetim 366dfec16c
DEV: Add plugin API to perform actions when the plugin is turned on/off (#28156)
Co-authored-by: David Taylor <david@taylorhq.com>
2024-07-31 15:38:10 -03:00
Blake Erickson 30048ab97f
FIX: Video thumbnails for missing videos (#28152)
Skip trying to find a thumbnail if the video src cannot be found.

Bug report: https://meta.discourse.org/t/317423
2024-07-31 10:54:52 -06:00
Krzysztof Kotlarek 5830f2c9a1
FIX: double reviewable items bug (#28161)
When `SiteSetting.review_every_post` is true and the category `require_topic_approval` system creates two reviewable items.
1. Firstly, because the category needs approval, the `ReviewableQueuePost` record` is created - at this stage, no topic is created.
2. Admin is approving the review. The topic and first post are created.
3. Because `review_every_post` is true `queue_for_review_if_possible` callback is evaluated and `ReviewablePost` is created.
4. Then `ReviewableQueuePost` is linked to the newly generated topic and post.

At the beginning, we were thinking about hooking to those guards:
```
  def self.queue_for_review_if_possible(post, created_or_edited_by)
    return unless SiteSetting.review_every_post
    return if post.post_type != Post.types[:regular] || post.topic.private_message?
    return if Reviewable.pending.where(target: post).exists?
...
```
And add something like
```
 return if Reviewable.approved.where(target: post).exists?
```

However, because the callback happens in point 3. before the `ReviewableQueuePost` is linked to the `Topic`, it was not possible.

Therefore, when `ReviewableQueuePost` is creating a `Topic`, a new option called `:reviewed_queued_post` is passed to `PostCreator` to avoid creating a second `Reviewable`.
2024-07-31 12:45:00 +10:00
Alan Guo Xiang Tan e9c0a6dffe
FIX: Move downloading of Maxmind databases to after `assets:precompile` (#28157)
We have been seeing `ZLib::BufError` when running the `assets:precompile` rake
task.

```
I, [2024-07-30T05:19:58.807019 #1059]  INFO -- : Writing /var/www/discourse/public/assets/scripts/discourse-test-listen-boot-9b14a0fc65c689577e6a428dcfd680205516fe211700a71c7adb5cbcf4df2cc5.js
rake aborted!
Zlib::BufError: buffer error (Zlib::BufError)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache/file_store.rb💯in `<<'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache/file_store.rb💯in `set'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache.rb:212:in `set'
```

The hypothesis here is that some thread unsafe issue is causing the
problem since we download the Maxmind databases in a thread and run
decompression operations once the gzip file is downloaded.

In the near term, we plan to move downloading of Maxmind databases out
of the Rake task into a scheduled job so this patch should be considered
a temporary solution.

The trade-off here is that build time will slightly increase since we
are not longer downloading Maxmind databases while precompiling assets
at the same time.
2024-07-31 08:55:01 +08:00
Nat b29f49be97
Bump version to v3.4.0.beta1-dev 2024-07-30 15:35:33 +08:00
Nat a4bfe8adb4
Bump version to v3.3.0.beta6 2024-07-30 15:35:32 +08:00
Régis Hanol d5cd669464
FIX: race condition in Discourse.cache.fetch (#28124)
When using `Discourse.cache.fetch` with an expiry, there's a potential for a race condition due to how we read the data from redis.

The code used to be

```ruby
raw = redis.get(key) if !force
entry = read_entry(key) if raw
return entry if raw && !(entry == :__corrupt_cache__)
```

with `read_entry` defined as follow

```ruby
def read_entry(key)
  if data = redis.get(key)
    Marshal.load(data)
  end
rescue => e
  :__corrupt_cache__
end
```

If the value at "key" expired in redis between `raw = redis.get` and `entry = read_entry`, the `entry` variable would be `nil` despite `raw` having a value.

We would then proceed to return `entry` (which is `nil`) thinking it had a value, when it didn't.

The first `redis.get` can be skipped altogether and we can rely only on `read_entry` to read the data from redis. Thus avoiding the race condition and removing the double read operations.

Internal ref - t/132507
2024-07-30 09:08:12 +02:00
Nat 1239287697
Bump version to v3.3.0.beta6-dev 2024-07-30 14:35:57 +08:00
Nat f4cbf025b5
Bump version to v3.3.0.beta5 2024-07-30 14:35:56 +08:00
Natalie Tay 188cb58daa
SECURITY: Fixes for main (#28137)
* SECURITY: Update default allowed iframes list

Change the default iframe url list to all include 3 slashes.

* SECURITY: limit group tag's name length

Limit the size of a group tag's name to 100 characters.

Internal ref - t/130059

* SECURITY: Improve sanitization of SVGs in Onebox

---------

Co-authored-by: Blake Erickson <o.blakeerickson@gmail.com>
Co-authored-by: Régis Hanol <regis@hanol.fr>
Co-authored-by: David Taylor <david@taylorhq.com>
2024-07-30 14:19:01 +08:00
Alan Guo Xiang Tan 3193afe7ca
FIX: Rescue and warn when error is encountered in `DiscourseIpInfo.mmdb_download` (#28134)
Since switching to Maxmind permalinks to download the databases in
7079698cdf, we have received multiple
reports about rebuilds failing as `maxminddb:refresh` runs during
the rebuilds and failing to download the databases cases the rebuilds to
fail.

Downloading Maxmind databases should not sit in the critical rebuild
path but since we are close to the Discourse 3.3 release, we have opted
to just rescue all errors encountered when downloading the databases.

In the near future after the Discourse 3.3 release, we will be looking
at moving the downloading of maxmind databases out of the rebuild path.
2024-07-30 11:33:20 +08:00
Loïc Guitaut 1f5cbb9a44
DEV: Refactor translation overrides a bit (#28125)
This is a small followup of
https://github.com/discourse/discourse/pull/28037.
2024-07-30 09:56:46 +08:00
Osama Sayegh e9aa2c96e1
FIX: Add new/missing email templates to the email templates editor (#28075)
We have a dedicated admin page (`/admin/customize/email_templates`) that lets admins customize all emails that Discourse sends to users. The way this page works is that it lists all translations strings that are used for emails, and the list of translation strings is currently hardcoded and hasn't been updated in years. We've had a number of new emails that Discourse sends, so we should add those templates to the list to let admins easily customize those templates.

Meta topic: https://meta.discourse.org/t/3-2-x-still-ignores-some-custom-email-templates/308203.
2024-07-30 00:27:41 +03:00
Daniel Waterworth 1a95543e93
PERF: Don't use unaccent on string literals (#28120)
unaccent isn't marked as a pure function, so it gets evaluated per row
instead of once.
2024-07-29 15:37:25 -05:00
锦心 319075e4dd
FIX: Ensure JsLocaleHelper to not output deprecated translations (#28037)
* FIX: Ensure JsLocaleHelper to obly outputs up-to-date translations

The old implementation forgot to filter out deprecated
translations, causing these translations to incorrectly override the new
locale in the frontend.

This commit fills in the forgotten where clause, filtering only the
up-to-date part.

Related meta topic: https://meta.discourse.org/t/outdated-translation-replacement-causing-missing-translation/314352
2024-07-29 15:21:25 +08:00
Ted Johansson 3126c50baa
DEV: Update member access wizard step to use toggle group (#28013)
We want to change the design of the "member experience" step of the wizard from using checkbox switches to using radio toggle groups.
2024-07-29 14:07:06 +08:00
Loïc Guitaut cec8445f14 DEV: Ensure MF locales are checked properly
This patch fixes the `i18n:check` rake task which has been broken by
the `MessageFormat` upgrade.

It also adds a spec to ensure we generate valid MF code for all our
available locales.
2024-07-26 16:20:25 +02:00
Loïc Guitaut c4845acf5e FIX: Allow invalid plural keys in MF translations
We can get translations with invalid plural keys from Crowdin
or from custom overrides. Currently, this will raise an error and the
locales won’t be outputted at all.

This patch addresses this issue by using the new `strict: false` option
of our `messageformat-wrapper` gem, allowing to generate locales even if
there are invalid plural keys present.
2024-07-25 14:16:28 +02:00
锦心 5b05cdfbd9
FIX: Add post id to the anchor to prevent two identical anchors (#28070)
* FIX: Add post id to the anchor to prevent two identical anchors

We generate anchors for headings in posts. This works fine if there is
only one post in a topic with anchors. The problem comes when you have
two or more posts with the same heading. PrettyText generates anchors
based on the heading text using the raw context of each post, so it is
entirely possible to generate the same anchor for two posts in the same
topic, especially for topics with template replies

    Post1:
    # heading
    context
    Post2:
    # heading
    context

When both posts are on the page at the same time, the anchor will only
work for the first post, according to the [HTML specification](https://html.spec.whatwg.org/multipage/browsing-the-web.html#scroll-to-the-fragment-identifier).

> If there is an a element in the document tree whose root is document
> that has a name attribute whose value is equal to fragment, then
> return the *first* such element in tree order.

This bug is particularly serious in forums with non-Latin languages,
such as Chinese. We do not generate slugs for Chinese, which results in
the heading anchors being completely dependent on their order.

```ruby
[2] pry(main)> PrettyText.cook("# 中文")
=> "<h1><a name=\"h-1\" class=\"anchor\" href=\"#h-1\"></a>中文</h1>"
```

Therefore, the anchors in the two posts must be in exactly the same by
order, causing almost all of the anchors in the second post to be
invalid.

This commit solves this problem by adding the `post_id` to the anchor.
The new anchor generation method will add `p-{post_id}` as a prefix when
post_id is available:

```ruby
[3] pry(main)> PrettyText.cook("# 中文", post_id: 1234)
=> "<h1><a name=\"p-1234-h-1\" class=\"anchor\" href=\"#p-1234-h-1\"></a>中文</h1>"
```

This way we can ensure that each anchor name only appears once on the
same topic. Using post id also prevents the potential possibility of the
same anchor name when splitting/merging topics.
2024-07-25 13:50:30 +08:00
Krzysztof Kotlarek b64d01bc10
FIX: store information about the login method in the database. (#28054)
Previously in these 2 PRs, we introduced a new site setting `SiteSetting.enforce_second_factor_on_external_auth`.

https://github.com/discourse/discourse/pull/27547
https://github.com/discourse/discourse/pull/27674

When disabled, it should enforce 2FA for local login with username and password and skip the requirement when authenticating with oauth2.

We stored information about the login method in a secure session but it is not reliable. Therefore, information about the login method is moved to the database.
2024-07-24 17:19:58 +10:00
Vinoth Kannan d681decf01
FEATURE: use new site setting for onebox custom user agent. (#28045)
Previously, we couldn't change the user agent name dynamically for onebox requests. In this commit, a new hidden site setting `onebox_user_agent` is created to override the default user agent value specified in the [initializer](c333e9d6e6/config/initializers/100-onebox_options.rb (L15)).

Co-authored-by: Régis Hanol <regis@hanol.fr>
2024-07-24 04:45:30 +05:30
Régis Hanol 73ce3589ad
PERF: improves TextSentinel's seems_unpretentious check (#28044)
by scanning the text for the first word that is bigger than `max_word_length` instead of extracting (segmenting) all the words, computing their size, and comparing the maximum with `max_word_length`.

Idea from @mentalstring in https://meta.discourse.org/t/body-seems-unclear-error-when-users-are-typing-in-chinese/88715/14
2024-07-23 17:12:29 +02:00
锦心 a749387c80
FEATURE: Clean up previously logged information after permanently deleting posts (#28033)
* FEATURE: Clean up previously logged information after permanently deleting posts

When soft deleteing a topic or post, we will log some details in the
staff log, including the raw content of the post. Before this commit, we
will not clear the information in these records. Therefore, after
permanently deleting the post, `UserHistory` still retains copy of the
permanently deleted post. This is an unexpected behaviour and may raise
some potential legal issues.

This commit adds a behavior that when a post is permanently deleted, the
details column of the `UserHistory` associated with the post will be
overwritten to "(permanently deleted)". At the same time, for permanent
deletion, a new `action_id` is introduced to distinguish it from soft
deletion.

Related meta topic: https://meta.discourse.org/t/introduce-a-way-to-also-permanently-delete-the-sensitive-info-from-the-staff-logs/292546
2024-07-23 15:27:11 +08:00
Alan Guo Xiang Tan 2d59795e28
DEV: Log sidekiq job opts as string instead of hash (#28012)
This ensures that elasticsearch doesn't parse it as an object. There are
too many combination of job opts so we don't want elasticsearch to be
parsing and indexing this field as an object.
2024-07-23 06:27:43 +08:00
Régis Hanol 23aa88d203
FIX: Allow all caps within CJK text (#28018)
This improves the `TextSentinel` so that we don't consider CJK text as being uppercase and thus failing the validator.

It also optimizes the entropy computation by using native ruby `.bytes` to get all the bytes from the text.

It also tweaks the `seems_pronounceable?` and `seems_unpretentious?` check to use the `\p{Alnum}` unicode regexp group to account for non-latin languages.

Reference - https://meta.discourse.org/t/body-seems-unclear-error-when-users-are-typing-in-chinese/88715

Inspired by https://github.com/discourse/discourse/pull/27900

Co-authored-by: Paulo Magalhaes <mentalstring@gmail.com>
2024-07-22 17:35:52 +02:00
David Taylor eb8a224fd2
FIX: Improve error handling for messageformat compilation (#28015)
We are investigating reports of errors with messageformat compilation following 301713ef. This commit does not fix the issue, but it introduces some basic error handling to avoid completely breaking affected sites.

We will have a fix for the root cause soon.
2024-07-22 13:15:10 +01:00
Krzysztof Kotlarek fb7cc2d375
FIX: stop memoize PostActionTypes (#28005)
Memoizing all_flags on PostActionType was a mistake. This commit brings back the cache on the serialize level.
2024-07-22 17:35:49 +10:00
David Battersby 43aa47b118
FIX: experimental topics filter should allow tags with underscore (#27994)
When tags contain an underscore we should allow filtering in the same way, previously due to the regex those with underscores were not being found when filtering.
2024-07-20 00:58:29 +04:00
Alan Guo Xiang Tan 5038cad68e
DEV: Restore `missing_s3_uploads` stats count if site was restored (#27984)
This commit ensures that we reset the `missing_s3_uploads` status count
if there are no inventory files which are at least 2 days older than the
site's restored date.

Otherwise, a site with missing uploads but was subsequntly restored will
be continue to report missing uploads for 2 days.
2024-07-19 14:22:58 +08:00
Martin Brennan f5cbc3e3b8
FEATURE: Allow oneboxing private GitHub repo URLs and add private indicator to HTML (#27947)
Followup 560e8aff75

The linked commit allowed oneboxing private GitHub PRs,
issues, commits, and so on, but it didn't actually allow
oneboxing the root repo e.g https://github.com/discourse/discourse-reactions

We didn't have an engine for this, we were relying on OpenGraph
tags on the HTML rendering of the page like we do with other
oneboxes.

To fix this, we needed a new github engine for repos specifically.

Also, this commit adds a `data-github-private-repo` attribute to
PR, issue, and repo onebox HTML so we have an indicator of
whether the repo was private, which can be used for theme components
and so on.
2024-07-19 12:21:45 +10:00
Sérgio Saquetim 4b20021033
DEV: Restrict `include:unlisted` search option to users that can view unlisted topics (#27977) 2024-07-18 16:33:14 -03:00
Sérgio Saquetim 6a3e12a39c
FEATURE: Include advanced search option to include unlisted topics in the results (#27958)
---------

Co-authored-by: Régis Hanol <regis@hanol.fr>
2024-07-18 13:43:53 -03:00
Martin Brennan 48d13cb231
UX: Use a dropdown for SSL mode for group SMTP (#27932)
Our old group SMTP SSL option was a checkbox,
but this was not ideal because there are actually
3 different ways SSL can be used when sending
SMTP:

* None
* SSL/TLS
* STARTTLS

We got around this before with specific overrides
for Gmail, but it's not flexible enough and now people
want to use other providers. It's best to be clear,
though it is a technical detail. We provide a way
to test the SMTP settings before saving them so there
should be little chance of messing this up.

This commit also converts GroupEmailSettings to a glimmer
component.
2024-07-18 10:33:14 +10:00
Krzysztof Kotlarek c975c7fe1b
FEATURE: custom flag can require additional message (#27908)
Allow admin to create custom flag which requires an additional message.

I decided to rename the old `custom_flag` into `require_message` as it is more descriptive.
2024-07-18 10:10:22 +10:00
Isaac Janzen b3e0e920ed
DEV: Support adding a custom filter on `/filter` (#27927)
# Context

Currently there is no way to add a custom filter to the experimental `/filter` endpoint. While you can implement a custom `status:` there is no way to include the user's input in a custom query. 

# PR

This PR adds the ability to implement a custom filter. eg. `CUSTOM_FILTER:foo`

- Add `add_filter_custom_filter` for extension
- Add specs
2024-07-17 11:36:38 -05:00
Martin Brennan 0783bfbbfe
FIX: Use login SMTP auth for office365 in group mailer (#27931)
Followup 7b627dc14b

In this other commit, I changed the email settings validator
to always use the `login` authentication method for
office365 and outlook, but I didn't change the actual
group SMTP mailer to do this.

This commit fixes that issue and does some minor refactoring.
2024-07-16 16:21:14 +10:00
Alan Guo Xiang Tan 25778d9861
FIX: Return 400 response codes when topic list query params are invalid (#27930)
This commit updates `TopicQuery.validators` to cover all of the
public options listed in `TopicQuery.public_valid_options`. This is done
to fix the app returning a 500 response code when an invalid value, such
as a hash, is passed as a query param when accessing the various topic
list routes.
2024-07-16 10:30:04 +08:00
Nat d32675ace2
Bump version to v3.3.0.beta5-dev 2024-07-15 20:30:15 +08:00
Nat c9e24eca39
Bump version to v3.3.0.beta4 2024-07-15 20:30:14 +08:00
Martin Brennan 97e2b353f6
FEATURE: Allow for multiple GitHub onebox tokens (#27887)
Followup 560e8aff75

GitHub auth tokens cannot be made with permissions to
access multiple organisations. This is quite limiting.
This commit changes the site setting to be a "secret list"
type, which allows for a key/value mapping where the value
is treated like a password in the UI.

Now when a GitHub URL is requested for oneboxing, the
org name from the URL is used to determine which token
to use for the request.

Just in case anyone used the old site setting already,
there is a migration to create a `default` entry
with that token in the new list setting, and for
a period of time we will consider that token valid to
use for all GitHub oneboxes as well.
2024-07-15 13:07:36 +10:00
Krzysztof Kotlarek 9e4e591d60
Revert "FEATURE: custom flag can require additional message (#27706)" (#27906)
This reverts commit c0bcd979e3.
2024-07-15 09:45:57 +10:00
Krzysztof Kotlarek c0bcd979e3
FEATURE: custom flag can require additional message (#27706)
Allow admin to create custom flag which requires an additional message.

I decided to rename the old `custom_flag` into `require_message` as it is more descriptive.
2024-07-15 08:48:01 +10:00
Régis Hanol 8fe73ab404
DEV: better log around nil client_settings_json (#27868)
Internal ref t/132507
2024-07-11 11:30:53 +02:00
Loïc Guitaut 5ec227334a FIX: Don’t list values from disabled plugins
Currently, when a plugin registers a new reviewable type or extends a
list method (through `register_reviewble_type` and `extend_list_method`
respectively), the new array is statically computed and always returns
the same value. It will continue to return the same value even if the
plugin is disabled (it can be a problem in a multisite env too).

To address this issue, this patch changes how `extend_list_method`
works. It’s now using `DiscoursePluginRegistry.define_filtered_register`
to create a register on the fly and store the extra values from various
plugins. It then combines the original values with the ones from the
registry. The registry is already aware of disabled plugins, so when a
plugin is disabled, its registered values won’t be returned.
2024-07-11 10:51:48 +02:00
Alan Guo Xiang Tan 66878a9e80
DEV: Improve logging of Sidekiq errors when logstash logger is enabled (#27855)
This commit improves the logging of Sidekiq errors when
`ENABLE_LOGSTASH_LOGGER` is set to 1. Prior to this change, we would
only log the message and the backtrace. After this change, useful
information like `job.class`, `job.opts`, `job.problem_db`,
`exception.class` and `exception.message` are included in the log line
as well.
2024-07-11 14:17:18 +08:00
David Taylor ce3d91f422
DEV: Load `@json-editor/json-editor` using webpack async import (#27773) 2024-07-10 11:15:42 +01:00
Régis Hanol 758b9dd0ba
FEATURE: email attachments in a details (#27804)
This change how we present attachments from incoming emails to now be "hidden" in a "[details]" so they don't "hang" at the end of the post.

This is especially useful when using Discourse as a support tool where email is the main communication channel. For various reasons, images are often duplicated by email user agents, and hiding them behind the details block help keep the conversation focused on the isssue at hand.

Internal ref t/122333
2024-07-10 09:59:27 +02:00
Loïc Guitaut 301713ef96 DEV: Upgrade the MessageFormat library (JS)
This patch upgrades the MessageFormat library to version 3.3.0 from
0.1.5.

Our `I18n.messageFormat` method signature is unchanged, and now uses the
new API under the hood.

We don’t need dedicated locale files for handling pluralization rules
anymore as everything is now included by the library itself.

The compilation of the messages now happens through our
`messageformat-wrapper` gem. It then outputs an ES module that includes
all its needed dependencies.

Most of the changes happen in `JsLocaleHelper` and in the `ExtraLocales`
controller.

A new method called `.output_MF` has been introduced in
`JsLocaleHelper`. It handles all the fetching, compiling and
transpiling to generate the proper MF messages in JS. Overrides and
fallbacks are also handled directly in this method.

The other main change is that now the MF translations are served through
the `ExtraLocales` controller instead of being statically compiled in a
JS file, then having to patch the messages using overrides and
fallbacks. Now the MF translations are just another bundle that is
created on the fly and cached by the client.
2024-07-10 09:51:25 +02:00
Alan Guo Xiang Tan c9775d5f72
DEV: Apply `Logster.store.ignore` to `DiscourseLogstashLogger` as well (#27819)
This commit updates `DiscourseLogstashLogger#add_with_opts` to avoid
logging messages that matches regexp patterns configured in
`Logster.store.ignore`. Those error logs are mostly triggered by clients
and do not serve any useful purpose.
2024-07-10 13:51:42 +08:00
Alan Guo Xiang Tan 3fee52eae3
PERF: Update `docker:test` to run QUnit with more cores if available (#27816)
This commit updates the `docker:test` rake task to run core and plugin
QUnit tests in parallel using half the number of available CPU
processors to speed up time it takes to run the tests on hardware with
more CPU cores.

Before this commit, core QUnit tests ran by the `docker:test` rake task was capped at 3 parallel
processes while plugin QUnit tests was not ran in parallel.
2024-07-10 11:27:20 +08:00
Alan Guo Xiang Tan b4b7fa17af
DEV: Add exception class/message to `DiscourseLogstashLogger` take 2 (#27815)
This is the second take of af2bd4cc50 to
account for messages which contains newlines.
2024-07-10 11:04:17 +08:00