Commit Graph

1298 Commits

Author SHA1 Message Date
Martin Brennan e58e7a49f5
FIX: Bookmark reminder was clearing incorrectly (#28506)
Followup 76c56c8284

The change introduced above made it so the expired
bookmark reminders were cleared when using the bulk
action menu for bookmarks. However this also affected
clearing reminders for bookmarks when sending notifications.

When clearing bookmark reminders after sending notifications,
we take into account the auto delete preference:

* never          - The bookmark `reminder_at` date should not be cleared,
                   and the bookmark is kept.
* clear_reminder - The bookmark `reminder_at` date is cleared and
                   the bookmark is kept

The `never` option made it so "expired" bookmark reminder show
on the user's bookmark list.

This commit fixes the change from the other commit and only
forces clearing of `reminder_at` if using the bookmark bulk
action service.
2024-08-26 09:17:39 +10:00
锦心 ac365a86a2
FIX: Trigger user_logged_out event when the user logs out (#28482)
What did this fix?
===============

Previously, we only triggered this event in the `user.logged_out` method.
This resulted in the event being triggered only when the user was logged
out by the administrator or the site had strict logout mode enabled.
This bug affected customers who managed user status via webhooks.

meta topic: https://meta.discourse.org/t/user-log-out-event-not-triggered-in-webhooks/249464
2024-08-22 21:48:27 +08:00
Osama Sayegh 10ae7ef44a
FEATURE: Add estimated number of global and EU visitors to the about page (#28382)
This commit implements 2 new metrics/stats in the /about page for the _estimated_ numbers of unique visitors from the EU and the rest of the world. This new feature is currently off by default, but it can be enabled by turning on the hidden `display_eu_visitor_stats` site settings via the rails console.

There are a number of assumptions that we're making here in order to estimate the number of unique visitors, specifically:

1. we're assuming that the average of page views per anonymous visitor is similar to the average number of page views that a logged-in visitor makes, and
2. we're assuming that the ratio of logged in visitors from the EU is similar to the ratio of anonymous visitors from the EU

Discourse keeps track of the number of both logged-in and anonymous page views, and also the number of unique logged-in visitors and where they're from. So with those numbers and the assumptions above, we can estimate the number of unique anonymous visitors from the EU and the rest of the world.

Internal topic: t/128480.
2024-08-21 00:03:42 +03:00
Loïc Guitaut 0636855706 DEV: Allow using an AR relation as a model in services
This patch allows using an AR relation as a model in services without
fetching associated records. It will just check if the relation is empty
or not. In the former case, the execution will stop at that point, as
expected.
2024-08-20 16:32:46 +02:00
Joffrey JAFFEUX 5b17e85fe1
FIX: broken mentioned users with capitalized usernames (#28421)
This commit fixes two codepaths which where incorrectly working with capitalized usernames as we were doing a mix of username_lower and non lower username.

Also adds two specs for these cases.
2024-08-19 20:57:45 +02:00
Loïc Guitaut 86f1c177d6 DEV: Remove unnecessary freedom patches
This patch removes two freedom patches:

- `mail_disable_starttls.rb`: this has been fixed in the 2.8 release of
  the mail gem, so we don’t need it anymore.
- `rails4.rb`: those methods have been deprecated for a while now and
  should have been dropped with Discourse v3.2.
2024-08-19 12:06:54 +02:00
Alan Guo Xiang Tan 9d5f4bf2f1
DEV: Add tests for `Guardian#is_developer?` (#28419)
DEV: Add tests for `Guardian#is_developer?`

Follow up to e1c4cf2566
2024-08-19 14:20:09 +08:00
Martin Brennan dbafa10b3c
DEV: Add backup helpers for specs (#28394)
This has been split out from https://github.com/discourse/discourse/pull/28051
so we can use this same code in plugin specs before merging the core PR,
adds some helpers for creating local backup temp files
and cleaning them up.
2024-08-16 14:51:57 +10:00
Gabriel Grubba fc33826dc5
DEV: Removal of create_post_for_category_and_tag_changes setting (#28321)
* DEV: Removal of create_post_for_category_and_tag_changes setting

reverting commit: #65f35e1
and adding a migration to remove the setting
ref: t/132320

* DEV: change checks for zeros to check for nils

* DEV: remove create_post_for_category_and_tag_changes migration file

If anything goes wrong, we can always revert back to the previous state.
2024-08-14 12:12:21 -03:00
Krzysztof Kotlarek e82e255531
FIX: serialize Flags instead of PostActionType (#28362)
### Why?
Before, all flags were static. Therefore, they were stored in class variables and serialized by SiteSerializer. Recently, we added an option for admins to add their own flags or disable existing flags. Therefore, the class variable had to be dropped because it was unsafe for a multisite environment. However, it started causing performance problems. 

### Solution
When a new Flag system is used, instead of using PostActionType, we can serialize Flags and use fragment cache for performance reasons. 

At the same time, we are still supporting deprecated `replace_flags` API call. When it is used, we fall back to the old solution and the admin cannot add custom flags. In a couple of months, we will be able to drop that API function and clean that code properly. However, because it may still be used, redis cache was introduced to improve performance.

To test backward compatibility you can add this code to any plugin
```ruby
  replace_flags do |flag_settings|
    flag_settings.add(
      4,
      :inappropriate,
      topic_type: true,
      notify_type: true,
      auto_action_type: true,
    )
    flag_settings.add(1001, :trolling, topic_type: true, notify_type: true, auto_action_type: true)
  end
```
2024-08-14 12:13:46 +10:00
Krzysztof Kotlarek 559c9dfe0a
REVERT: FIX: serialize Flags instead of PostActionType (#28334) 2024-08-13 18:32:11 +10:00
Krzysztof Kotlarek 094052c1ff
FIX: serialize Flags instead of PostActionType (#28259)
### Why?
Before, all flags were static. Therefore, they were stored in class variables and serialized by SiteSerializer. Recently, we added an option for admins to add their own flags or disable existing flags. Therefore, the class variable had to be dropped because it was unsafe for a multisite environment. However, it started causing performance problems. 

### Solution
When a new Flag system is used, instead of using PostActionType, we can serialize Flags and use fragment cache for performance reasons. 

At the same time, we are still supporting deprecated `replace_flags` API call. When it is used, we fall back to the old solution and the admin cannot add custom flags. In a couple of months, we will be able to drop that API function and clean that code properly. However, because it may still be used, redis cache was introduced to improve performance.

To test backward compatibility you can add this code to any plugin
```ruby
  replace_flags do |flag_settings|
    flag_settings.add(
      4,
      :inappropriate,
      topic_type: true,
      notify_type: true,
      auto_action_type: true,
    )
    flag_settings.add(1001, :trolling, topic_type: true, notify_type: true, auto_action_type: true)
  end
```
2024-08-13 11:22:37 +10:00
Régis Hanol d10fd36319
FEATURE: participating users statistics (#28322)
Adds a new statistics (hidden from the UI, but available via the API) that tracks daily participating users.

A user is considered as "participating" if they have

- Reacted to a post
- Replied to a topic
- Created a new topic
- Created a new PM
- Sent a chat message
- Reacted to a chat message

Internal ref - t/131013
2024-08-12 23:47:13 +02:00
Loïc Guitaut 84823550d4 DEV: Enable Rails 7.1 defaults 2024-08-12 10:41:13 +02:00
Isaac Janzen 5b1d9d602f
DEV: Add `guardian` modifier to prevent sending PM (#28282) 2024-08-08 12:57:13 -05:00
Alan Guo Xiang Tan 4c0af24173
DEV: Log a warning message when a MiniScheduler scheduled job is stuck (#28258)
This commit adds a `MiniSchedulerLongRunningJobLogger` class which will
poll every 60 seconds for mini_scheduler jobs which are stuck. When it
detects that a job is stuck, it will log a warning message with the
current backtrace of the thread that is executing the job.

Note that for scheduled jobs which are executed at a frequency of less
than 30 minutes, we will log when the job has been executing for 30
minutes.

For scheduled jobs executed at a frequency of less than 2 hours, we will
log when the job has been executing for a duration greater than its
specified frequency.

For scheduled jobs executed at a frequency greater than 2 hours, we will
log as long as the job has been executing for more than 2 hours.
2024-08-08 12:20:16 +08:00
Natalie Tay a49a6941c6
DEV: Extend /filter's ability to order (#28242) 2024-08-07 16:37:00 +08:00
Loïc Guitaut 9e9d88f078 DEV: Use rspec mocks to properly verify a race condition
This is a small followup of https://github.com/discourse/discourse/pull/28124.
2024-08-06 15:57:04 +02:00
Krzysztof Kotlarek fc2259d1c8
FIX: limit the number of custom flags to 50 (#28221)
Admin can create up to 50 custom flags. It is limited for performance reasons.

When the limit is reached "Add button" is disabled and backend is protected by guardian.
2024-08-06 10:50:12 +10:00
Blake Erickson 6ee6b1f1d1
DEV: Add validation for allowed iframes setting (#28178)
- Adds a validator for the allowed iframes site setting
- Adds a migration to update any values that don't pass the validator

Follow up to: 188cb58daa
2024-08-01 06:51:02 -06:00
Sérgio Saquetim 366dfec16c
DEV: Add plugin API to perform actions when the plugin is turned on/off (#28156)
Co-authored-by: David Taylor <david@taylorhq.com>
2024-07-31 15:38:10 -03:00
Blake Erickson 30048ab97f
FIX: Video thumbnails for missing videos (#28152)
Skip trying to find a thumbnail if the video src cannot be found.

Bug report: https://meta.discourse.org/t/317423
2024-07-31 10:54:52 -06:00
Krzysztof Kotlarek 5830f2c9a1
FIX: double reviewable items bug (#28161)
When `SiteSetting.review_every_post` is true and the category `require_topic_approval` system creates two reviewable items.
1. Firstly, because the category needs approval, the `ReviewableQueuePost` record` is created - at this stage, no topic is created.
2. Admin is approving the review. The topic and first post are created.
3. Because `review_every_post` is true `queue_for_review_if_possible` callback is evaluated and `ReviewablePost` is created.
4. Then `ReviewableQueuePost` is linked to the newly generated topic and post.

At the beginning, we were thinking about hooking to those guards:
```
  def self.queue_for_review_if_possible(post, created_or_edited_by)
    return unless SiteSetting.review_every_post
    return if post.post_type != Post.types[:regular] || post.topic.private_message?
    return if Reviewable.pending.where(target: post).exists?
...
```
And add something like
```
 return if Reviewable.approved.where(target: post).exists?
```

However, because the callback happens in point 3. before the `ReviewableQueuePost` is linked to the `Topic`, it was not possible.

Therefore, when `ReviewableQueuePost` is creating a `Topic`, a new option called `:reviewed_queued_post` is passed to `PostCreator` to avoid creating a second `Reviewable`.
2024-07-31 12:45:00 +10:00
Régis Hanol d5cd669464
FIX: race condition in Discourse.cache.fetch (#28124)
When using `Discourse.cache.fetch` with an expiry, there's a potential for a race condition due to how we read the data from redis.

The code used to be

```ruby
raw = redis.get(key) if !force
entry = read_entry(key) if raw
return entry if raw && !(entry == :__corrupt_cache__)
```

with `read_entry` defined as follow

```ruby
def read_entry(key)
  if data = redis.get(key)
    Marshal.load(data)
  end
rescue => e
  :__corrupt_cache__
end
```

If the value at "key" expired in redis between `raw = redis.get` and `entry = read_entry`, the `entry` variable would be `nil` despite `raw` having a value.

We would then proceed to return `entry` (which is `nil`) thinking it had a value, when it didn't.

The first `redis.get` can be skipped altogether and we can rely only on `read_entry` to read the data from redis. Thus avoiding the race condition and removing the double read operations.

Internal ref - t/132507
2024-07-30 09:08:12 +02:00
Natalie Tay 188cb58daa
SECURITY: Fixes for main (#28137)
* SECURITY: Update default allowed iframes list

Change the default iframe url list to all include 3 slashes.

* SECURITY: limit group tag's name length

Limit the size of a group tag's name to 100 characters.

Internal ref - t/130059

* SECURITY: Improve sanitization of SVGs in Onebox

---------

Co-authored-by: Blake Erickson <o.blakeerickson@gmail.com>
Co-authored-by: Régis Hanol <regis@hanol.fr>
Co-authored-by: David Taylor <david@taylorhq.com>
2024-07-30 14:19:01 +08:00
Alan Guo Xiang Tan 3193afe7ca
FIX: Rescue and warn when error is encountered in `DiscourseIpInfo.mmdb_download` (#28134)
Since switching to Maxmind permalinks to download the databases in
7079698cdf, we have received multiple
reports about rebuilds failing as `maxminddb:refresh` runs during
the rebuilds and failing to download the databases cases the rebuilds to
fail.

Downloading Maxmind databases should not sit in the critical rebuild
path but since we are close to the Discourse 3.3 release, we have opted
to just rescue all errors encountered when downloading the databases.

In the near future after the Discourse 3.3 release, we will be looking
at moving the downloading of maxmind databases out of the rebuild path.
2024-07-30 11:33:20 +08:00
Loïc Guitaut 1f5cbb9a44
DEV: Refactor translation overrides a bit (#28125)
This is a small followup of
https://github.com/discourse/discourse/pull/28037.
2024-07-30 09:56:46 +08:00
Discourse Translator Bot f5fc49f5db
Update translations (#28115)
* Update translations

* DEV: Spec failed because of translation update

---------

Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2024-07-29 15:16:40 +02:00
锦心 319075e4dd
FIX: Ensure JsLocaleHelper to not output deprecated translations (#28037)
* FIX: Ensure JsLocaleHelper to obly outputs up-to-date translations

The old implementation forgot to filter out deprecated
translations, causing these translations to incorrectly override the new
locale in the frontend.

This commit fills in the forgotten where clause, filtering only the
up-to-date part.

Related meta topic: https://meta.discourse.org/t/outdated-translation-replacement-causing-missing-translation/314352
2024-07-29 15:21:25 +08:00
Ted Johansson 3126c50baa
DEV: Update member access wizard step to use toggle group (#28013)
We want to change the design of the "member experience" step of the wizard from using checkbox switches to using radio toggle groups.
2024-07-29 14:07:06 +08:00
Loïc Guitaut cec8445f14 DEV: Ensure MF locales are checked properly
This patch fixes the `i18n:check` rake task which has been broken by
the `MessageFormat` upgrade.

It also adds a spec to ensure we generate valid MF code for all our
available locales.
2024-07-26 16:20:25 +02:00
Loïc Guitaut 53210841c8 FIX: Validate MF strings when adding overrides
Currently, when adding translation overrides, values aren’t validated
for MF strings. This results in being able to add invalid plural keys or
even strings containing invalid syntax.

This patch addresses this issue by compiling the string when saving an
override if the key is detected as an MF one.

If there’s an error from the compiler, it’s added to the model errors,
which in turn is displayed to the user in the admin UI, helping them to
understand what went wrong.
2024-07-26 12:20:00 +02:00
Loïc Guitaut c4845acf5e FIX: Allow invalid plural keys in MF translations
We can get translations with invalid plural keys from Crowdin
or from custom overrides. Currently, this will raise an error and the
locales won’t be outputted at all.

This patch addresses this issue by using the new `strict: false` option
of our `messageformat-wrapper` gem, allowing to generate locales even if
there are invalid plural keys present.
2024-07-25 14:16:28 +02:00
Vinoth Kannan d681decf01
FEATURE: use new site setting for onebox custom user agent. (#28045)
Previously, we couldn't change the user agent name dynamically for onebox requests. In this commit, a new hidden site setting `onebox_user_agent` is created to override the default user agent value specified in the [initializer](c333e9d6e6/config/initializers/100-onebox_options.rb (L15)).

Co-authored-by: Régis Hanol <regis@hanol.fr>
2024-07-24 04:45:30 +05:30
Alan Guo Xiang Tan 2d59795e28
DEV: Log sidekiq job opts as string instead of hash (#28012)
This ensures that elasticsearch doesn't parse it as an object. There are
too many combination of job opts so we don't want elasticsearch to be
parsing and indexing this field as an object.
2024-07-23 06:27:43 +08:00
Régis Hanol 23aa88d203
FIX: Allow all caps within CJK text (#28018)
This improves the `TextSentinel` so that we don't consider CJK text as being uppercase and thus failing the validator.

It also optimizes the entropy computation by using native ruby `.bytes` to get all the bytes from the text.

It also tweaks the `seems_pronounceable?` and `seems_unpretentious?` check to use the `\p{Alnum}` unicode regexp group to account for non-latin languages.

Reference - https://meta.discourse.org/t/body-seems-unclear-error-when-users-are-typing-in-chinese/88715

Inspired by https://github.com/discourse/discourse/pull/27900

Co-authored-by: Paulo Magalhaes <mentalstring@gmail.com>
2024-07-22 17:35:52 +02:00
David Battersby 43aa47b118
FIX: experimental topics filter should allow tags with underscore (#27994)
When tags contain an underscore we should allow filtering in the same way, previously due to the regex those with underscores were not being found when filtering.
2024-07-20 00:58:29 +04:00
Alan Guo Xiang Tan 5038cad68e
DEV: Restore `missing_s3_uploads` stats count if site was restored (#27984)
This commit ensures that we reset the `missing_s3_uploads` status count
if there are no inventory files which are at least 2 days older than the
site's restored date.

Otherwise, a site with missing uploads but was subsequntly restored will
be continue to report missing uploads for 2 days.
2024-07-19 14:22:58 +08:00
Martin Brennan f5cbc3e3b8
FEATURE: Allow oneboxing private GitHub repo URLs and add private indicator to HTML (#27947)
Followup 560e8aff75

The linked commit allowed oneboxing private GitHub PRs,
issues, commits, and so on, but it didn't actually allow
oneboxing the root repo e.g https://github.com/discourse/discourse-reactions

We didn't have an engine for this, we were relying on OpenGraph
tags on the HTML rendering of the page like we do with other
oneboxes.

To fix this, we needed a new github engine for repos specifically.

Also, this commit adds a `data-github-private-repo` attribute to
PR, issue, and repo onebox HTML so we have an indicator of
whether the repo was private, which can be used for theme components
and so on.
2024-07-19 12:21:45 +10:00
Sérgio Saquetim 4b20021033
DEV: Restrict `include:unlisted` search option to users that can view unlisted topics (#27977) 2024-07-18 16:33:14 -03:00
Sérgio Saquetim 6a3e12a39c
FEATURE: Include advanced search option to include unlisted topics in the results (#27958)
---------

Co-authored-by: Régis Hanol <regis@hanol.fr>
2024-07-18 13:43:53 -03:00
Martin Brennan 48d13cb231
UX: Use a dropdown for SSL mode for group SMTP (#27932)
Our old group SMTP SSL option was a checkbox,
but this was not ideal because there are actually
3 different ways SSL can be used when sending
SMTP:

* None
* SSL/TLS
* STARTTLS

We got around this before with specific overrides
for Gmail, but it's not flexible enough and now people
want to use other providers. It's best to be clear,
though it is a technical detail. We provide a way
to test the SMTP settings before saving them so there
should be little chance of messing this up.

This commit also converts GroupEmailSettings to a glimmer
component.
2024-07-18 10:33:14 +10:00
Krzysztof Kotlarek c975c7fe1b
FEATURE: custom flag can require additional message (#27908)
Allow admin to create custom flag which requires an additional message.

I decided to rename the old `custom_flag` into `require_message` as it is more descriptive.
2024-07-18 10:10:22 +10:00
Isaac Janzen b3e0e920ed
DEV: Support adding a custom filter on `/filter` (#27927)
# Context

Currently there is no way to add a custom filter to the experimental `/filter` endpoint. While you can implement a custom `status:` there is no way to include the user's input in a custom query. 

# PR

This PR adds the ability to implement a custom filter. eg. `CUSTOM_FILTER:foo`

- Add `add_filter_custom_filter` for extension
- Add specs
2024-07-17 11:36:38 -05:00
Discourse Translator Bot 6dd09b0868
Update translations (#27936)
* Update translations

* DEV: Spec failed after recent translation changes

---------

Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2024-07-17 15:49:33 +02:00
Martin Brennan 97e2b353f6
FEATURE: Allow for multiple GitHub onebox tokens (#27887)
Followup 560e8aff75

GitHub auth tokens cannot be made with permissions to
access multiple organisations. This is quite limiting.
This commit changes the site setting to be a "secret list"
type, which allows for a key/value mapping where the value
is treated like a password in the UI.

Now when a GitHub URL is requested for oneboxing, the
org name from the URL is used to determine which token
to use for the request.

Just in case anyone used the old site setting already,
there is a migration to create a `default` entry
with that token in the new list setting, and for
a period of time we will consider that token valid to
use for all GitHub oneboxes as well.
2024-07-15 13:07:36 +10:00
Krzysztof Kotlarek 9e4e591d60
Revert "FEATURE: custom flag can require additional message (#27706)" (#27906)
This reverts commit c0bcd979e3.
2024-07-15 09:45:57 +10:00
Krzysztof Kotlarek c0bcd979e3
FEATURE: custom flag can require additional message (#27706)
Allow admin to create custom flag which requires an additional message.

I decided to rename the old `custom_flag` into `require_message` as it is more descriptive.
2024-07-15 08:48:01 +10:00
Loïc Guitaut b0480dd34e DEV: Avoid instance variables in specs
Small followup of https://github.com/discourse/discourse/pull/27705
2024-07-11 14:31:20 +02:00
Loïc Guitaut 5ec227334a FIX: Don’t list values from disabled plugins
Currently, when a plugin registers a new reviewable type or extends a
list method (through `register_reviewble_type` and `extend_list_method`
respectively), the new array is statically computed and always returns
the same value. It will continue to return the same value even if the
plugin is disabled (it can be a problem in a multisite env too).

To address this issue, this patch changes how `extend_list_method`
works. It’s now using `DiscoursePluginRegistry.define_filtered_register`
to create a register on the fly and store the extra values from various
plugins. It then combines the original values with the ones from the
registry. The registry is already aware of disabled plugins, so when a
plugin is disabled, its registered values won’t be returned.
2024-07-11 10:51:48 +02:00
Alan Guo Xiang Tan 66878a9e80
DEV: Improve logging of Sidekiq errors when logstash logger is enabled (#27855)
This commit improves the logging of Sidekiq errors when
`ENABLE_LOGSTASH_LOGGER` is set to 1. Prior to this change, we would
only log the message and the backtrace. After this change, useful
information like `job.class`, `job.opts`, `job.problem_db`,
`exception.class` and `exception.message` are included in the log line
as well.
2024-07-11 14:17:18 +08:00
Kris 0e3ed7ea2a
A11Y: improve topic list table markup for screenreaders (#27808)
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-07-10 13:14:36 -04:00
Régis Hanol 758b9dd0ba
FEATURE: email attachments in a details (#27804)
This change how we present attachments from incoming emails to now be "hidden" in a "[details]" so they don't "hang" at the end of the post.

This is especially useful when using Discourse as a support tool where email is the main communication channel. For various reasons, images are often duplicated by email user agents, and hiding them behind the details block help keep the conversation focused on the isssue at hand.

Internal ref t/122333
2024-07-10 09:59:27 +02:00
Loïc Guitaut 301713ef96 DEV: Upgrade the MessageFormat library (JS)
This patch upgrades the MessageFormat library to version 3.3.0 from
0.1.5.

Our `I18n.messageFormat` method signature is unchanged, and now uses the
new API under the hood.

We don’t need dedicated locale files for handling pluralization rules
anymore as everything is now included by the library itself.

The compilation of the messages now happens through our
`messageformat-wrapper` gem. It then outputs an ES module that includes
all its needed dependencies.

Most of the changes happen in `JsLocaleHelper` and in the `ExtraLocales`
controller.

A new method called `.output_MF` has been introduced in
`JsLocaleHelper`. It handles all the fetching, compiling and
transpiling to generate the proper MF messages in JS. Overrides and
fallbacks are also handled directly in this method.

The other main change is that now the MF translations are served through
the `ExtraLocales` controller instead of being statically compiled in a
JS file, then having to patch the messages using overrides and
fallbacks. Now the MF translations are just another bundle that is
created on the fly and cached by the client.
2024-07-10 09:51:25 +02:00
Alan Guo Xiang Tan c9775d5f72
DEV: Apply `Logster.store.ignore` to `DiscourseLogstashLogger` as well (#27819)
This commit updates `DiscourseLogstashLogger#add_with_opts` to avoid
logging messages that matches regexp patterns configured in
`Logster.store.ignore`. Those error logs are mostly triggered by clients
and do not serve any useful purpose.
2024-07-10 13:51:42 +08:00
Alan Guo Xiang Tan b4b7fa17af
DEV: Add exception class/message to `DiscourseLogstashLogger` take 2 (#27815)
This is the second take of af2bd4cc50 to
account for messages which contains newlines.
2024-07-10 11:04:17 +08:00
Alan Guo Xiang Tan 0b64cb9d8c
Revert "DEV: Remove `git_version` from `DiscourseLogstashLogger` log event (#27730)" (#27814)
This reverts commit bb0daa33cd.

This commit was not causing the problems we thought it was.
2024-07-10 10:36:22 +08:00
Alan Guo Xiang Tan af2bd4cc50
DEV: Add exception class and message fields to `DiscourseLogstashLogger` (#27787)
This commit updates `DiscourseLogstashlogger` to add the
`exception_class` and `exception_message` field to the log line when the
`progname` of the log message is `web-exception` which is Logster's
logging of exceptions during a web request.

The `exception_class` and `exception_message` fields allows consumers of
the logs to easily group logs together.
2024-07-10 08:54:39 +08:00
Martin Brennan 560e8aff75
FEATURE: Allow oneboxing private GitHub URLs (#27705)
This commit adds the ability to onebox private GitHub
commits, pull requests, issues, blobs, and actions using
a new `github_onebox_access_token` site setting. The token
must be set up in correctly to have access to the repos needed.

To do this successfully with the Oneboxer, we need to skip
redirects on the github.com host, otherwise we get a 404
on the URL before it is translated into a GitHub API URL
and has the appropriate headers added.
2024-07-10 09:39:31 +10:00
Alan Guo Xiang Tan 86e5f46175
DEV: Add hidden `s3_inventory_bucket_region` site setting (#27786)
This commit adds a hidden `s3_inventory_bucket_region` site setting to
specify the region of the `s3_inventory_bucket` when the `S3Inventory`
class initializes an instance of the `S3Helper`. By default, the
`S3Helper` class uses the value of the `s3_region` site setting but the
region of the `s3_inventory_bucket` is not always the same as the
`s3_region` configured.
2024-07-09 12:03:43 +08:00
Alan Guo Xiang Tan 28f5550886
DEV: Redo `DiscourseLogstashLogger` to not rely on `logstash-logger` (#27759)
This reverts commit 92d7d24d0f.
2024-07-08 14:03:11 +08:00
David Taylor 1bc18a5bf4
DEV: Remove spec following revert in 92d7d24d0f (#27736)
This spec was introduced as part of the original commit (8e10878e1a), but was missed from the revert.
2024-07-05 12:28:16 +01:00
Alan Guo Xiang Tan bb0daa33cd
DEV: Remove `git_version` from `DiscourseLogstashLogger` log event (#27730)
In our official Docker image, running git commands results in the
following error:

```
fatal: detected dubious ownership in repository at '/var/www/discourse'
To add an exception for this directory, call:

	git config --global --add safe.directory /var/www/discourse
```
2024-07-05 15:22:03 +08:00
Alan Guo Xiang Tan 8e10878e1a
DEV: Redo `DiscourseLogstashLogger` to not rely on `logstash-logger` (#27663)
This commit rewrites `DiscourseLogstashLogger` to not be an instance
of `LogstashLogger`. The reason we don't want it to be an instance of
`LogstashLogger` is because we want the new logger to be chained to
Logster's logger which can then pass down useful information like the
request's env and error backtraces which Logster has already gathered.

Note that this commit does not bother to maintain backwards
compatibility and drops the `LOGSTASH_URI` and `UNICORN_LOGSTASH_URI`
ENV variables which were previously used to configure the destination in
which `logstash-logger` would send the logs to. Instead, we introduce
the `ENABLE_LOGSTASH_LOGGER` ENV variable to replace both ENV and remove
the need for the log paths to be specified. Note that the previous
feature was considered experimental as stated in d888d3c54c
and the new feature should be considered experimental as well. The code
may be moved into a plugin in the future.
2024-07-05 09:41:52 +08:00
Loïc Guitaut 8d249457e8 DEV: Upgrade Rails to version 7.1
---------

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-07-04 10:58:21 +02:00
Jan Cernik 311b737c91
SECURITY: Fix Stored-dom XSS via Facebook Oneboxes 2024-07-03 20:49:22 +08:00
Ted Johansson 67e7808603
SECURITY: Add FastImage SSRF safe freedom patch 2024-07-03 20:49:15 +08:00
Natalie Tay 751750c7f8
FIX: Do not schedule avatar download if image is '' (#27687) 2024-07-03 19:50:31 +08:00
Martin Brennan 527f02e99f
FEATURE: Only count topic views for explicit/deferred tracked views (#27533)
Followup 2f2da72747

This commit moves topic view tracking from happening
every time a Topic is requested, which is susceptible
to inflating numbers of views from web crawlers, to
our request tracker middleware.

In this new location, topic views are only tracked when
the following headers are sent:

* HTTP_DISCOURSE_TRACK_VIEW - This is sent on every page navigation when
  clicking around the ember app. We count these as browser page views
  because we know it comes from the AJAX call in our app. The topic ID
  is extracted from HTTP_DISCOURSE_TRACK_VIEW_TOPIC_ID
* HTTP_DISCOURSE_DEFERRED_TRACK_VIEW - Sent when MessageBus initializes
  after first loading the page to count the initial page load view. The
  topic ID is extracted from HTTP_DISCOURSE_DEFERRED_TRACK_VIEW.

This will bring topic views more in line with the change we
made to page views in the referenced commit and result in
more realistic topic view counts.
2024-07-03 10:38:49 +10:00
Keegan George ea58140032
DEV: Remove summarization code (#27373) 2024-07-02 08:51:47 -07:00
Jan Cernik 6599b85a75
DEV: Block accidental serialization of entire AR models (#27668) 2024-07-01 17:08:48 -03:00
Osama Sayegh 4c1c25027d
DEV: Wiring for the admin about config page (#27492)
This commit continues work laid out by ffec8163b0 for the admin config page for the /about page. The last commit set up the user interface, and this one sets up all the wiring needed to make the input fields and save buttons actually work.

Internal topic: t/128544.
2024-07-01 05:40:37 +03:00
Régis Hanol a56321efb5 FIX: topic search order
When using the full page search and filtering down to a specific topic, the sort order was overwritten to by by "post_number".

This was confusing because we allow different type of sort order in the full search page.

This fixes it by only sorting by post_number when there's no "global" sort order defined.

Since the "new topic map" uses the search endpoint behind the scene, this also fixes the "most likes" popup.

Context - https://meta.discourse.org/t/searching-order-seems-to-be-broken-when-searching-in-topic/312303
2024-06-27 18:13:26 +02:00
Loïc Guitaut f58b844f45
Revert "DEV: Upgrade Rails to version 7.1" (#27625)
This reverts commit ce00f83173.
2024-06-26 18:55:05 +02:00
Régis Hanol 54a59be617 FEATURE: new 'should_add_email_attachments' plugin modifier
That can be used by plugins to control whether email attachments should be sent.

Internal ref - t/132149
2024-06-26 12:36:35 +02:00
Martin Brennan a128ce5c4c
FIX: Missing multipart/mixed boundary on emails (#27599)
Followup 96a0781bc1

When sending emails where secure uploads is enabled
and secure_uploads_allow_embed_images_in_emails is
true, we attach the images to the email, and we
do some munging with the final email so the structure
of the MIME parts looks like this:

```
multipart/mixed
  multipart/alternative
    text/plain
    text/html
  image/jpeg
  image/jpeg
  image/png
```

However, we were not specifying the `boundary` of the
`multipart/mixed` main content-type of the email, so
sometimes the email would come through appearing to
have an empty body with the entire thing attached as
one attachment, and some mail parsers considered the
entire email as the "epilogue" and/or "preamble".

This commit fixes the issue by specifying the boundary
in the content-type header per https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html
2024-06-25 13:43:10 +10:00
Loïc Guitaut ce00f83173 DEV: Upgrade Rails to version 7.1
---------

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-06-24 11:16:14 +02:00
Loïc Guitaut 160011793a Revert "DEV: Upgrade Rails to version 7.1 (#27539)"
This reverts commit ca4af53be8.
2024-06-21 11:20:40 +02:00
Kelv 60d5170587
DEV: add validation on content_security_policy_script_src site setting (#27564)
* DEV: add validation on content_security_policy_script_src site setting
2024-06-21 17:00:22 +08:00
Loïc Guitaut ca4af53be8 DEV: Upgrade Rails to version 7.1 (#27539)
* DEV: Upgrade Rails to 7.1

* FIX: Remove references to `Rails.logger.chained`

`Rails.logger.chained` was provided by Logster before Rails 7.1
introduced their broadcast logger. Now all the loggers are added to
`Rails.logger.broadcasts`.

Some code in our initializers was still using `chained` instead of
`broadcasts`.

* DEV: Make parameters optional to all FakeLogger methods

* FIX: Set `override_level` on Logster loggers (#27519)

A followup to f595d599dd

* FIX: Don’t duplicate Rack response

---------

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-06-21 09:44:06 +02:00
Penar Musaraj 33de5abb6e
DEV: Extract theme resolution to a helper (#27426)
This ensures that the theme id is resolved as early as possible in the
request cycle. This is necessary for the custom homepage to skip
preloading the wrong data.
2024-06-20 11:33:46 -04:00
Loïc Guitaut 982c005979 Revert "DEV: Upgrade Rails to version 7.1 (#27539)"
This reverts commit 2301dddcff.
2024-06-20 11:43:35 +02:00
Loïc Guitaut 2301dddcff
DEV: Upgrade Rails to version 7.1 (#27539)
* DEV: Upgrade Rails to 7.1

* FIX: Remove references to `Rails.logger.chained`

`Rails.logger.chained` was provided by Logster before Rails 7.1
introduced their broadcast logger. Now all the loggers are added to
`Rails.logger.broadcasts`.

Some code in our initializers was still using `chained` instead of
`broadcasts`.

* DEV: Make parameters optional to all FakeLogger methods

* FIX: Set `override_level` on Logster loggers (#27519)

A followup to f595d599dd

* FIX: Don’t duplicate Rack response

---------

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2024-06-20 10:33:01 +02:00
Rafael dos Santos Silva b2a9676f0b
FEATURE: Support Spotify Onebox (#27540) 2024-06-19 13:27:27 -03:00
Ted Johansson 96a0781bc1
FIX: Avoid duplicating e-mail body in summary e-mail (#27535)
We recently fixed a problem where secure upload images weren't re-attached when sending the activity summary e-mail.

This fix contained a bug that would lead to n copies of the e-mail body being included, n being the number of duplicates. This is because #fix_parts_after_attachments! was called once per attachment, and adding more parts to the multipart e-mail.

This PR fixes that by:

Adding a failing test case for the above.
Moving the looping over multiple posts into #fix_parts_after_attachments! itself.
2024-06-19 20:11:47 +08:00
Bianca Nenciu 2ce83b4d2e
FIX: Do not extract links for hotlinked images (#27538)
When a post is cooked the links are extracted and `TopicLink` instances
are created for each of them. These links are used in various places,
including the topic view, user summary page, etc.

In previous commit 48e5d1a, hotlinked images from Oneboxes have been
ignored from the texts, but hotlinked images turned into Lightboxes
were still extracted.
2024-06-19 14:25:29 +03:00
Ted Johansson 9cc030fe8d
DEV: Ensure digest e-mail secure image test uses actual digest e-mail (#27532)
The test that checks that securely uploaded images are re-attached to the digest e-mail wasn't rendering the actual digest e-mail template. This change fixes that.
2024-06-19 14:33:57 +08:00
Martin Brennan 83361b2fc5
FEATURE: Introduce site settings which require confirmation (#27315)
Many site settings can be distructive or have huge side-effects
for a site that the admin may not be aware of when changing it.

This commit introduces a `requires_confirmation` attribute that
can be added to any site setting. When it is true, a confirmation
dialog will open if that setting is changed in the admin UI,
optionally with a custom message that is defined in client.en.yml.

If the admin does not confirm, we reset the setting to its previous
clean value and do not save the new value.
2024-06-19 16:01:24 +10:00
Krzysztof Kotlarek cc4c199680
FEATURE: optional 2FA enforcement (#27506)
A new admin setting called `enforce_second_factor_on_external_auth`. It allows users to authenticate using external providers even when 2FA is forced with `enforce_second_factor` site setting.
2024-06-19 09:32:30 +10:00
Jarek Radosz 5cb84f8dcf
DEV: Revert rails 7.1 upgrade (#27522)
* Revert "FIX: Set `override_level` on Logster loggers (#27519)"

This reverts commit c1b0488c54.

* Revert "DEV: Make parameters optional to all FakeLogger methods"

This reverts commit 3318dad7b4.

* Revert "FIX: Remove references to `Rails.logger.chained`"

This reverts commit f595d599dd.

* Revert "DEV: Upgrade Rails to 7.1"

This reverts commit 081b00391e.
2024-06-18 23:48:30 +02:00
Loïc Guitaut 081b00391e DEV: Upgrade Rails to 7.1 2024-06-18 15:58:05 +02:00
Loïc Guitaut 2a22a3b51d FIX: Treat corrupt cache as cache miss
Currently when a cache entry is corrupt, we log the event without doing
anything else. It means the cache is still corrupt, and the proper value
isn’t computed again.

Normally, it’s very rare the cache becomes corrupt, but it can happen
when upgrading Rails for example and the cache format changes. This is
normally handled automatically by Rails but since we’re using a custom
cache class, we have to do it ourselves.

This patch takes the same approach the Rails team did, when a cache
entry is corrupt, we treat it as a miss, recomputing the proper value
and caching it in the new format.
2024-06-18 14:47:33 +02:00
Régis Hanol 53b3d2f0dc FIX: BBCode tag parser
Wasn't quite handling the cases where a closing bracket `]` was used in the value of one of the attributes.

```markdown
[chat quote=user channel="[broken]"]
```

Would not be correctly parsed because we would _greedily_ use the first `]` as the end of the tag even though it might be a valid character when inside proper quotes.

c39a4de139/app/assets/javascripts/discourse-markdown-it/src/features/bbcode-block.js (L62)

Re-wrote the `parseBBCodeTag` to properly handle the following cases

- A closing tag (aka `[/name]`) which are easy since they don't have any attributes
- An old `[quote=...]` format we used that doesn't uses quotes but still has various attributes of the form `key:value`
- All three valid BBCode opening tag formats we support
  - `[name]` without any attributes
  - `[name=foo]` with a default value
  - `[name foo=bar]` with some attributes

Ended up having to fix/rewrite the few bbcode rules that were using the `parseBBCodeTag` function, namely `d-wrap` and `discourse-local-dates`.

While working on this, I think I also found a way to get rid the of shims we had in place so that plugins could use the `parseBBCodeTag` function.

Reference - https://meta.discourse.org/t/having-a-right-bracket-in-a-channel-name-breaks-all-quotes-from-that-channel/308439
2024-06-18 10:47:18 +02:00
Kelv 2393234be5
DEV: remove legacy CSP implementation to make strict-dynamic only accepted behaviour (#27486)
* DEV: remove legacy CSP implementation that allowed for non-strict-dynamic behaviour
2024-06-18 16:40:53 +08:00
Renato Atilio 49fdccbb1d
FIX: restrict a href protocols on form template description (#27472) 2024-06-14 11:39:43 -03:00
Gabriel Grubba 93c2ae585c
FEATURE: Add tag_group option in `/filter` (#27427)
* FEATURE: Add tag_group option in `/filter`

* DEV: Update tag_group_filter in `/filter` to use SQL

* DEV: Add guardian to `/filter` by tag_group

* DEV: remove unused code

* DEV: Update tag_group filter implementation

* DEV: Add tests to tag_group filter
2024-06-13 14:03:49 -03:00
Loïc Guitaut c790c57415 DEV: Show parameters on a service contract failure
Now, when calling `StepsInspector#error` on a contract failure, the
output will contain the parameters provided to the contract.
2024-06-13 12:19:26 +02:00
Alan Guo Xiang Tan ae1d8c50da
DEV: Add ability to log a warning when Sidekiq job runs for too long (#27446)
This commits introduces the `sidekiq_report_long_running_jobs_minutes`
global setting which allows a site administrator to log a warning in the
Rails log when a Sidekiq job has been running for too long.

The warning is logged with the backtrace of the thread that is
processing the Sidekiq job to make it easier to figure out what a
sidekiq job is stuck on.
2024-06-13 14:24:44 +08:00
Martin Brennan e94ab11477
DEV: Allow for setting a message with SiteSetting.set_and_log (#27447)
When we turn on settings automatically for customers,
we sometimes use `.set_and_log` which will make a staff
action log for the site setting change. This is fine, but
there is no context for customers.

This change allows setting a message with `.set_and_log`, which
will be stored in the `details` column of the staff action log
created, which will show up on `/admin/logs/staff_action_logs`

---------

Co-authored-by: Kelv <kelv@discourse.org>
2024-06-13 14:59:49 +10:00
Ted Johansson a5df029be3
FIX: Email::Sender expects type to be a string (#27463)
In #26642 we introduced a change that re-attaches securely uploaded images in the digest e-mail. However, this change assumed that the type argument to the Email::Sender constructor would be a symbol, but when it is coming from the UserEmail job it is a string. This PR fixes that.
2024-06-13 11:53:25 +08:00