Regressed in eec10efc3d. It means that backend plugin spec failures in CI were not failing the spec suite.
Fixes recent regressions and skips two of them - to be handled next week.
---------
Co-authored-by: Andrei Prigorshnev <a.prigorshnev@gmail.com>
https://meta.discourse.org/t/improving-mailman-email-parsing/253041
When mirroring a public mailling list which uses mailman, there were some cases where the incoming email was not associated to the proper user.
As it happens, for various (undertermined) reasons, the email from the sender is often not in the `From` header but can be in any of the following headers: `Reply-To`, `CC`, `X-Original-From`, `X-MailFrom`.
It might be in other headers as well, but those were the ones we found the most reliable.
The old method updated only existing records, without considering that
new tags might have been created or some tags might not exist anymore.
This was usually not a problem because the stats were also updated by
other code paths.
However, the ensure consistency job should be more solid and help when
other code paths fail or after importing data.
Also, update category tag stats too should happen when updating other
category stats as well.
Currently processing emails that are blank or have a nil value for the mail will cause several errors.
This update allows emails with blank body or missing sender to log the blank email error to the mail logs rather than throwing an error.
In #21498, we split `BaseStore#download` into a "safe" version which returns nil on errors, and an "unsafe" version which raises an exception, which was the old behaviour of `#download`.
This change updates call sites that used the old `#download`, which raised exceptions, to use the new `#download!` to preserve behaviour (and silence deprecation warnings.)
It also silences the deprecation warning in tests.
* DEV: add new thread icon
* FIX: Use new thread icon, fix typo in SVG
UX: move the thread list icon to the right of
the collapse button
---------
Co-authored-by: Martin Brennan <martin@discourse.org>
* FIX: Displaying the wrong number of minimum tags in the composer
When the minimum number of tags set for the category is larger than the minimum number of tags
set in the category tag-groups, the composer was displaying the wrong value.
This commit fixes the value displayed in the composer to show the max value between the required
for the category and the tag-groups set for the category.
This bug was reported on Meta in https://meta.discourse.org/t/tags-from-multiple-tag-groups-required-only-suggest-select-at-least-one-tag/263817
* FIX: Limiting tags in categories not working as expected
When a category was restricted to a tag group A, which was set to only allow
one tag from the group per topic, selecting a tag belonging only to A returned
other tags from A that also belonged to other group/s (if any).
Example:
Tag group A: alpha, beta, gamma, epsilon, delta
Tag group B: alpha, beta, gamma
Both tag groups set to only allow one tag from the group per topic.
If Category 1 was set to only allow tags from the tag group A, and the first tag
selected was epsilon, then, because they also belonged to tag group B, the tags
alpha, beta, and gamma were still returned as valid options when they should not be.
This commit ensures that once a tag from a tag group that restricts its tags to
one per topic is selected, no other tag from this group is returned.
This bug was reported on Meta in https://meta.discourse.org/t/limiting-tags-to-categories-not-working-as-expected/263143.
* FIX: Moving topics does not prompt to add required tag for new category
When a topic moved from a category to another, the tag requirements
of the new category were not being checked.
This allowed a topic to be created and moved to a category:
- that limited the tags to a tag group, with the topic containing tags
not allowed.
- that required N tags from a tag group, with the topic not containing
the required tags.
This bug was reported on Meta in https://meta.discourse.org/t/moving-tagged-topics-does-not-prompt-to-add-required-tag-for-new-category/264138.
* FIX: Editing topics with tag groups from parents allows incorrect tagging
When there was a combination between parent tags defined in a tag group
set to allow only one tag from the group per topic, and other tag groups
relying on this restriction to combine the children tag types with the
parent tag, editing a topic could allow the user to insert an invalid
combination of these tags.
Example:
Automakers tag group: landhover, toyota
- group set to limit one tag from the group per topic
Toyota models group: land-cruiser, hilux, corolla
Landhover models group: evoque, defender, discovery
If a topic was initially set up with the tags toyota, land-cruiser it was
possible to edit it by removing the tag toyota and adding the tag landhover
and other landhover model tags like evoque for example.
In this case, the topic would end up with the tags toyota, land-cruiser,
landhover, evoque because Discourse will automatically insert the
missing parent tag toyota when it detects the tag land-cruiser.
This combination of tags would violate the restriction specified in
the Automakers tag group resulting in an invalid combination of tags.
This commit enforces that the "one tag from the group per topic"
restriction is verified before updating the topic tags and also
make sure the verification checks the compatibility of parent tags that
would be automatically inserted.
After the changes, the user will receive an error similar to:
The tags land-cruiser, landhover cannot be used simultaneously.
Please include only one of them.
Watched words were converted to regular expressions containing \W, which
handled only ASCII characters. Using [^[:word]] instead ensures that
UTF-8 characters are also handled correctly.
* Color for turbo_rspec in CI (`progress` and `documentation` formats)
* Show "DONE" only when `documentation` formatter is used
* Fix formatting
* Collapse RSpec commands
* Add line wrapping to the `progress` formatter (to mitigate GH Actions issue)
- Update welcome topic copy
- Edit the welcome topic automatically when the title or description changes
- Remove “Create your Welcome Topic” banner/CTA
- Add "edit welcome topic" user tip
### Background
When SSRF detection fails, the exception bubbles all the way up, causing a log alert. This isn't actionable, and should instead be ignored. The existing `rescue` does already ignore network errors, but fails to account for SSRF exceptions coming from `FinalDestination`.
### What is this change?
This PR does two things.
---
Firstly, it introduces a common root exception class, `FinalDestination::SSRFError` for SSRF errors. This serves two functions: 1) it makes it easier to rescue both errors at once, which is generally what one wants to do and 2) prevents having to dig deep into the class hierarchy for the constant.
This change is fully backwards compatible thanks to how inheritance and exception handling works.
---
Secondly, it rescues this new exception in `UserAvatar.import_url_for_user`, which is causing sporadic errors to be logged in production. After this SSRF errors are handled the same as network errors.
After this change, in order to join a chat channel, a user needs to be in a group with at least “Reply” permission for the category. If the user only has “See” permission, they are able to preview the channel, but not join it or send messages. The auto-join function also follows this new restriction.
---------
Co-authored-by: Martin Brennan <martin@discourse.org>
A category's slug can be encoded when
`SiteSetting.slug_generation_method` has been set to "encoded". As a
result, we have to support non ASCII characters as well.
This commit adds support for excluding categories when using the
`category:` filter with the `-` prefix. For example,
`-category:category-slug` will exclude all topics that belong to the
category with slug "category-slug" and all of its sub-categories.
To only exclude a particular category and not all of its sub-categories,
the `-` prefix can be used with the `=` prefix. For example,
`-=category:category-slug` will only exclude topics that belong to the
category with slug "category-slug". Topics in the sub-categories of
"category-slug" will still be included.
Specifying more than two tag names when using the `tag:` filter was not
working because of a bug in the code where only the first two value in
the `tag:` filter was being selected.
What is the problem?
Consider the following timeline:
1. OP starts a topic.
2. Troll responds snarkily.
3. Flagger flags the post as “inappropriate”.
4. Admin agrees and hides the post.
5. Troll ninja-edits the post within the grace period, but still snarky.
6. Flagger flags the post as inappropriate again.
The current behaviour is that the flagger is met with an error saying the post has been reviewed and can't be flagged again for the same reason.
The desired behaviour is after someone has edited a post, it should be flaggable again.
Why is this happening?
This is related to the ninja-edit feature, where within a set grace period no new revision is created, but a new revision is required to flag the same post for the same reason.
So essentially there is a window between the naughty corner cooldown where a flagged post can't be edited, and the ninja-edit grace period, where an edit can be made without a new revision. Posts that are edited within this window can't be re-flagged by the same user.
|-----------------|-------------------------------|
^ Flag accepted | ~~~~~~~~~~~~~ 🥷🏻 ~~~~~~~~~~~~ |
| ^ Editing grace period over
^ Naughty corner cooldown over
How does this fix it?
We already create a new revision when ninja-editing a post with a pending flag. The issue above happens only in the case where the flag is already accepted.
This change extends the existing behaviour so that a new revision is created when ninja-editing any flagged post, regardless of the status of the flag. (Deleted flags excluded.)
This should also help with posterity, avoiding situations where a successfully flagged post looks innocuous in the history because it was ninja-edited, and vice versa.
This header is used by Microsoft Exchange to indicate when certain types of
autoresponses should not be generated for an email.
It triggers our "is this mail autogenerated?" detection, but should not be used
for this purpose.
Clearing modifiers during a plugin spec run will affect all future specs. Instead, this commit introduces a more surgical `.unregister_modifier` API which plugins can use if they need to add/remove a modifier during a specific spec.
`TopicQuery#latest_results` which was being used by
`TopicQuery#list_filter` defaults to ordering by `Topic#bumped_at` in
descending order and that was taking precedent over the order scopes
being applied by `TopicsFilter`.
This allows multiple ordering to be specified by using a comma seperated string.
For example, `order:created,views` would order the topics by
`Topic#created_at` and then `Topic#views.
An older change about optimising images caused the selector that adds lightboxing not to apply on quoted images. This fixes that. The selector is now not applicable as optimisation occurs in a separate place.
This change allows quoted images to be opened in a lightbox.
This new modifier can be used by plugins to modify search ordering.
Specifically plugins such as discourse_solved can amend search ordering
so solved topics bump to the top.
Also correct edge case where low and high sort priority categories did not
order correctly when it came to closed/archived
This commit adds support for the following ordering filters:
1. `order:activity` which orders the topics by `Topic#bumped_at` in descending order
2. `order:activity-asc` which orders the topics by `Topic#bumped_at` in ascending order
3. `order:latest-post` which orders the topics by `Topic#last_posted_at` in descending order
4. `order:latest-post-asc` which orders the topics by `Topic#last_posted_at` in ascending order
5. `order:created` which orders the topics by `Topic#created_at` in descending order
6. `order:created-asc` which orders the topics by `Topic#created_at` in ascending order
7. `order:views` which orders the topics by `Topic#views` in descending order
8. `order:views-asc` which orders the topics by `Topic#views` in ascending order
9. `order:likes` which orders the topics by `Topic#likes` in descending order
10. `order:likes-asc` which orders the topics by `Topic#likes` in ascending order
11. `order:likes-op` which orders the topics by `Post#like_count` of the first post in the topic in descending order
12. `order:likes-op-asc` which orders the topics by `Post#like_count` of the first post in the topic in ascending order
13. `order:posters` which orders the topics by `Topic#participant_count` in descending order
14. `order:posters-asc` which orders the topics by `Topic#participant_count` in ascending order
15. `order:category` which orders the topics by `Category#name` of the topic's category in descending order
16. `order:category-asc` which orders the topics by `Category#name` of the topic's category in ascending order
Multiple order filters can be composed together and the order of ordering is applied based on the position of the filter
in the query string. For example, `order:views order:created` will order the topics by `Topic#views` in descending order
and then order the topics by `Topics#created_at` in descending order.
This commit adds support for the following date filters:
1. `activity-before:<YYYY-MM-DD>` which filters for topics that have been bumped at or before given date
2. `activity-after:<YYYY-MM-DD>` which filters for topics that have been bumped at or after given date
3. `created-before:<YYYY-MM-DD>` which filters for topics that have been created at or before given date
4. `created-after:<YYYY-MM-DD>` which filters for topics that have been created at or after given date
5. `latest-post-before:<YYYY-MM-DD>` which filters for topics with the
latest post posted at or before given date
6. `latest-post-after:<YYYY-MM-DD>` which filters for topics with the
latest post posted at or after given date
If the filter has an invalid value, i.e string that cannot be converted
into a proper date in the `YYYY-MM-DD` format, the filter will be ignored.
If either of each filter is specify multiple times, only the last
occurrence of each filter will be taken into consideration.
Large or broken images are removed from oneboxes, but sometimes images
were removed when they were oneboxed too. The reason is that images can
be oneboxed by the AllowlistedGenericOnebox or ImageOnebox and only
AllowlistedGenericOnebox was handled correctly.
- Move the old '`define_include_method`' arg to a `respect_plugin_enabled` kwarg
- Introduce an `include_condition` kwarg which can be passed a lambda with inclusion logic. Lambda will be run via `instance_exec` in the context of the serializer instance
This is backwards compatible - old-style invocations will trigger a deprecation message
- Move the old '`define_include_method`' arg to a `respect_plugin_enabled` kwarg
- Introduce an `include_condition` kwarg which can be passed a lambda with inclusion logic. Lambda will be run via `instance_exec` in the context of the serializer instance
This is backwards compatible - old-style invocations will trigger a deprecation message
Update chat and poll plugins to new pattern
By default, the Sprockets DirectiveProcessor introduces a newline between possible 'header' comments and the rest of the JS file. This causes sourcemaps to be offset by 1 line, and therefore breaks browser tooling. We know that Ember-Cli assets do not use Sprockets directives, so we can totally bypass the DirectiveProcessor for those files.
We're using v3 of Sprockets, which is no longer supported - upstreaming a fix will be difficult. Long term, we intend to move away from sprockets.
This PR adds the ability to destroy reviewables for a passed user via the API. This was not possible before as this action was reserved for reviewables for you created only.
If a user is an admin and calls the `#destroy` action from the API they are able to destroy a reviewable for a passed user. A user can be targeted by passed either their:
- username
- external_id (for SSO)
to the request.
In the case you attempt to destroy a non-personal reviewable and
- You are not an admin
- You do not access the `#destroy` action via the API
you will raise a `Discourse::InvalidAccess` (403) and will not succeed in destroying the reviewable.
Responding to negative behaviour tends to solicit more of the same. Common wisdom states: "don't feed the trolls".
This change codifies that advice by introducing a new nudge when hitting the reply button on a flagged post. It will be shown if either the current user, or two other users (configurable via a site setting) have flagged the post.
This is to help generate random channels and chat
messages for local dev. This was removed in 12a18d4d55
presumably because it was not worth refactoring at the
time.
I've only added these tasks:
- `rake chat:message:populate\[113,20\]` (channel_id, count)
- Generates the count of messages for a channel ID provided,
otherwise uses a random channel and 200 count.
- `rake chat:category_channel:populate`
- Creates a chat channel for a random category.
- `rake chat🧵populate\[132,5\]` (channel_id, message_count)
- Creates a thread with N messages in the specified channel,
and enables threading in that channel if necessary
It switches to a different command for detecting the current git branch because the old command always returned HEAD as branch when the git repository is on a detached head (e.g. tag). The new command doesn't return a branch when the repository is on a detached head, which allows us to fall back to the `version` variable that is stored in the git config since https://github.com/discourse/discourse_docker/pull/707. It contains the value of the `version` from `app.yml`.
It also includes a small change to specs, because our tests usually run on specific commits instead of a branch or tag, so Discourse.git_branch always returns "unknown". We can use the "unknown" branch for tests, so it makes sense to ignore it only in other envs.
This fixes a 500 error that occurs when adding a tag to a category's
restricted tag list if the category's restricted tags already included a
synonym tag.
* DEV: Support `likes-(min:max):<count>` on `/filter` route
This commit adds support for the following filters:
1. `likes-min`
2. `likes-max`
3. `views-min`
4. `views-max`
5. `likes-op-min`
6. `likes-op-max`
If the filter has an invalid value, i.e string that cannot be converted
into an integer, the filter will be ignored.
If either of each filter is specify multiple times, only the last
occurrence of each filter will be taken into consideration.
This commit adds support for the `posters-min:<count>` and
`posters-max:<count>` filters for the topics filtering query language.
`posters-min:1` will filter for topics with at least a one poster while
`posters-max:3` will filter for topics with a maximum of 3 posters.
If the filter has an invalid value, i.e string that cannot be converted
into an integer, the filter will be ignored.
If either of each filter is specify multiple times, only the last
occurence of each filter will be taken into consideration.
This commit adds support for the `posts-min:<count>` and
`posts-max:<count>` filters for the topics filtering query language.
`posts-min:1` will filter for topics with at least a one post while
`posts-max:3` will filter foor topics with a maximum of 3 posts.
If the filter has an invalid value, i.e string that cannot be converted
into an integer, the filter will be ignored.
If either of each filter is specify multiple times, only the last
occurence of each filter will be taken into consideration.
When we "pull hotlinked images" on onebox images, they are added to the uploads table and their dominant color is calculated. This commit adds the data to the HTML so that it can be used by the client in the same way as non-onebox images. It also adds specific handling to the new `discourse-lazy-videos` plugin.
This feature will allow sites to define which emoji are not allowed. Emoji in this list should be excluded from the set we show in the core emoji picker used in the composer for posts when emoji are enabled. And they should not be allowed to be chosen to be added to messages or as reactions in chat.
This feature prevents denied emoji from appearing in the following scenarios:
- topic title and page title
- private messages (topic title and body)
- inserting emojis into a chat
- reacting to chat messages
- using the emoji picker (composer, user status etc)
- using search within emoji picker
It also takes into account the various ways that emojis can be accessed, such as:
- emoji autocomplete suggestions
- emoji favourites (auto populates when adding to emoji deny list for example)
- emoji inline translations
- emoji skintones (ie. for certain hand gestures)
Why this change?
Previously `TopicsFilter` was designed in such a way that we act on a
filter sequentially based on the order it was matched. However, this
made it hard to support filters composition where a similar filter may
be present further in the query string. Because of this limitation, I
previously introduced a private API `TopicsFilter.register_scope` which
allows us to act on a filter only after the entire query string has been
scanned. However, I felt that it made the code complicated and hard to
reason about.
In thie commit, I've changed it such that we scan through the entire
query string and group the values of each filter together. This allows
us to act on the values of a given filter in one go which I find easier
to reason about. This also opens up the possibility for us to ignore
certain filters when it has been specified multiple times.
This commit adds support for the `created-by:<username>` query filter
which will return topics created by the specified user. Multiple
usernames can be specified by comma seperating the usernames like so:
`created-by:username1,username2`. This will filter for topics created by
either of the specified users. Multiple `created-by:<username>` can also
be composed together. `created-by:username1 created-by:username2` is
equivalent to `created-by:username1,username2`.
This was inadvertently removed in 4c46c7e. In very specific scenarios,
this could be used execute arbitrary JavaScript.
Only affects instances where SVGs are allowed as uploads and CDN is not
configured.
Previously, Discourse's password hashing was hard-coded to a specific algorithm and parameters. Any changes to the algorithm or parameters would essentially invalidate all existing user passwords.
This commit introduces a new `password_algorithm` column on the `users` table. This persists the algorithm/parameters which were use to generate the hash for a given user. All existing rows in the users table are assumed to be using Discourse's current algorithm/parameters. With this data stored per-user in the database, we'll be able to keep existing passwords working while adjusting the algorithm/parameters for newly hashed passwords.
Passwords which were hashed with an old algorithm will be automatically re-hashed with the new algorithm when the user next logs in.
Values in the `password_algorithm` column are based on the PHC string format (https://github.com/P-H-C/phc-string-format/blob/master/phc-sf-spec.md). Discourse's existing algorithm is described by the string `$pbkdf2-sha256$i=64000,l=32$`
To introduce a new algorithm and start using it, make sure it's implemented in the `PasswordHasher` library, then update `User::TARGET_PASSWORD_ALGORITHM`.
This commit adds support for the `in:<topic notification level>` query
filter. As an example, `in:tracking` will filter for topics that the
user is watching. Filtering for multiple topic notification levels can
be done by comma separating the topic notification level keys. For
example, `in:muted,tracking` or `in:muted,tracking,watching`.
Alternatively, the user can also compose multiple filters with `in:muted
in:tracking` which translates to the same behaviour as
`in:muted,tracking`.
This commit adds support for the `in:pinned` filter to the topics filtering
query language. When the filter is present, it will filter for topics
where `Topic#pinned_until` is greater than `Topic#pinned_at`.
This adds a SiteSetting, which when enabled, creates a small_action post for tag/category changes to the topic. It uses `topic.add_moderator_post, and passes raw text in, to describe the change.
This was introduced to the standard library in Ruby 2.4. In my testing, it produces the same result, and is around 8x faster than our pure-ruby implementation
Before this commit, composing multiple category filters with a query such as category:category1 and category:category2 would not return any results. This is because we were filtering for topics that belonged to both category1 and category2, which is impossible since a topic can only belong to a single category.
With this commit, specifying a query like category:category1 category:category2 will now translate to filtering for topics that belong to either the category1 or category2 category.
Invite only and Discourse connect could not be enabled at the same time
because of some legacy reason. This is a follow up commit to ce04db8,
355d51a and 40f6ceb.
Followup to #20915. If we're grouping search results that don't rely on core's search, we won't have access to pg headlines. This is now configurable via the constructor, defaulting to `SiteSetting.use_pg_headlines_for_excerpt`
We use schema_migration_details to determine the age of a site in multiple migrations. This commit moves the logic into a dedicated `Migration::Helpers` module so that it doesn't need to be re-implemented every time.
This commit adds support for filtering for topics in specific
subcategories via the categories filter query language.
For example: `category:documentation:admins` will filter for topics and
subcategory topics in
the category with slug "admins" whose parent category has the slug
"documentation".
The `=` prefix can also be used such that
`=category:documentation:admins` will exclude subcategory topics of the
category with slug "admins" whose parent category has the slug
"documentation".
On the `/filter` route, the categories filtering query language is now
supported in the input per the example provided below:
```
category:bug => topics in the bug category AND all subcategories
=category:bug => topics in the bug category excluding subcategories
category:bug,feature => allow for categories either in bug or feature
=category:bug,feature => allow for exact categories match excluding sub cats
categories: => alias for category
```
Currently composing multiple category filters is not supported as we
have yet to determine what behaviour it should result in. For example,
`category:bug category:feature` would now return topics that are in both
the `bug` and `feature` category but it is not possible for a topic to
belong to two categories.
Add a modifier that will allow us to tune the results returned by suggested.
At the moment the modifier allows us to toggle including random results.
This was created for the discourse-ai module. It needs to switch off random
results when it returns related topics.
Longer term we can use it to toggle unread/new and other aspects.
This also demonstrates how to test the contract when adding modifiers.
This was added in d3f02a1270
for hashtags but later removed usage in
b2acc416e7. It was removed because
serializing the user does not include things like their
secure_categories.
It is not used by any other plugins or themes, and can cause
issues where it will error when operating on a null user. Better
to just pass in the user_id and use it to look up a user
directly in a PrettyText::Helper
Similar to the _map added for group_list SiteSettings in
e62e93f83a, this commit adds
the same extension for simple and compact `list` type SiteSettings,
so developers do not have to do the `.to_s.split("|")` dance
themselves all the time.
For example:
```
SiteSetting.markdown_linkify_tlds
=> "com|net|org|io|onion|co|tv|ru|cn|us|uk|me|de|fr|fi|gov|ddd"
SiteSetting.markdown_linkify_tlds_map
=> ["com", "net", "org", "io", "onion", "co", "tv", "ru", "cn", "us", "uk", "me", "de", "fr", "fi", "gov"]
```
Krzysztof Kotlarek
Jarek Radosz
David Taylor
Régis Hanol
Mark VanLandingham
Bianca Nenciu
David Battersby
Ted Johansson
Joffrey JAFFEUX
Daniel Waterworth
chapoi
Sérgio Saquetim
Richard
Jan Cernik
Alan Guo Xiang Tan
Blake Erickson
Michael Brown
Natalie Tay
Sam
Penar Musaraj
Selase Krakani
Isaac Janzen
Martin Brennan
Gerhard Schlager
Roman Rizzi
Renato Atilio
