Previously, if the sso= payload was invalid Base64, but signed correctly, there would be no useful log or error. This commit improves things by:
- moving the base64 check before the signature checking so that it's properly surfaced
- split the ParseError exception into PayloadParseError and SignatureError
- add user-facing errors for both of those
- add/improve spec for both
There are a couple of reasons for this.
The first one is practical, and related to eager loading. Since /lib is not eager loaded, when the application boots, ProblemCheck["identifier"] will be nil because the child classes aren't loaded.
The second one is more conceptual. There turns out to be a lot of inter-dependencies between the part of the problem check system that live in /app and the parts that live in /lib, which probably suggests it should all go in /app.
Why this change?
On the `/admin/customize/themes/<:id>` route, we allow admins to edit
all settings via a settings editor. Prior to this change, trying to edit
and save a typed objects theme settings will result in an error on the
server.
This change creates a user setting that they can toggle if
they don't want to receive unread notifications when someone closes a
topic they have read and are watching/tracking it.
* A11Y: Update bulk selection keyboard shortcuts
Still a draft, but in current state this:
- adds `shift+b` as a keyboard shortcut to toggle bulk select
- adds `shift+d` as a keyboard shortcut to dismiss selected topic(s) (this
replaces `x r` and `x t` shortcuts)
- adds `x` as a keyboard shortcut to toggle selection (while in bulk select mode)
- fixes a bug with the `shift+a` shortcut, which was not working properly
Note that there is a breaking change here. Previously we had:
- `x r` to dismiss new topics
- `x t` to dismiss unread topics
However, this meant that we couldn't use `x` for selection, because the
itsatrap library does not allow the same character to be used both as a
single character shortcut and as the start of a sequence. The proposed
solution here is more consistent with other apps (Gmail, Github) that use
`x` to toggle selection.
Also, we never show both "Dismiss New" and "Dismiss Unread" in the same
screen, hence it makes sense to consolidate both actions under `shift+d`.
* Address review
Why this change?
Before this change, the new navigation item in the topic list will be
hidden when there are no new or unread topics for the user. We have
started to find this behaviour confusing UX wise so we decided to stop
hiding it.
Why this change?
This is a regression from introduced in
5c1147adf3 where dismissing unread topics
was changing the notification level of the topics instead of just
dismissing the unread posts.
What does this change do?
1. Bring back the previous implementation of the action
2. Fix the system test that was supposed to catch the problem but did
not.
When "lazy load categories" is enabled, the CategoryDrop component will
render at most 15 categories. If there are more categories, a "Show
more" link pointing to the categories page will be displayed.
Why this change?
Assertions against the database in a system test is not reliable because
the request sent from the client side may not have been processed when
the query to the database has been run.
The test was added to prevent a regression for 63119144ff
but it turns out that the test will still prevent the regression even if
we do not assert against the state in the database.
The build is broken due to some changes not being staged when I pushed the previous PR. The assertions that check that a job has been scheduled needs to be updated to reflect the new name.
Doing the following renames:
Jobs::ProblemChecks → Jobs::RunProblemChecks
Jobs::ProblemCheck → Jobs::RunProblemCheck
This is to disambiguate the ProblemCheck class name, ease fuzzy finding, and avoid needing to use :: in a bunch of places.
Before, the `back to forum` link was part of experimental admin navigation. It means that the link could be filtered out.
Because it is essential navigation, it should not be part of sidebar links and should be moved above the filter.
Why this change?
There are two problematic queries in question here when loading
notifications in various tabs in the user menu:
```
SELECT "notifications".*
FROM "notifications"
LEFT JOIN topics ON notifications.topic_id = topics.id
WHERE "notifications"."user_id" = 1338 AND (topics.id IS NULL OR topics.deleted_at IS NULL)
ORDER BY notifications.high_priority AND NOT notifications.read DESC,
NOT notifications.read AND notifications.notification_type NOT IN (5,19,25) DESC,
notifications.created_at DESC
LIMIT 30;
```
and
```
EXPLAIN ANALYZE SELECT "notifications".*
FROM "notifications"
LEFT JOIN topics ON notifications.topic_id = topics.id
WHERE "notifications"."user_id" = 1338
AND (topics.id IS NULL OR topics.deleted_at IS NULL)
AND "notifications"."notification_type" IN (5, 19, 25)
ORDER BY notifications.high_priority AND NOT notifications.read DESC, NOT notifications.read DESC, notifications.created_at DESC LIMIT 30;
```
For a particular user, the queries takes about 40ms and 26ms
respectively on one of our production instance where the user has 10K notifications while the site has 600K notifications in total.
What does this change do?
1. Adds the `index_notifications_user_menu_ordering` index to the `notifications` table which is
indexed on `(user_id, (high_priority AND NOT read) DESC, (NOT read)
DESC, created_at DESC)`.
1. Adds a second index `index_notifications_user_menu_ordering_deprioritized_likes` to the `notifications`
table which is indexed on `(user_id, (high_priority AND NOT read) DESC, (NOT read AND notification_type NOT IN (5,19,25)) DESC, created_at DESC)`. Note that we have to hardcode the like typed notifications type here as it is being used in an ordering clause.
With the two indexes above, both queries complete in roughly 0.2ms. While I acknowledge that there will be some overhead in insert,update or delete operations. I believe this trade-off is worth it since viewing notifications in the user menu is something that is at the core of using a Discourse forum so we should optimise this experience as much as possible.
Why this change?
The `/admin/customize/themes/:id/schema/name` route is a work in
progress but we want to be able to start navigating to it from the
`/admin/customize/themes/:id` route.
What does this change do?
1. Move `adminCustomizeThemes.schema` to a child route of
`adminCustomizeThemes.show`. This is because we need the model
from the parent route and if it isn't a child route we end up
having to load the theme model again from the server.
1. Add the `objects_schema` attribute to `ThemeSettingsSerializer`
1. Refactor `SiteSettingComponent` to be able to render a button
so that we don't have to hardcode the button rendering into the
`SiteSettings::String` component
Why this change?
Prior this change, we were using `URI.regexp` which was too strict as it
doesn't allow a URL path.
What does this change do?
Just parse the string using `URI.parse` and if it doesn't raise an error
we consider the string to be a valid URL
When a post is created by an incoming email, we show
an envelope icon on it which then opens a modal with the
raw email contents. Previously this was staff (admin+mod)
only, but now this commit adds the `view_raw_email_allowed_groups`
site setting, so any group can be added to give users permission
to see this.
Currently, the trust level method is calculating trust level based on maximum value from:
- locked trust level
- group automatic trust level
- previously granted trust level by admin
https://github.com/discourse/discourse/blob/main/lib/trust_level.rb#L33
Let's say the user belongs to groups with automatic trust level 1 and in the meantime meets all criteria to get trust level 2.
Each time, a user is removed from a group with automatic trust_level 1, they will be downgraded to trust_level 1 and promoted to trust_level 2
120a2f70a9/lib/promotion.rb (L142)
This will cause duplicated promotion messages.
Therefore, we have to check if the user meets the criteria, before downgrading.
## What?
Depending on the email software used, when you reply to an email that has some attachments, they will be sent along, since they're part of the embedded (replied to) email.
When Discourse processes the reply as an incoming email, it will automatically add all the (valid) attachments at the end of the post. Including those that were sent as part of the "embedded reply".
This generates posts in Discourse with duplicate attachments 🙁
## How?
When processing attachments of an incoming email, before we add it to the bottom of the post, we check it against all the previous uploads in the same topic. If there already is an `Upload` record, it means that it's a duplicate and it is _therefore_ skipped.
All the inline attachments are left untouched since they're more likely new attachments added by the sender.
CategoryChooser component usually displays just categories, but
sometimes it can show two none values: a "no category" or Uncategorized.
This commit makes sure that these are rendered correctly.
The problem was that the "none" item was automatically inserted in the
list of options, but that should not always happen. Toggling option
`autoInsertNoneItem` requires setting `none` too.
Why this change?
Prior to this change, the `CategoryList#find_relevant_topics` method was
loading and allocating all `CategoryFeaturedTopic` records in the
database to eventually only just use its `category_id` and `topic_id`
column. On a site with many `CategoryFeaturedTopic` records, the loading
of the ActiveRecord objects is a source of bottleneck.
The other problem with the `CategoryList#find_relevant_topics` method is
that it is unconditionally loading all records from the database even if
the user does not have access to the category. This again is wasteful.
What does this change do?
This commit makes it such that `CategoryList#find_relevant_topics` is
called only after `CategoryList#find_categories` in the `CategoryList#initialize`
method so that we can filter featured topics against categories that the
user has access to.
The second change is that Instead of loading `CategoryFeaturedTopic` records, we make an
inner join agains the `topics` table instead and skip any allocation of
`CatgoryFeaturedTopic` ActiveRecord objects.
It's February 29th, you know what that means...date-based flaky specs! If today is
February 29th 2024:
```
freeze_time(1.year.ago) -> Tue, 28 Feb 2023 01:38:42.732875000 UTC +00:00
```
Then
```
freeze_time(1.year.from_now) -> Wed, 28 Feb 2024 01:38:42.732875000 UTC +00:00
```
So then our "now" for the insert query ends up being "yesterday"
```
WHERE topic_hot_scores.topic_id IS NULL
AND topics.deleted_at IS NULL
AND topics.archetype <> :private_message
AND topics.created_at <= :now
```
When we send a bookmark reminder, there is an option to delete
the underlying bookmark. The Notification record stays around.
However, if you want to filter your notifications user menu
to only bookmark-based notifications, we were not showing unread
bookmark notifications for deleted bookmarks.
This commit fixes the issue _going forward_ by adding the
bookmarkable_id and bookmarkable_type to the Notification data,
so we can look up the underlying Post/Topic/Chat::Message
for a deleted bookmark and check user access in this way. Then,
it doesn't matter if the bookmark was deleted.
Returned results are ordered by ID, but the fabricated category and
private_category IDs are not always generated in the same order. This
caused the expected and actual results to be in different orders.
Why this change?
This is a follow up to 408d2f8e69. When
`ActiveRecord::ConnectionAdapaters::PostgreSQLAdatper#reload_type_map`
is called, we need to clear the type map cache otherwise migrations
adding an array column will end up throwing errors.
Why this change?
This patch has been added to address the problems identified in https://github.com/rails/rails/issues/35311. For every,
new connection created using the PostgreSQL adapter, 3 queries are executed to fetch type map information from the `pg_type`
system catalog, adding about 1ms overhead to every connection creation.
On multisite clusters where connections are reaped more aggressively, the 3 queries executed
accounts for a significant portion of CPU usage on the PostgreSQL cluster. This patch works around the problem by
caching the type map in a class level attribute to reuse across connections.
In rspec request specs, we do a huge verbose backtrace
when there is an error. However, 99% of the time you don't
care about pages and pages of activesupport/rspec gem
LOC in the backtrace...so this commit introduces an
env var RSPEC_EXCLUDE_GEMS_IN_BACKTRACE to allow for
turning this off.
Prior to this fix even when the user was not part of a group allowing sending pm we would show the prompt: "You've replied to ... X times, did you know you could send them a personal message instead?"
Why this change?
Before this change, the error messages returned when validating theme
settings of typed objects was an array of array instead of just an
array.
Why this change?
Previously in cac60a2c6b, I added support
for `type: "category"` for a property in the theme objects schema. This
commit extend the work previously to add support for types `topic`,
`post`, `group`, `upload` and `tag`.
As part of problem checks refactoring, we're moving some data to be DB backed. In this PR it's the tracking of problem check execution. When was it last run, when was the last problem, when should it run next, how many consecutive checks had problems, etc.
This allows us to implement the perform_every feature in scheduled problem checks for checks that don't need to be run every 10 minutes.
Why this change?
```
some_setting:
default: 0
type: string
```
A theme setting like the above will cause an error to be thrown on the
server when importing the theme because the default would be parsed as
an integer which caused an error to be thrown when we are validating the
value of the setting.
What does this change do?
Convert the value to a string when working with string typed theme
settings.
Why this change?
This change adds validation for the default value for `type: objects` theme
settings when a setting theme field is uploaded. This helps the theme
author to ensure that the objects which they specifc in the default
value adhere to the schema which they have declared.
When an error is encountered in one of the objects, the error
message will look something like:
`"The property at JSON Pointer '/0/title' must be at least 5 characters
long."`
We use a JSON Pointer to reference the property in the object which is
something most json-schema validator uses as well.
What does this change do?
1. This commit once again changes the shape of hash returned by
`ThemeSettingsObjectValidator.validate`. Instead of using the
property name as the key previously, we have decided to avoid
multiple levels of nesting and instead use a JSON Pointer as the key
which helps to simplify the implementation.
2 Introduces `ThemeSettingsObjectValidator.validate_objects` which
returns an array of validation error messages for all the objects
passed to the method.
Before this commit, we had a yarn package set up in the root directory and also in `app/assets/javascripts`. That meant two `yarn install` calls and two `node_modules` directories. This commit merges them both into the root location, and updates references to node_modules.
A previous attempt can be found at https://github.com/discourse/discourse/pull/21172. This commit re-uses that script to merge the `yarn.lock` files.
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Users can hide their public profile and presence information by checking
“Hide my public profile and presence features” on the
`u/{username}/preferences/interface` page. In that case, we also don't
want to return user status from the server.
This work has been started in https://github.com/discourse/discourse/pull/23946.
The current PR fixes all the remaining places in Core.
Note that the actual fix is quite simple – a5802f484d.
But we had a fair amount of duplication in the code responsible for
the user status serialization, so I had to dry that up first. The refactoring
as well as adding some additional tests is the main part of this PR.
This PR adds a new scheduled problem check that simply tries to connect to Twitter OAuth endpoint to check that it's working. It is using the default retry strategy of 2 retries 30 seconds apart.
Now forums can enroll their sites to be showcased in the Discourse [Discover](https://discourse.org/discover) directory. Once they enable the site setting `include_in_discourse_discover` to enroll their forum the `CallDiscourseHub` job will ping the `api.discourse.org/api/discover/enroll` endpoint. Then the Discourse Hub will fetch the basic details from the forum and add it to the review queue. If the site is approved then the forum details will be displayed in the `/discover` page.
Also, remove experimental setting and simply use top_menu for feature detection
This means that when people eventually enable the hot top menu, there will
be topics in it
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Previously, problem checks were all added as either class methods or blocks in AdminDashboardData. Another set of class methods were used to add and run problem checks.
As of this PR, problem checks are promoted to first-class citizens. Each problem check receives their own class. This class of course contains the implementation for running the check, but also configuration items like retry strategies (for scheduled checks.)
In addition, the parent class ProblemCheck also serves as a registry for checks. For example we can get a list of all existing check classes through ProblemCheck.checks, or just the ones running on a schedule through ProblemCheck.scheduled.
After this refactor, the task of adding a new check is significantly simplified. You add a class that inherits ProblemCheck, you implement it, add a test, and you're good to go.
Why this change?
This reverts 725561cf4b as it did not
address the root cause of the problem even though it fixed the failing tests we were seeing
when running `bundle exec rspec --tag ~type:multisite --order random:776 spec/system/admin_customize_form_templates_spec.rb spec/system/admin_sidebar_navigation_spec.rb spec/system/admin_site_setting_search_spec.rb spec/system/composer/dont_feed_the_trolls_popup_spec.rb spec/system/composer/review_media_unless_trust_level_spec.rb spec/system/create_account_spec.rb spec/system/editing_sidebar_tags_navigation_spec.rb spec/system/email_change_spec.rb spec/system/emojis/emoji_deny_list_spec.rb spec/system/group_activity_spec.rb spec/system/hashtag_autocomplete_spec.rb spec/system/network_disconnected_spec.rb spec/system/post_menu_spec.rb spec/system/post_small_action_spec.rb spec/system/tags_intersection_spec.rb spec/system/topic_list_focus_spec.rb spec/system/topic_page_spec.rb spec/system/user_page/user_profile_info_panel_spec.rb spec/system/viewing_group_members_spec.rb spec/system/viewing_navigation_menu_preferences_spec.rb`.
The root cause here is that `before_action`s added to a controller is
order dependent. As such, some requests were not setting the cookie
because the `before_action` callback was not even hit as a prior
`before_action` callbacks has raised an error such as the `check_xhr`
`before_action` callback.
To resolve the problem, we need to add the `prepend: true` option in
our monkey patch of `ApplicationController` to ensure that the
`before_action` callback which we have added is always run first.
This change also makes a couple of changes:
1. Improve the response body when a request is blocked by the `BlockRequestsMiddleware` middleware
so that it makes debugging easier.
2. Only set the cookies for non-xhr HTML format requests. Setting it for
other formats is kind of pointless.
Why this change?
We noticed that running `LOAD_PLUGINS=1 rspec --seed=38855 plugins/chat/spec/system/chat_new_message_spec.rb` locally
results in the system tests randomly failing. When we inspected the
request logs closely, we noticed that a `/presence/get` request from a
previous rspec example was being processed when a new rspec example is
already being run. We know it was from the previous rspec example
because inspecting the auth token showed the request using the auth
token of a user from the previous example. However, when a request using
an auth token from a previous example is used it ends up logging out the
same user on the server side because the user id in the cookie is the same
due to the use of `fab!`.
I did some research and there is apparently no way to wait until all
inflight requests by the browser has completed through capybara or
selenium. Therefore, we will add an identifier by attaching a cookie to all non-xhr requests so that
xhr requests which are triggered subsequently will contain the cookie in the request.
In the `BlockRequestsMiddleware` middleware, we will then reject any
requests when the value of the identifier in the cookie does not match the current rspec's example
location.
To see the problem locally, change `Auth::DefaultCurrentUserProvider.find_v1_auth_cookie` to the following:
```
def self.find_v1_auth_cookie(env)
return env[DECRYPTED_AUTH_COOKIE] if env.key?(DECRYPTED_AUTH_COOKIE)
env[DECRYPTED_AUTH_COOKIE] = begin
request = ActionDispatch::Request.new(env)
cookie = request.cookies[TOKEN_COOKIE]
# don't even initialize a cookie jar if we don't have a cookie at all
if cookie&.valid_encoding? && cookie.present?
puts "#{env["REQUEST_PATH"]} #{request.cookie_jar.encrypted[TOKEN_COOKIE]&.with_indifferent_access}"
request.cookie_jar.encrypted[TOKEN_COOKIE]&.with_indifferent_access
end
end
end
```
After which run the following command: `LOAD_PLUGINS=1 rspec --format documentation --seed=38855 plugins/chat/spec/system/chat_new_message_spec.rb`
It takes a few tries but the last spec should fail and you should see something like this:
```
assets/chunk.c16f6ba8b6824baa47ac.d41d8cd9.js {"token"=>"37d995a4b65395d3b343ec70fff915b4", "user_id"=>3382, "username"=>"bruce0", "trust_level"=>1, "issued_at"=>1708591735}
/assets/chunk.050148142e1d2dc992dd.d41d8cd9.js {"token"=>"37d995a4b65395d3b343ec70fff915b4", "user_id"=>3382, "username"=>"bruce0", "trust_level"=>1, "issued_at"=>1708591735}
/chat/api/channels/527/messages {"token"=>"37d995a4b65395d3b343ec70fff915b4", "user_id"=>3382, "username"=>"bruce0", "trust_level"=>1, "issued_at"=>1708591735}
/uploads/default/test_0/optimized/1X/_129430568242d1b7f853bb13ebea28b3f6af4e7_2_512x512.png {"token"=>"37d995a4b65395d3b343ec70fff915b4", "user_id"=>3382, "username"=>"bruce0", "trust_level"=>1, "issued_at"=>1708591735}
redirects to existing chat channel
redirects to chat channel if recipients param is missing (PENDING: Temporarily skipped with xit)
with multiple users
/favicon.ico {"token"=>"9a75c114c4d3401509a23d240f0a46d4", "user_id"=>3382, "username"=>"bruce0", "trust_level"=>1, "issued_at"=>1708591736}
/chat/new-message {"token"=>"9a75c114c4d3401509a23d240f0a46d4", "user_id"=>3382, "username"=>"bruce0", "trust_level"=>1, "issued_at"=>1708591736}
/presence/get {"token"=>"37d995a4b65395d3b343ec70fff915b4", "user_id"=>3382, "username"=>"bruce0", "trust_level"=>1, "issued_at"=>1708591735}
```
Note how the `/presence/get` request is using a token from the previous example.
Co-authored-by: David Taylor <david@taylorhq.com>
Why this change?
`current_url` does not rely on Capybara waiters so opt to use
`have_current_path` matcher instead. Also assert for email against
element displayed on the page instead of querying the database for it
which isn't really what system tests are meant for.
Add categories to the serialized search results together with the topics
when lazy load categories is enabled. This is necessary in order for the
results to be rendered correctly and display the category information.
Why this change?
The current shape of errors returns the error messages after it has been
translated but there are cases where we want to customize the error
messages and the current way return only translated error messages is
making customization of error messages difficult. If we
wish to have the error messages in complete sentences like
"`some_property` property must be present in #link 1", this is not
possible at the moment with the current shape of the errors we return.
What does this change do?
This change introduces the `ThemeSettingsObjectValidator::ThemeSettingsObjectErrors`
and `ThemeSettingsObjectValidator::ThemeSettingsObjectError` classes to
hold the relevant error key and i18n translation options.
We have separated and combined modes for sidebar panels.
Separated means the panels show only their own sections,
combined means sections from all panels are shown.
The admin sidebar only shows its own panels, so it must set
the mode to separated; however when we navigate to chat or
home we must revert to the initial mode setttings.
When hiding/showing the sidebar, as is the case on mobile
and using the toggle in the top left on desktop, we delete
and recreate the ember component on the page. This causes
the `sections` for each sidebar panel to get re-evaluated
every time.
For the admin sidebar, this means that we were constantly
re-adding the plugin links to the sidebar, causing duplication.
This can be fixed by just adding @cached to the getter for
sections.
The Digital Services Act requires a checkbox for any user who's flagging a post as illegal to confirm that they are flagging in good faith. This PR adds that.
Why this change?
This change supports a property of `type: category` in the schema that
is declared for a theme setting object. Example:
```
sections:
type: objects
schema:
name: section
properties:
category_property:
type: category
```
The value of a property declared as `type: category` will have to be a
valid id of a row in the `categories` table.
What does this change do?
Adds a property value validation step for `type: category`. Care has
been taken to ensure that we do not spam the database with a ton of
requests if there are alot of category typed properties. This is done by
walking through the entire object and collecting all the values for
properties typed category. After which, a single database query is
executed to validate which values are valid.
Why this change?
Firstly, note that this is not a security commit because this feature is
still in development and should not be used anywhere.
The reason we want to set a limit here is to greatly reduce the
possibility of a DoS attack in the future via `ThemeSetting` where
someone would set an arbituary large json string in
`ThemeSetting#json_value` and causing the server to run out of resources
trying to serialize/deserialize the value.
What does this change do?
Adds an ActiveRecord validation to ensure that the bytesize of the json
string being stored is smaller than or equal to 0.5mb. We believe 0.5mb
is a decent limit for now but we can review the limit in the future if
we believe it is too small.
Why this change?
The logic for validating a theme setting's value and default value was
not consistent as each part of the code would implement its own logic.
This is not ideal as the default value may be validated differently than
when we are setting a new value. Therefore, this commit seeks to
refactor all the validation logic for a theme setting's value into a
single service class.
What does this change do?
Introduce the `ThemeSettingsValidator` service class which holds all the
necessary helper methods required to validate a theme setting's value
A while ago we increased group SMTP read and open timeouts
to address issues we were seeing with Gmail sometimes giving
really long timeouts for these values. The commit was:
3e639e4aa7
Now, we want to increase all SMTP read timeouts to 30s,
since the 5s is too low sometimes, and the ruby Net::SMTP
stdlib also defaults to 30s.
Also, we want to slightly tweak the group smtp email job
not to fail if the IncomingEmail log fails to create, or if
a ReadTimeout is encountered, to avoid retrying the job in sidekiq
again and sending the same email out.
Why this change?
This commit updates `ThemeSettingsObjectValidator` to validate a
property's value against the validations listed in the schema.
For string types, `min_length`, `max_length` and `url` are supported.
For integer and float types, `min` and `max` are supported.
This commit adds another plugin modifier related to post
actions, similar to ae24e04a5e.
This will be used to exclude users who liked _and_ reacted to
the post, since now in discourse-reactions we make a Like when
a user reacts too. This will affect the display of the post footer.
Why this change?
This change adds property value validation to `ThemeSettingsObjectValidator`
for the following types: "string", "integer", "float", "boolean", "enum". Note
that this class is not being used anywhere yet and is still in
development.
When "lazy load categories" is enabled, only the categories present in
the sidebar are preloaded. This is insufficient because the parent
categories are necessary too for the sidebar to be rendered properly.
The strict-dynamic CSP directive is supported in all our target browsers, and makes for a much simpler configuration. Instead of allowlisting paths, we use a per-request nonce to authorize `<script>` tags, and then those scripts are allowed to load additional scripts (or add additional inline scripts) without restriction.
This becomes especially useful when admins want to add external scripts like Google Tag Manager, or advertising scripts, which then go on to load a ton of other scripts.
All script tags introduced via themes will automatically have the nonce attribute applied, so it should be zero-effort for theme developers. Plugins *may* need some changes if they are inserting their own script tags.
This commit introduces a strict-dynamic-based CSP behind an experimental `content_security_policy_strict_dynamic` site setting.
Followup to 978d52841a
It's complicated...we have multiple "anonymous" user concepts
in core, and even two classes called the exact same thing --
AnonymousUser.
The first case is Guardian::AnonymousUser, which is used for
people who are browsing the forum without being authenticated.
The second case is the model AnonymousUser, which is used when
a user is liking or posting anonymously via allow_anonymous_likes
or allow_anonymous_posting site settings.
We will untangle this naming nightmare later on...but for the
time being, only authenticated users who are pretending to be
anonymous should be able to like posts if allow_anonymous_likes
is on.
Why this change?
This is a first pass at adding an objects validator which main's job is
to validate an object against a defined schema which we will support. In
this pass, we are simply validating that properties that has been marked
as required are present in the object.
Reactions needs this to be able to filter out likes received
actions, where there is also an associated reaction, since
now most reactions also count as a like.
Why this change?
This is a follow up to c30aeafd9d. The
commit was calling `BlockRequestsMiddleware.allow_requests!` only before
`type: :system` tests but non system type tests could be running as well
and needs the `BlockRequestsMiddleware.allow_requests!` middleware to be
disabled too.
When making sensitive changes to an account (adding 2FA or passkeys), we
require users to confirm their password. This is to prevent an attacker
from adding 2FA to an account they have access to.
However, on newly created accounts, we should not require this, it's an
extra step and it doesn't provide extra security (since the account was
just created). This commit makes it so that we don't require session
confirmation for accounts created less than 5 minutes ago.
Why this change?
We have been debugging flaky system tests and noticed in https://github.com/discourse/discourse/actions/runs/7911902047/job/21596791343?pr=25690
that ActiveRecord connection checkout timeouts are encountered because
the Capybara server thread is processing requests even after
`Capybara.reset_session!` and ActiveRecord's `teardown_fixtures` have already been call.
The theory here is that an inflight request can still hit the Capybara
server even after `Capybara.reset_session!` has been called and end up
eating up an ActiveRecord connection for too long and also messing with
the database outside of a transaction.
What does this change do?
This change adds a `BlockRequestsMiddleware` middleware in the test
environment which is enabled to reject all incoming requests at the end
of each system test and before `Capybara.reset_session!` is called. At
the start of each RSpec test, the middleware is disabled again.
Why this change?
On CI, we have been seeing flaky system tests because ActiveRecord is
unable to checkout a connection. This patch is meant to help us debug
which thread is not returning the connection to the queue.
This would allow a theme component (or an API call) to reset the bump
date of a topic to a given post's created_at date.
I picked `post_id` as the parameter here because it provides a bit of
extra protection against accidentally resetting the bump date to a date
that doesn't make sense.
Fixes an issue where private topics that are quoted have an incorrectly formatted url when using a subfolder install.
This update returns a relative url that includes the base_path rather than a combination of base_url + base_path.
This commit includes several changes to make hashtags work when "lazy
load categories" is enabled. The previous hashtag implementation use the
category colors CSS variables, but these are not defined when the site
setting is enabled because categories are no longer preloaded.
This commit implements two fundamental changes:
1. load colors together with the other hashtag information
2. load cooked hashtag data asynchronously
The first change is implemented by adding "colors" to the HashtagItem
model. It is a list because two colors are returned for subcategories:
the color of the parent category and subcategory.
The second change is implemented on the server-side in a new route
/hashtags/by-ids and on the client side by loading previously unseen
hashtags, generating the CSS on the fly and injecting it into the page.
There have been minimal changes outside of these two fundamental ones,
but a refactoring will be coming soon to reuse as much of the code
and maybe favor use of `style` rather than injecting CSS into the page,
which can lead to page rerenders and indefinite grow of the styles.
This commit changes `max_image_megapixels` to be used
as is without multiplying by 2 to give extra leway.
We found in reality this was just causing confusion
for admins, especially with the already permissive
40MP default.
When we show the links to installed plugins in the admin
sidebar (for plugins that have custom admin routes) we were
previously only doing this if you opened /admin, not if you
navigated there from the main forum. We should just always
preload this data if the user is admin.
This commit also changes `admin_sidebar_enabled_groups` to
not be sent to the client as part of ongoing efforts to
not check groups on the client, since not all a user's groups
may be serialized.
When we insert into the hot set we add things with a score of 0
This means that if hot has more than batch size items in it with a score, then the 0s don't get an initial score
This corrects the situation by always ensuring we re-score:
1. batch size high scoring topics
2. (new) batch size recently bumped topics
* Update spec/models/topic_hot_scores_spec.rb
Co-authored-by: Isaac Janzen <50783505+janzenisaac@users.noreply.github.com>
---------
Co-authored-by: Isaac Janzen <50783505+janzenisaac@users.noreply.github.com>
This reverts commit 767b49232e.
If anything else (e.g. GTM integration) introduces a nonce/hash, then this change stops the splash screen JS to fail and makes sites unusable.
Removes duplication from LimitedEdit to see who can edit
posts, and also removes the old trust level setting check
since it's no longer necessary.
Also make it so staff can always edit since can_edit_post?
already has a staff escape hatch.
Why this change?
This commit introduces an experimental `type: objects` theme setting
which will allow theme developers to store a collection of objects as
JSON in the database. Currently, the feature is still in development and
this commit is simply setting up the ground work for us to introduce the
feature in smaller pieces.
What does this change do?
1. Adds a `json_value` column as `jsonb` data type to the `theme_settings` table.
2. Adds a `experimental_objects_type_for_theme_settings` site setting to
determine whether `ThemeSetting` records of with the `objects` data
type can be created.
3. Updates `ThemeSettingsManager` to support read/write access from the
`ThemeSettings#json_value` column.
Followup fb087b7ff6
post_links_allowed_groups is an odd check tied to
unrestricted_link_posting? in PostGuardian, in that
it doesn't have an escape hatch for staff like most
of the rest of these group based settings.
It doesn't make sense to exclude admins or mods from
posting links, so just always allow them to avoid confusion.
Browsers will ignore unsafe-inline if nonces or hashes are included in the CSP. When unsafe-inline is enabled, nonces and hashes are not required, so we can skip them.
Our strong recommendation remains that unsafe-inline should not be used in production.
Why this change?
This is caused by a regression in
59839e428f, where we stopped saving the
`Theme` object because it was unnecessary. However, it resulted in the
`after_save` callback not being called and hence
`Theme#update_javascript_cache!` not being called. As a result, some
sites were reporting that after runing a theme migration, the defaults
for the theme settings were used instead of the settings overrides
stored in the database.
What does this change do?
Add a call to `Theme#update_javascript_cache!` after running theme
migrations.
This fixes a bug where the sidebar categories would not be loaded when
the categories were lazy loaded because the sidebar uses the preloaded
category list, which was empty.
We just completed the 3.2 release, which marks a good time to drop some previously deprecated columns.
Since the column has been marked in ignored_columns, it has been inaccessible to application code since then. There's a tiny risk that this might break a Data Explorer query, but given the nature of the column, the years of disuse, and the fact that such a breakage wouldn't be critical, we accept it.
Why this change?
We are getting the following error on CI:
`Text file busy -
/github/home/.cache/selenium/chromedriver/linux64/121.0.6167.85/chromedriver`
This happens when two processes tries to download the chromedriver at
the same time. I'm not entirely sure why the previous implementatio is
not locking properly since we still saw the `Text file busy` error but
by accquring the lock before we even check for the existence of
`~/.cache/selenium`, we should be able to eliminate the chance of two
processes trying to download the chromedriver binary at the same time.
1. Don't show visited line for hot filter, it is in random order
2. Don't count likes on non regular posts (eg: whispers / small actions)
3. Don't count participants in non regular posts
Checking group permissions on the client does not work,
since not all groups are serialized to the client all
the time. We can check `uploaded_avatars_allowed_groups`
on the server side and serialize to the current user
instead.
CI runs on slower machines, so we need to use longer wait times. `Capybara.default_max_wait_time` is automatically reconfigured based on the environment.
JS assets added by plugins via `register_asset` will be outside the `assets/javascripts` directory, and are therefore exempt from being transpiled. That means that there isn't really any need to run them through DiscourseJsProcessor. Instead, we can just concatenate them together, and avoid the need for all the sprockets-wrangling.
This commit also takes the opportunity to clean up a number of plugin-asset-related codepaths which are no longer required (e.g. globs, handlebars)
1. Serial likers will just like a bunch of posts on the same topic, this will
heavily inflate hot score. To avoid artificial "heat" generated by one user only count
the first like on the topic within the recent_cutoff range per topic
2. When looking at recent topics prefer "unique likers", defer to total likes on
older topics cause we do not have an easy count for unique likers
3. Stop taking 1 off like_count, it is not needed - platforms like reddit
allow you to like own post so they need to remove it.
Why this change?
Returning an array makes it hard to immediately retrieve a setting by
name and makes the retrieval an O(N) operation. By returning an array,
we make it easier for us to lookup a setting by name and retrieval is
O(1) as well.
Why this change?
Since 1dba1aca27, we have been remapping
the `<->` proximity operator in a tsquery to `&`. However, there is
another variant of it which follows the `<N>` pattern. For example, the
following text "end-to-end" will eventually result in the following
tsquery `end-to-end:* <-> end:* <2> end:*` being generated by Postgres.
Before this fix, the tsquery is remapped to `end-to-end:* & end:* <2>
end:*` by us. This is requires the search data which we store to contain
`end` at exactly 2 position apart. Due to the way we limit the
number of duplicates in our search data, the search term may end up not
matching anything. In bd32912c5e, we made
it such that we do not allow any duplicates when indexing a topic's
title. Therefore, search for `end-to-end` against a topic title with
`end-to-end` will never match because our index will only contain one
`end` term.
What does this change do?
We will remap the `<N>` variant of the proximity operator.
This change removes the regex we used previously, which only allowed ASCII characters in fast-edit. Now multi-language content can be used with fast-edit.
It also removes the string replacement we relied on in the past to catch various forms of punctuation marks, as this no longer appears necessary (possibly since this component was updated to use Glimmer).
Why this change?
In workflow runs, we have seen processes being stuck on a flock lock and
I'm guessing because we are using `"w"` when opening the file which the
ruby documentation advises against as it states "don't use "w" because it truncates the file before lock."
Stuck workflow run: https://github.com/discourse/discourse/actions/runs/7690278010/job/20953851469
We were having a minor issue with emails with embedded images
that had newlines in the alt string; for example:
```
<p class="MsoNormal"><span style="font-size:11.0pt"><img width="898"
height="498" style="width:9.3541in;height:5.1875in" id="Picture_x0020_5"
src="cid:image003.png@01DA4EBA.0400B610" alt="A screenshot of a computer
program
Description automatically generated"></span><span
style="font-size:11.0pt"><o:p></o:p></span></p>
```
Once this was parsed and converted to markdown (or directly to HTML
in some cases), this caused an issue in the composer and the post
UI, where the markdown parser didn't know how to deal with this,
making the HTML show directly instead of showing an image.
The easiest way to deal with this is to just strip \n from image
alt and title attrs in the HTMLToMarkdown class.
This patch allows running system specs on an aarch64 Linux system
(typically our `discourse_dev` docker image).
As Chrome isn’t available for the aarch64 architecture (yet), we have to
rely on Firefox instead. This has some drawbacks like not being able to
access the browser logs like we do with the Chrome webdriver.
These routes were previously rendered using Rails, and had a fairly fragile 2fa implementation in vanilla-js. This commit refactors the routes to be handled in the Ember app, removes the custom vanilla-js bundles, and leans on our centralized 2fa implementation. It also introduces a set of system specs for the behavior.
Internal links always notify and add internal connections in topics.
This adds a special feature that lets you append `?silent=true` to a link
to have it excluded from:
1. Notifications - users will not be notified for these links
2. Post links below posts in the UI
This is specifically useful for large reports where adding all these connections
just results in noise.
In a handful of situations, we need to verify a user's 2fa credentials before `current_user` is assigned. For example: login, email_login and change-email confirmation. This commit adds an explicit `target_user:` parameter to the centralized 2fa system so that it can be used for those situations.
For safety and clarity, this new parameter only works for anon. If some user is logged in, and target_user is set to a different user, an exception will be raised.
For performance reasons we don't automatically add fabricated users to trust level auto-groups. However, when explicitly passing a trust level to the fabricator, in 99% of cases it means that trust level is relevant for the test, and we need the groups.
This change makes it so that when a trust level is explicitly passed to the fabricator, the auto-groups are refreshed. There's no longer a need to also pass refresh_auto_groups: true, which means clearer tests, fewer mistakes, and less confusion.
When exporting a csv file and the size of the file exceeded the
max_export_file_size_kb it will still send the PM that the export
succeeded with a broken link to a missing export file. This change
ensures that a failed message will be sent instead.
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_tag_topics site setting to tag_topic_allowed_groups.
We have all these calls to Group.refresh_automatic_groups! littered throughout the tests. Including tests that are seemingly unrelated to groups. This is because automatic group memberships aren't fabricated when making a vanilla user. There are two places where you'd want to use this:
You have fabricated a user that needs a certain trust level (which is now based on group membership.)
You need the system user to have a certain trust level.
In the first case, we can pass refresh_auto_groups: true to the fabricator instead. This is a more lightweight operation that only considers a single user, instead of all users in all groups.
The second case is no longer a thing after #25400.
We want to exclude the system user from group user counts, since intuitively admins wouldn't include them.
Originally this was accomplished by booting said system user from the groups, but this is causing problems, because the system user needs TL group membership to perform certain tasks.
After this PR, system user is still in the TL groups, but excluded when refreshing the user count.
Previously, it was not possible to modify the sorting order of the `TopicQuery` result from a plugin. This feature adds support to specify custom sorting functionality in a plugin. We're using the `apply_modifier` method in the `DiscoursePluginRegistry` module to achieve it.
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Some preparatory refactoring as we're working on TL groups for the system user. On User we have a scope #human_users to exclude the system user, DiscoBot, etc. This PR adds the same scope (delegated to User) on Group.
Why this change?
We have been seeing specs time out on GitHub CI but the problem is that
we are unable to debug those timeouts due to a lack of information. This
change seeks to print out the backtraces of all threads right before a
spec times out on CI.
What does this change do?
1. Starts a thread on CI which will wait for a spec to start running.
1. Once a spec starts running, the thread will sleep for
`PER_SPEC_TIMEOUT_SECONDS -1` seconds.
1. After sleeping, the thread checks if the spec is still running and
prints out the backtraces of all threads if it is. Otherwise, the
thread does nothing and runs the next loop.
1. At the end of each spec run, we ensure that the thread is in a
waiting state for the next spec run to start.
Note that there is no need for us to teardown or cleanup the thread
since the process terminates after running all the tests.
This is a temporary fix to address an issue where the
system user is losing its automatic groups when the server
is running. If any auto groups are provided, and the user is
a system user, then we return true. The system user is admin,
moderator, and TL4, so they usually have all auto groups.
We can remove this when we get to the bottom of why the auto
groups are being deleted.
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_for_user_api_key site setting to user_api_key_allowed_groups.
This isn't used by any of our plugins or themes, so very little fallout.
If configuring only moderators in a group based access setting, the mapping to the old setting wouldn't work correctly, because the case was unaccounted for.
This PR accounts for moderators group when doing the mapping.
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_post_links site setting to post_links_allowed_groups.
This isn't used by any of our plugins or themes, so very little fallout.
The old strategy used to load 25 categories at a time, including the
subcategories. The new strategy loads 20 parent categories and at most
5 subcategories for each parent category, for a maximum of 120
categories in total.
- Decrease gravity, we come in too hot prioritizing too many new topics
- Remove all muted topics / categories and tags from the hot list
- Punish topics with zero likes in algorithm
Why this change?
We have been looking into a flaky system tests in one of our plugins
where the DB transaction flow can be messed up from time to time. Our
debugging effort is complicated by that fact that `test-prof` starts a
DB transaction in a `before(:all)` block which makes it hard to properly
log information. By allowing test-prof to be disabled completely, we
believe it will make it easier for us to isolate the problem we are
investigating.
What does this change do?
1. Avoid loading test-prof files if `PREFABRICATION` env has been set to
`0`.
2. Set `PREFABRICATION=0` for plugin system tests in Github actions
This introduces a new experimental hot sort ordering.
It attempts to float top conversations by first prioritizing a topics with lots of recent activity (likes and users responding)
The schedule that updates hot topics is disabled unless the hidden site setting: `experimental_hot_topics` is enabled.
You can control "decay" with `hot_topic_gravity` and `recency` with `hot_topics_recent_days`
Data is stored in the new `topic_hot_scores` table and you can check it out on the `/hot` route once
enabled.
---------
Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
Why this change?
Currently, is it hard to iteratively write a theme settings migrations
because our theme migrations system does not rollback. Therefore, we
want to allow theme developers to be able to write QUnit tests for their
theme migrations files enabling them to iteratively write their theme
migrations.
What does this change do?
1. Update `Theme#baked_js_tests_with_digest` to include all `ThemeField`
records of `ThemeField#target` equal to `migrations`. Note that we do
not include the `settings` and `themePrefix` variables for migration files.
2. Don't minify JavaScript test files becasue it makes debugging in
development hard.
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_tag_topics site setting to tag_topic_allowed_groups.
This commit makes it so the admin sidebar (when enabled)
will hide the other forum sidebar sections on mobile, the
same way it does on desktop. It was not happening automatically
because the sidebar component is also inside the hamburger-dropdown
component, which is used on mobile.
* UX: add sorting params to groups table plugin outlet
* FEATURE: allow sorting group members by custom field via API
---------
Co-authored-by: Jean Perez <jmperez127@gmail.com>
* FIX: respect creation date when paginating group activity posts
There are scenarios where the chronological order of posts doesn't match the order of their IDs. For instance, when moving the first post from one topic or PM to another, a new post (with a higher ID) will be created, but it will retain the original creation time.
This PR changes the group activity page and endpoint to paginate posts using created_at instead of relying on ID ordering.
Why this change?
Importing theme with the `bundle` params is used mainly by
`discourse_theme` CLI in the development environment. However, we do not
want migrations to automatically run in the development environment
and instead want the developer to be intentional about running theme
migrations. As such, this commit adds support for a
`skip_migrations` param when importing a theme with the `bundle` params.
This commit also adds a `migrated` attribute for migrations theme fields
to indicate whether a migrations theme field has been migrated or not.
When navigating straight to a topic the category was not displayed at
all because the categories were not loaded. Similarly, the categories
for suggested topics were not loaded either.
This commit adds a list of categories to topic view model class and
serializer.
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_to_send_email_messages site setting to send_email_messages_allowed_groups.
Merges the design experiment at
https://meta.discourse.org/t/post-quote-copy-to-clipboard-button-feedback/285376
into core.
This adds a new button by default to the menu that pops up when text is
selected in a post.
The normal Quote button that is shown when selecting text within a post
will open the composer with the quote markdown prefilled.
This new "Copy Quote" button copies the quote markdown directly to the
user’s clipboard. This is useful for when you want to copy the quote
elsewhere – to another topic or a chat message for instance – without
having to manually copy from the opened composer, which then has to be
dismissed afterwards. An example of quote markdown:
```
[quote="someuser, post:7, topic:285376"]
In this moment, I am euphoric.
[/quote]
```
Some plugins have names (e.g. discourse-x-yz) that
are totally different from what they are actually called,
and that causes issues when showing them in a sorted way
in the admin plugin list.
Now, we should use the setting category name from client.en.yml
if it exists, otherwise fall back to the name, for sorting.
This is what we do on the client to determine what text to
show for the plugin name as well.
* FEATURE: Cache embed contents in the database
This will be useful for features that rely on the semantic content of topics, like the many AI features
Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
We're changing the implementation of trust levels to use groups. Part of this is to have site settings that reference trust levels use groups instead. It converts the min_trust_level_to_create_tag site setting to create_tag_allowed_groups.
This PR maintains backwards compatibility until we can update plugins and themes using this.
What we have now:
```
~~~~~~~ SERVER EXCEPTIONS ~~~~~~~Error encountered while proccessing /tag/tag24/l/latest ArgumentError: wrong number of arguments (given 1, expected 0) /__w/discourse/discourse/lib/site_setting_extension.rb:521:in `block in setup_methods'
```
What we actually want:
```
~~~~~~~ SERVER EXCEPTIONS ~~~~~~~
Error encountered while proccessing /tag/tag24/l/latest ArgumentError: wrong number of arguments (given 1, expected 0) /__w/discourse/discourse/lib/site_setting_extension.rb:521:in `block in setup_methods'
```
Alan Guo Xiang Tan
David Taylor
Arpit Jalan
Ted Johansson
Osama Sayegh
Blake Erickson
Penar Musaraj
Isaac Janzen
Bianca Nenciu
Krzysztof Kotlarek
Sérgio Saquetim
Kris
Loïc Guitaut
Martin Brennan
Kelv
Régis Hanol
Mark VanLandingham
dependabot[bot]
Jarek Radosz
Joffrey JAFFEUX
Natalie Tay
Andrei Prigorshnev
Vinoth Kannan
Sam
Keegan George
Daniel Waterworth
David Battersby
Gerhard Schlager
Neil Lalonde
marstall
Leonardo Mosquera
Roman Rizzi
Jean
Renato Atilio
Kyle
Rafael dos Santos Silva