Commit Graph

9871 Commits

Author SHA1 Message Date
Penar Musaraj 71c74a262d
FEATURE: Include participants in PN search data (#16855)
This makes it easier to find PMs involving a particular user, for
example by searching for `in:messages thisUser` (previously, that query
would only return results in posts where `thisUser` was in the post body).
2022-05-18 10:34:01 -04:00
Daniel Waterworth 6e53f4d913
DEV: New readonly mode. Only applies to non-staff (#16243) 2022-05-17 13:06:08 -05:00
Bianca Nenciu 985afe1092
FEATURE: Add page title to 404 pages (#16846)
The title had to be added both on the 404 page generated by the server
side, displayed when the user reaches a bad page directly and the 404
page rendered by Ember when a user reaches a missing topic while
navigating the forum.
2022-05-17 18:37:43 +03:00
Loïc Guitaut 73de203843 FIX: Apply 'hide email account' for invites 2022-05-17 09:56:06 +02:00
David Taylor 38216f6f0b
DEV: Make user field validation more specific (#16746)
- Only validate if custom_fields are loaded, so that we don't trigger a db query
- Only validate public user fields, not all custom_fields

This commit also reverts the unrelated spec changes in ba148e08, which were required to work around these issues
2022-05-16 14:21:33 +01:00
Daniel Waterworth 66a04c5cfe
FIX: Prevent all kinds of login in readonly mode (#16743) 2022-05-13 10:52:01 -05:00
Alan Guo Xiang Tan de9fe907ee
DEV: Readonly Redis support for `DiscourseRedis#multi/pipelined` (#16744)
Follow-up to 2df3c65ba9
2022-05-13 16:18:13 +08:00
Isaac Janzen 20740f196c
FIX: handle quote rendering for external Discourse instance (#16722)
Gracefully handle quotes from an external discourse instance by stripping quote-controls and including username in the title
2022-05-12 10:07:43 -05:00
Alan Guo Xiang Tan fd1dc91eed
DEV: Don't cache watched words in test env (#16731)
The cache was causing state to leak between tests since the `WatchedWord` record in the DB would have been rolled back but `WordWatcher` still had the word in the cache.
2022-05-12 14:45:05 +08:00
Martin Brennan 8e9164fb60
DEV: Minor bookmark tweaks for polymorphism (#16728)
* Make the modal for bookmarks display more consistently
* Make sure bookmark query can handle empty results for certain
  bookmarkable queries
2022-05-12 10:29:01 +10:00
David Taylor 9a5acc5cbc
DEV: Ensure service-worker sourcemap logic works with brotli/gzip (#16718)
The logic in 06893380 only works for `.js` files. It breaks down for `.br.js` and `.gz.js` files. This commit makes things more robust by extracting only the base_url from the service-worker JS, and taking the map filename from the original `sourceMappingURL` comment.
2022-05-11 13:42:34 +01:00
David Taylor 67b23c0e22
PERF: Update all user_histories with one query in UserDestroyer (#16717)
7a284164 previously switched the UserDestroyer to use find_each when iterating over UserHistory records. Unfortunately, since this logic is wrapped in a transaction, this didn't actually solve the memory usage problem. ActiveRecord maintains references to all modified models within a transaction.

This commit updates the logic to use a single SQL query, rather than updating models one-by-one
2022-05-11 13:39:31 +01:00
David Taylor 476bd1d237
DEV: Fix production sourcemaps with Ember CLI (#16707)
22a7905f restructured how we load Ember CLI assets in production. Unfortunately, it also broke sourcemaps for those assets. This commit fixes that regression via a couple of changes:

- It adds the necessary `.map` paths to `config.assets.precompile`
- It swaps Sprockets' default `SourcemappingUrlProcessor` with an extended version which maintains relative URLs of maps
2022-05-11 10:23:32 +01:00
David Taylor 0689338060
DEV: Ensure service-worker sourcemap URLs do not duplicate extension (#16704)
`script_asset_path('.../blah.js.map')` was appending `.js`, which would result in a filename like `.js.map.js`. It would also lose the `/assets` prefix, since the map files are not included in the sprockets manifest.

This commit updates the sourceMappingURL rewriting logic to calculate the service-worker's own JS url, and then append `.map`.
2022-05-11 09:51:06 +01:00
Arpit Jalan 977da1da77
FIX: do not log category custom fields changes if the value is unchanged (#16712) 2022-05-11 13:51:51 +05:30
Martin Brennan a6be4972a8
FIX: Use our header value instead of custom header on duplicates (#16711)
When we build and send emails using MessageBuilder and Email::Sender
we add custom headers defined in SiteSetting.email_custom_headers.
However this was causing errors in cases where the custom headers
defined a header that we already specify in outbound emails (e.g.
the Precedence: list header for topic/post emails).

This commit makes it so we always use the header value defined in Discourse
core if there is a duplicate, discarding the custom header value
from the site setting.

cf. https://meta.discourse.org/t/email-notifications-fail-if-duplicate-headers-exist/222960/14
2022-05-11 13:47:12 +10:00
Martin Brennan 4037cdb6db
FIX: Allow .ics for polymorphic bookmarks (#16694)
We have a .ics endpoint for user bookmarks, this
commit makes it so polymorphic bookmarks work on
that endpoint, using the serializer associated with
the RegisteredBookmarkable.
2022-05-11 09:29:24 +10:00
Bianca Nenciu 4760cf604c
FEATURE: Allow category moderators to post consecutively (#16706)
First posters and staff are already allowed to have unlimited
consecutive posts. This adds the same capabilities to category
moderators.
2022-05-10 22:18:17 +03:00
Isaac Janzen 1a12e4cfc8
FEATURE: Introduce site setting to allow for non staff pm tagging (#16671)
Currently the only way to allow tagging on pms is to use the `allow_staff_to_tag_pms` site setting.  We are removing that site setting and replacing it with `pm_tags_allowed_for_groups` which will allow for non staff tagging. It will be group based permissions instead of requiring the user to be staff.

If the existing value of `allow_staff_to_tag_pms` is `true` then we include the `staff` groups as a default for `pm_tags_allowed_for_groups`.
2022-05-10 10:02:28 -05:00
Loïc Guitaut ba148e082d FIX: Apply watched words to user fields
Currently we don’t apply watched words to custom user fields nor user
profile fields.
This led to users being able to use blocked words in their bio, location
or some custom user fields.

This patch addresses this issue by adding some validations so it’s not
possible anymore to save the User model or the UserProfile model if they
contain blocked words.
2022-05-10 11:37:52 +02:00
Martin Brennan 3b3c505f3c
FIX: Polymorphic bookmarks for bookmark report (#16693)
This allows the category_id filter for the bookmark
report to work with polymorphic bookmarks. Honestly this
is a little hardcode-y at the moment but until we go and
make this report a lot more flexible with more filters
I don't think it's worth the work to add extra interfaces
to RegisteredBookmarkable and BaseBookmarkable to make
this more flexible. This is enough for now.
2022-05-10 11:14:59 +10:00
Martin Brennan 244836ddd4
FIX: Use hidden site setting for batch presign rate limit (#16692)
This was causing issues on some sites, having the const, because this really is heavily
dependent on upload speed. We request 5-10 URLs at a time with this endpoint; for
a 1.5GB upload with 5mb parts this could mean 60 requests to the server to get all
the part URLs. If the user's upload speed is super fast they may request all 60
batches in a minute, if it is slow they may request 5 batches in a minute.

The other external upload endpoints are not hit as often, so they can stay as constant
values for now. This commit also increases the default to 20 requests/minute.
2022-05-10 11:14:26 +10:00
Martin Brennan fbcc35b417
DEV: Remove PostAction/UserAction bookmark refs (#16681)
We have not used anything related to bookmarks for PostAction
or UserAction records since 2020, bookmarks are their own thing
now. Deleting all this is just cleaning up old cruft.
2022-05-10 10:42:18 +10:00
Martin Brennan 955d47bbd0
FIX: Use polymorphic bookmarks for in:bookmarks search (#16684)
This commit makes sure the in:bookmarks post advanced
search filter works with polymorphic bookmarks.
2022-05-10 09:08:01 +10:00
Sam 2df3c65ba9
FIX: add support for pipelined and multi redis commands (#16682)
Latest redis interoduces a block form of multi / pipelined, this was incorrectly
passed through and not namespaced.

Fix also updates logster, we held off on upgrading it due to missing functions
2022-05-10 08:19:02 +10:00
Rafael dos Santos Silva 919f71537e
FIX: Background like count update didn't account for own user actions (#16688)
This fixes a corner case of the perf optimization in d4e35f5.

When you have the the same post showing in multiple tab/devices and like
said post in one place, we updated the like count but didn't flip the
`acted` bool in the front-end. This caused a small visual desync.

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
2022-05-09 17:23:39 -03:00
Roman Rizzi 6bdcd7afb2
FEATURE: Promote the "delete group" staff action log. (#16656)
We used to log group deletion as custom, which means we can't later search for them. Include group ID in the details.
2022-05-09 12:12:52 -03:00
Martin Brennan 222c8d9b6a
FEATURE: Polymorphic bookmarks pt. 3 (reminders, imports, exports, refactors) (#16591)
A bit of a mixed bag, this addresses several edge areas of bookmarks and makes them compatible with polymorphic bookmarks (hidden behind the `use_polymorphic_bookmarks` site setting). The main ones are:

* ExportUserArchive compatibility
* SyncTopicUserBookmarked job compatibility
* Sending different notifications for the bookmark reminders based on the bookmarkable type
* Import scripts compatibility
* BookmarkReminderNotificationHandler compatibility

This PR also refactors the `register_bookmarkable` API so it accepts a class descended from a `BaseBookmarkable` class instead. This was done because we kept having to add more and more lambdas/properties inline and it was very messy, so a factory pattern is cleaner. The classes can be tested independently as well.

Some later PRs will address some other areas like the discourse narrative bot, advanced search, reports, and the .ics endpoint for bookmarks.
2022-05-09 09:37:23 +10:00
Rafael dos Santos Silva 94cfe98ee4
FEATURE: Validate setting combination between exif strip and img opt (#16662)
Admins won't be able to disable strip_image_metadata if they don't
disable composer_media_optimization_image_enabled first since the later
will strip the same metadata on client during upload, making disabling
the former have no effect.

Bug report at https://meta.discourse.org/t/-/223350
2022-05-05 15:13:17 -03:00
Alan Guo Xiang Tan 0328757ffb
FIX: Topic view breaks with topic timer to publish to restricted category. (#16385)
When a user views a topic that contains a topic timer to publish to a
restricted category, an error occurs on the client side because the user
does not have access to information about the category.

This commit fixes it such that the topic timer is not shown to the user
if the user does not have access to the category.
2022-05-05 11:21:01 +08:00
Jarek Radosz 3f0e767106
DEV: Use `FakeLogger` in RequestTracker specs (#16640)
`TestLogger` was responsible for some flaky specs runs:

```
Error during failsafe response: undefined method `debug' for #<TestLogger:0x0000556c4b942cf0 @warnings=1>
Did you mean?  debugger
```

This commit also cleans up other uses of `FakeLogger`
2022-05-05 09:53:54 +08:00
Jarek Radosz fb1a3a1dbb
DEV: Drop `TrackingLogger` for `FakeLogger` (#16642) 2022-05-05 09:50:43 +08:00
Gerhard Schlager 8442a07c13
DEV: Compatibility with TruffleRuby (#16641) 2022-05-05 09:50:02 +08:00
Alan Guo Xiang Tan 8271828948
FIX: Users with unicode usernames unable to load more topics in activity (#16627)
This was due to a server side bug when unicode usernames have been
enabled. We were double encoding the unicode username in the URL
resulting in a invalid URL.
2022-05-05 09:48:22 +08:00
Rafael dos Santos Silva 63f239d0b2
FIX: Add email to admin user list when show_emails is enabled (#16636)
Fixes a regression on be519d2 where this case wasn't accounted for.

Reported at https://meta.discourse.org/t/-/226094
2022-05-04 14:07:22 -03:00
Isaac Janzen 2381f18eba
DEV: Convert notify_about_queued_posts_after to accept a float (#16637)
Add support for `notify_about_queued_posts_after` to be set to a float to allow for 15 min increments
2022-05-04 11:33:43 -05:00
Isaac Janzen dcc7f2a55e
DEV: Convert notify_about_flags_after to float (#16633)
Add support for `notify_about_flags_after` to be set to a float.
2022-05-04 11:19:43 -05:00
Gerhard Schlager 28e8ae553d
FIX: Missing translation when translation override contained a `%{key}` (#16625)
This happened only for languages other than "en" and when `I18n.t` was called without any interpolation keys. The lib still tried to interpolate keys because it interpreted the `overrides` option as interpolation key.
2022-05-04 17:35:22 +02:00
Bianca Nenciu 8695449cfc
FIX: Validate permalink_normalizations setting (#16604)
When an admin enters a badly formed regular expression in the
permalink_normalizations site setting, a RegexpError exception is
generated everytime a URL is normalized (see Permalink.normalize_url).

The new validator validates every regular expression present in the
setting value (delimited by '|').
2022-05-04 14:33:06 +03:00
David Taylor 485fc4636a
FEATURE: Add fallback to suggested value when auth_overrides_username (#16612)
If the identity provider does not provide a precise username value, then we should use our UserNameSuggester to generate one and use it for the override. This makes the override consistent with initial account creation.
2022-05-04 11:22:09 +01:00
David Taylor c1db968740
DEV: Move hotlinked image information into a dedicated table (#16585)
This will make future changes to the 'pull hotlinked images' system easier. This commit should not introduce any functional change.

For now, the old post_custom_field data is kept in the database. This will be dropped in a future commit.
2022-05-03 13:53:32 +01:00
Angus McLeod 9fc3d46003
Update wordpress scopes and add ``session/scopes`` endpoint (#15366)
* Update wordpress scopes && add ``session/scopes`` endpointt

* Fix failing spec

* Add users#show scope to discourse_connect

* Update app/controllers/session_controller.rb

Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>

Co-authored-by: Roman Rizzi <rizziromanalejandro@gmail.com>
2022-05-02 12:15:32 -03:00
David Taylor 0f772bdf5b
FEATURE: Optionally skip using full_name when suggesting usernames (#16592)
This commit introduces a new site setting: `use_name_for_username_suggestions` (default true)

Admins can disable it if they want to stop using Name values when generating usernames for users. This can be useful if you want to keep real names private-by-default or, when used in conjunction with the `use_email_for_username_and_name_suggestions` setting, you would prefer to use email-based username suggestions.
2022-04-29 14:00:13 +01:00
Penar Musaraj b266a36967
FEATURE: Add `group_messages:` keyword to advanced search (#16584) 2022-04-28 10:47:40 -04:00
Loïc Guitaut 008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
Alan Guo Xiang Tan 98c49acad5
DEV: Setup experimental sidebar skeleton (#16575)
* hidden siteSetting to enable experimental sidebar
* user preference to enable experimental sidebar
* `experimental_sidebar_enabled` attribute for current user
* Empty glimmer component for Sidebar
2022-04-28 15:27:06 +08:00
Penar Musaraj eebce8f80a
FEATURE: Add in:messages search modifier (#16567)
This adds `in:messages` as a synonym for `in:personal` and sets it up as our default nomenclature (`in:personal` will still work).
2022-04-26 16:47:01 -04:00
Roman Rizzi 2a96bca7a1
FIX: Correctly handle the print param on topics#show. (#16555)
The controller incorrectly sets print to true when passing `print=false`, which causes the rate limit to perform.
2022-04-25 16:04:13 -03:00
Roman Rizzi 068e93534c
FIX: Check 2FA is disabled before enabling DiscourseConnect. (#16542)
Both settings are incompatible. We validated that DiscourseConnect is disabled before enabling 2FA but were missing the other way around.
2022-04-25 14:49:36 -03:00
David Taylor 922fbe82da
DEV: Ensure `custom_fields_clean?` returns false when values change (#16552)
We were calling `dup` on the hash and using that to check for changes. However, we were not duplicating the values, so changes to arrays or nested hashes would not be detected.
2022-04-25 17:19:39 +01:00
Gerhard Schlager 3d62e335f7
FEATURE: Detect emoji from Emoji 14.0 (#16531) 2022-04-22 08:42:15 +02:00
Martin Brennan 3e4621c2cb
FEATURE: Polymorphic bookmarks pt. 2 (lists, search) (#16335)
This pull request follows on from https://github.com/discourse/discourse/pull/16308. This one does the following:

* Changes `BookmarkQuery` to allow for querying more than just Post and Topic bookmarkables
* Introduces a `Bookmark.register_bookmarkable` method which requires a model, serializer, fields and preload includes for searching. These registered `Bookmarkable` types are then used when validating new bookmarks, and also when determining which serializer to use for the bookmark list. The `Post` and `Topic` bookmarkables are registered by default.
* Adds new specific types for Post and Topic bookmark serializers along with preloading of associations in `UserBookmarkList`
* Changes to the user bookmark list template to allow for more generic bookmarkable types alongside the Post and Topic ones which need to display in a particular way

All of these changes are gated behind the `use_polymorphic_bookmarks` site setting, apart from the .hbs changes where I have updated the original `UserBookmarkSerializer` with some stub methods.

Following this PR will be several plugin PRs (for assign, chat, encrypt) that will register their own bookmarkable types or otherwise alter the bookmark serializers in their own way, also gated behind `use_polymorphic_bookmarks`.

This commit also removes `BookmarkQuery.preloaded_custom_fields` and the functionality surrounding it. It was added in 0cd502a558 but only used by one plugin (discourse-assign) where it has since been removed, and is now used by no plugins. We don't need it anymore.
2022-04-22 08:23:42 +10:00
David Taylor fc56bd36c9
DEV: Ensure a broken tag_group relation doesn't raise an error (#16529)
A category_required_tag_group should always have an associated tag_group. However, this is only enforced at the application layer, so it's technically possible for the database to include a category_required_tag_group without a matching tag_group.

Previously that situation would cause the whole site to go offline. With this change, it will cause some unexpected behavior, but the site serializer will not raise an error.
2022-04-21 18:18:35 +01:00
David Taylor 22a7905f2d
DEV: Allow Ember CLI assets to be used by development Rails app (#16511)
Previously, accessing the Rails app directly in development mode would give you assets from our 'legacy' Ember asset pipeline. The only way to run with Ember CLI assets was to run ember-cli as a proxy. This was quite limiting when working on things which are bypassed when using the ember-cli proxy (e.g. changes to `application.html.erb`). Also, since `ember-auto-import` introduced chunking, visiting `/theme-qunit` under Ember CLI was failing to include all necessary chunks.

This commit teaches Sprockets about our Ember CLI assets so that they can be used in development mode, and are automatically collected up under `/public/assets` during `assets:precompile`. As a bonus, this allows us to remove all the custom manifest modification from `assets:precompile`.

The key changes are:
- Introduce a shared `EmberCli.enabled?` helper
- When ember-cli is enabled, add ember-cli `/dist/assets` as the top-priority Rails asset directory
- Have ember-cli output a `chunks.json` manifest, and teach `preload_script` to read it and append the correct chunks to their associated `afterFile`
- Remove most custom ember-cli logic from the `assets:precompile` step. Instead, rely on Rails to take care of pulling the 'precompiled' assets into the `public/assets` directory. Move the 'renaming' logic to runtime, so it can be used in development mode as well.
- Remove fingerprinting from `ember-cli-build`, and allow Rails to take care of things

Long-term, we may want to replace Sprockets with the lighter-weight Propshaft. The changes made in this commit have been made with that long-term goal in mind.

tldr: when you visit the rails app directly, you'll now be served the current ember-cli assets. To keep these up-to-date make sure either `ember serve`, or `ember build --watch` is running. If you really want to load the old non-ember-cli assets, then you should start the server with `EMBER_CLI_PROD_ASSETS=0`. (the legacy asset pipeline will be removed very soon)
2022-04-21 16:26:34 +01:00
David Taylor e5fb884695
FEATURE: Show prompt for required tag groups (#16458) 2022-04-21 13:13:52 +01:00
Gerhard Schlager 1a56ce3674 FEATURE: Site setting to cap the recipient list in notification emails
* Adds a hidden site setting: `max_participant_names`
* Replaces duplicate code in `GroupSmtpMailer` and `UserNotifications`
* Groups are sorted by the number of users (decreasing)
* Replaces the query to count users of each group with `Group#user_count`)
* Users are sorted by their last reply in the topic (most recent first)
* Adds lots of tests
2022-04-21 10:43:13 +02:00
David Taylor 1551eaab01
FIX: Do not error when json-serialized cookies are used (#16522)
We intend to switch to the `:json` serializer, which will stringify all keys. However, we need a clean revert path. This commit ensures that our `_t` cookie handling works with both marshal (the current default) and json (the new default) serialization.
2022-04-20 19:15:40 +01:00
Isaac Janzen 692e0140e2
FEATURE: Enables support for dark mode emails (#16520)
This PR enables custom email dark mode styles by default that were added here.

There is currently poor support for dark mode queries in mail clients. The main beneficiary of these changes will be Apple Mail and Outlook.

Enjoy the darkness 🕶️
2022-04-20 13:00:04 -05:00
Isaac Janzen 196b791365
DEV: Prioritize full name when display_name_on_posts active (#16078)
See: https://meta.discourse.org/t/display-full-name-not-username-when-attributing-quote-or-reply/203533?u=isaacjanzen for context

The initial release [broke quoting](https://meta.discourse.org/t/quoting-broken-when-name-matches-username/217633?u=isaacjanzen) but we now pass the username when 
```
siteSettings.display_name_on_posts && !siteSettings.prioritize_username_in_ux && post.name
```
as well as the full name to guarantee that we are not getting any mismatches when querying for user / avatar.

eg. 
```
[quote="Isaac Janzen, post:3, topic:7, full:true, username:isaac.janzen"]
bing bong
[/quote]
```
2022-04-20 10:07:51 -05:00
Alan Guo Xiang Tan 935838adb6 DEV: Upgrade Sidekiq to 6.4.2 2022-04-20 16:26:38 +08:00
Martin Brennan 154afa60eb
FIX: Skip upload extension validation when changing security (#16498)
When changing upload security using `Upload#update_secure_status`,
we may not have the context of how an upload is being created, because
this code path can be run through scheduled jobs. When calling
update_secure_status, the normal ActiveRecord validations are run,
and ours include validating extensions. In some cases the upload
is created in an automated way, such as user export zips, and the
security is applied later, with the extension prohibited from
use when normally uploading.

This caused the upload to fail validation on `update_secure_status`,
causing the security change to silently fail. This fixes the issue
by skipping the file extension validation when the upload security
is being changed.
2022-04-20 14:11:39 +10:00
Rafael dos Santos Silva 9d5241d347
FEATURE: Block indexing the embed topic list (#16495)
This adds a robots tag header to disallow indexing the topic list that
powers our embed topic list feature. It also uses a new tag that allows
the content to be indexed in the parent page.

See https://developers.google.com/search/blog/2022/01/robots-meta-tag-indexifembedded

https://meta.discourse.org/t/-/125911/127
2022-04-19 18:24:38 -03:00
Bianca Nenciu 056c7a3f30
DEV: Add test for web hooks and topic tags changes (#16493) 2022-04-19 20:48:32 +03:00
Daniel Waterworth f55edd54fd
FIX: Don't allow DiscourseConnect logins in readonly mode (#16508) 2022-04-19 12:33:31 -05:00
Roman Rizzi 48417b59d6
FIX: Return a 404 when a sitemap request doesn't have a format (#16506) 2022-04-19 11:07:25 -03:00
David Taylor 137e06a316
FIX: Ensure 'crop' always returns requested dimensions (#16437)
Previously, 'crop' would resize the image to have the requested width, then crop the height to the requested value. This works when cropping images vertically, but not when cropping them horizontally.

For example, trying to crop a 500x500 image to 200x500 was actually resulting in a 200x200 image. Having an OptimizedImage with width/height columns mismatching the actual OptimizedImage width/height causes some unusual issues.

This commit ensures that a call to `OptimizedImage.crop(from, to, width, height)` will always return an image of the requested width/height. The `w x h^` syntax defines minimum width/height, while maintaining aspect ratio.
2022-04-19 11:21:24 +01:00
Martin Brennan c6c633e041
FIX: Issues with incorrect unread and private message topic tracking state (#16474)
This commit fixes two issues at play. The first was introduced
in f6c852b (or maybe not introduced
but rather revealed). When a user posted a new message in a topic,
they received the unread topic tracking state MessageBus message,
and the Unread (X) indicator was incremented by one, because with the
aforementioned perf commit we "guess" the correct last read post
for the user, because we no longer calculate individual users' read
status there. This meant that every time a user posted in a topic
they tracked, the unread indicator was incremented. To get around
this, we can just exclude the user who created the post from the
target users of the unread state message.

The second issue was related to the private message topic tracking
state, and was somewhat similar. Whenever a user created a new private
message, the New (X) indicator was incremented, and could not be
cleared until the page was refreshed. To solve this, we just don't
update the topic state for the user when the new_topic tracking state
message comes through if the user who created the topic is the
same as the current user.

cf. https://meta.discourse.org/t/bottom-of-topic-shows-there-is-1-unread-remaining-when-there-are-actually-0-unread-topics-remaining/220817
2022-04-19 11:37:01 +10:00
Mark VanLandingham 1e8a666003
DEV: Accept `force_respect_seen_recently` argument in UserEmail job (#16460) 2022-04-18 13:32:11 -05:00
Bianca Nenciu 234cf65e39
FIX: Do not show visibility topic if visible (#16478)
It used to show the warning that said only members of certain groups
could view the topic even if the group "everyone" was listed in
category's permission list.
2022-04-18 11:16:30 +03:00
David Taylor 8f786268be
SECURITY: Ensure user-agent-based responses are cached separately (#16475) 2022-04-14 14:25:52 +01:00
Gerhard Schlager b7230d14a3 REFACTOR: Add `full_url` to `Group`
Lets stop writing the same code over and over again.
2022-04-14 11:53:57 +02:00
Vinoth Kannan c863244382
FEATURE: add `user_suspended` attribute in post serialize. (#16413)
This PR will include `suspended` attribute in post serializer to check it in post widget and add a CSS class name.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2022-04-13 19:58:09 +05:30
Osama Sayegh eb5a3cfded
FEATURE: Add 2FA support to the Discourse Connect Provider protocol (#16386)
Discourse has the Discourse Connect Provider protocol that makes it possible to
use a Discourse instance as an identity provider for external sites. As a
natural extension to this protocol, this PR adds a new feature that makes it
possible to use Discourse as a 2FA provider as well as an identity provider.

The rationale for this change is that it's very difficult to implement 2FA
support in a website and if you have multiple websites that need to have 2FA,
it's unrealistic to build and maintain a separate 2FA implementation for each
one. But with this change, you can piggyback on Discourse to take care of all
the 2FA details for you for as many sites as you wish.

To use Discourse as a 2FA provider, you'll need to follow this guide:
https://meta.discourse.org/t/-/32974. It walks you through what you need to
implement on your end/site and how to configure your Discourse instance. Once
you're done, there is only one additional thing you need to do which is to
include `require_2fa=true` in the payload that you send to Discourse.

When Discourse sees `require_2fa=true`, it'll prompt the user to confirm their
2FA using whatever methods they've enabled (TOTP or security keys), and once
they confirm they'll be redirected back to the return URL you've configured and
the payload will contain `confirmed_2fa=true`. If the user has no 2FA methods
enabled however, the payload will not contain `confirmed_2fa`, but it will
contain `no_2fa_methods=true`.

You'll need to be careful to re-run all the security checks and ensure the user
can still access the resource on your site after they return from Discourse.
This is very important because there's nothing that guarantees the user that
will come back from Discourse after they confirm 2FA is the same user that
you've redirected to Discourse.

Internal ticket: t62183.
2022-04-13 15:04:09 +03:00
Bianca Nenciu 86c7e07428
FEATURE: Improve screened IPs roll up and extend for IPv6 (#15585)
This commit improves the logic for rolling up IPv4 screened IP
addresses and extending it for IPv6. IPv4 addresses will roll up only
up to /24. IPv6 can rollup to /48 at most. The log message that is
generated contains the list of original IPs and new subnet.
2022-04-12 21:07:37 +03:00
Roman Rizzi 6f76a12e0a
FEATURE: Let sites add a sitemap.xml file. (#16357)
* FEATURE: Let sites add a sitemap.xml file.

This PR adds the same features discourse-sitemap provides to core. Sitemaps are only added to the robots.txt file if the `enable_sitemap` setting is enabled and `login_required` disabled.

After merging discourse/discourse-sitemap#34, this change will take priority over the sitemap plugin because it will disable itself. We're also using the same sitemaps table, so our migration won't try to create it
again using `if_not_exists: true`.
2022-04-12 10:33:59 -03:00
Martin Brennan 9f2138dc92
FEATURE: Add a sidekiq job for syncing S3 ACLs (#16449)
Sometimes we need to update a _lot_ of ACLs on S3 (such as when secure media
is enabled), and since it takes ~1s per upload to update the ACL, this is
best spread out over many jobs instead of having to do the whole thing serially.

In future, it will be better to have a job that can be run based on
a column on uploads (e.g. acl_stale) so we can track progress, similar
to how we can set the baked_version to nil to rebake posts.
2022-04-12 14:26:42 +10:00
Blake Erickson 4c0a9fd4c3
FIX: Email logs not finding attached incoming email (#16446) 2022-04-11 21:22:15 -06:00
Isaac Janzen 6c0abe15e0
FEATURE: Add email dark mode (#16104)
implement dark mode emails when `SiteSetting.dark_mode_emails_active` is active.
2022-04-11 12:27:50 -05:00
Alan Guo Xiang Tan 0f7b9878ff SECURITY: Category group permissions leaked to normal users.
After this commit, category group permissions can only be seen by users
that are allowed to manage a category. In the past, we inadvertently
included a category's group permissions settings in `CategoriesController#show`
and `CategoriesController#find_by_slug` endpoints for normal users when
those settings are only a concern to users that can manage a category.
2022-04-08 13:46:20 +08:00
Bianca Nenciu 07d8189edd FIX: Serialize permissions for everyone group
The permissions for the 'everyone' group were not serialized because
the list of groups a user can view did not include it. This bug was
introduced in commit dfaf9831f7.
2022-04-08 09:25:29 +08:00
Sam cedcdb0057
FEATURE: allow for local theme js assets (#16374)
Due to default CSP web workers instantiated from CDN based assets are still
treated as "same-origin" meaning that we had no way of safely instansiating
a web worker from a theme.

This limits the theme system and adds the arbitrary restriction that WASM
based components can not be safely used.

To resolve this limitation all js assets in about.json are also cached on
local domain.

{
  "name": "Header Icons",
  "assets" : {
    "worker" : "assets/worker.js"
  }
}

This can then be referenced in JS via:

settings.theme_uploads_local.worker

local_js_assets are unconditionally served from the site directly and
bypass the entire CDN, using the pre-existing JavascriptCache

Previous to this change this code was completely dormant on sites which
used s3 based uploads, this reuses the very well tested and cached asset
system on s3 based sites.

Note, when creating local_js_assets it is highly recommended to keep the
assets lean and keep all the heavy working in CDN based assets. For example
wasm files can still live on the CDN but the lean worker that loads it can
live on local.

This change unlocks wasm in theme components, so wasm is now also allowed
in `theme_authorized_extensions`

* more usages of upload.content

* add a specific test for upload.content

* Adjust logic to ensure that after upgrades we still get a cached local js
on save
2022-04-07 07:58:10 +10:00
Jarek Radosz 8e809149d2
DEV: Fix "overridden" typos (#16399)
There are still some, but those are in actual code that's used outside core, so the change there would need to go through the deprecation cycle. That's a task for another day.
2022-04-06 23:17:20 +02:00
Penar Musaraj df10a27067
FIX: Exclude automatic anchors from search index (#16396) 2022-04-06 16:06:45 -04:00
Daniel Waterworth 6e9a068e44
FIX: Limit max word length in search index (#16380)
Long words bloat the index for little benefit.
2022-04-06 12:23:30 -05:00
David Taylor 68c74e9b93
FEATURE: Allow multiple required tag groups for a category (#16381)
Previously we only supported a single 'required tag group' for a category. This commit allows admins to specify multiple required tag groups, each with their own minimum tag count.

A new category_required_tag_groups database table replaces the existing columns on the categories table. Data is automatically migrated.
2022-04-06 14:08:06 +01:00
Sam 254f48e568
FIX: include crawler content on old mobile browsers (#16387)
Previous to this change an optimisation stripped crawler content from
all mobile browsers.

This had a side effect that meant that when we dropped support for an old
mobile platform we would stop rendering topic and topic list pages.

The new implementation ensures we only perform the optimisation on modern
mobile browsers.
2022-04-06 11:09:12 +01:00
Loïc Guitaut 357011eb3b DEV: Clean up freedom patches
This patch removes some of our freedom patches that have been deprecated
for some time now.
Some of them have been updated so we’re not shipping code based on an
old version of Rails.
2022-04-06 10:07:14 +02:00
Sam 1598e6b489
FIX: users watching tags in open tag groups not notified (#16384)
All users are members of the EVERYONE group, but this group is special and
is omitted from the group_users table. When checking permission we need to
make sure we also add a bypass.

This also fixes a very buggy test in post_alerter, it was confirming the
broken behavior due to fabricator flow.

When it defined the tag group the everyone group automatically had full access
then the additional permission fabricated just added one more group. After
fix was made to code the test started failing. Fabricators can be risky.
2022-04-06 11:43:57 +10:00
Jarek Radosz 3c44bed545
DEV: Make DistributedMemoizer use DistributedMutex (#16229)
Its implementation was already distributed-mutex-like, with slight differences that did not seem necessary.
2022-04-05 19:29:58 +02:00
Dan Ungureanu e90815a429
FIX: Redirect user to topic they were invited to (#16298)
This did not work properly everytime because the destination URL was
saved in a cookie and that can be lost for various reasons. This commit
redirects the user to invited topic if it exists.
2022-04-05 14:57:00 +03:00
David Taylor 39ac476db6 FIX: Do not attempt to pull_hotlinked_image for raw_html
raw_html posts (i.e. those which are pulled as part of our comments integration) don't go through our markdown pipeline, so `upload://` URLs are not supported. Running pull_hotlinked_images will break any images in the post.

In future we may add support for pulling hotlinked images in these posts. But for now, disabling it will stop it breaking images.
2022-04-05 16:39:38 +08:00
Martin Brennan b982992ef7
FIX: Auto-generated emails causing group SMTP email storm (#16372)
When emailing a group inbox and including other support-type
emails (or even just regular ones with autoresponders) in the
CC field, each automated reply to the group inbox triggered
more emails to be sent out to all CC addresses to notify them
of the new reply, which in turn caused more automated emails
to be sent to the group inbox.

This commit fixes the issue by preventing any emails being sent
by the PostAlerter when the new post has an incoming email record
which is_auto_generated, which we detect in Email::Receiver.
2022-04-05 13:18:49 +10:00
Blake Erickson ec2930712d
FIX: 500 error when creating a user with an integer username (#16370)
Via the API it is possible to create a user with an integer username. So
123 instead of "123". This causes the following 500 error:

```
NoMethodError (undefined method `unicode_normalize' for 1:Integer)
app/models/user.rb:276:in `normalize_username'
```

See: https://meta.discourse.org/t/222281
2022-04-04 15:15:32 -06:00
Faizaan Gagan 1da4b9eeb3
FIX: update 'posted' column on post owner change (#16367)
Fixes the issue where making a user x as owner of a post doesn't
cause the concerned topic to be listed in new owner's `My Posts`
top menu filter

per https://meta.discourse.org/t/199369
2022-04-04 22:42:38 +05:30
Bianca Nenciu 819038537c
FIX: can_permanently_delete should check for admin (#16348)
can_permanently_delete field in Post and TopicViewDetails serializers
cannot use Guardian's can_permanently_delete beause their use is
different. The field from the serializers is used to show the button
and the button is shown even if the post cannot be removed forever
because not enough time has passed since it was first deleted. The
guardian method is used by the controller to check that the post can
really be deleted.
2022-04-01 12:03:39 +11:00
Sam b023d88b09
FIX: Abort theme creation if unable to create uploads (#16336)
Previous to this change if any of the assets were not allowed extensions
they would simply be silently ignored, this could lead to broken themes
that are very hard to debug
2022-04-01 12:03:14 +11:00
Alan Guo Xiang Tan e7c3d01aaa
DEV: Restore order assertion in category serializer tests. (#16344)
Our group fabrication creates groups with name "my_group_#{n}" where n
is the sequence number of the group being created. However, this can
cause the test to be flaky if and when a group with name `my_group_10`
is created as it will be ordered before
`my_group_9`. This commits makes the group names determinstic to
eliminate any flakiness.

This reverts commit 558bc6b746.
2022-04-01 08:58:06 +08:00
David Taylor 558bc6b746
DEV: Fix flaky specs (#16340)
`group_permissions` are not serialized in a consistent order

Follow-up to dfaf9831f7
2022-03-31 12:06:19 +01:00
Alan Guo Xiang Tan dfaf9831f7
SECURITY: Avoid leaking private group name when viewing category. (#16337)
In certain instances when viewing a category, the name of a group with
restricted visilbity may be revealed to users which do not have the
required permission.
2022-03-31 14:39:01 +08:00
Rafael dos Santos Silva 4d809e984d
FEATURE: DiscourseEvent hook for sync_sso (#16329) 2022-03-30 16:22:22 -03:00
Mark VanLandingham b974375239
DEV: Skip notifications without topic_id in `ensure_consistency` (#16299) 2022-03-30 09:56:35 -05:00
Martin Brennan b8828d4a2d
FEATURE: Polymorphic bookmarks pt. 1 (CRUD) (#16308)
This commit introduces a new use_polymorphic_bookmarks site setting
that is default false and hidden, that will be used to help continuous
development of polymorphic bookmarks. This setting **should not** be
enabled anywhere in production yet, it is purely for local development.

This commit uses the setting to enable create/update/delete actions
for polymorphic bookmarks on the server and client side. The bookmark
interactions on topics/posts are all usable. Listing, searching,
sending bookmark reminders, and other edge cases will be handled
in subsequent PRs.

Comprehensive UI tests will be added in the final PR -- we already
have them for regular bookmarks, so it will just be a matter of
changing them to be for polymorphic bookmarks.
2022-03-30 12:43:11 +10:00
David Taylor ff93833fdf
UX: Use committed date for GitHub oneboxes (#16318)
Our copy says 'committed {date}`, but we were previously using the commit's authored date
2022-03-30 09:16:28 +08:00
Alan Guo Xiang Tan 5743a6ec1e DEV: Remove Zeitwerk inflection monkey patch.
There isn't a good reason we need to patch the inflector.

Co-authored-by: Loïc Guitaut <loic@discourse.org>
2022-03-29 16:04:49 +02:00
David Taylor b2a8dc4c0f
FIX: Maintain HTML `<img` when downloading remote images (#16278)
Under some conditions, replacing an `<img` with `![]()` can break rendering, and make the image disappear.

Context at https://meta.discourse.org/t/152801
2022-03-29 10:55:10 +01:00
Alan Guo Xiang Tan 8e5614b1bf
DEV: Remove hardcoded ID in test fabrication. (#16313)
This hardcoded ID can cause fabrication to fail once we create 999
users across the entire test suite.
2022-03-29 15:23:55 +08:00
Sam 24f327f7ad
DEV: add testing for multi del on keys (#16271)
* DEV: add testing for multi del on keys

Following #15905 we were missing some tests, this covers cases where
del is used in the form of .del(key1,key2)
2022-03-29 09:52:17 +11:00
Osama Sayegh e40c4bb7f9
FIX: Validate category tag restrictions before sending new topics to review (#16292)
Tags (and tag groups) can be configured so that they can only be used in specific categories and (optionally) restrict topics in these categories to be able to add/use only these tags. These restrictions work as expected when a topic is created without going through the review queue; however, if the topic has to be reviewed by a moderator then these restrictions currently aren't checked before the topic is sent to the review queue, but they're checked later when a moderator tries to approve the topic. This is because if a user manages to submit a topic that doesn't meet the restrictions, moderators won't be able to approve and it'll be stuck in the review queue.

This PR prevents topics that don't meet the tags requirements from being sent to the review queue and shows the poster an error message that indicates which tags that cannot be used.

Internal ticket: t60562.
2022-03-28 21:25:26 +03:00
Penar Musaraj b1211bee97
FIX: Resetting selectable avatars was failing (#16302) 2022-03-28 14:15:28 -04:00
David Taylor 720e1ca9e7
FEATURE: Support upload:// urls in img tags (#16277)
Previously, our `upload://` protocol urls were only supported in markdown image tags. This meant that our PullHotlinkedImages job was forced to convert `<img` tags to markdown. Depending on the exact syntax, this can actually cause the image to break.

This commit adds support for `upload://` inside regular HTML `<img` tags. In a future commit, we'll be able to use this to make our PullHotlinkedImages job much more robust.

Context at https://meta.discourse.org/t/152801
2022-03-28 16:46:47 +01:00
Alan Guo Xiang Tan fc40a572bb
DEV: Register question_answer_user_commented notification type. (#16297)
The notification type is used by https://github.com/discourse/discourse-question-answer
2022-03-28 16:03:19 +08:00
Bianca Nenciu 6eb3d658ca
FIX: Do not wrap unaccent around tsqueries (#16284)
tsqueries use quotes and having other characters that when unaccented
become quotes results in invalid tsqueries.
2022-03-25 19:10:05 +02:00
Mark VanLandingham a3563336db
FIX: Bug setting notification level to muted/ignored on user page (#16268) 2022-03-25 10:51:45 -05:00
Daniel Waterworth 9ce6280f51
DEV: Make tests more resilient (#16279)
Since we give a 200 response for login errors, we should be checking
whether the error key exists in each case or not.

Some tests were broken, because they weren't checking.
2022-03-25 10:44:12 -05:00
Dan Ungureanu 03ad88f2c2
FIX: Add `errors` field if group update confirmation (#16260)
* FIX: Redirect if Discourse-Xhr-Redirect is present

`handleRedirect` was passed an wrong argument type (a string) instead of
a jqXHR object and missed the fields checked in condition, thus always
evaluating to `false`.

* FIX: Add `errors` field if group update confirmation

An explicit confirmation about the effect of the group update is
required if the default notification level changes. Previously, if the
confirmation was missing the API endpoint failed silently returning
a 200 response code and a `user_count` field. This change ensures that
a proper error code is returned (422), a descriptive error message and
the additional information in the `user_count` field.

This commit also refactors the API endpoint to use the
`Discourse-Xhr-Redirect` header to redirect the user if the group is
no longer visible.
2022-03-24 14:50:44 +02:00
Martin Brennan 9d5737fd28
SECURITY: Hide private categories in user activity export (#16273)
In some of the user's own activity export data,
we sometimes showed a secure category's name or
exposed the existence of a secure category.
2022-03-24 15:38:44 +10:00
Bianca Nenciu cbaf7c949b
FIX: Make sure max_oneboxes_per_post is enforced (#16215)
PostAnalyzer and CookedPostProcessor both replace URLs with oneboxes.
PostAnalyzer did not use the max_oneboxes_per_post site and setting and
CookedPostProcessor replaced at most max_oneboxes_per_post URLs ignoring
the oneboxes that were replaced already by PostAnalyzer.
2022-03-23 17:36:08 +02:00
Dan Ungureanu 8e9cbe9db4
FIX: Do not raise if title cannot be crawled (#16247)
If the crawled page returned an error, `FinalDestination#safe_get`
yielded `nil` for `uri` and `chunk` arguments. Another problem is that
`get` did not handle the case when `safe_get` failed and did not return
the `location` and `set_cookie` headers.
2022-03-22 20:13:27 +02:00
David Taylor 80dd769530 FIX: Issue 404 for invalid `/tags/c/...` routes
Previously we would issue a 403 for all invalid routes under `/tags/c/...`, which is not semantically correct. In some cases, these 403'd routes would then be handled successfully in the Ember app, leading to some very confusing behavior.
2022-03-22 15:26:00 +00:00
Jarek Radosz be3dceccfa
DEV: Merge two spec files (#16244)
Also reenabled two specs on macOS as they're green now.
2022-03-22 09:23:06 +08:00
Vinoth Kannan 8a9a7a77d8
FIX: staff should not be able to PM groups that "Nobody" can message (#16163)
If a group's messageable_level is set to nobody then staff can't should not be able to send PMs to it.

Co-authored-by: Martin Brennan <martin@discourse.org>
2022-03-22 10:23:14 +10:00
Blake Erickson 61248652cd
DEV: Allow params to be passed on topic redirects (#16218)
* DEV: Allow params to be passed on topic redirects

There are several places where we redirect a url to a standard topic url
like `/t/:slug/:topic_id` but we weren't always passing query parameters
to the new url.

This change allows a few more query params to be included on the
redirect. The new params that are permitted are page, print, and
filter_top_level_replies. Any new params will need to be specified.

This also prevents the odd trailing empty page param that would
sometimes appear on a redirect. `/t/:slug/:id.json?page=`

* rubocop: fix missing space after comma

* fix another page= reference
2022-03-17 19:27:51 -06:00
Rafael dos Santos Silva 100a84d345
DEV: Reserve assigned notification type (#16207)
* DEV: Reserve assigned notification type

* update constants
2022-03-16 17:08:10 -03:00
Jarek Radosz 383f164672
FIX: Always include the first poster when converting to PM (#16187)
Regressed in #15626
2022-03-15 10:51:10 +01:00
Sam de9a031073
FEATURE: use canonical links in posts.rss feed (#16190)
* FEATURE: use canonical links in posts.rss feed

Previously we used non canonical links in posts.rss

These links get crawled frequently by crawlers when discovering new
content forcing crawlers to hop to non canonical pages just to end up
visiting canonical pages

This uses up expensive crawl time and adds load on Discourse sites

Old links were of the form:

`https://DOMAIN/t/SLUG/43/21`

New links are of the form

`https://DOMAIN/t/SLUG/43?page=2#post_21`

This also adds a post_id identified element to crawler view that was
missing.

Note, to avoid very expensive N+1 queries required to figure out the
page a post is on during rss generation, we cache that information.

There is a smart "cache breaker" which ensures worst case scenario is
a "page drift" - meaning we would publicize a post is on page 11 when
it is actually on page 10 due to post deletions. Cache holds for up to
12 hours.

Change only impacts public post RSS feeds (`/posts.rss`)
2022-03-15 20:17:06 +11:00
David Taylor 8664712c1a
PERF: Fix n+1 for categories + featured topics (#16188)
`topic.featured_topic` and `topic.category` are used by `TopicGuardian#can_see_topic?`
2022-03-14 22:23:39 +00:00
Blake Erickson 02fa04e333
FIX: Update topic route id param (#16166)
This update topic route has never worked. Better late than never. I am
in favor of using non-slug urls when using the api so I do think we
should fix this route.

Just thought I would update the `:id` param to `:topic_id` here in the
routes file instead of updating the controller to handle both params.

Added a spec to test this route.

Also added the same constraint we have on other topic routes to ensure
we only pass in an ID that is a digit.
2022-03-11 11:01:08 -07:00
Penar Musaraj 94750c81fa
FIX: Update group inbox notifications on archive/unarchive (#16152) 2022-03-11 11:57:47 +01:00
Osama Sayegh b0656f3ed0
FIX: Apply onebox blocked domain checks on every redirect (#16150)
The `blocked onebox domains` setting lets site owners change what sites
are allowed to be oneboxed. When a link is entered into a post,
Discourse checks the domain of the link against that setting and blocks
the onebox if the domain is blocked. But if there's a chain of
redirects, then only the final destination website is checked against
the site setting.

This commit amends that behavior so that every website in the redirect
chain is checked against the site setting, and if anything is blocked
the original link doesn't onebox at all in the post. The
`Discourse-No-Onebox` header is also checked in every response and the
onebox is blocked if the header is set to "1".

Additionally, Discourse will now include the `Discourse-No-Onebox`
header with every response if the site requires login to access content.
This is done to signal to a Discourse instance that it shouldn't attempt
to onebox other Discourse instances if they're login-only. Non-Discourse
websites can also use include that header if they don't wish to have
Discourse onebox their content.

Internal ticket: t59305.
2022-03-11 09:18:12 +03:00
Sam 43da88db6c
PERF: avoid following links in topic RSS feeds (#16145)
Topic RSS feeds contain many non canonical links such as:

- https://site.com/t/a-b-c/111/1
- https://site.com/t/a-b-c/111/2
- https://site.com/t/a-b-c/111/3
- https://site.com/t/a-b-c/111/4
- https://site.com/t/a-b-c/111/5
- https://site.com/t/a-b-c/111/6

Previously we were not indexing RSS feeds yet still following these
links.


This change means we totally ignore links in the RSS feeds which
avoids expensive work scanning them just to find we should not
include them.
2022-03-09 18:25:20 +11:00
Ayke Halder 28bb9e11f4
FEATURE: add nofollow to RSS alternate link in topics and categories (#16013)
* FEATURE: add nofollow to RSS alternate link in topics and categories

* Rspec tests for category and topic view: add nofollow to RSS alternate link
2022-03-09 16:34:02 +11:00
Martin Brennan ca93e5e68b
FIX: Allow admins to change user ignore list (#16129)
Previously, if an admin user tried to add/remove
users to another user's ignored list, it would
be added to their own ignore list because the
controller used current_user. Now for admins only
a source_user_id parameter can be passed through,
which will be used to ignore the target user for
that source user.
2022-03-09 14:51:30 +10:00
Jarek Radosz bf252752e7
FIX: Make reply-ids public by fixing a typo (#16137)
…and spec the endpoint
2022-03-08 21:08:15 +01:00
Bianca Nenciu 6d422a8033
FEATURE: Highlight expired bookmark reminders (#15317)
The user can select what happens with a bookamrk after it expires. New
option allow bookmark's reminder to be kept even after it has expired.
After a bookmark's reminder notification is created, the reminder date
will be highlighted in red until the user resets the reminder date.
User can do that using the new Clear Reminder button from the dropdown.
2022-03-08 19:44:18 +02:00
Jarek Radosz 14109ea92c
FIX: Show 404 html on /posts/:id/raw and /p/:id (#16131)
It returned a blank page before.
2022-03-08 17:42:07 +01:00
Jarek Radosz 768c80c2a4
DEV: Make `find_post_using` raise only `Discourse::NotFound` (#16133) 2022-03-08 17:39:52 +01:00
Bianca Nenciu 34b4b53bac
FEATURE: Use Postgres unaccent to ignore accents (#16100)
The search_ignore_accents site setting can be used to make the search
indexer remove the accents before indexing the content. The unaccent
function from PostgreSQL is better than Ruby's unicode_normalize(:nfkd).
2022-03-07 23:03:10 +02:00
Martin Brennan 7af01d88d2
FIX: Better 0 file size detection and logging (#16116)
When creating files with create-multipart, if the file
size was somehow zero we were showing a very unhelpful
error message to the user. Now we show a nicer message,
and proactively don't call the API if we know the file
size is 0 bytes in JS, along with extra console logging
to help with debugging.
2022-03-07 12:39:33 +10:00
jbrw fc30669db2
FIX: Support new layout on Amazon product pages (#16091)
Some product pages on Amazon are using a new HTML structure, meaning the previous Onebox engine was unable to gather the price and/or description. This change should allow these pages to be Oneboxed.
2022-03-04 18:31:53 -05:00
Blake Erickson d760fd4074
DEV: Add API scope for categories endpoint (#16105)
This change adds support for the categories endpoint to have an api
scope. Only adds GET scope for listing categories and for fetching a
single category.

See: https://meta.discourse.org/t/218080/4
2022-03-04 14:29:47 -07:00
David Taylor eb2e3b510d
FEATURE: Introduce 'Subcategories with featured topics' view (#16083)
This categories view is designed for sites which make heavy use of subcategories, and use top-level categories mainly for grouping
2022-03-04 21:11:59 +00:00
Osama Sayegh 8c71878ff5
UX: Add description to the 2FA page when adding new admins (#16098)
This PR adds an extra description to the 2FA page when granting a user admin access. It also introduces a general system for adding customized descriptions that can be used by future actions.

(Follow-up to dd6ec65061)
2022-03-04 06:43:06 +03:00
Dan Ungureanu 39ab14531a
FEATURE: API to create user's associated account (#15737)
Discourse users and associated accounts are created or updated when a
user logins or connects the account using their account preferences.
This new API can be used to create associated accounts and users too,
if necessary.
2022-03-03 18:17:02 +02:00
Arpit Jalan 567be512c9
FIX: save email rejection error class names for incoming email logs (#16095) 2022-03-03 18:58:13 +05:30
Joffrey JAFFEUX a558c5bd30
FIX: correctly notifies subscribers with post_action_destroyer (#16084) 2022-03-03 09:49:36 +01:00
Sam 3bf5692c72
FEATURE: prioritize group search order based on prefix match (#16093)
Our @mention user search prioritized users based on prefix matches.

So if searching for `sa` we will display `sam`, `asam` in that order

Previously, we did not prioritize group matches based on prefix. This change ensures better parity.

Implementation notes:

1. User search only prioritizes based on username prefix, not name prefix. TBD if we want to change that.
2. @mention on client side will show 0 group matches if we fill up all the spots with user matches. TBD if we want to unconditionally show the first / second group match.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2022-03-03 16:57:52 +11:00
Dan Ungureanu e3b4998efc
DEV: Remove notify user topic from share modal (#16085)
This feature was rarely used, could be used for spamming users and was
impossible to add a context to why the user was notified of a topic. A
simple private messages that includes the link and personalized message
can be used instead.
2022-03-03 09:27:45 +11:00
Bianca Nenciu b21bf840cb
FEATURE: Return geoname IDs from MaxMindDb (#16082)
The geoname IDs can be used by plugins to locate the user more precisely.
2022-03-02 23:51:42 +02:00
Blake Erickson df2441ee37
Top level replies (#16087)
* DEV: Show only top level replies

Adds a new query param to the topic view so that we can filter out posts
that aren't top level replies. If a post is a reply to another post
instead of the original topic post we should not include it in the
response if the `filter_top_level_replies` query param is present.

* add rspec test
2022-03-02 13:25:36 -07:00
Krzysztof Kotlarek ea3a58d051
FIX: indirectly muted categories for topic-tracking-state (#16067)
Topics belonging to indirectly muted categories should be excluded from topic-tracking-state report.
2022-03-02 15:02:09 +11:00
David Taylor c9dab6fd08
DEV: Automatically require 'rails_helper' in all specs (#16077)
It's very easy to forget to add `require 'rails_helper'` at the top of every core/plugin spec file, and omissions can cause some very confusing/sporadic errors.

By setting this flag in `.rspec`, we can remove the need for `require 'rails_helper'` entirely.
2022-03-01 17:50:50 +00:00
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Jarek Radosz c97b4d530a
DEV: Freeze time in time-sensitive specs (#16062)
Hopefully fixes a flake 🙂
2022-02-28 14:15:13 +08:00
Jarek Radosz 6f6406ea03
DEV: Fix random typos (#16066) 2022-02-28 10:20:58 +08:00
Jarek Radosz 4020738eed
DEV: Fix specs deprecations (#16059) 2022-02-26 03:51:39 +01:00
Roman Rizzi 54ad50eda1
FIX: Respect the cooldown window when editing a flagged topic. (#16046)
When staff decides to hide a flagged post, and it's the first post on the topic, the post owner shouldn't be able to edit either of them until the cooldown finishes. Edit either of them automatically, unhides the post, and makes the topic visible when there's a flag involved.

Reported on meta: https://meta.discourse.org/t/users-can-edit-flagged-topic-title-when-they-should-not-be-able-to/217796
2022-02-25 11:09:31 -03:00
Sam c71afdfdb0
FIX: avoid validations when destroying posts (#16049)
Previously email validations could fire when deleting posts if for
certain reasons any user validations fail on the user objects

This kind of condition could happen in core due to a corruption of a
user record, or via a plugin that introduces a new validation on User
2022-02-25 11:20:54 +11:00
Jeff Wong 85f1ec643d
FIX: Move selectable_avatars_mode_validator_spec to new directory (#16048)
Lib specs moved in 45cc16098d
Move the new selectable_avatars_mode_validator_spec to the new location
Remove the old selectable_avatars_enabled_validator_spec

follow-up of d1bdb6c65d
2022-02-24 13:57:26 -08:00
Jeff Wong d1bdb6c65d
FEATURE: upload an avatar option for uploading avatars with selectable avatars (#15878)
* FEATURE: upload an avatar option for uploading avatars with selectable avatars

Allow staff or users at or above a trust level to upload avatars even when the site
has selectable avatars enabled.

Everyone can still pick from the list of avatars. The option to upload is shown
below the selectable avatar list.

refactored boolean site setting into an enum with the following values:

disabled: No selectable avatars enabled (default)
everyone: Show selectable avatars, and allow everyone to upload custom avatars
tl1: Show selectable avatars, but require tl1+ and staff to upload custom avatars
tl2: Show selectable avatars, but require tl2+ and staff to upload custom avatars
tl3: Show selectable avatars, but require tl3+ and staff to upload custom avatars
tl4: Show selectable avatars, but require tl4 and staff to upload custom avatars
staff: Show selectable avatars, but only allow staff to upload custom avatars
no_one: Show selectable avatars. No users can upload custom avatars

Co-authored-by: Régis Hanol <regis@hanol.fr>
2022-02-24 12:57:39 -08:00
Loïc Guitaut e871865a61 FIX: Sanitize parameters provided to user actions
Currently, providing things like `filter[%24acunetix]=1` to
`UserActionsController#index` will throw an exception because instead of
getting a string as expected, we get a hash instead.

This patch simply uses `#permit` from strong parameters properly: first
we apply it on the whole parameters, this way it filters the keys we’re
interested in. By doing this, if the value is a hash for example, the
whole key/value pair will be ignored completely.
2022-02-23 15:46:40 +01:00
Jarek Radosz cecfc7ce79
FIX: Don't warn on empty .discourse-compatibility (#16032) 2022-02-23 09:44:37 +01:00
Osama Sayegh 586d572e05
FIX: Don't advance draft sequence when editing topic title (#16002)
This commit handles the edge case where a draft is lost with no warnings if the user edits the title (or category/tags) of a topic while they're replying.to the same topic. Repro steps are as follows:

1. Start replying to a topic and type enough to get a draft saved.
2. Scroll up to the topic title and click the pencil icon next to the topic title, change the title, category and/or tags, and then save the changes.
3. Reload the page and you'll see that the draft is gone.

This happens because we only allow 1 draft per topic per user and when you edit the title of a topic that you're replying to, from the server perspective it'll look like as if you've submitted your reply so it will advance the draft sequence for the topic and delete the draft.

The fix in this commit makes `PostRevisor` skip advancing the draft sequence when a topic's title is edited using the pencil button next to the title.

Internal ticket: t60854.

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2022-02-23 10:39:54 +03:00
Martin Brennan 599a72768c
DEV: Add force_quote_link option to PrettyText (#16034)
This option will make it so the [quote] bbcode will always
include the HTML link to the quoted post, even if a topic_id
is not provided in the PrettyText#cook options. This is so
[quote] bbcode can be used in other places, like chat messages,
that always need the link and do not have an "off-topic" ID
to use.
2022-02-23 16:13:46 +10:00
Sam d4d3580761
PERF: perform all cached counting in background (#15991)
Previously cached counting made redis calls in main thread and performed
the flush in main thread.

This could lead to pathological states in extreme heavy load.

This refactor reduces load and cleans up the interface
2022-02-22 16:45:25 +00:00
David Taylor f6c852bf8e
PERF: Use a shared message for replies to tracked topics (#16022)
Previously we were publishing one messagebus message per user which was 'tracking' a topic. On large sites, this can easily be 1000+ messages. The important information in the message is common between all users, so we can manage with a single message on a shared channel, which will be much more efficient.

For user-specific values (notification_level and last_read_post_number), the JS app can infer values which are 'good enough'. Correct values will be loaded as soon as a topic-list containing the topic is visited.
2022-02-22 15:27:46 +00:00
Joffrey JAFFEUX f5ec32bc8c
FEATURE: adds the user_promoted event to webhooks (#15996) 2022-02-22 10:57:18 +01:00
Martin Brennan fa0c796baf
DEV: Fix SMTP bounce regexp (#16019)
Never trust me with regexp. Follow up to
01ef1d08fc,
which did not take into account codes in
the format X.X.XX (with the 2 digits on the end)
2022-02-22 08:54:01 +10:00
Bianca Nenciu 90c3695ab0
FEATURE: Rename Reset Read bulk action to Defer (#15972)
It is enabled only if defer is enabled in user options too and if the
button shows up in the topic's footer.
2022-02-21 22:45:01 +02:00
Bianca Nenciu 94883dd326
FIX: Show links with 0 clicks in top links section (#16016)
This is useful to moderators to see if users post spam links.
2022-02-21 22:15:38 +02:00
Bianca Nenciu ae1d2d957f
FEATURE: Replace share post popup with share modal (#15875)
This uniformizes the topic share modal and the post link popup. It also
introduces a new feature which can notify the user of a post.
2022-02-21 22:14:28 +02:00
David Taylor 5d6d3fb244
DEV: Remove child theme settings/variables from parent compilation (#16001)
aa1442fdc3 split theme stylesheets so that every component gets its own stylesheet. Therefore, there is now no need for parent themes to collate the settings/variables of its children during scss compilation.

Technically this is a breaking change for any themes which depend on the settings/variables of their child components. That was never a supported/recommended arrangement, so we don't expect this to cause issues.
2022-02-21 11:15:35 +00:00
David Taylor c8d956374d FIX: Use fresh theme setting values when compiling stylesheets
If a theme is updated to introduce a new setting AND immediately make use of it in a stylesheet, then an error was being shown. This is because the stylesheet compilation was using the theme's cached settings, and the cache is only cleared **after** the theme has finished compiling.

This commit updates the SCSS compilation to use uncached values for settings. A similar fix was applied to other parts of theme compilation back in 2020: (a51b8d9c66)
2022-02-21 09:45:14 +00:00
David Taylor 51afa579f7 DEV: Use the theme cache helper for settings
The previous Discourse.cache usage was different to how other theme-related caching is handled, and also requires reaching out to redis every time. The common theme cache is held in memory (as a DistributedCache)
2022-02-21 09:45:14 +00:00
Martin Brennan 01ef1d08fc
FIX: Conform EmailLog#bounce_error_code to RFC (#16010)
This commit makes sure that the email log's bounce_error_code
conforms to the SMTP error code RFC on save, so that
it is always in the format X.X.X or XXX without any
additional string details. Also included is a migration
to fix this issue for past records.
2022-02-21 11:26:39 +10:00
Jarek Radosz 45cc16098d
DEV: Move spec/components to spec/lib (#15987)
Lib specs were inexplicably split into two directories (`lib` and `components`)

This moves them all into `lib`.
2022-02-18 19:41:54 +01:00
Bianca Nenciu 38cbca3f67
FIX: Count clicks on links with query params (#15969)
This did not work sometimes if a topic had the same URL with and without
query params because it did not try to select the best matching URL.
2022-02-18 14:47:56 +02:00
Bianca Nenciu 53f9a1a469
FEATURE: Add settings to scale daily flags limit (#15983)
Similar site settings exist for likes and edits and the new ones work
in a similar way.

By default, users below TL2 have a limit of 20, the limit is increased
by 1.5 for TL2 users up to 30, by 2 for TL3 users up to 40 and by 3 for
TL4 users up to 60.
2022-02-18 14:44:32 +02:00
Michael Brown 3bf3b9a4a5 DEV: pull email address validation out to a new EmailAddressValidator
We validate the *format* of email addresses in many places with a match against
a regex, often with very slightly different syntax.

Adding a separate EmailAddressValidator simplifies the code in a few spots and
feels cleaner.

Deprecated the old location in case someone is using it in a plugin.

No functionality change is in this commit.

Note: the regex used at the moment does not support using address literals, e.g.:
* localpart@[192.168.0.1]
* localpart@[2001:db8::1]
2022-02-17 21:49:22 -05:00
Jarek Radosz aaf432df86
DEV: Remove mock_redis (#15985)
Was used just in one spec file. And we prefer to run specs against a real redis server.
2022-02-18 01:14:38 +01:00
David Taylor ffcd2e9faf
FIX: Handle `nil` values in `DistributedCache#defer_get_set` (#15978)
Themes often cache `nil` values in a DistributedCache. This bug meant that we were re-calculating some values on every request, AND triggering message-bus publishing on every request.

This fix should provide a significant performance improvement for busy sites.
2022-02-17 14:52:14 +00:00
Osama Sayegh dd6ec65061
FEATURE: Centralized 2FA page (#15377)
2FA support in Discourse was added and grown gradually over the years: we first
added support for TOTP for logins, then we implemented backup codes, and last
but not least, security keys. 2FA usage was initially limited to logging in,
but it has been expanded and we now require 2FA for risky actions such as
adding a new admin to the site.

As a result of this gradual growth of the 2FA system, technical debt has
accumulated to the point where it has become difficult to require 2FA for more
actions. We now have 5 different 2FA UI implementations and each one has to
support all 3 2FA methods (TOTP, backup codes, and security keys) which makes
it difficult to maintain a consistent UX for these different implementations.
Moreover, there is a lot of repeated logic in the server-side code behind these
5 UI implementations which hinders maintainability even more.

This commit is the first step towards repaying the technical debt: it builds a
system that centralizes as much as possible of the 2FA server-side logic and
UI. The 2 main components of this system are:

1. A dedicated page for 2FA with support for all 3 methods.
2. A reusable server-side class that centralizes the 2FA logic (the
`SecondFactor::AuthManager` class).

From a top-level view, the 2FA flow in this new system looks like this:

1. User initiates an action that requires 2FA;

2. Server is aware that 2FA is required for this action, so it redirects the
user to the 2FA page if the user has a 2FA method, otherwise the action is
performed.

3. User submits the 2FA form on the page;

4. Server validates the 2FA and if it's successful, the action is performed and
the user is redirected to the previous page.

A more technically-detailed explanation/documentation of the new system is
available as a comment at the top of the `lib/second_factor/auth_manager.rb`
file. Please note that the details are not set in stone and will likely change
in the future, so please don't use the system in your plugins yet.

Since this is a new system that needs to be tested, we've decided to migrate
only the 2FA for adding a new admin to the new system at this time (in this
commit). Our plan is to gradually migrate the remaining 2FA implementations to
the new system.

For screenshots of the 2FA page, see PR #15377 on GitHub.
2022-02-17 12:12:59 +03:00
megothss c71c107649
FIX: Don't accept accents in slug if generation_method == 'ascii' (#15702)
* FIX: Don't accept accents in slug if generation_method == 'ascii'

Fixes bug reported in:
- https://meta.discourse.org/t/404-when-trying-to-edit-category-with-accent-in-slug/214762
- https://meta.discourse.org/t/formatting-and-accents-in-urls/215734/5

Assuming `SiteSetting.slug_generation_method == 'ascii'.

If the user provides a slug containing non-ascii characters while
creating the category, the user will receive a 404 error just
after saving the category since the slug will be escaped anyway but
Category.find_by_slug_path won't escape the category slug
causing the Edit Page of the category to be inaccessible.

This commit checks the provided slug and raises an error if the
provided slugcontains non-ascii characters ensuring that the
provided value is consistent with the site settings.

It also changes Category.find_by_slug_path to always escape the slug,
since if present, it is escaped anyway in Category.ensure_slug to
prevent the 404 in the Edit Category Page in case the user already
have some category with a non-ascii slug.

* Removed trailing whitespace
2022-02-17 13:46:06 +11:00
Krzysztof Kotlarek a7d43cf1ec
FEATURE: mute subcategory when parent category is muted (#15966)
When parent category or grandparent category is muted, then category should be muted as well.

Still, it can be overridden by setting individual subcategory notification level.

CategoryUser record is not created, mute for subcategories is purely virtual.
2022-02-17 00:42:02 +01:00
Dan Ungureanu effbd6d3e4
FEATURE: Show error if invite to topic is invalid (#15959)
This can happen if the topic to which a user is invited is in a private
category and the user was not invited to one of the groups that can see
that specific category.

This used to be a warning and this commit makes it an error.
2022-02-16 18:35:02 +02:00
David Taylor e945f301d1
PERF: Skip running 'auto_leave' during every PresenceChannel method (#15970)
These calls were originally introduced to ensure that any stale users were cleaned up regularly. This is quite an expensive process to run on every `GET /presence/get` call, and will also cause errors during readonly mode.

Since the original introduction of this logic, we added the `Jobs::PresenceChannelAutoLeave` which runs every minute. That should be enough to clean up any stale users.

Note that users which explicitly `leave` a channel are still removed immediately. This auto_leave logic just takes care of clients which have disappeared without leaving.
2022-02-16 11:18:13 +00:00
Bianca Nenciu 5eaf214594
FEATURE: New plugin API to check if upload is used (#15545)
This commit introduces two new APIs for handling unused uploads, one
can be used to exclude uploads in bulk when the data model allow and
the other one excludes uploads one by one.
2022-02-16 09:00:30 +02:00
Martin Brennan 2d30dd439f
DEV: Add chat_quoted notification type (#15968)
This is needed for the notification sent when quoting
chat messages inside a post.
2022-02-16 15:22:08 +10:00
Vinoth Kannan 1ea19a4d51
FIX: unable to filter user directory when sorted by user field. (#15951)
Since the "users" table is already added in the "includes" method it gives unexpected results while using it again in the "joins" method.
2022-02-16 07:57:35 +05:30
Alan Guo Xiang Tan 6c374cf158
DEV: Improve ArgumenError raised in PostOwnerChanger. (#15907)
Currently, it doesn't provide any context about the arguement which
caused the error.
2022-02-16 12:52:20 +11:00
Martin Brennan f9ec2b90a0
DEV: Drop user_stats count column constraints (#15949)
We added this constraint in 5bd55acf83
but it is causing problems in hosted sites and is catching the
issue too far down the line. This commit removes the constraint
for now, and also fixes an issue found with PostDestroyer
which wasn't using the UserStatCountUpdater when updating post_count
and thus was causing negative numbers to occur.
2022-02-16 12:49:11 +11:00
Sam 33a0ad1b69
PERF: introduce site/global emoji cache (#15899)
Previously calls such as `Emoji["smile"]` would force a full dehydration of
objects from Redis.

This introduces a version safe site and global emoji cache so lookups are
cheap. It eliminates iterating through the list of emojis and pulling from
redis.

Distributed cache uses a normalized name as the key and stores an Array tuple
with version and Emoji. Successful hits always confirm version matches.

Interface to Emoji object remains unchanged.

We opted for 2 caches to improve reuse on multisites. misses though will be
stored in both caches. If there is a hit on the global cache we can avoid
looking up in site local cache and storing a miss there.
2022-02-16 12:46:17 +11:00
David Taylor 94a47d037f
PERF: Reduce number of EXPIRE calls from CachedCounting (#15958)
Previously we were calling `EXPIRE` every time we incremented a given key. Instead, we can call EXPIRE once when the key is first populated. A LUA script is used to make this as efficient as possible.

Consumers of this Concern use daily keys. Since we're now calling EXPIRE only at the beginning of the day, rather than throughout the day, the expire time has been increased from 3 to 4 days.
2022-02-15 16:55:21 +00:00
David Taylor 11c93342dc
DEV: Consolidate Redis evalsha logic into DiscourseRedis::EvalHelper (#15957) 2022-02-15 16:06:12 +00:00
Dan Ungureanu dd5373cc4c
FIX: Do not increase invite count for current user (#15952)
The current user could redeem an invite created by themselves.
2022-02-15 17:35:58 +02:00
Rafael dos Santos Silva 4d3da70bc6
PERF: Use Redis `SET EX GET` instead of LUA script for counting (#15939)
This will prevent Discourse from booting on Redis < 6.2.0
2022-02-15 10:36:07 -03:00
Martin Brennan 4086ee551e
DEV: Add bounce_error_code to EmailLog (#15948)
Whenever we got a bounced email in the Email::Receiver we
previously would just set bounced: true on the EmailLog and
discard the status/diagnostic code. This commit changes this
flow to store the bounce error code (defined in the RFC at
https://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xhtml)
not just in the Email::Receiver, but also via webhook events
from other mail services and from SNS.

This commit does not surface the bounce error in the UI,
we can do that later if necessary.
2022-02-15 14:17:26 +10:00
David Taylor b8526c6d75
DEV: Ensure stylesheet sourcemap URLs include __ws hostname param (#15934)
Without this parameter, requests for sourcemaps on shared-CDN multisites will not be routed to the correct database, resulting in a 404.

The stylesheet content now depends on the site hostname, so the hostname has been added to the digest.
2022-02-14 19:53:39 +00:00
David Taylor 07893779df
DEV: Correct service-worker sourceMappingURL (#15916)
We serve `service-worker.js` in an unusual way, which means that the sourcemap is not available on an adjacent path. This means that the browser fails to fetch the map, and shows an error in the console.

This commit re-writes the source map reference in the static_controller to be an absolute link to the asset (including the appropriate CDN, if enabled), and adds a spec for the behavior.

It's important to do this at runtime, rather than JS precompile time, so that changes to CDN configuration do not require re-compilation to take effect.
2022-02-14 12:47:56 +00:00
David Taylor af24c10314 DEV: Improve theme error handling UX
- Update UI to improve contrast
- Make it clear that the message is only shown to administrators
- Add theme name and id to the console output
- Parse the error backtrace to identify the theme-id for post-decoration errors
- Improve console output to include the theme name / URL
- Add `?safe_mode=no_custom` to the admin panel link, so that it will work even if the theme is causing the site to break
2022-02-14 10:11:19 +00:00
Martin Brennan 4b4f2330da
DEV: Expand UploadMarkdown generation capabilities (#15930)
The chat quoting mechanism will need to be able to generate
markdown for all kinds of uploads. The UploadMarkdown class
was missing generation for video and audio uploads. This
commit adds that in, and also expands the server-side regex
recognition of FileHelper types to match those in uploads.js,
and adds a spec for UploadMarkdown
2022-02-14 15:48:27 +10:00
Martin Brennan 88a8584348
FIX: Cooking custom emojis should not use a secure URL (#15929)
When a site has secure media enabled and a post is with secure
media, we were incorrectly cooking custom emoji URLs and using the
secure URL for those emojis, even though they should not be
considered secure (their corresponding upload records in the
database are _not_ secure). Now instead of the blanket
post.with_secure_media? boolean for the secure: param, we also
want to make sure the image whose URL is being cooked is also
_not_ a custom emoji.
2022-02-14 13:02:42 +10:00
Krzysztof Kotlarek a34075d205
SECURITY: Onebox response timeout and size limit (#15927)
Validation to ensure that Onebox request is no longer than 10 seconds and response size is not bigger than 1 MB
2022-02-14 12:11:09 +11:00
Bianca Nenciu 694205cc0c
DEV: Add include_all_pms option to TopicQuery (#15742)
This is intended for use by plugins which are building their own
topic lists, and want to include PMs alongside regular topics (e.g.
discourse-assign). It does not get used directly in core.
2022-02-11 14:46:23 +02:00
Alan Guo Xiang Tan 8e5b945b0f
Revert "DEV: Improve theme error handling UX" (#15900)
`PrettyText.cook` is breaking on some sites. Revert for now while we
investigate.

This reverts commit c81d369ab6.
2022-02-11 11:30:36 +08:00
Blake Erickson febc2361bd
DEV: Document external topic id endpoints (#15897)
* DEV: Document external topic id endpoints

This commit documents the existing Create Topic endpoint with the
`external_id` param and documents the new get topic by external id
endpoint.

It also refactors the existing topic show endpoint to use the new format
where we load the expected json schema response from a file.

See: 71f7f7ed49

* clean up unused test variables
2022-02-10 19:01:19 -07:00
Alan Guo Xiang Tan b876ff6281
FIX: Update user stat counts when post/topic visibility changes. (#15883)
Breakdown of fixes in this commit:

* `UserStat#topic_count` was not updated when visibility of
the topic changed.

* `UserStat#post_count` was not updated when post was hidden or
unhidden.

* `TopicConverter` was only incrementing or decrementing the counts by 1
even if a user has multiple posts in the topic.

* The commit turns off the verbose logging by default as it is just
noise to normal users who are not debugging this problem.
2022-02-11 09:00:58 +08:00
David Taylor c81d369ab6 DEV: Improve theme error handling UX
- Update UI to improve contrast
- Make it clear that the message is only shown to administrators
- Add theme name and id to the console output
- Parse the error backtrace to identify the theme-id for post-decoration errors
- Improve console output to include the theme name / URL
- Add `?safe_mode=no_custom` to the admin panel link, so that it will work even if the theme is causing the site to break
2022-02-10 22:56:11 +00:00
Blake Erickson e66241af7f
DEV: Update api docs to update username and to get post replies (#15881)
Added api docs for:

 - update username
 - get post replies
2022-02-09 15:04:36 -07:00
Dan Ungureanu 2d8ebe989a
FEATURE: Redeem invites for existent users (#15866)
This adds logic to increase an `InvitedUser` record, increase
`redemption_count` and create a `:invitee_accepted` to let the inviter
know that the invitee used the invite.

Initial support for this was implemented in commit 9969631.
2022-02-09 17:22:30 +02:00
Alan Guo Xiang Tan ae0625323a
FIX: Avoid errors when updating post and topic count user stats. (#15876)
In ab5361d69a, we rescue from the PG error
but the transaction is already aborted causing any DB query after to
fail. As such, we avoid triggering the error in the first place by
checking that we would not be insertin a negative number into the
counter cache.

Follow-up to ab5361d69a
2022-02-09 21:48:18 +08:00
Dan Ungureanu 1fb97f8bba
FIX: Replace Twitter handles one at a time (#15870)
Previously, all handles and hashtags were replaced in one go which could
result in a wrong result if a handle was a substring of another one.
2022-02-09 13:54:02 +02:00
Ayke Halder 5ff3a9c4bb
DEV: add native lazy loading for emojis (#15830) 2022-02-09 12:18:59 +01:00
Joffrey JAFFEUX b3ecf00c98
DEV: adds initial support for custom blocks using code fencing (#15743)
Allows to write custom code blocks:

```
   ```mermaid height=200,foo=bar
   test
   ```
```

Which will then get converted to:

```
<pre data-code-wrap="mermaid" data-code-height="200" data-code-foo="bar">
  <code class="lang-nohighlight">
    test
  </code>
</pre>
```
2022-02-09 11:23:44 +01:00
Bianca Nenciu c38114f0c6
FIX: Sort group owners and members together (#15708)
Sorting group members worked always kept the group owners at the top of
the list. This commit keeps the group owners at the top of the list only
when no order exists.
2022-02-09 11:43:58 +02:00
Bianca Nenciu f704deca17
FIX: Clear drafts only when post is created by real user (#15720)
This commits adds a new advance_draft to PostCreator that controls if
the draft sequence will be advanced or not. If the draft sequence is
advanced then the old drafts will be cleared. This used to happen for
posts created by plugins or through the API and cleared user drafts
by mistake.
2022-02-09 10:37:38 +02:00
Alan Guo Xiang Tan ab5361d69a
FIX: Avoid raising error when updating post and topic count user stats. (#15873)
There are still spots in the code base which results in us trying to turn the post and topic count negative. However,
we have a job that runs on a daily basis which will correct the count. Therefore, avoid raising an error for now
and log the exception instead.
2022-02-09 12:51:13 +08:00
Blake Erickson 71f7f7ed49
FEATURE: Add external_id to topics (#15825)
* FEATURE: Add external_id to topics

This commit allows for topics to be created and fetched by an
external_id. These changes are API only for now as there aren't any
front changes.

* add annotations

* add external_id to this spec

* Several PR feedback changes

- Add guardian to find topic
- 403 is returned for not found as well now
- add `include_external_id?`
- external_id is now case insensitive
- added test for posts_controller
- added test for topic creator
- created constant for max length
- check that it redirects to the correct path
- restrain external id in routes file

* remove puts

* fix tests

* only check for external_id in webhook if exists

* Update index to exclude external_id if null

* annotate

* Update app/controllers/topics_controller.rb

We need to check whether the topic is present first before passing it to the guardian.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>

* Apply suggestions from code review

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2022-02-08 20:55:32 -07:00
Martin Brennan 59343c3057
DEV: Add Windows Hello webauthn authentication spec (#15871)
Follow up to 6f7364e48b to add a spec
that tests the full authentication of a Windows Hello algorithm (-257)
webauthn verification. The test added in that commit only tested that
we know about that algorithm, not whether it was actually usable.
2022-02-09 12:47:47 +10:00
Dan Ungureanu 6f7364e48b
FEATURE: RS512, RS384 and RS256 COSE algorithms (#15804)
* FEATURE: RS512, RS384 and RS256 COSE algorithms

These algorithms are not implemented by cose-ruby, but used in the web
authentication API and were marked as supported.

* FEATURE: Use all algorithms supported by cose-ruby

Previously only a subset of the algorithms were allowed.
2022-02-08 14:07:47 +02:00
Alan Guo Xiang Tan 7afe768d60
DEV: Add tests for wistia onebox. (#15860)
Follow-up to 4ef56b0ca4
2022-02-08 13:04:32 +08:00
Jarek Radosz 6758835387
FIX: Overridden MessageFormat fallbacks (#15855)
…were missing pluralization rules

This resulted in errors like `MessageFormat.locale.en is not a function`
2022-02-08 12:31:08 +11:00
David Taylor c8c23ba557
DEV: Introduce deprecation warning for non-json Job arguments (#15842)
This commit introduces our own handling and warning for Sidekiq's new 'non-json-serializable' warning. This decouples us from Sidekiq's own deprecation cycle, and allows us to use our own deprecation system. It also means that the dump/parse happens in test mode, which will help us to catch occurrences before they reach production.
2022-02-07 17:59:55 +00:00
David Taylor 64be371749
DEV: Improve handling of invalid requests (#15841)
Our discourse_public_exceptions middleware is designed to catch bubbled exceptions from lower in the stack, and then use `ApplicationController.rescue_with_handler` to render an appropriate error response.

When the request itself is invalid, we had an escape-hatch to skip re-dispatching the request to ApplicationController. However, it was possible to work around this by 'layering' the errors. For example, if you made a request which resulted in a 404, but **also** had some other invalidity, the escape hatch would not be triggered.

This commit ensures that these kind of 'layered' errors are properly handled, without logging warnings. It also adds detection for invalid JSON bodies and badly-formed multipart requests.

The user-facing behavior is unchanged. This commit simply prevents warnings being logged for invalid requests.
2022-02-07 13:16:57 +00:00
Andrei Prigorshnev 404f882e5c
FEATURE: in result.rb don't use email for username suggestions until enabled in settings (#15586) 2022-02-07 16:02:26 +04:00
Martin Brennan 0a738bd5bc
FEATURE: Allow sending group SMTP emails with from alias (#15687)
This commit allows group SMTP emails to be sent with a
different from email address that has been set up as an
alias in the email provider. Emails from the alias will
be grouped correctly using Message-IDs in the mail client,
and replies to the alias go into the correct group inbox.
2022-02-07 13:52:01 +10:00
Alan Guo Xiang Tan 454d3740b4
DEV: Remove deprecated site settings. (#15725) 2022-02-07 11:26:06 +08:00
Alan Guo Xiang Tan b7eacaed21
FIX: Handle addressable error when parsing an invalid URL. (#15836)
Passing in an invalid URL would result in an `Addressable::URI::InvalidURIError` which we were not catching.
2022-02-07 11:25:42 +08:00
Alan Guo Xiang Tan 5bd55acf83
FIX: Add DB constraints for post & topic counter cache for `UserStat` (#15626)
Ensures that `UserStat#post_count` and `UserStat#topic_count` does not
go below 0. When it does like it did now, we tend to have bugs in our
code since we're usually coding with the assumption that the count isn't
negative.

In order to support the constraints, our post and topic fabricators in
tests will now automatically increment the count for the respective
user's `UserStat` as well. We have to do this because our fabricators
bypasss `PostCreator` which holds the responsibility of updating `UserStat#post_count` and
`UserStat#topic_count`.
2022-02-07 11:23:34 +08:00
Ayke Halder 81e175e6ba
FIX: allow native lazy loading attribute for quoted avatar image (#15834) 2022-02-07 09:28:26 +08:00
Alan Guo Xiang Tan 65f46ad4ed FIX: Better param guards for `wiki` and `post_type` posts controller. 2022-02-07 09:21:24 +08:00
Alan Guo Xiang Tan 930f51e175 FEATURE: Split up text segmentation for Chinese and Japanese.
* Chinese segmenetation will continue to rely on cppjieba
* Japanese segmentation will use our port of TinySegmenter
* Korean currently does not rely on segmentation which was dropped in c677877e4f
* SiteSetting.search_tokenize_chinese_japanese_korean has been split
into SiteSetting.search_tokenize_chinese and
SiteSetting.search_tokenize_japanese respectively
2022-02-07 09:21:14 +08:00
janzenisaac cffc2836cb
DEV: Don't allow users to immediately reinvite (#15722)
- Limit bulk re-invite to 1 time per day
- Move bulk invite by csv behind a site setting (hidden by default)
- Bump invite expiry from 30 -> 90 days

## Updates to rate_limiter
When limiting reinvites I found that **staff** are never limited in any way. So I updated the **rate_limiter** model to allow for a few things:
- add an optional param of `staff_limit`, which (when included and passed values, and the user passes `.staff?`) will override the default `max` & `secs` values and apply them to the user.
- in the case you **do** pass values to `staff_limit` but the user **does not** pass `staff?` the standard `max` & `secs` values will be applied to the user.

This should give us enough flexibility to 
1. continue to apply a strict rate limit to a standard user
2. but also apply a secondary (less strict) limit to staff
2022-02-03 13:07:40 -06:00
Rafael dos Santos Silva 5b5cbbfe5c
FEATURE: Onebox for news.ycombinator.com (#15781) 2022-02-03 13:39:21 -03:00
Alan Guo Xiang Tan 4aa9a813ec
FIX: Liking whispers should not contribute to `Topic#like_count`. (#15703)
Non-staff users are not allowed to see whisper so this change prevents
non-staff user from seeing a like count that does not make sense to
them. In the future, we might consider adding another like count column
for staff user.

Follow-up to 4492718864
2022-02-03 16:24:33 +11:00
Natalie Tay 23a8341b28
FEATURE: Validate domain settings for blocked_onebox_domain only (#15754)
We want to prevent the user from adding ? or * minimally
when setting domains in sitesettings
2022-02-03 11:26:53 +08:00
Martin Brennan 82cb67e67b
FIX: Canonical Message-ID was incorrect for some cases (#15701)
When creating a direct message to a group with group SMTP
set up, and adding another person to that message in the OP,
we send an email to the second person in the OP via the group_smtp
job. This in turn creates an IncomingEmail record to guard against
IMAP double sync.

The issue with this was that this IncomingEmail (which is essentialy
a placeholder/dummy one) was having its Message-ID used as the canonical
References Message-ID for subsequent emails sent out to user_private_message
recipients (such as members of the group), causing threading issues in
the mail client. The canonical <topic/ID@HOST> format should be used
instead for these cases.

This commit fixes the issue by only using the IncomingEmail for the
OP's Message-ID if the OP was created via our handle_mail email receiver
pipeline. It does not make sense to use it in other cases.
2022-02-03 10:36:32 +10:00
Krzysztof Kotlarek bb01563682
FIX: post mover validation color and message (#15688)
When the record is not saved, we should display a proper message.
One potential reason can be plugins for example discourse-calendar is specifying that only first post can contain event
2022-02-02 16:22:52 +11:00
Rafael dos Santos Silva 2e2fee9da3
FIX: Remove svg icons from webmanifest shortcuts (#15765)
* FIX: Remove svg icons from webmanifest shortcuts

While SVGs are valid in the webmanifest, Chromium has not implemented
support for it in this specific manifest member.

Revert when https://bugs.chromium.org/p/chromium/issues/detail?id=1091612
lands.

* fix test
2022-02-01 15:26:58 -03:00
Natalie Tay aac9f43038
Only block domains at the final destination (#15689)
In an earlier PR, we decided that we only want to block a domain if 
the blocked domain in the SiteSetting is the final destination (/t/59305). That 
PR used `FinalDestination#get`. `resolve` however is used several places
 but blocks domains along the redirect chain when certain options are provided.

This commit changes the default options for `resolve` to not do that. Existing
users of `FinalDestination#resolve` are
- `Oneboxer#external_onebox`
- our onebox helper `fetch_html_doc`, which is used in amazon, standard embed 
and youtube
  - these folks already go through `Oneboxer#external_onebox` which already
  blocks correctly
2022-01-31 15:35:12 +08:00
Martin Brennan 0b8d0a14d7
DEV: Add markdown_additional_options to Site (#15738)
Sometimes plugins need to have additional data or options available
when rendering custom markdown features/rules that are not available
on the default opts.discourse object. These additional options should
be namespaced to the plugin adding them.

```
Site.markdown_additional_options["chat"] = { limited_pretty_text_markdown_rules: [] }
```

These are passed down to markdown rules on opts.discourse.additionalOptions.

The main motivation for adding this is the chat plugin, which currently stores
chat_pretty_text_features and chat_pretty_text_markdown_rules on
the Site object via additions to the serializer, and the Site object is
not accessible to import via markdown rules (either through
Site.current() or through container.lookup). So, to have this working
for both front + backend code, we need to attach these additional options
from the Site object onto the markdown options object.
2022-01-28 13:02:02 +10:00
Gerhard Schlager 1fef96a2e7
FIX: Prevent "integer out of range" when merging post timings (#15723) 2022-01-26 23:34:28 +01:00
Dan Ungureanu f5b94f152f
FIX: Allow staff to reset passwords by username (#15709)
When staff visits the user profile of another user, the `email` field
in the model is empty. In this case, staff cannot send the reset email
password because nothing is passed in the `login` field.

This commit changes the behavior for staff users to allow resetting
password by username instead.
2022-01-26 10:39:58 +02:00
Alan Guo Xiang Tan 77137c5d29 FIX: Single line emojis has emoji metadata indexed twice.
This commit fixes a bug where we our `HTMLScrubber` was only searching
for emoji img tags which contains only the "emoji" class. However, our emoji image tags
may contain more than just the "emoji" class like "only-emoji" when an
emoji exists by itself on a single line.
2022-01-24 14:03:17 +08:00
Bianca Nenciu 48e5d1af03
FIX: Improve top links section from user summary (#15675)
* Do not extract links for hotlinked images
* Include only links that have been clicked at least once in user
summary
2022-01-24 11:33:23 +11:00
Bianca Nenciu a0c040060a
FIX: Select best link from Atom feed (#15663)
Some Atom feeds can contain more than one link and it used to return
only the first link.
2022-01-21 17:54:18 +02:00
Martin Brennan 70af45055a
DEV: Clear custom field preload proxy on preload_custom_fields (#15671)
If a model class calls preload_custom_fields twice then
we have to clear this otherwise the fields are cached inside the
already existing proxy and no new ones are added, so when we check
for custom_fields[KEY] an error is likely to occur
2022-01-21 14:29:51 +10:00
Alan Guo Xiang Tan e4e37257cc FIX: Handle malformed URLs in `TopicEmbed.absolutize_urls`. 2022-01-21 11:18:54 +08:00
David Taylor 820564826e
FIX: Ensure that login does not fail for users with invite records (#15647)
In the unlikely, but possible, scenario where a user has no email_tokens, and has an invite record for their email address, login would fail. This commit fixes the `Invite` `user_doesnt_already_exist` validation so that it only applies to new invites, or when changing the email address.

This regressed in d8fe0f4199 (based on `git bisect`)
2022-01-20 10:54:38 +00:00
Dan Ungureanu 5b7bddf966
FIX: Prefer email when resetting password (#15650)
The UI used to request a password reset by username when the user was
logged in. This did not work when hide_email_already_taken site setting
was enabled, which disables the lookup-by-username functionality.

This commit also introduces a check to ensure that the parameter is an
email when hide_email_already_taken is enabled as the single allowed
type is email (no usernames are allowed).
2022-01-20 10:04:45 +02:00
Natalie Tay f5ea00c73f
FIX: Respect blocked domains list when redirecting (#15656)
Our previous implementation used a simple `blocked_domain_array.include?(hostname)`
so some values were not matching. Additionally, in some configurations like ours, we'd used
"cat.*.dog.com" with the assumption we'd support globbing.

This change implicitly allows globbing by blocking "http://a.b.com" if "b.com" is a blocked 
domain but does not actively do anything for "*".

An upcoming change might include frontend validation for values that can be inserted.
2022-01-20 14:12:34 +08:00