Commit Graph

935 Commits

Author SHA1 Message Date
Jarek Radosz 20e34b5da6
DEV: Stabilize watched words order (#17215)
Fixes a flaky spec:

```
  1) WordWatcher.word_matcher_regexp format of the result regexp is correct when watched_words_regular_expressions = true
     Failure/Error: expect(regexp.inspect).to eq("/(#{word1})|(#{word2})/i")

       expected: "/(word35)|(word36)/i"
            got: "/(word36)|(word35)/i"

       (compared using ==)
     # ./spec/services/word_watcher_spec.rb:19:in `block (4 levels) in <main>'
```
2022-06-23 15:38:12 +02:00
Martin Brennan a176b57be0
FIX: Use bookmarkable pattern for bookmark cleanup (#17202)
We have a `cleanup!` class method on bookmarks that deletes
bookmarks X days after their related record (post/topic) are
deleted. This commit changes this method to use the
registered_bookmarkables for this instead, and each bookmarkable
type can delete related bookmarks in their own way.
2022-06-23 14:09:39 +10:00
Isaac Janzen 6ae761604a
FIX: Send quote notifications to correct users when prioritizing full names (#17030) 2022-06-09 11:52:28 -05:00
Sam 7b4e338c0e
FIX: fallback to default push notification icon if none exists (#16961)
Due to some changes we started notifying via push notifications on other
families of notifications. There are a total of about 30 or so possible
notification you could get, some can be pushed.

This fallback means that if for any reason we are unable to find an icon
for a push notification we just fallback to the Discourse logo.

Also go with a simple reply icon for watching first post.

Note, that in production `image_url` can return an exception if an image is
missing. This is not the case in test / development.
2022-06-01 12:00:05 +10:00
Bianca Nenciu 7328a2bfb0
FIX: Apply censored words to inline onebox (#16873)
Censored watched words were not censored inside the title of an inline
oneboxes. Malicious users could exploit this behaviour to insert bad
words. The same issue has been fixed for regular Oneboxes in commit
d184fe59ca.
2022-05-25 14:51:47 +03:00
Penar Musaraj ebdfc536dd
Revert "FEATURE: Include participants in PN search data (#16855)" (#16904)
This reverts commit 71c74a262d.
2022-05-25 15:08:36 +10:00
Alan Guo Xiang Tan 6edf101d5f
DEV: Minor improvements to WordWatcher (#16735)
Follow-up to fd1dc91eed
2022-05-24 10:23:54 +08:00
David Taylor bf6f8299a7 FEATURE: Pull hotlinked images immediately after posting
Previously, with the default `editing_grace_period`, hotlinked images were pulled 5 minutes after a post is created. This delay was added to reduce the chance of automated edits clashing with user edits.

This commit refactors things so that we can pull hotlinked images immediately. URLs are immediately updated in the post's `cooked` HTML. The post's raw markdown is updated later, after the `editing_grace_period`.

This involves a number of behind-the-scenes changes including:

- Schedule Jobs::PullHotlinkedImages immediately after Jobs::ProcessPost. Move scheduling to after the `update_column` call to avoid race conditions

- Move raw changes into a separate job, which is delayed until after the ninja-edit window

- Move disable_if_low_on_disk_space logic into the `pull_hotlinked_images` job

- Move raw-parsing/replacing logic into `InlineUpload` so it can be easily be shared between `UpdateHotlinkedRaw` and `PullUserProfileHotlinkedImages`
2022-05-23 14:28:02 +01:00
Gerhard Schlager 1e1b85c214
FIX: Site setting changes for boolean should be logged as true/false (#16888)
Previously true/false sometimes was logged as t or f
2022-05-23 11:23:10 +08:00
Penar Musaraj 71c74a262d
FEATURE: Include participants in PN search data (#16855)
This makes it easier to find PMs involving a particular user, for
example by searching for `in:messages thisUser` (previously, that query
would only return results in posts where `thisUser` was in the post body).
2022-05-18 10:34:01 -04:00
Alan Guo Xiang Tan fd1dc91eed
DEV: Don't cache watched words in test env (#16731)
The cache was causing state to leak between tests since the `WatchedWord` record in the DB would have been rolled back but `WordWatcher` still had the word in the cache.
2022-05-12 14:45:05 +08:00
David Taylor 67b23c0e22
PERF: Update all user_histories with one query in UserDestroyer (#16717)
7a284164 previously switched the UserDestroyer to use find_each when iterating over UserHistory records. Unfortunately, since this logic is wrapped in a transaction, this didn't actually solve the memory usage problem. ActiveRecord maintains references to all modified models within a transaction.

This commit updates the logic to use a single SQL query, rather than updating models one-by-one
2022-05-11 13:39:31 +01:00
Arpit Jalan 977da1da77
FIX: do not log category custom fields changes if the value is unchanged (#16712) 2022-05-11 13:51:51 +05:30
Martin Brennan 907adce1cb
FIX: Use registered bookmarkables for BookmarkManager (#16695)
These validate/after_create/after_destroy methods were added
back in b8828d4a2d before
the RegisteredBookmarkable API and pattern was nailed down.
This commit updates BookmarkManager to call out to the
relevant bookmarkable for these and bookmark_metadata for
consistency.
2022-05-11 09:51:03 +10:00
Martin Brennan fbcc35b417
DEV: Remove PostAction/UserAction bookmark refs (#16681)
We have not used anything related to bookmarks for PostAction
or UserAction records since 2020, bookmarks are their own thing
now. Deleting all this is just cleaning up old cruft.
2022-05-10 10:42:18 +10:00
Sam 2df3c65ba9
FIX: add support for pipelined and multi redis commands (#16682)
Latest redis interoduces a block form of multi / pipelined, this was incorrectly
passed through and not namespaced.

Fix also updates logster, we held off on upgrading it due to missing functions
2022-05-10 08:19:02 +10:00
Roman Rizzi 6bdcd7afb2
FEATURE: Promote the "delete group" staff action log. (#16656)
We used to log group deletion as custom, which means we can't later search for them. Include group ID in the details.
2022-05-09 12:12:52 -03:00
Martin Brennan 222c8d9b6a
FEATURE: Polymorphic bookmarks pt. 3 (reminders, imports, exports, refactors) (#16591)
A bit of a mixed bag, this addresses several edge areas of bookmarks and makes them compatible with polymorphic bookmarks (hidden behind the `use_polymorphic_bookmarks` site setting). The main ones are:

* ExportUserArchive compatibility
* SyncTopicUserBookmarked job compatibility
* Sending different notifications for the bookmark reminders based on the bookmarkable type
* Import scripts compatibility
* BookmarkReminderNotificationHandler compatibility

This PR also refactors the `register_bookmarkable` API so it accepts a class descended from a `BaseBookmarkable` class instead. This was done because we kept having to add more and more lambdas/properties inline and it was very messy, so a factory pattern is cleaner. The classes can be tested independently as well.

Some later PRs will address some other areas like the discourse narrative bot, advanced search, reports, and the .ics endpoint for bookmarks.
2022-05-09 09:37:23 +10:00
Joffrey JAFFEUX 81580e559c
FIX: prevents double user_badge_granted event (#16654)
This bug was causing double events to be fired as :user_badge_granted is already called when a `user_badge` is created. More over the signature of the block in the UserBadge code is `badge_id, user_id` not `badge, user_id`.
2022-05-05 15:09:41 +02:00
Loïc Guitaut 008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
Alan Guo Xiang Tan 98c49acad5
DEV: Setup experimental sidebar skeleton (#16575)
* hidden siteSetting to enable experimental sidebar
* user preference to enable experimental sidebar
* `experimental_sidebar_enabled` attribute for current user
* Empty glimmer component for Sidebar
2022-04-28 15:27:06 +08:00
Martin Brennan 3e4621c2cb
FEATURE: Polymorphic bookmarks pt. 2 (lists, search) (#16335)
This pull request follows on from https://github.com/discourse/discourse/pull/16308. This one does the following:

* Changes `BookmarkQuery` to allow for querying more than just Post and Topic bookmarkables
* Introduces a `Bookmark.register_bookmarkable` method which requires a model, serializer, fields and preload includes for searching. These registered `Bookmarkable` types are then used when validating new bookmarks, and also when determining which serializer to use for the bookmark list. The `Post` and `Topic` bookmarkables are registered by default.
* Adds new specific types for Post and Topic bookmark serializers along with preloading of associations in `UserBookmarkList`
* Changes to the user bookmark list template to allow for more generic bookmarkable types alongside the Post and Topic ones which need to display in a particular way

All of these changes are gated behind the `use_polymorphic_bookmarks` site setting, apart from the .hbs changes where I have updated the original `UserBookmarkSerializer` with some stub methods.

Following this PR will be several plugin PRs (for assign, chat, encrypt) that will register their own bookmarkable types or otherwise alter the bookmark serializers in their own way, also gated behind `use_polymorphic_bookmarks`.

This commit also removes `BookmarkQuery.preloaded_custom_fields` and the functionality surrounding it. It was added in 0cd502a558 but only used by one plugin (discourse-assign) where it has since been removed, and is now used by no plugins. We don't need it anymore.
2022-04-22 08:23:42 +10:00
Alan Guo Xiang Tan 6fc52d95de DEV: Remove scoped order warning in PostDestroyer.
`Scoped order is ignored, it's forced to be batch order.`

`find_each` ignores the `order` scope and triggers a warning in
production which is noisy.

Follow-up to 7a284164ce
2022-04-13 09:34:36 +08:00
Bianca Nenciu 86c7e07428
FEATURE: Improve screened IPs roll up and extend for IPv6 (#15585)
This commit improves the logic for rolling up IPv4 screened IP
addresses and extending it for IPv6. IPv4 addresses will roll up only
up to /24. IPv6 can rollup to /48 at most. The log message that is
generated contains the list of original IPs and new subnet.
2022-04-12 21:07:37 +03:00
Penar Musaraj df10a27067
FIX: Exclude automatic anchors from search index (#16396) 2022-04-06 16:06:45 -04:00
Daniel Waterworth 6e9a068e44
FIX: Limit max word length in search index (#16380)
Long words bloat the index for little benefit.
2022-04-06 12:23:30 -05:00
Sam 1598e6b489
FIX: users watching tags in open tag groups not notified (#16384)
All users are members of the EVERYONE group, but this group is special and
is omitted from the group_users table. When checking permission we need to
make sure we also add a bypass.

This also fixes a very buggy test in post_alerter, it was confirming the
broken behavior due to fabricator flow.

When it defined the tag group the everyone group automatically had full access
then the additional permission fabricated just added one more group. After
fix was made to code the test started failing. Fabricators can be risky.
2022-04-06 11:43:57 +10:00
Martin Brennan b982992ef7
FIX: Auto-generated emails causing group SMTP email storm (#16372)
When emailing a group inbox and including other support-type
emails (or even just regular ones with autoresponders) in the
CC field, each automated reply to the group inbox triggered
more emails to be sent out to all CC addresses to notify them
of the new reply, which in turn caused more automated emails
to be sent to the group inbox.

This commit fixes the issue by preventing any emails being sent
by the PostAlerter when the new post has an incoming email record
which is_auto_generated, which we detect in Email::Receiver.
2022-04-05 13:18:49 +10:00
Faizaan Gagan 1da4b9eeb3
FIX: update 'posted' column on post owner change (#16367)
Fixes the issue where making a user x as owner of a post doesn't
cause the concerned topic to be listed in new owner's `My Posts`
top menu filter

per https://meta.discourse.org/t/199369
2022-04-04 22:42:38 +05:30
David Taylor 7a284164ce
PERF: Update UserDestroyer to fetch histories and actions in batch (#16316)
This is much more memory-efficient when there are 1000s of rows
2022-03-29 16:56:55 +01:00
David Taylor b2a8dc4c0f
FIX: Maintain HTML `<img` when downloading remote images (#16278)
Under some conditions, replacing an `<img` with `![]()` can break rendering, and make the image disappear.

Context at https://meta.discourse.org/t/152801
2022-03-29 10:55:10 +01:00
Penar Musaraj 94750c81fa
FIX: Update group inbox notifications on archive/unarchive (#16152) 2022-03-11 11:57:47 +01:00
Bianca Nenciu 34b4b53bac
FEATURE: Use Postgres unaccent to ignore accents (#16100)
The search_ignore_accents site setting can be used to make the search
indexer remove the accents before indexing the content. The unaccent
function from PostgreSQL is better than Ruby's unicode_normalize(:nfkd).
2022-03-07 23:03:10 +02:00
Dan Ungureanu 39ab14531a
FEATURE: API to create user's associated account (#15737)
Discourse users and associated accounts are created or updated when a
user logins or connects the account using their account preferences.
This new API can be used to create associated accounts and users too,
if necessary.
2022-03-03 18:17:02 +02:00
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Jarek Radosz 6f6406ea03
DEV: Fix random typos (#16066) 2022-02-28 10:20:58 +08:00
Martin Brennan 2d30dd439f
DEV: Add chat_quoted notification type (#15968)
This is needed for the notification sent when quoting
chat messages inside a post.
2022-02-16 15:22:08 +10:00
Alan Guo Xiang Tan 6c374cf158
DEV: Improve ArgumenError raised in PostOwnerChanger. (#15907)
Currently, it doesn't provide any context about the arguement which
caused the error.
2022-02-16 12:52:20 +11:00
Martin Brennan f9ec2b90a0
DEV: Drop user_stats count column constraints (#15949)
We added this constraint in 5bd55acf83
but it is causing problems in hosted sites and is catching the
issue too far down the line. This commit removes the constraint
for now, and also fixes an issue found with PostDestroyer
which wasn't using the UserStatCountUpdater when updating post_count
and thus was causing negative numbers to occur.
2022-02-16 12:49:11 +11:00
Alan Guo Xiang Tan 81791a821c
DEV: Fix small typo. (#15904) 2022-02-11 11:51:51 +08:00
Alan Guo Xiang Tan b876ff6281
FIX: Update user stat counts when post/topic visibility changes. (#15883)
Breakdown of fixes in this commit:

* `UserStat#topic_count` was not updated when visibility of
the topic changed.

* `UserStat#post_count` was not updated when post was hidden or
unhidden.

* `TopicConverter` was only incrementing or decrementing the counts by 1
even if a user has multiple posts in the topic.

* The commit turns off the verbose logging by default as it is just
noise to normal users who are not debugging this problem.
2022-02-11 09:00:58 +08:00
Alan Guo Xiang Tan ae0625323a
FIX: Avoid errors when updating post and topic count user stats. (#15876)
In ab5361d69a, we rescue from the PG error
but the transaction is already aborted causing any DB query after to
fail. As such, we avoid triggering the error in the first place by
checking that we would not be insertin a negative number into the
counter cache.

Follow-up to ab5361d69a
2022-02-09 21:48:18 +08:00
Alan Guo Xiang Tan ab5361d69a
FIX: Avoid raising error when updating post and topic count user stats. (#15873)
There are still spots in the code base which results in us trying to turn the post and topic count negative. However,
we have a job that runs on a daily basis which will correct the count. Therefore, avoid raising an error for now
and log the exception instead.
2022-02-09 12:51:13 +08:00
Dan Ungureanu 820fea835c
FIX: Further reduce the input of to_tsvector (#15716)
Random strings can result into much longer tsvectors. For example
parsing a Base64 string of ~600kb can result in a tsvector of over 1MB,
which is the maximum size of a tsvector.

Follow-up-to: 823c3f09d4
2022-02-07 23:03:01 +02:00
Alan Guo Xiang Tan 5bd55acf83
FIX: Add DB constraints for post & topic counter cache for `UserStat` (#15626)
Ensures that `UserStat#post_count` and `UserStat#topic_count` does not
go below 0. When it does like it did now, we tend to have bugs in our
code since we're usually coding with the assumption that the count isn't
negative.

In order to support the constraints, our post and topic fabricators in
tests will now automatically increment the count for the respective
user's `UserStat` as well. We have to do this because our fabricators
bypasss `PostCreator` which holds the responsibility of updating `UserStat#post_count` and
`UserStat#topic_count`.
2022-02-07 11:23:34 +08:00
David Taylor 863262a5de
DEV: Update more Jobs#enqueue calls to use strings (#15823)
Symbols are converted to strings anyway, so there is no change in behaviour. The latest version of sidekiq introduced a warning for this.
2022-02-04 19:58:34 +00:00
Gerhard Schlager 1fef96a2e7
FIX: Prevent "integer out of range" when merging post timings (#15723) 2022-01-26 23:34:28 +01:00
Alan Guo Xiang Tan 77137c5d29 FIX: Single line emojis has emoji metadata indexed twice.
This commit fixes a bug where we our `HTMLScrubber` was only searching
for emoji img tags which contains only the "emoji" class. However, our emoji image tags
may contain more than just the "emoji" class like "only-emoji" when an
emoji exists by itself on a single line.
2022-01-24 14:03:17 +08:00
Jarek Radosz 5a50f18c0c
DEV: Avoid `$` globals (#15453)
Also:
* Remove an unused method (#fill_email)
* Replace a method that was used just once (#generate_username) with `SecureRandom.alphanumeric`
* Remove an obsolete dev puma `tmp/restart` file logic
2022-01-08 23:39:46 +01:00
Peter Zhu c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00