Commit Graph

1361 Commits

Author SHA1 Message Date
jbrw ea5d303a07
DEV: Increase timeout when pulling hotlinked image (#18036)
We observed that some sites seemingly put us in a tarpit when we attempt to pull hotlinked images. Increasing the timeout will help in these situations.
2022-08-23 09:38:10 -04:00
Jarek Radosz 2361833844
FIX: Don't raise on deleted topic in UpdateHotlinkedRaw (#17901)
Fixes:

```
Job exception: undefined method `acting_user=' for nil:NilClass
```

in
```
/var/www/discourse/lib/post_revisor.rb:181:in `revise!'
/var/www/discourse/app/models/post.rb:646:in `revise'
/var/www/discourse/app/jobs/regular/update_hotlinked_raw.rb:24:in `execute'
```
2022-08-13 22:40:26 +02:00
Roman Rizzi 58b135d6d3
DEV:Using symbols is deprecated, use strings instead (#17874) 2022-08-11 11:33:41 -03:00
Osama Sayegh ce9eec8606
DEV: Combine all header notification bubbles into one in the new user menu (#17718)
Extracted from https://github.com/discourse/discourse/pull/17379.
2022-08-03 08:57:59 +03:00
Roman Rizzi f1c3670d74
FIX: Publish membership update events when refreshing automatic groups. (#17668)
Adding or removing users from automatic groups is now consistent with `Group#add` and `Group#remove`.
2022-07-27 11:34:08 -03:00
Bianca Nenciu f75a99e932
FIX: Notify tag watchers when publishing topic (#17576)
When a topic was published from a shared draft and it had tags, the
users watching the tags were not notified. The problem was that the
topics are usually created in a secret category and publishing it just
moves an existent topic to the target category, without making any
changes to the tags.
2022-07-20 19:07:18 +03:00
Bianca Nenciu 4a996825fd
FIX: Skip job if tag edit notification is disabled (#17508)
Previous commit did not schedule any more new jobs, but the jobs in
queue could still create notifications.

Follow up to commit e8d802eb86.
2022-07-15 15:36:27 +03:00
Alan Guo Xiang Tan 7d41e980c9
FIX: Uninitialized class variable error in sidekiq (#17227)
Follow-up to 4199ada1ce
2022-06-24 14:17:39 +10:00
Martin Brennan 3f5e19c62a
FIX: Typo in log_thread (#17226)
Follow up to 4199ada1ce
2022-06-24 12:12:30 +08:00
Alan Guo Xiang Tan 4199ada1ce
DEV: Ensure Sidekiq logging thread is always running (#17211) 2022-06-24 10:28:18 +08:00
Vinoth Kannan deee3c6f02
DEV: drop unused column `flair_url` from groups table. (#17179)
It's already included in the `ignored_columns` list in the group model. 03ffb0bf27/app/models/group.rb (L9)

Also, removed the `MigrateGroupFlairImages` onceoff job and spec.
2022-06-22 00:15:05 +05:30
Jarek Radosz fcb4e5a1a1
DEV: Make wizard an ember addon (#17027)
Co-authored-by: David Taylor <david@taylorhq.com>
2022-06-17 14:50:21 +02:00
Bianca Nenciu 9db8f00b3d
FEATURE: Create upload_references table (#16146)
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.

* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
2022-06-09 09:24:30 +10:00
David Taylor 5238f6788c
FEATURE: Allow hotlinked media to be blocked (#16940)
This commit introduces a new site setting: `block_hotlinked_media`. When enabled, all attempts to hotlink media (images, videos, and audio) will fail, and be replaced with a linked placeholder. Exceptions to the rule can be added via `block_hotlinked_media_exceptions`.

`download_remote_image_to_local` can be used alongside this feature. In that case, hotlinked images will be blocked immediately when the post is created, but will then be replaced with the downloaded version a few seconds later.

This implementation is purely server-side, and does not impact the composer preview.

Technically, there are two stages to this feature:

1. `PrettyText.sanitize_hotlinked_media` is called during `PrettyText.cook`, and whenever new images are introduced by Onebox. It will iterate over all src/srcset attributes in the post HTML and check if they're allowed. If not, the attributes will be removed and replaced with a `data-blocked-hotlinked-src(set)` attribute

2. In the `CookedPostProcessor`, we iterate over all `data-blocked-hotlinked-src(set)` attributes and check whether we have a downloaded version of the media. If yes, we update the src to use the downloaded version. If not, the entire media element is replaced with a placeholder. The placeholder is labelled 'external media', and is a link to the offsite media.
2022-06-07 15:23:04 +01:00
David Taylor 19f583c449
FIX: Skip pulling hotlinked images for nil user bio (#16901) 2022-05-24 11:52:13 +01:00
David Taylor bf6f8299a7 FEATURE: Pull hotlinked images immediately after posting
Previously, with the default `editing_grace_period`, hotlinked images were pulled 5 minutes after a post is created. This delay was added to reduce the chance of automated edits clashing with user edits.

This commit refactors things so that we can pull hotlinked images immediately. URLs are immediately updated in the post's `cooked` HTML. The post's raw markdown is updated later, after the `editing_grace_period`.

This involves a number of behind-the-scenes changes including:

- Schedule Jobs::PullHotlinkedImages immediately after Jobs::ProcessPost. Move scheduling to after the `update_column` call to avoid race conditions

- Move raw changes into a separate job, which is delayed until after the ninja-edit window

- Move disable_if_low_on_disk_space logic into the `pull_hotlinked_images` job

- Move raw-parsing/replacing logic into `InlineUpload` so it can be easily be shared between `UpdateHotlinkedRaw` and `PullUserProfileHotlinkedImages`
2022-05-23 14:28:02 +01:00
David Taylor 0baabafa9d DEV: Map already-downloaded hotlinked images in post_process_cooked
Previously this mapping of **cooked** images was only being run for oneboxes. Now it runs for all images, so we can transform hotlinked images without needing to immediately update `raw`
2022-05-23 14:28:02 +01:00
Martin Brennan faf5b4d3e9
PERF: Speed up secure media and ACL sync rake tasks (#16849)
Incorporates learnings from /t/64227:

* Changes the code to set access control posts in the rake
  task to be an efficient UPDATE SQL query.
  The original version was timing out with 312017 post uploads,
  the new query took ~3s to run.
* Changes the code to mark uploads as secure/not secure in
  the rake task to be an efficient UPDATE SQL query rather than
  using UploadSecurity. This took a very long time previously,
  and now takes only a few seconds.
* Spread out ACL syncing for uploads into jobs with batches of
  100 uploads at a time, so they can be parallelized instead
  of having to wait ~1.25 seconds for each ACL to be changed
  in S3 serially.

One issue that still remains is post rebaking. Doing this serially
is painfully slow. We have a way to do this in sidekiq via PeriodicalUpdates
but this is limited by max_old_rebakes_per_15_minutes. It would
be better to fan this rebaking out into jobs like we did for the
ACL sync, but that should be done in another PR.
2022-05-23 13:14:11 +10:00
Martin Brennan fcc2e7ebbf
FEATURE: Promote polymorphic bookmarks to default and migrate (#16729)
This commit migrates all bookmarks to be polymorphic (using the
bookmarkable_id and bookmarkable_type) columns. It also deletes
all the old code guarded behind the use_polymorphic_bookmarks setting
and changes that setting to true for all sites and by default for
the sake of plugins.

No data is deleted in the migrations, the old post_id and for_topic
columns for bookmarks will be dropped later on.
2022-05-23 10:07:15 +10:00
Martin Brennan fbcc35b417
DEV: Remove PostAction/UserAction bookmark refs (#16681)
We have not used anything related to bookmarks for PostAction
or UserAction records since 2020, bookmarks are their own thing
now. Deleting all this is just cleaning up old cruft.
2022-05-10 10:42:18 +10:00
Martin Brennan 222c8d9b6a
FEATURE: Polymorphic bookmarks pt. 3 (reminders, imports, exports, refactors) (#16591)
A bit of a mixed bag, this addresses several edge areas of bookmarks and makes them compatible with polymorphic bookmarks (hidden behind the `use_polymorphic_bookmarks` site setting). The main ones are:

* ExportUserArchive compatibility
* SyncTopicUserBookmarked job compatibility
* Sending different notifications for the bookmark reminders based on the bookmarkable type
* Import scripts compatibility
* BookmarkReminderNotificationHandler compatibility

This PR also refactors the `register_bookmarkable` API so it accepts a class descended from a `BaseBookmarkable` class instead. This was done because we kept having to add more and more lambdas/properties inline and it was very messy, so a factory pattern is cleaner. The classes can be tested independently as well.

Some later PRs will address some other areas like the discourse narrative bot, advanced search, reports, and the .ics endpoint for bookmarks.
2022-05-09 09:37:23 +10:00
Isaac Janzen 2381f18eba
DEV: Convert notify_about_queued_posts_after to accept a float (#16637)
Add support for `notify_about_queued_posts_after` to be set to a float to allow for 15 min increments
2022-05-04 11:33:43 -05:00
Isaac Janzen dcc7f2a55e
DEV: Convert notify_about_flags_after to float (#16633)
Add support for `notify_about_flags_after` to be set to a float.
2022-05-04 11:19:43 -05:00
David Taylor c1db968740
DEV: Move hotlinked image information into a dedicated table (#16585)
This will make future changes to the 'pull hotlinked images' system easier. This commit should not introduce any functional change.

For now, the old post_custom_field data is kept in the database. This will be dropped in a future commit.
2022-05-03 13:53:32 +01:00
Loïc Guitaut 008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
Mark VanLandingham 1e8a666003
DEV: Accept `force_respect_seen_recently` argument in UserEmail job (#16460) 2022-04-18 13:32:11 -05:00
Roman Rizzi 6f76a12e0a
FEATURE: Let sites add a sitemap.xml file. (#16357)
* FEATURE: Let sites add a sitemap.xml file.

This PR adds the same features discourse-sitemap provides to core. Sitemaps are only added to the robots.txt file if the `enable_sitemap` setting is enabled and `login_required` disabled.

After merging discourse/discourse-sitemap#34, this change will take priority over the sitemap plugin because it will disable itself. We're also using the same sitemaps table, so our migration won't try to create it
again using `if_not_exists: true`.
2022-04-12 10:33:59 -03:00
Martin Brennan 9f2138dc92
FEATURE: Add a sidekiq job for syncing S3 ACLs (#16449)
Sometimes we need to update a _lot_ of ACLs on S3 (such as when secure media
is enabled), and since it takes ~1s per upload to update the ACL, this is
best spread out over many jobs instead of having to do the whole thing serially.

In future, it will be better to have a job that can be run based on
a column on uploads (e.g. acl_stale) so we can track progress, similar
to how we can set the baked_version to nil to rebake posts.
2022-04-12 14:26:42 +10:00
David Taylor 39ac476db6 FIX: Do not attempt to pull_hotlinked_image for raw_html
raw_html posts (i.e. those which are pulled as part of our comments integration) don't go through our markdown pipeline, so `upload://` URLs are not supported. Running pull_hotlinked_images will break any images in the post.

In future we may add support for pulling hotlinked images in these posts. But for now, disabling it will stop it breaking images.
2022-04-05 16:39:38 +08:00
Martin Brennan b8828d4a2d
FEATURE: Polymorphic bookmarks pt. 1 (CRUD) (#16308)
This commit introduces a new use_polymorphic_bookmarks site setting
that is default false and hidden, that will be used to help continuous
development of polymorphic bookmarks. This setting **should not** be
enabled anywhere in production yet, it is purely for local development.

This commit uses the setting to enable create/update/delete actions
for polymorphic bookmarks on the server and client side. The bookmark
interactions on topics/posts are all usable. Listing, searching,
sending bookmark reminders, and other edge cases will be handled
in subsequent PRs.

Comprehensive UI tests will be added in the final PR -- we already
have them for regular bookmarks, so it will just be a matter of
changing them to be for polymorphic bookmarks.
2022-03-30 12:43:11 +10:00
Alan Guo Xiang Tan 5743a6ec1e DEV: Remove Zeitwerk inflection monkey patch.
There isn't a good reason we need to patch the inflector.

Co-authored-by: Loïc Guitaut <loic@discourse.org>
2022-03-29 16:04:49 +02:00
Martin Brennan 9d5737fd28
SECURITY: Hide private categories in user activity export (#16273)
In some of the user's own activity export data,
we sometimes showed a secure category's name or
exposed the existence of a secure category.
2022-03-24 15:38:44 +10:00
Bianca Nenciu 30f3e78834
FIX: Reset last sent for existent bookmarks (#16202)
The meaning of reminder_at and reminder_last_sent_at changed after
commit 6d422a8033. A bookmark reminder
will fire only if reminder_last_sent_at is null, but before that it
fired everytime reminder_at was set. This is no longer true because
sometimes reminder_at continues to exist even after a reminder fired.
2022-03-18 16:31:35 +02:00
Bianca Nenciu 6a295ea9e9
DEV: Log more when verbose_upload_logging is enabled (#16177)
A message was logged when download started, but it was not known if a error
during the download.
2022-03-15 23:55:05 +02:00
Penar Musaraj 94750c81fa
FIX: Update group inbox notifications on archive/unarchive (#16152) 2022-03-11 11:57:47 +01:00
Bianca Nenciu 6d422a8033
FEATURE: Highlight expired bookmark reminders (#15317)
The user can select what happens with a bookamrk after it expires. New
option allow bookmark's reminder to be kept even after it has expired.
After a bookmark's reminder notification is created, the reminder date
will be highlighted in red until the user resets the reminder date.
User can do that using the new Clear Reminder button from the dropdown.
2022-03-08 19:44:18 +02:00
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Michael Brown 3bf3b9a4a5 DEV: pull email address validation out to a new EmailAddressValidator
We validate the *format* of email addresses in many places with a match against
a regex, often with very slightly different syntax.

Adding a separate EmailAddressValidator simplifies the code in a few spots and
feels cleaner.

Deprecated the old location in case someone is using it in a plugin.

No functionality change is in this commit.

Note: the regex used at the moment does not support using address literals, e.g.:
* localpart@[192.168.0.1]
* localpart@[2001:db8::1]
2022-02-17 21:49:22 -05:00
Bianca Nenciu 5eaf214594
FEATURE: New plugin API to check if upload is used (#15545)
This commit introduces two new APIs for handling unused uploads, one
can be used to exclude uploads in bulk when the data model allow and
the other one excludes uploads one by one.
2022-02-16 09:00:30 +02:00
Martin Brennan 4086ee551e
DEV: Add bounce_error_code to EmailLog (#15948)
Whenever we got a bounced email in the Email::Receiver we
previously would just set bounced: true on the EmailLog and
discard the status/diagnostic code. This commit changes this
flow to store the bounce error code (defined in the RFC at
https://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xhtml)
not just in the Email::Receiver, but also via webhook events
from other mail services and from SNS.

This commit does not surface the bounce error in the UI,
we can do that later if necessary.
2022-02-15 14:17:26 +10:00
Alan Guo Xiang Tan 6f03b2694d
DEV: Fix typo. (#15857) 2022-02-08 09:04:53 +08:00
David Taylor 15cff27bfe
DEV: Stringify keys of nested hashes in job arguments (#15850)
This provides symmetry with the `.with_indifferent_access` usage in `Jobs#perform`, which is also recursive.
2022-02-07 20:28:45 +00:00
David Taylor c8c23ba557
DEV: Introduce deprecation warning for non-json Job arguments (#15842)
This commit introduces our own handling and warning for Sidekiq's new 'non-json-serializable' warning. This decouples us from Sidekiq's own deprecation cycle, and allows us to use our own deprecation system. It also means that the dump/parse happens in test mode, which will help us to catch occurrences before they reach production.
2022-02-07 17:59:55 +00:00
David Taylor 9ddd1f739e
DEV: Update :critical_user_email calls to use strings (#15827)
Symbols are converted to strings anyway, so there is no change in behaviour. The latest version of sidekiq introduced a warning for this.
2022-02-04 23:43:53 +00:00
David Taylor f53d70ac63
DEV: Ensure `delay_for` and `queue` are not passed as job arguments (#15824)
This regressed in 3a85c4d680 because deep_stringify_keys makes a copy of the `opts` hash
2022-02-04 20:11:03 +00:00
David Taylor c3a54eb30f DEV: Use strings for :user_email job type argument
Job arguments go via JSON, and so symbols will appear as strings in the Job's `#execute` method. The latest version of Sidekiq has started warning about this to reduce developer confusion.
2022-02-04 18:28:18 +00:00
David Taylor 3a85c4d680 DEV: Ensure Sidekiq job arguments have stringified keys
The latest version of Sidekiq introduced a warning when jobs are queued with arguments which 'do not stringify to JSON safely'. In the vast majority of cases, this is because a hash is passed with symbols as keys. When those args are passed to the job, the keys will be stringified.

Our job wrapper already takes care of this issue by calling '.with_indifferent_access' on the args before passing them to `#execute`, so we don't need to change anything about our use. All we need to do is satisfy Sidekiq's warning system by 'stringifying' all the keys before enqueuing the job.
2022-02-04 18:28:18 +00:00
Peter Zhu c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
Martin Brennan 2d68e5d942
FEATURE: Scheduled problem checks for admin dashboard (#15327)
This commit introduces scheduled problem checks for the admin dashboard, which are long running or otherwise cumbersome problem checks that will be run every 10 minutes rather than every time the dashboard is loaded. If these scheduled checks add a problem, the problem will remain until it is cleared or until the scheduled job runs again.

An example of a check that should be scheduled is validating credentials against an external provider.

This commit also introduces the concept of a `priority` to the problems generated by `AdminDashboardData` and the scheduled checks. This is `low` by default, and can be set to `high`, but this commit does not change any part of the UI with this information, only adds a CSS class.

I will be making a follow up PR to check group SMTP credentials.
2021-12-20 09:59:11 +10:00
Roman Rizzi 3602f83cf4
FEATURE: Delete previous reviewable reminders. (#15250)
We send the reminder using the GroupMessage class, which supports removing previous messages. We can't match them by raw because they could mention different moderators. Also, I had to change the subject to remove dynamically generated values, which is necessary for finding them.
2021-12-10 10:17:39 -03:00