Commit Graph

519 Commits

Author SHA1 Message Date
Martin Brennan f49e3e5731
DEV: Add security_last_changed_at and security_last_changed_reason to uploads (#11860)
This PR adds security_last_changed_at and security_last_changed_reason to uploads. This has been done to make it easier to track down why an upload's secure column has changed and when. This necessitated a refactor of the UploadSecurity class to provide reasons why the upload security would have changed.

As well as this, a source is now provided from the location which called for the upload's security status to be updated as they are several (e.g. post creator, topic security updater, rake tasks, manual change).
2021-01-29 09:03:44 +10:00
Martin Brennan 0034cbda8a
DEV: Change Topic Timer from enqueue_at scheduled jobs to incrementally executed jobs (#11698)
Moves the topic timer jobs from being scheduled ahead of time with enqueue_at to a 5 minute scheduled run like bookmark reminders, in a new job called Jobs::EnqueueTopicTimers. Backwards compatibility is maintained by checking if an existing topic timer job is enqueued in sidekiq for the timer, and if it is not running it inside the new job.

The functionality to close/open a topic if it is in the opposite state still remains in the after_save block of TopicTimer, with further commentary, which is used for Open/Close Temporarily.

This also removes the ensure_consistency! functionality of topic timers as it is no longer needed; the new job will always pick up the timers because they are not stored in a fragile state of sidekiq.
2021-01-19 13:30:58 +10:00
Martin Brennan bd25627198
FIX: IMAP post alerter race condition and code improvements (#11711)
This PR fixes a race condition with the IMAP notification code. In the `Email::Receiver` we call the `NewPostManager` to create the post and enqueue jobs and sends alerts via `PostAlerter`. However, if the post alerter reaches the `notify_pm_users` and the `group_notifying_via_smtp` method _before_ the incoming email is updated with the post and topic, we unnecessarily send a notification to the person who just posted. The result of this is that the IMAP syncer re-imports the email sent to the user about their own post, which looks like this in the group inbox:

To fix this, we skip the jobs enqueued by `NewPostManager` and only enqueue them with `PostJobsEnqueuer` manually _after_ the incoming email record has been updated with the post and topic.

Other improvements:

* Moved code to calculate email addresses from `IncomingEmail` records into the topic, with a group passed in, for easier testing and debugging. It is not the responsibility of the post alerter to figure this stuff out.
* Add shortcut methods on `IncomingEmail` to split or provide an empty array for to and cc addresses to avoid repetition.
2021-01-15 10:54:46 +10:00
Martin Brennan 2404fa7a23
DEV: Split toggle topic close job (#11679)
Splits the `ToggleTopicClosed` job into two distinct `OpenTopic` and `CloseTopic` jobs to make the code clearer. The old job cannot be deleted yet because of outstanding sidekiq schedules, so a todo has been added to do so later this year.

Also replaced mentions of `topic_status_update` with `topic_timer` in some files, because the `topic_status_update` model is obsolete and replaced by topic timer.

Added some shortcut methods for checking if a topic is open/whether a user can change an open topic.
2021-01-13 08:49:29 +10:00
David Taylor 13e39d8b9f
PERF: Improve cook_url performance for topic thumbnails (#11609)
- Only initialize the S3Helper when needed
- Skip initializing the S3Helper for S3Store#cdn_url
- Allow cook_url to be passed a `local` hint to skip unnecessary checks
2020-12-30 18:13:13 +00:00
Krzysztof Kotlarek 3ea4f36f26
FIX: use sql_fragment instead of sanitize_sql_array (#11460)
This is a follow up to comment under this PR https://github.com/discourse/discourse/pull/11441

Sam suggested using sql_fragment instead of sanitize_sql_array
2020-12-11 10:56:26 +11:00
Mark VanLandingham 9f786306bd
FIX: Actually error when topic timer time is in the past (#11434) 2020-12-08 11:13:45 -06:00
Krzysztof Kotlarek 9c5ee4923b
FEATURE: silently close topic (#11392)
New TopicTimer to silently close topic. It will be used by discourse-solved plugin

Meta: https://meta.discourse.org/t/allow-auto-close-for-solved-to-do-so-silently/169300
2020-12-03 10:43:19 +11:00
Bianca Nenciu 0863c36221
FIX: Improve errors when invite to topic fails (#11133)
It used to simply say "not allowed" without giving any hint what the
problem could be. This commit refactors the code and tries to improve
readability.
2020-11-06 16:58:10 +02:00
tshenry d778d99b55
FIX: limit_topics_per_day should only apply to regular topics (#11127)
PMs were being evaluted by both the limit_topics_per_day and limit_private_messages_per_day rate limiters when it should only be the latter.
2020-11-04 16:23:49 -08:00
Arpit Jalan 1476e17c35
FEATURE: new setting to create a linked topic on autoclosing mega topics (#11001)
This commit adds a site setting `auto_close_topics_create_linked_topic`
which when enabled works in conjunction with `auto_close_topics_post_count`
setting and creates a new linked topic for the topic just closed.

The auto-created new topic contains a link for all the previous topics
and the topic titles are appended with `(Part {n})`.

The setting is enabled by default.
2020-11-02 12:18:48 +05:30
Arpit Jalan c6bf70c870
DEV: annotate models (#11047) 2020-10-27 23:42:33 +05:30
Roman Rizzi 21c53ed249
FEATURE: Topic slow mode. (#10904)
Adds a new slow mode for topics that are heating up. Users will have to wait for a period of time before being able to post again.

We store this interval inside the topics table and track the last time a user posted using the last_posted_at datetime in the TopicUser relation.
2020-10-16 16:24:38 -03:00
Daniel Waterworth 721ee36425
Replace `base_uri` with `base_path` (#10879)
DEV: Replace instances of Discourse.base_uri with Discourse.base_path

This is clearer because the base_uri is actually just a path prefix. This continues the work started in 555f467.
2020-10-09 12:51:24 +01:00
Krzysztof Kotlarek c934a0f759
FIX: don't error Topic#similar_to when prepared raw is blank (#10711)
If raw contains incorrect URL, `prepare_data` returns empty string:

https://github.com/discourse/discourse/blob/master/lib/search.rb#L91

Therefore we should not only check if the cooked post is not blank but also if prepared data is not blank.
2020-09-22 07:53:12 +10:00
Martin Brennan 5268568d23
FEATURE: Remove user topic timers and migrate to bookmarks with reminders (#10474)
This PR removes the user reminder topic timers, because that system has been supplanted and improved by bookmark reminders. The option is removed from the UI and all existing user reminder topic timers are migrated to bookmark reminders.

Migration does this:

* Get all topic_timers with status_type 5 (reminders)
* Gets all bookmarks where the user ID and topic ID match
* Loops through the found topic timers
  * If there is no bookmark for the OP of the topic, then we just create a bookmark with a reminder
  * If there is a bookmark for the OP of the topic and it does **not** have a reminder set, then just 
update it with the topic timer reminder
  * If there is a bookmark for the OP of the topic with a reminder then just discard the topic timer
* Cancels all outstanding user reminder topic timers
* **Trashes (not deletes) all user reminder topic timers**

Notes:

* For now I have left the user reminder topic timer job class in place; this is so the jobs can be cancelled in the migration. It and the specs will be deleted in the next PR.
* At a later date I will write a migration to delete all trashed user topic timers. They are not deleted here in case there are data issues and they need to be recovered.
* A future PR will change the UI of the topic timer modal to make it look more like the bookmark modal.
2020-09-14 11:11:55 +10:00
Guo Xiang Tan 1e8f216e17
FIX: Ensure autoclose based on last post is executed by system user. 2020-08-26 12:08:46 +08:00
Guo Xiang Tan aae9e6e5fd
FIX: `Topic.similar_to` results in error if cooked raw is blank. 2020-08-21 10:54:08 +08:00
Blake Erickson c2f3c0dc44 FIX: generate_topic_thumbnails job infinitely running for corrupted images
It's possible that the original topic image is broken in some form, so
we shouldn't try and generate a topic thumbnail for it. The fix will
prevent the generate_topic_thumbnails job being enqueued every time the
topic is viewed.
2020-08-13 17:08:32 -06:00
Robin Ward a1ec711373 FIX: Previous fix comitted the wrong thing and was broken :( 2020-08-04 13:46:26 -04:00
Robin Ward 6da7a97eee FIX: Exclude shared drafts from digests 2020-08-04 13:35:48 -04:00
Blake Erickson 8de635fe92
DEV: Show message when cannot invite user to PM (#10336)
* DEV: Show message when cannot invite user to PM

When inviting a user to a PM return a message that says, "Sorry, this
user can't be invited." if they have been muted or are not in a users
allowed pm users list.

* Minor refactor & improved some text
2020-07-31 09:52:19 -06:00
Guo Xiang Tan 4f8262e0d5
FIX: Cooked snippet of raw in `Topic.similar_to`.
If we don't cook the raw, we end up trying to match uncooked raw against
`TopicSearchData#search_data` which consists of cooked raw.
2020-07-28 15:20:18 +08:00
Guo Xiang Tan 4b21b5aac1
FIX: Handle case where `Post#raw` is blank in `Topic.similar_to`. 2020-07-28 13:23:53 +08:00
Guo Xiang Tan dcfe765dac
Make rubocop happy. 2020-07-28 12:06:38 +08:00
Guo Xiang Tan 597d542c33
FIX: Improve `Topic.similar_to` with better `Topic#title` matches.
This changes PG text search to only match the given title against
lexemes that are formed from the title. Likewise, the given raw will
only be matched against lexemes that are formed from the post's raw.
2020-07-28 12:00:27 +08:00
jbrw 0ed784b4fc
FEATURE: Create logs for Group Moderator changes (#10271) 2020-07-21 14:29:02 -04:00
Guo Xiang Tan ff7678e210
FIX: Reindex posts when `Topic#title` or `Category#name` changes. 2020-07-17 11:12:31 +08:00
Blake Erickson cc8540701e FIX: Be sure to use same units when comparing thumbnail size 2020-07-16 14:30:29 -06:00
Blake Erickson 1e9ce51151
FIX: Prevent thumbnail gen if image too large (#10247)
It's possible through an import or other means to have images larger
than the current max allowed image size in the db.

If this happens the thumbnail generation job will keep running
indefinitely trying to download a new copy of the original but
discarding it because it is larger than the max_file_size eventually
causing this error

`Job exception: undefined method `path' for nil:NilClass`

because the newly downloaded image is now nil.

This fix stops the enqueuing of the `GenerateTopicThumbnails` job for
all images that happen to be larger than the max image size.
2020-07-16 11:15:53 +10:00
Bianca Nenciu 4492718864
FIX: Skip whisper posts when updating topic like count (#10157) 2020-07-13 16:30:00 +10:00
Dan Ungureanu c72bc27888
FEATURE: Implement support for IMAP and SMTP email protocols. (#8301)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-07-10 12:05:55 +03:00
Guo Xiang Tan cbe1dd8ec7
Revert "FIX: Delete related search data when record has been deleted."
This reverts commit ecc799ab56.

This commit does not fix anything because we've always been deleting
records in `Searchable`.
2020-07-09 10:08:35 +08:00
Blake Erickson 2e13ed061d FIX: Generate Topic Thumbnails nil to Array error
If any value, including nil, is passed in as an argument the default
won't be set, so we need to handle when a non-Array value is passed in
to the `generate_thumbnails!` method.
2020-07-06 15:35:33 -06:00
David Taylor cb1b472a0f
PERF: Do not include thumbnail information in default topic list payload (#10163)
Now it is only included when a theme/plugin has requested it.
2020-07-06 10:59:21 +01:00
Guo Xiang Tan ecc799ab56
FIX: Delete related search data when record has been deleted. 2020-07-02 12:24:59 +08:00
Bernhard Suttner e31471585a
DEV: allow to have duplicate topic titles if categegory is different (#10034)
Co-authored-by: Robin Ward <robin.ward@gmail.com>

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2020-06-18 11:19:47 -04:00
David Taylor 75b1298e99
DEV: Drop unused image_url column from posts and topics (#9953)
This has been superseded by image_upload_id. The image_url value in API responses is now generated dynamically from the upload record.
2020-06-02 16:21:38 +10:00
Michael Brown d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood 20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Martin Brennan f9d55b4941
FEATURE: Update the topic excerpt when the OP is rebaked (#9852)
* We now have a site setting "topic_excerpt_maxlength" that is used when the OP is created or revised to generate a topic excerpt.
* However, posts created before this setting was introduced cannot benefit from this change unless they are revised, and if the topic excerpt length setting is changed that situation is also not covererd.
* This PR makes a change to rebake! to update the topic excerpt IF the post is the OP.
2020-05-22 13:04:15 +10:00
David Taylor 725e38f9d7
DEV: Allow plugins to request topic thumbnail sizes (#9828)
In plugin.rb, you can register new sizes like

```
register_topic_thumbnail_size [512, 512]
```

For more information about thumbnails see 03818e642a
2020-05-19 10:38:58 +01:00
David Taylor f968155288
FIX: Ensure no image downloads during topic thumbnail serialization
If image uploads are missing a width/height, we re-download them to update the width/height in the database. However, this takes significant resources so we don't want to run this logic during topic serialization.
2020-05-18 12:04:29 +01:00
David Taylor 0495a748d0
FIX: Use CDN URLs for topic thumbnails 2020-05-15 13:35:20 +01:00
Kane York 1059557ce1
DEV: Standardize ignored_columns removal comments (#9771) 2020-05-13 13:08:15 -07:00
David Taylor 03818e642a
FEATURE: Include optimized thumbnails for topics (#9215)
This introduces new APIs for obtaining optimized thumbnails for topics. There are a few building blocks required for this:

- Introduces new `image_upload_id` columns on the `posts` and `topics` table. This replaces the old `image_url` column, which means that thumbnails are now restricted to uploads. Hotlinked thumbnails are no longer possible. In normal use (with pull_hotlinked_images enabled), this has no noticeable impact

- A migration attempts to match existing urls to upload records. If a match cannot be found then the posts will be queued for rebake

- Optimized thumbnails are generated during post_process_cooked. If thumbnails are missing when serializing a topic list, then a sidekiq job is queued

- Topic lists and topics now include a `thumbnails` key, which includes all the available images:
   ```
   "thumbnails": [
   {
     "max_width": null,
     "max_height": null,
     "url": "//example.com/original-image.png",
     "width": 1380,
     "height": 1840
   },
   {
     "max_width": 1024,
     "max_height": 1024,
     "url": "//example.com/optimized-image.png",
     "width": 768,
     "height": 1024
   }
   ]
  ```

- Themes can request additional thumbnail sizes by using a modifier in their `about.json` file:
   ```
    "modifiers": {
      "topic_thumbnail_sizes": [
        [200, 200],
        [800, 800]
      ],
      ...
  ```
  Remember that these are generated asynchronously, so your theme should include logic to fallback to other available thumbnails if your requested size has not yet been generated

- Two new raw plugin outlets are introduced, to improve the customisability of the topic list. `topic-list-before-columns` and `topic-list-before-link`
2020-05-05 09:07:50 +01:00
Sam Saffron d0d5a138c3
DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
Sam Saffron 310a7edee5
DEV: remove unused columns from posts and topics
avg_time on posts and topics have not been used in a year.

This uses a re-runnable ddl transaction diasabled migration to
drop the column, cause it touchs very high traffic table and may
deadlock
2020-04-30 11:22:33 +10:00
Dan Ungureanu e733701887
FEATURE: Make report filters reusable (#9444)
This commit also adds 'include subcategories' report filter
2020-04-22 11:52:50 +03:00
Dan Ungureanu 3d9c320aab
PERF: Cache Category.subcategory_ids (#9350)
Also reset category cache after backup restore.
2020-04-09 15:42:24 +03:00