Commit Graph

1368 Commits

Author SHA1 Message Date
Martin Brennan 0034cbda8a
DEV: Change Topic Timer from enqueue_at scheduled jobs to incrementally executed jobs (#11698)
Moves the topic timer jobs from being scheduled ahead of time with enqueue_at to a 5 minute scheduled run like bookmark reminders, in a new job called Jobs::EnqueueTopicTimers. Backwards compatibility is maintained by checking if an existing topic timer job is enqueued in sidekiq for the timer, and if it is not running it inside the new job.

The functionality to close/open a topic if it is in the opposite state still remains in the after_save block of TopicTimer, with further commentary, which is used for Open/Close Temporarily.

This also removes the ensure_consistency! functionality of topic timers as it is no longer needed; the new job will always pick up the timers because they are not stored in a fragile state of sidekiq.
2021-01-19 13:30:58 +10:00
Martin Brennan 5710d5d771
FIX: Do not process pop3 mails > 1 week old (#11740)
This adds a safe default to not process pop3 emails when the pop3 polling option is set up that are > 1 week old. This is to avoid the situation where an older mailbox is used, which causes us to go and process all emails in that mailbox, sending out error emails to the senders of emails which cannot be parsed successfully.
2021-01-19 09:49:50 +10:00
Krzysztof Kotlarek 06b7c44593
FEATURE: reason to reject user signup (#11700)
Feature for `Must Approve Users` setup. When a user is rejected, a staff member can optionally set a reason for audit purposes. In addition, feedback email can be sent to the user.

Meta: https://meta.discourse.org/t/account-rejection-email/103112/8
2021-01-15 09:43:26 +11:00
Martin Brennan 2404fa7a23
DEV: Split toggle topic close job (#11679)
Splits the `ToggleTopicClosed` job into two distinct `OpenTopic` and `CloseTopic` jobs to make the code clearer. The old job cannot be deleted yet because of outstanding sidekiq schedules, so a todo has been added to do so later this year.

Also replaced mentions of `topic_status_update` with `topic_timer` in some files, because the `topic_status_update` model is obsolete and replaced by topic timer.

Added some shortcut methods for checking if a topic is open/whether a user can change an open topic.
2021-01-13 08:49:29 +10:00
jbrw b1f32f2f57
DEV: TopicTrackingState calls should happen in the background (#11682)
* DEV: TopicTrackingState calls should happen in the background

It was observed that calling TopicTrackingState on popular topics could result in a large number of calls to redis, resulting in slow response times when posting replies.

These calls should be moved to a background job.

* DEV: PostUpdateTopicTrackingState should execute on default queue
2021-01-11 15:58:27 -05:00
Mark VanLandingham 4601f3be7e
FEATURE: Send notification emails when users leave do not disturb mode (#11643) 2021-01-07 10:49:49 -06:00
Martin Brennan 64ba5b1d21
FIX: Group SMTP email improvements (#11633)
Fixes a rare race condition causing the `Imap::Sync` class to create an incoming email and associated post/topic, which then kicks off the PostAlerter to notify others in the PM about a reply in the topic, but for the OP which is not necessary (because the person emailing the IMAP inbox already knows about the OP). Basically, we should never be sending the group SMTP email for the first post in a topic.

Also in this PR:

* Custom attribute accessors for the to/from/cc addresses on `IncomingEmail`, to parse them from an array to a joined string so the logic for this is only in one place.
* Store extra detail against the `IncomingEmail` created in `GroupSmtpMailer`
* regex test Mail header Reply-To as string instead of Field, which fixes `warning: deprecated Object#=~ is called on Mail::Field; it always returns nil`
* Add DEBUG_IMAP to log all IMAP logs as warnings for easier debugging
* Changed the Rails logging to `ImapSyncLog` in the `GroupSmtpMailer`
2021-01-05 15:32:04 +10:00
Bianca Nenciu 755627caa5
FEATURE: Introduce skip_auto_delete_reply_likes site setting (#11562)
osts from topics with 'auto delete replies timer' with more than
skip_auto_delete_reply_likes likes will no longer be deleted. If 0,
all posts will be deleted.
2020-12-23 16:30:10 +11:00
Roman Rizzi 8a7fe3b276
FIX: Don't enqueue imported users when there're multiple custom fields. (#11559)
My initial implementation didn't consider this case. We should skip imported users if the "imported_id" field is present, even if there're other custom fields.
2020-12-22 14:28:07 -03:00
Krzysztof Kotlarek 6dd3f986b2
FIX: no email when the category is muted and mailing list is on (#11490)
When `mute_all_categories_by_default` setting is enabled we should not send mailing list until category, tag or topic is explicitly watched.
2020-12-16 09:30:21 +11:00
Roman Rizzi c7b9f044a4
FEATURE: Automatically disable slow mode. (#11461)
Staff and TL4 users can decide the slow mode duration. We'll internally set a topic timer to disable it.
2020-12-14 14:06:50 -03:00
Vinoth Kannan eb60fc86dc
PERF: run user merging task in a background job. (#10961)
* PERF: run user merging task in a background job.

Currently, admin page is timing out while merging the users with lots of posts.
2020-12-10 15:52:08 +11:00
Kane York 901a45eeb3
FEATURE: Add likes, flags to user data export (#11439)
This commit is dedicated to https://twitter.com/FiloSottile/status/1335666583126073354 for reminding me that like timestamps are valuable data.

Likes additionally include the topic_id and post_number of the acted post, to aid in analysis. Flag export does not include the disposition by staff.
2020-12-09 15:34:13 -08:00
David Taylor c69bb5d5be
DEV: Always enqueue sidekiq jobs after database transaction commit (#11293)
When jobs are enqueued inside a transaction, it's possible that they will be executed before the necessary data is available in the database. This commit ensures all jobs are enqueued in an ActiveRecord after_commit hook.

One potential downside here is if the job fails to enqueue, the transaction will no longer be aborted. However, the chance of that happening is reasonably low, and the impact is significantly lower than the current issue where jobs are scheduled before their data is ready.
2020-12-08 11:05:01 +11:00
Krzysztof Kotlarek 9c5ee4923b
FEATURE: silently close topic (#11392)
New TopicTimer to silently close topic. It will be used by discourse-solved plugin

Meta: https://meta.discourse.org/t/allow-auto-close-for-solved-to-do-so-silently/169300
2020-12-03 10:43:19 +11:00
Krzysztof Kotlarek b7490e894f FIX: don't error-index category job when missing category
When the category is missing we should not throw an error but just finish the job.
2020-12-01 09:30:53 +08:00
Gerhard Schlager 333f0af0ec
FIX: Cached badge_count isn't updated after backfilling badges (#11281) 2020-11-18 22:01:56 +01:00
Joffrey JAFFEUX de174ef0c4
DEV: cap notifications per run at 300 as stated in comment (#11252) 2020-11-17 09:08:12 +10:00
Arpit Jalan fec9d6e578
FIX: when creating linked topics make sure they belong to same category (#11188)
This PR fixes a bug where the newly created linked topic was being
assigned "uncategorized" category. Now we'll assign linked topics the
same category as that of parent topic.

Meta reference: https://meta.discourse.org/t/linked-topics-splitting-and-managing-megatopics/168992/10?u=techapj
2020-11-11 00:44:27 +05:30
Guo Xiang Tan 650da7b626 PERF: Update index for category in a background job.
Search indexing can get expensive and there is no need for us to block
the entire request just to wait for index to finish.
2020-11-09 13:51:26 +08:00
Arpit Jalan 436bd48512
UX: update topic small action post to add link for new linked topic URL (#11132)
https://meta.discourse.org/t/linked-topics-splitting-and-managing-megatopics/168992/4?u=techapj
2020-11-05 22:39:21 +05:30
Arpit Jalan d237ada12f
FIX: enqueue PostCreator jobs outside transaction block (#11120)
The `enqueue_jobs` is not correctly post-processing the post since the
post is being created inside a transaction block. This commit explicitly
enqueues the job outside transaction block.
2020-11-04 21:41:05 +05:30
Arpit Jalan 24d1b1f159
UX: when creating linked topic use URL and let onebox work its magic (#11118)
When the linked topic is created we'll not hardcode the topic title and
let onebox work its magic instead so that the title can be updated
automatically.
2020-11-04 17:19:09 +05:30
David Taylor 5140ec9acf
DEV: Cleanup ignored user logic (#11107)
- IgnoredUser records should all now have an expiring_at value. This commit enforces that in the DB, and fixes any corrupt rows
- Changes to the ignored user list are now handled by the `/u/{username}/notification_level` endpoint. This allows setting expiration dates on the ignore. This commit removes the old logic for saving a list of usernames in the user preferences.
- Many specs were calling `IgnoredUser.create`. This commit changes them to use `Fabricate(:ignored_user)` for consistency
2020-11-03 12:38:54 +00:00
David Taylor 1b7d39fa85
FIX: Remove 4 month limit on IgnoredUser records (#11105)
b8c676e7 added the 'forever' option to the UI, and this is correctly stored in the database. However, we had a hard-coded limit of 4 months in the cleanup job. This commit removes the limit, so ignores can last forever.
2020-11-03 12:12:43 +00:00
Arpit Jalan 1476e17c35
FEATURE: new setting to create a linked topic on autoclosing mega topics (#11001)
This commit adds a site setting `auto_close_topics_create_linked_topic`
which when enabled works in conjunction with `auto_close_topics_post_count`
setting and creates a new linked topic for the topic just closed.

The auto-created new topic contains a link for all the previous topics
and the topic titles are appended with `(Part {n})`.

The setting is enabled by default.
2020-11-02 12:18:48 +05:30
Martin Brennan 632942e697
FIX: Ensure group SMTP and message builder always uses from address for Reply-To when IMAP is enabled (#11037)
There is a site setting reply_by_email_enabled which when combined with reply_by_email_address creates a Reply-To header in emails in the format "test+%{reply_key}@test.com" along with a PostReplyKey record, so when replying Discourse knows where to route the reply.

However this conflicts with the IMAP implementation. Since we are sending the email for a group via SMTP and from their actual email account, we want all replys to go to that email account as well so the IMAP sync job can pick them up and put them in the correct place. So if the group has IMAP enabled and configured, then the reply-to header will be correct.

This PR also makes a further fix to 64b0b50 by using the correct recipient user for the PostReplyKey record. If the post user is used we encounter this error:

if destination.user_id != user.id && !forwarded_reply_key?(destination, user)
  raise ReplyUserNotMatchingError, "post_reply_key.user_id => #{destination.user_id.inspect}, user.id => #{user.id.inspect}"
end
This is because the user above is found from the from_address, but the destination which is the PostReplyKey is made by the post.user, which will be different people.
2020-10-28 07:01:58 +10:00
Kane York e35fcd3340
FEATURE: Include rejected queued posts in the user archive export (#10859)
Requested at https://meta.discourse.org/t/where-can-a-user-find-the-post-that-was-rejected-by-the-moderator/165671?u=riking

Field whitelisting is applied to the json field using Hash#slice, which was activesupport until Ruby 2.5.
2020-10-27 07:48:48 -07:00
Martin Brennan 64b0b50ac0
FIX: Pass user to Email::Sender to avoid broken reply key for group_smtp email (#10978)
Our Email::Sender class accepts an optional user argument, which is used to create a PostReplyKey record when present. This record is used to sub out the %{reply_key} placeholder in the Reply-To mail header, so if we do not pass in the user we get a broken Reply-To header.

This is especially problematic in the IMAP group SMTP situation, because these emails go to customers that we are replying to, and when they reply to us the email bounces! This fixes the issue by passing user to the Email::Sender when sending a group_smtp email but there is still more to do in another PR.

This Email::Sender optional user is a bit of a footgun IMO, especially because most of the time we use it there is a user we can source. I would like to do another PR for this after this one to make the parameter not optional, so we don't end up with these reply issues down the line again.
2020-10-22 10:49:08 +10:00
Bianca Nenciu be5efc9410
FIX: Ensure old uploads can have animated field updated (#10963)
If admins decreased the maximum filesize limit the ActiveRecord
validations would fail.
2020-10-20 19:11:43 +03:00
Bianca Nenciu 43e52a7dc1
DEV: Remove gifsicle dependency (#10357)
Dependency on gifsicle, allow_animated_avatars and allow_animated_thumbnails
site settings were all removed. Animated GIF images are still allowed, but
the generated optimized images are no longer animated for those (which were
used for avatars and thumbnails).

The added 'animated' is populated by extracting information using FastImage.
This field was used to selectively reoptimize old animations. This process
happens in the background.
2020-10-16 13:41:27 +03:00
Martin Brennan c3cede697d
FEATURE: Add weekly bookmark cleanup code (#10899)
When posts or topics are deleted we don't want to immediately delete associated bookmarks, so we have a grace period to recover them and their reminders if the post or topic is un-deleted. This PR adds a task to the Weekly scheduled job to go and delete bookmarks attached to posts or topics deleted > 3 days ago.
2020-10-14 09:38:57 +10:00
Bianca Nenciu 25b8ed740b
DEV: Make site setting type uploaded_image_list use upload IDs (#10401)
It used to be a list of concatenated upload URLs which was prone to
break.
2020-10-13 16:17:06 +03:00
Gerhard Schlager bdbee36961 DEV: Fix typo 2020-10-07 23:43:11 +02:00
David Taylor 16daf3c986
FIX: Compare Jobs::UserEmail type argument as string (#10851)
In specs, symbols are passed through correctly. But in production, all arguments are provided to jobs as strings.

Followup to c0293339b8
2020-10-07 16:55:32 +01:00
David Taylor c0293339b8
PERF: Do not enqueue digest emails when attempted recently (#10849)
Previously, Jobs::EnqueueDigestEmails would enqueue a digest job for every user, even if there are no topics to send. The digest job would exit, no email would send, and last_emailed_at would not change. 30 minutes later, Jobs::EnqueueDigestEmails would run again and re-enqueue jobs for the same users.

120fa8ad introduced a temporary mitigation for this issue, by randomly selecting a subset of those users each time.

This commit adds a new `digest_attempted_at` column to the `user_stats` table. This column is updated every time a digest job completes for a user. Using this, we can avoid scheduling digest jobs for the same user every 30 minutes. This also removes the random user selection in 120fa8ad, and instead prioritizes users who had digests attempted the longest time ago.
2020-10-07 15:30:38 +01:00
Sam 120fa8ad2f
PERF: Introduce absolute limit of digests per 30 minutes (#10845)
To avoid blocking the sidekiq queue a limit of 10,000 digests per 30 minutes
is introduced.

This acts as a safety measure that makes sure we don't keep pouring oil on
a fire.

On multisites it is recommended to set the number way lower so sites do not
dominate the backlog. A reasonable default for multisites may be 100-500.

This can be controlled with the environment var

DISCOURSE_MAX_DIGESTS_ENQUEUED_PER_30_MINS_PER_SITE
2020-10-07 17:30:15 +11:00
Martin Brennan 6e2be3e60b
FIX: When admin changes an email for the user the user must confirm the change (#10830)
See https://meta.discourse.org/t/changing-a-users-email/164512 for additional context.

Previously when an admin user changed a user's email we assumed that they would need a password reset too because they likely did not have access to their account. This proved to be incorrect, as there are other reasons a user needs admin to change their email. This PR:

* Changes the admin change email for user flow so the user is sent an email to confirm the change
* We now record who the email change request was requested by
* If the requested by user is admin and not the user we note this in the email sent to the user
* We also make the confirm change email route open to anonymous users, so it can be clicked by the user even if they do not have access to their account. If there is a logged in user we make sure the confirmation matches the current user.
2020-10-07 13:02:24 +10:00
Kane York 68e87bb58e
User export: profile as json, export auth token logs (#10819)
* FEATURE: Export the entire user profile as json, not just bio/website

* FEATURE: Add session log information to user export

Even though the columns are named 'auth_token' etc, the content is not actually usable to log into the forum with. Despite all that, it is still truncated for export, to avoid any 'token hash cracking' situations.
2020-10-06 15:51:53 -07:00
Roman Rizzi a8c47e7c7f
SECURITY: Ensure users can see the topic before setting a topic timer. (#10841) 2020-10-06 16:49:06 -03:00
Arpit Jalan 3684337e4a FIX: email always settings were not being respected
https://meta.discourse.org/t/email-notification-for-messages/163937/10?u=techapj
2020-09-23 22:00:15 +05:30
Martin Brennan 5268568d23
FEATURE: Remove user topic timers and migrate to bookmarks with reminders (#10474)
This PR removes the user reminder topic timers, because that system has been supplanted and improved by bookmark reminders. The option is removed from the UI and all existing user reminder topic timers are migrated to bookmark reminders.

Migration does this:

* Get all topic_timers with status_type 5 (reminders)
* Gets all bookmarks where the user ID and topic ID match
* Loops through the found topic timers
  * If there is no bookmark for the OP of the topic, then we just create a bookmark with a reminder
  * If there is a bookmark for the OP of the topic and it does **not** have a reminder set, then just 
update it with the topic timer reminder
  * If there is a bookmark for the OP of the topic with a reminder then just discard the topic timer
* Cancels all outstanding user reminder topic timers
* **Trashes (not deletes) all user reminder topic timers**

Notes:

* For now I have left the user reminder topic timer job class in place; this is so the jobs can be cancelled in the migration. It and the specs will be deleted in the next PR.
* At a later date I will write a migration to delete all trashed user topic timers. They are not deleted here in case there are data issues and they need to be recovered.
* A future PR will change the UI of the topic timer modal to make it look more like the bookmark modal.
2020-09-14 11:11:55 +10:00
Kane York e0a0928420
FEATURE: Add bookmarks to the user export (#10591) 2020-09-11 11:03:22 +10:00
Gerhard Schlager ac70c48be4 FIX: Prevent "uploads are missing in S3" alerts after restoring a backup
After restoring a backup it takes up to 48 hours for uploads stored on S3 to appear in the S3 inventory. This change prevents alerts about missing uploads by preventing the EnsureS3UploadsExistence job from running in the first 48 hours after a restore. During the restore it  deletes the count of missing uploads from the PluginStore, so that an alert isn't triggered by an old number.
2020-09-10 21:37:48 +02:00
Guo Xiang Tan 9b75d95fc6 PERF: Keep track of first unread PM and first unread group PM for user.
This optimization helps to filter away topics so that the joins on
related tables when querying for unread messages is not expensive.
2020-09-09 14:05:41 +08:00
Blake Erickson 67dec38f31 FIX: Gravatar download attempt if user is missing their email
It is possible that a user could exist without an email, if so we should
not enqueue a job to download their gravatar.

This commit resolves this error that can occur:

```
Job exception: undefined method `email' for nil:NilClass
/var/www/discourse/app/models/user.rb:1204:in `email'
/var/www/discourse/app/jobs/regular/update_gravatar.rb:12:in `execute'
```

This commit also fixes the original spec which actually was wrong. The
job never enqueued in the original spec and so the gravatar was never
actually updated and the test was checking if the two values were the
same, but they were both null and never updated, so of course they were
the same!

A new test has also been added to make sure the gravatar job isn't
enqueued when a user's email is missing.
2020-09-02 20:19:46 -06:00
Krzysztof Kotlarek 9954a677ab
FIX: don't send mailing list for post with empty content (#10577)
discourse-assign is creating posts with empty content to show that a specific user was assign/unassigned for a specific topic.

It is causing confusing emails with empty content

The bug was mentioned here: https://meta.discourse.org/t/again-on-empty-emails-and-notifications-generated-on-topic-assignment/162213
2020-09-03 08:58:25 +10:00
Guo Xiang Tan 01600492de
FIX: Don't raise error in update username job if user has been deleted. 2020-09-02 11:17:17 +08:00
Kane York 26ec4fd25b
FIX: User export category preferences on a deleted category. (#10573)
Tests from a1dd761bd9 were incomplete and did not test a deleted category's category_users record.
2020-09-01 13:22:59 -07:00
Guo Xiang Tan 069a109cbb
DEV: Require scheduled job in development to avoid loading file twice.
This removes the need to memoize constant in order to avoid the "warning: already initialized constant".
2020-09-01 10:14:40 +08:00
Kane York a1dd761bd9
FIX: Handle deleted categories in post export (#10567)
Fixes a crash when exporting my own archive on Meta.
2020-08-31 17:33:28 -07:00
Kane York 5ec5fbd7ba
User export improvements 2 (#10560)
* FEATURE: Use predictable filenames inside the user archive export

* FEATURE: Include badges in user archive export

* FEATURE: Add user_visits table to the user archive export
2020-08-31 15:26:51 -07:00
Kane York c5dc729e77
FEATURE: Add category tracking state to user archive export (#10557)
Tackling a simple table for the first actual new file in the user archive export.
2020-08-28 13:16:31 -07:00
Kane York 225cdba676 DEV: drop the explicit .each in UserArchive CSV writing
I think this is mostly stylistic, but this helps prevent explosive typos in the enum_for() line.
2020-08-28 11:46:53 -07:00
Kane York 4aed861336 DEV: minor refactors to ExportUserArchive(Spec) 2020-08-28 11:46:53 -07:00
Kane York 7bf199b0c4 DEV: Switch to new ExportUserArchive job
We now use the newly created job class from the previous commit.
2020-08-28 11:46:53 -07:00
Kane York a8560d741f DEV: Create ExportUserArchive as clone of ExportCsvFile
This is in preparation for improvements to the user archive export data.
Some refactors happened along the way, including calling the different _export methods 'components' of the zip file.

Additionally, make the test for post export much more comprehensive.

Copy sources:
  app/jobs/regular/export_csv_file.rb
  spec/jobs/export_csv_file_spec.rb
2020-08-28 11:46:53 -07:00
Guo Xiang Tan a4ec4c06e3
Fix the build. 2020-08-27 13:50:07 +08:00
Guo Xiang Tan 40c6d90df3 PERF: Create a partial regular post_search_data index on large sites.
With the addition of `PostSearchData#private_message`, a partial
index consisting of only search data from regular posts can be created.
The partial index helps to speed up searches on large sites since PG
will not have to do an index scan on the entire search data index which
has shown to be a bottle neck.
2020-08-27 13:42:00 +08:00
Guo Xiang Tan 3cc761ac44
FIX: Clean up toggle closed topic timer when user is not authorized. 2020-08-26 12:59:05 +08:00
Guo Xiang Tan 5bca1aec48
FIX: Clean up topic_timers when no longer valid.
This was causing a bug where the jobs for the topic_timers will keep
getting enqueued over and over again.
2020-08-26 12:18:51 +08:00
Martin Brennan 4670b62969
DEV: IMAP log to database (#10435)
Convert all IMAP logging to write to a database table for easier inspection. These logs are cleaned up daily if they are > 5 days old.

Logs can easily be watched in dev by setting DISCOURSE_DEV_LOG_LEVEL=\"debug\" and running tail -f development.log | grep IMAP
2020-08-14 12:01:31 +10:00
Krzysztof Kotlarek 7194b31443
FEATURE: don't notify about changed tags for a private message (#10408)
* FEATURE: don't notify about changed tags for a private message

Only staff members observing specific tag should receive a notification

* FIX: remove other category which is not used

* FIX: improved specs to ensure that revise was succesful
2020-08-13 17:22:34 +10:00
Régis Hanol bc63232d2e
FIX: sync reviewable count when opening the hamburger menu (#10368)
When a tab is open but left unattended for a while, the red, green, and blue
pills tend to go out of sync.

So whevener we open the notifications menu, we sync up the notification count
(eg. blue and green pills) with the server.

However, the reviewable count (eg. the red pill) is not a notification and
is located in the hamburger menu. This commit adds a new route on the server
side to retrieve the reviewable count for the current user and a ping
(refreshReviewableCount) from the client side to sync the reviewable count
whenever they open the hamburger menu.

REFACTOR: I also refactored the hamburger-menu widget code to prevent repetitive uses
of "this.".

PERF: I improved the performance of the 'notify_reviewable' job by doing only 1 query
to the database to retrieve all the pending reviewables and then tallying based on the
various rights.
2020-08-07 18:13:02 +02:00
David Taylor cb12a721c4
REFACTOR: Refactor pull_hotlinked_images job
This commit should cause no functional change
- Split into functions to avoid deep nesting
- Register custom field type, and remove manual json parse/serialize
- Recover from deleted upload records

Also adds a test to ensure pull_hotlinked_images redownloads secure images only once
2020-08-05 12:14:59 +01:00
Gerhard Schlager 957e851ffe Revert "FIX: Regularly reset unknown extension of uploads"
This reverts commit cc7b24b88b as it shouldn't be needed anymore for new uploads.
2020-08-03 13:37:32 +02:00
Régis Hanol da7e6b1f7e DEV: make rubocop 🤖 happy 2020-07-31 18:16:54 +02:00
Régis Hanol 3f59d0cffe
DEV: prevent initialized constant warning
Was tired of seeing the following warnings in the logs

```
/discourse/app/jobs/scheduled/old_keys_reminder.rb:7: warning: already initialized constant Jobs::OldKeysReminder::OLD_CREDENTIALS_PERIOD
/discourse/app/jobs/scheduled/old_keys_reminder.rb:7: warning: previous definition of OLD_CREDENTIALS_PERIOD was here
```
2020-07-31 18:05:12 +02:00
David Taylor 4ef389bc04
FIX: Define s3_helper in EnsureS3UploadsExistence job
Followup to 16c65a94
2020-07-29 12:23:04 +01:00
David Taylor 16c65a94f7
PERF: Preload S3 inventory data for multisite clusters 2020-07-29 10:31:55 +01:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Guo Xiang Tan c6202af005
Update rubocop to 2.3.1. 2020-07-24 17:19:21 +08:00
Guo Xiang Tan 3766122a82
DEV: Allow developmental post search index versions. 2020-07-23 15:19:46 +08:00
Guo Xiang Tan 609ba50fe8
DEV: Add more granularity to `SearchIndexer` versions.
Sometimes, we just want to reindex a specific model and not all the
things.
2020-07-23 14:24:06 +08:00
jbrw 06073fe8c6
FEATURE: Allow group moderators to close/archive topics
* FEATURE: Allow group moderators to close/archive topics
2020-07-14 12:36:19 -04:00
Robin Ward a73da42691 FIX: Don't award new user of the month in the wrong month
see: https://meta.discourse.org/t/new-user-of-the-month-badge-awarded-before-registraton-date/157347/2?u=eviltrout
2020-07-13 15:05:30 -04:00
Dan Ungureanu c72bc27888
FEATURE: Implement support for IMAP and SMTP email protocols. (#8301)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-07-10 12:05:55 +03:00
Penar Musaraj 67582e7d27
FIX: Do not send system emails to suspended users (#10192) 2020-07-08 13:30:32 -04:00
Vinoth Kannan 9426d12c1d FEATURE: trigger `user_updated` event if username is changed. 2020-07-03 00:14:19 +05:30
Sam Saffron 9ffc022cf4
DEV: improve verbose mode for reindexer
This makes the verbose mode provide a bit of progress notification
while reindexing as it can take many hours to do a giant site
2020-06-24 17:29:45 +10:00
Sam Saffron dcad720a4c
DEV: add optional verbose logging to re-index job
This verbose logging can be useful when executing the job by hand
for debugging purposes

In general people will not use this
2020-06-24 15:37:08 +10:00
David Taylor d473ddd02c
FIX: Use raw content when checking for watched words to flag (#10104)
This makes flagged watched words match the behavior of censor, requires_approval and blocked watched words.
2020-06-22 19:48:06 +01:00
Guo Xiang Tan 3370ef188e
FEATURE: Remove deprecated uploads url site settings.
The site settings have been replaced with direct image upload since
Discourse 2.3.
2020-06-22 14:32:29 +08:00
David Taylor 17c4f76eac
FIX: Do not attempt to pull_hotlinked on emoji images when CDN enabled (#10091) 2020-06-19 20:21:05 +01:00
David Taylor a99bb0ded4
Revert "FIX: Do not attempt to pull_hotlinked on emoji images when CDN enabled"
This changed cause plugin spec failures and needs further investigation

This reverts commit 78626d2832.
2020-06-19 14:39:16 +01:00
David Taylor 9f2e7e4651
FIX: Handle invalid URLs gracefully when pulling hotlinked images 2020-06-19 12:52:51 +01:00
David Taylor 78626d2832
FIX: Do not attempt to pull_hotlinked on emoji images when CDN enabled 2020-06-19 12:45:06 +01:00
Dan Ungureanu 5bfe1ee4f1
FEATURE: Improve UX support for multiple email addresses (#9691) 2020-06-10 19:11:49 +03:00
Dan Ungureanu 3a7ca97c36
FIX: Use include-subcategories filter in report export (#10007)
Some filters were renamed and the conversion of the filter names and arguments
was removed.
2020-06-10 18:57:39 +03:00
Arpit Jalan 3094459cd9
FEATURE: multiple use invite links (#9813) 2020-06-09 20:49:32 +05:30
Robin Ward 4e3a84c687 FIX: If creating a flag for a watched word, include the reason 2020-06-02 11:49:02 -04:00
David Taylor 3106f85983
FIX: Support exporting reports which reference topics (#9957) 2020-06-01 18:23:58 +01:00
Krzysztof Kotlarek 9a6ef80739
FEATURE: notify admins about old credentials (#9918)
* FEATURE: notify admins about old credentials

Security and API keys should be renewed periodically.
This additional notification should help admins keep their Discourse safe and secure.
2020-06-01 13:49:27 +10:00
David Taylor ecfce93f28
FIX: Support IRIs (unicode URIs) when pulling hotlinked images (#9928) 2020-05-29 17:47:05 +01:00
David Taylor 28f46c171c
FIX: Pull hotlinked images even when edited by system users (#9890)
Previously the pull hotlinked images job was skipped after system edits. This ensured that we never had an infinite loop of system-edit/pull-hotlinked/system-edit/pull-hotlinked etc.

A side effect was that edits made by system for any other reason (e.g. API, removing full quotes) would prevent pulling hotlinked images. This commit removes the system edit check, and replaces it with another method to avoid an infinite job scheduling loop.
2020-05-29 13:07:47 +01:00
Andrew Schleifer 74d28a43d1
new S3 backup layout (#9830)
* DEV: new S3 backup layout

Currently, with $S3_BACKUP_BUCKET of "bucket/backups", multisite backups
end up in "bucket/backups/backups/dbname/" and single-site will be in
"bucket/backups/".

Both _should_ be in "bucket/backups/dbname/"

- remove MULTISITE_PREFIX,
- always include dbname,
- method to move to the new prefix
- job to call the method

* SPEC: add tests for `VacateLegacyPrefixBackups` onceoff job.

Co-authored-by: Vinoth Kannan <vinothkannan@vinkas.com>
2020-05-29 00:28:23 +05:30
David Taylor 8a3d9d7036
DEV: Run jobs sequentially in test mode (#9897)
When running jobs in tests, we use `Jobs.run_immediately!`. This means that jobs are run synchronously when they are enqueued. Jobs sometimes enqueue other jobs, which are also executed synchronously. This means that the outermost job will block until the inner jobs have finished executing. In some cases (e.g. process_post with hotlinked images) this can lead to a deadlock.

This commit changes the behavior slightly. Now we will never run jobs inside other jobs. Instead, we will queue them up and run them sequentially in the order they were enqueued. As a whole, they are still executed synchronously. Consider the example

```ruby
class Jobs::InnerJob < Jobs::Base
  def execute(args)
    puts "Running inner job"
  end
end

class Jobs::OuterJob < Jobs::Base
  def execute(args)
    puts "Starting outer job"
    Jobs.enqueue(:inner_job)
    puts "Finished outer job"
  end
end

Jobs.enqueue(:outer_job)
puts "All jobs complete"
```

The old behavior would result in:

```
Starting outer job
Running inner job
Finished outer job
All jobs complete
```

The new behavior will result in:
```
Starting outer job
Finished outer job
Running inner job
All jobs complete
```
2020-05-28 12:52:27 +01:00
Blake Erickson a788427b46
FIX: broken query if upload id is missing (#9900)
Log warnings if there are any errors trying to upload group flair image
and ensure upload.id exists before trying to use it in a sql query.
2020-05-27 12:39:30 -06:00
Régis Hanol 2a4db15544 FIX: don't send digests to users with no primary email
It might happen that some User records have no associated primary emails.
In which case we don't ever want to send them a digest.

Also added a new "user_email_no_email" skipped email log to ensure these cases
are properly handled and surfaced.
2020-05-27 17:09:40 +02:00
Krzysztof Kotlarek 34e5f0a9a3
Revert "FEATURE: notify admins about old credentials (#9854)" (#9886)
This reverts commit 349a67bee6.
2020-05-27 09:52:53 +10:00
Krzysztof Kotlarek 349a67bee6
FEATURE: notify admins about old credentials (#9854)
* FEATURE: notify admins about old credentials

Security and API keys should be renewed periodically.
This additional notification should help admins keep their Discourse safe and secure.
2020-05-27 08:13:47 +10:00
Vinoth Kannan 5fb9271878
DEV: ignore `flair_url` column in group model. (#9873) 2020-05-26 00:43:50 +05:30
Vinoth Kannan 505122bb45 FIX: skip onceoff job for groups with invalid flair URL. 2020-05-25 13:11:00 +05:30
Vinoth Kannan 8e56197728
UX: use "icon-picker" & "image-uploader" fields to set group flair. (#9779) 2020-05-25 11:08:47 +05:30
Kane York 869f9b20a2
PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
Sam Saffron 451e9c4bb9
DEV: minor SQL formatting change
Moved join prior to left join to make query less confusing.

Has no material impact on performance.
2020-05-12 16:55:42 +10:00
Rafael dos Santos Silva 4d137b4f89
FIX: Handle a deleted topic in thumbnail generation
Follow up to 03818e642a

A topic can be deleted between job enqueue and job execution
2020-05-11 13:32:37 -03:00
Krzysztof Kotlarek a3e1e9ef16
FIX: remove deprecated URI.escape (#9697)
During Nokogumbo changes I introduced back URI.escape which is deprecated.
2020-05-08 11:14:59 +10:00
Martin Brennan fa572d3a7a
DEV: Remove all code referencing at_desktop bookmark reminders (#9650)
We have found no need for these reminder types, so we are removing the code for them.
2020-05-06 15:22:43 +10:00
David Taylor 03818e642a
FEATURE: Include optimized thumbnails for topics (#9215)
This introduces new APIs for obtaining optimized thumbnails for topics. There are a few building blocks required for this:

- Introduces new `image_upload_id` columns on the `posts` and `topics` table. This replaces the old `image_url` column, which means that thumbnails are now restricted to uploads. Hotlinked thumbnails are no longer possible. In normal use (with pull_hotlinked_images enabled), this has no noticeable impact

- A migration attempts to match existing urls to upload records. If a match cannot be found then the posts will be queued for rebake

- Optimized thumbnails are generated during post_process_cooked. If thumbnails are missing when serializing a topic list, then a sidekiq job is queued

- Topic lists and topics now include a `thumbnails` key, which includes all the available images:
   ```
   "thumbnails": [
   {
     "max_width": null,
     "max_height": null,
     "url": "//example.com/original-image.png",
     "width": 1380,
     "height": 1840
   },
   {
     "max_width": 1024,
     "max_height": 1024,
     "url": "//example.com/optimized-image.png",
     "width": 768,
     "height": 1024
   }
   ]
  ```

- Themes can request additional thumbnail sizes by using a modifier in their `about.json` file:
   ```
    "modifiers": {
      "topic_thumbnail_sizes": [
        [200, 200],
        [800, 800]
      ],
      ...
  ```
  Remember that these are generated asynchronously, so your theme should include logic to fallback to other available thumbnails if your requested size has not yet been generated

- Two new raw plugin outlets are introduced, to improve the customisability of the topic list. `topic-list-before-columns` and `topic-list-before-link`
2020-05-05 09:07:50 +01:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Kane York 784bf2a173
FIX: Unassign user titles when a badge is deleted (#9573) 2020-05-02 18:02:28 -07:00
Sam Saffron d0d5a138c3
DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
Dan Ungureanu 7a946b9f5c
DEV: Fix typo
Follow-up to 0a85a7aef9.
2020-04-29 12:38:57 +03:00
Dan Ungureanu 0a85a7aef9
FEATURE: Add user_profile to user_archive CSV export (#9571) 2020-04-29 12:09:50 +03:00
Roman Rizzi 394babcae3
FIX: Only show the review page to users that can see it. Do not publish the reviewable count update message to everyone. (#9556) 2020-04-27 14:51:25 -03:00
Guo Xiang Tan 16f0d1c503
PERF: Remove database query when publishing to staff users. 2020-04-27 11:50:21 +08:00
Jarek Radosz e0aec23844 DEV: Don't warn about valid data 2020-04-26 22:18:44 +02:00
Vinoth Kannan df0c386f8a
UX: drop the `automatic_membership_retroactive` column from groups model. (#9430) 2020-04-22 22:07:39 +05:30
Martin Brennan 628ba9d1e2
FEATURE: Promote bookmarks with reminders to core functionality (#9369)
The main thrust of this PR is to take all the conditional checks based on the `enable_bookmarks_with_reminders` away and only keep the code from the `true` path, making bookmarks with reminders the core bookmarks feature. There is also a migration to create `Bookmark` records out of `PostAction` bookmarks for a site.

### Summary

* Remove logic based on whether enable_bookmarks_with_reminders is true. This site setting is now obsolete, the old bookmark functionality is being removed. Retain the setting and set the value to `true` in a migration.
* Use the code from the rake task to create a database migration that creates bookmarks from post actions.
* Change the bookmark report to read from the new table.
* Get rid of old endpoints for bookmarks
* Link to the new bookmarks list from the user summary page
2020-04-22 13:44:19 +10:00
Jarek Radosz 5a81e3999c
DEV: Remove `bypass_bump` from CookedPostProcessor (#9468)
It was only passing it along to `PullHotlinkedImages` and that class have not used that arg since April 2014 (c52ee665b4)
2020-04-21 03:48:19 +02:00
Robin Ward 25bed4f643 FIX: Concurrency issues with making topic embedded posts visible 2020-04-20 15:11:59 -04:00
David Taylor b6c19cba20
FIX: Abort emit_web_hook_event job cleanly if web hook was deleted (#9445)
Raising an error causes the job to be retried, and causes a lot of noise in the logs
2020-04-16 21:24:09 +01:00
Neil Lalonde 074509fd95
FIX: don't demote users to TL2 when default trust level is 3
Within 24 hours of signing up, new users were losing their
default trust level of 3. With this fix, demotions from
trust level 3 won't happen when the "default trust level"
setting is 3 or 4.
2020-04-16 12:28:16 -04:00
Robin Ward ce663d67e1 FIX: CSV Exports were throwing errors with invalid dates
This fix will consider any invalid dates to be non-existant.
2020-04-07 14:05:27 -04:00
David Taylor 3cfe086a94
FIX: Limit personal message participants when converting from topic (#9343)
Previously all topic posters would be added which could lead to major performance issues. Now if there are too many posters, only the acting user will be added as a participant.
2020-04-03 16:42:01 +01:00
Daniel Waterworth 76bb01c31c DEV: Increase process_post lock timeout
The process_post job uses CookedPostProcessor which also uses a
DistributedMutex. There's no good reason for the timeout of the outer
lock to be smaller than the timeout of the inner lock.
2020-04-03 13:34:44 +01:00
Vinoth Kannan 82201fa466 FIX: jobs/delete_replies: Add Time+Duration, not Time+Time #9314
Co-authored-by: Kane York <kanepyork@gmail.com>
2020-04-03 09:23:40 +05:30
Joffrey JAFFEUX aeaea3c154
FIX: correctly take category/group filters into csv export (#9300) 2020-03-30 19:08:47 +02:00
Joffrey JAFFEUX 96e841a635
DEV: trigger :username_changed when a user is renamed (#9301) 2020-03-30 18:02:34 +02:00
Martin Brennan 097851c135
FIX: Change secure media to encompass attachments as well (#9271)
If the “secure media” site setting is enabled then ALL files uploaded to Discourse (images, video, audio, pdf, txt, zip etc. etc.) will follow the secure media rules. The “prevent anons from downloading files” setting will no longer have any bearing on upload security. Basically, the feature will more appropriately be called “secure uploads” instead of “secure media”.

This is being done because there are communities out there that would like all attachments and media to be secure based on category rules but still allow anonymous users to download attachments in public places, which is not possible in the current arrangement.
2020-03-26 07:16:02 +10:00
Vinoth Kannan aad12822b7
FEATURE: automatically delete replies on a topic after N days. (#9209) 2020-03-19 21:06:31 +05:30
Roman Rizzi 27bc4f51c7
FIX: Ignore suspect users that were migrated or users who were created more than six months ago (#9205) 2020-03-14 08:47:53 -03:00
Martin Brennan 793f39139a
FEATURE: Send notifications for time-based and At Desktop bookmark reminders (#9071)
* This PR implements the scheduling and notification system for bookmark reminders. Every 5 minutes a schedule runs to check any reminders that need to be sent before now, limited to **300** reminders at a time. Any leftover reminders will be sent in the next run. This is to avoid having to deal with fickle sidekiq and reminders in the far-flung future, which would necessitate having a background job anyway to clean up any missing `enqueue_at` reminders.

* If a reminder is sent its `reminder_at` time is cleared and the `reminder_last_sent_at` time is filled in. Notifications are only user-level notifications for now.

* All JavaScript and frontend code related to displaying the bookmark reminder notification is contained here. The reminder functionality is now re-enabled in the bookmark modal as well.

* This PR also implements the "Remind me next time I am at my desktop" bookmark reminder functionality. When the user is on a mobile device they are able to select this option. When they choose this option we set a key in Redis saying they have a pending at desktop reminder. The next time they change devices we check if the new device is desktop, and if it is we send reminders using a DistributedMutex. There is also a job to ensure consistency of these reminders in Redis (in case Redis drops the ball) and the at desktop reminders expire after 20 days.

* Also in this PR is a fix to delete all Bookmarks for a user via `UserDestroyer`
2020-03-12 10:16:00 +10:00
Roman Rizzi b9aaa9718d
FIX: When must_approve_users is enabled, we don't want to send suspect users to the review queue. Only non-approved users should be sent. Provide a migration to auto-approve every problematic review item (#9179) 2020-03-11 17:05:44 -03:00
Roman Rizzi 826b4793c0
FEATURE: Approve suspect users is now true by default. The suspect users list was removed (#9151) 2020-03-10 08:56:42 -03:00
Jarek Radosz 85e03a7f68
DEV: Replace `Time.new` with `Time.now` (#9142)
(or `Time.zone.now`)
2020-03-09 17:37:49 +01:00
Gerhard Schlager edc8d58ac3 FEATURE: Add site setting to disable staged user cleanup
... and disabled the cleanup during imports, otherwise a running Sidekiq might delete users before posts are created
2020-03-09 10:26:41 +01:00
Bianca Nenciu f14dd1f82d
FIX: Prevent race condition when post processing post (#8819)
If a post is being cooked twice (for example after an edit), there is a
chance the 'raw' and 'cooked' column to be inconsistent. This reduces
the chances of that happening.
2020-03-07 14:36:54 +02:00
Kane York 10ddb8a9c4 FIX: Use destroy_all instead of delete_all for shared drafts
Rails has an odd behavior for calling .delete_all on a has_many relation - the
default behavior is to nullify the foreign key fields instead of actually
'DELETE'ing the records.

Additionally, publishing a shared draft topic creates a PostRevision that the
NotifyPostRevision job picks up which is then promptly deleted.

Use destroy_all when cleaning up the revisions and have the NotifyPostRevision
job tolerate deleted PostRevision records.

This takes a small performance hit (several SQL DELETEs instead of just one)
but shouldn't be too much of an issue (high cardinalities range from 30-100).
2020-03-05 11:13:43 -08:00
tshenry a09e5d12c2
FIX: Topics should honor auto-close when published to category (#8963)
* FIX: Topics should honor auto-close when published to category

* Add test
2020-03-02 14:21:35 -05:00
Vinoth Kannan 5774107a2d
FIX: downloaded image URLs incorrectly replaced in post raw. (#9014)
Previously, while replacing the downloaded image URL `http://wiki.mozilla.org/images/2/2e/Longcat1.png` similar non-image URL `http://wiki.mozilla.org/images/2` was replaced wrongly.
2020-02-27 10:22:55 +05:30
Sam Saffron 372f6f4f22
FEATURE: limit number of notifications per user to 10,000
Introduces a new site setting `max_notifications_per_user`.

Out-of-the-box this is set to 10,000. If a user exceeds this number of
notifications, we will delete the oldest notifications keeping only 10,000.

To disable this safeguard set the setting to 0.

Enforcement happens weekly.

This is in place to protect the system from pathological states where a
single user has enormous amounts of notifications causing various queries
to time out. In practice nobody looks back more than a few hundred notifications.
2020-02-24 11:42:50 +11:00
Dan Ungureanu cf0c6d5761
FIX: Ensure web hooks are retried at most 5 times 2020-02-21 17:02:40 +02:00
Robin Ward 345764565f FIX: Respect muted tags for mailing list mode
If a user has a tag muted, don't send them emails about that tag.
We've done this forever for categories so it makes sense to do it
for tags too.
2020-02-19 15:14:42 -05:00
Robin Ward c954d083df Link website when reviewing users 2020-02-19 10:18:05 -05:00
Roman Rizzi 9441362c72
FEATURE: Support uploading a csv with either user emails or usernames (#8971) 2020-02-18 10:53:12 -03:00
Sam Saffron 28292d2759
PERF: avoid shelling to get hostname aggressively
Previously we had many places in the app that called `hostname` to get
hostname of a server. This commit replaces the pattern in 2 ways

1. We cache the result in `Discourse.os_hostname` so it is only ever called once

2. We prefer to use Socket.gethostname which avoids making a shell command

This improves performance as we are not spawning hostname processes throughout
the app lifetime
2020-02-18 15:13:19 +11:00
Roman Rizzi 2ee6a615b7
FEATURE: Send suspect users to the review queue (#8811) 2020-01-29 15:38:27 -03:00
Martin Brennan ab3bda6cd0
FIX: Mitigate issue where legacy pre-secure hotlinked media would not be redownloaded (#8802)
Basically, say you had already downloaded a certain image from a certain URL
using pull_hotlinked_images and the onebox. The upload would be stored
by its sha as an upload record. Whenever you linked to the same URL again
in a post (e.g. in our case an og:image on review.discourse) we would
would reuse the original upload record because of the sha1.

However when you turned on secure media this could cause problems as
the first post that uses that upload after secure media is enabled
will set the access control post for the upload to the new post.
Then if the post is deleted every single onebox/link to that same image
URL will fail forever with 403 as the secure-media-uploads URL fails
if the access control post has been deleted.

To fix this when cooking posts and pulling hotlinked images, we only
allow using an original upload by URL if its access control post
matches the current post, and if the original_sha1 is filled in,
meaning it was uploaded AFTER secure media was enabled. otherwise
we just redownload the media again to be safe, as the URL will always
be new then.
2020-01-29 10:11:38 +10:00
Krzysztof Kotlarek 20e7fb1c95
FIX: correct notification when tag or category is added (#8801)
Regression was created here:
https://github.com/discourse/discourse/pull/8750

When tag or category is added and the user is watching that category/tag
we changed notification type to `edited` instead of `new post`.

However, the logic here should be a little bit more sophisticated.

If the user has already seen the post, notification should be `edited`.

However, when user hasn't yet seen post, notification should be "new
reply". The case for that is when for example topic is under private
category and set for publishing later. In that case, we modify an
existing topic, however, for a user, it is like a new post.

Discussion on meta:
https://meta.discourse.org/t/publication-of-timed-topics-dont-trigger-new-topic-notifications/139335/13
2020-01-29 11:03:47 +11:00
Gerhard Schlager 9621af2214 FIX: Award "User of the month" badge at the beginning of month
Previously the badge was granted one month after the last time the badge was granted. The exact date shifted by one day each month. The new logic tries to grant the badge always at the beginning of a new month by looking at new users of the previous month. The "granted at" date is set to the end of the previous month.
2020-01-28 22:59:13 +01:00
David Taylor a9d0d55817 FEATURE: Add message to log when admins are automatically deactivated 2020-01-28 12:16:24 +00:00
Dan Ungureanu 5060811464
FIX: Export all category names in user archives (#8790) 2020-01-28 12:52:07 +02:00
Gerhard Schlager ea11ad4d99 DEV: Drop unused columns 2020-01-27 15:28:56 +01:00
Jarek Radosz 8a82ceb3bc
FIX: Improve downsize_uploads (#8409)
With this change the script:
* Actually removes original large-sized images
* Doesn't save processed files if their size has increased
* Prevents inconsistent state
2020-01-27 03:31:11 +01:00
Martin Brennan 45b37a8bd1
FIX: Resolve pull hotlinked image and broken link issues for secure media URLs (#8777)
When pull_hotlinked_images tried to run on posts with secure media (which had already been downloaded from external sources) we were getting a 404 when trying to download the image because the secure endpoint doesn't allow anon downloads.

Also, we were getting into an infinite loop of pull_hotlinked_images because the job didn't consider the secure media URLs as "downloaded" already so it kept trying to download them over and over.

In this PR I have also refactored secure-media-upload URL checks and mutations into single source of truth in Upload, adding a SECURE_MEDIA_ROUTE constant to check URLs against too.
2020-01-24 11:59:30 +10:00
Martin Brennan 1b3b0708c0
FEATURE: Update upload security status on post move, topic conversion, category change (#8731)
Add TopicUploadSecurityManager to handle post moves. When a post moves around or a topic changes between categories and public/private message status the uploads connected to posts in the topic need to have their secure status updated, depending on the security context the topic now lives in.
2020-01-23 12:01:10 +10:00
Bianca Nenciu 7b7e1717f2
FIX: Quoting a quote preserves the original post information (#8746)
Let's say post #2 quotes post number #1. If a user decides to quote the
quote in post #2, it should keep the information of post #1
("user_1, post: 1, topic: X"), instead of replacing with current post
info ("user_2, post: 2, topic: X").
2020-01-22 16:10:23 +02:00
Bianca Nenciu 0a27086764
FEATURE: Export all types of reports (#8748)
There is a single stacked_chart which was not exportable
2020-01-21 18:43:19 +02:00
Krzysztof Kotlarek 0420be88a6
FIX: when tag or category is added notify users that topic was modified (#8750)
There is a feature, that when tag or category is added to the topic,
customers who are watching that category or tag are notified.

The problem is that it is using default notification type "new post"

It would be better to use "new post" only when there really is a new
post and "edited" when categories or tags were modified.
2020-01-21 08:41:13 +11:00
Gerhard Schlager f216c6d60b FEATURE: Drop "backup" schema 7 days after restore
The "backup" schema is used to rollback a failed restore. It isn't useful after a longer period of time and turns into a waste of disk space.
2020-01-16 17:48:47 +01:00
Martin Brennan 7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
Gerhard Schlager c351ffe580 FEATURE: Add hidden setting to disable configuration of inventory bucket 2020-01-14 17:23:12 +01:00
David Taylor cff6e941de
PERF: Cache ranks for featured badges, to simplify user serialization (#8698) 2020-01-14 14:26:49 +00:00
Roman Rizzi d69c5eebcf
Feature: Mass award badge (#8694)
* UI: Mass grant a badge from the admin ui

* Send the uploaded CSV and badge ID to the backend

* Read the CSV and grant badge in batches

* UX: Communicate the result to the user

* Don't award if badge is disabled

* Create a 'send_notification' method to remove duplicated code, slightly shrink badge image. Replace router transition with href.

* Dynamically discover current route
2020-01-13 11:20:26 -03:00
Martin Brennan edbc356593
FIX: Replace deprecated URI.encode, URI.escape, URI.unescape and URI.unencode (#8528)
The following methods have long been deprecated in ruby due to flaws in their implementation per http://blade.nagaokaut.ac.jp/cgi-bin/vframe.rb/ruby/ruby-core/29293?29179-31097:

URI.escape
URI.unescape
URI.encode
URI.unencode
escape/encode are just aliases for one another. This PR uses the Addressable gem to replace these methods with its own encode, unencode, and encode_component methods where appropriate.

I have put all references to Addressable::URI here into the UrlHelper to keep them corralled in one place to make changes to this implementation easier.

Addressable is now also an explicit gem dependency.
2019-12-12 12:49:21 +10:00
Sam Saffron 0c52537f10 DEV: update rubocop to version 0.77
We like to stay as close as possible to latest with rubocop cause the cops
get better.

This update required some code changes, specifically the default is to avoid
explicit returns where implicit is done

Also this renames a few rules
2019-12-10 11:48:39 +11:00
Joffrey JAFFEUX 0d3d2c43a0
DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
Martin Brennan e7c7a05097
FIX: Mark secure media upload insecure automatically if used for theme component (#8413)
When uploading a file to a theme component, and that file is existing and has already been marked as secure, we now automatically mark the file as secure: false, change the ACL, and log the action as the user (also rebake the posts for the upload)
2019-11-28 07:32:17 +10:00
David Taylor e1fcbf4aef DEV: Remove new_version_emails global setting reference
All site settings are now shadowed by global settings, so there is no need to lookup the global setting explicitly
2019-11-20 15:54:09 +00:00
Martin Brennan 02cb01406e
FIX: Allow secure uploads if global s3 setting active and enable_s3_uploads validations (#8373)
The secure media functionality relied on `SiteSetting.enable_s3_uploads?` which, as we found in dev, did not take into account global S3 settings via `GlobalSetting.use_s3?`. We now use `SiteSetting.Upload.enable_s3_uploads` instead to be more consistent.

Also, we now validate `enable_s3_uploads` changes, because if `GlobalSetting.use_s3?` is true users should NOT be enabling S3 uploads manually.
2019-11-20 07:46:44 +10:00
Krzysztof Kotlarek 975165f25f
FIX: Don't error CleanUpInactiveUserJob when user is missing (#8362) 2019-11-18 16:14:15 +11:00
Penar Musaraj 102909edb3 FEATURE: Add support for secure media (#7888)
This PR introduces a new secure media setting. When enabled, it prevent unathorized access to media uploads (files of type image, video and audio). When the `login_required` setting is enabled, then all media uploads will be protected from unauthorized (anonymous) access. When `login_required`is disabled, only media in private messages will be protected from unauthorized access. 

A few notes: 

- the `prevent_anons_from_downloading_files` setting no longer applies to audio and video uploads
- the `secure_media` setting can only be enabled if S3 uploads are already enabled and configured
- upload records have a new column, `secure`, which is a boolean `true/false` of the upload's secure status
- when creating a public post with an upload that has already been uploaded and is marked as secure, the post creator will raise an error
- when enabling or disabling the setting on a site with existing uploads, the rake task `uploads:ensure_correct_acl` should be used to update all uploads' secure status and their ACL on S3
2019-11-18 11:25:42 +10:00
Penar Musaraj 067696df8f DEV: Apply Rubocop redundant return style 2019-11-14 15:10:51 -05:00
David Taylor 836b3f4d82
FIX: Do not deactivate admin accounts with recent posts or api keys (#8342)
This prevents 'bot' users being deactivated
2019-11-12 16:56:01 +00:00
Mark VanLandingham 3dd2f2f701
DEV: Remove RSS feed polling in favor of plugin (#8233) 2019-11-12 09:49:02 -06:00
Krzysztof Kotlarek 69266f60ed FIX: tag and category watchers regression (#8336)
I made a regression here 17366d3bcc (diff-ddeebb36d131f89ca91be9d04c2baefaR10)

When the tag is added, people watching specific tag are notified but also people watching specific category.

Therefore, `notify_post_users` should accept options who should be notified.

So when `category` is added to the topic, users watching topic and users watching category are notified.

When `tag` is added to the topic, users watching topic and users watching tag are notified

Finally, when a new post is created, everybody is notified, topic watchers, category watchers, tag watchers.
2019-11-12 16:44:46 +11:00
Martin Brennan 56d3e29a69
FIX: Badge and user title interaction fixes (#8282)
* Fix user title logic when badge name customized
* Fix an issue where a user's title was not considered a badge granted title when the user used a badge for their title and the badge name was customized. this affected the effectiveness of revoke_ungranted_titles! which only operates on badge_granted_titles.
* When a user's title is set now it is considered a badge_granted_title if the badge name OR the badge custom name from TranslationOverride is the same as the title
* When a user's badge is revoked we now also revoke their title if the user's title matches the badge name OR the badge custom name from TranslationOverride
* Add a user history log when the title is revoked to remove confusion about why titles are revoked
* Add granted_title_badge_id to user_profile, now when we set badge_granted_title on a user profile when updating a user's title based on a badge, we also remember which badge matched the title
* When badge name (or custom text) changes update titles of users in a background job
* When the name of a badge changes, or in the case of system badges when their custom translation text changes, then we need to update the title of all corresponding users who have a badge_granted_title and matching granted_title_badge_id. In the case of system badges we need to first get the proper badge ID based on the translation key e.g. badges.regular.name
* Add migration to backfill all granted_title_badge_ids for both normal badge name titles and titles using custom badge text.
2019-11-08 15:34:24 +10:00
Krzysztof Kotlarek 17366d3bcc
FEATURE: notify tag watchers when tag was added to post (#8299)
Issue was mentioned in this [meta topic](https://meta.discourse.org/t/send-a-notification-to-watching-users-when-adding-tag/125314)

It is working well when category is changed because NotifyCategoryChange job already got that code:
```
if post&.topic&.visible?
  post_alerter = PostAlerter.new
  post_alerter.notify_post_users(post, User.where(id: args[:notified_user_ids]))
  post_alerter.notify_first_post_watchers(post, post_alerter.category_watchers(post.topic))
end
```

For NotifyTagChange job notify post users were missing so it worked only when your notification was set to `watching first post`
2019-11-07 08:20:15 +11:00
David Taylor 52c5cf33f8
FEATURE: Overhaul of admin API key system (#8284)
- Allow revoking keys without deleting them
- Auto-revoke keys after a period of no use (default 6 months)
- Allow multiple keys per user
- Allow attaching a description to each key, for easier auditing
- Log changes to keys in the staff action log
- Move all key management to one place, and improve the UI
2019-11-05 14:10:23 +00:00
Arpit Jalan b7327d2c34 UX: show user email address on "grant admin access" email and UI 2019-11-04 14:47:00 +05:30
Arpit Jalan 2ae71b7a87 FEATURE: allow sending bulk invites to staged users 2019-10-30 11:40:03 +05:30
Krzysztof Kotlarek c32bd8ae48 FEATURE: Remove attachments and truncate raw field for incoming emails (#8253)
Adds the settings: 

raw_email_max_length, raw_rejected_email_max_length, delete_rejected_email_after_days. 

These settings control retention of the "raw" emails logs.

raw_email_max_length ensures that if we get incoming email that is huge we will truncate it removing uploads from the raw log.

raw_rejected_email_max_length introduces an even more aggressive truncation for rejected incoming mail. 

delete_rejected_email_after_days controls how many days we will keep rejected emails for (default 90)
2019-10-30 16:54:35 +11:00
Sam Saffron 3d85cc1e69 PERF: run expensive clean up uploads less frequently
Previously every hour we would run a full scan of the entire DB searching
for expired uploads that need to be moved to the tombstone folder.

This commit amends it so we only run the job 2 times per clean_orpha_uploads_grace_period_hours

There is a upper bound of 7 days so even if the grace period is set really
high it will still run at least once a week.

By default we have a 48 grace period so this amends it to run this cleanup
daily instead of hourly. This eliminates 23 times we run this ultra expensive
query.
2019-10-28 11:14:52 +11:00
Dan Ungureanu 1358312584
FIX: Zeitwerk-related fixes for jobs. (#8219) 2019-10-21 20:25:35 +03:00
Daniel Waterworth 55a1394342 DEV: pluck_first
Doing .pluck(:column).first is a very common pattern in Discourse and in
most cases, a limit cause isn't being added. Instead of adding a limit
clause to all these callsites, this commit adds two new methods to
ActiveRecord::Relation:

pluck_first, equivalent to limit(1).pluck(*columns).first

and pluck_first! which, like other finder methods, raises an exception
when no record is found
2019-10-21 12:08:20 +01:00
Joffrey JAFFEUX b6ef473a31
DEV: prevents redefinition of various constants (#8199)
CLEANUP_GRACE_PERIOD
MAX_AWARDED
POLL_MAILBOX_TIMEOUT_ERROR_KEY
2019-10-17 09:18:07 +02:00
Rafael dos Santos Silva 6e9c8fe854
FIX: More encoded slug fixes (#8191)
* FIX: Do not encode the URL twice

Now that we encode slugs in the server we don't need this anymore.

Reverts fe5na33

* FIX: More places do deal with encoded slugs

* the param is a string now, not a hash

* FIX: Handle the nil slug on /categories

* DEV: Add seeded? method to identity default categories

* DEV: Use SiteSetting to keep track of seeded categories
2019-10-16 17:08:43 -03:00
Penar Musaraj e4424d7c2e FIX: Do not make notification API call if push_url is blank
Followup to 77643931bc
2019-10-16 10:59:06 -04:00
Dan Ungureanu 086b46051c
FIX: Zeitwerk-related fixes for jobs. (#8187) 2019-10-14 13:03:22 +03:00
Rafael dos Santos Silva 76ab0350f1
FIX: Properly encoded slugs when configured to (#8158)
When an admin changes the site setting slug_generation_method to
encoded, we weren't really encoding the slug, but just allowing non-ascii
characters in the slug (unicode).

That brings problems when a user posts a link to topic without the slug, as
our topic controller tries to redirect the user to the correct URL that contains
the slug with unicode characters. Having unicode in the Location header in a
response is a RFC violation and some browsers end up in a redirection loop.

Bug report: https://meta.discourse.org/t/-/125371?u=falco

This commit also checks if a site uses encoded slugs and clear all saved slugs
in the db so they can be regenerated using an onceoff job.
2019-10-11 12:38:16 -03:00
Penar Musaraj 77643931bc FIX: Ensure push_url exists before making push notification API call 2019-10-04 11:52:10 -04:00
Roman Rizzi 10565e4623
SECURITY: Safely decompress files. (#8124)
* FEATURE: Adds an extra protection layer when decompressing files.

* Rename exporter/importer to zip importer. Update old locale

* Added a new composite class to decompress a file with multiple strategies

* Set max file size inside a site setting

* Ensure that file is deleted after compression

* Sanitize path and files before compressing/decompressing
2019-10-03 10:19:35 -03:00
Krzysztof Kotlarek 427d54b2b0 DEV: Upgrading Discourse to Zeitwerk (#8098)
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains. 

We no longer need to use Rails "require_dependency" anywhere and instead can just use standard 
Ruby patterns to require files.

This is a far reaching change and we expect some followups here.
2019-10-02 14:01:53 +10:00
Robin Ward d251f12c9c Tweak calculation for reviewable sensitivities/priorities
Previously, calculating thresholds for reviewables was done based on the
50th and 85th percentile across all reviewables. However, many forum
owners provided feedback that these thresholds were too easy to hit, in
particular when it came to auto hiding content.

The calculation has been adjusted to base the priorities on reviewables
that have a minimum of 2 scores (flags). This should push the amount of
flags required to hide something higher then before.
2019-09-19 14:07:56 -04:00
Robin Ward d5b52abf2f FIX: Require a min amount of reviewables before calculating thresholds
On forums with very few flags you don't want to calculate averages
because they won't be very useful. Stick with the defaults until we hit
15 reviewables at least.
2019-09-19 13:42:50 -04:00
Roman Rizzi 568232052e
DEV: Remove FlagQuery class and old code (#8064) 2019-09-12 13:21:33 -03:00
David Taylor 67a98946b8 FIX: Do not log 'pull_hotlinked_images' edits in the staff action log 2019-09-12 15:55:45 +01:00
Gerhard Schlager 56c6e8e961 FIX: POP3 doesn't work with TLS 1.3 2019-09-11 18:43:19 +02:00