Commit Graph

546 Commits

Author SHA1 Message Date
Roman Rizzi 60059a7190
FEATURE: A low priority filter for the review queue. (#12822)
This filter hides reviewables with a score lower than the "reviewable_low_priority_threshold" setting. We only use reviewables that already met this threshold to calculate the Medium and High priority filters.
2021-04-23 15:34:24 -03:00
Gerhard Schlager 92e222d246
FIX: POP3 polling shouldn't stop after exception or old email (#12742) 2021-04-19 10:27:29 +02:00
Osama Sayegh a23d0f9961
UX: Add image uploader widget for uploading badge images (#12377)
Currently the process of adding a custom image to badge is quite clunky; you have to upload your image to a topic, and then copy the image URL and pasting it in a text field. Besides being clucky, if the topic or post that contains the image is deleted, the image will be garbage-collected in a few days and the badge will lose the image because the application is not that the image is referenced by a badge.

This commit improves that by adding a proper image uploader widget for badge images.
2021-03-17 08:55:23 +03:00
David Taylor 4430bc153d
FIX: Do not clean up uploads when they're used by theme settings (#12326)
We intend to move ThemeSetting to use an upload_id column, rather than storing the URL. So this is a short-term solution.
2021-03-09 19:16:45 +00:00
Gerhard Schlager a96a5db0fb
DEV: Add option to send system message to groups (#12256) 2021-03-02 18:51:50 +01:00
Roman Rizzi 4e716e9ce5
FIX: Reduce the time_read threshold to one minute. (#12159)
* FIX: Reduce the time_read threshold to one minute.

Five minutes is too much and could fill the queue with false positives.

* Update spec/jobs/enqueue_suspect_users_spec.rb

Co-authored-by: Arpit Jalan <arpit@techapj.com>

Co-authored-by: Arpit Jalan <arpit@techapj.com>
2021-02-20 08:25:32 -03:00
Roman Rizzi 95fb363c2a
FEATURE: Use the "time_read" stat to flag users as suspicious. (#12145)
Completing the discobot tutorial gives you ~3m of reading time, so we set the limit at 5m. Additionally, we use an "OR" clause to cover the case when you just scroll through a single topic.
2021-02-19 13:10:19 -03:00
Krzysztof Kotlarek ad3ec5809f
FIX: Dismiss new with better migration (#12062)
Original PR was reverted because of broken migration https://github.com/discourse/discourse/pull/12058

I fixed it by adding this line
```
          AND topics.id IN(SELECT id FROM topics ORDER BY created_at DESC LIMIT :max_new_topics)
```

This time it is left joining a limited amount of topics. I tested it on few databases and it worked quite smooth
2021-02-15 08:50:33 +11:00
Penar Musaraj 900d4187ef
DEV: Prevents rate limits for new feature checks on multisite (#12053) 2021-02-12 08:52:59 -05:00
Krzysztof Kotlarek a696cc07d2
Revert "FEATURE: Ability to dismiss all new topics (#12018)" (#12058)
This reverts commits 7426764af4 and f5b18e2a31
2021-02-12 08:50:25 +11:00
Krzysztof Kotlarek f5b18e2a31
FEATURE: Ability to dismiss all new topics (#12018)
Follow up https://github.com/discourse/discourse/pull/11968

Dismiss all new topics using the same DismissTopicService. In addition, MessageBus receives exact topic ids which should be marked as `seen`.
2021-02-11 13:35:09 +11:00
Krzysztof Kotlarek f39e7fe81d
FEATURE: New way to dismiss new topics (#11927)
This is a try to simplify logic around dismiss new topics to have one solution to work in all places - dismiss all-new, dismiss new in a specific category or even in a specific tag.
2021-02-04 11:27:34 +11:00
Penar Musaraj 49e97279c7 FEATURE: Add daily job to check for new features 2021-02-01 10:31:44 +08:00
Mark VanLandingham 809274fe0d
DEV: Replace 'processed' column on notifications with new table (#11864) 2021-01-27 10:29:24 -06:00
Régis Hanol aa1138ff71
FIX: reindex_search job should work on model with no search data (#11819)
Lots of changes but it's mostly a refactoring.

The interesting part that was fix are the 'load_problem_<model>_ids' methods.
They will now return records with no search data associated so they can be properly indexed for the search.
This "bad" state usually happens after a migration.
2021-01-25 11:23:36 +01:00
Mark VanLandingham 1a7922bea2
FEATURE: Create notification schedule to automatically set do not disturb time (#11665)
This adds a new table UserNotificationSchedules which stores monday-friday start and ends times that each user would like to receive notifications (with a Boolean enabled to remove the use of the schedule). There is then a background job that runs every day and creates do_not_disturb_timings for each user with an enabled notification schedule. The job schedules timings 2 days in advance. The job is designed so that it can be run at any point in time, and it will not create duplicate records.

When a users saves their notification schedule, the schedule processing service will run and schedule do_not_disturb_timings. If the user should be in DND due to their schedule, the user will immediately be put in DND (message bus publishes this state).

The UI for a user's notification schedule is in user -> preferences -> notifications. By default every day is 8am - 5pm when first enabled.
2021-01-20 10:31:52 -06:00
Martin Brennan fb184fed06
DEV: Add created_via column to IncomingEmail (#11751)
This should make it easier to track down how the incoming email was created, which is one of four locations:

The POP3 poller (which picks up reply via email replies)
The admin email controller #handle_mail (which is where hosted mail is sent)
The IMAP sync tool
The group SMTP mailer, which sends emails when replying to IMAP topics, pre-emptively creating IncomingEmail records to avoid double syncing
2021-01-20 13:22:41 +10:00
Martin Brennan 0034cbda8a
DEV: Change Topic Timer from enqueue_at scheduled jobs to incrementally executed jobs (#11698)
Moves the topic timer jobs from being scheduled ahead of time with enqueue_at to a 5 minute scheduled run like bookmark reminders, in a new job called Jobs::EnqueueTopicTimers. Backwards compatibility is maintained by checking if an existing topic timer job is enqueued in sidekiq for the timer, and if it is not running it inside the new job.

The functionality to close/open a topic if it is in the opposite state still remains in the after_save block of TopicTimer, with further commentary, which is used for Open/Close Temporarily.

This also removes the ensure_consistency! functionality of topic timers as it is no longer needed; the new job will always pick up the timers because they are not stored in a fragile state of sidekiq.
2021-01-19 13:30:58 +10:00
Martin Brennan 5710d5d771
FIX: Do not process pop3 mails > 1 week old (#11740)
This adds a safe default to not process pop3 emails when the pop3 polling option is set up that are > 1 week old. This is to avoid the situation where an older mailbox is used, which causes us to go and process all emails in that mailbox, sending out error emails to the senders of emails which cannot be parsed successfully.
2021-01-19 09:49:50 +10:00
Mark VanLandingham 4601f3be7e
FEATURE: Send notification emails when users leave do not disturb mode (#11643) 2021-01-07 10:49:49 -06:00
Roman Rizzi 8a7fe3b276
FIX: Don't enqueue imported users when there're multiple custom fields. (#11559)
My initial implementation didn't consider this case. We should skip imported users if the "imported_id" field is present, even if there're other custom fields.
2020-12-22 14:28:07 -03:00
Gerhard Schlager 333f0af0ec
FIX: Cached badge_count isn't updated after backfilling badges (#11281) 2020-11-18 22:01:56 +01:00
Joffrey JAFFEUX de174ef0c4
DEV: cap notifications per run at 300 as stated in comment (#11252) 2020-11-17 09:08:12 +10:00
David Taylor 5140ec9acf
DEV: Cleanup ignored user logic (#11107)
- IgnoredUser records should all now have an expiring_at value. This commit enforces that in the DB, and fixes any corrupt rows
- Changes to the ignored user list are now handled by the `/u/{username}/notification_level` endpoint. This allows setting expiration dates on the ignore. This commit removes the old logic for saving a list of usernames in the user preferences.
- Many specs were calling `IgnoredUser.create`. This commit changes them to use `Fabricate(:ignored_user)` for consistency
2020-11-03 12:38:54 +00:00
David Taylor 1b7d39fa85
FIX: Remove 4 month limit on IgnoredUser records (#11105)
b8c676e7 added the 'forever' option to the UI, and this is correctly stored in the database. However, we had a hard-coded limit of 4 months in the cleanup job. This commit removes the limit, so ignores can last forever.
2020-11-03 12:12:43 +00:00
Bianca Nenciu be5efc9410
FIX: Ensure old uploads can have animated field updated (#10963)
If admins decreased the maximum filesize limit the ActiveRecord
validations would fail.
2020-10-20 19:11:43 +03:00
Bianca Nenciu 43e52a7dc1
DEV: Remove gifsicle dependency (#10357)
Dependency on gifsicle, allow_animated_avatars and allow_animated_thumbnails
site settings were all removed. Animated GIF images are still allowed, but
the generated optimized images are no longer animated for those (which were
used for avatars and thumbnails).

The added 'animated' is populated by extracting information using FastImage.
This field was used to selectively reoptimize old animations. This process
happens in the background.
2020-10-16 13:41:27 +03:00
Martin Brennan c3cede697d
FEATURE: Add weekly bookmark cleanup code (#10899)
When posts or topics are deleted we don't want to immediately delete associated bookmarks, so we have a grace period to recover them and their reminders if the post or topic is un-deleted. This PR adds a task to the Weekly scheduled job to go and delete bookmarks attached to posts or topics deleted > 3 days ago.
2020-10-14 09:38:57 +10:00
Bianca Nenciu 25b8ed740b
DEV: Make site setting type uploaded_image_list use upload IDs (#10401)
It used to be a list of concatenated upload URLs which was prone to
break.
2020-10-13 16:17:06 +03:00
Gerhard Schlager bdbee36961 DEV: Fix typo 2020-10-07 23:43:11 +02:00
David Taylor c0293339b8
PERF: Do not enqueue digest emails when attempted recently (#10849)
Previously, Jobs::EnqueueDigestEmails would enqueue a digest job for every user, even if there are no topics to send. The digest job would exit, no email would send, and last_emailed_at would not change. 30 minutes later, Jobs::EnqueueDigestEmails would run again and re-enqueue jobs for the same users.

120fa8ad introduced a temporary mitigation for this issue, by randomly selecting a subset of those users each time.

This commit adds a new `digest_attempted_at` column to the `user_stats` table. This column is updated every time a digest job completes for a user. Using this, we can avoid scheduling digest jobs for the same user every 30 minutes. This also removes the random user selection in 120fa8ad, and instead prioritizes users who had digests attempted the longest time ago.
2020-10-07 15:30:38 +01:00
Sam 120fa8ad2f
PERF: Introduce absolute limit of digests per 30 minutes (#10845)
To avoid blocking the sidekiq queue a limit of 10,000 digests per 30 minutes
is introduced.

This acts as a safety measure that makes sure we don't keep pouring oil on
a fire.

On multisites it is recommended to set the number way lower so sites do not
dominate the backlog. A reasonable default for multisites may be 100-500.

This can be controlled with the environment var

DISCOURSE_MAX_DIGESTS_ENQUEUED_PER_30_MINS_PER_SITE
2020-10-07 17:30:15 +11:00
Gerhard Schlager ac70c48be4 FIX: Prevent "uploads are missing in S3" alerts after restoring a backup
After restoring a backup it takes up to 48 hours for uploads stored on S3 to appear in the S3 inventory. This change prevents alerts about missing uploads by preventing the EnsureS3UploadsExistence job from running in the first 48 hours after a restore. During the restore it  deletes the count of missing uploads from the PluginStore, so that an alert isn't triggered by an old number.
2020-09-10 21:37:48 +02:00
Guo Xiang Tan 9b75d95fc6 PERF: Keep track of first unread PM and first unread group PM for user.
This optimization helps to filter away topics so that the joins on
related tables when querying for unread messages is not expensive.
2020-09-09 14:05:41 +08:00
Guo Xiang Tan 069a109cbb
DEV: Require scheduled job in development to avoid loading file twice.
This removes the need to memoize constant in order to avoid the "warning: already initialized constant".
2020-09-01 10:14:40 +08:00
Guo Xiang Tan a4ec4c06e3
Fix the build. 2020-08-27 13:50:07 +08:00
Guo Xiang Tan 40c6d90df3 PERF: Create a partial regular post_search_data index on large sites.
With the addition of `PostSearchData#private_message`, a partial
index consisting of only search data from regular posts can be created.
The partial index helps to speed up searches on large sites since PG
will not have to do an index scan on the entire search data index which
has shown to be a bottle neck.
2020-08-27 13:42:00 +08:00
Martin Brennan 4670b62969
DEV: IMAP log to database (#10435)
Convert all IMAP logging to write to a database table for easier inspection. These logs are cleaned up daily if they are > 5 days old.

Logs can easily be watched in dev by setting DISCOURSE_DEV_LOG_LEVEL=\"debug\" and running tail -f development.log | grep IMAP
2020-08-14 12:01:31 +10:00
Gerhard Schlager 957e851ffe Revert "FIX: Regularly reset unknown extension of uploads"
This reverts commit cc7b24b88b as it shouldn't be needed anymore for new uploads.
2020-08-03 13:37:32 +02:00
Régis Hanol da7e6b1f7e DEV: make rubocop 🤖 happy 2020-07-31 18:16:54 +02:00
Régis Hanol 3f59d0cffe
DEV: prevent initialized constant warning
Was tired of seeing the following warnings in the logs

```
/discourse/app/jobs/scheduled/old_keys_reminder.rb:7: warning: already initialized constant Jobs::OldKeysReminder::OLD_CREDENTIALS_PERIOD
/discourse/app/jobs/scheduled/old_keys_reminder.rb:7: warning: previous definition of OLD_CREDENTIALS_PERIOD was here
```
2020-07-31 18:05:12 +02:00
David Taylor 4ef389bc04
FIX: Define s3_helper in EnsureS3UploadsExistence job
Followup to 16c65a94
2020-07-29 12:23:04 +01:00
David Taylor 16c65a94f7
PERF: Preload S3 inventory data for multisite clusters 2020-07-29 10:31:55 +01:00
Guo Xiang Tan 3766122a82
DEV: Allow developmental post search index versions. 2020-07-23 15:19:46 +08:00
Guo Xiang Tan 609ba50fe8
DEV: Add more granularity to `SearchIndexer` versions.
Sometimes, we just want to reindex a specific model and not all the
things.
2020-07-23 14:24:06 +08:00
Robin Ward a73da42691 FIX: Don't award new user of the month in the wrong month
see: https://meta.discourse.org/t/new-user-of-the-month-badge-awarded-before-registraton-date/157347/2?u=eviltrout
2020-07-13 15:05:30 -04:00
Sam Saffron 9ffc022cf4
DEV: improve verbose mode for reindexer
This makes the verbose mode provide a bit of progress notification
while reindexing as it can take many hours to do a giant site
2020-06-24 17:29:45 +10:00
Sam Saffron dcad720a4c
DEV: add optional verbose logging to re-index job
This verbose logging can be useful when executing the job by hand
for debugging purposes

In general people will not use this
2020-06-24 15:37:08 +10:00
Guo Xiang Tan 3370ef188e
FEATURE: Remove deprecated uploads url site settings.
The site settings have been replaced with direct image upload since
Discourse 2.3.
2020-06-22 14:32:29 +08:00
Krzysztof Kotlarek 9a6ef80739
FEATURE: notify admins about old credentials (#9918)
* FEATURE: notify admins about old credentials

Security and API keys should be renewed periodically.
This additional notification should help admins keep their Discourse safe and secure.
2020-06-01 13:49:27 +10:00