Commit Graph

943 Commits

Author SHA1 Message Date
Vinoth Kannan 4c6bfb9b39 DEV: Don't destroy webhook in case of error 2019-03-21 18:34:54 +05:30
Tarek Khalil a31a35b334 FEATURE: Ignored user notification behaviour should be as a muted user (#7227) 2019-03-21 12:15:34 +01:00
Tarek Khalil fed2dd9148 FEATURE: Add scheduled job to purge expired ignored users (#7211) 2019-03-20 11:01:43 +01:00
Régis Hanol 31e06dbcd2 FIX: SES webhook wasn't parsing the message 2019-03-19 11:40:19 +01:00
Guo Xiang Tan 4020c87680 DEV: Refactor tests for `Jobs::CleanUpInactiveUsers`.
* Remove use of 0 in favor of `TrustLevel.levels[:newuser]`.
* Consolidate two tests into a single one.
* Test that disabling the feature works.
* Avoid loading full ActiveRecord object in test when we only need to
know the existence of the record.
2019-03-19 09:57:21 +08:00
Bianca Nenciu 2347661a74 FEATURE: Clean up inactive users. (#7172) 2019-03-18 16:25:15 +01:00
Bianca Nenciu 5114ef958a FIX: Do not trigger post alerts for empty posts. (#7138) 2019-03-15 17:58:43 +01:00
Penar Musaraj 9334d2f4f7
FEATURE: add more granular user option levels for email notifications (#7143)
Migrates email user options to a new data structure, where `email_always`, `email_direct` and `email_private_messages` are replace by

* `email_messages_level`, with options: `always`, `only_when_away` and `never` (defaults to `always`)
* `email_level`, with options: `always`, `only_when_away` and `never` (defaults to `only_when_away`)
2019-03-15 10:55:11 -04:00
Tarek Khalil bd6d31c9ec
FEATURE: Add `IgnoredUsersSummary` daily job (#7144)
* FEATURE: Add `IgnoredUsersSummary` daily job

## Why?

This is part of the [Ability to ignore a user feature](https://meta.discourse.org/t/ability-to-ignore-a-user/110254/8).

We want to:

1. Send an automatic group PM that goes out to moderators
2. When {x} users have Ignored the same user, threshold defined by a site setting, default of 5
3. Only send this message every X days which is defined by another site setting
2019-03-14 22:51:43 +00:00
Robin Ward fa5a158683 REFACTOR: Move `queue_jobs` out of `SiteSetting`
It is not a setting, and only relevant in specs. The new API is:

```
Jobs.run_later!        # jobs will be thrown on the queue
Jobs.run_immediately!  # jobs will run right away, avoid the queue
```
2019-03-14 10:47:38 -04:00
Guo Xiang Tan b0c8fdd7da FIX: Properly support defaults for upload site settings. 2019-03-13 16:36:57 +08:00
Bianca Nenciu c6ed86220e FIX: Notify on tag change. (#7119) 2019-03-12 18:09:34 +01:00
Guo Xiang Tan 34b29f62db DEV: Remove the use of stubs and mocks in `Jobs::UserEmail` tests.
We can only be sure that an email is sent when we get a mailer in
`ActionMailer::Deliveries`. A couple of tests were actually incorrect
because it didn't flow through our email sender where there are more
conditions in determining whether an email is sent or not.
2019-03-12 09:39:16 +08:00
Dan Ungureanu 32bae48fd3 DEV: Use User#human? User#bot? (#7140) 2019-03-12 07:58:14 +08:00
David Taylor 0a4562253e DEV: Add 'starting' event to sidekiq log when interval logging enabled 2019-03-08 10:56:36 +00:00
David Taylor e2510d79cc DEV: Improve thread-safety of sidekiq logging 2019-03-08 10:31:49 +00:00
David Taylor 9db05a895a DEV: Add job_id to sidkiq log
This makes it easier to correlate 'pending' logs from the same job
2019-03-08 09:16:13 +00:00
Gerhard Schlager 1121514799 UX: Localize date format in "new user of the month" message 2019-03-06 21:58:25 +01:00
David Taylor df474bceee DEV: Further sidekiq logging stability improvements
- Open the log file in "append" mode. This avoids issues if the file does not exist (and matches standard rails log behavior)
- Correctly parse the interval logging environment variable
2019-03-06 12:50:15 +00:00
David Taylor fe62de68dd DEV: Correct sidekiq logging to avoid thread leak 2019-03-06 10:11:31 +00:00
David Taylor 8b30ed5b7a DEV: Serialize the job parameters in sidekiq logs
Otherwise this can lead to some very large data structures when processing the logs later
2019-03-05 17:44:49 +00:00
David Taylor 8963f1af30
FEATURE: Optional detailed performance logging for Sidekiq jobs (#7091)
By default, this does nothing. Two environment variables are available:

- `DISCOURSE_LOG_SIDEKIQ`

  Set to `"1"` to enable logging. This will log all completed jobs to `log/rails/sidekiq.log`, along with various db/redis/network statistics. This is useful to track down poorly performing jobs.

- `DISCOURSE_LOG_SIDEKIQ_INTERVAL`

  (seconds) Check running jobs periodically, and log their current duration. They will appear in the logs with `status:pending`. This is useful to track down jobs which take a long time, then crash sidekiq before completing.
2019-03-05 11:19:11 +00:00
Régis Hanol 326d892f5e Aadd 'secondary_emails' field in users export
FIX: escape_comma wasn't working in CSV exports
FIX: group_names field wasn't properly serialized
2019-02-27 10:12:20 +01:00
Régis Hanol e3a23116d2
FIX: don't update gravatar if the user has no email 2019-02-20 22:34:43 +01:00
Régis Hanol fc14847c14 PERF: only require aws-sdk-sns gem when it's being used 2019-02-14 11:08:21 +01:00
Vinoth Kannan 484bd82278 FIX: Add onceoff job to remove double quotes from s3 etags 2019-02-14 05:19:41 +05:30
Régis Hanol 4d674acc25 FEATURE: AWS SNS bounce notifications webhooks 2019-02-13 21:26:40 +01:00
Vinoth Kannan b4f713ca52
FEATURE: Use amazon s3 inventory to manage upload stats (#6867) 2019-02-01 10:10:48 +05:30
Robin Ward 78ddc82952 FIX: Respect min_flags_staff_visibility for new flags too
There was a situation where if:

* There were new flags to review that met the visibility threshold

AND

* There were old flags that *didn't* meet the threshold

THEN

a pending flags notification would be sent out. This fixes that case.
Staff should not be notified of flags if they do not meet the threshold
and are old.
2019-01-25 11:27:43 -05:00
Robin Ward 96b2585a91 REFACTOR: Remove unncessary stubs from pending flags reminder
They seem to be calculated fine by the application, and stubbing
makes the tests more brittle and prone to regression.
2019-01-24 13:45:58 -05:00
Robin Ward f32de88dfc FIX: Don't notify of pending flags if min_flags_staff_visibility not met 2019-01-22 11:01:18 -05:00
Guo Xiang Tan 95e3841974 FIX: Remove old reference and use `MiniScheduler::Stat`. 2019-01-18 16:36:11 +08:00
Sam 384135845b FEATURE: introduce ultra_low priority queue
This commit introduces an ultra low priority queue for post rebakes. This
way rebakes can never interfere with regular sidekiq processing for cases
where we perform a large scale rebake.

Additionally it allows Post.rebake_old to be run with rate_limiter: false
to avoid triggering the limiter when rebaking. This is handy for cases
where you want to just force the full rebake and not wait for it to trickle
2019-01-17 14:53:19 +11:00
Bianca Nenciu 7d84648d11 FEATURE: Remove full quotes only from new posts. (#6862) 2019-01-17 13:24:32 +11:00
Saurabh Patel b63b399799 DEV: remove uploaded_meta_id column from category (#6725)
* DEV: remove uploaded_meta_id column from category

* remove uploaded_meta part
2019-01-10 09:37:21 +08:00
Sam e08a3f719c FEATURE: push post rebake regular task to low priority queue
This allows us to run regular rebakes without starving the normal queue.

It additionally adds the ability to specify queue with `Jobs.enqueue` so
we can specifically queue a job with lower priority using the `queue` arg.
2019-01-09 08:57:20 +11:00
Vinoth Kannan 8f602be2fe FEATURE: keep the topic in closed status until the community flags are handled 2019-01-08 16:13:10 +05:30
David Taylor 5bf16d7d10 FEATURE: Topic timer for bumping a topic in the future 2019-01-04 13:08:04 +00:00
Gerhard Schlager 1a8ca68ea3 FEATURE: Improve backup stats on admin dashboard
* Dashboard doesn't timeout anymore when Amazon S3 is used for backups
* Storage stats are now a proper report with the same caching rules
* Changing the backup_location, s3_backup_bucket or creating and deleting backups removes the report from the cache
* It shows the number of backups and the backup location
* It shows the used space for the correct backup location instead of always showing used space on local storage
* It shows the date of the last backup as relative date
2018-12-17 11:35:11 +01:00
Sam f74ef71130 DEV: add transaction and active check to invalidate job
Follows up on 3b76f19668 this per @tgxworld a transaction makes sense here

Very minor
2018-12-13 14:59:56 +11:00
Neil Lalonde 3b76f19668 FIX: invalidating inactive admin emails should mark them as not active 2018-12-12 17:07:49 -05:00
Neil Lalonde a1db15fead FEATURE: require admins to re-validate their email addresses if they haven't been seen for a number of days, configurable with the invalidate_inactive_admin_email_after_days site setting. Social logins are also revoked. Default is 365 days. 2018-12-12 15:32:38 -05:00
David Taylor 9db829134c
FIX: Use database to persist metadata during social registration (#6750)
Previously was using the cookie_store, which is limited to 4kb. This caused issues for providers sending large volumes of metadata about a user.
2018-12-10 15:10:06 +00:00
Saurabh Patel 9e3143445b DEV:add uploaded_meta option in category for category meta image (#6724) 2018-12-07 16:24:07 +01:00
Sam 8e307e633e FIX: posts would not auto rebake unless gravatar download was enabled 2018-12-07 17:03:22 +11:00
Vinoth Kannan d33d031742
FEATURE: Filter topic and post web hook events by tags (#6726)
* FEATURE: Filter topic and post web hook events by tags

* Add a spec test with unmatched tags
2018-12-05 14:44:06 +05:30
Régis Hanol 3c9c95ac83 Update Rubocop to 0.60 2018-12-04 10:48:16 +01:00
Guo Xiang Tan eecd1a7d8c FIX: `Jobs::CleanUpUploads` fails when value of upload data_type is an empty string. 2018-11-30 10:46:39 +08:00
Guo Xiang Tan ba280f9cf3 FIX: Give up migrating url site setting if endpoint doesn't return data. 2018-11-22 12:16:52 +08:00
Guo Xiang Tan a8603e04ab FIX: Give up migration of URL site settings if there is an SSL error. 2018-11-22 12:14:27 +08:00
Guo Xiang Tan df72674f24 Warn when migrate url site settings job encounters an error. 2018-11-21 07:55:50 +08:00
Guo Xiang Tan 81b3bdaabd FIX: Remove site settings override for deprecated url site settings. 2018-11-20 11:42:39 +08:00
Guo Xiang Tan 9fd704735e Rescue from `SocketError` in migrate url onceoff job. 2018-11-19 13:07:34 +08:00
Guo Xiang Tan 9e86b425bc FIX: Job to clean up old URL settings when new setting has been set.
Related to 44391ee8ab
2018-11-16 09:33:31 +08:00
Guo Xiang Tan 901611cb4e Fix migrate upload url site settings to rescue from invalid URLs. 2018-11-15 15:16:14 +08:00
Guo Xiang Tan 5b320e6652 DEV: Don't warn about using deprecated site settings in clean up uploads job. 2018-11-14 20:58:06 +08:00
Guo Xiang Tan df111259fe More URL site settings into a onceoff job.
* Doing it in a post migration was a bad idea
  because the migration will fail if the site
  is down while trying to download uploads
  which points to the instance. This mainly
  affects self-hosters using `discourse_docker`
  where `./launcher rebuild` will take the
  existing container down.
2018-11-14 20:29:20 +08:00
Guo Xiang Tan 44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Guo Xiang Tan 7b44339529 FIX: Prevent uploads used in site settings from being deleted. 2018-11-13 09:15:16 +08:00
Gerhard Schlager 24e5be3f0c FIX: Relative links in translations should work with subfolder 2018-11-08 23:31:05 +00:00
Bianca Nenciu 2070edf889 FIX: Clarify User.group_locked_trust_level.
* Rename User.group_locked_trust_level to User.group_granted_trust_level.

* Remove the column from users table.
2018-11-07 10:27:44 +08:00
Bianca Nenciu e0ccd36dbe FEATURE: Suspicious logins report. (#6544) 2018-10-30 22:51:58 +00:00
Bianca Nenciu 6a3767cde7 FEATURE: Warn users via email about suspicious logins. (#6520)
* FEATURE: Warn users via email about suspicious logins.

* DEV: Move suspicious login check to a job.
2018-10-25 09:45:31 +00:00
Sam de6b585368 minor, bypass gravatar update if user does not match
this protects against a race condition that can happen when a user record
is destroyed reasonably quickly
2018-10-23 12:20:41 +11:00
Kyle Zhao 0f1afad6da FIX: extracted theme JavaScripts for multisite (#6502)
* FIX: extracted theme javascripts for multisite

* onceoff to rebake all theme fields
2018-10-18 17:05:34 +11:00
Sam 8d06731484 FIX: reduce amount of work onceoff does
In the past onceoff was forcing inline download of gravatars,
this can be so expensive that it will never finish

This fix ensures it only marks avatars stale which will be picked
up by regular schedules
2018-10-16 10:29:16 +11:00
Guo Xiang Tan 84d4c81a26 FEATURE: Support backup uploads/downloads directly to/from S3.
This reverts commit 3c59106bac.
2018-10-15 09:43:31 +08:00
Régis Hanol 09961fb425 FIX: properly escape name of custom emoji 2018-10-11 09:35:23 +02:00
Guo Xiang Tan 3c59106bac Revert "FEATURE: Support backup uploads/downloads directly to/from S3."
This reverts commit c29a4dddc1.

We're doing a beta bump soon so un-revert this after that is done.
2018-10-11 11:08:23 +08:00
Gerhard Schlager c29a4dddc1 FEATURE: Support backup uploads/downloads directly to/from S3. 2018-10-11 10:38:43 +08:00
Bianca Nenciu e0e0a91e7d FIX: Retry sending email in case of temporary issue. (#6375) 2018-10-04 22:56:24 +08:00
Gerhard Schlager 3a6ab02563 FEATURE: Notify admins when scheduling of backup fails 2018-10-02 15:48:16 +02:00
Guo Xiang Tan d8fa7fbbab Fix onceoff job in cfa7173da3 not running. 2018-10-01 17:14:04 +08:00
Guo Xiang Tan cfa7173da3 FIX: Onceoff job to fix missing user profile backgrounds. 2018-10-01 16:26:40 +08:00
Robin Ward 02da022c70
PERF: Quit out of the email job quickly if disabled (#6423)
This prevents sidekiq from doing a bunch of queries when email is
disabled.

Critical emails are a special case and will be sent.
2018-10-01 01:15:45 +08:00
Guo Xiang Tan 767f27929d Rename `Jobs::RecoverPostUploads` to rerun the job take 2. 2018-09-19 22:40:32 +08:00
Guo Xiang Tan 6dbe1d832d Rename `Jobs::RecoverPostUploads` to rerun the job.
Running a migration to destroy the onceoff job doesn't work
because an instance running the old code may execute the job.
2018-09-19 21:59:44 +08:00
Guo Xiang Tan 667e68ec58 Prefer `create` over `create!` when not checking for response. 2018-09-19 18:22:54 +08:00
Guo Xiang Tan aa1af9fc22 FIX: Onceoff job to recover missing post uploads.
This fixes the regression due to 1f636c445b
2018-09-14 09:04:01 +08:00
Guo Xiang Tan e1b16e445e Rename `FileHelper.is_image?` -> `FileHelper.is_supported_image?`. 2018-09-12 09:22:28 +08:00
Gerhard Schlager 797cbf8653 FIX: Remove user fields when anonymizing user 2018-09-07 00:02:56 +02:00
Guo Xiang Tan b6a139b581 Fix broken spec. 2018-09-06 12:41:43 +08:00
Sam d9c0dc8687 correct prev commit
s3. did not exists it is s3-
2018-09-05 16:11:44 +10:00
Sam 83e1315e42 FIX: correct urls in uploads table to point at dualstack
Last week we added support for dual stack urls but did not remap the
the old records in the uploads and optimized images table

This caused a few minor edge cases worst was that if you rebaked old
images S3 CDN was not repopulated.
2018-09-05 15:58:04 +10:00
Sam e4498d2a8a FIX: keep db and job correctly in multisite logs
This ensures we report job and db correctly, previously we were
only reporting this on default
2018-09-04 16:05:44 +10:00
Sam e1975e293f FIX: when uploads are destroyed clear up avatar refs in user table
This also auto corrects twice daily when we ensure consistency
2018-08-31 14:46:42 +10:00
Guo Xiang Tan 4f81d4cc8e Minor fixes to `Jobs::BulkInvite`. 2018-08-30 15:35:16 +08:00
Gerhard Schlager b2cf725700 FIX: Don't try to send invite email when invite was deleted 2018-08-29 12:43:12 +02:00
Sam 44cf3cf975 FIX: queue heartbeats in readonly modes
If sidekiq is paused or Discourse is in readonly continue to queue
heartbeats

If we do not do that then a master process can end up reaping sidekiq
workers and causing various badness

This also impacts restore which can do weird stuff TM in cases like this
2018-08-29 12:36:59 +10:00
Sam 740308675b FEATURE: erode bounce score every time an email is sent
Introduces a hidden setting (default is 0.1) that erodes bounce score
every time we send an email. This means that erratic failures are less
painful cause system auto corrects
2018-08-28 17:02:12 +10:00
Sam 9ab1fb7dfc FEATURE: correctly store width and height on uploads
Previously we used width and height for thumbnails, new code ensures

1. We auto correct width and height
2. We added extra columns for thumbnail_width and height, this is determined
 by actual upload and no longer passed in as a side effect
3. Optimized Image now stores filesize which can be used for analysis, decisions

Also

- fixes Android image manifest as a side effect
- fixes issue where a thumbnail generated that is smaller than the upload is no longer used
2018-08-28 12:59:22 +10:00
Guo Xiang Tan 36a7028f19 FEATURE: Clean up `PostReplyKey` records.
* Default retention of 90 days.
2018-08-23 10:40:02 +08:00
Guo Xiang Tan 8bdf14834b PERF: Restrict number of skipped email log for `Jobs::UserEmail`. 2018-08-21 11:14:43 +08:00
Guo Xiang Tan 2c70d3f443 Take 2 on ba6f11c521. 2018-08-21 10:06:36 +08:00
Bianca Nenciu 860c1c3dcd FEATURE: Automatically expire keys if not used for a configurable amount of time. (#6264) 2018-08-20 17:36:14 +02:00
Guo Xiang Tan 16c0ebe8a8 Fix the build. 2018-08-17 16:53:07 +08:00
Sam 796164b58c FIX: automatically correct bad avatars on access
Also start relying on upload extension for optimized images
2018-08-16 16:32:56 +10:00
Sam 1f17b84b63 FEATURE: more context for error reporting on jobs fails 2018-08-16 12:38:49 +10:00
Régis Hanol de92913bf4 FIX: store the topic links using the cooked upload url 2018-08-14 12:23:32 +02:00
Guo Xiang Tan ba022234c6 Add onceoff job to fix incorrect upload extensions. 2018-08-13 10:44:08 +08:00