Commit Graph

450 Commits

Author SHA1 Message Date
David Taylor f524f8f811
Remove Yahoo login support from core and deprecate OpenID2.0 (#7310)
- Plugin developers using OpenID2.0 should migrate to OAuth2 or OIDC. OpenID2.0 APIs will be removed in v2.4.0

- For sites requiring Yahoo login, it can be implemented using the OpenID Connect plugin: https://meta.discourse.org/t/103632

For more information, see https://meta.discourse.org/t/113249
2019-04-08 10:38:25 +01:00
Guo Xiang Tan 108c231d1c FIX: Clean up `topic_search_data` of trashed topics.
This keeps the index and table smaller.
2019-04-08 16:53:39 +08:00
Guo Xiang Tan c4997ce85f FIX: Don't try and reindex posts that have been trashed. 2019-04-08 16:38:43 +08:00
Robin Ward c9ddc6ce30 FIX: Make sure the site setting works and fix build 2019-04-04 17:44:10 -04:00
Vinoth Kannan 64ec4a6a1b fix the EnsurePostUploadsExistence job spec 2019-04-04 21:53:58 +05:30
Vinoth Kannan b3fb0a7039 FEATURE: ensure consistency of post uploads in cooked content 2019-04-04 02:23:28 +05:30
Robin Ward 111a502231 FIX: Deleting Users should work nicely with Reviewable Users
"Rejecting" a user in the queue is equivalent to deleting them, which
would then making it impossible to review rejected users. Now we store
information about the user in the payload so if they are deleted things
still display in the Rejected view.

Secondly, if a user is destroyed outside of the review queue, it will
now automatically "Reject" that queue item.
2019-04-03 16:42:39 -04:00
Guo Xiang Tan d151425353
PERF: Delete search data of posts from trashed topics periodically. (#7302)
This keeps both the index and table smaller.
2019-04-03 10:10:41 +08:00
Guo Xiang Tan d8704c11ca PERF: Better use of index when queueing a topci for search reindex.
Also move `Search::INDEX_VERSION` to `SearchIndexer` which is where the
version is actually being used.
2019-04-02 09:53:37 +08:00
Guo Xiang Tan aa2311a7b0 FIX: Don't reindex posts belonging to a deleted topic for search.
Posts belonging to a deleted topic can't be index for search so we need
to avoid loading those post ids.
2019-04-02 07:36:53 +08:00
Guo Xiang Tan 3fc5dbb045 FIX: Don't attempt to reindex posts that have an empty raw.
If the post ids keep loading, we might end up in a situations where
we're always loading the same post ids over and over again without
indexing anything new.

Follow up to daeda80ada.
2019-04-02 07:13:33 +08:00
Robin Ward 76669bb5a6 FIX: Don't refer to pending review items as flags
They could be queued posts or users, and the notice should reflect that
properly.
2019-04-01 14:46:56 -04:00
Guo Xiang Tan daeda80ada
FIX: Don't index posts with empty `Post#raw` for search. (#7263)
* DEV: Remove unnecessary join in `Jobs::ReindexSearch`.

* FIX: Don't index posts with empty `Post#raw` for search.
2019-04-01 10:06:27 +08:00
Guo Xiang Tan 730ebdfcba DEV: Refactor `Jobs::EmitWebHookEvent` specs. 2019-04-01 09:46:39 +08:00
Vinoth Kannan 904ba266cf SPEC: Add test case in emit_web_hook_event_spec for commit 4c6bfb9 2019-03-31 16:28:40 +05:30
Tarek Khalil b1cb95fc23
FEATURE: Introduce ignore duration selection (#7266)
* FEATURE: Introducing new UI for tracking User's ignored or muted states
2019-03-29 10:14:53 +00:00
Robin Ward 118f98b6ed Linting error 2019-03-28 12:52:46 -04:00
Robin Ward b58867b6e9 FEATURE: New 'Reviewable' model to make reviewable items generic
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.

Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
2019-03-28 12:45:10 -04:00
Sam Saffron 9ebabc1de8 FEATURE: unconditionally update Topic updated_at when posts change in topic
Previously we would bypass touching `Topic.updated_at` for whispers and post
recovery / deletions.

This meant that certain types of caching can not be done where we rely on
this information for cache accuracy.

For example if we know we have zero unread topics as of yesterday and whisper
is made I need to bump this date so the cache remains accurate

This is only half of a larger change but provides the groundwork.

Confirmed none of our serializers leak out Topic.updated_at so this is safe
spot for this info

At the moment edits still do not change this but it is not relevant for the
unread cache.

This commit also cleans up some specs to use the new `eq_time` matcher for
millisecond fidelity comparison of times

Previously `freeze_time` would fudge this which is not that clean.
2019-03-28 17:28:01 +11:00
David Taylor 95d5819218 FIX: Re-download hotlinked optimized images (#7249)
* FIX: Download local images, even if download remote is disabled
2019-03-27 21:31:12 +01:00
David Taylor bb3f8e32e5 DEV: Run pull_hotlinked_images onebox specs without synchronous jobs
This was changed in fa5a1586, and caused a mutex to lock up, adding 60 seconds to the test suite.
2019-03-22 19:27:37 +00:00
Neil Lalonde 399e937a38 FIX: prevent sending multiple summary emails due to Sidekiq delays 2019-03-22 12:34:34 -04:00
David Taylor a9d5ffbe3d FIX: Prevent critical emails bypassing disable, and improve email test logic
- The test_email job is removed, because it was always being run synchronously (not in sidekiq)
- 34b29f62 added a bypass for critical emails, to match the spec. This removes the bypass, and removes the spec.
- This adapts the specs for 72ffabf6, so that they check for emails being sent
- This reimplements c2797921, allowing test emails to be sent even when emails are disabled
2019-03-22 17:28:43 +08:00
Tarek Khalil a31a35b334 FEATURE: Ignored user notification behaviour should be as a muted user (#7227) 2019-03-21 12:15:34 +01:00
Tarek Khalil 3b59ff0d02 [FEATURE] Disallow ignoring self, admins or moderators users (#7202) 2019-03-20 11:18:46 +01:00
Tarek Khalil fed2dd9148 FEATURE: Add scheduled job to purge expired ignored users (#7211) 2019-03-20 11:01:43 +01:00
Guo Xiang Tan 4020c87680 DEV: Refactor tests for `Jobs::CleanUpInactiveUsers`.
* Remove use of 0 in favor of `TrustLevel.levels[:newuser]`.
* Consolidate two tests into a single one.
* Test that disabling the feature works.
* Avoid loading full ActiveRecord object in test when we only need to
know the existence of the record.
2019-03-19 09:57:21 +08:00
Bianca Nenciu 2347661a74 FEATURE: Clean up inactive users. (#7172) 2019-03-18 16:25:15 +01:00
Penar Musaraj 9334d2f4f7
FEATURE: add more granular user option levels for email notifications (#7143)
Migrates email user options to a new data structure, where `email_always`, `email_direct` and `email_private_messages` are replace by

* `email_messages_level`, with options: `always`, `only_when_away` and `never` (defaults to `always`)
* `email_level`, with options: `always`, `only_when_away` and `never` (defaults to `only_when_away`)
2019-03-15 10:55:11 -04:00
Tarek Khalil bd6d31c9ec
FEATURE: Add `IgnoredUsersSummary` daily job (#7144)
* FEATURE: Add `IgnoredUsersSummary` daily job

## Why?

This is part of the [Ability to ignore a user feature](https://meta.discourse.org/t/ability-to-ignore-a-user/110254/8).

We want to:

1. Send an automatic group PM that goes out to moderators
2. When {x} users have Ignored the same user, threshold defined by a site setting, default of 5
3. Only send this message every X days which is defined by another site setting
2019-03-14 22:51:43 +00:00
Robin Ward fa5a158683 REFACTOR: Move `queue_jobs` out of `SiteSetting`
It is not a setting, and only relevant in specs. The new API is:

```
Jobs.run_later!        # jobs will be thrown on the queue
Jobs.run_immediately!  # jobs will run right away, avoid the queue
```
2019-03-14 10:47:38 -04:00
Guo Xiang Tan b0c8fdd7da FIX: Properly support defaults for upload site settings. 2019-03-13 16:36:57 +08:00
Penar Musaraj 95532814df DEV: Make Rubocop happy 2019-03-11 22:33:24 -04:00
Guo Xiang Tan 34b29f62db DEV: Remove the use of stubs and mocks in `Jobs::UserEmail` tests.
We can only be sure that an email is sent when we get a mailer in
`ActionMailer::Deliveries`. A couple of tests were actually incorrect
because it didn't flow through our email sender where there are more
conditions in determining whether an email is sent or not.
2019-03-12 09:39:16 +08:00
Robin Ward d1d9a4f128 Add new `run_jobs_synchronously!` helper for tests
Previously if you wanted to have jobs execute in test mode, you'd have
to do `SiteSetting.queue_jobs = false`, because the opposite of queue
is to execute.

I found this very confusing, so I created a test helper called
`run_jobs_synchronously!` which is much more clear about what it does.
2019-03-11 16:58:35 -04:00
David Taylor fc7938f7e0
REFACTOR: Migrate GoogleOAuth2Authenticator to use ManagedAuthenticator (#7120)
https://meta.discourse.org/t/future-social-authentication-improvements/94691/3
2019-03-07 11:31:04 +00:00
Joffrey JAFFEUX 703c724cf3
REFACTOR: Migrate InstagramAuthenticator to use ManagedAuthenticator (#7081) 2019-03-04 14:54:28 +01:00
Régis Hanol 326d892f5e Aadd 'secondary_emails' field in users export
FIX: escape_comma wasn't working in CSV exports
FIX: group_names field wasn't properly serialized
2019-02-27 10:12:20 +01:00
Vinoth Kannan 484bd82278 FIX: Add onceoff job to remove double quotes from s3 etags 2019-02-14 05:19:41 +05:30
Vinoth Kannan ba9cc83d4c FIX: Destination prefix in S3 inventory configuration is incorrect 2019-02-06 20:51:28 +05:30
Vinoth Kannan ff12c4b2d4 FIX: Bucket name is missing in S3 inventory data path 2019-02-06 19:16:08 +05:30
Vinoth Kannan b4f713ca52
FEATURE: Use amazon s3 inventory to manage upload stats (#6867) 2019-02-01 10:10:48 +05:30
Robin Ward 78ddc82952 FIX: Respect min_flags_staff_visibility for new flags too
There was a situation where if:

* There were new flags to review that met the visibility threshold

AND

* There were old flags that *didn't* meet the threshold

THEN

a pending flags notification would be sent out. This fixes that case.
Staff should not be notified of flags if they do not meet the threshold
and are old.
2019-01-25 11:27:43 -05:00
Robin Ward 96b2585a91 REFACTOR: Remove unncessary stubs from pending flags reminder
They seem to be calculated fine by the application, and stubbing
makes the tests more brittle and prone to regression.
2019-01-24 13:45:58 -05:00
Robin Ward f32de88dfc FIX: Don't notify of pending flags if min_flags_staff_visibility not met 2019-01-22 11:01:18 -05:00
Sam 384135845b FEATURE: introduce ultra_low priority queue
This commit introduces an ultra low priority queue for post rebakes. This
way rebakes can never interfere with regular sidekiq processing for cases
where we perform a large scale rebake.

Additionally it allows Post.rebake_old to be run with rate_limiter: false
to avoid triggering the limiter when rebaking. This is handy for cases
where you want to just force the full rebake and not wait for it to trickle
2019-01-17 14:53:19 +11:00
Sam 824c3420e9 DEV: make Jobs.enqueue tests less fragile
Previously we depended on non Sidekiq specific mocking which is not the
official way of testing Sidekiq, this made these tests very fragile

New testing is more robust and complete
2019-01-09 09:51:11 +11:00
Sam e08a3f719c FEATURE: push post rebake regular task to low priority queue
This allows us to run regular rebakes without starving the normal queue.

It additionally adds the ability to specify queue with `Jobs.enqueue` so
we can specifically queue a job with lower priority using the `queue` arg.
2019-01-09 08:57:20 +11:00
David Taylor 5bf16d7d10 FEATURE: Topic timer for bumping a topic in the future 2019-01-04 13:08:04 +00:00
David Taylor 1960236822
FIX: Suspicious login detection (#6772) 2018-12-14 16:30:34 +00:00
Neil Lalonde 3b76f19668 FIX: invalidating inactive admin emails should mark them as not active 2018-12-12 17:07:49 -05:00
Neil Lalonde a1db15fead FEATURE: require admins to re-validate their email addresses if they haven't been seen for a number of days, configurable with the invalidate_inactive_admin_email_after_days site setting. Social logins are also revoked. Default is 365 days. 2018-12-12 15:32:38 -05:00
David Taylor 3fedb2ad20 DEV: Style and performance improvements
Follow-up from 9db8291
2018-12-11 09:58:20 +00:00
David Taylor 9db829134c
FIX: Use database to persist metadata during social registration (#6750)
Previously was using the cookie_store, which is limited to 4kb. This caused issues for providers sending large volumes of metadata about a user.
2018-12-10 15:10:06 +00:00
Rishabh da41a515cd Add test to check if posts are auto rebaked by PeriodicalUpdates job when gravatar download is disabled
follow-up for 8e307e633e
2018-12-10 11:10:57 +11:00
Vinoth Kannan d33d031742
FEATURE: Filter topic and post web hook events by tags (#6726)
* FEATURE: Filter topic and post web hook events by tags

* Add a spec test with unmatched tags
2018-12-05 14:44:06 +05:30
Guo Xiang Tan eecd1a7d8c FIX: `Jobs::CleanUpUploads` fails when value of upload data_type is an empty string. 2018-11-30 10:46:39 +08:00
Penar Musaraj 03deda2147
Upgrade to FontAwesome 5 (take two) (#6673)
* Add missing icons to set

* Revert FA5 revert

 This reverts commit 42572ff

* use new SVG syntax in locales

* Noscript page changes (remove login button, center "powered by" footer text)

* Cast wider net for SVG icons in settings

- include any _icon setting for SVG registry (offers better support for plugin settings)

- let themes store multiple pipe-delimited icons in a setting

- also replaces broken onebox image icon with SVG reference in cooked post processor

* interpolate icons in locales

* Fix composer whisper icon alignment

* Add support for stacked icons

* SECURITY: enforce hostname to match discourse hostname

This ensures that the hostname rails uses for various helpers always matches
the Discourse hostname

* load SVG sprite with pre-initializers

* FIX: enable caching on SVG sprites

* PERF: use JSONP for SVG sprites so they are served from CDN

This avoids needing to deal with CORS for loading of the SVG

Note, added the svg- prefix to the filename so we can quickly tell in
dev tools what the file is

* Add missing SVG sprite JSONP script to CSP

* Upgrade to FA 5.5.0

* Add support for all FA4.7 icons

- adds complete frontend and backend for renamed FA4.7 icons

- improves performance of SvgSprite.bundle and SvgSprite.all_icons

* Fix group avatar flair preview

- adds an endpoint at /svg-sprites/search/:keyword

- adds frontend ajax call that pulls icon in avatar flair preview even when it is not in subset

* Remove FA 4.7 font files
2018-11-26 16:49:57 -05:00
Guo Xiang Tan 81b3bdaabd FIX: Remove site settings override for deprecated url site settings. 2018-11-20 11:42:39 +08:00
Guo Xiang Tan 9e86b425bc FIX: Job to clean up old URL settings when new setting has been set.
Related to 44391ee8ab
2018-11-16 09:33:31 +08:00
Guo Xiang Tan df111259fe More URL site settings into a onceoff job.
* Doing it in a post migration was a bad idea
  because the migration will fail if the site
  is down while trying to download uploads
  which points to the instance. This mainly
  affects self-hosters using `discourse_docker`
  where `./launcher rebuild` will take the
  existing container down.
2018-11-14 20:29:20 +08:00
Bianca Nenciu fce0a0ccc8 FEATURE: Compute distance between logins to generate login alerts. (#6562) 2018-11-14 13:26:47 +01:00
Guo Xiang Tan 44391ee8ab
FEATURE: Upload Site Settings. (#6573) 2018-11-14 15:03:02 +08:00
Guo Xiang Tan 7b44339529 FIX: Prevent uploads used in site settings from being deleted. 2018-11-13 09:15:16 +08:00
Gerhard Schlager 24e5be3f0c FIX: Relative links in translations should work with subfolder 2018-11-08 23:31:05 +00:00
Bianca Nenciu 2070edf889 FIX: Clarify User.group_locked_trust_level.
* Rename User.group_locked_trust_level to User.group_granted_trust_level.

* Remove the column from users table.
2018-11-07 10:27:44 +08:00
Guo Xiang Tan bdb8e9efdb DEV: Remove mocks from specs. 2018-11-07 09:55:58 +08:00
Bianca Nenciu e0ccd36dbe FEATURE: Suspicious logins report. (#6544) 2018-10-30 22:51:58 +00:00
Régis Hanol d17c8df926 Only check for suspicious login for staff members 2018-10-26 00:29:28 +02:00
Bianca Nenciu 6a3767cde7 FEATURE: Warn users via email about suspicious logins. (#6520)
* FEATURE: Warn users via email about suspicious logins.

* DEV: Move suspicious login check to a job.
2018-10-25 09:45:31 +00:00
Kyle Zhao 0f1afad6da FIX: extracted theme JavaScripts for multisite (#6502)
* FIX: extracted theme javascripts for multisite

* onceoff to rebake all theme fields
2018-10-18 17:05:34 +11:00
Neil Lalonde 12f132736b FIX: error looking at users in admin when tl3_promotion_min_duration is set to a very high value 2018-10-11 15:11:48 -04:00
Guo Xiang Tan 1b1ef21481 DEV: Improve `Jobs::CleanUpEmailLogs` specs. 2018-10-08 11:11:20 +08:00
Vinoth Kannan 8430ea927e FIX: Generate webhook payloads before destroy events (#6325) 2018-10-05 16:53:59 +08:00
Guo Xiang Tan cfa7173da3 FIX: Onceoff job to fix missing user profile backgrounds. 2018-10-01 16:26:40 +08:00
Robin Ward 02da022c70
PERF: Quit out of the email job quickly if disabled (#6423)
This prevents sidekiq from doing a bunch of queries when email is
disabled.

Critical emails are a special case and will be sent.
2018-10-01 01:15:45 +08:00
Guo Xiang Tan 767f27929d Rename `Jobs::RecoverPostUploads` to rerun the job take 2. 2018-09-19 22:40:32 +08:00
Guo Xiang Tan aa1af9fc22 FIX: Onceoff job to recover missing post uploads.
This fixes the regression due to 1f636c445b
2018-09-14 09:04:01 +08:00
Sam d9c0dc8687 correct prev commit
s3. did not exists it is s3-
2018-09-05 16:11:44 +10:00
Sam 83e1315e42 FIX: correct urls in uploads table to point at dualstack
Last week we added support for dual stack urls but did not remap the
the old records in the uploads and optimized images table

This caused a few minor edge cases worst was that if you rebaked old
images S3 CDN was not repopulated.
2018-09-05 15:58:04 +10:00
Sam 1866a8e8da correct invalid spec 2018-08-31 15:06:30 +10:00
Gerhard Schlager b2cf725700 FIX: Don't try to send invite email when invite was deleted 2018-08-29 12:43:12 +02:00
Sam 44cf3cf975 FIX: queue heartbeats in readonly modes
If sidekiq is paused or Discourse is in readonly continue to queue
heartbeats

If we do not do that then a master process can end up reaping sidekiq
workers and causing various badness

This also impacts restore which can do weird stuff TM in cases like this
2018-08-29 12:36:59 +10:00
Sam 740308675b FEATURE: erode bounce score every time an email is sent
Introduces a hidden setting (default is 0.1) that erodes bounce score
every time we send an email. This means that erratic failures are less
painful cause system auto corrects
2018-08-28 17:02:12 +10:00
Kyle Zhao e25a6e085e FIX: drop title updates through RSS feeds
can create an update loop
2018-08-28 16:25:04 +10:00
Guo Xiang Tan 36a7028f19 FEATURE: Clean up `PostReplyKey` records.
* Default retention of 90 days.
2018-08-23 10:40:02 +08:00
Guo Xiang Tan 8bdf14834b PERF: Restrict number of skipped email log for `Jobs::UserEmail`. 2018-08-21 11:14:43 +08:00
Guo Xiang Tan 2c70d3f443 Take 2 on ba6f11c521. 2018-08-21 10:06:36 +08:00
Bianca Nenciu 860c1c3dcd FEATURE: Automatically expire keys if not used for a configurable amount of time. (#6264) 2018-08-20 17:36:14 +02:00
Guo Xiang Tan 16c0ebe8a8 Fix the build. 2018-08-17 16:53:07 +08:00
Joffrey JAFFEUX 10a3499d68
uses emoji versions for specs (#6276) 2018-08-16 13:45:30 +02:00
Gerhard Schlager b73950692b FIX: Parsing non-existent feed should not fail 2018-08-10 18:37:14 +02:00
Simon Cossar 3cd4dc0f5f Allow users with group_locked_trust_level to be promoted to tl3 (#6249) 2018-08-10 10:42:23 +10:00
Guo Xiang Tan ba6f11c521 PERF: Only log the first skipped email when user exceeds daily limit.
https://meta.discourse.org/t/cleaning-up-e-mail-logs/39132
2018-08-08 16:25:00 +08:00
Guo Xiang Tan 4e11811321 FIX: `UserAvatar#update_gravatar!` does not update `User#uploaded_avatar`.
https://meta.discourse.org/t/missing-user-profile-pictures/93844/4
2018-08-06 13:29:44 +08:00
Gerhard Schlager 5d421fb946 FIX: Try respecting charset in HTTP header of RSS feed 2018-08-01 10:41:20 +02:00
Gerhard Schlager ff942ed2f3 FIX: Try detecting encoding of RSS feed 2018-08-01 10:41:20 +02:00
Guo Xiang Tan 87537b679c Drop `reply_key`, `skipped` and `skipped_reason` from `email_logs`. 2018-07-30 11:39:28 +08:00
Guo Xiang Tan fad9c2b971 PERF: Move `EmailLog#reply_key` into new `post_reply_keys` table. 2018-07-24 13:51:53 +08:00
Guo Xiang Tan ae8b0a517f PERF: Split skipped email logs into a seperate table. 2018-07-24 13:14:37 +08:00