Commit Graph

1255 Commits

Author SHA1 Message Date
David Taylor b22ea7911c
DEV: Drop old SSO site setting rows from the database (#12148)
These were copied to their new names in 821bb1e8cb
2021-02-19 19:05:49 +00:00
Sam 58de9e85be
FIX: ensure corrected migration runs (#12137)
Some instances may have ran earier version of the migration. Ensure newer
one runs instead.
2021-02-19 11:48:32 +11:00
Sam 4ef642b300
FIX: optimise MoveNewSinceToTable (#12136)
* FIX: optimise MoveNewSinceToTable

Avoids shuffling all ids around to the app (only use min / max)
Ensure the query for boundaries is ordered by user_id
2021-02-19 11:35:52 +11:00
Dan Ungureanu bddf94c0ab
FIX: Delete topic timers far in the future (#12125)
The migration used to fail because the same duration in minutes was out
of the integer range. The '20 years' limit was introduced in e0f0fe5.
2021-02-18 14:18:43 +02:00
Krzysztof Kotlarek 7829558c6d
FIX: dismiss new when topic_user exists without last read (#12103)
The bug was mentioned on meta: https://meta.discourse.org/t/pressing-dismiss-new-doesnt-clear-new-topics/179858

Problem is that sometimes the user has TopicUser records with `last_read_post_number` set as NULL. In that case, the topic is still "new" to them and should be dismissed when they click dismiss button.

In addition, I added that condition to post_migration and bumped the number to fix existing records. Migration is written to be idempotent so it will make no harm to already deployed instances.
2021-02-18 10:39:05 +11:00
Martin Brennan 9f0f801ae3
FIX: Do not mark group_flair images as secure on upload (#12081)
See https://meta.discourse.org/t/secure-media-uploads-breaks-group-flair-image/173671/4

Group flair image uploads definitely do not need to be secure.
2021-02-16 12:34:03 +10:00
Krzysztof Kotlarek ad3ec5809f
FIX: Dismiss new with better migration (#12062)
Original PR was reverted because of broken migration https://github.com/discourse/discourse/pull/12058

I fixed it by adding this line
```
          AND topics.id IN(SELECT id FROM topics ORDER BY created_at DESC LIMIT :max_new_topics)
```

This time it is left joining a limited amount of topics. I tested it on few databases and it worked quite smooth
2021-02-15 08:50:33 +11:00
Krzysztof Kotlarek a696cc07d2
Revert "FEATURE: Ability to dismiss all new topics (#12018)" (#12058)
This reverts commits 7426764af4 and f5b18e2a31
2021-02-12 08:50:25 +11:00
Krzysztof Kotlarek 7426764af4
FIX: Optimize move to dismiss_new_topics migration (#12041)
This migration is quite heavy because of join to all potential topics which should be `dismissed` for each user. To make it a little bit more efficient I did two things:
- move conditions to join so it should use fewer rows
- do that in batches - 1000 users at the time
2021-02-11 16:05:38 -05:00
Krzysztof Kotlarek f5b18e2a31
FEATURE: Ability to dismiss all new topics (#12018)
Follow up https://github.com/discourse/discourse/pull/11968

Dismiss all new topics using the same DismissTopicService. In addition, MessageBus receives exact topic ids which should be marked as `seen`.
2021-02-11 13:35:09 +11:00
David Taylor 821bb1e8cb
FEATURE: Rename 'Discourse SSO' to DiscourseConnect (#11978)
The 'Discourse SSO' protocol is being rebranded to DiscourseConnect. This should help to reduce confusion when 'SSO' is used in the generic sense.

This commit aims to:
- Rename `sso_` site settings. DiscourseConnect specific ones are prefixed `discourse_connect_`. Generic settings are prefixed `auth_`
- Add (server-side-only) backwards compatibility for the old setting names, with deprecation notices
- Copy `site_settings` database records to the new names
- Rename relevant translation keys
- Update relevant translations

This commit does **not** aim to:
- Rename any Ruby classes or methods. This might be done in a future commit
- Change any URLs. This would break existing integrations
- Make any changes to the protocol. This would break existing integrations
- Change any functionality. Further normalization across DiscourseConnect and other auth methods will be done separately

The risks are:
- There is no backwards compatibility for site settings on the client-side. Accessing auth-related site settings in Javascript is fairly rare, and an error on the client side would not be security-critical.
- If a plugin is monkey-patching parts of the auth process, changes to locale keys could cause broken error messages. This should also be unlikely. The old site setting names remain functional, so security-related overrides will remain working.

A follow-up commit will be made with a post-deploy migration to delete the old `site_settings` rows.
2021-02-08 10:04:33 +00:00
Martin Brennan 18da1d5b07
FIX: Topic timer duration_minutes was not backfilled correctly (#12004)
Because of a where clause of duration_minutes != duration, where
duration_minutes was NULL, the previous migration to fill the new
duration_minutes column failed. This corrects the failed migration
by just running the update where duration_minutes is NULL and duration
IS NOT NULL.

Previous commit is 4af77f1
2021-02-08 09:33:08 +10:00
Martin Brennan 4af77f1e38
FEATURE: Allow durations < 1 hour and < 1 day for topic timers where duration is specified (auto delete replies, close based on last post) (#11961)
This PR allows entering a float value for topic timers e.g. 0.5 for 30 minutes when entering hours, 0.5 for 12 hours when entering days. This is achieved by adding a new column to store the duration of a topic timer in minutes instead of the ambiguous both hours and days that it could be before.

This PR has ommitted the post migration to delete the duration column in topic timers; it will be done in a subsequent PR to ensure that no data is lost if the UPDATE query to set duration_mintues fails.

I have to keep the old keyword of duration in set_or_create_topic_timer for backwards compat, will remove at a later date after plugins are updated.
2021-02-05 10:12:56 +10:00
Krzysztof Kotlarek f39e7fe81d
FEATURE: New way to dismiss new topics (#11927)
This is a try to simplify logic around dismiss new topics to have one solution to work in all places - dismiss all-new, dismiss new in a specific category or even in a specific tag.
2021-02-04 11:27:34 +11:00
Martin Brennan 4af4d36175
FIX: IMAP allow unknown senders to reply to group topics via email (#11877)
Adds a new column/setting to groups, allow_unknown_sender_topic_replies, which is default false. When enabled, this scenario is allowed via IMAP:

* OP sends an email to the support email address which is synced to a group inbox via IMAP, creating a group topic
* Group user replies to the group topic
* An email notification is sent to the OP of the topic via GroupSMTPMailer
* The OP has several email accounts and the reply is sent to all of them, or they forward their reply to another email account
* The OP replies from a different email address than the OP (gloria@gmail.com instead of gloria@hey.com for example)
* The a new staged user is created, the new reply is accepted and added to the topic, and the staged user is added to the topic allowed users

Without allow_unknown_sender_topic_replies enabled the new reply creates an entirely new topic (because the email address it is sent from is not previously part of the topic email chain).
2021-01-29 09:59:10 +10:00
Martin Brennan f49e3e5731
DEV: Add security_last_changed_at and security_last_changed_reason to uploads (#11860)
This PR adds security_last_changed_at and security_last_changed_reason to uploads. This has been done to make it easier to track down why an upload's secure column has changed and when. This necessitated a refactor of the UploadSecurity class to provide reasons why the upload security would have changed.

As well as this, a source is now provided from the location which called for the upload's security status to be updated as they are several (e.g. post creator, topic security updater, rake tasks, manual change).
2021-01-29 09:03:44 +10:00
Mark VanLandingham 809274fe0d
DEV: Replace 'processed' column on notifications with new table (#11864) 2021-01-27 10:29:24 -06:00
Régis Hanol cd3d24ed8c
FIX: move post_search_data migration into onceoff job (#11851)
And reduce the size of the batches to 100k.

That should hopefully make the migrations run smoother...
2021-01-26 16:29:00 +01:00
Régis Hanol c56ba6c9bd
PERF: batch expensive post-migration (#11845)
Run the 'MigrateSearchDataAfterDefaultLocaleRename' post migration in batches of 500k records.

This will hopefully prevent any potential deadlocks on large tables.
2021-01-25 22:58:58 +01:00
Gerhard Schlager e65c5b0aad
PERF: Migrate search data after locale rename (#11831)
The default locale `en_US` has been renamed into `en`. This tries to migrate existing search data to avoid resource intensive reindexing.
2021-01-25 14:30:17 +01:00
Martin Brennan 9ee8a01c3a
FIX: Change default for IncomingEmail#created_via to 0 (unknown) and make NOT NULL (#11782)
Follow up to https://review.discourse.org/t/dev-add-created-via-column-to-incomingemail-pr-11751/18366/6
2021-01-21 12:59:50 +10:00
Gerhard Schlager 3b2f6e129a
FEATURE: Add English (UK) as locale (#11768)
* "English" gets renamed into "English (US)"
* "English (UK)" replaces "English"

@discourse-translator-bot keep_translations_and_approvals
2021-01-20 21:32:22 +01:00
Mark VanLandingham 1a7922bea2
FEATURE: Create notification schedule to automatically set do not disturb time (#11665)
This adds a new table UserNotificationSchedules which stores monday-friday start and ends times that each user would like to receive notifications (with a Boolean enabled to remove the use of the schedule). There is then a background job that runs every day and creates do_not_disturb_timings for each user with an enabled notification schedule. The job schedules timings 2 days in advance. The job is designed so that it can be run at any point in time, and it will not create duplicate records.

When a users saves their notification schedule, the schedule processing service will run and schedule do_not_disturb_timings. If the user should be in DND due to their schedule, the user will immediately be put in DND (message bus publishes this state).

The UI for a user's notification schedule is in user -> preferences -> notifications. By default every day is 8am - 5pm when first enabled.
2021-01-20 10:31:52 -06:00
Martin Brennan fb184fed06
DEV: Add created_via column to IncomingEmail (#11751)
This should make it easier to track down how the incoming email was created, which is one of four locations:

The POP3 poller (which picks up reply via email replies)
The admin email controller #handle_mail (which is where hosted mail is sent)
The IMAP sync tool
The group SMTP mailer, which sends emails when replying to IMAP topics, pre-emptively creating IncomingEmail records to avoid double syncing
2021-01-20 13:22:41 +10:00
Krzysztof Kotlarek 06b7c44593
FEATURE: reason to reject user signup (#11700)
Feature for `Must Approve Users` setup. When a user is rejected, a staff member can optionally set a reason for audit purposes. In addition, feedback email can be sent to the user.

Meta: https://meta.discourse.org/t/account-rejection-email/103112/8
2021-01-15 09:43:26 +11:00
Martin Brennan 87961534ea
FEATURE: IMAP detect spammed email and delete associated Discourse topic (#11654)
This PR adds functionality for the IMAP sync code to detect if a UID that is missing from the mail group mailbox is in the Spam/Junk folder for the mail account, and if so delete the associated Discourse topic. This is identical to what we do for emails that are moved for Trash.

If an email is missing but not in Spam or Trash, then we mark the incoming email record with imap_missing: true. This may be used in future to further filter or identify these emails, and perhaps go hunting for them in the email account in bulk.

Note: This adds some code duplication because the trash and spam email detection and handling is very similar. I intend to do more refactors/improvements to the IMAP sync code in time because there is a lot of room for improvement.
2021-01-14 09:54:18 +10:00
Bianca Nenciu ec0212e56b
FIX: Make category slugs lowercase (#11277)
Admins could specify category slug with upper case characters and same slug,
but with different cases could be used simultaneously.
2021-01-12 17:28:33 +02:00
Mark VanLandingham 4601f3be7e
FEATURE: Send notification emails when users leave do not disturb mode (#11643) 2021-01-07 10:49:49 -06:00
Sam 53f4d54f23
PERF: add indexes to speed up profile pages (#11598)
These 2 indexes optimise performance on profile pages.

The summary page displays:

1. A list of "Top Link" - links sorted by number of clicks posted by user
2. A list of "Top Replies" - replies made by a user that go the most hearts

These two areas could devolve into full index or table scans, new indexes are there to avoid this cost on large dbs

One minor downside is that storage requirements go a tiny bit up to maintain the new indexes
2020-12-29 15:54:05 +11:00
Alan Guo Xiang Tan d2a0462186 FIX: Re-run remove flag and queued post web hook migrations.
There have been production instances where the migrations has run but the records
that were meant to be deleted have not been deleted. The root cause is
unknown but the migration is safe and simple to re-run. The problem is
not reproducible locally so we're not spending too much time on digging
up the root. Time vs business cost tradeoff.

Follow-up fb15da43da
2020-12-21 10:17:08 +08:00
Mark VanLandingham 649ed24bb4
FEATURE: Do not disturb (#11484) 2020-12-18 09:03:51 -06:00
David Taylor 02278109d6
DEV: Maintain github_user_info primary key values during migration (#11286)
This will only happen if the user_associated_accounts table is currently empty. It's useful for people that may be depending on primary key values in data explorer queries. This change will only have an effect on sites which have not already run this migration.
2020-11-19 10:30:18 +00:00
Dan Ungureanu aee5e80038
FIX: Disable auto_update for existent themes (#11244) 2020-11-16 15:35:07 +02:00
Dan Ungureanu bc8423a1bf
FEATURE: Add auto update field to themes (#11102)
Themes marked for auto update will be automatically updated when
Discourse is updated. This is triggered by discourse_docker or
docker_manager running Rake task 'themes:update'.
2020-11-16 14:44:09 +02:00
Roman Rizzi f2bef7ea8f
FIX: Store Reviewable's force_review as a boolean. (#11219)
* FIX: Store Reviewable's force_review as a boolean.

Using the `force_review` flag raises the score to hit the minimum visibility threshold. This strategy turned out to be ineffective on sites with a high number of flags, where these values could rapidly fluctuate.

This change adds a `force_review` column on the reviewables table and modifies the `Reviewable#list_for` method to show these items when passing the `status: :pending` option, even if the score is not high enough. ReviewableQueuedPosts and ReviewableUsers are always created using this option.
2020-11-13 08:19:01 -03:00
Dan Ungureanu ab314218d3
FEATURE: Implement edit functionality for post notices (#11140)
All post notice related custom fields were moved to a single one.
2020-11-11 14:49:53 +02:00
David Taylor e8452a55a6
DEV: Drop github_user_infos table (#11181)
Follow-up to cf21de0e7a
2020-11-10 11:33:27 +00:00
David Taylor cf21de0e7a
DEV: Migrate Github authentication to ManagedAuthenticator (#11170)
This commit adds an additional find_user_by_email hook to ManagedAuthenticator so that GitHub login can continue to support secondary email addresses

The github_user_infos table will be dropped in a follow-up commit.

This is the last core authenticator to be migrated to ManagedAuthenticator 🎉
2020-11-10 10:09:15 +00:00
David Taylor 5140ec9acf
DEV: Cleanup ignored user logic (#11107)
- IgnoredUser records should all now have an expiring_at value. This commit enforces that in the DB, and fixes any corrupt rows
- Changes to the ignored user list are now handled by the `/u/{username}/notification_level` endpoint. This allows setting expiration dates on the ignore. This commit removes the old logic for saving a list of usernames in the user preferences.
- Many specs were calling `IgnoredUser.create`. This commit changes them to use `Fabricate(:ignored_user)` for consistency
2020-11-03 12:38:54 +00:00
Arpit Jalan 1476e17c35
FEATURE: new setting to create a linked topic on autoclosing mega topics (#11001)
This commit adds a site setting `auto_close_topics_create_linked_topic`
which when enabled works in conjunction with `auto_close_topics_post_count`
setting and creates a new linked topic for the topic just closed.

The auto-created new topic contains a link for all the previous topics
and the topic titles are appended with `(Part {n})`.

The setting is enabled by default.
2020-11-02 12:18:48 +05:30
Jarek Radosz 2f4a1ff61b
DEV: Update rubocop-discourse from 2.3.2 to 2.4.0 (#11079)
Also fixes whitespace related issues raised by rubocop.
2020-10-30 15:04:29 +01:00
David Taylor 23e5c605f6 DEV: Add support for allowed parameters in user api key scopes
Initially, this feature is only intended for use in core/plugins, so there is no API for requesting a parameter-scoped key. That may change in future.
2020-10-19 10:40:55 +01:00
Roman Rizzi 21c53ed249
FEATURE: Topic slow mode. (#10904)
Adds a new slow mode for topics that are heating up. Users will have to wait for a period of time before being able to post again.

We store this interval inside the topics table and track the last time a user posted using the last_posted_at datetime in the TopicUser relation.
2020-10-16 16:24:38 -03:00
Bianca Nenciu 43e52a7dc1
DEV: Remove gifsicle dependency (#10357)
Dependency on gifsicle, allow_animated_avatars and allow_animated_thumbnails
site settings were all removed. Animated GIF images are still allowed, but
the generated optimized images are no longer animated for those (which were
used for avatars and thumbnails).

The added 'animated' is populated by extracting information using FastImage.
This field was used to selectively reoptimize old animations. This process
happens in the background.
2020-10-16 13:41:27 +03:00
Penar Musaraj 5763309953
FEATURE: WCAG compliant color schemes (#10882)
Co-authored-by: Kris <kris.aubuchon@discourse.org>
2020-10-15 14:05:48 -04:00
Penar Musaraj 74de7a49f5
Simplify theme and color scheme seeding (#10872)
Now that we have support for user-selectable color schemes, it makes sense
to simplify seeding and theme updates in the wizard. 

We now:

- seed only one theme, named "Default" (previously "Light")
- seed a user-selectable Dark color scheme
- rename the "Themes" wizard step to "Colors"
- update the default theme's color scheme if a default is set
(a new theme is created if there is no default)
2020-10-14 10:18:02 -04:00
Bianca Nenciu 25b8ed740b
DEV: Make site setting type uploaded_image_list use upload IDs (#10401)
It used to be a list of concatenated upload URLs which was prone to
break.
2020-10-13 16:17:06 +03:00
Sam Saffron cd3254e170 DEV: remove legacy code
This code is a no-op on all sites, even though it looks rather dangerous
this migration has long run prior to people trying exploit it.

That said ... hygiene here ... is not good.

Remove this legacy, we do not want it, even in historical migrations.
2020-10-08 10:53:07 +08:00
David Taylor c0293339b8
PERF: Do not enqueue digest emails when attempted recently (#10849)
Previously, Jobs::EnqueueDigestEmails would enqueue a digest job for every user, even if there are no topics to send. The digest job would exit, no email would send, and last_emailed_at would not change. 30 minutes later, Jobs::EnqueueDigestEmails would run again and re-enqueue jobs for the same users.

120fa8ad introduced a temporary mitigation for this issue, by randomly selecting a subset of those users each time.

This commit adds a new `digest_attempted_at` column to the `user_stats` table. This column is updated every time a digest job completes for a user. Using this, we can avoid scheduling digest jobs for the same user every 30 minutes. This also removes the random user selection in 120fa8ad, and instead prioritizes users who had digests attempted the longest time ago.
2020-10-07 15:30:38 +01:00
Martin Brennan 6e2be3e60b
FIX: When admin changes an email for the user the user must confirm the change (#10830)
See https://meta.discourse.org/t/changing-a-users-email/164512 for additional context.

Previously when an admin user changed a user's email we assumed that they would need a password reset too because they likely did not have access to their account. This proved to be incorrect, as there are other reasons a user needs admin to change their email. This PR:

* Changes the admin change email for user flow so the user is sent an email to confirm the change
* We now record who the email change request was requested by
* If the requested by user is admin and not the user we note this in the email sent to the user
* We also make the confirm change email route open to anonymous users, so it can be clicked by the user even if they do not have access to their account. If there is a logged in user we make sure the confirmation matches the current user.
2020-10-07 13:02:24 +10:00