Commit Graph

1462 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan 3266350e80
FEATURE: Decouple category/tag presence in sidebar from notifi level (#17273) 2022-06-30 14:54:20 +08:00
Andrei Prigorshnev 033f72c65f
FEATURE: user status emoji (#17025) 2022-06-22 18:15:33 +04:00
Vinoth Kannan deee3c6f02
DEV: drop unused column `flair_url` from groups table. (#17179)
It's already included in the `ignored_columns` list in the group model. 03ffb0bf27/app/models/group.rb (L9)

Also, removed the `MigrateGroupFlairImages` onceoff job and spec.
2022-06-22 00:15:05 +05:30
Osama Sayegh 5176c689e9
UX: Change wording for 'regular' categories to 'normal' (#17134)
At some point in the past we decided to rename the 'regular' notification state of topics/categories to 'normal'. However, some UI copy was missed when the initial renaming was done so this commit changes the spots that were missed to the new name.
2022-06-20 06:49:33 +03:00
David Taylor 410ab5d587
FIX: Skip invalid values in site setting upload references migration (#17138) 2022-06-19 19:02:55 +01:00
Martin Brennan 4d3c1ceb44
FEATURE: Log the SMTP response in EmailLog (#17056)
When sending emails with delivery_method_options -> return_response
set to true, the SMTP sending code inside Mail will return the SMTP
response when calling deliver! for mail within the app. This commit
ensures that Email::Sender captures this response if it is returned
and stores it against the EmailLog created for the sent email.

A follow up PR will make this visible within the admin email UI.
2022-06-15 10:28:30 +10:00
David Taylor 002d62b847
FIX: Cleanup invalid historic site setting data (#17049)
`selectable_avatars_urls` contains invalid data (it's a backup from 20200810194943_change_selectable_avatars_site_setting.rb)

This migration is deliberately backdated so that it runs before `20220330160747_copy_site_settings_uploads_to_upload_references`
2022-06-09 09:51:29 +01:00
Martin Brennan 20f118c4c9
FIX: Handle empty string in theme_settings for upload_references (#17047)
Follow up to 9db8f00b3d,
the theme_settings.value field is not an integer and so
can be '', we need to account for this in the migration
otherwise we get this error:

> PG::InvalidTextRepresentation: ERROR:  invalid input syntax for type integer: ""
2022-06-09 10:19:44 +02:00
Bianca Nenciu 9db8f00b3d
FEATURE: Create upload_references table (#16146)
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.

* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
2022-06-09 09:24:30 +10:00
Andrei Prigorshnev 5c596273a0
FEATURE: user status (#16875) 2022-05-27 13:15:14 +04:00
Martin Brennan 5ebad3be0a
FIX: Further refine duplicate bookmark delete query (#16931)
In f00e282067 we added this
DELETE query to delete duplicate for_topic bookmarks, we
just need this further refinement to the WHERE clause to
avoid deleting post bookmarks.
2022-05-27 08:22:23 +10:00
Daniel Waterworth f00e282067
FIX: Delete extraneous topic bookmarks (#16926)
Otherwise the migration can fail

Co-authored-by: David Taylor <david@taylorhq.com>
2022-05-26 11:58:15 -05:00
Martin Brennan fcc2e7ebbf
FEATURE: Promote polymorphic bookmarks to default and migrate (#16729)
This commit migrates all bookmarks to be polymorphic (using the
bookmarkable_id and bookmarkable_type) columns. It also deletes
all the old code guarded behind the use_polymorphic_bookmarks setting
and changes that setting to true for all sites and by default for
the sake of plugins.

No data is deleted in the migrations, the old post_id and for_topic
columns for bookmarks will be dropped later on.
2022-05-23 10:07:15 +10:00
David Taylor 991b62b6f1
DEV: Drop old hotlinked image data from post_custom_fields (#16594)
`20220428094026_create_post_hotlinked_media` moved this data into a dedicated table
2022-05-12 15:34:35 +01:00
David Taylor d90065e0ef
Revert "DEV: Update add_column migration to remove transaction (#16715)" (#16716)
This reverts commit e599b5b08f.

This didn't solve the problem we were trying to fix - reverting back to the standard migration pattern
2022-05-11 12:36:11 +01:00
David Taylor e599b5b08f
DEV: Update add_column migration to remove transaction (#16715)
This migration is failing to acquire a lock under some production conditions. We're only performing one action, so removing the transaction is safe and may help to resolve the issue.
2022-05-11 11:28:34 +01:00
Isaac Janzen 1a12e4cfc8
FEATURE: Introduce site setting to allow for non staff pm tagging (#16671)
Currently the only way to allow tagging on pms is to use the `allow_staff_to_tag_pms` site setting.  We are removing that site setting and replacing it with `pm_tags_allowed_for_groups` which will allow for non staff tagging. It will be group based permissions instead of requiring the user to be staff.

If the existing value of `allow_staff_to_tag_pms` is `true` then we include the `staff` groups as a default for `pm_tags_allowed_for_groups`.
2022-05-10 10:02:28 -05:00
Martin Brennan fbcc35b417
DEV: Remove PostAction/UserAction bookmark refs (#16681)
We have not used anything related to bookmarks for PostAction
or UserAction records since 2020, bookmarks are their own thing
now. Deleting all this is just cleaning up old cruft.
2022-05-10 10:42:18 +10:00
Roman Rizzi 6bdcd7afb2
FEATURE: Promote the "delete group" staff action log. (#16656)
We used to log group deletion as custom, which means we can't later search for them. Include group ID in the details.
2022-05-09 12:12:52 -03:00
Osama Sayegh 7e026e8ddf
DEV: Add last_seen_reviewable_id to the users table (#16666)
We're adding this column now in preparation for a future commit(s) that will
redesign the avatar/notifications menu. The reason the column is added in a
separate commit is because the redesign changes are going to be complex with a
high risk of getting (temporarily) reverted and if they included a database
migration, they wouldn't revert cleanly/easily.

Internal ticket: t65045.
2022-05-06 10:11:16 +03:00
David Taylor db9ae32e41
FIX: Skip invalid custom_field json in hotlinked_media migration (#16630)
Truly testing for JSON validity would require defining a new postgres function. Checking just the first character should take care of all the cases of invalid historic data that we've seen.
2022-05-04 15:05:18 +01:00
David Taylor 1c9d8b4999
FIX: Handle invalid JSON from downloaded_images custom fields (#16613)
custom_field data on some sites has duplicate values for a given url key in the JSON value. This is invalid, so we should drop the extra data.
2022-05-03 18:51:25 +01:00
David Taylor bebb4fd111
FIX: Handle duplicates in hotlinked_media migration (#16611)
In the old custom_field-based system, it was possible for a url to be both 'downloaded' and 'broken'. The new table enforces uniqueness, so we need to drop invalid data.
2022-05-03 17:17:27 +01:00
David Taylor 19d2d55011
FIX: Ensure post_hotlinked_media index does not exceed size limit (#16609)
On some installations, this would fail with 'index row size exceeds btree version 4 maximum'. This commit replaces the (post_id, url)` index with a `(post_id, md5(url))` index, which is much more space efficient.
2022-05-03 15:47:58 +01:00
David Taylor c1db968740
DEV: Move hotlinked image information into a dedicated table (#16585)
This will make future changes to the 'pull hotlinked images' system easier. This commit should not introduce any functional change.

For now, the old post_custom_field data is kept in the database. This will be dropped in a future commit.
2022-05-03 13:53:32 +01:00
Loïc Guitaut 008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
Alan Guo Xiang Tan 98c49acad5
DEV: Setup experimental sidebar skeleton (#16575)
* hidden siteSetting to enable experimental sidebar
* user preference to enable experimental sidebar
* `experimental_sidebar_enabled` attribute for current user
* Empty glimmer component for Sidebar
2022-04-28 15:27:06 +08:00
Martin Brennan c841e34b62
FIX: Check if bookmarkable column exists before adding (#16497)
Because in 8040b95e8c we removed
a previous post migrate file, some people may not have had those
original polymorphic bookmark columns removed, and the migration
from this PR running will cause duplicate column errors.

cf. https://meta.discourse.org/t/duplicatecolumn-and-multisite-migrate-failed/224480
2022-04-19 12:01:18 +10:00
Roman Rizzi 6f76a12e0a
FEATURE: Let sites add a sitemap.xml file. (#16357)
* FEATURE: Let sites add a sitemap.xml file.

This PR adds the same features discourse-sitemap provides to core. Sitemaps are only added to the robots.txt file if the `enable_sitemap` setting is enabled and `login_required` disabled.

After merging discourse/discourse-sitemap#34, this change will take priority over the sitemap plugin because it will disable itself. We're also using the same sitemaps table, so our migration won't try to create it
again using `if_not_exists: true`.
2022-04-12 10:33:59 -03:00
David Taylor 1e436f242e
DEV: Move tag_group cleanup from post_migrate -> migrate (#16434)
There is no need to wait until after the deploy for this cleanup. In fact, running it later will mean there could be a window of a few minutes during which the site is broken.

The only requirement is that it runs after the broken `20220401130745_create_category_required_tag_groups` migration.

Followup to 39a6de3d73
2022-04-11 13:28:24 +01:00
Roman Rizzi 39a6de3d73
FIX: Clean required category tag groups with invalid tag_group ids (#16414)
The category table's required_tag_group_id contained references to deleted tag groups, which we copied to the new table. The new serializer tries to get the associated tag group name but fails because the tag group is nil.

This PR adds an inner join in the original migration to make sure tag groups still exist and adds a new post-migration to fix already migrated sites.
2022-04-07 17:02:11 -04:00
David Taylor 68c74e9b93
FEATURE: Allow multiple required tag groups for a category (#16381)
Previously we only supported a single 'required tag group' for a category. This commit allows admins to specify multiple required tag groups, each with their own minimum tag count.

A new category_required_tag_groups database table replaces the existing columns on the categories table. Data is automatically migrated.
2022-04-06 14:08:06 +01:00
Alan Guo Xiang Tan d0c2eb3359
DEV: Drop `enable_long_polling` and `long_polling_interval` settings (#16323)
Scheduled to drop in 2.9.

Co-authored-by: Loïc Guitaut <loic@discourse.org>
2022-03-30 16:32:40 +08:00
Martin Brennan 230e82e948
DEV: Make Bookmark#post_id column nullable (#16287)
As we are gradually moving to having a polymorphic
bookmarkable relationship on the Bookmark table,
we need to make the post_id column nullable to be
able to develop and test the new columns, and
for cutover/migration purposes later as well.
2022-03-28 13:09:13 +10:00
David Taylor f3aab19829
DEV: Promote historic post_deploy migrations (#16288)
This commit promotes all post_deploy migrations which existed in Discourse v2.7.13 (timestamp <= 20210328233843)

This reduces the likelihood of issues relating to migration run order

Also fixes a couple of typos in `script/promote_migrations`
2022-03-25 15:48:20 +00:00
Martin Brennan 8040b95e8c
DEV: Re-add polymorphic bookmark columns (#16261)
This commit is a redo of2f1ddadff7dd47f824070c8a3f633f00a27aacde
which we reverted because it blew up an internal CI check. I looked
into it, and it happened because the old migration to add the bookmark
columns still existed, and those columns were dropped in a post migrate,
so the two migrations to add the columns were conflicting before
the post migrate was run.

------

This commit only includes the creation of the new columns and index,
and does not add any triggers, backfilling, or new data.

A backfill will be done in the final PR when we switch this over.
Intermediate PRs will look something like this:

Add an experimental site setting for using polymorphic bookmarks,
and make sure in the places where bookmarks are created or updated
we fill in the columns. This setting will be used in subsequent
PRs as well.
Listing and searching bookmarks based on polymorphic associations
Creating post and topic bookmarks using polymorphic associations,
and changing special for_topic logic to just rely on the Topic
bookmarkable_type
Querying bookmark reminders based on polymorphic associations
Make sure various other areas like importers, bookmark guardian,
and others all rely on the associations
Prepare plugins that rely on the Bookmark model to use polymorphic
associations
The final core PR will remove all the setting gates and switch over
to using the polymorphic associations, backfill the bookmarks
table columns, and ignore the old post_id and for_topic colummns.
Then it will just be a matter of dropping the old columns down the
line.
2022-03-23 12:43:08 +10:00
Alan Guo Xiang Tan 4e6f7a1a6e
Revert "DEV: Re-add polymorphic bookmark columns (#16245)" (#16246)
This reverts commit 2f1ddadff7.
2022-03-22 13:13:42 +08:00
Martin Brennan 2f1ddadff7
DEV: Re-add polymorphic bookmark columns (#16245)
This commit is a redo of e21c640a3c
which we reverted to not include half-done work in a release.

This commit is slightly different though, in that it only includes
the creation of the new columns and index, and does not add any
triggers, backfilling, or new data.

A backfill will be done in the final PR when we switch this over.
Intermediate PRs will look something like this:

1. Add an experimental site setting for using polymorphic bookmarks,
   and make sure in the places where bookmarks are created or updated
   we fill in the columns. This setting will be used in subsequent
   PRs as well.
2. Listing and searching bookmarks based on polymorphic associations
3. Creating post and topic bookmarks using polymorphic associations,
   and changing special for_topic logic to just rely on the Topic
   bookmarkable_type
4. Querying bookmark reminders based on polymorphic associations
5. Make sure various other areas like importers, bookmark guardian,
   and others all rely on the associations
6. Prepare plugins that rely on the Bookmark model to use polymorphic
   associations

The final core PR will remove all the setting gates and switch over
to using the polymorphic associations, backfill the bookmarks
table columns, and ignore the old post_id and for_topic colummns.
Then it will just be a matter of dropping the old columns down the
line.
2022-03-22 14:26:13 +10:00
Bianca Nenciu 30f3e78834
FIX: Reset last sent for existent bookmarks (#16202)
The meaning of reminder_at and reminder_last_sent_at changed after
commit 6d422a8033. A bookmark reminder
will fire only if reminder_last_sent_at is null, but before that it
fired everytime reminder_at was set. This is no longer true because
sometimes reminder_at continues to exist even after a reminder fired.
2022-03-18 16:31:35 +02:00
Bianca Nenciu 6d422a8033
FEATURE: Highlight expired bookmark reminders (#15317)
The user can select what happens with a bookamrk after it expires. New
option allow bookmark's reminder to be kept even after it has expired.
After a bookmark's reminder notification is created, the reminder date
will be highlighted in red until the user resets the reminder date.
User can do that using the new Clear Reminder button from the dropdown.
2022-03-08 19:44:18 +02:00
Bianca Nenciu 34b4b53bac
FEATURE: Use Postgres unaccent to ignore accents (#16100)
The search_ignore_accents site setting can be used to make the search
indexer remove the accents before indexing the content. The unaccent
function from PostgreSQL is better than Ruby's unicode_normalize(:nfkd).
2022-03-07 23:03:10 +02:00
David Taylor 241e34e6cf
FIX: Update and rebake uses of the old centralized avatar service (#16086)
This URL was originally updated in 89cb537fae. However, some sites are not using the proxy, and have configured their forum to hotlink images directly to avatars.discourse.org.

We intend to shut down this domain in favor of `avatars.discourse-cdn.com`, so this migration will re-write any matching site setting values and queue affected posts for rebaking.
2022-03-07 09:45:20 +00:00
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Alan Guo Xiang Tan e613aef419
DEV: Remove use of Discourse application code in migration. (#16068)
This makes the migration less brittle to changes in the future.

Follow-up to 21a356e3af
2022-02-28 10:48:46 +08:00
Jeff Wong d1bdb6c65d
FEATURE: upload an avatar option for uploading avatars with selectable avatars (#15878)
* FEATURE: upload an avatar option for uploading avatars with selectable avatars

Allow staff or users at or above a trust level to upload avatars even when the site
has selectable avatars enabled.

Everyone can still pick from the list of avatars. The option to upload is shown
below the selectable avatar list.

refactored boolean site setting into an enum with the following values:

disabled: No selectable avatars enabled (default)
everyone: Show selectable avatars, and allow everyone to upload custom avatars
tl1: Show selectable avatars, but require tl1+ and staff to upload custom avatars
tl2: Show selectable avatars, but require tl2+ and staff to upload custom avatars
tl3: Show selectable avatars, but require tl3+ and staff to upload custom avatars
tl4: Show selectable avatars, but require tl4 and staff to upload custom avatars
staff: Show selectable avatars, but only allow staff to upload custom avatars
no_one: Show selectable avatars. No users can upload custom avatars

Co-authored-by: Régis Hanol <regis@hanol.fr>
2022-02-24 12:57:39 -08:00
Joffrey JAFFEUX f5ec32bc8c
FEATURE: adds the user_promoted event to webhooks (#15996) 2022-02-22 10:57:18 +01:00
Martin Brennan fa0c796baf
DEV: Fix SMTP bounce regexp (#16019)
Never trust me with regexp. Follow up to
01ef1d08fc,
which did not take into account codes in
the format X.X.XX (with the 2 digits on the end)
2022-02-22 08:54:01 +10:00
Martin Brennan 01ef1d08fc
FIX: Conform EmailLog#bounce_error_code to RFC (#16010)
This commit makes sure that the email log's bounce_error_code
conforms to the SMTP error code RFC on save, so that
it is always in the format X.X.X or XXX without any
additional string details. Also included is a migration
to fix this issue for past records.
2022-02-21 11:26:39 +10:00
Martin Brennan f9ec2b90a0
DEV: Drop user_stats count column constraints (#15949)
We added this constraint in 5bd55acf83
but it is causing problems in hosted sites and is catching the
issue too far down the line. This commit removes the constraint
for now, and also fixes an issue found with PostDestroyer
which wasn't using the UserStatCountUpdater when updating post_count
and thus was causing negative numbers to occur.
2022-02-16 12:49:11 +11:00
Martin Brennan 4086ee551e
DEV: Add bounce_error_code to EmailLog (#15948)
Whenever we got a bounced email in the Email::Receiver we
previously would just set bounced: true on the EmailLog and
discard the status/diagnostic code. This commit changes this
flow to store the bounce error code (defined in the RFC at
https://www.iana.org/assignments/smtp-enhanced-status-codes/smtp-enhanced-status-codes.xhtml)
not just in the Email::Receiver, but also via webhook events
from other mail services and from SNS.

This commit does not surface the bounce error in the UI,
we can do that later if necessary.
2022-02-15 14:17:26 +10:00
Martin Brennan ad9e791ba7
FIX: Post migration to fix custom emojis with secure URL (#15944)
Follow up to 88a8584348. Sets
the baked version of all posts with custom emoji and a secure
media URL in the cooked content to 0. Then our periodic rebake
posts job will rebake them to apply the fix in the linked
commit. This only matters on sites with secure media enabled.
2022-02-15 10:37:16 +10:00
Blake Erickson 71f7f7ed49
FEATURE: Add external_id to topics (#15825)
* FEATURE: Add external_id to topics

This commit allows for topics to be created and fetched by an
external_id. These changes are API only for now as there aren't any
front changes.

* add annotations

* add external_id to this spec

* Several PR feedback changes

- Add guardian to find topic
- 403 is returned for not found as well now
- add `include_external_id?`
- external_id is now case insensitive
- added test for posts_controller
- added test for topic creator
- created constant for max length
- check that it redirects to the correct path
- restrain external id in routes file

* remove puts

* fix tests

* only check for external_id in webhook if exists

* Update index to exclude external_id if null

* annotate

* Update app/controllers/topics_controller.rb

We need to check whether the topic is present first before passing it to the guardian.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>

* Apply suggestions from code review

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2022-02-08 20:55:32 -07:00
Martin Brennan 0a738bd5bc
FEATURE: Allow sending group SMTP emails with from alias (#15687)
This commit allows group SMTP emails to be sent with a
different from email address that has been set up as an
alias in the email provider. Emails from the alias will
be grouped correctly using Message-IDs in the mail client,
and replies to the alias go into the correct group inbox.
2022-02-07 13:52:01 +10:00
Alan Guo Xiang Tan 5bd55acf83
FIX: Add DB constraints for post & topic counter cache for `UserStat` (#15626)
Ensures that `UserStat#post_count` and `UserStat#topic_count` does not
go below 0. When it does like it did now, we tend to have bugs in our
code since we're usually coding with the assumption that the count isn't
negative.

In order to support the constraints, our post and topic fabricators in
tests will now automatically increment the count for the respective
user's `UserStat` as well. We have to do this because our fabricators
bypasss `PostCreator` which holds the responsibility of updating `UserStat#post_count` and
`UserStat#topic_count`.
2022-02-07 11:23:34 +08:00
Alan Guo Xiang Tan 930f51e175 FEATURE: Split up text segmentation for Chinese and Japanese.
* Chinese segmenetation will continue to rely on cppjieba
* Japanese segmentation will use our port of TinySegmenter
* Korean currently does not rely on segmentation which was dropped in c677877e4f
* SiteSetting.search_tokenize_chinese_japanese_korean has been split
into SiteSetting.search_tokenize_chinese and
SiteSetting.search_tokenize_japanese respectively
2022-02-07 09:21:14 +08:00
Alan Guo Xiang Tan 4aa9a813ec
FIX: Liking whispers should not contribute to `Topic#like_count`. (#15703)
Non-staff users are not allowed to see whisper so this change prevents
non-staff user from seeing a like count that does not make sense to
them. In the future, we might consider adding another like count column
for staff user.

Follow-up to 4492718864
2022-02-03 16:24:33 +11:00
Andrei Prigorshnev dad2e5e513
FEATURE: make the use_email_for_username_and_name_suggestions setting visible and on by default on existing sites (#15751) 2022-02-01 14:55:17 +04:00
Martin Brennan 6b6c0fef4c
DEV: Drop bookmark trigger correctly (#15486)
Apparently `DROP FUNCTION X CASCADE` is the better way to
do this, we were running into cross-schema errors on try.
2022-01-07 15:20:07 +10:00
Martin Brennan 04c7776650
DEV: Rolling back bookmarkable column changes (#15482)
It is too close to release of 2.8 for incomplete
feature shenanigans. Ignores and drops the columns and drops
the trigger/function introduced in
e21c640a3c.
Will pick this feature back up post-release.
2022-01-07 12:16:43 +10:00
Martin Brennan e21c640a3c
DEV: Add polymorphic bookmarkable columns (#15454)
We are planning on attaching bookmarks to more and
more other models, so it makes sense to make a polymorphic
relationship to handle this. This commit adds the new
columns and backfills them in the bookmark table, and
makes sure that any new bookmark changes fill in the columns
via DB triggers.

This way we can gradually change the frontend and backend
to use these new columns, and eventually delete the
old post_id and for_topic columns in `bookmarks`.
2022-01-06 08:56:05 +10:00
Martin Brennan b22450c7a8
DEV: Drop old bookmark columns (#15405)
As per 22208836c5
these are no longer needed, a follow up commit will remove the ignored
columns.
2022-01-04 11:19:27 +10:00
Osama Sayegh 8e6988163f
FIX: Don't allow NULL values for `notification_level` in `category_users` (#15407) 2021-12-29 09:19:39 +11:00
Martin Brennan a263743268
DEV: Remove old enable_experimental_composer_uploader site setting (#15406)
No longer used, the uppy method is now the default for
composer uploads and the old code is deleted.
2021-12-24 15:27:25 +11:00
Osama Sayegh 0192efb37b
FIX: Remove length constraint of `fancy_title` in `topics` (#15381)
The `fancy_title` column in the `topics` table currently has a constraint that limits the column to 400 characters. We need to remove that constraint because it causes some automatic topics/PMs from the system to fail when using Discourse in locales that need more than 400 characters to the translate the content of those automatic messages.

Internal ticket: t58030.
2021-12-21 21:24:23 +03:00
Martin Brennan d5e380e5c1
DEV: Promote uppy backup uploader to primary uploader (#15363)
This commit removes the enable_experimental_backup_uploader site
setting and the flags in backups-index.hbs to make the uppy
backup uploader the main one from now on.

A follow-up commit will delete the old backup uploader code and
also remove resumable.js from the project.
2021-12-20 13:39:35 +10:00
Penar Musaraj 60851037d8
DEV: Remove unused composer editor preview sync (#15322) 2021-12-16 17:18:54 -05:00
David Taylor 7d72041fff
FIX: Correct email_tokens_token_to_nullable migration (#15277)
We were checking for the existence of the column in any schema, including the `backup` schema. This can cause 'column does not exist' errors. In fact, we should only be checking in the `public` schema.
2021-12-13 16:19:09 +00:00
Sam 5fc42bf769
DEV: allow nulls in email_tokens token column (#15271)
This column was dropped in a previous commit, in post migrations.
Unfortunatly that causes smoke tests to fail as there is a period between
migration and post migrations where records can not be inserted into the
table.
2021-12-13 17:38:06 +11:00
Dan Ungureanu 3d4aee1487
DEV: Drop unused column email_tokens.token (#15203) 2021-12-13 16:29:47 +11:00
Angus McLeod df3886d6e5
FEATURE: Experimental support for group membership via google auth (#14835)
This commit introduces a new site setting "google_oauth2_hd_groups". If enabled, group information will be fetched from Google during authentication, and stored in the Discourse database. These 'associated groups' can be connected to a Discourse group via the "Membership" tab of the group preferences UI. 

The majority of the implementation is generic, so we will be able to add support to more authentication methods in the near future.

https://meta.discourse.org/t/managing-group-membership-via-authentication/175950
2021-12-09 12:30:27 +00:00
Dan Ungureanu d8fe0f4199
FEATURE: Restrict link invites to email domain (#15211)
Allow multiple emails to redeem a link invite only if the email domain
name matches the one specified in the link invite.
2021-12-08 17:06:57 +02:00
Dan Ungureanu 9a6ec1d0c6 PERF: Add index on email_tokens.token_hash 2021-12-07 10:17:45 +08:00
Sam 3a73028a70
PERF: migrate normalized_emails in a migrations (#15166)
Old OnceOff job could perform pretty slowly on sites with millions of emails

New implementation operates in batches in a migration, minimizing locking.
2021-12-02 15:08:22 +11:00
Krzysztof Kotlarek 9cabd3721b
FEATURE: ability to add description to tags (#15125)
Ability to add description to tags, which will be displayed on hover.
2021-12-01 09:18:56 +11:00
Loïc Guitaut a5fbb90df4 FEATURE: Display pending posts on user’s page
Currently when a user creates posts that are moderated (for whatever
reason), a popup is displayed saying the post needs approval and the
total number of the user’s pending posts. But then this piece of
information is kind of lost and there is nowhere for the user to know
what are their pending posts or how many there are.

This patch solves this issue by adding a new “Pending” section to the
user’s activity page when there are some pending posts to display. When
there are none, then the “Pending” section isn’t displayed at all.
2021-11-29 10:26:33 +01:00
Penar Musaraj d99deaf1ab
FEATURE: show recent searches in quick search panel (#15024) 2021-11-25 15:44:15 -05:00
David Taylor 03998e0a29
FIX: Use CDN URL for internal onebox avatars (#15077)
This commit will also trigger a background rebake for all existing posts with internal oneboxes
2021-11-25 12:07:34 +00:00
Dan Ungureanu fa8cd629f1
DEV: Hash tokens stored from email_tokens (#14493)
This commit adds token_hash and scopes columns to email_tokens table.
token_hash is a replacement for the token column to avoid storing email
tokens in plaintext as it can pose a security risk. The new scope column
ensures that email tokens cannot be used to perform a different action
than the one intended.

To sum up, this commit:

* Adds token_hash and scope to email_tokens

* Reuses code that schedules critical_user_email

* Refactors EmailToken.confirm and EmailToken.atomic_confirm methods

* Periodically cleans old, unconfirmed or expired email tokens
2021-11-25 09:34:39 +02:00
Bianca Nenciu 3ea8937157
FEATURE: Add email normalization rules setting (#14593)
When this setting is turned on, it will check that normalized emails
are unique. Normalized emails are emails without any dots or plus
aliases.

This setting can be used to block use of aliases of the same email
address.
2021-11-24 11:30:06 +02:00
Alan Guo Xiang Tan 8226ab1099
PERF: Updating first unread PM for user not respecting limits. (#15056)
In b8c8909a9d, we introduced a regression
where users may have had their `UserStat.first_unread_pm_at` set
incorrectly. This commit introduces a migration to reset `UserStat.first_unread_pm_at` back to
`User#created_at`.

Follow-up to b8c8909a9d.
2021-11-23 12:51:54 +08:00
Arpit Jalan d1fc759ac4
FIX: remove 'crawl_images' site setting (#14646) 2021-10-19 17:12:29 +05:30
Martin Brennan 2364626ded
FEATURE: Change all core to use uppy-image-uploader (#14428)
Instead of using image-uploader, which relies on the old
UploadMixin, we can now use the uppy-image-uploader which
uses the new UppyUploadMixin which is stable enough and
supports both regular XHR uploads and direct S3 uploads,
controlled by a site setting (default to XHR).

At some point it may make sense to rename uppy-image-uploader
back to image-uploader, once we have gone through plugins
etc. and given a bit of deprecation time period.

This commit also fixes `for_private_message`, `for_site_setting`,
and `pasted` flags not being sent via uppy uploads onto the
UploadCreator, both via regular XHR uploads and also through
external/multipart uploads.

The uploaders changed are:

* site setting images
* badge images
* category logo
* category background
* group flair
* profile background
* profile card background
2021-10-19 13:25:42 +10:00
Krzysztof Kotlarek cb5b0cb9d8
FEATURE: save local date to calendar (#14486)
It allows saving local date to calendar.
Modal is giving option to pick between ics and google. User choice can be remembered as a default for the next actions.
2021-10-06 14:11:52 +11:00
Roman Rizzi 90a3fbc07b
DEV: Remove HTML setting type and sanitization logic. (#14440)
* DEV: Remove HTML setting type and sanitization logic.

We concluded that we don't want settings to contain HTML, so I'm removing the setting type and sanitization logic. Additionally, we no longer allow the global-notice text to contain HTML.

I searched for usages of this setting type in the `all-the-plugins` repo and found none, so I haven't added a migration for existing settings.

* Mark Global notices containing links as HTML Safe.
2021-10-04 15:40:35 -03:00
Alan Guo Xiang Tan cd64e88711
PERF: Improve database query perf when loading topics for a category. (#14416)
* PERF: Improve database query perf when loading topics for a category.

Instead of left joining the `topics` table against `categories` by filtering with `categories.id`,
we can improve the query plan by filtering against `topics.category_id`
first before joining which helps to reduce the number of rows in the
topics table that has to be joined against the other tables and also
make better use of our existing index.

The following is a before and after of the query plan for a category
with many subcategories.

Before:

```
                                                                                                       QUERY PLAN

-------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
 Limit  (cost=1.28..747.09 rows=30 width=12) (actual time=85.502..2453.727 rows=30 loops=1)
   ->  Nested Loop Left Join  (cost=1.28..566518.36 rows=22788 width=12) (actual time=85.501..2453.722 rows=30 loops=1)
         Join Filter: (category_users.category_id = topics.category_id)
         Filter: ((topics.category_id = 11) OR (COALESCE(category_users.notification_level, 1) <> 0) OR (tu.notification_level > 1))
         ->  Nested Loop Left Join  (cost=1.00..566001.58 rows=22866 width=20) (actual time=85.494..2453.702 rows=30 loops=1)
               Filter: ((COALESCE(tu.notification_level, 1) > 0) AND ((topics.category_id <> 11) OR (topics.pinned_at IS NULL) OR ((t
opics.pinned_at <= tu.cleared_pinned_at) AND (tu.cleared_pinned_at IS NOT NULL))))
               Rows Removed by Filter: 1
               ->  Nested Loop  (cost=0.57..528561.75 rows=68606 width=24) (actual time=85.472..2453.562 rows=31 loops=1)
                     Join Filter: ((topics.category_id = categories.id) AND ((categories.topic_id <> topics.id) OR (categories.id = 1
1)))
                     Rows Removed by Join Filter: 13938306
                     ->  Index Scan using index_topics_on_bumped_at on topics  (cost=0.42..100480.05 rows=715549 width=24) (actual ti
me=0.010..633.015 rows=464623 loops=1)
                           Filter: ((deleted_at IS NULL) AND ((archetype)::text <> 'private_message'::text))
                           Rows Removed by Filter: 105321
                     ->  Materialize  (cost=0.14..36.04 rows=30 width=8) (actual time=0.000..0.002 rows=30 loops=464623)
                           ->  Index Scan using categories_pkey on categories  (cost=0.14..35.89 rows=30 width=8) (actual time=0.006.
.0.040 rows=30 loops=1)
                                 Index Cond: (id = ANY ('{11,53,57,55,54,56,112,94,107,115,116,117,97,95,102,103,101,105,99,114,106,1
13,104,98,100,96,108,109,110,111}'::integer[]))
               ->  Index Scan using index_topic_users_on_topic_id_and_user_id on topic_users tu  (cost=0.43..0.53 rows=1 width=16) (a
ctual time=0.004..0.004 rows=0 loops=31)
                     Index Cond: ((topic_id = topics.id) AND (user_id = 1103877))
         ->  Materialize  (cost=0.28..2.30 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=30)
               ->  Index Scan using index_category_users_on_user_id_and_last_seen_at on category_users  (cost=0.28..2.29 rows=1 width
=8) (actual time=0.004..0.004 rows=0 loops=1)
                     Index Cond: (user_id = 1103877)
 Planning Time: 1.359 ms
 Execution Time: 2453.765 ms
(23 rows)
```

After:

```
                                                                                                                            QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=1.28..438.55 rows=30 width=12) (actual time=38.297..657.215 rows=30 loops=1)
   ->  Nested Loop Left Join  (cost=1.28..195944.68 rows=13443 width=12) (actual time=38.296..657.211 rows=30 loops=1)
         Filter: ((categories.topic_id <> topics.id) OR (topics.category_id = 11))
         Rows Removed by Filter: 29
         ->  Nested Loop Left Join  (cost=1.13..193462.59 rows=13443 width=16) (actual time=38.289..657.092 rows=59 loops=1)
               Join Filter: (category_users.category_id = topics.category_id)
               Filter: ((topics.category_id = 11) OR (COALESCE(category_users.notification_level, 1) <> 0) OR (tu.notification_level > 1))
               ->  Nested Loop Left Join  (cost=0.85..193156.79 rows=13489 width=20) (actual time=38.282..657.059 rows=59 loops=1)
                     Filter: ((COALESCE(tu.notification_level, 1) > 0) AND ((topics.category_id <> 11) OR (topics.pinned_at IS NULL) OR ((topics.pinned_at <= tu.cleared_pinned_at) AND (tu.cleared_pinned_at IS NOT NULL))))
                     Rows Removed by Filter: 1
                     ->  Index Scan using index_topics_on_bumped_at on topics  (cost=0.42..134521.06 rows=40470 width=24) (actual time=38.267..656.850 rows=60 loops=1)
                           Filter: ((deleted_at IS NULL) AND ((archetype)::text <> 'private_message'::text) AND (category_id = ANY ('{11,53,57,55,54,56,112,94,107,115,116,117,97,95,102,103,101,105,99,114,106,113,104,98,100,96,108,109,110,111}'::integer[])))
                           Rows Removed by Filter: 569895
                     ->  Index Scan using index_topic_users_on_topic_id_and_user_id on topic_users tu  (cost=0.43..1.43 rows=1 width=16) (actual time=0.003..0.003 rows=0 loops=60)
                           Index Cond: ((topic_id = topics.id) AND (user_id = 1103877))
               ->  Materialize  (cost=0.28..2.30 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=59)
                     ->  Index Scan using index_category_users_on_user_id_and_last_seen_at on category_users  (cost=0.28..2.29 rows=1 width=8) (actual time=0.004..0.004 rows=0 loops=1)
                           Index Cond: (user_id = 1103877)
         ->  Index Scan using categories_pkey on categories  (cost=0.14..0.17 rows=1 width=8) (actual time=0.001..0.001 rows=1 loops=59)
               Index Cond: (id = topics.category_id)
 Planning Time: 1.633 ms
 Execution Time: 657.255 ms
(22 rows)
```

* PERF: Optimize index on topics bumped_at.

Replace `index_topics_on_bumped_at` index with a partial index on `Topic#bumped_at` filtered by archetype since there is already another index that covers private topics.
2021-09-28 10:05:00 +08:00
jbrw ae09b46098
DEV: update comments on unused Uploads. -4 is now considered reserved. (#14362) 2021-09-16 17:44:51 -04:00
Vinoth Kannan a6de4a5ce9
DEV: use upload id to save in theme setting instead of URL. (#14341)
When we use URL instead it creates the problem while changing the CDN hostname.
2021-09-16 07:58:53 +05:30
Martin Brennan 41e19adb0d
DEV: Ignore reminder_type for bookmarks (#14349)
We don't actually use the reminder_type for bookmarks anywhere;
we are just storing it. It has no bearing on the UI. It used
to be relevant with the at_desktop bookmark reminders (see
fa572d3a7a)

This commit marks the column as readonly, ignores it, and removes
the index, and it will be dropped in a later PR. Some plugins
are relying on reminder_type partially so some stubs have been
left in place to avoid errors.
2021-09-16 09:56:54 +10:00
Martin Brennan d1d2298a4c
DEV: Add for_topic column to bookmarks (#14343)
This new column will be used to indicate that a bookmark
is at the topic level. The first post of a topic can be
bookmarked twice after this change -- with for_topic set
to true and with for_topic set to false.

A later PR will use this column for logic to bookmark the
topic, and then topic-level bookmark links will take you
to the last unread post in the topic.

See also 22208836c5
2021-09-15 11:29:22 +10:00
Martin Brennan 22208836c5
DEV: Ignore bookmarks.topic_id column and remove references to it in code (#14289)
We don't need no stinkin' denormalization! This commit ignores
the topic_id column on bookmarks, to be deleted at a later date.
We don't really need this column and it's better to rely on the
post.topic_id as the canonical topic_id for bookmarks, then we
don't need to remember to update both columns if the bookmarked
post moves to another topic.
2021-09-15 10:16:54 +10:00
Martin Brennan 581482003a
DEV: Change uploads.filesize column to bigint (#14334)
This is necessary to allow for large file uploads via
the direct S3 upload mechanism, as we convert the external
file to an Upload record via ExternalUploadManager once
it is complete.

This will allow for files larger than 2,147,483,647 bytes (2.14GB)
to be referenced in the uploads table.

This is a table locking migration, but since it is not as highly
trafficked as posts, topics, or users, the disruption should be minimal.
2021-09-14 12:20:56 +10:00
Penar Musaraj 90a23c6fc8
FEATURE: Enable auto dark mode on new instances (#14208) 2021-09-02 14:55:38 -04:00
Daniel Waterworth d11f19f099
PERF: Remove redundant post_timings_summary index (#14164)
It's redundant since post_timings_unique exists which has a superset of
the columns with the same prefix.
2021-08-26 10:50:34 -05:00
Martin Brennan d295a16dab
FEATURE: Uppy direct S3 multipart uploads in composer (#14051)
This pull request introduces the endpoints required, and the JavaScript functionality in the `ComposerUppyUpload` mixin, for direct S3 multipart uploads. There are four new endpoints in the uploads controller:

* `create-multipart.json` - Creates the multipart upload in S3 along with an `ExternalUploadStub` record, storing information about the file in the same way as `generate-presigned-put.json` does for regular direct S3 uploads
* `batch-presign-multipart-parts.json` - Takes a list of part numbers and the unique identifier for an `ExternalUploadStub` record, and generates the presigned URLs for those parts if the multipart upload still exists and if the user has permission to access that upload
* `complete-multipart.json` - Completes the multipart upload in S3. Needs the full list of part numbers and their associated ETags which are returned when the part is uploaded to the presigned URL above. Only works if the user has permission to access the associated `ExternalUploadStub` record and the multipart upload still exists.

  After we confirm the upload is complete in S3, we go through the regular `UploadCreator` flow, the same as `complete-external-upload.json`, and promote the temporary upload S3 into a full `Upload` record, moving it to its final destination.
* `abort-multipart.json` - Aborts the multipart upload on S3 and destroys the `ExternalUploadStub` record if the user has permission to access that upload.

Also added are a few new columns to `ExternalUploadStub`:

* multipart - Whether or not this is a multipart upload
* external_upload_identifier - The "upload ID" for an S3 multipart upload
* filesize - The size of the file when the `create-multipart.json` or `generate-presigned-put.json` is called. This is used for validation.

When the user completes a direct S3 upload, either regular or multipart, we take the `filesize` that was captured when the `ExternalUploadStub` was first created and compare it with the final `Content-Length` size of the file where it is stored in S3. Then, if the two do not match, we throw an error, delete the file on S3, and ban the user from uploading files for N (default 5) minutes. This would only happen if the user uploads a different file than what they first specified, or in the case of multipart uploads uploaded larger chunks than needed. This is done to prevent abuse of S3 storage by bad actors.

Also included in this PR is an update to vendor/uppy.js. This has been built locally from the latest uppy source at d613b849a6. This must be done so that I can get my multipart upload changes into Discourse. When the Uppy team cuts a proper release, we can bump the package.json versions instead.
2021-08-25 08:46:54 +10:00
Bianca Nenciu ff367e22fb
FEATURE: Make allow_uploaded_avatars accept TL (#14091)
This gives admins more control over who can upload custom profile
pictures.
2021-08-24 10:46:28 +03:00
Jarek Radosz 07c6b720bc
DEV: Remove `PostProcessed` trigger option (#13916)
It was deprecated 5 years ago in e55e2aff94

I've seen it still being used in the wild, even though it doesn't do anything anymore as I understand it.
2021-08-04 22:24:47 +02:00
Jean e7b8e75583
FEATURE: Add post edits count to user activity (#13495) 2021-08-02 10:15:53 -04:00
Martin Brennan b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
Bianca Nenciu 760c9a5698
FEATURE: Show draft count in user menu and activity (#13812)
This commit adds the number of drafts a user has next to the "Draft"
label in the user preferences menu and activity tab. The count is
updated via MessageBus when a draft is created or destroyed.
2021-07-27 14:05:33 +03:00
mintsaxon 7162ecfb04 FEATURE: Per-category default slow mode duration for topics.
When configured, all topics in the category inherits the slow mode
duration from the category's default.

Note that currently there is no way to remove the slow mode from the
topics once it has been set.
2021-07-21 12:32:07 +08:00