Commit Graph

1123 Commits

Author SHA1 Message Date
Martin Brennan fa572d3a7a
DEV: Remove all code referencing at_desktop bookmark reminders (#9650)
We have found no need for these reminder types, so we are removing the code for them.
2020-05-06 15:22:43 +10:00
David Taylor 0be68f147a
FIX: Avoid using a temporary table in image url database migration
When using pgbouncer without a transaction, it's possible for the connection to change between statements in a migration. Instead, create a real table, and drop it when the migration is finished
2020-05-05 16:29:39 +01:00
David Taylor 03818e642a
FEATURE: Include optimized thumbnails for topics (#9215)
This introduces new APIs for obtaining optimized thumbnails for topics. There are a few building blocks required for this:

- Introduces new `image_upload_id` columns on the `posts` and `topics` table. This replaces the old `image_url` column, which means that thumbnails are now restricted to uploads. Hotlinked thumbnails are no longer possible. In normal use (with pull_hotlinked_images enabled), this has no noticeable impact

- A migration attempts to match existing urls to upload records. If a match cannot be found then the posts will be queued for rebake

- Optimized thumbnails are generated during post_process_cooked. If thumbnails are missing when serializing a topic list, then a sidekiq job is queued

- Topic lists and topics now include a `thumbnails` key, which includes all the available images:
   ```
   "thumbnails": [
   {
     "max_width": null,
     "max_height": null,
     "url": "//example.com/original-image.png",
     "width": 1380,
     "height": 1840
   },
   {
     "max_width": 1024,
     "max_height": 1024,
     "url": "//example.com/optimized-image.png",
     "width": 768,
     "height": 1024
   }
   ]
  ```

- Themes can request additional thumbnail sizes by using a modifier in their `about.json` file:
   ```
    "modifiers": {
      "topic_thumbnail_sizes": [
        [200, 200],
        [800, 800]
      ],
      ...
  ```
  Remember that these are generated asynchronously, so your theme should include logic to fallback to other available thumbnails if your requested size has not yet been generated

- Two new raw plugin outlets are introduced, to improve the customisability of the topic list. `topic-list-before-columns` and `topic-list-before-link`
2020-05-05 09:07:50 +01:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Jeff Wong 2cb9e85d14
FEATURE: add category banner for why a user cannot post (#9576)
* FEATURE: add category banner for why a user cannot post

Adds a category banner for why a user is unable to post in a category.

Also adds an extra alert for the user when a user is unable to create a topic in a
category and they still try and click on the disabled-looking new topic
button.
2020-04-30 10:39:11 -07:00
Sam Saffron 53e0d8fbab
DEV: correct drop logic for columns in post table
We need to recreate the badge_posts view each time we change
column on the posts table.

The p.* is auto expanded on view create, cascade should not have
been used
2020-04-30 12:58:09 +10:00
Sam Saffron 310a7edee5
DEV: remove unused columns from posts and topics
avg_time on posts and topics have not been used in a year.

This uses a re-runnable ddl transaction diasabled migration to
drop the column, cause it touchs very high traffic table and may
deadlock
2020-04-30 11:22:33 +10:00
Martin Brennan ca539fdccf
FIX: Rename all instances of bookmarkWithReminder to just bookmark (#9579)
* Rename all instances of bookmarkWithReminder and bookmark_with_reminder to just bookmark
* Delete old bookmark code at the same time
* Add migration to remove the bookmarkWithReminder post menu item if people have it set in site settings
2020-04-30 10:09:22 +10:00
Martin Brennan 5108cf8ddf
FIX: Topic user bookmarked column logic was not correct (#9563)
Make sure the topic_user.bookmarked column is set correctly when user bookmarks/unbookmarks any post in a topic. For example, you bookmarked a post in the topic that was not the OP, the bookmark icon in the topic list would not be shown. Same if deleting a bookmark for the last bookmarked post in a topic, the bookmark icon in the topic list would not be removed.

Previously this was only setting it to true if bookmarking the OP/topic, which was not correct -- we want to show the icon on the topic list if any post is bookmarked.
Also set to false if unbookmarking the last bookmarked post in the topic.
Also in this PR is a migration to correct any out of sync topic_user.bookmarked columns, based on the new logic.
2020-04-28 16:19:25 +10:00
Sam Saffron 6a18c9aa0b
Revert "FEATURE: enforce_canonical_emails site setting"
This reverts commit 6f9177e2ed.

We decided on a completely different approach to the problem.

Instead we will let blocked emails be treated as canonical.
2020-04-24 13:52:06 +10:00
Vinoth Kannan df0c386f8a
UX: drop the `automatic_membership_retroactive` column from groups model. (#9430) 2020-04-22 22:07:39 +05:30
Dan Ungureanu 4e5f9d4cd1
DEV: Drop 'key' column from user_api_keys (#9388) 2020-04-22 12:13:19 +03:00
Martin Brennan 628ba9d1e2
FEATURE: Promote bookmarks with reminders to core functionality (#9369)
The main thrust of this PR is to take all the conditional checks based on the `enable_bookmarks_with_reminders` away and only keep the code from the `true` path, making bookmarks with reminders the core bookmarks feature. There is also a migration to create `Bookmark` records out of `PostAction` bookmarks for a site.

### Summary

* Remove logic based on whether enable_bookmarks_with_reminders is true. This site setting is now obsolete, the old bookmark functionality is being removed. Retain the setting and set the value to `true` in a migration.
* Use the code from the rake task to create a database migration that creates bookmarks from post actions.
* Change the bookmark report to read from the new table.
* Get rid of old endpoints for bookmarks
* Link to the new bookmarks list from the user summary page
2020-04-22 13:44:19 +10:00
Kane York ea8b40a53b
FEATURE: Add created_at column to user_badges (#9463)
* FEATURE: Add created_at column to user_badges

This makes scraping for newly granted badges easier.

Patch from @eviltrout applied.

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2020-04-20 10:03:25 -07:00
Robin Ward 8b2e7483b2 DEV: Only warn about invalid email setting in production
In development and test mode you don't care about email domains. Now
that `rake db:migrate` outputs a lot less this felt like an error when
it's really a warning.
2020-04-15 11:21:21 -04:00
Sam Saffron 6f9177e2ed
FEATURE: enforce_canonical_emails site setting
The new `enforce_canonical_emails` site setting ensures that emails in the
canonical form are unique.

This mean that if `s.a.m+1@gmail.com` is registered `sam@gmail.com` will
not be allowed.

The commit contains a blanket "tag strip" (stripping everything after +)
it also contains special handling of a "dot strip" for googlemail and gmail.

The setting only impacts new registrations after `enforce_canonical_emails`

The setting is default false so it will not impact any existing installs.
2020-04-14 14:16:30 +10:00
Kane York 138d4aebde FEATURE: Webhooks and Event for user being granted a badge
Adding a webhook for badge revocation is left for future work as it's relatively rare.
2020-04-08 14:38:30 -07:00
Robin Ward e1f8014acd
FEATURE: Support for publishing topics as pages (#9364)
If the feature is enabled, staff members can construct a URL and publish a
topic for others to browse without the regular Discourse chrome.

This is useful if you want to use Discourse like a CMS and publish
topics as articles, which can then be embedded into other systems.
2020-04-08 12:52:36 -04:00
Dan Ungureanu b64b590cfb
FIX: Add index on user_api_keys.key_hash (#9387) 2020-04-08 19:49:18 +03:00
Dan Ungureanu 2e8f54e9ee
DEV: Fix build
Follow up to 0653750fbf.
2020-04-07 17:10:05 +03:00
Dan Ungureanu 0653750fbf
FEATURE: Hash user API keys in the database (#9344)
The 'key' column will be dropped in a future commit.
2020-04-07 16:42:52 +03:00
Martin Brennan 393215266f
FIX: Set null high_priority columns to false in high priority notification migration (#9335)
Some DBs failed on this migration, possibly due to new records being added, with the error

ActiveRecord::NotNullViolation: PG::NotNullViolation: ERROR:  column "high_priority" contains null values

Luckily, the migration is retryable as is.
2020-04-02 09:32:39 +10:00
Martin Brennan b79ea986ac
FEATURE: High priority bookmark reminder notifications (#9290)
Introduce the concept of "high priority notifications" which include PM and bookmark reminder notifications. Now bookmark reminder notifications act in the same way as PM notifications (float to top of recent list, show in the green bubble) and most instances of unread_private_messages in the UI have been replaced with unread_high_priority_notifications.

The user email digest is changed to just have a section about unread high priority notifications, the unread PM section has been removed.

A high_priority boolean column has been added to the Notification table and relevant indices added to account for it.

unread_private_messages has been kept on the User model purely for backwards compat, but now just returns unread_high_priority_notifications count so this may cause some inconsistencies in the UI.
2020-04-01 09:09:20 +10:00
Joffrey JAFFEUX 0996c3b7b3
FEATURE: allows multiple custom emoji groups (#9308)
Note: DBHelper would fail with a sql syntax error on columns like "group".

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2020-03-30 20:16:10 +02:00
David Taylor 2ed0f8d120
DEV: Remove migration which was erroneously added in d62d258f 2020-03-25 14:49:59 +00:00
David Taylor d62d258fe5
Revert "FIX: word boundary regex (\b) not working in Unicode languages. (#9163)"
Lookbehind regex is not supported in Firefox or IE11

This reverts commit 572bb5988f.
2020-03-25 14:34:45 +00:00
Roman Rizzi c4bc734b11
FIX: Backfill topic timer duration (#9249) 2020-03-23 09:38:18 +05:30
Vinoth Kannan aad12822b7
FEATURE: automatically delete replies on a topic after N days. (#9209) 2020-03-19 21:06:31 +05:30
Martin Brennan 793f39139a
FEATURE: Send notifications for time-based and At Desktop bookmark reminders (#9071)
* This PR implements the scheduling and notification system for bookmark reminders. Every 5 minutes a schedule runs to check any reminders that need to be sent before now, limited to **300** reminders at a time. Any leftover reminders will be sent in the next run. This is to avoid having to deal with fickle sidekiq and reminders in the far-flung future, which would necessitate having a background job anyway to clean up any missing `enqueue_at` reminders.

* If a reminder is sent its `reminder_at` time is cleared and the `reminder_last_sent_at` time is filled in. Notifications are only user-level notifications for now.

* All JavaScript and frontend code related to displaying the bookmark reminder notification is contained here. The reminder functionality is now re-enabled in the bookmark modal as well.

* This PR also implements the "Remind me next time I am at my desktop" bookmark reminder functionality. When the user is on a mobile device they are able to select this option. When they choose this option we set a key in Redis saying they have a pending at desktop reminder. The next time they change devices we check if the new device is desktop, and if it is we send reminders using a DistributedMutex. There is also a job to ensure consistency of these reminders in Redis (in case Redis drops the ball) and the at desktop reminders expire after 20 days.

* Also in this PR is a fix to delete all Bookmarks for a user via `UserDestroyer`
2020-03-12 10:16:00 +10:00
Roman Rizzi b9aaa9718d
FIX: When must_approve_users is enabled, we don't want to send suspect users to the review queue. Only non-approved users should be sent. Provide a migration to auto-approve every problematic review item (#9179) 2020-03-11 17:05:44 -03:00
David Taylor d1474e94a1
FEATURE: Allow themes to specify modifiers in their about.json file (#9097)
There are three modifiers:
- serialize_topic_excerpts (boolean)
- csp_extensions (array of strings)
- svg_icons (array of strings)

When multiple themes are active, the values will be combined. The combination method varies based on the setting. CSP/SVG arrays will be combined. serialize_topic_excerpts will use `Enumerable#any`.
2020-03-11 13:30:45 +00:00
Blake Erickson 43b54c631d
DEV: Reserve webhook event types to be used in plugins (#9110)
* DEV: Reserve webhook event types to be used in plugins

Based on feedback on the following PR's:

https://github.com/discourse/discourse-solved/pull/85

https://github.com/discourse/discourse-assign/pull/61

This commit reserves ID's to be used for webhook event types to ensure
that some other webhook or plugin doesn't end up using the same ID.

* Fix broken test

I don't think this test has to test ALL event types to verify that this
feature is working. Now that we added some event types that plugins are
using this test was failing for missing fabricators that exist in the
respective plugins.

* remove loop and just test first record
2020-03-06 10:16:19 -07:00
Martin Brennan e1e74abd4f
FEATURE: Improving bookmarks part 2 -- Topic Bookmarking (#8954)
### UI Changes

If `SiteSetting.enable_bookmarks_with_reminders` is enabled:

* Clicking "Bookmark" on a topic will create a new Bookmark record instead of a post + user action
* Clicking "Clear Bookmarks" on a topic will delete all the new Bookmark records on a topic
* The topic bookmark buttons control the post bookmark flags correctly and vice-versa
Disabled selecting the "reminder type" for bookmarks in the UI because the backend functionality is not done yet (of sending users notifications etc.)

### Other Changes

* Added delete bookmark route (but no UI yet)
* Added a rake task to sync the old PostAction bookmarks to the new Bookmark table, which can be run as many times as we want for a site (it will not create duplicates).
2020-02-13 16:26:02 +10:00
Arpit Jalan 33abd68bdf fix the build. 2020-01-30 17:51:55 +05:30
Arpit Jalan a3115e430e DEV: migration to remove 'bounce_score_threshold_deactivate' setting record 2020-01-30 17:32:57 +05:30
Gerhard Schlager ea11ad4d99 DEV: Drop unused columns 2020-01-27 15:28:56 +01:00
Gerhard Schlager cb54bf4f45 FIX: Don't cause exceptions due to rename of `reply_id` column
This syncs the value of the `reply_id` column into the `reply_post_id` column until all servers have been deployed and the post migrations ran.

Follow-up to ab07b945c2
2020-01-17 19:43:00 +01:00
Gerhard Schlager ab07b945c2
Merge pull request #8736 from gschlager/rename_reply_id_column
REFACTOR: Rename `post_replies.reply_id` column to `post_replies.reply_post_id`
2020-01-17 17:24:49 +01:00
Martin Brennan 7c32411881
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664)
### General Changes and Duplication

* We now consider a post `with_secure_media?` if it is in a read-restricted category.
* When uploading we now set an upload's secure status straight away.
* When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. 
* Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is).
* When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload.

### Viewing Secure Media

* The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions
* If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor`

### Removed

We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled.

* We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context.
* We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-16 13:50:27 +10:00
David Taylor cff6e941de
PERF: Cache ranks for featured badges, to simplify user serialization (#8698) 2020-01-14 14:26:49 +00:00
Jarek Radosz 531016f99b
DEV: Add missing indexes to user_profiles (#8691)
* DEV: Update model annotations
* DEV: Add missing indexes to user_profiles

The columns were changed in 24347ace10 (diff-baa5914c0c7cddf3c8b5cd9139e0d091)
2020-01-09 17:08:55 +01:00
Blake Erickson b7b85f9ade
FEATURE: Turn csp on by default (#8665)
* turn csp on by default

* remove csp migration for new sites now that is is on by default

* Ensure CSP is off before starting qunit
2020-01-06 13:42:21 -07:00
David Taylor 9348d2cfdf
PERF: Cache user badge count in user_stats table (#8610)
This means that we no longer need to run a `count()` on the user_badges table when serializing a user
2019-12-30 11:19:59 +00:00
David Taylor e8beccdaa4 DEV: Drop unused api_keys.key column
This completes the migration to hashed API keys which was started in 4c9ca24ccf
2019-12-19 11:21:49 +00:00
Penar Musaraj ea05a68df7
FIX: Use updated_at date to denote expired invites (#8521) 2019-12-17 10:13:49 -05:00
David Taylor 4c9ca24ccf
FEATURE: Hash API keys in the database (#8438)
API keys are now only visible when first created. After that, only the first four characters are stored in the database for identification, along with an sha256 hash of the full key. This makes key usage easier to audit, and ensures attackers would not have access to the live site in the event of a database leak.

This makes the merge lower risk, because we have some time to revert if needed. Once the change is confirmed to be working, we will add a second commit to drop the `key` column.
2019-12-12 11:45:00 +00:00
Martin Brennan 6261339da9
Improving bookmarks part 1 (#8466)
Note: All of this functionality is hidden behind a hidden, default false, site setting called `enable_bookmarks_with_reminders`. Also, any feedback on Ember code would be greatly appreciated!

This is part 1 of the bookmark improvements. The next PR will address the backend logic to send reminder notifications for bookmarked posts to users. This PR adds the following functionality:

* We are adding a new `bookmarks` table and `Bookmark` model to make the bookmarks a first-class citizen and to allow attaching reminders to them.
* Posts now have a new button in their actions menu that has the icon of an actual book
* Clicking the button opens the new bookmark modal.
* Both name and the reminder type are optional.
* If you close the modal without doing anything, the bookmark is saved with no reminder.
* If you click the Cancel button, no bookmark is saved at all.
* All of the reminder type tiles are dynamic and the times they show will be based on your user timezone set in your profile (this should already be set for you).
* If for some reason a user does not have their timezone set they will not be able to set a reminder, but they will still be able to create a bookmark.
* A bookmark can be deleted by clicking on the book icon again which will be red if the post is bookmarked.

This PR does NOT do anything to migrate or change existing bookmarks in the form of `PostActions`, the two features live side-by-side here. Also this does nothing to the topic bookmarking.
2019-12-11 14:04:02 +10:00
Sam Saffron 0c52537f10 DEV: update rubocop to version 0.77
We like to stay as close as possible to latest with rubocop cause the cops
get better.

This update required some code changes, specifically the default is to avoid
explicit returns where implicit is done

Also this renames a few rules
2019-12-10 11:48:39 +11:00
Mark VanLandingham 14cb386f1e
FEATURE: Featured topic for user profile & card (#8461) 2019-12-09 11:15:47 -08:00
Neil Lalonde 875f0d8fd8
FEATURE: Tag synonyms
This feature adds the ability to define synonyms for tags, and the ability to merge one tag into another while keeping it as a synonym. For example, tags named "js" and "java-script" can be synonyms of "javascript". When searching and creating topics using synonyms, they will be mapped to the base tag.

Along with this change is a new UI found on each tag's page (for example, `/tags/javascript`) where more information about the tag can be shown. It will list the synonyms, which categories it's restricted to (if any), and which tag groups it belongs to (if tag group names are public on the `/tags` page by enabling the "tags listed by group" setting). Staff users will be able to manage tags in this UI, merge tags, and add/remove synonyms.
2019-12-04 13:33:51 -05:00