Commit Graph

3795 Commits

Author SHA1 Message Date
Martin Brennan 08e625c446
FIX: Use previous chunk to check if local backup chunk upload complete (#14896)
Uppy and Resumable slice up their chunks differently, which causes a difference
in this algorithm. Let's take a 131.6MB file (137951695 bytes) with a 5MB (5242880 bytes)
chunk size. For resumable, there are 26 chunks, and uppy there are 27. This is
controlled by forceChunkSize in resumable which is false by default. The final
chunk size is 6879695 (chunk size + remainder) whereas in uppy it is 1636815 (just remainder).

This means that the current condition of uploaded_file_size + current_chunk_size >= total_size
is hit twice by uppy, because it uses a more correct number of chunks. This
can be solved for both uppy and resumable by checking the _previous_ chunk
number * chunk_size as the uploaded_file_size.

An example of what is happening before that change, using the current
chunk number to calculate uploaded_file_size.

chunk 26: resumable: uploaded_file_size (26 * 5242880) + current_chunk_size (6879695) = 143194575 >= total_size (137951695) ? YES
chunk 26: uppy: uploaded_file_size (26 * 5242880) + current_chunk_size (5242880) = 141557760 >= total_size (137951695) ? YES
chunk 27: uppy: uploaded_file_size (27 * 5242880) + current_chunk_size (1636815) = 143194575 >= total_size (137951695) ? YES

An example of what this looks like after the change, using the previous
chunk number to calculate uploaded_file_size:

chunk 26: resumable: uploaded_file_size (25 * 5242880) + current_chunk_size (6879695) = 137951695 >= total_size (137951695) ? YES
chunk 26: uppy: uploaded_file_size (25 * 5242880) + current_chunk_size (5242880) = 136314880 >= total_size (137951695) ? NO
chunk 27: uppy: uploaded_file_size (26 * 5242880) + current_chunk_size (1636815) = 137951695 >= total_size (137951695) ? YES
2021-11-15 15:08:21 +10:00
Daniel Waterworth e7c0bbb9c0
DEV: Let's always give a drop_from param to deprecate (#14901)
So that we know when deprecations can be removed in the future.
2021-11-12 08:52:59 -06:00
Roman Rizzi a3814b1e56
FIX: Display top posts from private categories if the user has access. (#14878)
Users viewing the top topics from the categories page should see those belonging to a private category if they have access to it.
2021-11-11 13:35:03 -03:00
Martin Brennan e4350bb966
FEATURE: Direct S3 multipart uploads for backups (#14736)
This PR introduces a new `enable_experimental_backup_uploads` site setting (default false and hidden), which when enabled alongside `enable_direct_s3_uploads` will allow for direct S3 multipart uploads of backup .tar.gz files.

To make multipart external uploads work with both the S3BackupStore and the S3Store, I've had to move several methods out of S3Store and into S3Helper, including:

* presigned_url
* create_multipart
* abort_multipart
* complete_multipart
* presign_multipart_part
* list_multipart_parts

Then, S3Store and S3BackupStore either delegate directly to S3Helper or have their own special methods to call S3Helper for these methods. FileStore.temporary_upload_path has also removed its dependence on upload_path, and can now be used interchangeably between the stores. A similar change was made in the frontend as well, moving the multipart related JS code out of ComposerUppyUpload and into a mixin of its own, so it can also be used by UppyUploadMixin.

Some changes to ExternalUploadManager had to be made here as well. The backup direct uploads do not need an Upload record made for them in the database, so they can be moved to their final S3 resting place when completing the multipart upload.

This changeset is not perfect; it introduces some special cases in UploadController to handle backups that was previously in BackupController, because UploadController is where the multipart routes are located. A subsequent pull request will pull these routes into a module or some other sharing pattern, along with hooks, so the backup controller and the upload controller (and any future controllers that may need them) can include these routes in a nicer way.
2021-11-11 08:25:31 +10:00
Martin Brennan 6a68bd4825
DEV: Limit list multipart parts to 1 (#14853)
We are only using list_multipart_parts right now in the
uploads controller for multipart uploads to check if the
upload exists; thus we don't need up to 1000 parts.

Also adding a note for future explorers that list_multipart_parts
only gets 1000 parts max, and adding params for max parts
and starting parts.
2021-11-10 08:01:28 +10:00
Bianca Nenciu b203e316ac
FEATURE: Add pagination to API keys page (#14777) 2021-11-09 12:18:23 +02:00
Martin Brennan 9a72a0945f
FIX: Ensure CORS rules exist for S3 using rake task (#14802)
This commit introduces a new s3:ensure_cors_rules rake task
that is run as a prerequisite to s3:upload_assets. This rake
task calls out to the S3CorsRulesets class to ensure that
the 3 relevant sets of CORS rules are applied, depending on
site settings:

* assets
* direct S3 backups
* direct S3 uploads

This works for both Global S3 settings and Database S3 settings
(the latter set directly via SiteSetting).

As it is, only one rule can be applied, which is generally
the assets rule as it is called first. This commit changes
the ensure_cors! method to be able to apply new rules as
well as the existing ones.

This commit also slightly changes the existing rules to cover
direct S3 uploads via uppy, especially multipart, which requires
some more headers.
2021-11-08 09:16:38 +10:00
Martin Brennan fac9185421
FEATURE: Move emoji uploader to use uppy (#14814)
This commit changes the emoji uploader to use the UppyUploadMixin,
and makes some minor changes to the emoji uploader (tightening the
copy for drag and drop and adding a percentage for the upload).

Since no other uppy upload mixin components have needed to upload
multiple files so far, this necessitated adding a tracker for the
in progress uploads so we know when to reset the uploader once all
uploads are complete.

At the moment, the emoji uploader cannot be used for direct S3 uploads
because the admin emoji controller creates other records and does other
magic with the emojis. At some point we need to factor this kind of thing
into the ExternalUploadManager.transform! action to complete external
uploads.
2021-11-04 11:31:58 +10:00
Jean 8d73730c44
FEATURE: Add setting to disable notifications for topic tags edits (#14794) 2021-11-02 13:53:21 -04:00
jbrw cfc62dbace
FIX: allowed_theme_ids should not be persisted in GlobalSettings (#14756)
* FIX: allowed_theme_ids should not be persisted in GlobalSettings

It was observed that the memoized value of `GlobalSetting.allowed_theme_ids` would be persisted across requests, which could lead to unpredictable/undesired behaviours in a multisite environment.

This change moves that logic out of GlobalSettings so that the returned theme IDs are correct for the current site.

Uses get_set_cache, which ultimately uses DistributedCache, which will take care of multisite issues for us.
2021-10-29 11:46:52 -04:00
Jean 92f4cdd330
FEATURE: bypass topic bump when disable_category_edit_notifications is enabled (#14754) 2021-10-27 17:05:10 -04:00
David Taylor e073451eae
UX: Make PresenceChannel changes more responsive (#14733)
For very fast-paced things (e.g. replying... indicators), 5s resolution is not great. This commit improves the resolution to 1 update per second.
2021-10-26 21:15:20 +01:00
Roman Rizzi be38615afd
FEATURE: Mark omniauth failures as HTML safe. (#14713)
Plugins can add HTML elements to auth error messages.
2021-10-26 09:46:25 -03:00
David Taylor 9ac6f1d3bb
FIX: Include the Vary:Accept header on all Accept-based responses (#14647)
By default, Rails only includes the Vary:Accept header in responses when the Accept: header is included in the request. This means that proxies/browsers may cache a response to a request with a missing Accept header, and then later serve that cached version for a request which **does** supply the Accept header. This can lead to some very unexpected behavior in browsers.

This commit adds the Vary:Accept header for all requests, even if the Accept header is not present in the request. If a format parameter (e.g. `.json` suffix) is included in the path, then the Accept header is still omitted. (The format parameter takes precedence over any Accept: header, so the response is no longer varies based on the Accept header)
2021-10-25 12:53:50 +01:00
Jarek Radosz 3b90d7de66
FIX: 2N+1 sql in admin /api/keys endpoint (#14679) 2021-10-21 17:00:17 +02:00
Martin Brennan 2364626ded
FEATURE: Change all core to use uppy-image-uploader (#14428)
Instead of using image-uploader, which relies on the old
UploadMixin, we can now use the uppy-image-uploader which
uses the new UppyUploadMixin which is stable enough and
supports both regular XHR uploads and direct S3 uploads,
controlled by a site setting (default to XHR).

At some point it may make sense to rename uppy-image-uploader
back to image-uploader, once we have gone through plugins
etc. and given a bit of deprecation time period.

This commit also fixes `for_private_message`, `for_site_setting`,
and `pasted` flags not being sent via uppy uploads onto the
UploadCreator, both via regular XHR uploads and also through
external/multipart uploads.

The uploaders changed are:

* site setting images
* badge images
* category logo
* category background
* group flair
* profile background
* profile card background
2021-10-19 13:25:42 +10:00
Krzysztof Kotlarek 9062fd9b7a
FIX: improvements for download local dates (#14588)
* FIX: do not display add to calendar for past dates

There is no value in saving past dates into calendar

* FIX: remove postId and move ICS to frontend

PostId is not necessary and will make the solution more generic for dates which doesn't belong to a specific post.

Also, ICS file can be generated in JavaScript to avoid calling backend.
2021-10-14 09:22:44 +11:00
Bianca Nenciu c4843fc1c1
FEATURE: Allow admins to permanently delete posts and topics (#14406)
Sometimes administrators want to permanently delete posts and topics
from the database. To make sure that this is done for a good reasons,
administrators can do this only after one minute has passed since the
post was deleted or immediately if another administrator does it.
2021-10-13 12:53:23 +03:00
Alan Guo Xiang Tan d0595127cc
FIX: Missing excerpt for post small actions in topic timeline. (#14547) 2021-10-12 09:20:35 +08:00
David Taylor a55642a30a
DEV: Various behind-the-scenes improvements to PresenceChannel (#14518)
- Allow the `/presence/get` endpoint to return multiple channels in a single request (limited to 50)
- When multiple presence channels are initialized in a single Ember runloop, batch them into a single GET request
- Introduce the `presence-pretender` to allow easy testing of PresenceChannel-related features
- Introduce a `use_cache` boolean (default true) on the the server-side PresenceChannel initializer. Useful during testing.
2021-10-07 15:50:14 +01:00
Krzysztof Kotlarek cb5b0cb9d8
FEATURE: save local date to calendar (#14486)
It allows saving local date to calendar.
Modal is giving option to pick between ics and google. User choice can be remembered as a default for the next actions.
2021-10-06 14:11:52 +11:00
Blake Erickson fe676f334a
FEATURE: Return subcategories on categories endpoint (#14492)
* FEATURE: Return subcategories on categories endpoint

When using the API subcategories will now be returned nested inside of
each category response under the `subcategory_list` param. We already
return all the subcategory ids under the `subcategory_ids` param, but
you then would have to make multiple separate API calls to fetch each of
those subcategories. This way you can get **ALL** of the categories
along with their subcategories in a single API response.

The UI will not be affected by this change because you need to pass in
the `include_subcategories=true` param in order for subcategories to be
returned.

In a follow up PR I'll add the API scoping for fetching categories so
that a readonly API key can be used for the `/categories.json` endpoint. This
endpoint should be used instead of the `/site.json` endpoint for
fetching a sites categories and subcategories.

* Update PR based on feedback

- Have spec check for specific subcategory
- Move comparison check out of loop
- Only populate subcategory list if option present
- Remove empty array initialization
- Update api spec to allow null response

* More PR updates based on feedback

- Use a category serializer for the subcategory_list
- Don't include the subcategory_list param if empty
- For the spec check for the subcategory by id
- Fix spec to account for param not present when empty
2021-10-05 12:12:31 -06:00
Roman Rizzi 90a3fbc07b
DEV: Remove HTML setting type and sanitization logic. (#14440)
* DEV: Remove HTML setting type and sanitization logic.

We concluded that we don't want settings to contain HTML, so I'm removing the setting type and sanitization logic. Additionally, we no longer allow the global-notice text to contain HTML.

I searched for usages of this setting type in the `all-the-plugins` repo and found none, so I haven't added a migration for existing settings.

* Mark Global notices containing links as HTML Safe.
2021-10-04 15:40:35 -03:00
Andrei Prigorshnev b609f6c11c
FIX: restrict other user's notification routes (#14442)
It was possible to see notifications of other users using routes:
- notifications/responses
- notifications/likes-received
- notifications/mentions
- notifications/edits

We weren't showing anything private (like notifications about private messages), only things that're publicly available in other places. But anyway, it feels strange that it's possible to look at notifications of someone else. Additionally, there is a risk that we can unintentionally leak something on these pages in the future.

This commit restricts these routes.
2021-09-29 16:24:28 +04:00
Alan Guo Xiang Tan 9d5da2b383
PERF: Revert all inboxes from messages route. (#14445)
The all inboxes was introduced in
016efeadf6 but we decided to roll it back
for performance reasons. The main performance challenge here is that PG
has to basically loop through all the PMs that a user is allowed to view
before being able to order by `Topic#bumped_at`. The all inboxes was not
planned as part of the new/unread filter so we've decided not to tackle
the performance issue for the upcoming release.

Follow-up to 016efeadf6
2021-09-28 11:58:04 +08:00
Jarek Radosz db69b87187
DEV: Fix theme qunit error messages (#14420) 2021-09-22 20:00:19 +02:00
Martin Brennan d0e1c222f7
DEV: Add instrumentation for uploads (#14397)
This commit allows for measuring the time taken for
individual uploads via the new uppy interfaces, only
if the enable_upload_debug_mode site setting is enabled.

Also in this PR, for upload errors with a specific message
locally, we return the real message to show in the modal
instead of the upload.failed message so the developer
does not have to dig around in logs.
2021-09-22 08:43:02 +10:00
Martin Brennan dba6a5eabf
FEATURE: Humanize file size error messages (#14398)
The file size error messages for max_image_size_kb and
max_attachment_size_kb are shown to the user in the KB
format, regardless of how large the limit is. Since we
are going to support uploading much larger files soon,
this KB-based limit soon becomes unfriendly to the end
user.

For example, if the max attachment size is set to 512000
KB, this is what the user sees:

> Sorry, the file you are trying to upload is too big (maximum
size is 512000KB)

This makes the user do math. In almost all file explorers that
a regular user would be familiar width, the file size is shown
in a format based on the maximum increment (e.g. KB, MB, GB).

This commit changes the behaviour to output a humanized file size
instead of the raw KB. For the above example, it would now say:

> Sorry, the file you are trying to upload is too big (maximum
size is 512 MB)

This humanization also handles decimals, e.g. 1536KB = 1.5 MB
2021-09-22 07:59:45 +10:00
Martin Brennan 0c42a1e5f3
FEATURE: Topic-level bookmarks (#14353)
Allows creating a bookmark with the `for_topic` flag introduced in d1d2298a4c set to true. This happens when clicking on the Bookmark button in the topic footer when no other posts are bookmarked. In a later PR, when clicking on these topic-level bookmarks the user will be taken to the last unread post in the topic, not the OP. Only the OP can have a topic level bookmark, and users can also make a post-level bookmark on the OP of the topic.

I had to do some pretty heavy refactors because most of the bookmark code in the JS topics controller was centred around instances of Post JS models, but the topic level bookmark is not centred around a post. Some refactors were just for readability as well.

Also removes some missed reminderType code from the purge in 41e19adb0d
2021-09-21 08:45:47 +10:00
Bianca Nenciu c9ad9bff8a
FIX: Update only passed custom fields (#14357)
It used to replace custom fields instead of updating only the custom
fields that were passed. The changes to custom fields will also be
logged.
2021-09-17 13:37:56 +03:00
Martin Brennan 41e19adb0d
DEV: Ignore reminder_type for bookmarks (#14349)
We don't actually use the reminder_type for bookmarks anywhere;
we are just storing it. It has no bearing on the UI. It used
to be relevant with the at_desktop bookmark reminders (see
fa572d3a7a)

This commit marks the column as readonly, ignores it, and removes
the index, and it will be dropped in a later PR. Some plugins
are relying on reminder_type partially so some stubs have been
left in place to avoid errors.
2021-09-16 09:56:54 +10:00
Alan Guo Xiang Tan 27bad28c53
Partially revert "PERF: Improve query performance all inbox private messages. (#14304)" (#14344)
This partially reverts commit ddb458343d.

Seeing performance degrade on larger sites so back to drawing board on
this one. Instead of the DISTINCT LEFT JOIN, we switch back to
IN(subquery).
2021-09-15 11:32:10 +08:00
Alan Guo Xiang Tan ddb458343d
PERF: Improve query performance all inbox private messages. (#14304)
First reported in https://meta.discourse.org/t/-/202482/19

There are two optimizations being applied here:

1. Fetch a user's group ids in a seperate query instead of including it
   as a sub-query. When I tried a subquery, the query plan becomes very
inefficient.

1. Join against the `topic_allowed_users` and `topic_allowed_groups`
   table instead of doing an IN against a subquery where we UNION the
`topic_id`s from the two tables. From my profiling, this enables PG to
do a backwards index scan on the `index_topics_on_timestamps_private`
index.

This commit fixes a bug where listing all messages was incorrectly
excluding topics if a topic has been archived by a group even if the
user did not belong to the group.

This commit also fixes another bug where dismissing private messages
selectively was subjected to the default limit of 30.
2021-09-15 10:29:42 +08:00
Martin Brennan 22208836c5
DEV: Ignore bookmarks.topic_id column and remove references to it in code (#14289)
We don't need no stinkin' denormalization! This commit ignores
the topic_id column on bookmarks, to be deleted at a later date.
We don't really need this column and it's better to rely on the
post.topic_id as the canonical topic_id for bookmarks, then we
don't need to remember to update both columns if the bookmarked
post moves to another topic.
2021-09-15 10:16:54 +10:00
Bianca Nenciu 6a7ea66670
FEATURE: Use second factor for admin confirmation (#14293)
Administrators can use second factor to confirm granting admin access
without using email. The old method of confirmation via email is still
used as a fallback when second factor is unavailable.
2021-09-14 15:19:28 +03:00
Dan Ungureanu f517b6997c
FEATURE: Cook drafts excerpt in user activity (#14315)
The previous excerpt was a simple truncated raw message. Starting with
this commit, the raw content of the draft is cooked and an excerpt is
extracted from it. The logic for extracting the excerpt mimics the the
`ExcerptParser` class, but does not implement all functionality, being
a much simpler implementation.

The two draft controllers have been merged into one and the /draft.json
route has been changed to /drafts.json to be consistent with the other
route names.
2021-09-14 15:18:01 +03:00
Bianca Nenciu dde66b9e16
FIX: Update only present fields in request (#14310)
Some category fields were always updated, even if they were not present
in the request. When this happened, these field were erased.
2021-09-14 15:04:54 +03:00
Alan Guo Xiang Tan bc23dcd30b
FIX: Don't publish PM archive events to acting user. (#14291)
When a user archives a personal message, they are redirected back to the
inbox and will refresh the list of the topics for the given filter.
Publishing an event to the user results in an incorrect incoming message
because the list of topics has already been refreshed.

This does mean that if a user has two tabs opened, the non-active tab
will not receive the incoming message but at this point we do not think
the technical trade-offs are worth it to support this feature. We
basically have to somehow exclude a client from an incoming message
which is not easy to do.

Follow-up to fc1fd1b416
2021-09-10 09:20:50 +08:00
Alan Guo Xiang Tan 7b77dd5c05
FIX: Display unread/new PM links only when viewing own user. (#14290)
At this point in time, we do not think supporting unread and new when an
admin is looking at another user's messages is worth supporting.

Follow-up to fc1fd1b416
2021-09-09 14:02:17 +08:00
Alan Guo Xiang Tan ee8c943326
FIX: Remove dismissed new topics from PM topic tracking state. (#14288)
Follow-up to fc1fd1b416
2021-09-09 12:39:27 +08:00
Krzysztof Kotlarek e3793e6d7c
FIX: better filter for groups search (#14262)
Follow up of https://github.com/discourse/discourse/pull/14216

Allow plugins to register custom filter with block
2021-09-08 09:38:45 +10:00
Alan Guo Xiang Tan 1937474e84
PERF: Avoid additional database query when viewing own user. (#14239) 2021-09-06 10:38:07 +08:00
Krzysztof Kotlarek f859fd6bde
FEATURE: allow plugins to extend Groups (#14216)
* add_permitted_group_param API for plugins
* add groups-interaction-custom-options outlet
* custom search can use custom group scope
2021-09-06 10:18:51 +10:00
Jean 85c31c73ba
FIX: allow single string values on custom multiple select fields and not just arrays (#14236) 2021-09-03 09:26:57 -04:00
Vinoth Kannan 49b2bb294e
FEATURE: option to update default notification level of existing users. (#14084)
Previously, a group's `default_notification_level` change will only affect the users added after it.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-08-31 16:11:26 +05:30
Vinoth Kannan 08dce4f477
UX: use existing guardian method to check messageable group. (#14174)
We should display "Message" button only if personal messages are enabled. Currently, it's not respecting that site setting.
2021-08-30 10:38:33 +05:30
Vinoth Kannan 465774cf2c
UX: display correct replies count in embedded comments view. (#14175)
Previosuly, the reply count included the "small_action" posts too. It also caused the broken embed HTML issue.
2021-08-30 10:37:53 +05:30
David Taylor 31db83527b DEV: Introduce PresenceChannel API for core and plugin use
PresenceChannel aims to be a generic system for allow the server, and end-users, to track the number and identity of users performing a specific task on the site. For example, it might be used to track who is currently 'replying' to a specific topic, editing a specific wiki post, etc.

A few key pieces of information about the system:
- PresenceChannels are identified by a name of the format `/prefix/blah`, where `prefix` has been configured by some core/plugin implementation, and `blah` can be any string the implementation wants to use.
- Presence is a boolean thing - each user is either present, or not present. If a user has multiple clients 'present' in a channel, they will be deduplicated so that the user is only counted once
- Developers can configure the existence and configuration of channels 'just in time' using a callback. The result of this is cached for 2 minutes.
- Configuration of a channel can specify permissions in a similar way to MessageBus (public boolean, a list of allowed_user_ids, and a list of allowed_group_ids). A channel can also be placed in 'count_only' mode, where the identity of present users is not revealed to end-users.
- The backend implementation uses redis lua scripts, and is designed to scale well. In the future, hard limits may be introduced on the maximum number of users that can be present in a channel.
- Clients can enter/leave at will. If a client has not marked itself 'present' in the last 60 seconds, they will automatically 'leave' the channel. The JS implementation takes care of this regular check-in.
- On the client-side, PresenceChannel instances can be fetched from the `presence` ember service. Each PresenceChannel can be used entered/left/subscribed/unsubscribed, and the service will automatically deduplicate information before interacting with the server.
- When a client joins a PresenceChannel, the JS implementation will automatically make a GET request for the current channel state. To avoid this, the channel state can be serialized into one of your existing endpoints, and then passed to the `subscribe` method on the channel.
- The PresenceChannel JS object is an ember object. The `users` and `count` property can be used directly in ember templates, and in computed properties.
- It is important to make sure that you `unsubscribe()` and `leave()` any PresenceChannel objects after use

An example implementation may look something like this. On the server:

```ruby
register_presence_channel_prefix("site") do |channel|
  next nil unless channel == "/site/online"
  PresenceChannel::Config.new(public: true)
end
```

And on the client, a component could be implemented like this:

```javascript
import Component from "@ember/component";
import { inject as service } from "@ember/service";

export default Component.extend({
  presence: service(),
  init() {
    this._super(...arguments);
    this.set("presenceChannel", this.presence.getChannel("/site/online"));
  },
  didInsertElement() {
    this.presenceChannel.enter();
    this.presenceChannel.subscribe();
  },
  willDestroyElement() {
    this.presenceChannel.leave();
    this.presenceChannel.unsubscribe();
  },
});
```

With this template:

```handlebars
Online: {{presenceChannel.count}}
<ul>
  {{#each presenceChannel.users as |user|}} 
    <li>{{avatar user imageSize="tiny"}} {{user.username}}</li>
  {{/each}}
</ul>
```
2021-08-27 16:26:06 +01:00
Martin Brennan 99ec8eb6df
FIX: Capture S3 metadata when calling create_multipart (#14161)
The generate_presigned_put endpoint for direct external uploads
(such as the one for the uppy-image-uploader) records allowed
S3 metadata values on the uploaded object. We use this to store
the sha1-checksum generated by the UppyChecksum plugin, for later
comparison in ExternalUploadManager.

However, we were not doing this for the create_multipart endpoint,
so the checksum was never captured and compared correctly.

Also includes a fix to make sure UppyChecksum is the last preprocessor to run.
It is important that the UppyChecksum preprocessor is the last one to
be added; the preprocessors are run in order and since other preprocessors
may modify the file (e.g. the UppyMediaOptimization one), we need to
checksum once we are sure the file data has "settled".
2021-08-27 09:50:23 +10:00
Dan Ungureanu 3406a49e21
FEATURE: Create notification for redeemed invite (#14146)
Users can invite people to topic and they will be automatically
redirected to the topic when logging in after signing up. This commit
ensures a "invited_to_topic" notification is created when the invite is
redeemed.

The same notification is used for the "Notify" sharing method that is
found in share topic modal.
2021-08-26 10:43:56 +03:00
Martin Brennan 1646856974
FIX: Topic reset_new unscoped causing huge queries (#14158)
Since ad3ec5809f when a user chooses
the Dismiss New... option in the New topic list, we send a request
to topics/reset-new.json with ?tracked=false as the only parameter.

This then uses Topic as the scope for topics to dismiss, with no
other limitations. When we do topic_scope.pluck(:id), it gets the
ID of every single topic in the database (that is not deleted) to
pass to TopicsBulkAction, causing a huge query with severe performance
issues.

This commit changes the default scope to use
`TopicQuery.new(current_user).new_results(limit: false)`
which should only use the topics in the user's New list, which
will be a much smaller list, depending on the user's "new_topic_duration_minutes"
setting.
2021-08-26 11:25:20 +10:00
Alan Guo Xiang Tan f66007ec83
FEATURE: Display unread and new counts for messages. (#14059)
There are certain design decisions that were made in this commit.

Private messages implements its own version of topic tracking state because there are significant differences between regular and private_message topics. Regular topics have to track categories and tags while private messages do not. It is much easier to design the new topic tracking state if we maintain two different classes, instead of trying to mash this two worlds together.

One MessageBus channel per user and one MessageBus channel per group. This allows each user and each group to have their own channel backlog instead of having one global channel which requires the client to filter away unrelated messages.
2021-08-25 11:17:56 +08:00
Martin Brennan d295a16dab
FEATURE: Uppy direct S3 multipart uploads in composer (#14051)
This pull request introduces the endpoints required, and the JavaScript functionality in the `ComposerUppyUpload` mixin, for direct S3 multipart uploads. There are four new endpoints in the uploads controller:

* `create-multipart.json` - Creates the multipart upload in S3 along with an `ExternalUploadStub` record, storing information about the file in the same way as `generate-presigned-put.json` does for regular direct S3 uploads
* `batch-presign-multipart-parts.json` - Takes a list of part numbers and the unique identifier for an `ExternalUploadStub` record, and generates the presigned URLs for those parts if the multipart upload still exists and if the user has permission to access that upload
* `complete-multipart.json` - Completes the multipart upload in S3. Needs the full list of part numbers and their associated ETags which are returned when the part is uploaded to the presigned URL above. Only works if the user has permission to access the associated `ExternalUploadStub` record and the multipart upload still exists.

  After we confirm the upload is complete in S3, we go through the regular `UploadCreator` flow, the same as `complete-external-upload.json`, and promote the temporary upload S3 into a full `Upload` record, moving it to its final destination.
* `abort-multipart.json` - Aborts the multipart upload on S3 and destroys the `ExternalUploadStub` record if the user has permission to access that upload.

Also added are a few new columns to `ExternalUploadStub`:

* multipart - Whether or not this is a multipart upload
* external_upload_identifier - The "upload ID" for an S3 multipart upload
* filesize - The size of the file when the `create-multipart.json` or `generate-presigned-put.json` is called. This is used for validation.

When the user completes a direct S3 upload, either regular or multipart, we take the `filesize` that was captured when the `ExternalUploadStub` was first created and compare it with the final `Content-Length` size of the file where it is stored in S3. Then, if the two do not match, we throw an error, delete the file on S3, and ban the user from uploading files for N (default 5) minutes. This would only happen if the user uploads a different file than what they first specified, or in the case of multipart uploads uploaded larger chunks than needed. This is done to prevent abuse of S3 storage by bad actors.

Also included in this PR is an update to vendor/uppy.js. This has been built locally from the latest uppy source at d613b849a6. This must be done so that I can get my multipart upload changes into Discourse. When the Uppy team cuts a proper release, we can bump the package.json versions instead.
2021-08-25 08:46:54 +10:00
Bianca Nenciu ff367e22fb
FEATURE: Make allow_uploaded_avatars accept TL (#14091)
This gives admins more control over who can upload custom profile
pictures.
2021-08-24 10:46:28 +03:00
Bianca Nenciu eb6d66fe6f
FIX: Do not allow negative values for LIMIT (#14122)
Negative values generated invalid SQL queries.
2021-08-24 10:45:26 +03:00
Roman Rizzi a50cb61dd5
FIX: Deprecated method should still behave the same. (#14067) 2021-08-19 09:58:26 +08:00
Grayden 64ead3c3a1
FIX: Revoking admin or moderator status doesn't require refresh to delete/anonymize/merge user (#14073)
* FIX: Revoking admin or moderator status doesn't require refresh to delete/anonymize/merge user

On the /admin/users/<id>/<username> page, there are action buttons that are either visible or hidden depending on a few fields from the AdminDetailsSerializer: `can_be_deleted`, `can_be_anonymized`, `can_be_merged`, `can_delete_all_posts`.

These fields are updated when granting/revoking admin or moderator status. However, those updates were not being reflected on the page. E.g. if a user is granted moderation privileges, the 'anonymize user' and 'merge' buttons still appear on the page, which is inconsistent with the backend state of the user. It requires refreshing the page to update the state.

This commit fixes that issue, by syncing the client model state with the server state when handling a successful response from the server. Now, when revoking privileges, the buttons automatically appear without refreshing the page. Similarly, when granting moderator privileges, the buttons automatically disappear without refreshing the page.

* Add detailed user response to spec for changed routes.

Add tests to verify that the revoke_moderation, grant_moderation, and revoke_admin routes return a response formatted according to the AdminDetailedUserSerializer.
2021-08-19 09:57:16 +08:00
Arpit Jalan 7db3888f17
FIX: log proper error message when SSO nonce verification fails (#14077) 2021-08-18 18:44:12 +05:30
Penar Musaraj 08a3aa546b
DEV: Include `login_required` attribute in basic info endpoint (#14064)
This is useful in the DiscourseHub mobile app, currently the app queries
the `about.json` endpoint, which can raise a CORS issue in some cases,
for example when the site only accepts logins from an external provider.
2021-08-17 14:05:51 -04:00
Roman Rizzi 54ad0ecc72
DEV: Deprecate unused route instead of removing it. (#14030)
We'll no longer use this route, so we'll return a 301 with the new URL just in case someone is still using it.
2021-08-12 15:54:14 -03:00
Roman Rizzi 630d485f0f
DEV: Remove unused server-side route. (#14011)
We no longer use this route. When a staff member wants to see a user flagged posts, we redirect them to the review queue.
2021-08-11 17:29:19 -03:00
David Taylor 7dc8f8b794 FEATURE: Allow linking an existing account during external-auth signup
When a user signs up via an external auth method, a new link is added to the signup modal which allows them to connect an existing Discourse account. This will only happen if:

- There is at least 1 other auth method available

and

- The current auth method permits users to disconnect/reconnect their accounts themselves
2021-08-10 15:07:40 +01:00
David Taylor 46dc189850 DEV: Improve robustness of associate_accounts_controller
This handles a few edge cases which are extremely rare (due to the UI layout), but still technically possible:

- Ensure users are authenticated before attempting association.

- Add a message and logic for when a user already has an association for a given auth provider.
2021-08-10 15:07:40 +01:00
David Taylor 2cae29f644 DEV: Update associate_accounts_controller to use secure_session
This is much cleaner than using redis directly. It also opens the door to more complex association change flows which may happen during login.
2021-08-10 15:07:40 +01:00
Roman Rizzi 044de6d670
DEV: Give callback listeners access to the request object. (#13965)
Plugins listening on the `before_auth` callback can interact with the request object and access data like the user agent or the remote IP address. We'll later store this data in the user record, but it might not exist at this point if we're authenticating a new account.
2021-08-06 11:26:11 -03:00
Alan Guo Xiang Tan 3f59ccefd7 FIX: Remove limit on dismissing unread and new messages.
Follow-up to 2c046cc670
2021-08-05 14:55:38 +08:00
Alan Guo Xiang Tan 2c046cc670 FEATURE: Dismiss new and unread for PM inboxes. 2021-08-05 12:56:15 +08:00
Alan Guo Xiang Tan 016efeadf6
FEATURE: New and Unread messages for user personal messages. (#13603)
* FEATURE: New and Unread messages for user personal messages.

Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
2021-08-02 12:41:41 +08:00
Jean ac777440fd
FIX: Validate value of custom dropdown user fields - dropdowns and multiple selects (#13890) 2021-07-30 13:50:47 -04:00
Joffrey JAFFEUX 74f0631acd
FIX: allows authentication data to be present in bootstrap (#13885) 2021-07-29 15:01:11 +02:00
Alan Guo Xiang Tan 2b5625bbf0
FIX: Avoid creating a post revision when topic tags have not changed. (#13881)
Co-authored-by: jmperez127 <jmperez127@gmail.com>
2021-07-29 08:14:25 -04:00
David Taylor c94879ea43 DEV: Remove incorrect method descriptions
These do not accurately describe the methods. They were likely copy/pasted from another controller.
2021-07-29 18:23:56 +08:00
Andrei Prigorshnev 5a2ad7e386
DEV: remove calls to guardian from GroupActionLogger (#13835)
We shouldn't be checking if a user is allowed to do an action in the logger. We should be checking it just before we perform the action. In fact, guardians in the logger can make things even worse in case of a security bug. Let's say we forgot to check user's permissions before performing some action, but we still have a call to the guardian in the logger. In this case, a user would perform the action anyway, and this action wouldn't even be logged!

I've checked all cases and I confirm that we're safe to delete this calls from the logger.

I've added two calls to guardians in admin/user_controller. We didn't have security bugs there, because regular users can't access admin/... routes at all. But it's good to have calls to guardian in these methods anyway, neighboring methods have them.
2021-07-28 15:04:04 +04:00
Martin Brennan b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
Martin Brennan e1d2b67178
DEV: Fix stylesheet manager flaky spec (#13846)
The cache_fullpath for the Stylesheet::Manager was the same for
every test runner in a parallel test environment, so when other
specs or other places e.g. the stylesheets_controller_spec ran
rm -rf Stylesheet::Manager.cache_fullpath this caused errors
for other specs running that went through the
Stylesheet::Manager::Builder#compile path, causing the error

```
Errno::ENOENT:
       No such file or directory @ rb_sysopen
```


Also fixed the stylesheet_controller which was interpolating Rails.root + CACHE_PATH
itself instead of just using Stylesheet::Manager.cache_fullpath
2021-07-26 14:22:54 +10:00
Blake Erickson 6ac3f1f7b5
DEV: Return 400 instead of 500 for invalid top period (#13828)
* DEV: Return 400 instead of 500 for invalid top period

This change will prevent a fatal 500 error when passing in an invalid
period param value to the `/top` route.

* Check if the method exists first

I couldn't get `ListController.respond_to?` to work, but was still able
to check if the method exists with
`ListController.action_methods.include?`. This way we can avoid relying
on the `NoMethodError` exception which may be raised during the course
of executing the method.

* Just check if the period param value is valid

* Use the new TopTopic.validate_period method
2021-07-23 14:58:10 -06:00
Robin Ward 7b45a5ce55 FIX: Better and more secure validation of periods for TopicQuery
Co-authored-by: Martin Brennan <mjrbrennan@gmail.com>
2021-07-23 14:24:44 -04:00
Andrei Prigorshnev 8bc01c1bb5
DEV: extract leave_group method from the group#remove_member method (#13823)
* Copy remove_member to new `leave` method

* Remove unneeded code from the leave method

* Rearrange the leave method

* Remove unneeded code from the remove_member method

* Add tests

* Implement on the client side
2021-07-22 20:14:18 +04:00
Andrei Prigorshnev 73e8183ffb
FIX: update the list of users after user joined group (#13822)
* Make UI update after user have joined a public group

* Check if a group exists in the join method
2021-07-22 18:48:26 +04:00
Andrei Prigorshnev 3cf7a3766a
DEV: extract join_group method from groups#add_members method (#13807)
* Copy the add_members method to the new join method

* Remove unneeded code from the join method

* Rearrange the join method

* Remove unneeded stuff from the add_members method

* Extract add_user_to_group method

* Implement of the client side

* Tests

* Doesn't inline users.uniq

* Return promise from join.then()

* Remove unnecessary begin and end

* Revert "Return promise from join.then()"

This reverts commit bda84d8d

* Remove variable already_in_group
2021-07-22 11:11:23 +04:00
Robin Ward 78ab3f37b8 FIX: Don't proxy `/qunit` URL when using Ember CLI
This is confusing because you're running the tests on the older version
of Ember. Use `/tests` for Ember CLI, and `/qunit` when using Rails'
asset pipeline (but only if REALLY necessary!)
2021-07-21 15:21:09 -04:00
Penar Musaraj 2ce2c83bc9
FIX: Show user filter hints when typing `@` in search (#13799)
Will show the last 6 seen users as filtering suggestions when typing @ in quick search. (Previously the user suggestion required a character after the @.)

This also adds a default limit of 6 to the user search query, previously the backend was returning 20 results but a maximum of 6 results was being shown anyway.
2021-07-21 09:14:53 -04:00
Krzysztof Kotlarek 40f6ceb6f2
FIX: display warning when SSO email is different from invite email (#13804)
In this commit, we skipped frontend validation when email is obfuscated:
https://github.com/discourse/discourse/commit/534008ba24c

However, if email from SSO is different from email from invite, we should still display warning.
2021-07-21 17:03:04 +10:00
mintsaxon 7162ecfb04 FEATURE: Per-category default slow mode duration for topics.
When configured, all topics in the category inherits the slow mode
duration from the category's default.

Note that currently there is no way to remove the slow mode from the
topics once it has been set.
2021-07-21 12:32:07 +08:00
Andrei Prigorshnev 1a8c949900
UX: suspend forever time period messages (#13776)
When the Forever option is selected for suspending a user, the user is suspended for 1000 years. Without customizing the site’s text, this time period is displayed to the user in the suspension email that is sent to the user, and if the user attempts to log back into the site. Telling someone that they have been suspended for 1000 years seems likely to come across as a bad attempt at humour.

This PR special case messages when a user suspended or silenced forever.
2021-07-20 14:42:08 +04:00
Rafael dos Santos Silva e12b00eab7
FEATURE: Stop checking referer for embeds (#13756)
Flips content_security_policy_frame_ancestors default to enabled, and
removes HTTP_REFERER checks on embed requests, as the new referer
privacy options made the check fragile.
2021-07-16 15:25:49 -03:00
Andrei Prigorshnev 1cadae3879
FIX: simplify and improve choosing favorite badges (#13743)
* No need to return anything except a status code from the server

* Switch a badge state before sending a request and then switch it back in case of an error
2021-07-16 11:13:00 +08:00
Penar Musaraj 55bed48917
DEV: Remove stylesheet controller non-prod code (#13745) 2021-07-15 12:52:40 -04:00
Vinoth Kannan d6fc39c886
FEATURE: update existing users when group default notifications changed. (#13434)
Currently, the changes will only affect the users added after.
2021-07-15 19:53:57 +05:30
Andrei Prigorshnev 5cd447695e
FIX: problems with choosing favorite badges (#13731) 2021-07-15 14:51:44 +04:00
Osama Sayegh 31aa701518
FEATURE: Add option to grant badge multiple times to users using Bulk Award (#13571)
Currently when bulk-awarding a badge that can be granted multiple times, users in the CSV file are granted the badge once no matter how many times they're listed in the file and only if they don't have the badge already.

This PR adds a new option to the Badge Bulk Award feature so that it's possible to grant users a badge even if they already have the badge and as many times as they appear in the CSV file.
2021-07-15 05:53:26 +03:00
Martin Brennan 7911124d3d
FEATURE: Uppy image uploader with UppyUploadMixin (#13656)
This PR adds the first use of Uppy in our codebase, hidden behind a enable_experimental_image_uploader site setting. When the setting is enabled only the user card background uploader will use the new uppy-image-uploader component added in this PR.

I've introduced an UppyUpload mixin that has feature parity with the existing Upload mixin, and improves it slightly to deal with multiple/single file distinctions and validations better. For now, this just supports the XHRUpload plugin for uppy, which keeps our existing POST to /uploads.json.
2021-07-13 12:22:00 +10:00
Mark VanLandingham 6a9dc556bc
DEV: DiscourseEvent for category updated (#13707) 2021-07-12 14:23:07 -05:00
Arpit Jalan 953fd2cb50
FEATURE: add title tag for group detail page (#13702) 2021-07-12 20:05:57 +05:30
Dan Ungureanu 2d904ade6d
FIX: Let staged users choose their username (#13678)
When a staged user tried to redeem an invite, a different username was
suggested and manually typing the staged username failed because the
username was not available.
2021-07-12 07:57:38 +10:00
Bianca Nenciu 87c1e98571
FEATURE: Let users select flair (#13587)
User flair was given by user's primary group. This PR separates the
two, adds a new field to the user model for flair group ID and users
can select their flair from user preferences now.
2021-07-08 10:46:21 +03:00
Dan Ungureanu 9969631cef
FEATURE: Make invites work with existing users (#13532)
* FEATURE: Redirect logged in user to invite topic

Users who were already logged in and were given an invite link to a
topic used to see an error message saying that they already have an
account and cannot redeem the invite. This commit amends that behavior
and redirects the user directly to the topic, if they can see it.

* FEATURE: Add logged in user to invite groups

Users who were already logged in and were given an invite link to a
group used to see an error message saying that they already have an
account and cannot redeem the invite. This commit amends that behavior
and adds the user to the group.
2021-07-07 19:42:42 +03:00
Vinoth Kannan 33eae4cbd8
FEATURE: add period filter in top topics route for tags. (#13415)
And also move all the "top topics by period" routes to query string param.

/top/monthly => /top?period=monthly
/c/:slug/:id/l/top/monthly => /c/:slug/:id/l/top?period=monthly
/tag/:slug/l/top/daily => /tag/:slug/l/top?period=daily (new)
2021-07-06 15:25:11 +05:30
Dan Ungureanu 34387c5a38
FEATURE: Warn if invited user cannot see topic (#13548)
Users can invite people to topics from secured category, but they will
not be redirected to the topic after signing up unless they have the
permissions to view the topic. This commit shows a warning when invite
is saved if the topic is in a secured category and none of the invite
groups are allowed to see it.
2021-07-06 12:49:26 +03:00
Jarek Radosz fdb85b8025
DEV: Add theme-color meta to bootstrap (#13615)
It's rendered in Rails in:

ebcc70a5ba/app/views/layouts/_head.html.erb (L9-L9)
2021-07-02 16:43:10 +02:00