discourse/spec/fabricators
Martin Brennan d295a16dab
FEATURE: Uppy direct S3 multipart uploads in composer (#14051)
This pull request introduces the endpoints required, and the JavaScript functionality in the `ComposerUppyUpload` mixin, for direct S3 multipart uploads. There are four new endpoints in the uploads controller:

* `create-multipart.json` - Creates the multipart upload in S3 along with an `ExternalUploadStub` record, storing information about the file in the same way as `generate-presigned-put.json` does for regular direct S3 uploads
* `batch-presign-multipart-parts.json` - Takes a list of part numbers and the unique identifier for an `ExternalUploadStub` record, and generates the presigned URLs for those parts if the multipart upload still exists and if the user has permission to access that upload
* `complete-multipart.json` - Completes the multipart upload in S3. Needs the full list of part numbers and their associated ETags which are returned when the part is uploaded to the presigned URL above. Only works if the user has permission to access the associated `ExternalUploadStub` record and the multipart upload still exists.

  After we confirm the upload is complete in S3, we go through the regular `UploadCreator` flow, the same as `complete-external-upload.json`, and promote the temporary upload S3 into a full `Upload` record, moving it to its final destination.
* `abort-multipart.json` - Aborts the multipart upload on S3 and destroys the `ExternalUploadStub` record if the user has permission to access that upload.

Also added are a few new columns to `ExternalUploadStub`:

* multipart - Whether or not this is a multipart upload
* external_upload_identifier - The "upload ID" for an S3 multipart upload
* filesize - The size of the file when the `create-multipart.json` or `generate-presigned-put.json` is called. This is used for validation.

When the user completes a direct S3 upload, either regular or multipart, we take the `filesize` that was captured when the `ExternalUploadStub` was first created and compare it with the final `Content-Length` size of the file where it is stored in S3. Then, if the two do not match, we throw an error, delete the file on S3, and ban the user from uploading files for N (default 5) minutes. This would only happen if the user uploads a different file than what they first specified, or in the case of multipart uploads uploaded larger chunks than needed. This is done to prevent abuse of S3 storage by bad actors.

Also included in this PR is an update to vendor/uppy.js. This has been built locally from the latest uppy source at d613b849a6. This must be done so that I can get my multipart upload changes into Discourse. When the Uppy team cuts a proper release, we can bump the package.json versions instead.
2021-08-25 08:46:54 +10:00
..
allowed_pm_users.rb FEATURE: Allow List for PMs (#10270) 2020-07-20 15:23:49 -06:00
api_key_fabricator.rb FEATURE: Hash API keys in the database (#8438) 2019-12-12 11:45:00 +00:00
badge_fabricator.rb
bookmark_fabricator.rb FEATURE: Send notifications for time-based and At Desktop bookmark reminders (#9071) 2020-03-12 10:16:00 +10:00
category_fabricator.rb FIX: Make inline oneboxes work with secured topics in secured contexts (#8895) 2020-02-12 12:11:28 +02:00
category_group_fabricator.rb
color_scheme_color_fabricator.rb
color_scheme_fabricator.rb
dimissed_topic_user.rb FEATURE: New way to dismiss new topics (#11927) 2021-02-04 11:27:34 +11:00
do_not_disturb_fabricator.rb FEATURE: Do not disturb (#11484) 2020-12-18 09:03:51 -06:00
email_change_request_fabricator.rb
email_log_fabricator.rb
email_token_fabricator.rb
embeddable_host_fabricator.rb
external_upload_stub_fabricator.rb FEATURE: Uppy direct S3 multipart uploads in composer (#14051) 2021-08-25 08:46:54 +10:00
flag_fabricator.rb
group_fabricator.rb DEV: Add SMTP group ID to EmailLog (#13381) 2021-06-15 11:29:46 +10:00
group_history_fabricator.rb
group_request_fabricator.rb
group_user_fabricator.rb
ignored_user_fabricator.rb DEV: Cleanup ignored user logic (#11107) 2020-11-03 12:38:54 +00:00
incoming_email_fabricator.rb FIX: Change default for IncomingEmail#created_via to 0 (unknown) and make NOT NULL (#11782) 2021-01-21 12:59:50 +10:00
incoming_link_fabricator.rb
invite_fabricator.rb
invited_user_fabricator.rb FEATURE: multiple use invite links (#9813) 2020-06-09 20:49:32 +05:30
like_fabricator.rb
muted_user.rb
notification_fabricator.rb FEATURE: High priority bookmark reminder notifications (#9290) 2020-04-01 09:09:20 +10:00
optimized_image_fabricator.rb DEV: Fix OptimizedImage specs 2020-07-06 21:51:56 +02:00
permalink_fabricator.rb
post_action_fabricator.rb FIX: correct user serializer user method for extended serializer (#8590) 2019-12-19 09:48:01 -08:00
post_custom_field_fabricator.rb
post_detail_fabricator.rb
post_fabricator.rb Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse"" 2020-05-23 00:56:13 -04:00
post_reply_key_fabricator.rb
post_revision_fabricator.rb
published_page_fabricator.rb FEATURE: allows published pages to be public (#10053) 2020-06-17 12:42:20 +02:00
reviewable_claimed_topic_fabricator.rb
reviewable_fabricator.rb FIX: Granting staff status should auto-approve users waiting approval (#8533) 2019-12-12 16:26:38 -03:00
screened_email_fabricator.rb
screened_ip_address_fabricator.rb
screened_url_fabricator.rb
search_log_fabricator.rb
shared_draft_fabricator.rb
single_sign_on_record_fabricator.rb
skipped_email_log_fabricator.rb
tag_fabricator.rb
tag_group_fabricator.rb
theme_fabricator.rb
theme_field_fabricator.rb FIX: add theme field errors (#12880) 2021-04-28 15:00:37 -07:00
topic_allowed_group_fabricator.rb
topic_allowed_user_fabricator.rb
topic_embed_fabricator.rb
topic_fabricator.rb DEV: Add include_pms option to TopicQuery (#10647) 2020-09-14 12:07:35 +01:00
topic_tag_fabricator.rb
topic_timer_fabricator.rb DEV: Use dynamic/static fabricator attrs correctly (#9519) 2020-04-22 20:49:53 +02:00
topic_user_fabricator.rb
upload_fabricator.rb FEATURE: Use path from existing URL of uploads and optimized images (#13177) 2021-05-27 17:42:25 +02:00
user_action_fabricator.rb
user_api_key_fabricator.rb DEV: Introduce plugin API to contribute user api key scopes 2020-10-19 10:40:55 +01:00
user_avatar_fabricator.rb
user_email_fabricator.rb
user_fabricator.rb FEATURE: Disallow putting urls in the title for TL-0 users (#13947) 2021-08-05 13:38:39 +04:00
user_field_fabricator.rb
user_field_option_fabricator.rb.rb FIX: Validate value of custom dropdown user fields - dropdowns and multiple selects (#13890) 2021-07-30 13:50:47 -04:00
user_option_fabricator.rb
user_profile_fabricator.rb
user_second_factor_fabricator.rb
user_security_key_fabricator.rb
watched_word_fabricator.rb DEV: Use dynamic/static fabricator attrs correctly (#9519) 2020-04-22 20:49:53 +02:00
web_crawler_request_fabricator.rb DEV: Use dynamic/static fabricator attrs correctly (#9519) 2020-04-22 20:49:53 +02:00
web_hook_fabricator.rb FEATURE: add support for like webhooks (#12917) 2021-04-30 17:08:38 -07:00