discourse/spec/components
Martin Brennan b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
..
auth FEATURE: Add global admin api key rate limiter (#12527) 2021-06-03 10:52:43 +01:00
common_passwords DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
concern FIX: Nil-filled CF arrays were not being deleted (#13518) 2021-06-25 11:34:51 +02:00
email FEATURE: Use group SMTP job and mailer instead of UserNotifications change (#13489) 2021-06-28 08:55:13 +10:00
file_store FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787) 2021-07-28 08:42:25 +10:00
freedom_patches DEV: Don't user before(:all)/after(:all) (#13389) 2021-06-15 17:25:06 +02:00
guardian FIX: don't allow category and tag tracking settings on staged users (#13688) 2021-07-16 14:50:40 -04:00
highlight_js DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
imap FEATURE: Use group SMTP settings for sending user notification emails (initial) (#13220) 2021-06-03 14:47:32 +10:00
import DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
middleware DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
migration FIX: Allow post migrations using `#change` to carry out unsafe migration 2020-05-15 14:23:27 +08:00
plugin FEATURE: Add new plugin API to allow plugins to extend `Site#categories` (#13773) 2021-07-19 13:54:19 +08:00
pretty_text SPEC: 'lookup_upload_urls' method should use cdn url if available. 2019-10-14 12:57:33 +05:30
rate_limiter DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
scheduler DEV: reduce logging when no external id is specified 2020-04-08 12:42:28 +10:00
site_settings DEV: Isolate multisite specs (#13634) 2021-07-07 18:57:42 +02:00
stylesheet FIX: Add order to outputted stylesheet link tags (#13735) 2021-07-15 12:51:46 -04:00
svg_sprite FIX: Issues with custom icons in themes (#13732) 2021-07-14 15:18:29 -04:00
theme_store FEATURE: Allow themes to specify modifiers in their about.json file (#9097) 2020-03-11 13:30:45 +00:00
validators FIX: Validate email_accent_bg_color color (#13778) 2021-07-22 17:42:47 +03:00
wizard UX: Remove emoji step in wizard (#12169) 2021-02-22 12:14:18 -05:00
admin_confirmation_spec.rb Update rubocop to 2.3.1. 2020-07-24 17:19:21 +08:00
admin_user_index_query_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
archetype_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
cache_spec.rb FIX: ensures defined expired_in is passed from write to write_entry (#11622) 2021-01-04 10:34:44 +01:00
category_badge_spec.rb FIX: Correctly escape category description text (#8107) 2019-10-01 12:04:39 -04:00
composer_messages_finder_spec.rb FIX: Don't show the get a room composer message in private categories (#12702) 2021-04-14 12:34:13 -05:00
content_buffer_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
cooked_post_processor_spec.rb FEATURE: Enable pausing images from Giphy and Tenor (#13185) 2021-05-27 15:00:38 -04:00
crawler_detection_spec.rb FEATURE: Implement browser update in crawler view (#12448) 2021-03-22 19:41:42 +02:00
current_user_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
directory_helper_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
discourse_diff_spec.rb Remove focus from specs 2019-10-16 14:28:04 -04:00
discourse_event_spec.rb DEV: Plugin API to add directory columns (#13440) 2021-06-22 13:00:04 -05:00
discourse_hub_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
discourse_plugin_registry_spec.rb DEV: Remove deprecated plugins variables importer (#12168) 2021-02-23 16:20:59 -05:00
discourse_redis_spec.rb DEV: Remove specs that are no longer relevant. 2020-06-23 12:09:04 +08:00
discourse_spec.rb DEV: Improve Ember CLI's bootstrap logic (#12792) 2021-04-23 10:24:42 -04:00
discourse_tagging_spec.rb FIX: Show required tags to staff by default and override limit (#13242) 2021-06-02 12:43:34 -04:00
discourse_updates_spec.rb FIX: Sort admin dashboard new updates by latest (#12146) 2021-02-19 11:03:36 -05:00
distributed_memoizer_spec.rb DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
distributed_mutex_spec.rb DEV: Improve flaky time-sensitive specs (#9141) 2020-03-10 22:13:17 +01:00
email_cook_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
email_updater_spec.rb FEATURE: add maximum limit for secondary emails (#12599) 2021-04-05 20:31:42 +05:30
enum_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
excerpt_parser_spec.rb FIX: Make Oneboxer#apply insert block Oneboxes correctly (#11449) 2020-12-14 17:49:37 +02:00
feed_element_installer_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
feed_item_accessor_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
file_helper_spec.rb DEV: properly clean up temp files in FileHelper spec 2019-05-28 11:33:08 +10:00
filter_best_posts_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
final_destination_spec.rb DEV: Oneboxer wildcard subdomains (#13015) 2021-05-13 15:48:35 -04:00
flag_settings_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
gaps_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
global_path_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
guardian_spec.rb FIX: Allow only groups with flairs to be selected (#13744) 2021-07-21 14:41:04 +03:00
has_errors_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
hijack_spec.rb FIX: strip the trailing slash (/) of cors origins. (#10996) 2020-10-29 13:01:06 +11:00
html_prettify_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
html_to_markdown_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
image_sizer_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
inline_oneboxer_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
js_locale_helper_spec.rb FEATURE: Add English (UK) as locale (#11768) 2021-01-20 21:32:22 +01:00
json_error_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
letter_avatar_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
method_profiler_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
new_post_manager_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
new_post_result_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
oneboxer_spec.rb FEATURE: Censor Oneboxes (#12902) 2021-06-03 11:39:12 +10:00
onpdiff_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
pbkdf2_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
pinned_check_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
plain_text_to_markdown_spec.rb FIX: use URI.regexp to find URLs in plain text 2019-06-07 01:26:06 +02:00
post_action_creator_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
post_creator_spec.rb DEV: Make rubocop happy. 2021-07-23 16:04:41 +08:00
post_destroyer_spec.rb DEV: Remove unncessary fabrication in tests. 2021-07-26 09:14:23 +08:00
post_locker_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
post_merger_spec.rb FIX: TL4 users cannot delete others posts (#13554) 2021-06-30 15:51:35 +03:00
post_revisor_spec.rb FIX: Rename ninja edit terminology to grace period (#13598) 2021-07-01 11:27:11 +10:00
pretty_text_spec.rb FIX: manually adds frowning_face_with_open_mouth for apple (#13528) 2021-07-21 23:27:20 +02:00
promotion_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
quote_comparer_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
rate_limiter_spec.rb No need to disable rate limiter after running tests (#13093) 2021-05-19 16:04:35 +04:00
redis_store_spec.rb DEV: Implement a faster Discourse.cache 2019-11-27 16:11:49 +11:00
retrieve_title_spec.rb FEATURE: add title tag for group detail page (#13702) 2021-07-12 20:05:57 +05:30
rtl_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
s3_helper_spec.rb DEV: Clean up S3 specs, stubs, and helpers 2020-09-28 12:02:25 +01:00
s3_inventory_multisite_spec.rb DEV: Isolate multisite specs (#13634) 2021-07-07 18:57:42 +02:00
s3_inventory_spec.rb DEV: Isolate multisite specs (#13634) 2021-07-07 18:57:42 +02:00
score_calculator_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
scss_checker_spec.rb PERF: Eager load Theme associations in Stylesheet Manager. 2021-06-21 11:06:58 +08:00
search_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
secure_session_spec.rb DEV: correct implementation of expiry api 2019-11-11 11:18:12 +11:00
site_icon_manager_spec.rb DEV: enable frozen string literal on all files 2019-05-13 09:31:32 +08:00
site_setting_extension_multisite_spec.rb DEV: Isolate multisite specs (#13634) 2021-07-07 18:57:42 +02:00
site_setting_extension_spec.rb DEV: Isolate multisite specs (#13634) 2021-07-07 18:57:42 +02:00
slug_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
spam_handler_spec.rb FIX: use allowlist and blocklist terminology (#10209) 2020-07-27 10:23:54 +10:00
suggested_topics_builder_spec.rb DEV: Default to skipping creating a topic when fabricating categories (#7976) 2019-08-06 11:26:54 +01:00
system_message_spec.rb FIX: TL2 promotion message and advance training (#10679) 2020-09-22 10:17:52 +10:00
text_cleaner_spec.rb FEATURE: Correctly convert topic title to uppercase and lowercase for Turkish default locale (#13115) 2021-05-24 18:13:30 +10:00
text_sentinel_spec.rb FIX: prevents exception when text input is nil (#12922) 2021-05-03 09:21:35 +02:00
theme_settings_manager_spec.rb DEV: support json_schema in theme settings (#12294) 2021-03-10 20:15:04 -05:00
theme_settings_parser_spec.rb DEV: Don't user before(:all)/after(:all) (#13389) 2021-06-15 17:25:06 +02:00
timeline_lookup_spec.rb DEV: followup to 8edd2b38cb to use existing spec (#11830) 2021-01-25 12:04:27 +01:00
topic_creator_spec.rb FIX: Enforce tag group count validation before sending to review queue (#12728) 2021-04-19 09:43:50 +10:00
topic_publisher_spec.rb DEV: Improve flaky time-sensitive specs (#9141) 2020-03-10 22:13:17 +01:00
topic_query_spec.rb SECURITY: Validate period param for top topic routes (#13818) 2021-07-22 16:31:53 +10:00
topic_retriever_spec.rb FEATURE: Stop checking referer for embeds (#13756) 2021-07-16 15:25:49 -03:00
topic_view_spec.rb FIX: Show correct dates in topic timelines (#13670) 2021-07-12 19:35:24 +03:00
topics_bulk_action_spec.rb FEATURE: Add last visit indication to topic view page. (#13471) 2021-07-05 14:17:31 +08:00
trashable_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
trust_level_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
unread_spec.rb FEATURE: Add last visit indication to topic view page. (#13471) 2021-07-05 14:17:31 +08:00
url_helper_spec.rb FIX: errors loading secure uploads when secure uploads is disabled (#13047) 2021-06-08 13:25:51 -04:00
user_lookup_spec.rb PERF: Combine avatar_lookup and primary_group_lookup into user_lookup (#10253) 2020-07-17 10:48:08 +01:00
user_name_suggester_spec.rb FIX: allow for final sigma in suggested usernames (#11540) 2020-12-23 08:51:36 +11:00
version_spec.rb DEV: Fix an apparently "too modern" git command (#10894) 2020-10-12 22:54:56 +02:00