discourse/spec/components
Martin Brennan e4350bb966
FEATURE: Direct S3 multipart uploads for backups (#14736)
This PR introduces a new `enable_experimental_backup_uploads` site setting (default false and hidden), which when enabled alongside `enable_direct_s3_uploads` will allow for direct S3 multipart uploads of backup .tar.gz files.

To make multipart external uploads work with both the S3BackupStore and the S3Store, I've had to move several methods out of S3Store and into S3Helper, including:

* presigned_url
* create_multipart
* abort_multipart
* complete_multipart
* presign_multipart_part
* list_multipart_parts

Then, S3Store and S3BackupStore either delegate directly to S3Helper or have their own special methods to call S3Helper for these methods. FileStore.temporary_upload_path has also removed its dependence on upload_path, and can now be used interchangeably between the stores. A similar change was made in the frontend as well, moving the multipart related JS code out of ComposerUppyUpload and into a mixin of its own, so it can also be used by UppyUploadMixin.

Some changes to ExternalUploadManager had to be made here as well. The backup direct uploads do not need an Upload record made for them in the database, so they can be moved to their final S3 resting place when completing the multipart upload.

This changeset is not perfect; it introduces some special cases in UploadController to handle backups that was previously in BackupController, because UploadController is where the multipart routes are located. A subsequent pull request will pull these routes into a module or some other sharing pattern, along with hooks, so the backup controller and the upload controller (and any future controllers that may need them) can include these routes in a nicer way.
2021-11-11 08:25:31 +10:00
..
auth FIX: Don't leak unhashed user API keys to redis (#14682) 2021-10-21 19:43:26 +03:00
common_passwords
concern FIX: Nil-filled CF arrays were not being deleted (#13518) 2021-06-25 11:34:51 +02:00
email FIX: Remove List-Post email header (#14554) 2021-10-11 20:57:42 +03:00
file_store FEATURE: Direct S3 multipart uploads for backups (#14736) 2021-11-11 08:25:31 +10:00
freedom_patches FIX: Ensure id sequences are not reset during db:migrate (#14184) 2021-08-30 12:31:22 +01:00
guardian FEATURE: Allow admins to permanently delete posts and topics (#14406) 2021-10-13 12:53:23 +03:00
highlight_js
imap DEV: Move imap_helper to spec/support directory (#14776) 2021-10-29 20:46:25 +02:00
import
middleware FIX: Strip `discourse-logged-in` header during `force_anonymous!` (#14533) 2021-10-07 12:31:42 +01:00
migration
plugin FEATURE: Add new plugin API to allow plugins to extend `Site#categories` (#13773) 2021-07-19 13:54:19 +08:00
pretty_text
rate_limiter
scheduler
site_settings DEV: Remove HTML setting type and sanitization logic. (#14440) 2021-10-04 15:40:35 -03:00
stylesheet FIX: Order outputted theme stylesheets (#14133) 2021-08-25 09:37:07 +08:00
svg_sprite DEV: prevents flakky spec when deleting plugin (#14701) 2021-10-25 10:24:21 +02:00
theme_store
validators FEATURE: Humanize file size error messages (#14398) 2021-09-22 07:59:45 +10:00
wizard FEATURE: Enable auto dark mode on new instances (#14208) 2021-09-02 14:55:38 -04:00
admin_confirmation_spec.rb
admin_user_index_query_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
archetype_spec.rb
cache_spec.rb FIX: ensures defined expired_in is passed from write to write_entry (#11622) 2021-01-04 10:34:44 +01:00
category_badge_spec.rb
composer_messages_finder_spec.rb FEATURE: Make allow_uploaded_avatars accept TL (#14091) 2021-08-24 10:46:28 +03:00
content_buffer_spec.rb
cooked_post_processor_spec.rb FIX: remove 'crawl_images' site setting (#14646) 2021-10-19 17:12:29 +05:30
crawler_detection_spec.rb FEATURE: Implement browser update in crawler view (#12448) 2021-03-22 19:41:42 +02:00
current_user_spec.rb
directory_helper_spec.rb
discourse_diff_spec.rb Escape values of HTML attributes 2021-08-10 10:25:15 -04:00
discourse_event_spec.rb DEV: Plugin API to add directory columns (#13440) 2021-06-22 13:00:04 -05:00
discourse_hub_spec.rb
discourse_plugin_registry_spec.rb DEV: Remove deprecated plugins variables importer (#12168) 2021-02-23 16:20:59 -05:00
discourse_redis_spec.rb DEV: Pass kwargs to the redis gem when calling methods/commands that we don't wrap (#14530) 2021-10-06 17:42:04 +03:00
discourse_spec.rb DEV: prevents flakky spec when deleting plugin (#14701) 2021-10-25 10:24:21 +02:00
discourse_tagging_spec.rb FIX: Show required tags to staff by default and override limit (#13242) 2021-06-02 12:43:34 -04:00
discourse_updates_spec.rb FIX: Regression introduced in #14715 (#14842) 2021-11-09 17:20:09 +11:00
distributed_memoizer_spec.rb
distributed_mutex_spec.rb
email_cook_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
email_updater_spec.rb FEATURE: add maximum limit for secondary emails (#12599) 2021-04-05 20:31:42 +05:30
enum_spec.rb
excerpt_parser_spec.rb FIX: Make Oneboxer#apply insert block Oneboxes correctly (#11449) 2020-12-14 17:49:37 +02:00
feed_element_installer_spec.rb
feed_item_accessor_spec.rb
file_helper_spec.rb
filter_best_posts_spec.rb
final_destination_spec.rb FIX: Canonical URLs may be relative (#14825) 2021-11-05 14:20:14 -03:00
flag_settings_spec.rb
gaps_spec.rb
global_path_spec.rb
guardian_spec.rb FIX: allowed_theme_ids should not be persisted in GlobalSettings (#14756) 2021-10-29 11:46:52 -04:00
has_errors_spec.rb
hijack_spec.rb FEATURE: Cache CORS preflight requests for 2h (#14614) 2021-10-14 22:37:53 -03:00
html_prettify_spec.rb
html_to_markdown_spec.rb FIX: Hoisting linebreaks shouldn't fail for HTML5 elements (#14364) 2021-09-17 10:41:34 +02:00
image_sizer_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
inline_oneboxer_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
js_locale_helper_spec.rb FEATURE: Add English (UK) as locale (#11768) 2021-01-20 21:32:22 +01:00
json_error_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
letter_avatar_spec.rb
method_profiler_spec.rb
new_post_manager_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
new_post_result_spec.rb
oneboxer_spec.rb FIX: Display Instagram Oneboxes in an iframe (#14789) 2021-11-02 14:34:51 -04:00
onpdiff_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
pbkdf2_spec.rb DEV: Load rails_helper in pbkdf2_spec (#14775) 2021-10-29 20:15:10 +02:00
pinned_check_spec.rb
plain_text_to_markdown_spec.rb
post_action_creator_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
post_creator_spec.rb DEV: Ignore bookmarks.topic_id column and remove references to it in code (#14289) 2021-09-15 10:16:54 +10:00
post_destroyer_spec.rb FIX: Skip post validations for system revisions when author deletes post (#14824) 2021-11-08 09:33:41 +08:00
post_locker_spec.rb
post_merger_spec.rb FIX: TL4 users cannot delete others posts (#13554) 2021-06-30 15:51:35 +03:00
post_revisor_spec.rb FIX: Make PostRevisor more consistent (#14841) 2021-11-09 16:29:37 +02:00
presence_channel_spec.rb DEV: Introduce PresenceChannel API for core and plugin use 2021-08-27 16:26:06 +01:00
pretty_text_spec.rb FIX: Do not replace words in hashtags and mentions (#14760) 2021-10-29 17:53:09 +03:00
promotion_spec.rb FIX: check if BasicBadge is enabled for TL1 welcome message (#13983) 2021-08-11 08:39:25 +10:00
quote_comparer_spec.rb
rate_limiter_spec.rb No need to disable rate limiter after running tests (#13093) 2021-05-19 16:04:35 +04:00
redis_store_spec.rb
retrieve_title_spec.rb FIX: increase chunk size to fetch title tag correctly (#14144) 2021-09-03 13:15:58 +05:30
rtl_spec.rb
s3_helper_spec.rb FIX: Ensure CORS rules exist for S3 using rake task (#14802) 2021-11-08 09:16:38 +10:00
s3_inventory_multisite_spec.rb DEV: Isolate multisite specs (#13634) 2021-07-07 18:57:42 +02:00
s3_inventory_spec.rb DEV: Isolate multisite specs (#13634) 2021-07-07 18:57:42 +02:00
score_calculator_spec.rb
scss_checker_spec.rb PERF: Eager load Theme associations in Stylesheet Manager. 2021-06-21 11:06:58 +08:00
search_spec.rb FIX: Use the same mode for chinese search when indexing and querying. (#14780) 2021-11-01 10:14:47 +08:00
secure_session_spec.rb
site_icon_manager_spec.rb
site_setting_extension_multisite_spec.rb DEV: Isolate multisite specs (#13634) 2021-07-07 18:57:42 +02:00
site_setting_extension_spec.rb DEV: Remove HTML setting type and sanitization logic. (#14440) 2021-10-04 15:40:35 -03:00
slug_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
spam_handler_spec.rb
suggested_topics_builder_spec.rb
system_message_spec.rb FIX: TL2 promotion message and advance training (#10679) 2020-09-22 10:17:52 +10:00
text_cleaner_spec.rb FEATURE: Correctly convert topic title to uppercase and lowercase for Turkish default locale (#13115) 2021-05-24 18:13:30 +10:00
text_sentinel_spec.rb FIX: prevents exception when text input is nil (#12922) 2021-05-03 09:21:35 +02:00
theme_settings_manager_spec.rb DEV: use upload id to save in theme setting instead of URL. (#14341) 2021-09-16 07:58:53 +05:30
theme_settings_parser_spec.rb DEV: Don't user before(:all)/after(:all) (#13389) 2021-06-15 17:25:06 +02:00
timeline_lookup_spec.rb DEV: followup to 8edd2b38cb to use existing spec (#11830) 2021-01-25 12:04:27 +01:00
topic_creator_spec.rb FIX: include new tags in validation if user can create one. (#14744) 2021-10-28 11:59:46 +05:30
topic_publisher_spec.rb
topic_query_spec.rb FIX: use category's default sort order in latest & unseen filters only. (#14571) 2021-10-12 10:25:03 +05:30
topic_retriever_spec.rb FEATURE: Stop checking referer for embeds (#13756) 2021-07-16 15:25:49 -03:00
topic_view_spec.rb DEV: Remove `TopicView#first_post_id`. (#14631) 2021-10-18 14:47:47 +08:00
topics_bulk_action_spec.rb FEATURE: Dismiss new and unread for PM inboxes. 2021-08-05 12:56:15 +08:00
trashable_spec.rb
trust_level_spec.rb
unread_spec.rb FEATURE: Add last visit indication to topic view page. (#13471) 2021-07-05 14:17:31 +08:00
url_helper_spec.rb FIX: errors loading secure uploads when secure uploads is disabled (#13047) 2021-06-08 13:25:51 -04:00
user_lookup_spec.rb REVERT "FIX: do not show private group flair on user avatars" (#13991) 2021-08-10 17:25:11 +05:30
user_name_suggester_spec.rb DEV: simplify username suggester (#14531) 2021-10-27 14:41:24 +04:00
version_spec.rb DEV: Fix an apparently "too modern" git command (#10894) 2020-10-12 22:54:56 +02:00