discourse/spec/jobs
Martin Brennan b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
..
regular FIX: Change email from to not have via site_name for group SMTP (#13788) 2021-07-20 11:56:04 +10:00
about_stats_spec.rb DEV: s/\$redis/Discourse\.redis (#8431) 2019-12-03 10:05:53 +01:00
activation_reminder_emails_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
auto_expire_user_api_keys_spec.rb DEV: Improve flaky time-sensitive specs (#9141) 2020-03-10 22:13:17 +01:00
auto_queue_handler_spec.rb DEV: Remove stray code that has been commented out. 2020-07-29 09:58:29 +08:00
automatic_group_membership_spec.rb DEV: Ensure DiscourseEvent handlers cleaned up during specs (#11205) 2020-11-11 19:46:13 +00:00
bookmark_reminder_notifications_spec.rb FIX: When user has already hit bookmark limit, do not error for clear_reminder! or other updates (#12658) 2021-04-09 13:06:35 +10:00
bulk_grant_trust_level_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
bulk_invite_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
bump_topic_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
clean_dismissed_topic_users_spec.rb FIX: Dismiss new with better migration (#12062) 2021-02-15 08:50:33 +11:00
clean_up_associated_accounts_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
clean_up_crawler_stats_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
clean_up_email_change_requests_spec.rb DEV: Periodically delete old email change requests (#13054) 2021-05-14 10:34:56 +03:00
clean_up_email_logs_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
clean_up_inactive_users_spec.rb FIX: Don’t try to delete inactive moderators 2019-05-13 21:44:03 +02:00
clean_up_post_reply_keys_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
clean_up_unused_staged_users_spec.rb FEATURE: Add site setting to disable staged user cleanup 2020-03-09 10:26:41 +01:00
clean_up_uploads_spec.rb FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787) 2021-07-28 08:42:25 +10:00
clean_up_user_export_topics_spec.rb FIX: delete system generated message when user_export record is deleted (#7595) 2019-05-28 16:38:41 +05:30
cleanup_imap_sync_log_spec.rb DEV: IMAP debugging improvements (#11784) 2021-01-21 11:37:47 +10:00
close_topic_spec.rb FEATURE: Topic timer UI revamp (#11912) 2021-02-03 10:13:32 +10:00
correct_missing_dualstack_urls_spec.rb DEV: Clean up S3 specs, stubs, and helpers 2020-09-28 12:02:25 +01:00
crawl_topic_link_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
create_linked_topic_spec.rb FIX: when creating linked topics make sure they belong to same category (#11188) 2020-11-11 00:44:27 +05:30
create_missing_avatars_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
create_recent_post_search_indexes_spec.rb PERF: Create a partial regular post_search_data index on large sites. 2020-08-27 13:42:00 +08:00
create_user_reviewable_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
delete_replies_spec.rb FEATURE: Allow durations < 1 hour and < 1 day for topic timers where duration is specified (auto delete replies, close based on last post) (#11961) 2021-02-05 10:12:56 +10:00
delete_topic_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
disable_bootstrap_mode_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
download_avatar_from_url_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
download_backup_email_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
download_profile_background_from_url_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
emit_web_hook_event_spec.rb FEATURE: add support for like webhooks (#12917) 2021-04-30 17:08:38 -07:00
enable_bootstrap_mode_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
enqueue_digest_emails_spec.rb PERF: Do not enqueue digest emails when attempted recently (#10849) 2020-10-07 15:30:38 +01:00
enqueue_suspect_users_spec.rb FIX: Reduce the time_read threshold to one minute. (#12159) 2021-02-20 08:25:32 -03:00
ensure_s3_uploads_existence_spec.rb DEV: Clean up S3 specs, stubs, and helpers 2020-09-28 12:02:25 +01:00
export_csv_file_spec.rb FEATURE: Rename 'Discourse SSO' to DiscourseConnect (#11978) 2021-02-08 10:04:33 +00:00
export_user_archive_spec.rb DEV: ensure user export ordering is predictable (#13340) 2021-06-09 15:55:52 +10:00
feature_topic_users_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
fix_out_of_sync_user_uploaded_avatar_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
fix_primary_emails_for_staged_users_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
fix_s3_etags_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
fix_user_usernames_and_groups_names_clash_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
grant_anniversary_badges_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
grant_new_user_of_the_month_badges_spec.rb FIX: Don't award new user of the month in the wrong month 2020-07-13 15:05:30 -04:00
heartbeat_spec.rb DEV: Replace `Time.new` with `Time.now` (#9142) 2020-03-09 17:37:49 +01:00
ignored_users_summary_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
invalidate_inactive_admins_spec.rb DEV: Migrate Github authentication to ManagedAuthenticator (#11170) 2020-11-10 10:09:15 +00:00
invite_email_spec.rb FIX: Correctly use invite to topic email templates (#12411) 2021-03-16 17:08:54 +02:00
jobs_base_spec.rb DEV: Run jobs sequentially in test mode (#9897) 2020-05-28 12:52:27 +01:00
jobs_spec.rb DEV: Correct typos and spelling mistakes (#12812) 2021-05-21 11:43:47 +10:00
mass_award_badge_spec.rb FEATURE: Add option to grant badge multiple times to users using Bulk Award (#13571) 2021-07-15 05:53:26 +03:00
migrate_badge_image_to_uploads_spec.rb UX: Add image uploader widget for uploading badge images (#12377) 2021-03-17 08:55:23 +03:00
migrate_group_flair_images_spec.rb FIX: Group#flair_url must be a real URL (#11400) 2020-12-07 13:35:41 +02:00
notify_mailing_list_subscribers_spec.rb FIX: no email when the category is muted and mailing list is on (#11490) 2020-12-16 09:30:21 +11:00
notify_moved_posts_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
notify_reviewable_spec.rb FIX: sync reviewable count when opening the hamburger menu (#10368) 2020-08-07 18:13:02 +02:00
notify_tag_change_spec.rb FIX: do not notify when the hidden tag is added or removed (#12025) 2021-02-11 10:03:45 +11:00
old_keys_reminder_spec.rb DEV: remove instagram login site settings and auth classes. (#11073) 2020-10-30 09:09:56 +05:30
open_topic_spec.rb FEATURE: Topic timer UI revamp (#11912) 2021-02-03 10:13:32 +10:00
pending_queued_posts_reminder_spec.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
pending_reviewables_reminder_spec.rb FEATURE: Hide Reviewable scores, change score filter to Priority 2019-05-07 14:05:23 -04:00
pending_users_reminder_spec.rb FEATURE: Staff can receive pending user reminders more frequently. (#13422) 2021-06-24 10:02:56 -03:00
periodical_updates_spec.rb DEV: Improve flaky time-sensitive specs (#9141) 2020-03-10 22:13:17 +01:00
poll_mailbox_spec.rb FIX: POP3 polling shouldn't stop after exception or old email (#12742) 2021-04-19 10:27:29 +02:00
post_update_topic_tracking_state_spec.rb FIX: Check if post.topic exists before publishing topic updates (#11900) 2021-02-09 16:41:22 +11:00
post_uploads_recovery_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
process_bulk_invite_emails_spec.rb FEATURE: send max 200 emails every minute for bulk invites (#7875) 2019-07-19 11:29:12 +05:30
process_email_spec.rb DEV: Add created_via column to IncomingEmail (#11751) 2021-01-20 13:22:41 +10:00
process_post_spec.rb DEV: Move autotag to topic creator (#12790) 2021-04-23 16:55:34 +03:00
process_shelved_notifications_spec.rb DEV: Replace 'processed' column on notifications with new table (#11864) 2021-01-27 10:29:24 -06:00
publish_topic_to_category_spec.rb FIX: Auto close topic from category settings based on topic created_at (#12082) 2021-02-17 07:51:39 +10:00
pull_hotlinked_images_spec.rb FEATURE: Add a site setting to allow emojis to come from an external URL (#12180) 2021-03-02 16:04:16 -03:00
purge_expired_ignored_users_spec.rb DEV: Cleanup ignored user logic (#11107) 2020-11-03 12:38:54 +00:00
rebake_custom_emoji_posts_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
reindex_search_spec.rb FIX: reindex_search job should work on model with no search data (#11819) 2021-01-25 11:23:36 +01:00
remove_banner_spec.rb DEV: adds support for bannered until (#13417) 2021-06-24 11:35:36 +02:00
reviewable_priorities_spec.rb FIX: Use a better default for the low_priority_threshold setting. (#13161) 2021-05-26 13:16:16 -03:00
send_system_message_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
suspicious_login_spec.rb DEV: Prefabrication (test optimization) (#7414) 2019-05-07 13:12:20 +10:00
sync_topic_user_bookmarked_spec.rb FIX: Make sure topic_user.bookmarked is synced in more places (#13383) 2021-06-16 08:30:40 +10:00
tl3_promotions_spec.rb PERF: Dematerialize topic_reply_count (#9769) 2020-05-14 15:42:00 -07:00
toggle_topic_closed_spec.rb FIX: Clean up toggle closed topic timer when user is not authorized. 2020-08-26 12:59:05 +08:00
topic_timer_enqueuer_spec.rb FIX: Do not prevent other topic timers running on error (#13665) 2021-07-08 12:49:58 +10:00
truncate_user_flag_stats_spec.rb DEV: Improve flaky time-sensitive specs (#9141) 2020-03-10 22:13:17 +01:00
unsilence_users_spec.rb DEV: use #frozen_string_literal: true on all spec 2019-04-30 10:27:42 +10:00
update_animated_uploads_spec.rb FIX: Ensure old uploads can have animated field updated (#10963) 2020-10-20 19:11:43 +03:00
update_gravatar_spec.rb FIX: Gravatar download attempt if user is missing their email 2020-09-02 20:19:46 -06:00
update_s3_inventory_spec.rb DEV: Clean up S3 specs, stubs, and helpers 2020-09-28 12:02:25 +01:00
update_username_spec.rb FIX: Don't raise error in update username job if user has been deleted. 2020-09-02 11:17:17 +08:00
user_email_spec.rb DEV: Commit failing test for email substution bug 2021-07-08 15:56:09 -07:00
vacate_legacy_prefix_backups_spec.rb DEV: Don't user before(:all)/after(:all) (#13389) 2021-06-15 17:25:06 +02:00