discourse/app/jobs/scheduled
Martin Brennan b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
..
about_stats.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
activation_reminder_emails.rb FIX: Zeitwerk-related fixes for jobs. (#8219) 2019-10-21 20:25:35 +03:00
auto_expire_user_api_keys.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
auto_queue_handler.rb FEATURE: Blocking is optional when deleting a user from the review queue. (#13375) 2021-06-15 12:35:45 -03:00
badge_grant.rb FIX: Cached badge_count isn't updated after backfilling badges (#11281) 2020-11-18 22:01:56 +01:00
bookmark_reminder_notifications.rb DEV: cap notifications per run at 300 as stated in comment (#11252) 2020-11-17 09:08:12 +10:00
category_stats.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
check_new_features.rb DEV: Prevents rate limits for new feature checks on multisite (#12053) 2021-02-12 08:52:59 -05:00
check_out_of_date_themes.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
clean_dismissed_topic_users.rb FIX: Dismiss new with better migration (#12062) 2021-02-15 08:50:33 +11:00
clean_up_associated_accounts.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
clean_up_crawler_stats.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
clean_up_email_change_requests.rb DEV: Periodically delete old email change requests (#13054) 2021-05-14 10:34:56 +03:00
clean_up_email_logs.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
clean_up_exports.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
clean_up_inactive_users.rb FIX: Don't error CleanUpInactiveUserJob when user is missing (#8362) 2019-11-18 16:14:15 +11:00
clean_up_post_reply_keys.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
clean_up_search_logs.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
clean_up_stylesheet_cache.rb PERF: Add scheduled job to delete old stylesheet cache rows (#13747) 2021-07-16 10:58:01 -04:00
clean_up_unmatched_emails.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
clean_up_unmatched_ips.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
clean_up_unsubscribe_keys.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
clean_up_unused_api_keys.rb FEATURE: Overhaul of admin API key system (#8284) 2019-11-05 14:10:23 +00:00
clean_up_unused_staged_users.rb FEATURE: Add site setting to disable staged user cleanup 2020-03-09 10:26:41 +01:00
clean_up_uploads.rb FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787) 2021-07-28 08:42:25 +10:00
cleanup_imap_sync_log.rb DEV: IMAP log to database (#10435) 2020-08-14 12:01:31 +10:00
create_missing_avatars.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
create_recent_post_search_indexes.rb Fix the build. 2020-08-27 13:50:07 +08:00
dashboard_stats.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
destroy_old_deletion_stubs.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
destroy_old_hidden_posts.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
directory_refresh_daily.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
directory_refresh_older.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
disable_bootstrap_mode.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
drop_backup_schema.rb FEATURE: Drop "backup" schema 7 days after restore 2020-01-16 17:48:47 +01:00
enqueue_digest_emails.rb PERF: Do not enqueue digest emails when attempted recently (#10849) 2020-10-07 15:30:38 +01:00
enqueue_onceoffs.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
enqueue_suspect_users.rb FIX: Reduce the time_read threshold to one minute. (#12159) 2021-02-20 08:25:32 -03:00
ensure_db_consistency.rb DEV: Fix typo 2020-10-07 23:43:11 +02:00
ensure_s3_uploads_existence.rb FIX: Prevent "uploads are missing in S3" alerts after restoring a backup 2020-09-10 21:37:48 +02:00
fix_user_usernames_and_groups_names_clash.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
grant_anniversary_badges.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
grant_new_user_of_the_month_badges.rb DEV: Require scheduled job in development to avoid loading file twice. 2020-09-01 10:14:40 +08:00
heartbeat.rb FIX: Zeitwerk-related fixes for jobs. (#8187) 2019-10-14 13:03:22 +03:00
ignored_users_summary.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
invalidate_inactive_admins.rb FEATURE: Add message to log when admins are automatically deactivated 2020-01-28 12:16:24 +00:00
migrate_upload_scheme.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
old_keys_reminder.rb DEV: Require scheduled job in development to avoid loading file twice. 2020-09-01 10:14:40 +08:00
pending_queued_posts_reminder.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
pending_reviewables_reminder.rb DEV: stop freezing frozen strings 2020-04-30 16:48:53 +10:00
pending_users_reminder.rb FEATURE: Staff can receive pending user reminders more frequently. (#13422) 2021-06-24 10:02:56 -03:00
periodical_updates.rb DEV: Change Topic Timer from enqueue_at scheduled jobs to incrementally executed jobs (#11698) 2021-01-19 13:30:58 +10:00
poll_mailbox.rb FIX: POP3 polling shouldn't stop after exception or old email (#12742) 2021-04-19 10:27:29 +02:00
process_badge_backlog.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
process_shelved_notifications.rb DEV: Replace 'processed' column on notifications with new table (#11864) 2021-01-27 10:29:24 -06:00
process_user_notification_schedules.rb DEV: Improve logging of errors in `Jobs::ProcessUserNotificationSchedules` 2021-07-21 12:20:44 +08:00
purge_deleted_uploads.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
purge_expired_ignored_users.rb DEV: Cleanup ignored user logic (#11107) 2020-11-03 12:38:54 +00:00
purge_old_web_hook_events.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
purge_unactivated.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
reindex_search.rb FIX: reindex_search job should work on model with no search data (#11819) 2021-01-25 11:23:36 +01:00
reviewable_priorities.rb FIX: Recalculate scores only when approving or transitioning to pending. (#13009) 2021-05-10 14:09:04 -03:00
schedule_backup.rb DEV: Add option to send system message to groups (#12256) 2021-03-02 18:51:50 +01:00
tl3_promotions.rb PERF: Dematerialize topic_reply_count (#9769) 2020-05-14 15:42:00 -07:00
top_refresh_older.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
top_refresh_today.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
topic_timer_enqueuer.rb FIX: Do not prevent other topic timers running on error (#13665) 2021-07-08 12:49:58 +10:00
unsilence_users.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
update_animated_uploads.rb FIX: Ensure old uploads can have animated field updated (#10963) 2020-10-20 19:11:43 +03:00
update_heat_settings.rb DEV: Upgrading Discourse to Zeitwerk (#8098) 2019-10-02 14:01:53 +10:00
version_check.rb DEV: Remove new_version_emails global setting reference 2019-11-20 15:54:09 +00:00
weekly.rb FEATURE: Add weekly bookmark cleanup code (#10899) 2020-10-14 09:38:57 +10:00