This takes the uppy chunking algorithm and combines it with some
form submission from resumable.js for parity with the current
backup controller to make local backup uploads work with uppy.
We can then use this to replace the resumable-upload component
and the resumable.js library from our codebase, once stable.
This is disabled by default, so people using local backups will not
be affected. The enable_experimental_backup_uploader site setting
must be enabled for this to work.
Uppy V2 includes the S3 multipart batch presigning change
we contributed in d613b849a6
so we need to upgrade it. This also brings both package.json
files into line and accounts for the renaming of Plugin
to BasePlugin in Uppy.
This has been tested and is working locally for both
regular Ember and Ember CLI, for uploads.json
XHR uploads and for direct S3 uploads (single and multipart).
This rolls uppy back to the previous bundle that was used,
which will break multipart functionality (which is not yet
enabled anywhere).
No other upload functionality should be affected by this change,
it will be as if d295a16dab had
not been merged.
This pull request introduces the endpoints required, and the JavaScript functionality in the `ComposerUppyUpload` mixin, for direct S3 multipart uploads. There are four new endpoints in the uploads controller:
* `create-multipart.json` - Creates the multipart upload in S3 along with an `ExternalUploadStub` record, storing information about the file in the same way as `generate-presigned-put.json` does for regular direct S3 uploads
* `batch-presign-multipart-parts.json` - Takes a list of part numbers and the unique identifier for an `ExternalUploadStub` record, and generates the presigned URLs for those parts if the multipart upload still exists and if the user has permission to access that upload
* `complete-multipart.json` - Completes the multipart upload in S3. Needs the full list of part numbers and their associated ETags which are returned when the part is uploaded to the presigned URL above. Only works if the user has permission to access the associated `ExternalUploadStub` record and the multipart upload still exists.
After we confirm the upload is complete in S3, we go through the regular `UploadCreator` flow, the same as `complete-external-upload.json`, and promote the temporary upload S3 into a full `Upload` record, moving it to its final destination.
* `abort-multipart.json` - Aborts the multipart upload on S3 and destroys the `ExternalUploadStub` record if the user has permission to access that upload.
Also added are a few new columns to `ExternalUploadStub`:
* multipart - Whether or not this is a multipart upload
* external_upload_identifier - The "upload ID" for an S3 multipart upload
* filesize - The size of the file when the `create-multipart.json` or `generate-presigned-put.json` is called. This is used for validation.
When the user completes a direct S3 upload, either regular or multipart, we take the `filesize` that was captured when the `ExternalUploadStub` was first created and compare it with the final `Content-Length` size of the file where it is stored in S3. Then, if the two do not match, we throw an error, delete the file on S3, and ban the user from uploading files for N (default 5) minutes. This would only happen if the user uploads a different file than what they first specified, or in the case of multipart uploads uploaded larger chunks than needed. This is done to prevent abuse of S3 storage by bad actors.
Also included in this PR is an update to vendor/uppy.js. This has been built locally from the latest uppy source at d613b849a6. This must be done so that I can get my multipart upload changes into Discourse. When the Uppy team cuts a proper release, we can bump the package.json versions instead.
This PR adds the first use of Uppy in our codebase, hidden behind a enable_experimental_image_uploader site setting. When the setting is enabled only the user card background uploader will use the new uppy-image-uploader component added in this PR.
I've introduced an UppyUpload mixin that has feature parity with the existing Upload mixin, and improves it slightly to deal with multiple/single file distinctions and validations better. For now, this just supports the XHRUpload plugin for uppy, which keeps our existing POST to /uploads.json.
This PR adds uppy to the project with a custom JS build and the shims needed to import it into our JS code. We need a custom build of Uppy because we do not use webpack for our JS modules/build. The only way to get what you want from Uppy is to use the webpack modules or to include the entire Uppy project including all plugins in a single JS file. This way we can just use the plugins we actually want. Future PRs will actually use Uppy!
The emoji-picker is a specific piece of code as it has very strong performance requirements which are almost not found anywhere else in the app, as a result it was using various hacks to make it work decently even on old browsers.
Following our drop of Internet Explorer, and various new features in Ember and recent browsers we can now take advantage of this to reduce the amount of code needed, this rewrite most importantly does the following:
- use loading="lazy" preventing the full list of emojis to be loaded on opening
- uses InterserctionObserver to find the active section
- limits the use of native event listentes only for hover/click emojis (for performance reason we track click on the whole emoji area and delegate events), everything else is using ember events
- uses popper to position the emoji picker
- no jquery code
Martin Brennan
Joffrey JAFFEUX
Martin Brennan
Robin Ward
Alan Guo Xiang Tan
Jarek Radosz
Blake Erickson