discourse/lib/file_store/base_store.rb

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

225 lines
5.9 KiB
Ruby
Raw Normal View History

# frozen_string_literal: true
2013-11-05 13:04:47 -05:00
module FileStore
class BaseStore
UPLOAD_PATH_REGEX ||= %r{/(original/\d+X/.*)}
OPTIMIZED_IMAGE_PATH_REGEX ||= %r{/(optimized/\d+X/.*)}
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787) This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads` settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader. A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. ### Starting a direct S3 upload When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded. Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage. ### Completing a direct S3 upload Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`. 1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this. 2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues. We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large. 3. Finally we return the serialized upload record back to the client There are several errors that could happen that are handled by `UploadsController` as well. Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-27 18:42:25 -04:00
TEMPORARY_UPLOAD_PREFIX ||= "temp/"
2013-11-05 13:04:47 -05:00
2014-04-15 07:04:14 -04:00
def store_upload(file, upload, content_type = nil)
upload.url = nil
2015-05-29 12:39:47 -04:00
path = get_path_for_upload(upload)
store_file(file, path)
2013-11-05 13:04:47 -05:00
end
FEATURE: Secure media allowing duplicated uploads with category-level privacy and post-based access rules (#8664) ### General Changes and Duplication * We now consider a post `with_secure_media?` if it is in a read-restricted category. * When uploading we now set an upload's secure status straight away. * When uploading if `SiteSetting.secure_media` is enabled, we do not check to see if the upload already exists using the `sha1` digest of the upload. The `sha1` column of the upload is filled with a `SecureRandom.hex(20)` value which is the same length as `Upload::SHA1_LENGTH`. The `original_sha1` column is filled with the _real_ sha1 digest of the file. * Whether an upload `should_be_secure?` is now determined by whether the `access_control_post` is `with_secure_media?` (if there is no access control post then we leave the secure status as is). * When serializing the upload, we now cook the URL if the upload is secure. This is so it shows up correctly in the composer preview, because we set secure status on upload. ### Viewing Secure Media * The secure-media-upload URL will take the post that the upload is attached to into account via `Guardian.can_see?` for access permissions * If there is no `access_control_post` then we just deliver the media. This should be a rare occurrance and shouldn't cause issues as the `access_control_post` is set when `link_post_uploads` is called via `CookedPostProcessor` ### Removed We no longer do any of these because we do not reuse uploads by sha1 if secure media is enabled. * We no longer have a way to prevent cross-posting of a secure upload from a private context to a public context. * We no longer have to set `secure: false` for uploads when uploading for a theme component.
2020-01-15 22:50:27 -05:00
def store_optimized_image(file, optimized_image, content_type = nil, secure: false)
optimized_image.url = nil
2015-05-29 12:39:47 -04:00
path = get_path_for_optimized_image(optimized_image)
store_file(file, path)
end
def store_file(file, path, opts = {})
not_implemented
2013-11-05 13:04:47 -05:00
end
def remove_upload(upload)
remove_file(upload.url, get_path_for_upload(upload))
2013-11-05 13:04:47 -05:00
end
def remove_optimized_image(optimized_image)
remove_file(optimized_image.url, get_path_for_optimized_image(optimized_image))
2015-05-29 12:39:47 -04:00
end
def remove_file(url, path)
not_implemented
2013-11-05 13:04:47 -05:00
end
def upload_path
path = File.join("uploads", RailsMultisite::ConnectionManagement.current_db)
return path if !Rails.env.test?
File.join(path, "test_#{ENV["TEST_ENV_NUMBER"].presence || "0"}")
end
FEATURE: Direct S3 multipart uploads for backups (#14736) This PR introduces a new `enable_experimental_backup_uploads` site setting (default false and hidden), which when enabled alongside `enable_direct_s3_uploads` will allow for direct S3 multipart uploads of backup .tar.gz files. To make multipart external uploads work with both the S3BackupStore and the S3Store, I've had to move several methods out of S3Store and into S3Helper, including: * presigned_url * create_multipart * abort_multipart * complete_multipart * presign_multipart_part * list_multipart_parts Then, S3Store and S3BackupStore either delegate directly to S3Helper or have their own special methods to call S3Helper for these methods. FileStore.temporary_upload_path has also removed its dependence on upload_path, and can now be used interchangeably between the stores. A similar change was made in the frontend as well, moving the multipart related JS code out of ComposerUppyUpload and into a mixin of its own, so it can also be used by UppyUploadMixin. Some changes to ExternalUploadManager had to be made here as well. The backup direct uploads do not need an Upload record made for them in the database, so they can be moved to their final S3 resting place when completing the multipart upload. This changeset is not perfect; it introduces some special cases in UploadController to handle backups that was previously in BackupController, because UploadController is where the multipart routes are located. A subsequent pull request will pull these routes into a module or some other sharing pattern, along with hooks, so the backup controller and the upload controller (and any future controllers that may need them) can include these routes in a nicer way.
2021-11-10 17:25:31 -05:00
def self.temporary_upload_path(file_name, folder_prefix: "")
# We don't want to use the original file name as it can contain special
# characters, which can interfere with external providers operations and
# introduce other unexpected behaviour.
file_name_random = "#{SecureRandom.hex}#{File.extname(file_name)}"
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787) This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads` settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader. A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. ### Starting a direct S3 upload When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded. Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage. ### Completing a direct S3 upload Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`. 1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this. 2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues. We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large. 3. Finally we return the serialized upload record back to the client There are several errors that could happen that are handled by `UploadsController` as well. Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-27 18:42:25 -04:00
File.join(TEMPORARY_UPLOAD_PREFIX, folder_prefix, SecureRandom.hex, file_name_random)
end
2013-11-05 13:04:47 -05:00
def has_been_uploaded?(url)
not_implemented
2013-11-05 13:04:47 -05:00
end
2015-05-29 12:39:47 -04:00
def download_url(upload)
not_implemented
2013-11-05 13:04:47 -05:00
end
2015-05-29 12:39:47 -04:00
def cdn_url(url)
not_implemented
2013-11-05 13:04:47 -05:00
end
2015-05-29 12:39:47 -04:00
def absolute_base_url
not_implemented
2015-05-29 12:39:47 -04:00
end
def relative_base_url
not_implemented
end
def s3_upload_host
not_implemented
end
2013-11-05 13:04:47 -05:00
def external?
not_implemented
2013-11-05 13:04:47 -05:00
end
def internal?
2015-05-29 12:39:47 -04:00
!external?
2013-11-05 13:04:47 -05:00
end
def path_for(upload)
not_implemented
2013-11-05 13:04:47 -05:00
end
def list_missing_uploads(skip_optimized: false)
not_implemented
end
def download(object, max_file_size_kb: nil)
DistributedMutex.synchronize("download_#{object.sha1}", validity: 3.minutes) do
extension =
File.extname(
object.respond_to?(:original_filename) ? object.original_filename : object.url,
)
filename = "#{object.sha1}#{extension}"
file = get_from_cache(filename)
if !file
max_file_size_kb ||= [
SiteSetting.max_image_size_kb,
SiteSetting.max_attachment_size_kb,
].max.kilobytes
secure = object.respond_to?(:secure) ? object.secure? : object.upload.secure?
url =
(
if secure
Discourse.store.signed_url_for_path(object.url)
else
Discourse.store.cdn_url(object.url)
end
)
url = SiteSetting.scheme + ":" + url if url =~ %r{\A//}
file =
FileHelper.download(
url,
max_file_size: max_file_size_kb,
tmp_file_name: "discourse-download",
follow_redirect: true,
)
return nil if file.nil?
cache_file(file, filename)
file = get_from_cache(filename)
end
file
end
2013-11-05 13:04:47 -05:00
end
2013-11-27 16:01:41 -05:00
def purge_tombstone(grace_period)
end
def get_path_for(type, id, sha, extension)
depth = get_depth_for(id)
2018-12-26 11:34:49 -05:00
tree = File.join(*sha[0, depth].chars, "")
"#{type}/#{depth + 1}X/#{tree}#{sha}#{extension}"
end
def get_path_for_upload(upload)
# try to extract the path from the URL instead of calculating it,
# because the calculated path might differ from the actual path
if upload.url.present? && (path = upload.url[UPLOAD_PATH_REGEX, 1])
return prefix_path(path)
end
extension =
if upload.extension
".#{upload.extension}"
else
2018-12-26 11:34:49 -05:00
# Maintain backward compatibility before Jobs::MigrateUploadExtensions runs
File.extname(upload.original_filename)
end
get_path_for("original", upload.id, upload.sha1, extension)
end
def get_path_for_optimized_image(optimized_image)
# try to extract the path from the URL instead of calculating it,
# because the calculated path might differ from the actual path
if optimized_image.url.present? && (path = optimized_image.url[OPTIMIZED_IMAGE_PATH_REGEX, 1])
return prefix_path(path)
end
upload = optimized_image.upload
version = optimized_image.version || 1
extension =
"_#{version}_#{optimized_image.width}x#{optimized_image.height}#{optimized_image.extension}"
get_path_for("optimized", upload.id, upload.sha1, extension)
end
CACHE_DIR ||= "#{Rails.root}/tmp/download_cache/"
CACHE_MAXIMUM_SIZE ||= 500
def get_cache_path_for(filename)
"#{CACHE_DIR}#{filename}"
end
def get_from_cache(filename)
path = get_cache_path_for(filename)
File.open(path) if File.exist?(path)
end
def cache_file(file, filename)
path = get_cache_path_for(filename)
dir = File.dirname(path)
FileUtils.mkdir_p(dir) unless Dir.exist?(dir)
FileUtils.cp(file.path, path)
# Remove all but CACHE_MAXIMUM_SIZE most recent files
files = Dir.glob("#{CACHE_DIR}*")
2020-06-01 00:07:07 -04:00
files.sort_by! do |f|
begin
2020-06-01 00:07:07 -04:00
File.mtime(f)
rescue Errno::ENOENT
Time.new(0)
end
end
files.pop(CACHE_MAXIMUM_SIZE)
FileUtils.rm(files, force: true)
end
private
def not_implemented
raise "Not implemented."
end
def get_depth_for(id)
depths = [0]
depths << Math.log(id / 1_000.0, 16).ceil if id.positive?
depths.max
end
def prefix_path(path)
path
end
2013-11-05 13:04:47 -05:00
end
end