If there is no extended attribute support in the file system, the blobs
will not have their associated ETags available. In that case, the file
system blob store should rehash the content while producing the combined
blob and return the expected S3-style ETag.
Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore. Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider. Further optimizations are
possible for delimiter.
Previously jclouds attempted to slice non-repeatable InputStream
Payloads in order to upload sequentially. This never worked due to
mutating the single stream via skip and close. Also backfill test
which spuriously succeeded.
Prior commit introduced a bug in the computation of the MPU ETag value,
where it was concatenating strings, rather than operating on the bytes
of the integer value.
S3 uses a different ETag for multipart uploads (MPUs) than regular
objects. The ETag consists of the md5 hash of the concatenated ETags of
individual parts followed by the number of parts (separated by "-").
The patch changes the LocalBlobStore's implementation of
CompleteMultipartUpload to set the S3-style ETag before calling
putBlob() and return that ETag to the caller.
To simplify testing, a new protected method with a default NOOP
implementation is added to the BaseBlobIntegrationTest. It allows
providers to further verify MPUs (i.e. ensuring the correct ETag has
been stored alongside the object).
Previously with GCS and its maximum 32 parts, the slicing algorithm
would sliced 3.2 GB blobs into (31) 32 MB parts and (1) 2.3 GB part.
With this algorithm it creates (31) 100 MB parts and (1) smaller part.
MultipartUploadSlicingAlgorithm creates multiple equal-sized parts and
a remaining amount. BaseBlobStore.putMultipartBlob used this
interface incorrectly, which could create more parts than intended
since the remaining size could be larger than the part size. This
manifested with Google Cloud Storage which only allows 32 parts.
HTTP headers are case insensitive by nature (see RFC 2616). When addUserMetadataTo check if this is indeed a user metadata header, it must properly ignore case.
The fix make sure that both metadataPrefix and the header key are compared with toLowerCase().
This solves issue with minio metadata read
Previously S3, Swift, and local blobstores threw a generic
IllegalArgumentException for this uncommon error. Instead
consistently throw HttpResponseException.
Atmos only supports listing by directories while other blobstores
allow listing via arbitrary prefixes. Allow requests which list
directories via both prefix and delimiter = "/" to succeed instead of
failing all requests. Also change a test which specified recursive to
instead be delimiter = "/". Fixesgaul/s3proxy#244.
roded
Roded Bahat
Timur Alperovich
Andrew Gaul
Ignasi Barrera
Joe Meiring
Jesse Glick
Alin Dreghiciu
andreaturli
Yuval Kashtan
Timur Alperovich
Chaithanya Ganta
Timur Alperovich