JCLOUDS-1582: fixes a bug in the transient blobstore where after
uploading a multipart upload, GET/HEAD returns the hash of the content,
rather than the MPU ETag.
In the last commit (last section of squashed commit), the GSON library was integrated into the JClouds core module using maven-bundle plugins include resource instruction. Building OSGi instruction variables from the respective modules show a weakness when resources such as script builder shell scripts are required to be integrated into the bundle but not provide a dedicated variable declaration for the resource section.
The following commit demonstrates a change in strategy in declaration and integration of OSGi metadata.
- Replace old bundle-plugin with newest bnd-plugin (bundle-plugin uses bnd-plugin internally)
- Move OSGi metadata declarations from a maven variable passing strategy into dedicated bnd.bnd files
+ Cleaner pom files, no bundle packaging
+ Intellisense / Autocomplete support for .bnd files in terms of package exports etc.
For demonstration, the overall OSGi adjustments are limited to project, core, script builder, compute, blob store, and load balancer because most custom OSGi metadata is defined here.
Note: Other modules are currently disabled from build because some feedback is needed first.
Make GSON integration work.
To understand the changes, see the core modules' bnd file. GSON internal packages also define a version. Both already exported and new export declarations are fused. The global JClouds core module exports defined the entire set of GSON packages available.
Some minor modifications were made in the module project; replace maven jar plugin with a minified version of the declaration, outsourced in projects bnd file.
Previously this provider worked around a RestAnnotationProcessor quirk
by using multi-part uploads for InputStream payloads. Instead work
around the quirk another way which allows a single-part upload. This
allows inclusion of the Content-MD5 header during object creation.
Backfill tests with both ByteSource and InputStream inputs.
If there is no extended attribute support in the file system, the blobs
will not have their associated ETags available. In that case, the file
system blob store should rehash the content while producing the combined
blob and return the expected S3-style ETag.
Previously getBlobKeysInsideContainer returned all keys and filtered
in LocalBlobStore. Now getBlobKeysInsideContainer filters via prefix
which can dramatically decrease the number of keys returned,
especially for the filesystem provider. Further optimizations are
possible for delimiter.
Previously jclouds attempted to slice non-repeatable InputStream
Payloads in order to upload sequentially. This never worked due to
mutating the single stream via skip and close. Also backfill test
which spuriously succeeded.
Prior commit introduced a bug in the computation of the MPU ETag value,
where it was concatenating strings, rather than operating on the bytes
of the integer value.
S3 uses a different ETag for multipart uploads (MPUs) than regular
objects. The ETag consists of the md5 hash of the concatenated ETags of
individual parts followed by the number of parts (separated by "-").
The patch changes the LocalBlobStore's implementation of
CompleteMultipartUpload to set the S3-style ETag before calling
putBlob() and return that ETag to the caller.
To simplify testing, a new protected method with a default NOOP
implementation is added to the BaseBlobIntegrationTest. It allows
providers to further verify MPUs (i.e. ensuring the correct ETag has
been stored alongside the object).
Previously with GCS and its maximum 32 parts, the slicing algorithm
would sliced 3.2 GB blobs into (31) 32 MB parts and (1) 2.3 GB part.
With this algorithm it creates (31) 100 MB parts and (1) smaller part.
MultipartUploadSlicingAlgorithm creates multiple equal-sized parts and
a remaining amount. BaseBlobStore.putMultipartBlob used this
interface incorrectly, which could create more parts than intended
since the remaining size could be larger than the part size. This
manifested with Google Cloud Storage which only allows 32 parts.
HTTP headers are case insensitive by nature (see RFC 2616). When addUserMetadataTo check if this is indeed a user metadata header, it must properly ignore case.
The fix make sure that both metadataPrefix and the header key are compared with toLowerCase().
This solves issue with minio metadata read
Previously S3, Swift, and local blobstores threw a generic
IllegalArgumentException for this uncommon error. Instead
consistently throw HttpResponseException.
Atmos only supports listing by directories while other blobstores
allow listing via arbitrary prefixes. Allow requests which list
directories via both prefix and delimiter = "/" to succeed instead of
failing all requests. Also change a test which specified recursive to
instead be delimiter = "/". Fixesgaul/s3proxy#244.
