AWS S3 MPU ETag are hashes of the part ETag headers but some
implementations, specifically S3Proxy with the filesystem provider,
represent multi-part objects as a single object. Remove these checks
since they add nothing.
These are simpler than the full XML API and better supported by
non-AWS S3 implementations, e.g., Ceph, S3Proxy. Further this makes
the provider more consistent when creating a bucket or object which
only supports setting canned ACLs.
Swift implements multi-part upload with user-visible parts and an
explicit manifest. When deleting an MPU blob it can delete only the
manifest or both the manifest and parts. For consistency with other
providers, we now do the latter in the portable abstraction. Swift
ignores the multipart-manifest=delete parameter for single-part
objects. Fixesandrewgaul/s3proxy#92.
Previously we only used the blob name and not the blob
name/slo/timestamp/part size cookie which yield extra parts when
listing an MPU with parts from a previous MPU. Listing using the
stricter prefix gives the expected results.
Fixesandrewgaul/s3proxy#91.
On Windows, we need to avoid trailing spaces, as the test fails to
create the required blob. Specifically, we should not test blobs named
" " and "%20 ".
When constructing the expected HTTP requests in STS tests, STS API
should not pre-encode the strings. The form parameters are already
encoded and are otherwise encoded twice.
When constructing the query path, S3 does not properly handle encoded
paths. For example, if a blob named %20 is to be placed into the blob
store, S3 would end up placing blob named " " (what %20 represents).
This occurs because the S3 provider examines the URI's path portion
(which is presented in a decoded fasion to the caller). After
examining the path, it is not encoded again. Instead, we should call
getRawPath() to avoid this issue.
There are two issues on the decoding path:
1. Given a blob named " ", S3 API will throw a RuntimeException due to
a NULL check -- the key that it uses is NULL to represent the XML
content " " corresponding to the blob name.
2. Given a blob named "%20 ", S3 API will generate a URI for a blob
named "%20%20", which is also incorrect. The correct URI would be
"%2520%20" (escaping the first "%" and " " characters).
The first issue is due to the currentOrNull() helper, which calls
trim() on the string and then compares the string to an empty string.
This means that a blob named " " will be parsed as "" and then
converted to NULL as the result of that method. Passing "null" as the
key then fails in a number of places (notably, appendPath()).
The second issue is due to the appendPath() method in the jclouds Uris
class. The issue here is that appendPath() calls urlDecode() and
passes the result to path(). The path() method, in turn, also calls
urlDecode(). After these transformations, a properly encoded blob of
the form %2520%20 turns into "%20 " and then " " (two spaces). After
these transformations the path is encoded again, resulting in "%20%20"
(which is wrong).
Andrew Gaul
Zhao Jin
Rita Zhang
Zack Shoylev
Ignasi Barrera
Graeme-Miller
Josef Cacek
Ka-Hing Cheung
Stuart Hendren
John McDonnell
Timur Alperovich
Irmo Manie