Due to Swift's wonky MPU implementation we must try the multi-part
delete first followed by a single-part delete if it fails. Previously
delete would leave orphan subobjects. Fixesandrewgaul/s3proxy#92.
Swift implements multi-part upload with user-visible parts and an
explicit manifest. When deleting an MPU blob it can delete only the
manifest or both the manifest and parts. For consistency with other
providers, we now do the latter in the portable abstraction. Swift
ignores the multipart-manifest=delete parameter for single-part
objects. Fixesandrewgaul/s3proxy#92.
Previously we only used the blob name and not the blob
name/slo/timestamp/part size cookie which yield extra parts when
listing an MPU with parts from a previous MPU. Listing using the
stricter prefix gives the expected results.
Fixesandrewgaul/s3proxy#91.
The patch adds the delimiter support in the openstack-swift API. As
part of the change, the subdirectory support in results is introduced.
This occurs when a prefix and delimiter options are set and there are
subdirectories present in the listing (i.e. multiple objects under the
same prefix/delimiter). In this case, Swift will return a list of
"subdir" objects (similar to CommonPrefixes in S3), which need to be
treated differently.
The patch adds delimiter option support in the generic blob store
interface. A live integration test is added to verify that jclouds
correctly lists objects separated by a delimiter.
Openstack Swift ContainerApiLiveTest testListWithOptions assumes that
there are no other containers and that ContainerApiLiveTest container
will _always_ be the first container.
The patch changes the behavior of the test to instead pull out the
container from the returned list.
When listing blobs, Swift returns an array of "subdir" entries for
every common prefix. The openstack-swift provider needs to process
such entries and add them to the result set. The subdir array is an
array of strings, so additional metadata needs to be added to the
objects (bogus ETag, bogus LastModifiedDate, and so on).
When directory marker blobs are used, this means that potential _two_
entries are generated for every directory if:
1. the delimiter is set and matches the directory blob (e.g. dir/ and
delimiter /)
2. there are objects under the directory name (e.g. dir/blob), which
will result in results that include common prefixes (subdir)
In the above example, we should expect two results: "dir" and "dir/"
representing the directory marker blob and the common prefix,
respectively.
This is caught in the testDirectory integration test.
The patch changes the behavior of the Swift provider to correctly
handle the results in the subdir stanza and changes the test to expect
the directory marker to be returned in the list.
Plumbs the prefix option to the openstack-swift provider. In the
process, the support for the recursive option is modified to avoid
setting the _path_ parameter, but rather use the delimiter if required
(setting the delimiter is sufficient for a non-recursive listing).
Currently swift creates blob-1, blob-2... blob-n for multipart
upload segments, which are very common names that can easily
collide with normal objects. This changes the naming convention
to that of python-swiftclient (the `swift' command line).
python-swiftclient also uses a different segment container by
default, that's not implemented by this patch
Andrew Gaul
Reijhanniel Jearl Campos
Rita Zhang
Oliver Gondža
Timur Alperovich
Ka-Hing Cheung
Zack Shoylev
Everett Toews
Jeremy Daggett
Adrian Cole
Ignasi Barrera
Adrian Cole