Adds the delimiter support in the local blob store. The existing
directory implementation is preserved and is not compatible with the
delimiter option, as the existing implementation assumes the usage of
filesystem directories.
The patch adds the delimiter support in the openstack-swift API. As
part of the change, the subdirectory support in results is introduced.
This occurs when a prefix and delimiter options are set and there are
subdirectories present in the listing (i.e. multiple objects under the
same prefix/delimiter). In this case, Swift will return a list of
"subdir" objects (similar to CommonPrefixes in S3), which need to be
treated differently.
The patch adds delimiter option support in the generic blob store
interface. A live integration test is added to verify that jclouds
correctly lists objects separated by a delimiter.
Openstack Swift ContainerApiLiveTest testListWithOptions assumes that
there are no other containers and that ContainerApiLiveTest container
will _always_ be the first container.
The patch changes the behavior of the test to instead pull out the
container from the returned list.
There is no notion of marker blobs in the file system blob store,
which means that when listing results after issuing the
"createDirectory" request, the marker blob will not be in the result
set.
When the delimiter option is specified, local blob store should behave
similarly to swift and return the directory marker blobs (directory
blobs) and the common prefix. This undoes the filtering we were
previously doing.
When listing blobs, Swift returns an array of "subdir" entries for
every common prefix. The openstack-swift provider needs to process
such entries and add them to the result set. The subdir array is an
array of strings, so additional metadata needs to be added to the
objects (bogus ETag, bogus LastModifiedDate, and so on).
When directory marker blobs are used, this means that potential _two_
entries are generated for every directory if:
1. the delimiter is set and matches the directory blob (e.g. dir/ and
delimiter /)
2. there are objects under the directory name (e.g. dir/blob), which
will result in results that include common prefixes (subdir)
In the above example, we should expect two results: "dir" and "dir/"
representing the directory marker blob and the common prefix,
respectively.
This is caught in the testDirectory integration test.
The patch changes the behavior of the Swift provider to correctly
handle the results in the subdir stanza and changes the test to expect
the directory marker to be returned in the list.
Multiple constructors annotated with @SerializedNames confuses
NamingStrategies.translateName and causes failures with newer JDK.
Since the second constructor does not need this annotation we remote
it.
When filtering results, we have to consider that a result was added to
the common prefixes list in its entirety (i.e. a directory). Such
results should be filtered out from the delimiter test.
An example that demonstrates this problem is if one creates a
directory "foo" and an object within it called "file". When listing
the results, they will include the directory object "foo" and the
object under "foo/file". During a non-recursive listing, we create a
list of common prefixes ("foo"). Subsequently, jclouds should remove
all objects that include the delimiter ("/"), however, that would not
apply to "foo". With the change to include the delimiter in the
listing, we need to be careful not to return two values "foo" and
"foo/".
A unit test for the local blob store to highlight this problem is
included. An integration test "testDirectory" also catches this issue.
Relaxes the membership check in prefix testing. The check is no longer
sensitive to ordering of the results, as different providers may
append, prepend, or insert in sorted order the relative prefixes.
We should not append a "/" to the marker when returning list results
in the case of directories (RELATIVE_PATH), as the names will already
include the delimiter.
Added a jclouds test to catch such regressions in the local store in
the future.
Plumbs the prefix option to the openstack-swift provider. In the
process, the support for the recursive option is modified to avoid
setting the _path_ parameter, but rather use the delimiter if required
(setting the delimiter is sufficient for a non-recursive listing).
Plumbs support for the prefix query option in the Azure provider. This
option is not compatible with the "directory" list option and an
exception is thrown if both are set.
Implements prefix support for the local blob store. The patch allows
for correctly parsing prefixes that may not terminate with a delimiter
(i.e. foo with delimiter "/" and a key foobar/key, should return
foobar/ as the common path) and ones that do (i.e. foo/).
NOTE: there is a small change in behavior in this patch.
LocalBlobStore used to return the common prefixes without the
delimiter character ("/"). However, other providers do include the
delimiter (I checked S3 and Google Cloud Storage) and LocalBlobStore
should include it as well.
jclouds should issue the requests to create all of the containers in
the container pool before sleeping. The patch modifies the
createContainerAndEnsureEmpty() method to take an additional
parameter, which specifies whether awaitConsistency() should be called
or not.
refactor AccountApiExpectedTest to AccountApiMockTest
refactor DatacenterApiExpectTest to DatacenterApiMockTest
refactor SoftwareDescriptionApiExpectTest to SoftwareDescriptionApiMockTest
refactor VirtualGuestApiExpectTest to VirtualGuestApiMockTest
refactor VirtualGuestBlockDeviceTemplateGroupApiExpectTest to VirtualGuestBlockDeviceTemplateGroupApiMockTest
