When the delimiter option is specified, local blob store should behave
similarly to swift and return the directory marker blobs (directory
blobs) and the common prefix. This undoes the filtering we were
previously doing.
When listing blobs, Swift returns an array of "subdir" entries for
every common prefix. The openstack-swift provider needs to process
such entries and add them to the result set. The subdir array is an
array of strings, so additional metadata needs to be added to the
objects (bogus ETag, bogus LastModifiedDate, and so on).
When directory marker blobs are used, this means that potential _two_
entries are generated for every directory if:
1. the delimiter is set and matches the directory blob (e.g. dir/ and
delimiter /)
2. there are objects under the directory name (e.g. dir/blob), which
will result in results that include common prefixes (subdir)
In the above example, we should expect two results: "dir" and "dir/"
representing the directory marker blob and the common prefix,
respectively.
This is caught in the testDirectory integration test.
The patch changes the behavior of the Swift provider to correctly
handle the results in the subdir stanza and changes the test to expect
the directory marker to be returned in the list.
When filtering results, we have to consider that a result was added to
the common prefixes list in its entirety (i.e. a directory). Such
results should be filtered out from the delimiter test.
An example that demonstrates this problem is if one creates a
directory "foo" and an object within it called "file". When listing
the results, they will include the directory object "foo" and the
object under "foo/file". During a non-recursive listing, we create a
list of common prefixes ("foo"). Subsequently, jclouds should remove
all objects that include the delimiter ("/"), however, that would not
apply to "foo". With the change to include the delimiter in the
listing, we need to be careful not to return two values "foo" and
"foo/".
A unit test for the local blob store to highlight this problem is
included. An integration test "testDirectory" also catches this issue.
Relaxes the membership check in prefix testing. The check is no longer
sensitive to ordering of the results, as different providers may
append, prepend, or insert in sorted order the relative prefixes.
We should not append a "/" to the marker when returning list results
in the case of directories (RELATIVE_PATH), as the names will already
include the delimiter.
Added a jclouds test to catch such regressions in the local store in
the future.
Implements prefix support for the local blob store. The patch allows
for correctly parsing prefixes that may not terminate with a delimiter
(i.e. foo with delimiter "/" and a key foobar/key, should return
foobar/ as the common path) and ones that do (i.e. foo/).
NOTE: there is a small change in behavior in this patch.
LocalBlobStore used to return the common prefixes without the
delimiter character ("/"). However, other providers do include the
delimiter (I checked S3 and Google Cloud Storage) and LocalBlobStore
should include it as well.
jclouds should issue the requests to create all of the containers in
the container pool before sleeping. The patch modifies the
createContainerAndEnsureEmpty() method to take an additional
parameter, which specifies whether awaitConsistency() should be called
or not.
Providers yield different messages so remove parsing from test. Also
narrow exception handling scope.TTP 416 into IllegalArgumentException
Providers yield different messages so remove parsing from test. Also
narrow exception handling scope.
for range get in a LocalBlobStore, metadata.getSize() has the
original blob size and contentMetadata.getContentLength() has
the actual payload size. Other blobstores have the same size
in both
container may not be valid hostnames and the current blob uri
code can fail. there isn't really a way to use uri for transient
blobstore anyway, and removing it is the easier option. for
reference, filesystem blobstore also doesn't do this
for range get in a LocalBlobStore, metadata.getSize() has the
original blob size and contentMetadata.getContentLength() has
the actual payload size. Other blobstores have the same size
in both
Some object stores have additive semantics for metadata. As currently
implemented Azure, S3, and Swift consistently replace user metadata
but have different behavior for system metadata, e.g., Content-Type.
