AWS S3 MPU ETag are hashes of the part ETag headers but some
implementations, specifically S3Proxy with the filesystem provider,
represent multi-part objects as a single object. Remove these checks
since they add nothing.
These are simpler than the full XML API and better supported by
non-AWS S3 implementations, e.g., Ceph, S3Proxy. Further this makes
the provider more consistent when creating a bucket or object which
only supports setting canned ACLs.
Swift implements multi-part upload with user-visible parts and an
explicit manifest. When deleting an MPU blob it can delete only the
manifest or both the manifest and parts. For consistency with other
providers, we now do the latter in the portable abstraction. Swift
ignores the multipart-manifest=delete parameter for single-part
objects. Fixesandrewgaul/s3proxy#92.
Previously we only used the blob name and not the blob
name/slo/timestamp/part size cookie which yield extra parts when
listing an MPU with parts from a previous MPU. Listing using the
stricter prefix gives the expected results.
Fixesandrewgaul/s3proxy#91.
- bump api version to 1.21
- use `alpine/3.2` image for liveTests
- use `kwart/alpine-ext:3.2-ssh` image as ssh-able image
- assert request bodies created correctly in NetworkApiMockTest
On Windows, we need to avoid trailing spaces, as the test fails to
create the required blob. Specifically, we should not test blobs named
" " and "%20 ".
When constructing the expected HTTP requests in STS tests, STS API
should not pre-encode the strings. The form parameters are already
encoded and are otherwise encoded twice.
When constructing the query path, S3 does not properly handle encoded
paths. For example, if a blob named %20 is to be placed into the blob
store, S3 would end up placing blob named " " (what %20 represents).
This occurs because the S3 provider examines the URI's path portion
(which is presented in a decoded fasion to the caller). After
examining the path, it is not encoded again. Instead, we should call
getRawPath() to avoid this issue.
There are two issues on the decoding path:
1. Given a blob named " ", S3 API will throw a RuntimeException due to
a NULL check -- the key that it uses is NULL to represent the XML
content " " corresponding to the blob name.
2. Given a blob named "%20 ", S3 API will generate a URI for a blob
named "%20%20", which is also incorrect. The correct URI would be
"%2520%20" (escaping the first "%" and " " characters).
The first issue is due to the currentOrNull() helper, which calls
trim() on the string and then compares the string to an empty string.
This means that a blob named " " will be parsed as "" and then
converted to NULL as the result of that method. Passing "null" as the
key then fails in a number of places (notably, appendPath()).
The second issue is due to the appendPath() method in the jclouds Uris
class. The issue here is that appendPath() calls urlDecode() and
passes the result to path(). The path() method, in turn, also calls
urlDecode(). After these transformations, a properly encoded blob of
the form %2520%20 turns into "%20 " and then " " (two spaces). After
these transformations the path is encoded again, resulting in "%20%20"
(which is wrong).
deprecated hardware types in EC2 are flagged using this, according to latest AWS advice,
to fix situations where deprecated hardware types would be chosen when a non-deprecated alternative exists.
we also deprecate T2 because it requires a VPC.
also fixes semantics of ImagesToRegionAndIdMap to match the logic used in the TemplateBuilderImpl;
RegionAndName should contain the ID not the ProviderID. this is only really used in tests so no external impact.
(previously the cache only worked if provider ID matched the non-location segment of the image ID.)
however this does now assert that image id's are in the right format for AWS, as suggested by @nacx.
Move the prefix and delimiter tests from BaseContainerLiveTest to
BaseContainerIntegrationTest, as that class is subclassed in
Filesystem integration tests. This will make sure the file system blob
store will not regress.
The ListContainerTest is updated to _not_ use a file separator
character, as that leads to a possible creation of directories.
Instead, "-" is used as the delimiter for the delimiter option tests.
The patch adds the delimiter support in the openstack-swift API. As
part of the change, the subdirectory support in results is introduced.
This occurs when a prefix and delimiter options are set and there are
subdirectories present in the listing (i.e. multiple objects under the
same prefix/delimiter). In this case, Swift will return a list of
"subdir" objects (similar to CommonPrefixes in S3), which need to be
treated differently.
The patch adds delimiter option support in the generic blob store
interface. A live integration test is added to verify that jclouds
correctly lists objects separated by a delimiter.
Openstack Swift ContainerApiLiveTest testListWithOptions assumes that
there are no other containers and that ContainerApiLiveTest container
will _always_ be the first container.
The patch changes the behavior of the test to instead pull out the
container from the returned list.
There is no notion of marker blobs in the file system blob store,
which means that when listing results after issuing the
"createDirectory" request, the marker blob will not be in the result
set.
When listing blobs, Swift returns an array of "subdir" entries for
every common prefix. The openstack-swift provider needs to process
such entries and add them to the result set. The subdir array is an
array of strings, so additional metadata needs to be added to the
objects (bogus ETag, bogus LastModifiedDate, and so on).
When directory marker blobs are used, this means that potential _two_
entries are generated for every directory if:
1. the delimiter is set and matches the directory blob (e.g. dir/ and
delimiter /)
2. there are objects under the directory name (e.g. dir/blob), which
will result in results that include common prefixes (subdir)
In the above example, we should expect two results: "dir" and "dir/"
representing the directory marker blob and the common prefix,
respectively.
This is caught in the testDirectory integration test.
The patch changes the behavior of the Swift provider to correctly
handle the results in the subdir stanza and changes the test to expect
the directory marker to be returned in the list.
Plumbs the prefix option to the openstack-swift provider. In the
process, the support for the recursive option is modified to avoid
setting the _path_ parameter, but rather use the delimiter if required
(setting the delimiter is sufficient for a non-recursive listing).
- Tidy up DockerTemplateOptions and add tests for new options
- Make Dns a list of strings in HostConfig
- Change template option builder to remove use of optionals
- Update tests to verify HostConfig.Dns as list
When a filesystem blob does not have content metadata and when users
set jclouds.filesystem.auto-detect-content-type to tru, probe the file
type to return to clients. This is useful when using jclouds to serve
an existing filesystem.
Currently swift creates blob-1, blob-2... blob-n for multipart
upload segments, which are very common names that can easily
collide with normal objects. This changes the naming convention
to that of python-swiftclient (the `swift' command line).
python-swiftclient also uses a different segment container by
default, that's not implemented by this patch
Providers yield different messages so remove parsing from test. Also
narrow exception handling scope.TTP 416 into IllegalArgumentException
Providers yield different messages so remove parsing from test. Also
narrow exception handling scope.
The names created do not match those searched for.
They are created in FormatSharedNamesAndAppendUniqueStringToThoseWhichRepeat.java
and have are of this form jclouds#I-0#e96.
But for example jclouds#I-0#us-east-1#* is used as the search term.
- Change to portBindings and Map intead of List
- Tidy up formatting for DockerTemplateOptions
- Remove superfluous checkNotNull calls
- Document the Map contents for portBindings option
- Remove unused import to fix Checkstyle
* Improves extension namespace configuration to use Guice multibindings
so each provider can cleanly provide their own namespaces.
* Fixes the HPCloud Compute volume attachment namespace and adds the
corresponding live tests.
* Fixes the Rackspace CloudServers UK volume attachment namespace.
improve DeploymentToNodeMetadata
enhance Deployment value object
add support for get/set NetworkConfiguration
added AzureComputeTemplateOptions to manage networkConfigurations
add support for create/checkAvailable/list storageAccounts
add support for inboundPorts
add more RoleSize
add support for SecurityGroupExtension by using NetworkSecurityGroup
fix destroy node
fix LocationToLocation
fix OsImageToImage
Andrew Gaul
Zhao Jin
Rita Zhang
Zack Shoylev
Ignasi Barrera
Graeme-Miller
Josef Cacek
Ka-Hing Cheung
Andrea Turli
Stuart Hendren
Josef Cacek
John McDonnell
googlielmo
Timur Alperovich
Andrew Kennedy
Irmo Manie
Oliver Gondža
Nate Rosenblum
Adrian Bravo
Arvind Nadendla
Marcus Wanner
Csaba Palfi
Alex Heneveld
Andrew Bayer
Iván López
Karel Vervaeke
Jakub Bartecek
Everett Toews
Matt Hurne