Swift implements multi-part upload with user-visible parts and an
explicit manifest. When deleting an MPU blob it can delete only the
manifest or both the manifest and parts. For consistency with other
providers, we now do the latter in the portable abstraction. Swift
ignores the multipart-manifest=delete parameter for single-part
objects. Fixesandrewgaul/s3proxy#92.
Previously we only used the blob name and not the blob
name/slo/timestamp/part size cookie which yield extra parts when
listing an MPU with parts from a previous MPU. Listing using the
stricter prefix gives the expected results.
Fixesandrewgaul/s3proxy#91.
- bump api version to 1.21
- use `alpine/3.2` image for liveTests
- use `kwart/alpine-ext:3.2-ssh` image as ssh-able image
- assert request bodies created correctly in NetworkApiMockTest
On Windows, we need to avoid trailing spaces, as the test fails to
create the required blob. Specifically, we should not test blobs named
" " and "%20 ".
When constructing the expected HTTP requests in STS tests, STS API
should not pre-encode the strings. The form parameters are already
encoded and are otherwise encoded twice.
When constructing the query path, S3 does not properly handle encoded
paths. For example, if a blob named %20 is to be placed into the blob
store, S3 would end up placing blob named " " (what %20 represents).
This occurs because the S3 provider examines the URI's path portion
(which is presented in a decoded fasion to the caller). After
examining the path, it is not encoded again. Instead, we should call
getRawPath() to avoid this issue.
There are two issues on the decoding path:
1. Given a blob named " ", S3 API will throw a RuntimeException due to
a NULL check -- the key that it uses is NULL to represent the XML
content " " corresponding to the blob name.
2. Given a blob named "%20 ", S3 API will generate a URI for a blob
named "%20%20", which is also incorrect. The correct URI would be
"%2520%20" (escaping the first "%" and " " characters).
The first issue is due to the currentOrNull() helper, which calls
trim() on the string and then compares the string to an empty string.
This means that a blob named " " will be parsed as "" and then
converted to NULL as the result of that method. Passing "null" as the
key then fails in a number of places (notably, appendPath()).
The second issue is due to the appendPath() method in the jclouds Uris
class. The issue here is that appendPath() calls urlDecode() and
passes the result to path(). The path() method, in turn, also calls
urlDecode(). After these transformations, a properly encoded blob of
the form %2520%20 turns into "%20 " and then " " (two spaces). After
these transformations the path is encoded again, resulting in "%20%20"
(which is wrong).
deprecated hardware types in EC2 are flagged using this, according to latest AWS advice,
to fix situations where deprecated hardware types would be chosen when a non-deprecated alternative exists.
we also deprecate T2 because it requires a VPC.
also fixes semantics of ImagesToRegionAndIdMap to match the logic used in the TemplateBuilderImpl;
RegionAndName should contain the ID not the ProviderID. this is only really used in tests so no external impact.
(previously the cache only worked if provider ID matched the non-location segment of the image ID.)
however this does now assert that image id's are in the right format for AWS, as suggested by @nacx.
Move the prefix and delimiter tests from BaseContainerLiveTest to
BaseContainerIntegrationTest, as that class is subclassed in
Filesystem integration tests. This will make sure the file system blob
store will not regress.
The ListContainerTest is updated to _not_ use a file separator
character, as that leads to a possible creation of directories.
Instead, "-" is used as the delimiter for the delimiter option tests.
The patch adds the delimiter support in the openstack-swift API. As
part of the change, the subdirectory support in results is introduced.
This occurs when a prefix and delimiter options are set and there are
subdirectories present in the listing (i.e. multiple objects under the
same prefix/delimiter). In this case, Swift will return a list of
"subdir" objects (similar to CommonPrefixes in S3), which need to be
treated differently.
The patch adds delimiter option support in the generic blob store
interface. A live integration test is added to verify that jclouds
correctly lists objects separated by a delimiter.
Openstack Swift ContainerApiLiveTest testListWithOptions assumes that
there are no other containers and that ContainerApiLiveTest container
will _always_ be the first container.
The patch changes the behavior of the test to instead pull out the
container from the returned list.
There is no notion of marker blobs in the file system blob store,
which means that when listing results after issuing the
"createDirectory" request, the marker blob will not be in the result
set.
When listing blobs, Swift returns an array of "subdir" entries for
every common prefix. The openstack-swift provider needs to process
such entries and add them to the result set. The subdir array is an
array of strings, so additional metadata needs to be added to the
objects (bogus ETag, bogus LastModifiedDate, and so on).
When directory marker blobs are used, this means that potential _two_
entries are generated for every directory if:
1. the delimiter is set and matches the directory blob (e.g. dir/ and
delimiter /)
2. there are objects under the directory name (e.g. dir/blob), which
will result in results that include common prefixes (subdir)
In the above example, we should expect two results: "dir" and "dir/"
representing the directory marker blob and the common prefix,
respectively.
This is caught in the testDirectory integration test.
The patch changes the behavior of the Swift provider to correctly
handle the results in the subdir stanza and changes the test to expect
the directory marker to be returned in the list.
Plumbs the prefix option to the openstack-swift provider. In the
process, the support for the recursive option is modified to avoid
setting the _path_ parameter, but rather use the delimiter if required
(setting the delimiter is sufficient for a non-recursive listing).
- Tidy up DockerTemplateOptions and add tests for new options
- Make Dns a list of strings in HostConfig
- Change template option builder to remove use of optionals
- Update tests to verify HostConfig.Dns as list
When a filesystem blob does not have content metadata and when users
set jclouds.filesystem.auto-detect-content-type to tru, probe the file
type to return to clients. This is useful when using jclouds to serve
an existing filesystem.
Currently swift creates blob-1, blob-2... blob-n for multipart
upload segments, which are very common names that can easily
collide with normal objects. This changes the naming convention
to that of python-swiftclient (the `swift' command line).
python-swiftclient also uses a different segment container by
default, that's not implemented by this patch
Providers yield different messages so remove parsing from test. Also
narrow exception handling scope.TTP 416 into IllegalArgumentException
Providers yield different messages so remove parsing from test. Also
narrow exception handling scope.
The names created do not match those searched for.
They are created in FormatSharedNamesAndAppendUniqueStringToThoseWhichRepeat.java
and have are of this form jclouds#I-0#e96.
But for example jclouds#I-0#us-east-1#* is used as the search term.
- Change to portBindings and Map intead of List
- Tidy up formatting for DockerTemplateOptions
- Remove superfluous checkNotNull calls
- Document the Map contents for portBindings option
- Remove unused import to fix Checkstyle
* Improves extension namespace configuration to use Guice multibindings
so each provider can cleanly provide their own namespaces.
* Fixes the HPCloud Compute volume attachment namespace and adds the
corresponding live tests.
* Fixes the Rackspace CloudServers UK volume attachment namespace.
improve DeploymentToNodeMetadata
enhance Deployment value object
add support for get/set NetworkConfiguration
added AzureComputeTemplateOptions to manage networkConfigurations
add support for create/checkAvailable/list storageAccounts
add support for inboundPorts
add more RoleSize
add support for SecurityGroupExtension by using NetworkSecurityGroup
fix destroy node
fix LocationToLocation
fix OsImageToImage
- Some users get a DependencyVioloation, rather than InvalidGroup.InUse,
when attempting to delete the security group. This caused
cleanupIncidentalResources to propagate an exception.
- Fixes it by converting this to an IllegalStateException (in same
way as is done for “InUse”)
- Adds tests (using MockWebServer) for happy-path and for failing
to delete the security group with each of InUse and DependencyViolation
responses.
- Adds Timeouts.cleanupIncidentalResources
- Use that timeout in EC2, when retrying the deletion of security group
on VM deletion (previously hard-coded as 3 seconds).
- Configure that timeout in the tests, so deterministic number of retries
Java on OS X does not support extended attributes, which the
filesystem blobstore uses to implement user metadata. This disable
the relevant test on OS X
- update value objects (Container, Config)
- add new value object (Resource, StatusCode)
- add more options for ContainerApi
- add support for pause/unpause, restart, kill, copy and attach APIs + Mock and Live tests
- add *ParseTest
- refactor MockTests
- removed *Options.NONE
- cleanup low-value javadoc
- more coverage in LiveTests
- removed useless DockerProperties
- add comments for TLS issue
- use Date where needed
- fix listImages
- add package-private constructor to value classes
- remove KillOptions, RestartOptions, StopOptions in favour of @QueryParam
- use PEM instead of PKCS12 format
- update docker/pom.xml to use sshj dependencies
- explicit usage of OkHttp driver
- simplify DockerApiMetadata
- add bouncycastle driver explicitly
when you bind a host volume into docker, java does not correctly
detect that and checks the xattr support on the root fs instead
of the host fs. The root fs often does not support xattr, so
the check would fail even if the target really does support xattr.
the fix is just try setting the xattrs anyway, and let them fail
if there really isn't xattrs support
This allows callers to read the content length during container
listing. Tested against: atmosonline, aws-s3, azureblob, filesystem,
and transient. Intentionally not implemented for legacy swift
provider.
make the filesystem blob store distinguish between a/ and a. a/
must be a directory blob with no content and only metadata
on listing, only directories with md5 attribute is considered a
blob and returned
This commit also improves some internal behavior related to the
bootstrap configuration generation, and deprecated some methods that
will be removed in upcoming versions.
* Removes all methods that return the bootstrap DataBag or the raw
configuration json to return the BootstrapConfig object instead.
* Deprecates all methods that provide support for old Chef versions.
* Formats the Enterprise Chef provider according to the jclouds
formatting guidelines.
* Deprecates the ChefContext view. That view is not an abstraction and
only provides access to the ChefService. It can be obtained from the
ChefApi itself.
* Remove oauth as a standalone api.
* Rename redundant OAuthAuthenticationFilter to OAuthFilter.
* Make AuthorizationApi more simple by using form semantics.
* Simplified OAuth by only permitting RS256 and none algos.
Andrew Gaul
Josef Cacek
Ignasi Barrera
Zack Shoylev
Ka-Hing Cheung
Andrea Turli
Stuart Hendren
Josef Cacek
John McDonnell
googlielmo
Timur Alperovich
Andrew Kennedy
Irmo Manie
Oliver Gondža
Nate Rosenblum
Adrian Bravo
Arvind Nadendla
Marcus Wanner
Csaba Palfi
Alex Heneveld
Andrew Bayer
Iván López
Karel Vervaeke
Jakub Bartecek
Everett Toews
Matt Hurne
lbortman
Jeremy Daggett
Evgeny Tarasenko
Aled Sage
Taylor Jones
Chris Custine
Jeremy Daggett
Everett Toews
Nikolay Sokolov
William Chu
Adrian Cole
Reynald Borer