InputStream.read(byte[]) can return fewer bytes than requested.
Specifically ByteSource.concat(ByteSource...).openStream() will only
return as many bytes as the current ByteSource contains. Thus
ByteSources.repeatingArrayByteSource(byte[]).openStream() will return
short reads despite the byte[] input from its single logical
InputStream.
Callers should instead explicitly set contentMD5, usually with the
results from Guava Hashing.md5(). This narrows the API and removes a
strange IOException from callers. Further it removes a dangerous
rebuffering of arbitrarily-large non-repeatable Payloads.
The existing approach for deleting objects in a container suffers
from a head-of-line blocking problem. This commit implements a better
scheme which does not have that problem. This scheme uses a counting
semaphore for making sure that a certain number of futures are
issued in parallel. As each of these futures is completed, one
permit of the semaphore is released.
Added unit tests for testing this new scheme.
When issuing many simultaneous requests to Synaptic Atmos I observed:
HTTP/1.1 failed with code 500, error: AtmosError
[code=1040, message=The server is busy. Please try again.]
Previously all clients slept for fixed intervals and thus retried
around the same time. This commit adds a random delay which should
better distribute load on the provider.
Previously jclouds could use an unlimited number of threads on its
user ExecutorService. While this ExecutorService will go away when we
complete deasyncafication, we should prevent jclouds from misbehaving
until that time.
FileBackedOutputStream.asByteSource.getInput returns a FileInputStream
which we do not close. We later call FileBackedOutputStream.reset
which removes the underlying File. This fails on Windows which does
not support deleting an open file and leaks resources on other
platforms. Eagerly close to address this issue.
Payload.getInput must always call openStream to handle overridden
methods correctly. Previously this caused errors in jclouds-chef in
BaseCipherPayload.
Also deprecate byte[], File, InputSupplier<InputStream>, and String
Payloads. Callers should instead provide a ByteSource via
ByteSource.wrap(byte[]) and Files.asByteSource(File)
We plan to transition Payload to ByteSource in the next major release.
Unfortunately Payload.getInput masks its checked exception and
ByteSource.getInput is final so we cannot continue to mask the
exceptions. Deprecation of getInput and addition openStream allows us
to transition callers from the former to the latter.
This changeset introduces an alternative to PayloadSlicer,
IterablePayloadSlicer, with a method for returning a Payload iterator.
...swift.blobstore.strategy.internal.SequentialMultipartUploadStrategy
has been updated to to use a payload iterator.
This allows jclouds to factor out common headers into the Caller so they don't
have to be repeated in the Callee.
The Produces/Consumes annotations (Content-Type/Accept headers) will also
propagate from the Caller to the Callee.
This patch moves the Invokable Parameter cache to Reflection2 and adds
a convenience method for it to allow it to be shared by multiple
callers. The subsequent ability of S3Utils to use this cache results
in a ~40% improvement in performance for generating signed GETs and
PUTs for S3. This commit also converts a few others calls to
Invokable.getParameters() but the observed benefit from those was
small in microbenchmarks.
Atmos does not return a location header when writing zero-length
objects, which normally throws an HttpResponseException: no uri in
headers or content.
BlobStore.createContainerInLocation is supposed to return True if the
container was newly created and False if the container already
existed. This commit makes that happen for Swift blobstores.
Not all S3-compatible providers support virtual host buckets and thus
we should disable this feature by default. Continue to enable virtual
host buckets for AWS-S3 which supports this although this feature
suffers from DNS settling issues. Ran ran integration tests against
AWS-S3 and Scality using its S3 API.
By caching the results from Invokable.getParameters(), this commit
improves request signing performance (GETs and PUTs) for S3 by >
3X. These performance problems were seen in production and diagnosed
using the YourKit profiler.
Generalized the Arg0ToPagedIterable to allow to propagating
all arguments. This will help building PagedIterables for
api methods that require more than one argument to be invoked.
This is a TOCTOU violation and FilePayload.getInput already propagates
this. This commit allows external callers like jclouds-cli to
introspect on the exception type, returning a more friendly error
message in some situations.
Some providers (specifically HP Cloud and Google Cloud Storage) do not
properly support Expect: 100-continue headers. JDK7 is stricter in its
handling of the Expect header than JDK6 -- in particular, it expects
servers to properly respond to an expect header and times out only if a
prior timeout did not exist on the underlying HTTP connection. As a
result, JDK7 tests against these providers hang and fail.
This commit introduces a new filter -- appropriate called
StripExpectHeader -- that is controlled by the property
jclouds.strip-expect-header. The property defaults to false to preserve
existing behavior but allows applications to tweak Expect header
handling.
Tested by running HPCS live tests with JDK7 -- previously most of these
tests would fail with timeouts.
Closes JCLOUDS-181
- Adds the SecurityGroupExtension to compute, with tests and stub
support.
- Gets everything else to actually build against this.
- Unifies on compute's IpPermission/IpProtocol, eliminating EC2's.
- Converters from EC2/Nova/CloudStack SecurityGroup (and rules, for
the latter two) to the compute SecurityGroup (and rules, etc).
- EC2SecurityGroupExtension and tests.
- AWSEC2SecurityGroupExtension and tests - depends on JCLOUDS-99.
Andrew Gaul
Christopher Dancy
Ignasi Barrera
Jeremy Daggett
Pasi Niemi
Shri Javadekar
Andrew Gaul
Andrew Phillips
Zack Shoylev
Andrew Bayer
Eric Evans
Everett Toews
Ignacio Mulas
Niraj Tolia
Adrian Cole
Diwaker Gupta
Ignasi Barrera
JoshVote
Anton Panasenko