This change improves the performance of writing to sockets with the
default Java URL connection HTTP client, by enlarging the buffer used
for socket writes from an implicit hard-coded 4KB / 8KB buffer to a
configurable 32KB buffer.
The buffer size is now controlled by the following property with the
following default value:
jclouds.output-socket-buffer-size: 32768
The implementation is based on a variant of ByteStreams.copy (written as
ByteStreams2.copy) which accepts the buffer size as an argument, unlike
the original Guava code that uses a hard-coded size.
The change was done directly within the loop that copies the input
stream to the output stream, and not by wrapping a BufferedOutputStream
around the existing output stream, in order to avoid copying the payload
twice.
On some platforms this change can improve both the putBlob throughput
and the total CPU consumption.
These should provide a descriptive second argument, not the same as
the first argument which is null in the failure case. This also found
a logic error in CreateVolumeResponseHandler.
The x-amz-copy-source header on S3 CopyObject should be URL encoded (as
a path). This is not universally true of all headers though (for example
the = in x-amz-copy-source-range) therefore introducing a new parameter
on @Headers to indicate whether URL encoding should take place.
Previously read returned a value between -128 and 127. -1 indicates
end of stream, causing issues for callers. Instead return values
between 0 and 255 as intended.
Reflective creation of SimpleTimeLimiter to allow compatibility with
Guava 23.0. SimpleTimeLimiter.create(ExecutorService) was introduced
in Guava 22.0 to replace the SimpleTimeLimiter(ExecutorService)
constructor, which was deprecated in Guava 22.0 and removed in Guava
23.0.
With a shared credential store the configuration of one compute service leaks in all others, causing the wrong credentials to be used when not overriden.
The intention is to use @SinceApiVersion for this purpose, but that
would affect a number of APIs, and we would want to have good test
coverage before merging that change (in
FormSignerUtils#getAnnotatedApiVersion). However, there is some issue
wth certain tests at resent that means we cannot successfully test
all APIs that make use of @SinceApiVersion in order to assure
ourselves that FormSignerUtils will not introduce some problem.
See https://github.com/jclouds/jclouds/pull/1102#issuecomment-302682049
for details.
This annotation is introduced as a temporary measure in order to
decouple the functionality of FormSignerUtils#getAnnotatedApiVersion
from @SinceApiVersion and the tests in question. It can be removed and
replaced by @SinceApiVersion when those tests are fixed.
Designates that a method overrides the {@link ApiVersion} on the class
with a specific value.
For https://issues.apache.org/jira/browse/JCLOUDS-1235.
This change takes the approach of storing the information about the
overall list of groups within the `SecurityGroupInRegion` when it is
created, so that any subsequent conversion operation has access to all
the groups in the same region as the one to be converted.
It also collapses the functionality of `NovaSecurityGroupToSecurityGroup`,
`SecurityGroupRuleToIpPermission` and `FindSecurityGroupWithNameAndReturnTrue`
all into `NovaSecurityGroupInRegionToSecurityGroup`, and deletes the
now unused-classes SecurityGroupRuleToIpPermission,
NovaSecurityGroupToSecurityGroup and associated tests.
Jclouds sends default user agent string with each request to cloud
services. But some of the application would like to overide this and
send custom user agent instead.
This commit define a string property to overide this default user agent
string. This property will be applied to all outgoing http request to
cloud services
JCLOUDS-819
Readers can confuse this with 1. Found via error-prone. Fixed via:
find -name \*.java | xargs sed -i 's/\( [0-9][0-9]*\)l/\1L/g'
find -name \*.java | xargs sed -i 's/\(([0-9][0-9]*\)l/\1L/g'
* dynamically call TypeToken.isSupertypeOf with Guava 19 and later and
TypeToken.isAssignableFrom with Guava 18 and earlier
* consume or ignore values from methods with CheckReturnValue
* replace usage of removed Iterators.emptyIterator
When Class.forName is called for a class in a different bundle it will
fail as the default karaf class loader won't load classes from other
bundles.
I have fixed this by using the classloader of the original
(non-autovalue) type and assuming it will be in the same bundle as the
autovalue type (I think this is a reasonable assumtion).
So far the only place where I've actually seen this being an issue is
when using the jclouds-labs-google provider within karaf. It fails
when serialising the Firewall.Rule class within a FirewallOptions
object.
Related issues: JCLOUDS-1166, JCLOUDS-1160, JCLOUDS-1169
This change reverts commit da4614e and upgrades from Gson 2.3.1 to 2.5,
which is the last version that exports the internal packages, used in
jclouds-core, in the OSGi bundle.
This is a combination of 16 commits:
* First approach to ArbitraryCpuRamTemplateBuilderImpl
* Several fixes: refactoring some names, format, identation problems, some missing license headers and generateId method
* Refactored parse utility
* Added GoogleComputeEngineArbitraryCpuRamTemplateBuilderImpl to support GCE custom machine URI
* extracted hardware creation to automaticHardwareForCpuAndRam method
* Fixed ide automatic asterisk imports
* correcting WIP base case PR according to comments
* added machineTypeUriToHardware to set custom hardware in nodes
* fix checkstyle violations and other PR comments
* Set the providerId to custom machineType URI and fix adding node log
* Arbitrary hardware tests added to BaseTemplateBuilderLiveTest and GoogleComputeEngineTemplateBuilderLiveTest
* Added two more tests to BaseTemplateBuilderLiveTest
* Move repeated constants to TestUtils to reuse code
* Fix full path in the Hardware id and URI
* Add custom hardware tests to BaseComputeServiceLiveTest and GCEServiceLiveTest
* Change customHardware test to use buildTemplate and fix identation
Some providers need other APIs or portable services to perform certain
operations. This change allows to pass existing Context and View
instances when creating a context, so they are added to the injector of
the context being built. This allows to inject external apis and
providers into an existing provider to leverage the external
functionality.
This fallback was added by mistake and because of my bad advice
on a code review. Fallbacks only apply for failed responses which
status code is >= 300, so a fallback for a 204 would never
be triggered.
This reverts commit e65950b858.
This commit refactors the ImageCacheSupplier to act as a
proper cache. It is used by the ImageExtesion and all operations
on the images are propagated to the cache.
A method has also been added to the TemplateBuilder to let users
force a cache refresh. There have been several requests to provide a way
to disable image caching in the compute abstraction, and this new method
should fix that.
The patch implements a QueryValue class, which encodes the underlying
value based on whether the "encoded" flag is set. This class is used
by the RestAnnotationProcessor to propagate the @Encoded value set on
any parameters.
Since the encoding is now handled by the QueryValue instances, we
should no longer call encodeQueryLine() in the URI builder and instead
call buildQueryLine(). The caveat is that we need to make sure all of
the parameters that may need to be encoded are converted to QueryValue
objects. This is done by converting Object instances to QueryValue by
an instance of the TransformObjectToQueryValue when adding any query
parameters to the URI.
Adds support for the @Encoded option for the @QueryParam annotation.
The @Encoded params are not encoded, while all parameters that don't
have it are encoded. The change applies to the @QueryParam annotation
on a single parameter. There is no way to express @Encoded on the list
of parameters and their values in @QueryParams.
The big change is that query parameter encoding is now handled within
the annotation processor, as opposed to relying on the UriBuilder to
perform the encoding. This is required since the UriBuilder does not
have any information about additional annotations associated with each
of the query parameters.
Also, adds unit tests for making sure keys and values are properly
encoded when using the @QueryParams option.
Certain providers (e.g. Google Cloud Storage) place tokens that should
be encoded in the request path (e.g. GET
http://<host>/b/<bucket>/o/<object>) and expect them to be
percent-encoded. In the above example a GET request for "foo/bar"
should be translated to http://<host>/b/<bucket>/o/foo%2Fbar.
Currently, there is no way to express this in jclouds, as the entire
request path is encoded exactly once and there is no control over
whether a request parameter should be handled specially. In the
example above, "/" are not encoded in the path and the URL is
submitted as "http://<host>/b/<bucket>/o/foo/bar", which may be wrong.
This patch extends the annotation processor to support @Encoded for
the individual parameters of the request. However, this means that the
entire path is _NOT_ URL encoded. The caller *must* make sure that the
appropriate parameters are encoded -- ones that are marked with the
@Encoded annotation. Parameters not marked with the @Encoded
annotation are URI encoded prior to being added to the path. This
means that "/" characters will also be URI encoded in this case (i.e.
"foo/bar" is turned into "foo%2Fbar").
For the Google Storage provider, we will annotate the parameters that
are going to be pre-encoded (object names) and ensure the provider
encodes them prior to calling the API (separate patch in
jclouds-labs-google).
jclouds should not decode query strings that are passed to create HTTP
requests. This is problematic because in some cases a wrong request
may be generated. The most obvious example is if one passes the "+"
character. For example, the following query parameter: "users=me+you"
is stored by the URI builder as "me you" and subsequently appears in
the request as "users=me%20you", as opposed to "users=me%2Byou" (%2b
is percent encoding for "+").
This is not currently a problem because jclouds relies on the
isUrlEncoded() method to check if a query parameter should be decoded
and the situation above is avoided.
This PR attempts to suggest an alternative (and what I believe is
simpler) approach: on the path of crafting requests, jclouds should
only *encode*, not decode strings. Specifically, jclouds should
_never_ be in a situation where it relies on the isUrlEncoded()
method.
When constructing the query path, S3 does not properly handle encoded
paths. For example, if a blob named %20 is to be placed into the blob
store, S3 would end up placing blob named " " (what %20 represents).
This occurs because the S3 provider examines the URI's path portion
(which is presented in a decoded fasion to the caller). After
examining the path, it is not encoded again. Instead, we should call
getRawPath() to avoid this issue.
There are two issues on the decoding path:
1. Given a blob named " ", S3 API will throw a RuntimeException due to
a NULL check -- the key that it uses is NULL to represent the XML
content " " corresponding to the blob name.
2. Given a blob named "%20 ", S3 API will generate a URI for a blob
named "%20%20", which is also incorrect. The correct URI would be
"%2520%20" (escaping the first "%" and " " characters).
The first issue is due to the currentOrNull() helper, which calls
trim() on the string and then compares the string to an empty string.
This means that a blob named " " will be parsed as "" and then
converted to NULL as the result of that method. Passing "null" as the
key then fails in a number of places (notably, appendPath()).
The second issue is due to the appendPath() method in the jclouds Uris
class. The issue here is that appendPath() calls urlDecode() and
passes the result to path(). The path() method, in turn, also calls
urlDecode(). After these transformations, a properly encoded blob of
the form %2520%20 turns into "%20 " and then " " (two spaces). After
these transformations the path is encoded again, resulting in "%20%20"
(which is wrong).
jclouds should not check if the string is encoded, but rather expect
that all strings would be encoded prior to transmission. As part of
that change, we must make sure that no code relies on such behavior
within jclouds. This commit adds a blobstore test to check encoding
pattern for blobs. It also removes the encoding check in the Strings2
class and the related test.
Dori Polotsky
Andrew Gaul
David Currie
Mathieu Tortuyaux
Ignasi Barrera
Jesse Glick
andreaturli
Tim Peierls
Chaithanya Ganta
Svetoslav Neykov
William L. Thomson Jr
Geoff Macartney
Zack Shoylev
Vijay Panghal
Duncan Grant
Iván Lomba
Rita Zhang
Alex Heneveld
Josef Cacek
Timur Alperovich
Oliver Gondža