This commit refactors the ImageCacheSupplier to act as a
proper cache. It is used by the ImageExtesion and all operations
on the images are propagated to the cache.
A method has also been added to the TemplateBuilder to let users
force a cache refresh. There have been several requests to provide a way
to disable image caching in the compute abstraction, and this new method
should fix that.
The patch implements a QueryValue class, which encodes the underlying
value based on whether the "encoded" flag is set. This class is used
by the RestAnnotationProcessor to propagate the @Encoded value set on
any parameters.
Since the encoding is now handled by the QueryValue instances, we
should no longer call encodeQueryLine() in the URI builder and instead
call buildQueryLine(). The caveat is that we need to make sure all of
the parameters that may need to be encoded are converted to QueryValue
objects. This is done by converting Object instances to QueryValue by
an instance of the TransformObjectToQueryValue when adding any query
parameters to the URI.
Adds support for the @Encoded option for the @QueryParam annotation.
The @Encoded params are not encoded, while all parameters that don't
have it are encoded. The change applies to the @QueryParam annotation
on a single parameter. There is no way to express @Encoded on the list
of parameters and their values in @QueryParams.
The big change is that query parameter encoding is now handled within
the annotation processor, as opposed to relying on the UriBuilder to
perform the encoding. This is required since the UriBuilder does not
have any information about additional annotations associated with each
of the query parameters.
Also, adds unit tests for making sure keys and values are properly
encoded when using the @QueryParams option.
Certain providers (e.g. Google Cloud Storage) place tokens that should
be encoded in the request path (e.g. GET
http://<host>/b/<bucket>/o/<object>) and expect them to be
percent-encoded. In the above example a GET request for "foo/bar"
should be translated to http://<host>/b/<bucket>/o/foo%2Fbar.
Currently, there is no way to express this in jclouds, as the entire
request path is encoded exactly once and there is no control over
whether a request parameter should be handled specially. In the
example above, "/" are not encoded in the path and the URL is
submitted as "http://<host>/b/<bucket>/o/foo/bar", which may be wrong.
This patch extends the annotation processor to support @Encoded for
the individual parameters of the request. However, this means that the
entire path is _NOT_ URL encoded. The caller *must* make sure that the
appropriate parameters are encoded -- ones that are marked with the
@Encoded annotation. Parameters not marked with the @Encoded
annotation are URI encoded prior to being added to the path. This
means that "/" characters will also be URI encoded in this case (i.e.
"foo/bar" is turned into "foo%2Fbar").
For the Google Storage provider, we will annotate the parameters that
are going to be pre-encoded (object names) and ensure the provider
encodes them prior to calling the API (separate patch in
jclouds-labs-google).
jclouds should not decode query strings that are passed to create HTTP
requests. This is problematic because in some cases a wrong request
may be generated. The most obvious example is if one passes the "+"
character. For example, the following query parameter: "users=me+you"
is stored by the URI builder as "me you" and subsequently appears in
the request as "users=me%20you", as opposed to "users=me%2Byou" (%2b
is percent encoding for "+").
This is not currently a problem because jclouds relies on the
isUrlEncoded() method to check if a query parameter should be decoded
and the situation above is avoided.
This PR attempts to suggest an alternative (and what I believe is
simpler) approach: on the path of crafting requests, jclouds should
only *encode*, not decode strings. Specifically, jclouds should
_never_ be in a situation where it relies on the isUrlEncoded()
method.
When constructing the query path, S3 does not properly handle encoded
paths. For example, if a blob named %20 is to be placed into the blob
store, S3 would end up placing blob named " " (what %20 represents).
This occurs because the S3 provider examines the URI's path portion
(which is presented in a decoded fasion to the caller). After
examining the path, it is not encoded again. Instead, we should call
getRawPath() to avoid this issue.
There are two issues on the decoding path:
1. Given a blob named " ", S3 API will throw a RuntimeException due to
a NULL check -- the key that it uses is NULL to represent the XML
content " " corresponding to the blob name.
2. Given a blob named "%20 ", S3 API will generate a URI for a blob
named "%20%20", which is also incorrect. The correct URI would be
"%2520%20" (escaping the first "%" and " " characters).
The first issue is due to the currentOrNull() helper, which calls
trim() on the string and then compares the string to an empty string.
This means that a blob named " " will be parsed as "" and then
converted to NULL as the result of that method. Passing "null" as the
key then fails in a number of places (notably, appendPath()).
The second issue is due to the appendPath() method in the jclouds Uris
class. The issue here is that appendPath() calls urlDecode() and
passes the result to path(). The path() method, in turn, also calls
urlDecode(). After these transformations, a properly encoded blob of
the form %2520%20 turns into "%20 " and then " " (two spaces). After
these transformations the path is encoded again, resulting in "%20%20"
(which is wrong).
jclouds should not check if the string is encoded, but rather expect
that all strings would be encoded prior to transmission. As part of
that change, we must make sure that no code relies on such behavior
within jclouds. This commit adds a blobstore test to check encoding
pattern for blobs. It also removes the encoding check in the Strings2
class and the related test.
refactor AccountApiExpectedTest to AccountApiMockTest
refactor DatacenterApiExpectTest to DatacenterApiMockTest
refactor SoftwareDescriptionApiExpectTest to SoftwareDescriptionApiMockTest
refactor VirtualGuestApiExpectTest to VirtualGuestApiMockTest
refactor VirtualGuestBlockDeviceTemplateGroupApiExpectTest to VirtualGuestBlockDeviceTemplateGroupApiMockTest
ideally we shouldn't need this function and instead never double
encode strings, but auditing for that is beyond what I have time
for. currently, putBlob(" ") and putBlob("%20") behave the same
way which is arguably incorrect
If an IOException is thrown during the execution of an HttpCommand retry only if the HTTP method is idempotent (i.e. GET, DELETE, PUT). Otherwise the retry could cause unwanted side effects (i.e. creating and leaking multiple new nodes).
* Improves extension namespace configuration to use Guice multibindings
so each provider can cleanly provide their own namespaces.
* Fixes the HPCloud Compute volume attachment namespace and adds the
corresponding live tests.
* Fixes the Rackspace CloudServers UK volume attachment namespace.
- If get timeout of 0 (or negative), then still try once.
- Remove (unlikely) race in retry’s apply(T) where context-switching
delays could cause `before(end)` to return false the first time, even
though the timeout was positive.
- Ensure retries at end of the timeout (e.g. if timeout is 30 secs
and last sleep takes us up to the 30 secs mark, then test again
rather than returning immediately after the sleep!)
- Use `long` for time, rather than `java.util.Date`, for internal
calculations. Deprecates old protected methods that use Date.
ByteSourcePayloadIterator avoids buffering the entire slice like
InputStreamPayloadIterator does. Also rename PayloadIterator to
better reflect its intent and reduce its visibility.
Prefer ByteSourcePayload which offers a superset of its functionality.
Note that ByteArrayPayload implicitly set the contentLength while
users of ByteSourcePayload must do so explicitly.
Some providers, notably Azure, include a byte-order mark in their XML
responses. ParseSax.apply buffers these responses in a String when
users enable trace-level logging to include the response in any thrown
exceptions. InputSource(InputStream) skips these byte-order marks
while InputSource(Reader) does not, yielding a SAXParseException.
This method is dangerous since all ByteSource should provide a new
InputStream on every call to openStream while the method returns the
same InputStream for non-repeatable Payloads.
This avoids unneeded garbage, especially during XML parsing. Replaced
with:
find -name \*.java | xargs sed -i 's/^\( *[^ ]*\) = new StringBuilder();$/\1.setLength(0);/'
S3 compatible blobStores sometimes return date in the format:
"2014-07-23T20:53:17+0000" instead of the more common
"2014-07-23T18:09:39.944Z". This caused jclouds to barf with an
IllegalArgumentException.
This commit tries to parse both the formats for S3. The exception
is thrown if both fail.
Added unit tests for the same.
This commit replaces file resource-based test inputs with in-memory
equivalents. This is more consistent and efficient than the previous
approach. Also resized some test inputs to be partSize + 1 instead of
2 * partSize. Tested against aws-s3, blobstore, core, cloudfiles-us,
and filesystem.
InputStream.read(byte[]) can return fewer bytes than requested.
Specifically ByteSource.concat(ByteSource...).openStream() will only
return as many bytes as the current ByteSource contains. Thus
ByteSources.repeatingArrayByteSource(byte[]).openStream() will return
short reads despite the byte[] input from its single logical
InputStream.
Callers should instead explicitly set contentMD5, usually with the
results from Guava Hashing.md5(). This narrows the API and removes a
strange IOException from callers. Further it removes a dangerous
rebuffering of arbitrarily-large non-repeatable Payloads.
Rita Zhang
Alex Heneveld
Andrew Gaul
Ignasi Barrera
Josef Cacek
Zack Shoylev
Timur Alperovich
Oliver Gondža
Arvind Nadendla
Andrea Turli
Ka-Hing Cheung
Svetoslav Neykov
Aled Sage
Jeremy Daggett
Adrian Cole
Adrian Cole
Andrew Phillips
Shri Javadekar
Tom Manville
Christopher Dancy