The patch implements a QueryValue class, which encodes the underlying
value based on whether the "encoded" flag is set. This class is used
by the RestAnnotationProcessor to propagate the @Encoded value set on
any parameters.
Since the encoding is now handled by the QueryValue instances, we
should no longer call encodeQueryLine() in the URI builder and instead
call buildQueryLine(). The caveat is that we need to make sure all of
the parameters that may need to be encoded are converted to QueryValue
objects. This is done by converting Object instances to QueryValue by
an instance of the TransformObjectToQueryValue when adding any query
parameters to the URI.
On SUSE, the “-f” force option is not available for groupadd,
so `groupadd -f wheel` returns exit code 9 if the group already
exists. To avoid this, first check if the group exists.
In normal usage, this doesn’t matter: the script continues with the
next command anyway.
However, if the statements generated by UserAdd or AdminAccess are
used outside of that context (e.g. by code external to jclouds), then
this can cause them to fail.
Adds support for the @Encoded option for the @QueryParam annotation.
The @Encoded params are not encoded, while all parameters that don't
have it are encoded. The change applies to the @QueryParam annotation
on a single parameter. There is no way to express @Encoded on the list
of parameters and their values in @QueryParams.
The big change is that query parameter encoding is now handled within
the annotation processor, as opposed to relying on the UriBuilder to
perform the encoding. This is required since the UriBuilder does not
have any information about additional annotations associated with each
of the query parameters.
Also, adds unit tests for making sure keys and values are properly
encoded when using the @QueryParams option.
Certain providers (e.g. Google Cloud Storage) place tokens that should
be encoded in the request path (e.g. GET
http://<host>/b/<bucket>/o/<object>) and expect them to be
percent-encoded. In the above example a GET request for "foo/bar"
should be translated to http://<host>/b/<bucket>/o/foo%2Fbar.
Currently, there is no way to express this in jclouds, as the entire
request path is encoded exactly once and there is no control over
whether a request parameter should be handled specially. In the
example above, "/" are not encoded in the path and the URL is
submitted as "http://<host>/b/<bucket>/o/foo/bar", which may be wrong.
This patch extends the annotation processor to support @Encoded for
the individual parameters of the request. However, this means that the
entire path is _NOT_ URL encoded. The caller *must* make sure that the
appropriate parameters are encoded -- ones that are marked with the
@Encoded annotation. Parameters not marked with the @Encoded
annotation are URI encoded prior to being added to the path. This
means that "/" characters will also be URI encoded in this case (i.e.
"foo/bar" is turned into "foo%2Fbar").
For the Google Storage provider, we will annotate the parameters that
are going to be pre-encoded (object names) and ensure the provider
encodes them prior to calling the API (separate patch in
jclouds-labs-google).
Since commit 56e687f497, Linux line endings (LF) are enforced. But on
Windows, a common practice is to set core.autocrlf to 'auto', wich mean
that the local copy of the file has Windows line endings, whereas the
remote copy has Linux line endings (cf. https://help.github.com/articles/dealing-with-line-endings/#platform-windows).
With core.autoclrf=auto, Checkstyle will throw an error because local
files will have Windows line endings.
This setting will set Linux line endings for all text files, except
.cmd files.
AWS-S3 provider should not pre-encode parameters that are passed to
the jclouds request. This comes up in the AWSS3BlobRequestSigner as
the only place where a parameter is pre-encoded.
jclouds should not decode query strings that are passed to create HTTP
requests. This is problematic because in some cases a wrong request
may be generated. The most obvious example is if one passes the "+"
character. For example, the following query parameter: "users=me+you"
is stored by the URI builder as "me you" and subsequently appears in
the request as "users=me%20you", as opposed to "users=me%2Byou" (%2b
is percent encoding for "+").
This is not currently a problem because jclouds relies on the
isUrlEncoded() method to check if a query parameter should be decoded
and the situation above is avoided.
This PR attempts to suggest an alternative (and what I believe is
simpler) approach: on the path of crafting requests, jclouds should
only *encode*, not decode strings. Specifically, jclouds should
_never_ be in a situation where it relies on the isUrlEncoded()
method.
On Windows, we need to avoid trailing spaces, as the test fails to
create the required blob. Specifically, we should not test blobs named
" " and "%20 ".
When constructing the expected HTTP requests in STS tests, STS API
should not pre-encode the strings. The form parameters are already
encoded and are otherwise encoded twice.
When constructing a URI path for a blob in the listing results, the
Azure provider should re-encode any "bad" characters. Azure provider
also should not call trim() on the blob name, as that will remove any
leading or trailing space characters (which the provider supports). In
fact, the trim() call is only required because of the fact that when
parsing the XML document in the Azure tests, the white space between
the tags was also included in the values. Changing when the character
array is reset eliminates the need to do so.
When constructing the query path, S3 does not properly handle encoded
paths. For example, if a blob named %20 is to be placed into the blob
store, S3 would end up placing blob named " " (what %20 represents).
This occurs because the S3 provider examines the URI's path portion
(which is presented in a decoded fasion to the caller). After
examining the path, it is not encoded again. Instead, we should call
getRawPath() to avoid this issue.
There are two issues on the decoding path:
1. Given a blob named " ", S3 API will throw a RuntimeException due to
a NULL check -- the key that it uses is NULL to represent the XML
content " " corresponding to the blob name.
2. Given a blob named "%20 ", S3 API will generate a URI for a blob
named "%20%20", which is also incorrect. The correct URI would be
"%2520%20" (escaping the first "%" and " " characters).
The first issue is due to the currentOrNull() helper, which calls
trim() on the string and then compares the string to an empty string.
This means that a blob named " " will be parsed as "" and then
converted to NULL as the result of that method. Passing "null" as the
key then fails in a number of places (notably, appendPath()).
The second issue is due to the appendPath() method in the jclouds Uris
class. The issue here is that appendPath() calls urlDecode() and
passes the result to path(). The path() method, in turn, also calls
urlDecode(). After these transformations, a properly encoded blob of
the form %2520%20 turns into "%20 " and then " " (two spaces). After
these transformations the path is encoded again, resulting in "%20%20"
(which is wrong).
jclouds should not check if the string is encoded, but rather expect
that all strings would be encoded prior to transmission. As part of
that change, we must make sure that no code relies on such behavior
within jclouds. This commit adds a blobstore test to check encoding
pattern for blobs. It also removes the encoding check in the Strings2
class and the related test.
- Adding env_reset to the default configuration in /etc/sudoers
- Adding secure_path to the default configuration in /etc/sudoers
- secure_path value is
"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Andrew Gaul
Ignasi Barrera
Timur Alperovich
Nikolay Sokolov
Aled Sage
Stuart Hendren
John McDonnell
olivierlemasle
Andrew Kennedy
Michael Wilson
Irmo Manie
Oliver Gondža
Yavor Yanchev