Commit Graph

297 Commits

Author SHA1 Message Date
David Pilato 0bbcd94827 Merge branch 'master' into hlclient/add-delete-method
# Conflicts:
#	client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java
#	client/rest-high-level/src/test/java/org/elasticsearch/client/CrudIT.java
2017-02-15 16:08:29 +01:00
David Pilato 7df1df24e8 Add delete API to the High Level Rest Client 2017-02-15 16:00:50 +01:00
Tanguy Leroux 86993ad759 Add Index API to High Level Rest Client ()
This commit adds support for the Index API in the High Level Rest Client.
2017-02-15 13:53:41 +01:00
Simon Willnauer ecb01c15b9 Fold InternalSearchHits and friends into their interfaces ()
We have a bunch of interfaces that have only a single implementation
for 6 years now. These interfaces are pretty useless from a SW development
perspective and only add unnecessary abstractions. They also require
lots of casting in many places where we expect that there is only one
concrete implementation. This change removes the interfaces, makes
all of the classes final and removes the duplicate `foo` `getFoo` accessors
in favor of `getFoo` from these classes.
2017-02-08 14:40:08 +01:00
Luca Cavanna 3551106aa7 Add get/exists method to RestHighLevelClient ()
This commit adds support for get and exists api to the high level Java Rest Client. It also adds the infrastructure for other methods to be added in the future.
2017-02-04 14:54:26 +01:00
Tim Brooks f70188ac58 Remove connect SocketPermissions from core ()
This is related to . Core no longer needs `SocketPermission`
`connect`.

This permission is relegated to these modules/plugins:
- transport-netty4 module
- reindex module
- repository-url module
- discovery-azure-classic plugin
- discovery-ec2 plugin
- discovery-gce plugin
- repository-azure plugin
- repository-gcs plugin
- repository-hdfs plugin
- repository-s3 plugin

And for tests:
- mocksocket jar
- rest client
- httpcore-nio jar
- httpasyncclient jar
2017-02-03 09:39:56 -06:00
Jason Tedor 9a0b216c36 Upgrade checkstyle to version 7.5
This commit upgrades the checkstyle configuration from version 5.9 to
version 7.5, the latest version as of today. The main enhancement
obtained via this upgrade is better detection of redundant modifiers.

Relates 
2017-02-03 09:46:44 -05:00
Jay Modi 7520a107be Optionally require a valid content type for all rest requests with content ()
This change adds a strict mode for xcontent parsing on the rest layer. The strict mode will be off by default for 5.x and in a separate commit will be enabled by default for 6.0. The strict mode, which can be enabled by setting `http.content_type.required: true` in 5.x, will require that all incoming rest requests have a valid and supported content type header before the request is dispatched. In the non-strict mode, the Content-Type header will be inspected and if it is not present or not valid, we will continue with auto detection of content like we have done previously.

The content type header is parsed to the matching XContentType value with the only exception being for plain text requests. This value is then passed on with the content bytes so that we can reduce the number of places where we need to auto-detect the content type.

As part of this, many transport requests and builders were updated to provide methods that
accepted the XContentType along with the bytes and the methods that would rely on auto-detection have been deprecated.

In the non-strict mode, deprecation warnings are issued whenever a request with body doesn't provide the Content-Type header.

See 
2017-02-02 14:07:13 -05:00
Chris Earle f0f75b187a Support Preemptive Authentication with RestClient ()
This adds the necessary `AuthCache` needed to support preemptive authorization. By adding every host to the cache, the automatically added `RequestAuthCache` interceptor will add credentials on the first pass rather than waiting to do it after _each_ anonymous request is rejected (thus always sending everything twice when basic auth is required).
2017-01-24 11:34:05 -05:00
Nik Everett 6265ef1c1b Deguice rest handlers ()
There are presently 7 ctor args used in any rest handlers:
* `Settings`: Every handler uses it to initialize a logger and
  some other strange things.
* `RestController`: Every handler registers itself with it.
* `ClusterSettings`: Used by `RestClusterGetSettingsAction` to
  render the default values for cluster settings.
* `IndexScopedSettings`: Used by `RestGetSettingsAction` to get
  the default values for index settings.
* `SettingsFilter`: Used by a few handlers to filter returned
  settings so we don't expose stuff like passwords.
* `IndexNameExpressionResolver`: Used by `_cat/indices` to
  filter the list of indices.
* `Supplier<DiscoveryNodes>`: Used to fill enrich the response
  by handlers that list tasks.

We probably want to reduce these arguments over time but
switching construction away from guice gives us tighter
control over the list of available arguments.

These parameters are passed to plugins using
`ActionPlugin#initRestHandlers` which is expected to build and
return that handlers immediately. This felt simpler than
returning an reference to the ctors given all the different
possible args.

Breaks java plugins by moving rest handlers off of guice.
2017-01-20 11:48:51 -05:00
Luca Cavanna 193111919c move ignore parameter support from yaml test client to low level rest client ()
All the language clients support a special ignore parameter that doesn't get passed to elasticsearch with the request, but used to indicate which error code should not lead to an exception if returned for a specific request.

Moving this to the low level REST client will allow the high level REST client to make use of it too, for instance so that it doesn't have to intercept ResponseExceptions when the get api returns a 404.
2017-01-16 18:54:44 +01:00
Tim Brooks 7a8884d9fa Wrap rest httpclient with doPrivileged blocks ()
This is related to . A number of modules (reindex, etc) use the
rest client. The rest client opens connections using the apache http
client. To avoid throwing SecurityException when using the
SecurityManager these operations must be privileged. This is tricky
because connections are opened within the httpclient code on its
reactor thread. The way I confronted this was to wrap the creation
of the client (and creation of reactor thread) in a doPrivileged
block. The new thread inherits the existing security context.
2017-01-16 09:17:44 -06:00
Tanguy Leroux 3a3ce61186 Update Jackson to 2.8.6 ()
closes 
2017-01-13 09:05:48 +01:00
Jason Tedor 126efea56c Upgrade to Netty 4.1.7
This commit upgrades the Netty dependency to version 4.1.7.Final,
picking up some important bug fixes.

Relates 
2017-01-12 10:58:21 -05:00
javanna ded694fc83 Make StatusToXContent extend ToXContentObject and rename it to StatusToXContentObject
This also allows to make RestToXContentListener require ToXContentObject rather than ToXContent
2017-01-06 23:31:48 +01:00
javanna 0a6827a5cc Make RestHighLevelClient non final 2017-01-05 11:29:41 +01:00
javanna f0181b19f5 add REST high level client gradle submodule and first simple method
The RestHighLevelClient class takes as as an argument a low level client instance RestClient. The first method added is ping, which returns true if the call to HEAD / went ok and false if an IOException was thrown. Any other exception gets bubbled up.

There are two kinds of tests, a unit test (RestHighLevelClientTests) that verifies the interaction between high level and low level client, and an integration test (MainActionIT) which relies on an externally started es cluster to send requests to.
2017-01-05 10:55:47 +01:00
javanna 1899aea9ca [TEST] move randomHeaders method from RestClientTestCase to RestClientTestUtil and simplify headers assertions 2017-01-05 10:55:47 +01:00
Tim B be22a250b6 Replace Socket, ServerSocket, and HttpServer usages in tests with mocksocket versions ()
This integrates the mocksocket jar with elasticsearch tests. Mocksocket wraps actions requiring SocketPermissions in doPrivilege blocks. This will eventually allow SocketPermissions to be assigned to the mocksocket jar opposed to the entire elasticsearch codebase.
2017-01-04 14:38:51 -06:00
Jason Tedor 5185a9734d Add comment clarifying property setting in client
This commit adds a comment clarifying why we do not catch a security
exception in the pre-built transport client.
2016-12-23 18:14:36 -05:00
Jason Tedor 31bf279ec7 Tell Netty not to be unsafe in transport client
Today we ship with default jvm.options for server Elasticsearch that
prevents Netty from using some unsafe optimizations. Yet, the settings
do nothing for the transport client since it is embedded in other
applications that will not read and use those settings. This commit adds
these settings for the transport client, and is done so in a way that
still enables users to go unsafe if they want to go unsafe (they
shouldn't, but the option is there).

Relates 
2016-12-21 10:49:33 -05:00
Nik Everett 2aa89820f3 Don't use null charset in RequestLogger ()
If the response comes back with a content type with a `null`
charset we were blindly using it, causing `NullPointerException`s.

Closes 
2016-12-15 11:19:51 -05:00
javanna 478cd78047 Remove unused import from RequestLogger
Not only was StringJoiner unused, it's also a class only available in java 1.8, which is a problem given that the REST client has minimum java required set to 1.7
2016-12-13 15:01:51 +01:00
Luca Cavanna 9be778c5e5 Warn log deprecation warnings received from server ()
The warnings get printed out in a single line e.g. WARNING: request [DELETE http://localhost:9200/index/type/_api] returned 3 warnings:[this is warning number 0],[this is warning number 1],[this is warning number 2]
2016-12-12 12:11:42 +01:00
Nik Everett fc2060ba7e Don't close rest client from its callback ()
If you try to close the rest client inside one of its callbacks then
it blocks itself. The thread pool switches the status to one that
requests a shutdown and then waits for the pool to shutdown. When
another thread attempts to honor the shutdown request it waits
for all the threads in the pool to finish what they are working on.
Thus thread a is waiting on thread b while thread b is waiting
on thread a. It isn't quite that simple, but it is close.

Relates to 
2016-12-09 10:39:51 -05:00
Nik Everett 2087234d74 Timeout improvements for rest client and reindex ()
Changes the default socket and connection timeouts for the rest
client from 10 seconds to the more generous 30 seconds.

Defaults reindex-from-remote to those timeouts and make the
timeouts configurable like so:
```
POST _reindex
{
  "source": {
    "remote": {
      "host": "http://otherhost:9200",
      "socket_timeout": "1m",
      "connect_timeout": "10s"
    },
    "index": "source",
    "query": {
      "match": {
        "test": "data"
      }
    }
  },
  "dest": {
    "index": "dest"
  }
}
```

Closes 
2016-12-05 10:54:51 -05:00
Luca Cavanna 5b8bdba12e Remove subrequests method from CompositeIndicesRequest () 2016-11-30 15:03:58 +01:00
Simon Willnauer a9a2753f0b Add a HostFailureListener to notify client code if a node got disconnected ()
Today there is no way to get notified if a node is disconnected. Client code
must poll the TransportClient constantly to detect that a node is not connected
anymore in order to react and add new nodes or notify altering etc. For instance
if a hostname  gets resolved to an IP but that host is disconnected clients want
to reconnect by resolving the hostname again which is a common situation in cloud
environments.

Closes 
2016-11-22 20:46:28 +01:00
Simon Willnauer de04aad994 Remove `modules/transport_netty_3` in favor of `netty_4` ()
We kept `netty_3` as a fallback in the 5.x series but now that master
is 6.0 we don't need this or in other words all issues coming up with
netty 4 will be blockers for 6.0.
2016-11-17 12:44:42 +01:00
Ryan Ernst d14c470b89 Remove generics from ActionRequest
closes 
2016-11-14 15:32:01 -08:00
Luca Cavanna 293a3cab01 Rest client: don't reuse that same HttpAsyncResponseConsumer across multiple retries ()
* Rest client: don't reuse that same HttpAsyncResponseConsumer across multiple retries

Turns out that AbstractAsyncResponseConsumer from apache async http client is stateful and cannot be reused across multiple requests. The failover mechanism was mistakenly reusing that same instance, which can be provided by users, across retries in case nodes are down or return 5xx errors. The downside is that we have to change the signature of two public methods, as HttpAsyncResponseConsumer cannot be provided directly anymore, rather its factory needs to be provided which is going to be used to create one instance of the consumer per request attempt.

Up until now we tested our RestClient against multiple nodes only in a mock environment, where we don't really send http requests. In that scenario we can verify that retries etc. work properly but the interaction with the http client library in a real scenario is different and can catch other problems. With this commit we also add an integration test that sends requests to multiple hosts, and some of them may also get stopped meanwhile. The specific test for pathPrefix was also removed as pathPrefix is now randomly applied by default, hence implicitly tested. Moved also a small test method that checked the validity of the path argument to the unit test RestClientSingleHostTests.

Also increase default buffer limit to 100MB and make it required in default consumer

The default buffer limit used to be 10MB but that proved not to be high enough for scroll requests (see reindex from remote). With this commit we increase the limit to 100MB and make it a bit more visibile in the consumer factory.
2016-11-08 16:42:42 +01:00
Yuhao Bi 79090431af Provide error message when request path is null 2016-11-02 09:32:15 +01:00
Nik Everett a612e5988e Bump reindex-from-remote's buffer to 200mb
It was 10mb and that was causing trouble when folks reindex-from-remoted
with large documents.

We also improve the error reporting so it tells folks to use a smaller
batch size if they hit a buffer size exception. Finally, adds some docs
to reindex-from-remote mentioning the buffer and giving an example of
lowering the size.

Closes 
2016-11-01 13:19:28 -04:00
Adrien Grand b3cc54cf0d Upgrade to lucene-6.3.0-snapshot-ed102d6 ()
Lucene 6.3 is expected to be released in the next weeks so it'd be good to give
it some integration testing. I had to upgrade randomized-testing too so that
both Lucene and Elasticsearch are on the same version.
2016-10-28 14:47:15 +02:00
Areek Zillur 481f7909ae Merge branch 'master' into cleanup/transport_bulk 2016-10-11 16:04:47 -04:00
Areek Zillur 0e8b6532ec rename DocumentRequest to DocWriteRequest 2016-10-11 16:00:10 -04:00
Areek Zillur 396f80c963 Revert "rename DocumentRequest to DocumentWriteRequest"
This reverts commit b5079ce009.
2016-10-07 17:50:07 -04:00
Simon Willnauer 194a6b1df0 Remove LocalTransport in favor of MockTcpTransport ()
This change proposes the removal of all non-tcp transport implementations. The
mock transport can be used by default to run tests instead of local transport that has
roughly the same performance compared to TCP or at least not noticeably slower.

This is a master only change, deprecation notice in 5.x will be committed as a
separate change.
2016-10-07 11:27:47 +02:00
Areek Zillur b5079ce009 rename DocumentRequest to DocumentWriteRequest 2016-10-06 05:05:59 -04:00
Areek Zillur bd4a03a426 Merge branch 'master' into cleanup/transport_bulk 2016-10-04 14:06:17 -04:00
Jason Tedor 51d53791fe Remove lenient URL parameter parsing
Today when parsing a request, Elasticsearch silently ignores incorrect
(including parameters with typos) or unused parameters. This is bad as
it leads to requests having unintended behavior (e.g., if a user hits
the _analyze API and misspell the "tokenizer" then Elasticsearch will
just use the standard analyzer, completely against intentions).

This commit removes lenient URL parameter parsing. The strategy is
simple: when a request is handled and a parameter is touched, we mark it
as such. Before the request is actually executed, we check to ensure
that all parameters have been consumed. If there are remaining
parameters yet to be consumed, we fail the request with a list of the
unconsumed parameters. An exception has to be made for parameters that
format the response (as opposed to controlling the request); for this
case, handlers are able to provide a list of parameters that should be
excluded from tripping the unconsumed parameters check because those
parameters will be used in formatting the response.

Additionally, some inconsistencies between the parameters in the code
and in the docs are corrected.

Relates 
2016-10-04 12:45:29 -04:00
Areek Zillur 248ac240ed Merge branch 'master' into cleanup/transport_bulk 2016-10-03 16:12:11 -04:00
Daniel Mitterdorfer 7a1ad716fa Update client benchmarks to log4j2 2016-09-23 15:39:18 +02:00
Ryan Ernst 85b8f29415 Build: Remove old maven deploy support ()
* Build: Remove old maven deploy support

This change removes the old maven deploy that we have in parallel to
maven-publish, and makes maven-publish fully work with publishing to
maven local. Using `gradle publishToMavenLocal` should be used to
publish to .m2.

Note that there is an unfortunate hack that means for
zip artifacts we must first create/publish a dummy pom file, and then
follow that with the real pom file. It would be nice to have the pom
file contains packaging=zip, but maven central then requires sources and
javadocs. But our zips are really just attached artifacts, so we already
set the packaging type to pom for our zip files. This change just works
around a limitation of the underlying maven publishing library which
silently skips attached artifacts when the packaging type is set to pom.

relates 
closes 

* Remove unnecessary extra spacing
2016-09-19 15:10:41 -07:00
Jim Ferenczi 1764ec56b3 Fixed naming inconsistency for fields/stored_fields in the APIs ()
This change replaces the fields parameter with stored_fields when it makes sense.
This is dictated by the renaming we made in  for the search API.

The following list of endpoint has been changed to use `stored_fields` instead of `fields`:
* get
* mget
* explain

The documentation and the rest API spec has been updated to cope with the changes for the following APIs:
* delete_by_query
* get
* mget
* explain

The `fields` parameter has been deprecated for the following APIs (it is replaced by _source filtering):
* update: the fields are extracted from the _source directly.
* bulk: the fields parameter is used but fields are extracted from the source directly so it is allowed to have non-stored fields.

Some APIs still have the `fields` parameter for various reasons:
* cat.fielddata: the fields paramaters relates to the fielddata fields that should be printed.
* indices.clear_cache: used to indicate which fielddata fields should be cleared.
* indices.get_field_mapping: used to filter fields in the mapping.
* indices.stats: get stats on fields (stored or not stored).
* termvectors: fields are retrieved from the stored fields if possible and extracted from the _source otherwise.
* mtermvectors:
* nodes.stats: the fields parameter is used to concatenate completion_fields and fielddata_fields so it's not related to stored_fields at all.

Fixes 
2016-09-13 20:54:41 +02:00
Jason Tedor e166459bbe Merge branch 'master' into log4j2
* master:
  Increase visibility of deprecation logger
  Skip transport client plugin installed on JDK 9
  Explicitly disable Netty key set replacement
  percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
  Make it possible for Ingest Processors to access AnalysisRegistry
  Allow RestClient to send array-based headers
  Silence rest util tests until the bogusness can be simplified
  Remove unknown HttpContext-based test as it fails unpredictably on different JVMs
  Tests: Improve rest suite names and generated test names for docs tests
  Add support for a RestClient base path
2016-08-31 10:59:27 -04:00
Jason Tedor 6f8a047942 Skip transport client plugin installed on JDK 9
This commit adds an assumption to
PreBuiltTransportClientTests#testPluginInstalled on JDK 9. The
underlying issue is that Netty attempts to access sun.nio.ch but this
package is not exported from java.base on JDK 9. This throws an uncaught
InaccessibleObjectException causing the test to fail. This assumption
can be removed when Netty 4.1.6 is released as it will include a fix for
this scenario.

Relates 
2016-08-31 09:46:27 -04:00
Chris Earle b8f4c92d41 Allow RestClient to send array-based headers
This enables the RestClient to send array-based (multi-valued) header values, rather than only sending whatever happened to be the _last_ value of the header.
2016-08-30 18:02:07 -04:00
Chris Earle c05d5f9257 Remove unknown HttpContext-based test as it fails unpredictably on different JVMs 2016-08-30 17:18:01 -04:00
Chris Earle 335c020cd7 Add support for a RestClient base path
This enables simple support for proxies (beyond proxy host and proxy port, which is done via the RequestConfig)) to provide a base path in front of all requests performed by the RestClient.
2016-08-30 13:46:45 -04:00
Jason Tedor 7da0cdec42 Introduce Log4j 2
This commit introduces Log4j 2 to the stack.
2016-08-30 13:31:24 -04:00
Chris Earle bd0b06440e Add "Async" to the end of each Async RestClient method
This makes it much harder to accidentally miss the Response.
2016-08-26 10:51:33 -04:00
Daniel Mitterdorfer 4460998ff8 Remove obsolete NoopSearchRequestBuilder#setNoStoredFields() 2016-08-26 09:58:53 +02:00
Daniel Mitterdorfer 7b81c4ca59 Add client-benchmark-noop-api-plugin to stress clients even more in benchmarks () 2016-08-26 09:05:47 +02:00
Chris Earle e171d0e0a8 Un-final Core REST Client classes
This removes final from the RestClient, Response, and Sniffer classes so that outside code can mock them. Their constructors are already package private, so there's not much that can go wrong.
2016-08-25 16:02:04 -04:00
Jason Tedor a62740bbd2 Avoid early initializing Netty
Today when we load the Netty plugins, we indirectly cause several Netty
classes to initialize. This is because we attempt to load some classes
by name, and loading these classes is done in a way that triggers a long
chain of class initializers within Netty. We should not do this, this
can lead to log messages before the logger is loader, and it leads to
initialization in cases when the classes would never be needed (for
example, Netty 3 class initialization is never needed if Netty 4 is
used, and vice versa). This commit avoids this early initialization of
these classes by removing the need for the early loading.

Relates 
2016-08-05 14:58:33 -04:00
Tanguy Leroux 841d5a210e Update to Jackson 2.8.1
This commit updates Jackson to the 2.8.1 version, which is more strict when it comes to build objects. It also adds the snakeyaml dependency that was previously shaded in jackson libs.

It also closes 
2016-08-05 12:26:06 +02:00
Colin Goodheart-Smithe f1110f6f2a fix import statements 2016-08-04 16:45:05 +01:00
Colin Goodheart-Smithe f273981f37 Added failure message to test 2016-08-04 16:28:12 +01:00
Jason Tedor 785624e96b Restore interrupted status on when closing client
When closing a transport client that depends on Netty 4, interrupted
exceptions can be thrown while shutting down some Netty threads. This
commit refactors the handling of these exceptions to finish shutting
down and then just restore the interrupted status.
2016-08-04 11:22:13 -04:00
Jason Tedor 2936810c63 Setting exists equals false instead of not exists
This commit rewrites a boolean expression to check for equality with
false instead of negating the existence check.
2016-08-04 10:03:08 -04:00
Jason Tedor c08557d033 Wait for Netty 4 threads to terminate on close
Today if the PreBuiltTransportClient is using Netty 4 transport, on
shutdown some Netty 4 threads could linger. This commit causes the
client to wait for these threads to shutdown upon termination.
2016-08-04 09:48:38 -04:00
Colin Goodheart-Smithe b0730bb214 Fix PreBuiltTransportClientTests to run and pass
This change does three things:

1. Makes PreBuiltTransportClientTests run since it was silently
failing on a missing dependency
2. Makes PreBuiltTransportClientTests pass
3. Removes the http.type and transport.type from being set in the
transport clients additional settings since these are set to `netty4` by
default anyway.
2016-08-04 14:15:28 +01:00
Jason Tedor 227463c356 Default transport and HTTP to Netty 4
This commit sets the default transport and HTTP implementation to use
transport-netty4.

Relates 
2016-08-02 12:19:19 -04:00
Daniel Mitterdorfer b99a482992 Improve client benchmark ()
* Allow to run client benchmark as an uberjar
* Busy wait to avoid accidental skew on low target throughput rates
* Trigger and wait for full GC to happen between trials
* Add missing SuppressForbidden to allow System.gc in client benchmark
2016-08-02 14:17:51 +02:00
Ryan Ernst 05edb0367d Add client jars pom generation, and make transport client depend on
transport client plugin jars
2016-07-29 07:41:39 -07:00
javanna 1ea8f865d6 Add response body to ResponseException error message
Consuming the response body to make it part of the exception message means that it may not be readable anymore later, depending on whether the entity is repeatable or not. Turns out that the response body tells a lot about the error itself, and considering that we don't expect bodies to be incredibly big for errors, we can wrap the entity into a BufferedHttpEntity to make it repeatable.

Closes 
2016-07-28 14:44:20 +02:00
Luca Cavanna 8a51cfb5b3 Simplify Sniffer initialization and automatically create the default HostsSniffer ()
Simplify Sniffer initialization and automatically create the default HostsSniffer

Take Sniffer.Builder out to its own top level class. Remove HostsSniffer.Builder and let SnifferBuilder create the default HostsSniffer. This simplifies the Sniffer initialization as the HostsSniffer is not mandatory anymore. It can still be specified though in case the configuration needs to be changed or a different impl has to be used. Also make HostsSniffer an interface.
2016-07-26 17:28:37 +02:00
Daniel Mitterdorfer c33f85bc37 Add client benchmark
With this commit we add a benchmark for the new REST client and the
existing transport client.

Closes 
2016-07-26 11:01:22 +02:00
Jason Tedor 2d1b0587dd Introduce Netty 4
This commit adds transport-netty4, a transport and HTTP implementation
based on Netty 4.

Relates 
2016-07-22 22:26:35 -04:00
javanna 061ea1bd8c [TEST] move assertions outside of listener in testAsyncRequests for clearer test failures 2016-07-22 22:19:59 +02:00
javanna c9c7af791c update nextHost method javadocs 2016-07-22 21:42:42 +02:00
javanna 46cb3f36ff fix concurrency bug when getting the host for a given request
It can happen that the list of healthy hosts is empty, then we get one from the blacklist. but some other operation might have sneaked in and emptied the blacklist in the meantime, so we have to retry till we manage to get some host, either from the healthy list or from the blacklist.
2016-07-22 21:35:43 +02:00
javanna e6054a931e add async request unit test 2016-07-22 19:01:56 +02:00
javanna 4e8ee1f0ab add some javadocs to clarify internal listeners behaviour 2016-07-22 17:44:50 +02:00
javanna 835d8cecdc [TEST] add unit tests for internal TrackingFailureListener
Any provided listener will always be wrapped into FailureTrackingListener to handle retries
2016-07-22 17:31:11 +02:00
javanna a6a685b0f6 make Response class final 2016-07-22 16:36:14 +02:00
javanna a579866b42 rename mayRetry to isRetryStatus 2016-07-22 15:49:39 +02:00
javanna 37e075a506 Make SyncResponseListener safer
Throw explicit IllegalStateException in unexpected situations, like where both response and exception are set, or when both are unset. Add unit test for SyncResponseListener.
2016-07-22 15:48:15 +02:00
javanna 175c327e17 validate bufferLimit is positive in HeapBufferedAsyncResponseConsumer 2016-07-21 13:59:28 +02:00
javanna 59ccc88c73 rename mustRetry method to mayRetry 2016-07-20 16:24:57 +02:00
javanna fccfe7dcb8 RestClient javadocs adjustments 2016-07-20 16:24:29 +02:00
javanna 569d7b3ecc notify the listener if the request gets cancelled 2016-07-20 15:24:14 +02:00
javanna a9b5c5adbe restore throws IOException clause on all performRequest sync methods
We throw IOException, which is the exception that is going to be thrown in 99% of the cases. A more generic exception can happen, and if it is a runtime one we just let it bubble up as is, otherwise we wrap it into runtime one so that we don't require to catch Exception everywhere, which seems odd.

Also adjusted javadocs for all performRequest methods
2016-07-19 15:18:05 +02:00
javanna 8eccdff9ad add HeapBufferedAsyncResponseConsumer with configurable max buffer size
We keep the default async client behaviour like in BasicAsyncResponseConsumer, but we lower the maximum size of the buffer from Integer.MAX_VALUE (2GB) to 10 MB. This way users will realize they are buffering big responses in heap hence they'll know they have to do something about it, either write their own response consumer or increase the buffer size limit by providing their manually creeted instance of HeapBufferedAsyncResponseConsumer (constructor accept a bufferLimit int argument).
2016-07-19 15:17:12 +02:00
javanna 1bb33cf572 Remove RestClient#JSON_CONTENT_TYPE constant, already available in ContentType class 2016-07-19 15:17:12 +02:00
javanna f2ab597c84 Build: use license mapping for http* and commons-*
This way we reduce the number of LICENSE and NOTICE files with same content for client.rest and client.sniffer projects.
2016-07-19 15:16:45 +02:00
javanna 69309fb834 [TEST] remove one too many SuppressWarnings 2016-07-19 15:16:45 +02:00
javanna 06caea6b80 move RestClient#builder method on top for more visibility 2016-07-19 15:16:45 +02:00
javanna a3f9721751 replace till with until in RestClient javadocs 2016-07-19 15:16:45 +02:00
javanna b6b92c64c0 update Response javadocs 2016-07-19 15:16:45 +02:00
javanna 283090e2ae add check for null hosts in RestClientBuilder, so it fails early
Also delayed call to HttpAsyncClient#start so that if something goes wrong while creating the RestClient, the http client threads don't linger. In fact, if the constructor fails it is not possible to call close against the RestClient.
2016-07-19 15:16:45 +02:00
javanna e5006ed7b5 Rest Client: have RestClientBuilder callback also return the same type as their argument
HttpClientConfigCallback#customizeHttpClient now also returns the HttpClientBuilder so it can be completely replaced
RequestConfigCallback#customizeRequestConfig now also returns the HttpClientBuilder so it can be completely replaced
2016-07-19 15:16:45 +02:00
javanna e27203534a Rest Client: improve listener naming 2016-07-19 15:16:45 +02:00
javanna 41e97a7cb1 RestClient: take builder out to its own class
The RestClient class is getting bigger and bigger, its builder can definitely be taken out to its own top level class: RestClientBuilder
2016-07-19 15:16:45 +02:00
javanna bb21009772 [TEST] add async entities to the randomization for RequestLoggerTests 2016-07-19 15:15:58 +02:00
javanna 1fbec71243 Rest client: introduce async performRequest method and use async client under the hood for sync requests too
The new method accepts the usual parameters (method, endpoint, params, entity and headers) plus a response listener and an async response consumer. Shortcut methods are also added that don't require params, entity and the async response consumer optional.

There are a few relevant api changes as a consequence of the move to async client that affect sync methods:
- Response doesn't implement Closeable anymore, responses don't need to be closed
- performRequest throws Exception rather than just IOException, as that is the the exception that we get from the FutureCallback#failed method in the async http client
- ssl configuration is a bit simpler, one only needs to call setSSLStrategy from a custom HttpClientConfigCallback, that doesn't end up overridng any other default around connection pooling (it used to happen with the sync client and make ssl configuration more complex)

Relates to 
2016-07-19 15:15:58 +02:00
javanna 22aa40bb6d Build: add apache async http client dependencies 2016-07-19 15:11:40 +02:00
Simon Willnauer 8394544548 Add a dedicated client/transport project for transport-client ()
The `client/transport` project adds a new jar build project that
pulls in all dependencies and configures all required modules.

Preinstalled modules are:
 * transport-netty
 * lang-mustache
 * reindex
 * percolator

The `TransportClient` classes are still in core
while `TransportClient.Builder` has only a protected construcutor
such that users are redirected to use the new `TransportClientBuilder`
from the new jar.

Closes 
2016-07-18 15:42:24 +02:00
javanna 512b8be791 RestClient: simplify ssl configuration and make http config callback functional friendly 2016-07-12 13:25:55 +02:00
javanna fa0b354e66 Rest Client: add callback to customize http client settings
The callback replaces the ability to fully replace the http client instance. By doing that, one used to lose any default that the RestClient had set for the underlying http client. Given that you'd usually override one or two things only, like a couple of timeout values, the ssl factory or the default credentials providers, it is not uder friendly if by doing that users end up replacing the whole http client instance and lose any default set by us.
2016-07-12 12:31:28 +02:00
javanna 942e342662 Rest Client: use short performRequest methods when possible 2016-07-11 10:36:26 +02:00
javanna fd297637a2 Rest Client: add short performRequest method variants without params and/or body
Users wanting to send a request by providing only its method and endpoint, effectively the only two required arguments, shouldn't need to pass in an empty map and a null entity for the body. While at it we can also add a variant to send requests by specifying only method, endpoint and params, but not body. Headers remain a vararg as last argument, so they can always optionally be provided.

 Closes 
2016-07-11 10:36:04 +02:00
javanna 2a91a6ac37 Rest Client: check log level against tracer logger for trace logging 2016-07-08 12:52:29 +02:00
javanna c63719d085 Rest Client: wrap log statement in logger.isDebugEnabled 2016-07-08 12:51:48 +02:00
javanna cfc762b70d Rest Client: add slash to log line when missing between host and uri
Closes 
2016-07-08 12:25:20 +02:00
javanna 134b73ab33 Rest Client: HostsSniffer to set http as default scheme
The assumption is HostsSniffer is that all of the arguments have been properly provided and validated through HostsSniffer.Builder, except they weren't, as the scheme didn't have a default value and when not set would cause NPEs down the road. Improved tests to catch this also.
2016-07-07 19:11:37 +02:00
Tanguy Leroux 8c40b2b54e Fix order of modifiers 2016-07-01 16:57:14 +02:00
javanna c4b87149c2 Build: set group for client and sniffer, disable publishing for client-test
Closes 
2016-07-01 12:09:34 +02:00
Yannick Welsch 33313df0a2 Add ThreadLeakLingering option to Rest client tests
Some Rest tests use the Sun HTTP server which has lingering threads after shutdown. Similar to ESTestCase, this adds an
option to wait 5 seconds for these threads to terminate.
2016-06-23 10:21:37 +02:00
Nik Everett 0bf447c697 Group client projects under :client
:client ---------> :client:rest
:client-sniffer -> :client:sniffer
:client-test ----> :client:test

This lines the client up with how we do things like modules and
plugins.
2016-06-22 14:26:41 -04:00
javanna c2839c1577 [TEST] Add client-test module and make client tests use randomized runner directly
The lucene-test dependency caused issues with IDEs as they would always load the lucene 5 jar although they shouldn't have, which caused jarhell in es core tests.

If we depend directly on randomized runner we don't have this problem. It is luckily still compatible with java 1.7. This requires though adding a thin module that includes the base test class which can be shared between client and client-sniffer.
2016-06-22 19:37:08 +02:00
javanna f0b6abe439 rename onSuccess to onResponse
That makes it a bit clearer that it's about the response and whether we decide if it was a good one or a failure (based on status code)
2016-06-21 17:11:09 +02:00
javanna cb4bfcb864 Take SniffOnFailureListener out of Sniffer and make FailureListener final on RestClient 2016-06-21 15:29:59 +02:00
javanna 8c60374284 Build: do not load integ test class if --skip-integ-tests-in-disguise is specified in NamingConventionsCheck
Projects that don't depend on elasticsearch-test fail otherwise because org.elasticsearch.test.EsIntegTestCase (default integ test class) is not in the classpath. They should provide their onw integ test base class, but having integration tests should not be mandatory. One can simply set skipIntegTestsInDisguise to true to prevent loading of integ test class.
2016-06-17 13:46:27 +02:00
javanna 7c8013f9fd Build: remove explicit targetCompatibility from forbiddenapis config
targetCompatibility set to the project is enough.
2016-06-15 11:43:48 +02:00
javanna 70fb67cc7b Build: targetCompatibility and sourceCompatibility are enough to make sure we compile client and client-sniffer with target and source 1.7 2016-06-14 23:16:38 +02:00
javanna cf93e904e3 remove message parameter from RequestLogger methods
This prevents useless string allocation.
2016-06-14 14:14:03 +02:00
javanna 1932f6bc7c Rename RequestLogger#log methods to distinguish between the two
One method is to log a request that yielded a response, the other one for a failed request
2016-06-14 13:45:38 +02:00
javanna caa6c96259 Build: make client and client-sniffer depend on lucene-test version 5, last 1.7 compatible version 2016-06-14 09:43:31 +02:00
javanna 116805b28b remove TODO around copying hosts when rotating the collection, it's not a problem for now 2016-06-13 14:02:01 +02:00
javanna 8f7b7fb813 added comments to clarify RequestLogger and DeadHostState 2016-06-13 12:59:30 +02:00
javanna 3cd201e67e [TEST] add comment on using animal-sniffer suppress annotation 2016-06-13 12:02:09 +02:00
javanna 50b6f4c02f Build: changed forbidden-apis targetCompatibility to 1.7 for client and client-sniffer 2016-06-13 09:20:39 +02:00
javanna 777d438f48 [TEST] use jdk-internal bundled signature (rather than the previously removed jdk-non-portable) 2016-06-10 12:46:33 +02:00
javanna 3d7186c81f make DeadHostState final 2016-06-10 11:40:35 +02:00
javanna 6db90da25e [TEST] Remove usage of @SuppressForbidden due to sun HttpServer usage, resolve some violations that were hidden by it 2016-06-10 11:40:13 +02:00
javanna a5e329e563 [TEST] use animalsniffer annotation @IgnoreJRERequirement (similar to @SuppressForbidden) rather than exclusion pattern 2016-06-10 10:11:21 +02:00
javanna 0af9d2c767 Build: add animalsniffer to detect usage of java8 apis in client and client-sniffer 2016-06-10 09:51:47 +02:00
javanna f38ce72004 make forbiddenApisTest work for client and client-sniffer 2016-06-09 23:44:34 +02:00
javanna bd773359d5 [TEST] add SuppresForbidden annotation for client project 2016-06-09 17:39:52 +02:00
javanna d8c0fad08f fix failing tests 2016-06-09 17:06:58 +02:00
javanna 9a4971d3fd fix line length 2016-06-09 16:57:57 +02:00
javanna de8b9fd579 use switch for status codes and add comments 2016-06-09 16:51:08 +02:00
javanna cbdffc7965 s/elasticsearchResponse/response 2016-06-09 16:39:59 +02:00
javanna 853ea9385b add comments on retries 2016-06-09 16:38:43 +02:00
javanna 742f9c6eaa nextHost to return Iterable<HttpHost> rather than Iterator 2016-06-09 16:34:01 +02:00
javanna c028bf9250 rename deadUntil to deadUntilNanos 2016-06-09 16:31:09 +02:00
javanna 85606e8a4b renamed all time intervals arguments and members to include the time unit in their name 2016-06-09 16:27:52 +02:00
javanna 437c4f210b rename ElasticsearchResponse to Response and ElasticsearchResponseException to ResponseException 2016-06-09 14:38:32 +02:00
javanna 04d620da74 require hosts when creating RestClient.Builder
Also fix order of arguments when using assertEquals
2016-06-08 12:37:50 +02:00
javanna 2cf04c0877 wrap entity only when not repeatable and improve RequestLoggerTests 2016-06-06 15:41:11 +02:00
javanna 1f7f6e2709 wrap log statement with logger.isDebugEnabled 2016-06-06 15:13:42 +02:00
javanna a461dd84d2 Build: add hamcrest and securemock to version.properties 2016-06-06 15:02:52 +02:00
javanna 05e26a46d7 raise default socket timeout to 10s and default connect timeout to 1s 2016-06-04 01:06:18 +02:00
javanna f2318ed5ae Build: add missing licenses, SHAs and enable dependency licenses check 2016-06-04 00:56:42 +02:00
javanna b15279b5ef Allow to pass socket facttry registry to createDefaultHttpClient method 2016-06-03 23:59:26 +02:00
javanna f17f0f9247 rename ElasticsearchResponse#getFirstHeader to getHeader 2016-06-03 18:28:31 +02:00
javanna 29eee328fe [TEST] expand RequestLoggerTests to all the supported http methods 2016-06-03 18:23:52 +02:00
javanna 13a27a34f8 [TEST] add RestClient integration test
Relies on real HttpServer to test the actual interaction between RestClient and HttpClient, and how requests get sent in real life.
2016-06-03 16:20:12 +02:00