Commit Graph

467 Commits

Author SHA1 Message Date
javanna 59ccc88c73 rename mustRetry method to mayRetry 2016-07-20 16:24:57 +02:00
javanna fccfe7dcb8 RestClient javadocs adjustments 2016-07-20 16:24:29 +02:00
javanna 569d7b3ecc notify the listener if the request gets cancelled 2016-07-20 15:24:14 +02:00
javanna a9b5c5adbe restore throws IOException clause on all performRequest sync methods
We throw IOException, which is the exception that is going to be thrown in 99% of the cases. A more generic exception can happen, and if it is a runtime one we just let it bubble up as is, otherwise we wrap it into runtime one so that we don't require to catch Exception everywhere, which seems odd.

Also adjusted javadocs for all performRequest methods
2016-07-19 15:18:05 +02:00
javanna 8eccdff9ad add HeapBufferedAsyncResponseConsumer with configurable max buffer size
We keep the default async client behaviour like in BasicAsyncResponseConsumer, but we lower the maximum size of the buffer from Integer.MAX_VALUE (2GB) to 10 MB. This way users will realize they are buffering big responses in heap hence they'll know they have to do something about it, either write their own response consumer or increase the buffer size limit by providing their manually creeted instance of HeapBufferedAsyncResponseConsumer (constructor accept a bufferLimit int argument).
2016-07-19 15:17:12 +02:00
javanna 1bb33cf572 Remove RestClient#JSON_CONTENT_TYPE constant, already available in ContentType class 2016-07-19 15:17:12 +02:00
javanna f2ab597c84 Build: use license mapping for http* and commons-*
This way we reduce the number of LICENSE and NOTICE files with same content for client.rest and client.sniffer projects.
2016-07-19 15:16:45 +02:00
javanna 69309fb834 [TEST] remove one too many SuppressWarnings 2016-07-19 15:16:45 +02:00
javanna 06caea6b80 move RestClient#builder method on top for more visibility 2016-07-19 15:16:45 +02:00
javanna a3f9721751 replace till with until in RestClient javadocs 2016-07-19 15:16:45 +02:00
javanna b6b92c64c0 update Response javadocs 2016-07-19 15:16:45 +02:00
javanna 283090e2ae add check for null hosts in RestClientBuilder, so it fails early
Also delayed call to HttpAsyncClient#start so that if something goes wrong while creating the RestClient, the http client threads don't linger. In fact, if the constructor fails it is not possible to call close against the RestClient.
2016-07-19 15:16:45 +02:00
javanna e5006ed7b5 Rest Client: have RestClientBuilder callback also return the same type as their argument
HttpClientConfigCallback#customizeHttpClient now also returns the HttpClientBuilder so it can be completely replaced
RequestConfigCallback#customizeRequestConfig now also returns the HttpClientBuilder so it can be completely replaced
2016-07-19 15:16:45 +02:00
javanna e27203534a Rest Client: improve listener naming 2016-07-19 15:16:45 +02:00
javanna 41e97a7cb1 RestClient: take builder out to its own class
The RestClient class is getting bigger and bigger, its builder can definitely be taken out to its own top level class: RestClientBuilder
2016-07-19 15:16:45 +02:00
javanna bb21009772 [TEST] add async entities to the randomization for RequestLoggerTests 2016-07-19 15:15:58 +02:00
javanna 1fbec71243 Rest client: introduce async performRequest method and use async client under the hood for sync requests too
The new method accepts the usual parameters (method, endpoint, params, entity and headers) plus a response listener and an async response consumer. Shortcut methods are also added that don't require params, entity and the async response consumer optional.

There are a few relevant api changes as a consequence of the move to async client that affect sync methods:
- Response doesn't implement Closeable anymore, responses don't need to be closed
- performRequest throws Exception rather than just IOException, as that is the the exception that we get from the FutureCallback#failed method in the async http client
- ssl configuration is a bit simpler, one only needs to call setSSLStrategy from a custom HttpClientConfigCallback, that doesn't end up overridng any other default around connection pooling (it used to happen with the sync client and make ssl configuration more complex)

Relates to #19055
2016-07-19 15:15:58 +02:00
javanna 22aa40bb6d Build: add apache async http client dependencies 2016-07-19 15:11:40 +02:00
Simon Willnauer 8394544548 Add a dedicated client/transport project for transport-client (#19435)
The `client/transport` project adds a new jar build project that
pulls in all dependencies and configures all required modules.

Preinstalled modules are:
 * transport-netty
 * lang-mustache
 * reindex
 * percolator

The `TransportClient` classes are still in core
while `TransportClient.Builder` has only a protected construcutor
such that users are redirected to use the new `TransportClientBuilder`
from the new jar.

Closes #19412
2016-07-18 15:42:24 +02:00
javanna 512b8be791 RestClient: simplify ssl configuration and make http config callback functional friendly 2016-07-12 13:25:55 +02:00
javanna fa0b354e66 Rest Client: add callback to customize http client settings
The callback replaces the ability to fully replace the http client instance. By doing that, one used to lose any default that the RestClient had set for the underlying http client. Given that you'd usually override one or two things only, like a couple of timeout values, the ssl factory or the default credentials providers, it is not uder friendly if by doing that users end up replacing the whole http client instance and lose any default set by us.
2016-07-12 12:31:28 +02:00
javanna 942e342662 Rest Client: use short performRequest methods when possible 2016-07-11 10:36:26 +02:00
javanna fd297637a2 Rest Client: add short performRequest method variants without params and/or body
Users wanting to send a request by providing only its method and endpoint, effectively the only two required arguments, shouldn't need to pass in an empty map and a null entity for the body. While at it we can also add a variant to send requests by specifying only method, endpoint and params, but not body. Headers remain a vararg as last argument, so they can always optionally be provided.

 Closes #19312
2016-07-11 10:36:04 +02:00
javanna 2a91a6ac37 Rest Client: check log level against tracer logger for trace logging 2016-07-08 12:52:29 +02:00
javanna c63719d085 Rest Client: wrap log statement in logger.isDebugEnabled 2016-07-08 12:51:48 +02:00
javanna cfc762b70d Rest Client: add slash to log line when missing between host and uri
Closes #19314
2016-07-08 12:25:20 +02:00
javanna 134b73ab33 Rest Client: HostsSniffer to set http as default scheme
The assumption is HostsSniffer is that all of the arguments have been properly provided and validated through HostsSniffer.Builder, except they weren't, as the scheme didn't have a default value and when not set would cause NPEs down the road. Improved tests to catch this also.
2016-07-07 19:11:37 +02:00
Tanguy Leroux 8c40b2b54e Fix order of modifiers 2016-07-01 16:57:14 +02:00
javanna c4b87149c2 Build: set group for client and sniffer, disable publishing for client-test
Closes #19205
2016-07-01 12:09:34 +02:00
Yannick Welsch 33313df0a2 Add ThreadLeakLingering option to Rest client tests
Some Rest tests use the Sun HTTP server which has lingering threads after shutdown. Similar to ESTestCase, this adds an
option to wait 5 seconds for these threads to terminate.
2016-06-23 10:21:37 +02:00
Nik Everett 0bf447c697 Group client projects under :client
:client ---------> :client:rest
:client-sniffer -> :client:sniffer
:client-test ----> :client:test

This lines the client up with how we do things like modules and
plugins.
2016-06-22 14:26:41 -04:00
javanna c2839c1577 [TEST] Add client-test module and make client tests use randomized runner directly
The lucene-test dependency caused issues with IDEs as they would always load the lucene 5 jar although they shouldn't have, which caused jarhell in es core tests.

If we depend directly on randomized runner we don't have this problem. It is luckily still compatible with java 1.7. This requires though adding a thin module that includes the base test class which can be shared between client and client-sniffer.
2016-06-22 19:37:08 +02:00
javanna f0b6abe439 rename onSuccess to onResponse
That makes it a bit clearer that it's about the response and whether we decide if it was a good one or a failure (based on status code)
2016-06-21 17:11:09 +02:00
javanna cb4bfcb864 Take SniffOnFailureListener out of Sniffer and make FailureListener final on RestClient 2016-06-21 15:29:59 +02:00
javanna 8c60374284 Build: do not load integ test class if --skip-integ-tests-in-disguise is specified in NamingConventionsCheck
Projects that don't depend on elasticsearch-test fail otherwise because org.elasticsearch.test.EsIntegTestCase (default integ test class) is not in the classpath. They should provide their onw integ test base class, but having integration tests should not be mandatory. One can simply set skipIntegTestsInDisguise to true to prevent loading of integ test class.
2016-06-17 13:46:27 +02:00
javanna 7c8013f9fd Build: remove explicit targetCompatibility from forbiddenapis config
targetCompatibility set to the project is enough.
2016-06-15 11:43:48 +02:00
javanna 70fb67cc7b Build: targetCompatibility and sourceCompatibility are enough to make sure we compile client and client-sniffer with target and source 1.7 2016-06-14 23:16:38 +02:00
javanna cf93e904e3 remove message parameter from RequestLogger methods
This prevents useless string allocation.
2016-06-14 14:14:03 +02:00
javanna 1932f6bc7c Rename RequestLogger#log methods to distinguish between the two
One method is to log a request that yielded a response, the other one for a failed request
2016-06-14 13:45:38 +02:00
javanna caa6c96259 Build: make client and client-sniffer depend on lucene-test version 5, last 1.7 compatible version 2016-06-14 09:43:31 +02:00
javanna 116805b28b remove TODO around copying hosts when rotating the collection, it's not a problem for now 2016-06-13 14:02:01 +02:00
javanna 8f7b7fb813 added comments to clarify RequestLogger and DeadHostState 2016-06-13 12:59:30 +02:00
javanna 3cd201e67e [TEST] add comment on using animal-sniffer suppress annotation 2016-06-13 12:02:09 +02:00
javanna 50b6f4c02f Build: changed forbidden-apis targetCompatibility to 1.7 for client and client-sniffer 2016-06-13 09:20:39 +02:00
javanna 777d438f48 [TEST] use jdk-internal bundled signature (rather than the previously removed jdk-non-portable) 2016-06-10 12:46:33 +02:00
javanna 3d7186c81f make DeadHostState final 2016-06-10 11:40:35 +02:00
javanna 6db90da25e [TEST] Remove usage of @SuppressForbidden due to sun HttpServer usage, resolve some violations that were hidden by it 2016-06-10 11:40:13 +02:00
javanna a5e329e563 [TEST] use animalsniffer annotation @IgnoreJRERequirement (similar to @SuppressForbidden) rather than exclusion pattern 2016-06-10 10:11:21 +02:00
javanna 0af9d2c767 Build: add animalsniffer to detect usage of java8 apis in client and client-sniffer 2016-06-10 09:51:47 +02:00
javanna f38ce72004 make forbiddenApisTest work for client and client-sniffer 2016-06-09 23:44:34 +02:00
javanna bd773359d5 [TEST] add SuppresForbidden annotation for client project 2016-06-09 17:39:52 +02:00
javanna d8c0fad08f fix failing tests 2016-06-09 17:06:58 +02:00
javanna 9a4971d3fd fix line length 2016-06-09 16:57:57 +02:00
javanna de8b9fd579 use switch for status codes and add comments 2016-06-09 16:51:08 +02:00
javanna cbdffc7965 s/elasticsearchResponse/response 2016-06-09 16:39:59 +02:00
javanna 853ea9385b add comments on retries 2016-06-09 16:38:43 +02:00
javanna 742f9c6eaa nextHost to return Iterable<HttpHost> rather than Iterator 2016-06-09 16:34:01 +02:00
javanna c028bf9250 rename deadUntil to deadUntilNanos 2016-06-09 16:31:09 +02:00
javanna 85606e8a4b renamed all time intervals arguments and members to include the time unit in their name 2016-06-09 16:27:52 +02:00
javanna 437c4f210b rename ElasticsearchResponse to Response and ElasticsearchResponseException to ResponseException 2016-06-09 14:38:32 +02:00
javanna 04d620da74 require hosts when creating RestClient.Builder
Also fix order of arguments when using assertEquals
2016-06-08 12:37:50 +02:00
javanna 2cf04c0877 wrap entity only when not repeatable and improve RequestLoggerTests 2016-06-06 15:41:11 +02:00
javanna 1f7f6e2709 wrap log statement with logger.isDebugEnabled 2016-06-06 15:13:42 +02:00
javanna a461dd84d2 Build: add hamcrest and securemock to version.properties 2016-06-06 15:02:52 +02:00
javanna 05e26a46d7 raise default socket timeout to 10s and default connect timeout to 1s 2016-06-04 01:06:18 +02:00
javanna f2318ed5ae Build: add missing licenses, SHAs and enable dependency licenses check 2016-06-04 00:56:42 +02:00
javanna b15279b5ef Allow to pass socket facttry registry to createDefaultHttpClient method 2016-06-03 23:59:26 +02:00
javanna f17f0f9247 rename ElasticsearchResponse#getFirstHeader to getHeader 2016-06-03 18:28:31 +02:00
javanna 29eee328fe [TEST] expand RequestLoggerTests to all the supported http methods 2016-06-03 18:23:52 +02:00
javanna 13a27a34f8 [TEST] add RestClient integration test
Relies on real HttpServer to test the actual interaction between RestClient and HttpClient, and how requests get sent in real life.
2016-06-03 16:20:12 +02:00
javanna 4572b69011 [TEST] add RestClient unit tests
Unit tests rely on mockito to mock the internal HttpClient instance. No http request is performed, we only simulate interaction between RestClient and its internal HttpClient.
2016-06-03 16:20:12 +02:00
javanna 9ed2d610ec [TEST] rename restClientTests back to RestClientBuilderTests 2016-06-03 16:20:12 +02:00
javanna 24ea585c9e don't use setDefaultHeaders from HttpClient
Store default headers ourselves instead, otherwise default ones cannot be replaced. Don't allow for multiple headers with same key, last one wins and replaces previous ones with same key.

Also fail with null params or headers.
2016-06-03 16:20:11 +02:00
javanna 47e52044e4 [TEST] add setHosts test and rename RestClientBuilderTests to RestClientTests 2016-06-03 16:01:07 +02:00
javanna 35dbdeeae5 check hosts is not null nor empty earlier, remove check from nextHost
if we check at set time, we don't need to check each single time in nextHost
2016-06-03 16:01:07 +02:00
javanna 6d66fbd9c1 add toString to DeadHostState class 2016-06-03 16:01:07 +02:00
javanna c9db111387 add javadocs on closing responses 2016-06-03 16:01:07 +02:00
javanna 83c6e736de add support for PATCH and TRACE methods
Although elasticsearch doesn't support these methods (RestController doesn't even allow to register handler for them), the RestClient should allow to send requests using them.
2016-06-03 16:01:07 +02:00
javanna 51e487fa55 [TEST] remove okhttp test dependency
Use sun HttpServer instead and disable forbidden-apis for test classes. It turns out to be more flexible than okhttp as it allows get & delete with body.
2016-06-03 16:01:07 +02:00
javanna 044a97c740 move client sniffer to its own project
Create a new subproject called client-sniffer that contains the o.e.client.sniff package. Since it is going to go to a separate jar, due to its additional functionalities and dependency on jackson, it makes sense to have it as a separate project that depends on client. This way we make sure that client doesn't depend on it etc.
2016-06-03 16:01:07 +02:00
javanna 7f4807b29e add some javadocs 2016-06-03 16:01:07 +02:00
javanna 29b1f8d44a make some classes and methods package private
ElasticsearchResponseException, as well as ElasticsearchResponse, should only be created from o.e.client package.
RequestLogger should only be used from this package too.
2016-06-03 16:01:07 +02:00
javanna 6490355cb6 make host state immutable
Instead of having a Connection mutable object that holds the state of the connection to each host, we now have immutable objects only. We keep two sets, one with all the hosts, one with the blacklisted ones. Once we blacklist a host we associate it with a DeadHostState which keeps track of the number of failed attempts and when the host should be retried. A new state object is created each and every time the state of the host needs to be updated.
2016-06-03 16:01:07 +02:00
javanna c70e08c393 include response body in ElasticsearchResponseException error message 2016-06-03 16:01:07 +02:00
javanna eae914ae8e Replace rest test client with low level RestClient
We still have a wrapper called RestTestClient that is very specific to Rest tests, as well as RestTestResponse etc. but all the low level bits around http connections etc. are now handled by RestClient.
2016-06-03 16:01:07 +02:00
javanna 325b723930 [TEST] add rest client test dependency and replace usage of HttpRequestBuilder with RestClient in integration tests 2016-06-03 16:01:07 +02:00
javanna 6d3f6c7faf support missing OPTIONS method, it is supported in elasticsearch core 2016-06-03 16:01:07 +02:00
javanna 9a38d81bec Expose whole ElasticsearchResponse within ElasticsearchResponseException
The only small problem is that the response gets closed straightaway and its body read immediately into a string. Should be ok to load it all into memory eagerly though in case of errors. Otherwise it becomes cumbersome to have an exception implement Closeable...
2016-06-03 16:01:07 +02:00
javanna 16ab491016 add getFirstHeader method to ElasticsearchResponse 2016-06-03 16:01:07 +02:00
javanna 1d06916b07 adapt params argument, they can only be Strings in performRequest method 2016-06-03 16:01:07 +02:00
javanna 2735897b36 use versions from versions.properties 2016-06-03 16:01:07 +02:00
javanna 2d7a781195 fix usage of deprecated apis 2016-06-03 16:01:07 +02:00
javanna c0a72c1686 add support for headers: default ones and per request 2016-06-03 16:01:07 +02:00
javanna 85a7721185 simplify Connection class
The connection class can be greatly simplified now that we don't ping anymore. Pings required a special initial state (UNKNOWN) for connections, to indicate that they require pinging although they are not dead. At this point we don't need the State enum anymore, as connections can only be dead or alive based on the number of failed attempts. markResurrected is also not needed, as it was again a way to make pings required. RestClient can simply pick a dead connection now and use it, no need to change its state when picking the connection.
2016-06-03 16:01:07 +02:00
javanna cdffc3d15b remove notion of connection pool, turn around dependency between RestClient and Sniffer
RestClient exposes a setNodes method, which Sniffer can call to update its nodes.
2016-06-03 16:01:07 +02:00
javanna b38ef345e2 remove streams leftover
Given that we don't use streams anymore, we can check straightaway if the connection iterator is empty before returning it and resurrect a connection when needed directly in the connection pool, no lastResortConnection method required.
2016-06-03 16:01:07 +02:00
javanna 599dad560c remove streams and java 8 only api, build with source and target 1.7 2016-06-03 16:01:07 +02:00
javanna 9569ebc262 add builders for simple creation of RestClient and SniffingConnectionPool instances
We have quite some constructor parameters and some defaults should be applied, builders help simplifying creation of objects for users.
2016-06-03 16:01:07 +02:00
javanna e040d2fc77 remove ConnectionPool interface 2016-06-03 16:01:07 +02:00
javanna 17a21f0272 add curl format trace logging for requests and responses 2016-06-03 16:01:07 +02:00
javanna e7fe397c39 add missing parentheses 2016-06-03 16:01:07 +02:00
javanna 530ad227a2 prevent unclosed response entities in ElasticsearchResponseException, eagerly read response string in case of error status code 2016-06-03 16:01:07 +02:00
javanna e77ab87926 return the response as part of ElasticsearchResponseException 2016-06-03 16:01:07 +02:00
javanna e85ed3eb52 remove pinging from static connection pool, can be replaced by a low connect timeout on each request 2016-06-03 16:01:07 +02:00
javanna d7c41764f2 add some javadocs to connection pool classes 2016-06-03 16:01:07 +02:00
javanna 9ffdea9515 remove Scheme enum 2016-06-03 16:01:07 +02:00
javanna bd29dc1572 Remove Transport class, move all of it within RestClient class 2016-06-03 16:01:07 +02:00
javanna 062a21678c merge Connection and StatefulConnection into one class, remove generic type from Transport 2016-06-03 16:01:07 +02:00
javanna a472544ab4 move sniff related stuff to sniff package 2016-06-03 16:01:07 +02:00
javanna ce663e9703 get rid of connection selector predicate 2016-06-03 16:01:07 +02:00
javanna f6a5a0a4ad get rid of Node abstraction 2016-06-03 16:01:07 +02:00
javanna 94cf8437d0 get rid of retry timeout exception 2016-06-03 16:01:07 +02:00
javanna 25892351e7 remove Verb enum 2016-06-03 16:01:07 +02:00
javanna 6ae8be55bd Introduce notion of Transport, Connection and ConnectionPool
There are two implementations of connection pool, a static one that allows to enable/disable pings, and a sniffing one that sniffs nodes from the nodes info api.

Transport retrieves a stream of connections from the connection for each request and calls onSuccess or onFailure depending on the result of the request.

Transport also supports a max retry timeout to control the timeout for the request retries overall.
2016-06-03 16:01:07 +02:00
javanna 5970723945 clarify why jarhell and dependency license check are disabled, add okhttp mockwebserver dependency
Also enable forbiddenApisTest (will fail till we get rid of the sun http web server in RestClientTests that will be replaced with mockwebserver)
2016-06-03 16:01:07 +02:00
javanna c9c33a7719 Fix checkstyle issues, setup thirdPartyAudit checks and enable forbiddenApisMain 2016-06-03 16:01:07 +02:00
Simon Willnauer 3efbe95ca4 RestClient prototype 2016-06-03 16:01:07 +02:00