Commit Graph

788 Commits

Author SHA1 Message Date
Lee Hinman b4c78019b0
Remove all dependencies from XContentBuilder (#29225)
* Remove all dependencies from XContentBuilder

This commit removes all of the non-JDK dependencies from XContentBuilder, with
the exception of `CollectionUtils.ensureNoSelfReferences`. It adds a third
extension point around dealing with time-based fields and formatters to work
around the Joda dependency.

This decoupling allows us to be able to move XContentBuilder to a separate lib
so it can be available for things like the high level rest client.

Relates to #28504
2018-03-27 12:58:22 -06:00
Luca Cavanna 13f9e922f3
REST client: hosts marked dead for the first time should not be immediately retried (#29230)
This was the plan from day one but due to a silly bug nodes were immediately retried after they were marked as dead for the first time. From the second time on, the expected backoff was applied.
2018-03-27 16:15:44 +02:00
Christoph Büscher 318b0af953 Remove execute mode bit from source files
Some source files seem to have the execute bit (a+x) set, which doesn't
really seem to hurt but is a bit odd. This change removes those, making
the permissions similar to other source files in the repository.
2018-03-26 13:37:55 +02:00
Nik Everett 8c59e43ac7
Docs: HighLevelRestClient#multiSearch (#29144)
Adds docs for `HighLevelRestClient#multiSearch`. Unlike the `multiGet`
docs these are much more sparse because multi-search doesn't support
setting many options on the `MultiSearchRequest` and instead just wraps
a list of `SearchRequest`s.

Closes #28389
2018-03-23 10:11:50 -04:00
Yu 24c8d8f5ef REST high-level client: add force merge API (#28896)
Relates to #27205
2018-03-22 17:17:16 +01:00
Christoph Büscher e4b30071bb
RankEvalRequest should implement IndicesRequest (#29188)
Change RankEvalRequest to implement IndicesRequest, so it gets treated
in a similar fashion to regular search requests e.g. by security.
2018-03-22 11:58:55 +01:00
Luca Cavanna ff09c82319
REST high-level client: add clear cache API (#28866)
* REST high-level client: add clear cache API

Relates to #27205

Also Closes #26947 (rest-spec were outdated)
2018-03-20 10:39:36 +01:00
Christoph Büscher 80532229a9
Move indices field from RankEvalSpec to RankEvalRequest (#28341)
Currently we store the indices specified in the request URL together with all
the other ranking evaluation specification in RankEvalSpec. This is not ideal
since e.g. the indices are not rendered to xContent and so cannot be parsed
back. Instead we should keep them in RankEvalRequest.
2018-03-19 16:26:02 +01:00
Nik Everett 1d8c507684 Client: Add missing test
Previously I added wrapping for an SSL exception without a test. That
was lame. This adds the test.
2018-03-17 20:09:43 -04:00
Nik Everett cd165d1c4b Client: Wrap SSLHandshakeException in sync calls
Adds SSLHandshakeException to the list of Exceptions that are
specifically rethrown from the async thread so its type is preserved.

This should make it easier to debug synchronous calls with ssl issues.
2018-03-17 11:06:05 -04:00
Nik Everett 2f21dc7129
Docs: HighLevelRestClient#multiGet (#29095)
Add documentation for HighLevelRestClient#multiGet.

Relates to #28389.
2018-03-16 18:52:34 -04:00
Nik Everett 60cb476cc9
Client: Wrap synchronous exceptions (#28919)
In the past the Low Level REST Client was super careful not to wrap
any exceptions that it throws from synchronous calls so that callers can
catch the exceptions and work with them. The trouble with that is that
the exceptions are originally thrown on the async thread pool and then
transfered back into calling thread. That means that the stack trace of
the exception doesn't have the calling method which is *super* *ultra*
confusing.

This change always wraps exceptions transferred from the async thread
pool so that the stack trace of the thrown exception contains the
caller's stack. It tries to preserve the type of the throw exception but
this is quite a fiddly thing to get right. We have to catch every type
of exception that we want to preserve, wrap with the same type and
rethrow. I've preserved the types of all exceptions that we had tests
mentioning but no other exceptions. The other exceptions are either
wrapped in `IOException` or `RuntimeException`.

Closes #28399
2018-03-16 16:55:37 -04:00
Nik Everett cf60e93a21
Docs: HighLevelRestClient#exists (#29073)
Docs: HighLevelRestClient#exists

Add documentation for `HighLevelRestClient#exists`.

Relates to #28389
2018-03-15 12:10:30 -04:00
Lee Hinman 8e8fdc4f0e
Decouple XContentBuilder from BytesReference (#28972)
* Decouple XContentBuilder from BytesReference

This commit removes all mentions of `BytesReference` from `XContentBuilder`.
This is needed so that we can completely decouple the XContent code and move it
into its own dependency.

While this change appears large, it is due to two main changes, moving
`.bytes()` and `.string()` out of XContentBuilder itself into static methods
`BytesReference.bytes` and `Strings.toString` respectively. The rest of the
change is code reacting to these changes (the majority of it in tests).

Relates to #28504
2018-03-14 13:47:57 -06:00
Nik Everett ef6fc1e9fd
Docs: HighLevelRestClient#ping (#29070)
Add documentation for `HighLevelRestClient#ping`.

Relates to #28389
2018-03-14 14:27:01 -04:00
Martijn van Groningen 34a264c375
added docs for `wrapper` query.
Closes #11591
2018-03-14 11:51:22 +01:00
Jason Tedor 24d10adaab
Main response should not have status 503 when okay (#29045)
The REST status 503 means "I can not handle the request that you sent
me." However today we respond to a main request with a 503 when there
are certain cluster blocks despite still responding with an actual main
response. This is broken, we should respond with a 200 status. This
commit removes this silliness.
2018-03-14 06:36:37 -04:00
Holger Bartnick b42804d182 [Docs] REST high-level client: Fix code for most basic search request (#28916)
We also need to add the query to the request.
2018-03-08 04:14:50 -08:00
Luca Cavanna 184a8718d8
REST high-level client: add flush API (#28852)
Relates to #27205
2018-03-01 10:56:03 +01:00
Yu 95dea2408d Add Refresh API for RestHighLevelClient (#27799)
Relates to #27205
2018-02-28 11:49:14 +01:00
olcbean c5821f9645 [docs] Line breaks for High-level REST Client (#28825)
Wrap code snippets for better display in the generated docs.
2018-02-26 14:03:03 -05:00
javanna e5bceee4a7 [TEST] replace randomAsciiAlphanumOfLengthBetween with randomAsciiLettersOfLengthBetween
This was wrongly changed from the corresponding deprecated method.
2018-02-21 16:36:04 +01:00
olcbean 50d8a25d67 [Docs] Java high-level REST client : clean up (#28703)
Make doc titles consistent with the ES docs and wrapping 
the code snippets for better readability.
2018-02-20 18:37:53 +01:00
Luca Cavanna 8bbb3c9ffa
REST high-level client: add support for Rollover Index API (#28698)
Relates to #27205
2018-02-20 15:58:58 +01:00
javanna 3e17185010 REST high-level client: fix _cluster/settings endpoint
The recent addition of the _cluster/settings API was merged together with another change that added encoding of the different URL parts. That broke the cluster PUT settings API straight-away. This commit fixes this problem. The '/' that's part of the /_cluster/settings endpoint should not be encoded.
2018-02-15 17:40:43 +01:00
Luca Cavanna ebe5e8e635
REST high-level client: encode path parts (#28663)
The REST high-level client supports now encoding of path parts, so that for instance documents with valid ids, but containing characters that need to be encoded as part of urls (`#` etc.), are properly supported. We also make sure that each path part can contain `/` by encoding them properly too.

Closes #28625
2018-02-15 17:22:45 +01:00
olcbean 02fc16f10e Add Cluster Put Settings API to the high level REST client (#28633)
Relates to #27205
2018-02-15 17:21:45 +01:00
Catalin Ursachi ee00523737 Removed redundant JSON object from Put Mapping docs (#28514) 2018-02-13 17:20:35 +01:00
Luca Cavanna cb7159d4c9
REST high-level Client: add missing final modifiers (#28572)
A couple of methods in RestHighLevelClient were supposed to be final but the modifier was forgotten.
2018-02-08 17:14:20 +01:00
Lee Hinman eebff4d2b3
Use non deprecated xcontenthelper (#28503)
* Move to non-deprecated XContentHelper.createParser(...)

This moves away from one of the now-deprecated XContentHelper.createParser
methods in favor of specifying the deprecation logger at parser creation time.

Relates to #28449

Note that this doesn't move all the `createParser` calls because some of them
use the already-deprecated method that doesn't specify the XContentType.

* Remove the deprecated (and now non-needed) createParser method
2018-02-05 16:18:18 -07:00
javanna b5986c8dce fix checkstyle error in SearchDocumentationIT 2018-02-02 14:05:30 +01:00
javanna 9d11cfd652 [DOCS] Remove rawtypes suppressions and fix violations in REST high-level client docs tests 2018-02-02 11:57:55 +01:00
javanna 174e243f83 [DOCS] Adapt indices exists docs after recent listener changes 2018-02-02 11:46:05 +01:00
Haris Osmanagić 897ef458f3 Add support for indices exists to REST high level client (#27384)
Relates to #27205
2018-02-02 11:25:36 +01:00
Luca Cavanna d10dec3e99
[DOCS] expand examples on providing mappings for create index and put mapping (#28483)
* [DOCS] expand examples on providing mappings for create index and put mapping

The create index API and put mappings API docs the for high-level Java REST client didn't have a lot of info on how to provide mappings. This commit adds some examples.
2018-02-02 10:32:24 +01:00
Tanguy Leroux bb97c00556
[Docs] Docs tests should wait for async execution to complete (#28481)
This commit splits the async execution documentation into 2 parts, one
for the async method itself and one for the action listener. This allows
to add more doc and to use CountDownLatches in doc tests to wait for
asynchronous operations to be completed before moving to the next test.

It also renames few files.

Related to #28457
2018-02-01 17:56:13 +01:00
Luca Cavanna d860971572
REST high-level client: add support for split and shrink index API (#28425)
Relates to #27205
2018-02-01 16:37:01 +01:00
Tanguy Leroux ec187caeae
[Test] Fix CRUDDocumentationIT (#28457)
Similarly to other documentation tests in the high level client, the
asynchronous operation like update, index or delete can make the test
fail if it sneak in between the middle of another operation.

This commit moves the async doc tests to be the last ones executed and
adds assert busy loops to ensure that the asynchronous operations are
correctly terminated.

closes #28446
2018-02-01 16:30:17 +01:00
olcbean 344db6c0b1 Java high-level REST client : minor fixes (#28467)
Corrected method name and a parameter name
2018-02-01 11:50:43 +01:00
markharwood 77d2dd203e
Search - add allow_partial_search_results flag with default setting false (#28440)
Adds allow_partial_search_results flag to search requests with default setting = true.
When false, will error if search either timeouts, has partial errors or has missing shards rather
than returning partial search results. A cluster-level setting provides a default for search requests with no flag.

Closes #27435
2018-01-31 15:51:29 +00:00
Christoph Büscher 6731c76900
Add ranking evaluation API to High Level Rest Client (#28357)
This change adds support for the new ranking evaluation API to the High Level Rest Client.
This mostly means adding support for parsing the various response objects back from the
REST representation. It includes one change to the response syntax where previously we didn't
print the type of the metric details section but we now need it to pick the right parser to
parse this section back.

Closes #28198
2018-01-30 17:48:09 +01:00
Luca Cavanna b779fb392b
[TEST] add test for case where search source is null (#28398)
Closes #28393
2018-01-30 13:19:43 +01:00
Luca Cavanna 2c99bfc947
REST high-level client: Fix parsing of script fields (#28395)
Script fields can get a bit more complicated than just stored fields. A script can return null, an object and also an array. Extended parsing to support such valid values. Also renamed util method from `parseStoredFieldsValue` to `parseFieldsValue` given that it can parse stored fields but also script fields, anything that's returned as `fields`.

Closes #28380
2018-01-30 13:19:08 +01:00
olcbean e3846a9c06 Java high-level REST : minor code clean up (#28409) 2018-01-29 13:22:26 +01:00
olcbean aad750ec3e High level rest client : minor code clean up (#28386) 2018-01-26 17:13:21 +01:00
Luca Cavanna fd66c94ce1
REST high-level client: add support for exists alias (#28332)
Relates to #27205
2018-01-25 14:53:27 +01:00
Luca Cavanna 45e1fe8bfc
REST high-level client: move to POST when calling API to retrieve which support request body (#28342)
It has been pointed out that GET with body may cause problems to some proxies. We are then switching to POST the API that retrieve info and support a request body.

Closes #28326
2018-01-25 14:34:56 +01:00
olcbean 9db23e48cd Add Indices Aliases API to the high level REST client (#27876)
Relates to #27205
2018-01-25 14:34:06 +01:00
Catalin Ursachi cf61d792b2 Added Put Mapping API to high-level Rest client (#27869)
Relates to #27205
2018-01-23 11:03:32 +01:00
Adrien Grand 700d9ecc95
Remove the `update_all_types` option. (#28288)
This option is not useful in 7.x since no indices may have more than one type
anymore.
2018-01-22 12:03:07 +01:00
javanna 0dfb65a6ee [TEST] fix RequestTests#testSearch in case search source is not set
The search request body can never be null as `SearchRequest` doesn't allow the inner `SearchSourceBuilder` to be null. Instead, when search source is not set, the request body is going to be an empty json object (`{}``)
2018-01-18 12:56:54 +01:00
Luca Cavanna b4c1c4a78c
REST high-level client: remove index suffix from indices client method names (#28263)
Today, the way to call them API under the indices namespace is by doing e.g. `client.indices().createIndex()`. Our spec define the API under the indices namespace as e.g. `indices.create`, hence there is no need to repeat the index suffix for each method as that is already defined by the namespace. Using the `index` suffix in each method was an oversight which must be corrected.
2018-01-18 11:57:24 +01:00
olcbean b98514c6d9 Add Close Index API to the high level REST client (#27734)
Add support for _close endpoint to the high level REST client

Relates to #27205
2018-01-17 11:47:08 +01:00
Martijn van Groningen 853f7e8780
Added multi get api to the high level rest client.
Relates to #27205
2018-01-16 17:27:02 +01:00
Tim Brooks 3895add2ca
Introduce elasticsearch-core jar (#28191)
This is related to #27933. It introduces a jar named elasticsearch-core
in the lib directory. This commit moves the JarHell class from server to
elasticsearch-core. Additionally, PathUtils and some of Loggers are
moved as JarHell depends on them.
2018-01-15 09:59:01 -07:00
Tim Brooks 99f88f15c5
Rename core module to server (#28180)
This is related to #27933. It renames the core module to server. This is
the first step towards introducing an elasticsearch-core jar.
2018-01-11 11:30:43 -07:00
olcbean fd45a46ce8 Deprecate `isShardsAcked()` in favour of `isShardsAcknowledged()` (#27819)
Several responses include the shards_acknowledged flag (indicating whether the
requisite number of shard copies started before the completion of the operation)
and there are two different getters used : isShardsAcknowledged() and isShardsAcked().

This PR deprecates the isShardsAcked() in favour of isShardsAcknowledged() in 
CreateIndexResponse, RolloverResponse and CreateIndexClusterStateUpdateResponse.

Closes #27784
2018-01-08 10:57:45 +01:00
Daniel Mitterdorfer 8085ec504b Upgrade Gradle Shadow plugin to 2.0.2
With this commit we upgrade the Gradle Shadow plugin that is used in our
benchmarks to version 2.0.2. This version does not use APIs that are
deprecated in Gradle 4.x.
2017-12-29 10:57:11 +01:00
Maxime Gréau 771defb97c
Build: Add 3rd party dependencies report generation (#27727)
* Adds task dependenciesInfo to BuildPlugin to generate a CSV file with dependencies information (name,version,url,license)
* Adds `ConcatFilesTask.groovy` to concatenates multiple files into one
* Adds task `:distribution:generateDependenciesReport` to concatenate `dependencies.csv` files into a single file (`es-dependencies.csv` by default)

 # Examples:
      $ gradle dependenciesInfo :distribution:generateDependenciesReport

 ## Use `csv` system property to customize the output file path
     $ gradle dependenciesInfo :distribution:generateDependenciesReport -Dcsv=/tmp/elasticsearch-dependencies.csv

 ## When branch is not master, use `build.branch` system property to generate correct licenses URLs
     $ gradle dependenciesInfo :distribution:generateDependenciesReport -Dbuild.branch=6.x -Dcsv=/tmp/elasticsearch-dependencies.csv
2017-12-26 10:51:47 +01:00
Tanguy Leroux 0f80e7c5f6
[Test] Fix IndicesClientDocumentationIT (#27899)
The last operation executed in IndicesClientDocumentationIT.testCreate()
 is an asynchronous index creation. Because nothing waits for its
 completion, on slow machines the index can sometimes be created after
 the testCreate() test is finished, and it can fail the following test.

 Closes #27754
2017-12-20 09:31:10 +01:00
David Turner b1039164a1 Mute MigrationDocumentationIT#testClusterHealth()
Relates #27754
2017-12-19 12:16:49 +00:00
Jay Modi 717e2ddf42
Do not use system properties when building the HttpAsyncClient (#27829)
This commit removes the usage of system properties for the HttpAsyncClient as this overrides some
defaults that we intentionally change. In order to set the default SSLContext to the system context
we set the SSLContext on the builder explicitly.

Closes #27827
2017-12-15 12:54:22 -07:00
Christoph Büscher bb14b8f7c5 Merge branch 'rankeval'
This commit adds a new module that provides an endpoint that can be used to
evaluate search ranking results.

Closes #19195
2017-12-14 16:45:03 +01:00
Christoph Büscher 5406a9f30d Add rank-eval module to transport client and HL client dependencies 2017-12-13 18:05:43 +01:00
Tanguy Leroux 28f6512319
[Test] Fix MigrationDocumentationIT.testClusterHealth (#27774)
Closes #27754
2017-12-13 16:47:01 +01:00
Tanguy Leroux b69923f112
Remove some unused code (#27792)
This commit removes some unused code.
2017-12-13 16:45:55 +01:00
javanna e01643126b [TEST] extend wait_for_active_shards randomization to include 'all' value
This was already changed in 6.x as part of the backport of the recently added open and create index API. wait_for_active_shards can be a number but also "all", with this commit we verify that providing "all" works too.
2017-12-11 14:27:12 +01:00
javanna 27e157f67c [TEST] remove code duplications in RequestTests 2017-12-08 10:51:27 +01:00
olcbean bcc33f391f Add Open Index API to the high level REST client (#27574)
Add _open to the high level REST client

Relates to #27205
2017-12-07 18:16:03 +01:00
Catalin Ursachi f823cea79c Added Create Index support to high-level REST client (#27351)
Relates to #27205
2017-12-07 11:39:59 +01:00
Martijn van Groningen 4d78e1a9ad
Added msearch api to high level client 2017-12-05 10:17:47 +01:00
Luca Cavanna 3e8ca38fca
Deprecate the transport client in favour of the high-level REST client (#27085) 2017-12-01 12:24:16 +01:00
Simon Willnauer fadbe0de08
Automatically prepare indices for splitting (#27451)
Today we require users to prepare their indices for split operations.
Yet, we can do this automatically when an index is created which would
make the split feature a much more appealing option since it doesn't have
any 3rd party prerequisites anymore.

This change automatically sets the number of routinng shards such that
an index is guaranteed to be able to split once into twice as many shards.
The number of routing shards is scaled towards the default shard limit per index
such that indices with a smaller amount of shards can be split more often than
larger ones. For instance an index with 1 or 2 shards can be split 10x
(until it approaches 1024 shards) while an index created with 128 shards can only
be split 3x by a factor of 2. Please note this is just a default value and users
can still prepare their indices with `index.number_of_routing_shards` for custom
splitting.

NOTE: this change has an impact on the document distribution since we are changing
the hash space. Documents are still uniformly distributed across all shards but since
we are artificually changing the number of buckets in the consistent hashign space
document might be hashed into different shards compared to previous versions.

This is a 7.0 only change.
2017-11-23 09:48:54 +01:00
Jim Ferenczi 6319424e4a
Move composite aggregation to core (#27474)
This change removes the module named aggs-composite and adds the `composite` aggs
as a core aggregation. This allows other plugins to use this new aggregation
and simplifies the integration in the HL rest client.
2017-11-21 13:31:01 +01:00
Luca Cavanna 29450de7b5
Cross Cluster Search: make remote clusters optional (#27182)
Today Cross Cluster Search requires at least one node in each remote cluster to be up once the cross cluster search is run. Otherwise the whole search request fails despite some of the data (either local and/or remote) is available. This happens when performing the _search/shards calls to find out which remote shards the query has to be executed on. This scenario is different from shard failures that may happen later on when the query is actually executed, in case e.g. remote shards are missing, which is not going to fail the whole request but rather yield partial results, and the _shards section in the response will indicate that.

This commit introduces a boolean setting per cluster called search.remote.$cluster_alias.skip_if_disconnected, set to false by default, which allows to skip certain clusters if they are down when trying to reach them through a cross cluster search requests. By default all clusters are mandatory.

Scroll requests support such setting too when they are first initiated (first search request with scroll parameter), but subsequent scroll rounds (_search/scroll endpoint) will fail if some of the remote clusters went down meanwhile.

The search API response contains now a new _clusters section, similar to the _shards section, that gets returned whenever one or more clusters were disconnected and got skipped:

"_clusters" : {
    "total" : 3,
    "successful" : 2,
    "skipped" : 1
}
Such section won't be part of the response if no clusters have been skipped.

The per cluster skip_unavailable setting value has also been added to the output of the remote/info API.
2017-11-21 11:41:47 +01:00
Mayya Sharipova 858b2c7cb8
Standardize underscore requirements in parameters (#27414)
Stardardize underscore requirements in parameters across different type of
requests:
_index, _type, _source, _id keep their underscores
params like version and retry_on_conflict will be without underscores
Throw an error if older versions of parameters are used

BulkRequest, MultiGetRequest, TermVectorcRequest, MoreLikeThisQuery
were changed

Closes #26886
2017-11-17 15:31:52 -05:00
Jim Ferenczi 623367d793
Add composite aggregator (#26800)
* This change adds a module called `aggs-composite` that defines a new aggregation named `composite`.
The `composite` aggregation is a multi-buckets aggregation that creates composite buckets made of multiple sources.
The sources for each bucket can be defined as:
  * A `terms` source, values are extracted from a field or a script.
  * A `date_histogram` source, values are extracted from a date field and rounded to the provided interval.
This aggregation can be used to retrieve all buckets of a deeply nested aggregation by flattening the nested aggregation in composite buckets.
A composite buckets is composed of one value per source and is built for each document as the combinations of values in the provided sources.
For instance the following aggregation:

````
"test_agg": {
  "terms": {
    "field": "field1"
  },
  "aggs": {
    "nested_test_agg":
      "terms": {
        "field": "field2"
      }
  }
}
````
... which retrieves the top N terms for `field1` and for each top term in `field1` the top N terms for `field2`, can be replaced by a `composite` aggregation in order to retrieve **all** the combinations of `field1`, `field2` in the matching documents:

````
"composite_agg": {
  "composite": {
    "sources": [
      {
	"field1": {
          "terms": {
              "field": "field1"
            }
        }
      },
      {
	"field2": {
          "terms": {
            "field": "field2"
          }
        }
      },
    }
  }
````

The response of the aggregation looks like this:

````
"aggregations": {
  "composite_agg": {
    "buckets": [
      {
        "key": {
          "field1": "alabama",
          "field2": "almanach"
        },
        "doc_count": 100
      },
      {
        "key": {
          "field1": "alabama",
          "field2": "calendar"
        },
        "doc_count": 1
      },
      {
        "key": {
          "field1": "arizona",
          "field2": "calendar"
        },
        "doc_count": 1
      }
    ]
  }
}
````

By default this aggregation returns 10 buckets sorted in ascending order of the composite key.
Pagination can be achieved by providing `after` values, the values of the composite key to aggregate after.
For instance the following aggregation will aggregate all composite keys that sorts after `arizona, calendar`:

````
"composite_agg": {
  "composite": {
    "after": {"field1": "alabama", "field2": "calendar"},
    "size": 100,
    "sources": [
      {
	"field1": {
          "terms": {
            "field": "field1"
          }
        }
      },
      {
	"field2": {
          "terms": {
            "field": "field2"
          }
	}
      }
    }
  }
````

This aggregation is optimized for indices that set an index sorting that match the composite source definition.
For instance the aggregation above could run faster on indices that defines an index sorting like this:

````
"settings": {
  "index.sort.field": ["field1", "field2"]
}
````

In this case the `composite` aggregation can early terminate on each segment.
This aggregation also accepts multi-valued field but disables early termination for these fields even if index sorting matches the sources definition.
This is mandatory because index sorting picks only one value per document to perform the sort.
2017-11-16 15:13:36 +01:00
Nicholas Knize 8904fc8210 [Geo] Decouple geojson parse logic from ShapeBuilders
This is the first step to supporting WKT (and other future) format(s). The ShapeBuilders are quite messy and can be simplified by decoupling the parse logic from the build logic. This commit refactors the parsing logic into its own package separate from the Shape builders. It also decouples the GeoShapeType into a standalone enumerator that is responsible for validating the parsed data and providing the appropriate builder. This future-proofs the code making it easier to maintain and add new shape types.
2017-11-10 14:37:58 -06:00
Tanguy Leroux 43e7a4a349
Upgrade to Jackson 2.8.10 (#27230)
While it's not possible to upgrade the Jackson dependencies 
to their latest versions yet (see #27032 (comment) for more) 
it's still possible to upgrade to the latest 2.8.x version.
2017-11-06 10:20:05 +01:00
Luca Cavanna 5d7d01ba75
Adjust RestHighLevelClient method modifiers (#27238)
RestHighLevelClient can be subclassed to add support for additional methods, but its public and protected methods should be final.
2017-11-06 10:05:40 +01:00
Catalin Ursachi 8bf33241ed Add Delete Index API support to high-level REST client (#27019)
Relates to #25847
2017-10-26 09:52:46 +02:00
Ryan Ernst 2a8452b513 Reindex: Fix headers in reindex action (#26937)
The headers passed to reindex were skipped except for the last one. This
commit fixes the copying of the headers, as well as adds a base test
case for rest client builders to access the headers within the built
rest client.

relates #22976
2017-10-25 16:37:01 -07:00
Luca Cavanna 8caf7d4ff8 Decouple BulkProcessor from ThreadPool (#26727)
Introduce minimal thread scheduler as a base class for `ThreadPool`. Such a class can be used from the `BulkProcessor` to schedule retries and the flush task. This allows to remove the `ThreadPool` dependency from `BulkProcessor`, which requires to provide settings that contain `node.name` and also needed log4j for logging. Instead, it needs now a `Scheduler` that is much lighter and gets automatically created and shut down on close.

Closes #26028
2017-10-25 10:30:23 +02:00
Tanguy Leroux 463e7e6fa3 Revert "Upgrade to Jackson 2.9.2 (#27032)"
This reverts commit 0b9acc5ace.
2017-10-20 08:25:41 +02:00
Tanguy Leroux 0b9acc5ace Upgrade to Jackson 2.9.2 (#27032)
Upgrade to Jackson 2.9.2 and also use a boolean `closed` flag to
indicate that a FastStringReader instance is closed, so that length
is still correctly reported after the reader is closed.
2017-10-19 15:15:02 +02:00
Reto Schüttel ab94150a23 Use a dedicated ThreadGroup in rest sniffer (#26897)
This change adds a dedicated thread group, configures threads with a corresponding thread name and starts all threads as daemon threads.
2017-10-12 15:20:51 +02:00
Nhat bf4c3642b2 remove _primary and _replica shard preferences (#26791)
The shard preference _primary, _replica and its variants were useful
for the asynchronous replication. However, with the current impl, they
are no longer useful and should be removed.

Closes #26335
2017-10-08 11:03:06 -04:00
Luca Cavanna 25f154b8c6 [DOCS] improve encrypted communication example in java REST low-level client docs (#26705)
Include SSLContext creation and add the missing "https" scheme to the host creation.

Relates to #26534
Closes #26670
2017-09-25 18:03:57 +02:00
Tanguy Leroux b3819e7f30 Make RestHighLevelClient's Request class public (#26627)
Request class is currently package protected, making it difficult for
the users to extend the RestHighLevelClient and to use its protected
methods to execute requests. This commit makes the Request class public
and changes few methods of RestHighLevelClient to be protected.
2017-09-20 11:36:10 +02:00
Boaz Leskes 0814ea3200 fix testSniffNodes to use the new error message
relates to #26564
2017-09-16 10:43:48 +03:00
Itamar Syn-Hershko e9deb62546 Better message text for ResponseException
This avoids messages with malformed URLs, like
"org.elasticsearch.client.ResponseException: PUT
http://127.0.0.1:9502customer: HTTP/1.1 400 Bad Request".

Relates #26564
2017-09-14 21:10:12 -04:00
Tanguy Leroux ad355f3e0b Forbid direct usage of ContentType.create() methods (#26457)
It's easy to create a wrong Content-Type header when converting a 
XContentType to a Apache HTTP ContentType instance.

This commit the direct usages of ContentType.create() methods in favor of a Request.createContentType(XContentType) method that does the right thing.

Closes #26438
2017-09-06 09:58:46 +02:00
Tal Levy ed151d829d Migrate Search requests to use Writeable reading strategies (#26428)
Migrates many SearchRequest objects to use Writeable conventions and rejects usage of `readFrom` in these new classes.
2017-08-30 11:00:33 -07:00
Michael Basnight cfd14cd2b8 Revert shading for the low level rest client (#26367)
At current, we do not feel there is enough of a reason to shade the low
level rest client. It caused problems with commons logging and IDE's
during the brief time it was used. We did not know exactly how many
users will need this, and decided that leaving shading out until we
gather more information is best. Users can still shade the jar
themselves. For information and feeback, see issue #26366.

Closes #26328

This reverts commit 3a20922046.
This reverts commit 2c271f0f22.
This reverts commit 9d10dbea39.
This reverts commit e816ef89a2.
2017-08-25 14:13:12 -05:00
Nik Everett b3edd11aa0 Allow plugins to plug rescore implementations (#26368)
This allows plugins to plug rescore implementations into
Elasticsearch. While this is a fairly expert thing to do I've
done my best to point folks to the QueryRescorer as one that at
least documents the tradeoffs that it makes. I've attempted to
limit the API surface area by removing `SearchContext` from the
exposed interface, instead exposing just the IndexSearcher and
`QueryShardContext`. I also tried to make some of the class names
more consistent and do some general cleanup while I was there.

I entertained the notion of moving the `QueryRescorer` to module.
After all, it'd be a wonderful test to prove that you can plug
rescore implementation into Elasticsearch if the only built in
rescore implementation is in the module. But I decided against it
because the new module would require a client jar and it'd require
moving some more things around. I think if we really want to do
it, we should do it as a followup.

I did, on the other hand, create an "example" rescore plugin which
should both be a nice example for anyone wanting to plug in their
own rescore implementation and servers as a good integration test
to make sure that you can indeed plug one in.

Closes #26208
2017-08-25 13:46:57 -04:00
Christoph Büscher ad8f359deb Register ip_range aggregation with the high level client (#26383)
The parser for the `ip_range` aggregation response is currently missing from the
NamedXContentRegistry in the high level rest client. Also changes the testing
around the expected number of parsers so we at least check that we register all
the parsers that we also test in InternalAggregationTestCase.
2017-08-25 13:10:03 +02:00
Marco Monaco f6bbc91c0d Register ParsedTopHits aggregation with the rest high level client (#26370) 2017-08-25 11:18:16 +02:00
Luca Cavanna 6d8e2c6d4c Make RestHighLevelClient Closeable and simplify its creation (#26180)
By making RestHighLevelClient Closeable, its close method will close the internal low-level REST client instance by default, which simplifies the way most users interact with the high-level client.

Its constructor accepts now a RestClientBuilder, which clarifies that the low-level REST client is internally created and managed.

It is still possible to provide an already built `RestClient` instance, but that can only be done by subclassing `RestHighLevelClient` and calling the protected constructor that accepts a `RestClient`. In such case a consumer has also to be provided, which controls what has to be done when the high-level client gets done.

Closes #26086
2017-08-24 09:39:41 +02:00
Luca Cavanna 1309dfd44d Add links to external classes in clients javadoc (#25998)
The client sniffer depends on the low-level REST client, while the Java high-level REST client and the transport client depend on Elasticsearch itself. Javadoc are not that useful unless they have links to the Elasticsearch classes in the latter case, and to the low-level REST client in the sniffer javadoc. This commit adds those links.
2017-08-17 21:03:47 +02:00
desmorto 292dd8f992 (refactor) some opportunities to use diamond operator (#25585)
* (refactor) some opportunities to use diamond operator

* Update ExceptionRetryIT.java

update typo
2017-08-15 16:36:42 -06:00
Tanguy Leroux 69f8641568 [Docs] Add documentation for search queries in high-level rest client (#25984) 2017-08-02 09:57:47 +02:00
Tanguy Leroux 9c8d3d3569 [Docs] Add migration notes for the high-level rest client (#25911) 2017-08-01 10:38:56 +02:00
Michael Basnight 3a20922046 Fix eclipse issues related to rest client shading (#25874)
* A cycle was detected in eclipse, and was fixed in the same fashion as
  core and core-tests.
* The rest client deps jar was not properly exported in the generated
  eclipse classpath file for rest client.

Relates #25208
2017-07-27 10:20:53 -05:00
Michael Basnight 2c271f0f22 Remove the shadow plugin apply in the rest client (#25921)
The low level rest client does not need the shadow plugin applied, it
only needs the plugin jar in the classpath, in order to create a
ShadowJar task.

Relates #25208
2017-07-27 09:22:18 -05:00
Michael Basnight 9d10dbea39 Fix rest client causing jarHell for gradle 3.5+ (#25892)
The configuration removed from the runtime configuration did not
properly remove the deps jar from gradle versions > 3.3. The rest client
now removes both the 3.3 and 3.3+ configurations so this works on both
versions of gradle.

Closes #25884
Relates #25208
2017-07-26 11:25:25 +02:00
Tanguy Leroux 90ebaaa9a8 [Docs] Add profile section to the Search API documentation (#25880) 2017-07-26 10:31:46 +02:00
Michael Basnight e816ef89a2 Shade external dependencies in the rest client jar
This commit removes all external dependencies from the rest client jar
and shades them in an 'org.elasticsearch.client' package within the jar
using shadowJar gradle plugin. All projects that depended on the
existing jar have been converted to using the 'org.elasticsearch.client'
package prefixes to interact with the rest client.

Closes #25208
2017-07-24 12:55:43 -05:00
javanna c4c1e909a3 [TEST] SearchDocumentationIT#testSearch to sort on _uid instead of _id 2017-07-21 11:15:21 +02:00
Jay Modi 3e4bc027eb RestClient uses system properties and system default SSLContext (#25757)
This commit calls the `useSystemProperties` method on the HttpAsyncClientBuilder so that the jvm
system properties are used. The primary reason for doing this is to ensure the builder uses the
system default SSLContext rather than the default instance created by the http client library.

Closes #23231
2017-07-20 07:36:56 -06:00
Adrien Grand f1ff7f2454 Require a field when a `seed` is provided to the `random_score` function. (#25594)
We currently use fielddata on the `_id` field which is trappy, especially as we
do it implicitly. This changes the `random_score` function to use doc ids when
no seed is provided and to suggest a field when a seed is provided.

For now the change only emits a deprecation warning when no field is supplied
but this should be replaced by a strict check on 7.0.

Closes #25240
2017-07-19 14:11:15 +02:00
Christoph Büscher 43bfe06759 [Docs] Add sorting and source filtering section to client docs (#25767) 2017-07-18 16:58:46 +02:00
Christoph Büscher 56b1250a34 [Docs] Adding highlighting section to high level client docs (#25751)
Adding a section about how to use highlighting in the SearchSourceBuilder and
how to retrieve highlighted fragments from the SearchResponse.
2017-07-17 19:30:58 +02:00
Christoph Büscher 5387ed00d2 [Docs] Adding suggestion sections to high level client docs (#25724)
This adds a section about how to add suggestions to the SearchSourceBuilder and
how to retrieve them from a SearchResponse.
2017-07-14 18:33:28 +02:00
Christoph Büscher f809a12493 [Docs] Adding aggregation sections to high level client docs (#25707)
This adds a section about how to add aggregations to the SearchSourceBuilder and how
to retrieve them from a SearchRepsonse to the documentation for the high level rest client.
2017-07-14 12:47:47 +02:00
Martijn van Groningen 02fad9ac8c
docs: updated java client api to take this into account too to take into account the p/c queries are in parent-join module
Closes #25624
2017-07-13 11:24:22 +02:00
Luca Cavanna ec66d655b5 Rename client artifacts (#25693)
It was brought up that our current client artifacts have generic names like 'rest' that may cause conflicts with other artifacts.

This commit renames:

- rest -> elasticsearch-rest-client
- sniffer -> elasticsearch-rest-client-sniffer
- rest-high-level -> elasticsearch-rest-high-level-client

A couple of small changes are also preparing the high level client for its first release.

Closes #20248
2017-07-13 09:44:25 +02:00
Simon Willnauer e81804cfa4 Add a shard filter search phase to pre-filter shards based on query rewriting (#25658)
Today if we search across a large amount of shards we hit every shard. Yet, it's quite
common to search across an index pattern for time based indices but filtering will exclude
all results outside a certain time range ie. `now-3d`. While the search can potentially hit
hundreds of shards the majority of the shards might yield 0 results since there is not document
that is within this date range. Kibana for instance does this regularly but used `_field_stats`
to optimize the indexes they need to query. Now with the deprecation of `_field_stats` and it's upcoming removal a single dashboard in kibana can potentially turn into searches hitting hundreds or thousands of shards and that can easily cause search rejections even though the most of the requests are very likely super cheap and only need a query rewriting to early terminate with 0 results.

This change adds a pre-filter phase for searches that can, if the number of shards are higher than a the `pre_filter_shard_size` threshold (defaults to 128 shards), fan out to the shards
and check if the query can potentially match any documents at all. While false positives are possible, a negative response means that no matches are possible. These requests are not subject to rejection and can greatly reduce the number of shards a request needs to hit. The approach here is preferable to the kibana approach with field stats since it correctly handles aliases and uses the correct threadpools to execute these requests. Further it's completely transparent to the user and improves scalability of elasticsearch in general on large clusters.
2017-07-12 22:19:20 +02:00
Christoph Büscher f3e7a1c4a4 Adding basic search request documentation for high level client (#25651) 2017-07-12 17:06:46 +02:00
Jack Conradson d2b4f7ac5a Disallow lang to be used with Stored Scripts (#25610)
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.
2017-07-12 07:55:57 -07:00
Colin Goodheart-Smithe 3a5a54e83e Collapses package structure for some bucket aggs (#25579)
This change collapses some of the packages for the bucket aggregations into their parent packages. This was done for the following aggregations:
* The variants of the range aggregation (geo_distance, date and ip) were moved into the `o.e.s.a.bucket.range` package
* The `o.e.s.a.bucket.terms.support` package was removed and the classes were moved to `o.e.s.a.bucket.terms`
* The filter aggregation was moved to `o.e.s.a.bucket.filter`

Since this PR is already relatively large with only the above changes subsequent PRs will do similar operations on relevant metric and pipeline aggregations

Relates to #22868
2017-07-10 15:08:15 +01:00
Luca Cavanna 4f4f9e0af1 [DOCS] revise high level client Search Scroll API docs (#25599)
Moved the full example at the end of the page, reduced the number of bullet points for it, and added smaller examples at the beginning of the page.
2017-07-07 17:48:58 +02:00
Tanguy Leroux b06a744b05 [Docs] Document Scroll API for Java High Level REST Client (#25554)
This commit adds documentation for _search/scroll and clear scroll methods of the high level Java REST client
2017-07-07 12:19:33 +02:00
Tanguy Leroux d9bc0f48b4 [Docs] Document Bulk Processor for Java High Level REST Client (#25572) 2017-07-06 17:05:10 +02:00
Luca Cavanna bd6d7f1495 Update REST client deps license and notice files (#25573)
We previously grouped all the license and notice files for httpcore, httpcore-nio, httpclient and httpasyncclient under the same license and notice file. There were though subtle differences between those which we didn't keep track of. For instance the httpcore license file has slightly changed since 4.4 which we have missed to track.

This commit goes back to having one license and notice file for each jar, to be completely sure that each dependency is associated with exactly the right licene and notice file.

 Closes #25567
2017-07-06 16:52:17 +02:00
Luca Cavanna 26bc900058 [DOCS] Rewrite low-level REST client docs and verify snippets compile (#25559)
Using the infra that we now have in place, we can convert the low-level REST client docs so that they extract code snippets from real Java classes. This way we make sure that all the snippets properly compile. Compared to the high level REST client docs, in this case we don't run the tests themselves, as that would require depending on test-framework which requires java 8 while the low-level REST client is compatible with java 7. I think that compiling snippets is enough for now.
2017-07-06 10:05:50 +02:00
Tanguy Leroux fefcae3d45 [Docs] Document Update API for Java High Level REST Client (#25536)
This commit adds documentation for Java High Level REST Client's Update API.
2017-07-05 12:16:42 +02:00
Luca Cavanna 6f8c0453bc [DOCS] add docs for high level client get method (#25538)
Document high level client get method
2017-07-05 11:57:57 +02:00
Luca Cavanna d91d96e06e [DOCS] update index, type and id for indexed documents in high level client docs (#25546)
Especially the type change is convenient as it becomes `doc`
2017-07-05 11:32:59 +02:00
Tanguy Leroux 3da3632021 [Docs] Document Bulk API for Java High Level REST Client (#25532)
This commit adds documentation for Java High Level REST Client's Bulk API.
2017-07-05 09:26:26 +02:00
Luca Cavanna 99fef2490a [DOCS] add docs for REST high level client index method (#25501)
This commit restructures the existing high level client docs, adapts the existing delete method docs and adds docs for the index method.
2017-07-03 10:48:16 +02:00
Drew Raines 6deb18c0de Preliminary support for ARM
This commit adds preliminary support for 64-bit ARM architectures.

Relates #25318
2017-06-30 14:22:20 -04:00
Christoph Büscher e82be00890 Adapt `SearchIT#testSearchWithParentJoin` to new join field (#25379)
In order to remove changing the "index.mapping.single_type" setting in this test
we need to change it to use the new join field.

Closes #25372
2017-06-23 19:50:56 +02:00
Tanguy Leroux e4f4886d40 [Test] Extend parsing checks for DocWriteResponses (#25257)
This commit changes the parsing logic of DocWriteResponse, ReplicationResponse
and GetResult so that it skips any unknown additional fields (for forward compatibility 
reasons). This affects the IndexResponse, UpdateResponse,DeleteResponse and 
GetResponse objects.
2017-06-19 13:19:09 +02:00
Nik Everett 21b1db2965 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.
2017-06-16 17:19:14 -04:00
Nik Everett 7b358190d6 Remove assemble task when not used for publishing (#25228)
Removes the `assemble` task from projects that are not published.
This should speed up `gradle assemble` by skipping projects that
don't need to be built. Which is useful because `gradle assemble`
is how we cut releases.
2017-06-16 11:46:34 -04:00
Tanguy Leroux 27f1206999 Use SPI in High Level Rest Client to load XContent parsers (#25098)
This commit adds a NamedXContentProvider interface that can 
be implemented by plugins or modules using Java's SPI feature 
in order to provide additional NamedXContent parsers to external
applications like the Java High Level Rest Client.
2017-06-15 12:50:02 +02:00
Luca Cavanna 60687734a3 [TEST] test that low level REST client leaves path untouched (#25193)
Relates to #24987
2017-06-15 11:32:26 +02:00
Tanguy Leroux 29502107b9 [Test] Add test for custom requests in High Level Rest Client (#25106)
This commit adds a test that tests and demonstrates how
{@link RestHighLevelClient} can be extended to support
custom endpoint.
2017-06-09 17:03:57 +02:00
Tanguy Leroux 348884bda4 [Test] Remove redundant assertions in SearchIT
The SearchResponse's took time is already checked by the assertSearchHeader()
method so it does not need to be checked twice.
2017-06-09 09:44:57 +02:00
Christoph Büscher 542b0616ac [Test] Fix high-level-rest SearchIT so that quick responses don't fail the test 2017-06-08 16:06:39 +02:00
Nik Everett 64cbb93727 Fix compilation in eclipse (#25107)
It looks like eclipse blows up when you take a method reference to
varargs method in a finally method:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=517951
2017-06-07 11:36:29 -04:00
Luca Cavanna d47d47928b Add support for clear scroll to high level REST client (#25038) 2017-06-06 14:30:42 +02:00
Alex Benusovich 5463294ec4 Fixed NPEs caused by requests without content. (#23497)
REST handlers that require a body will throw an an ElasticsearchParseException "request body required".
REST handlers that require a body OR source param will throw an ElasticsearchParseException "request body or source param required".
Replaced asserts in BulkRequest parsing code with a more descriptive IllegalArgumentException if the line contains an empty object.
Updated bulk REST test to verify an empty action line is rejected properly.
Updated BulkRequestTests with randomized testing for an empty action line.
Used try-with-resouces for XContentParser in AbstractBulkByQueryRestHandler.
2017-06-05 09:08:14 -06:00
olcbean 6dea5f14c3 Java api: Remove unneeded getTookInMillis method (#23923)
Some response classes in the java api expose both `getTook()` which returns a `TimeValue` and `getTookInMillis` which returns a `long` value. `getTook()` is enough as one can do `getTook().millis()` to obtain the same result as `getTookInMillis()`, which can be removed.
2017-06-02 11:11:05 +02:00
Colin Goodheart-Smithe 779fb9a1c0 Adds nodes usage API to monitor usages of actions (#24169)
* Adds nodes usage API to monitor usages of actions

The nodes usage API has 2 main endpoints

/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.

At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.

Still to do:

* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation

* Rafactors UsageService so counts are done by the handlers

* Fixing up docs tests

* Adds a name to all rest actions

* Addresses review comments
2017-06-02 08:46:38 +01:00
javanna 45bdaeced5 [TEST] fix SearchIT assertion to also accept took set to 0
Closes #24997
2017-06-01 15:29:48 +02:00
Luca Cavanna 856235fac2 Add searchScroll method to high level REST client (#24938) 2017-06-01 10:56:17 +02:00
Luca Cavanna c120f8af89 Add search method to high level REST client (#24796)
Relates to #23331
2017-05-29 12:47:34 +02:00
olcbean e08e92d934 Deleting a document from a non-existing index creates the should not auto create it, unless using EXTERNAL* versioning (#24518)
Currently a `delete document` request against a non-existing index actually **creates** this index.

With this change the `delete document` no longer creates the previously non-existing index and throws an `index_not_found` exception instead.

However as discussed in https://github.com/elastic/elasticsearch/pull/15451#issuecomment-165772026, if an external version is explicitly used, the current behavior is preserved and the index is still created and the document is marked for deletion.

Fixes #15425
2017-05-22 10:00:22 +03:00
Jason Tedor e71a3ea1bb Disable the Netty recycler in the client
The Netty recycler is nothing but trouble, so let us disable this by
default in the client too.

Relates #24793
2017-05-19 09:05:30 -04:00
Koen De Groote 905eb422f6 Use StringBuilder to construct a String instead of relying on appending where possible (#24753)
This PR revolves around places in the code where introducing a StringBuilder might make the construction
of a String easier to follow and also, maybe avoid a case where the compiler's very safe way of introducing 
StringBuilder instead of String might not always be optimal for performance.
2017-05-18 12:02:29 +02:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Jim Ferenczi e804817ce4 #24638 Add the parent-join plugin in the prebuilt transport client by default 2017-05-12 17:25:34 +02:00
Jim Ferenczi 4291880a29 [Tests] #24638 Fix rest high level client docs build by restoring hasChild and hasParent in the tests 2017-05-12 17:03:26 +02:00
Jim Ferenczi 279a18a527 Add parent-join module (#24638)
* Add parent-join module

This change adds a new module named `parent-join`.
The goal of this module is to provide a replacement for the `_parent` field but as a first step this change only moves the `has_child`, `has_parent` queries and the `children` aggregation to this module.
These queries and aggregations are no longer in core but they are deployed by default as a module.

Relates #20257
2017-05-12 15:58:06 +02:00
Nik Everett db16038061 Docs: correct indentation on callout
I've aligned them all to 70 characters except this one....

Relates to #24354
2017-05-02 13:01:30 -04:00
Nik Everett 732741dd8d Build that java api docs from a test (#24354)
We've had `QueryDSLDocumentationTests` for a while but it had a very
hopeful comment at the top about how we want to make sure that the
example in the query-dsl docs match up with the test but we never
had anything that made *sure* that they did. This changes that!

Now the examples from the query-dsl docs are all built from the
`QueryDSLDocumentationTests`. All except for the percolator example
because that is hard to do as it stands now.

To make this easier this change moves `QueryDSLDocumentationTests`
from core and into the high level rest client. This is useful for
two reasons:
1. We expect the high level rest client to be able to use the builders.
2. The code that builds that docs doesn't check out all of
Elasticsearch. It only checks out certain directories. Since we're
already including snippets from that directory we don't have to
make any changes to that process.

Closes #24320
2017-05-02 13:00:56 -04:00
Nik Everett bc45d10e82 Remove most usages of 1-arg Script ctor (#24325)
The one argument ctor for `Script` creates a script with the
default language but most usages of are for testing and either
don't care about the language or are for use with
`MockScriptEngine`. This replaces most usages of the one argument
ctor on `Script` with calls to `ESTestCase#mockScript` to make
it clear that the tests don't need the default scripting language.

I've also factored out some copy and pasted script generation
code into a single place. I would have had to change that code
to use `mockScript` anyway, so it was easier to perform the
refactor.

Relates to #16314
2017-04-26 16:04:38 -04:00
Jason Tedor 972bdc09ee Reject empty IDs
When indexing a document via the bulk API where IDs can be explicitly
specified, we currently accept an empty ID. This is problematic because
such a document can not be obtained via the get API. Instead, we should
rejected these requets as accepting them could be a dangerous form of
leniency. Additionally, we already have a way of specifying
auto-generated IDs and that is to not explicitly specify an ID so we do
not need a second way. This commit rejects the individual requests where
ID is specified but empty.

Relates #24118
2017-04-15 10:36:03 -04:00
Luca Cavanna b73f87b0ea Make buffer limit configurable in HeapBufferedConsumerFactory (#23970)
The buffer limit should have been configurable already, but the factory constructor is package private so it is truly configurable only from the org.elasticsearch.client package. Also the HttpAsyncResponseConsumerFactory interface was package private, so it could only be implemented from the org.elasticsearch.client package.

Closes #23958
2017-04-10 12:27:42 +02:00
Nik Everett 1ad5398af7 Fix compilation in Eclipse (#23954)
Eclipse seems to have trouble with CrudIT. It throws an NPE back
to the user. I've filed a bug:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=514884
but for now we should work around it.
2017-04-06 20:40:55 -04:00
Tim Brooks 5b1fbe5e6c Decouple BulkProcessor from client implementation (#23373)
This commit modifies the BulkProcessor to be decoupled from the
client implementation. Instead it just takes a
BiConsumer<BulkRequest, ActionListener<BulkResponse>> that executes
the BulkRequest.
2017-04-05 12:12:43 -05:00
Jason Tedor 3136ed1490 Rename random ASCII helper methods
This commit renames the random ASCII helper methods in ESTestCase. This
is because this method ultimately uses the random ASCII methods from
randomized runner, but these methods actually only produce random
strings generated from [a-zA-Z].

Relates #23886
2017-04-04 11:04:18 -04:00
Nik Everett 23b8b97275 Use include-tagged macro for high level client docs (#23438)
This should make it simpler to include example snippets from
tests.
2017-03-20 14:50:19 -04:00
Tanguy Leroux 15c936ec02 Correctly parse BulkItemResponse.Failure's status (#23432)
Today the status is lost when parsing back a BulkItemResponse.Failure. This commit changes the BulkItemResponse.Failure parsing method so that it correctly instantiates a failure with the parsed status instead of realying on the parsed ElasticsearchException (that always return an internal server error status).
2017-03-02 12:37:07 +01:00
Jason Tedor cda0d5f86a Fix checkstyle violation in delete docs test
This commit addresses a checkstyle violation in
DeleteDocumentationIT.java.
2017-03-01 08:22:02 -05:00
David Pilato dc80a1fa75 Bad rendering of docs 2017-03-01 13:33:12 +01:00
David Pilato 2629c9896c Missing dot and parenthesis 2017-03-01 13:21:37 +01:00
David Pilato ee4a17a0e2 Adapt per Nik's comments 2017-03-01 11:51:06 +01:00
David Pilato 6c526adaed Merge branch 'master' into doc/hlclient-delete 2017-03-01 11:26:36 +01:00
Tanguy Leroux 02c919f0b0 [Tests] Fix RestClientTests 2017-02-28 09:38:59 +01:00
Tanguy Leroux 9da8531f60 RestClient asynchronous execution should not throw exceptions (#23307)
The current implementation of RestClient.performAsync() methods can throw exceptions before the request is asynchronously executed. Since it only throws unchecked exceptions, it's easy for the user/dev to forget to catch them. Instead I think async methods should never throw exceptions and should always call the listener onFailure() method.
2017-02-28 09:33:51 +01:00
Christoph Büscher 3215ac11ec Add info method to High Level Rest client (#23350)
This commit adds support for an info() method to the High Level Rest
client that returns the cluster information usually obtained by performing a
`GET hostname:9200` request.
2017-02-27 11:43:56 +01:00
David Pilato c373ed102f Extract documentation from test code 2017-02-24 17:38:18 +01:00
David Pilato fd509d015c Merge remote-tracking branch 'origin/master'
# Conflicts:
#	client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java
2017-02-24 11:52:56 +01:00
Luca Cavanna 7a96a38104 Add support for named xcontent parsers to high level REST client (#23328)
NamedXContentRegistry will be used by the high level REST client to parse aggregation responses, and any section whose class type depends on a json key. There are currently on entries in the standard registry, but soon aggs and most likely suggesters will be added. Also it is possible for subclasses to provide additional named xcontent parsers.
2017-02-24 11:46:59 +01:00
David Pilato 4ebc6dd0d0 Fix after last merge with master and apply last comments 2017-02-24 09:19:39 +01:00
David Pilato b680e62831 Merge branch 'master' into hlclient/add-delete-method
# Conflicts:
#	client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java
#	client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java
#	client/rest-high-level/src/test/java/org/elasticsearch/client/CrudIT.java
#	client/rest-high-level/src/test/java/org/elasticsearch/client/RequestTests.java
2017-02-24 08:58:22 +01:00
Tanguy Leroux 7e3c06c55d Add BulkRequest support to High Level Rest client (#23312)
This commit adds support for BulkRequest execution in the High Level Rest client.
2017-02-23 16:37:26 +01:00
Tanguy Leroux 09734e7469 Add UpdateRequest support to High Level Rest client (#23266)
This commit adds support for UpdateRequest to the High Level Rest client
2017-02-22 11:54:54 +01:00
Simon Willnauer ce625ebdcc Expose `batched_reduce_size` via `_search` (#23288)
In #23253 we added an the ability to incrementally reduce search results.
This change exposes the parameter to control the batch since and therefore
the memory consumption of a large search request.
2017-02-21 18:36:59 +01:00
David Pilato 9bde68e4d6 delete and index tests can share some part of the code 2017-02-20 10:43:28 +01:00
David Pilato bf8241eec5 Remove createSampleDocument method and use the sync'ed index method 2017-02-20 10:33:19 +01:00
David Pilato 0ee74f9c28 Remove createSampleIndexRequest method and provide ids 2017-02-20 10:09:44 +01:00
David Pilato efa28e05f8 Check that we correctly propagate delete request parameters 2017-02-20 09:56:42 +01:00
David Pilato b2ec4c1f17 Remove a non needed comment 2017-02-20 09:36:47 +01:00
Jay Modi b234644035 Enforce Content-Type requirement on the rest layer and remove deprecated methods (#23146)
This commit enforces the requirement of Content-Type for the REST layer and removes the deprecated methods in transport
requests and their usages.

While doing this, it turns out that there are many places where *Entity classes are used from the apache http client
libraries and many of these usages did not specify the content type. The methods that do not specify a content type
explicitly have been added to forbidden apis to prevent more of these from entering our code base.

Relates #19388
2017-02-17 14:45:41 -05:00
David Pilato 5aa2ab3bbc Fix after 1st review
* Sort alphabetically method names
* Add javadoc link to the ref guide
* Add more tests about routing, version and version type
2017-02-17 11:46:02 +01:00
Jason Tedor 2925a81cc9 Fix REST spec for exists
A previous change aligned the handling of the GET document and HEAD
document APIs. This commit aligns the specification for these two APIs
as well, and fixes a failing test.

Relates #23196
2017-02-16 08:56:38 -05:00
Jason Tedor 3f07fcef64 Mark CrudIT#testExists as awaits fix
This commit marks the test CrudIT#testExists as awaiting a fix due to a
behavior change from aligning the GET and HEAD document APIs.
2017-02-15 22:56:52 -05:00
David Pilato 14f04905f8 Merge with master 2017-02-15 16:14:33 +01:00
David Pilato 0bbcd94827 Merge branch 'master' into hlclient/add-delete-method
# Conflicts:
#	client/rest-high-level/src/main/java/org/elasticsearch/client/Request.java
#	client/rest-high-level/src/test/java/org/elasticsearch/client/CrudIT.java
2017-02-15 16:08:29 +01:00
David Pilato 7df1df24e8 Add delete API to the High Level Rest Client 2017-02-15 16:00:50 +01:00
Tanguy Leroux 86993ad759 Add Index API to High Level Rest Client (#23040)
This commit adds support for the Index API in the High Level Rest Client.
2017-02-15 13:53:41 +01:00
Simon Willnauer ecb01c15b9 Fold InternalSearchHits and friends into their interfaces (#23042)
We have a bunch of interfaces that have only a single implementation
for 6 years now. These interfaces are pretty useless from a SW development
perspective and only add unnecessary abstractions. They also require
lots of casting in many places where we expect that there is only one
concrete implementation. This change removes the interfaces, makes
all of the classes final and removes the duplicate `foo` `getFoo` accessors
in favor of `getFoo` from these classes.
2017-02-08 14:40:08 +01:00
Luca Cavanna 3551106aa7 Add get/exists method to RestHighLevelClient (#22706)
This commit adds support for get and exists api to the high level Java Rest Client. It also adds the infrastructure for other methods to be added in the future.
2017-02-04 14:54:26 +01:00
Tim Brooks f70188ac58 Remove connect SocketPermissions from core (#22797)
This is related to #22116. Core no longer needs `SocketPermission`
`connect`.

This permission is relegated to these modules/plugins:
- transport-netty4 module
- reindex module
- repository-url module
- discovery-azure-classic plugin
- discovery-ec2 plugin
- discovery-gce plugin
- repository-azure plugin
- repository-gcs plugin
- repository-hdfs plugin
- repository-s3 plugin

And for tests:
- mocksocket jar
- rest client
- httpcore-nio jar
- httpasyncclient jar
2017-02-03 09:39:56 -06:00
Jason Tedor 9a0b216c36 Upgrade checkstyle to version 7.5
This commit upgrades the checkstyle configuration from version 5.9 to
version 7.5, the latest version as of today. The main enhancement
obtained via this upgrade is better detection of redundant modifiers.

Relates #22960
2017-02-03 09:46:44 -05:00
Jay Modi 7520a107be Optionally require a valid content type for all rest requests with content (#22691)
This change adds a strict mode for xcontent parsing on the rest layer. The strict mode will be off by default for 5.x and in a separate commit will be enabled by default for 6.0. The strict mode, which can be enabled by setting `http.content_type.required: true` in 5.x, will require that all incoming rest requests have a valid and supported content type header before the request is dispatched. In the non-strict mode, the Content-Type header will be inspected and if it is not present or not valid, we will continue with auto detection of content like we have done previously.

The content type header is parsed to the matching XContentType value with the only exception being for plain text requests. This value is then passed on with the content bytes so that we can reduce the number of places where we need to auto-detect the content type.

As part of this, many transport requests and builders were updated to provide methods that
accepted the XContentType along with the bytes and the methods that would rely on auto-detection have been deprecated.

In the non-strict mode, deprecation warnings are issued whenever a request with body doesn't provide the Content-Type header.

See #19388
2017-02-02 14:07:13 -05:00
Chris Earle f0f75b187a Support Preemptive Authentication with RestClient (#21336)
This adds the necessary `AuthCache` needed to support preemptive authorization. By adding every host to the cache, the automatically added `RequestAuthCache` interceptor will add credentials on the first pass rather than waiting to do it after _each_ anonymous request is rejected (thus always sending everything twice when basic auth is required).
2017-01-24 11:34:05 -05:00
Nik Everett 6265ef1c1b Deguice rest handlers (#22575)
There are presently 7 ctor args used in any rest handlers:
* `Settings`: Every handler uses it to initialize a logger and
  some other strange things.
* `RestController`: Every handler registers itself with it.
* `ClusterSettings`: Used by `RestClusterGetSettingsAction` to
  render the default values for cluster settings.
* `IndexScopedSettings`: Used by `RestGetSettingsAction` to get
  the default values for index settings.
* `SettingsFilter`: Used by a few handlers to filter returned
  settings so we don't expose stuff like passwords.
* `IndexNameExpressionResolver`: Used by `_cat/indices` to
  filter the list of indices.
* `Supplier<DiscoveryNodes>`: Used to fill enrich the response
  by handlers that list tasks.

We probably want to reduce these arguments over time but
switching construction away from guice gives us tighter
control over the list of available arguments.

These parameters are passed to plugins using
`ActionPlugin#initRestHandlers` which is expected to build and
return that handlers immediately. This felt simpler than
returning an reference to the ctors given all the different
possible args.

Breaks java plugins by moving rest handlers off of guice.
2017-01-20 11:48:51 -05:00
Luca Cavanna 193111919c move ignore parameter support from yaml test client to low level rest client (#22637)
All the language clients support a special ignore parameter that doesn't get passed to elasticsearch with the request, but used to indicate which error code should not lead to an exception if returned for a specific request.

Moving this to the low level REST client will allow the high level REST client to make use of it too, for instance so that it doesn't have to intercept ResponseExceptions when the get api returns a 404.
2017-01-16 18:54:44 +01:00
Tim Brooks 7a8884d9fa Wrap rest httpclient with doPrivileged blocks (#22603)
This is related to #22116. A number of modules (reindex, etc) use the
rest client. The rest client opens connections using the apache http
client. To avoid throwing SecurityException when using the
SecurityManager these operations must be privileged. This is tricky
because connections are opened within the httpclient code on its
reactor thread. The way I confronted this was to wrap the creation
of the client (and creation of reactor thread) in a doPrivileged
block. The new thread inherits the existing security context.
2017-01-16 09:17:44 -06:00
Tanguy Leroux 3a3ce61186 Update Jackson to 2.8.6 (#22596)
closes #22266
2017-01-13 09:05:48 +01:00
Jason Tedor 126efea56c Upgrade to Netty 4.1.7
This commit upgrades the Netty dependency to version 4.1.7.Final,
picking up some important bug fixes.

Relates #22587
2017-01-12 10:58:21 -05:00
javanna ded694fc83 Make StatusToXContent extend ToXContentObject and rename it to StatusToXContentObject
This also allows to make RestToXContentListener require ToXContentObject rather than ToXContent
2017-01-06 23:31:48 +01:00
javanna 0a6827a5cc Make RestHighLevelClient non final 2017-01-05 11:29:41 +01:00
javanna f0181b19f5 add REST high level client gradle submodule and first simple method
The RestHighLevelClient class takes as as an argument a low level client instance RestClient. The first method added is ping, which returns true if the call to HEAD / went ok and false if an IOException was thrown. Any other exception gets bubbled up.

There are two kinds of tests, a unit test (RestHighLevelClientTests) that verifies the interaction between high level and low level client, and an integration test (MainActionIT) which relies on an externally started es cluster to send requests to.
2017-01-05 10:55:47 +01:00
javanna 1899aea9ca [TEST] move randomHeaders method from RestClientTestCase to RestClientTestUtil and simplify headers assertions 2017-01-05 10:55:47 +01:00
Tim B be22a250b6 Replace Socket, ServerSocket, and HttpServer usages in tests with mocksocket versions (#22287)
This integrates the mocksocket jar with elasticsearch tests. Mocksocket wraps actions requiring SocketPermissions in doPrivilege blocks. This will eventually allow SocketPermissions to be assigned to the mocksocket jar opposed to the entire elasticsearch codebase.
2017-01-04 14:38:51 -06:00
Jason Tedor 5185a9734d Add comment clarifying property setting in client
This commit adds a comment clarifying why we do not catch a security
exception in the pre-built transport client.
2016-12-23 18:14:36 -05:00
Jason Tedor 31bf279ec7 Tell Netty not to be unsafe in transport client
Today we ship with default jvm.options for server Elasticsearch that
prevents Netty from using some unsafe optimizations. Yet, the settings
do nothing for the transport client since it is embedded in other
applications that will not read and use those settings. This commit adds
these settings for the transport client, and is done so in a way that
still enables users to go unsafe if they want to go unsafe (they
shouldn't, but the option is there).

Relates #22284
2016-12-21 10:49:33 -05:00
Nik Everett 2aa89820f3 Don't use null charset in RequestLogger (#22197)
If the response comes back with a content type with a `null`
charset we were blindly using it, causing `NullPointerException`s.

Closes #22190
2016-12-15 11:19:51 -05:00
javanna 478cd78047 Remove unused import from RequestLogger
Not only was StringJoiner unused, it's also a class only available in java 1.8, which is a problem given that the REST client has minimum java required set to 1.7
2016-12-13 15:01:51 +01:00
Luca Cavanna 9be778c5e5 Warn log deprecation warnings received from server (#21895)
The warnings get printed out in a single line e.g. WARNING: request [DELETE http://localhost:9200/index/type/_api] returned 3 warnings:[this is warning number 0],[this is warning number 1],[this is warning number 2]
2016-12-12 12:11:42 +01:00
Nik Everett fc2060ba7e Don't close rest client from its callback (#22061)
If you try to close the rest client inside one of its callbacks then
it blocks itself. The thread pool switches the status to one that
requests a shutdown and then waits for the pool to shutdown. When
another thread attempts to honor the shutdown request it waits
for all the threads in the pool to finish what they are working on.
Thus thread a is waiting on thread b while thread b is waiting
on thread a. It isn't quite that simple, but it is close.

Relates to #22027
2016-12-09 10:39:51 -05:00
Nik Everett 2087234d74 Timeout improvements for rest client and reindex (#21741)
Changes the default socket and connection timeouts for the rest
client from 10 seconds to the more generous 30 seconds.

Defaults reindex-from-remote to those timeouts and make the
timeouts configurable like so:
```
POST _reindex
{
  "source": {
    "remote": {
      "host": "http://otherhost:9200",
      "socket_timeout": "1m",
      "connect_timeout": "10s"
    },
    "index": "source",
    "query": {
      "match": {
        "test": "data"
      }
    }
  },
  "dest": {
    "index": "dest"
  }
}
```

Closes #21707
2016-12-05 10:54:51 -05:00
Luca Cavanna 5b8bdba12e Remove subrequests method from CompositeIndicesRequest (#21873) 2016-11-30 15:03:58 +01:00
Simon Willnauer a9a2753f0b Add a HostFailureListener to notify client code if a node got disconnected (#21709)
Today there is no way to get notified if a node is disconnected. Client code
must poll the TransportClient constantly to detect that a node is not connected
anymore in order to react and add new nodes or notify altering etc. For instance
if a hostname  gets resolved to an IP but that host is disconnected clients want
to reconnect by resolving the hostname again which is a common situation in cloud
environments.

Closes #21424
2016-11-22 20:46:28 +01:00
Simon Willnauer de04aad994 Remove `modules/transport_netty_3` in favor of `netty_4` (#21590)
We kept `netty_3` as a fallback in the 5.x series but now that master
is 6.0 we don't need this or in other words all issues coming up with
netty 4 will be blockers for 6.0.
2016-11-17 12:44:42 +01:00
Ryan Ernst d14c470b89 Remove generics from ActionRequest
closes #21368
2016-11-14 15:32:01 -08:00
Luca Cavanna 293a3cab01 Rest client: don't reuse that same HttpAsyncResponseConsumer across multiple retries (#21378)
* Rest client: don't reuse that same HttpAsyncResponseConsumer across multiple retries

Turns out that AbstractAsyncResponseConsumer from apache async http client is stateful and cannot be reused across multiple requests. The failover mechanism was mistakenly reusing that same instance, which can be provided by users, across retries in case nodes are down or return 5xx errors. The downside is that we have to change the signature of two public methods, as HttpAsyncResponseConsumer cannot be provided directly anymore, rather its factory needs to be provided which is going to be used to create one instance of the consumer per request attempt.

Up until now we tested our RestClient against multiple nodes only in a mock environment, where we don't really send http requests. In that scenario we can verify that retries etc. work properly but the interaction with the http client library in a real scenario is different and can catch other problems. With this commit we also add an integration test that sends requests to multiple hosts, and some of them may also get stopped meanwhile. The specific test for pathPrefix was also removed as pathPrefix is now randomly applied by default, hence implicitly tested. Moved also a small test method that checked the validity of the path argument to the unit test RestClientSingleHostTests.

Also increase default buffer limit to 100MB and make it required in default consumer

The default buffer limit used to be 10MB but that proved not to be high enough for scroll requests (see reindex from remote). With this commit we increase the limit to 100MB and make it a bit more visibile in the consumer factory.
2016-11-08 16:42:42 +01:00
Yuhao Bi 79090431af Provide error message when request path is null 2016-11-02 09:32:15 +01:00
Nik Everett a612e5988e Bump reindex-from-remote's buffer to 200mb
It was 10mb and that was causing trouble when folks reindex-from-remoted
with large documents.

We also improve the error reporting so it tells folks to use a smaller
batch size if they hit a buffer size exception. Finally, adds some docs
to reindex-from-remote mentioning the buffer and giving an example of
lowering the size.

Closes #21185
2016-11-01 13:19:28 -04:00
Adrien Grand b3cc54cf0d Upgrade to lucene-6.3.0-snapshot-ed102d6 (#21150)
Lucene 6.3 is expected to be released in the next weeks so it'd be good to give
it some integration testing. I had to upgrade randomized-testing too so that
both Lucene and Elasticsearch are on the same version.
2016-10-28 14:47:15 +02:00
Areek Zillur 481f7909ae Merge branch 'master' into cleanup/transport_bulk 2016-10-11 16:04:47 -04:00
Areek Zillur 0e8b6532ec rename DocumentRequest to DocWriteRequest 2016-10-11 16:00:10 -04:00
Areek Zillur 396f80c963 Revert "rename DocumentRequest to DocumentWriteRequest"
This reverts commit b5079ce009.
2016-10-07 17:50:07 -04:00
Simon Willnauer 194a6b1df0 Remove LocalTransport in favor of MockTcpTransport (#20695)
This change proposes the removal of all non-tcp transport implementations. The
mock transport can be used by default to run tests instead of local transport that has
roughly the same performance compared to TCP or at least not noticeably slower.

This is a master only change, deprecation notice in 5.x will be committed as a
separate change.
2016-10-07 11:27:47 +02:00
Areek Zillur b5079ce009 rename DocumentRequest to DocumentWriteRequest 2016-10-06 05:05:59 -04:00
Areek Zillur bd4a03a426 Merge branch 'master' into cleanup/transport_bulk 2016-10-04 14:06:17 -04:00
Jason Tedor 51d53791fe Remove lenient URL parameter parsing
Today when parsing a request, Elasticsearch silently ignores incorrect
(including parameters with typos) or unused parameters. This is bad as
it leads to requests having unintended behavior (e.g., if a user hits
the _analyze API and misspell the "tokenizer" then Elasticsearch will
just use the standard analyzer, completely against intentions).

This commit removes lenient URL parameter parsing. The strategy is
simple: when a request is handled and a parameter is touched, we mark it
as such. Before the request is actually executed, we check to ensure
that all parameters have been consumed. If there are remaining
parameters yet to be consumed, we fail the request with a list of the
unconsumed parameters. An exception has to be made for parameters that
format the response (as opposed to controlling the request); for this
case, handlers are able to provide a list of parameters that should be
excluded from tripping the unconsumed parameters check because those
parameters will be used in formatting the response.

Additionally, some inconsistencies between the parameters in the code
and in the docs are corrected.

Relates #20722
2016-10-04 12:45:29 -04:00
Areek Zillur 248ac240ed Merge branch 'master' into cleanup/transport_bulk 2016-10-03 16:12:11 -04:00
Daniel Mitterdorfer 7a1ad716fa Update client benchmarks to log4j2 2016-09-23 15:39:18 +02:00
Ryan Ernst 85b8f29415 Build: Remove old maven deploy support (#20403)
* Build: Remove old maven deploy support

This change removes the old maven deploy that we have in parallel to
maven-publish, and makes maven-publish fully work with publishing to
maven local. Using `gradle publishToMavenLocal` should be used to
publish to .m2.

Note that there is an unfortunate hack that means for
zip artifacts we must first create/publish a dummy pom file, and then
follow that with the real pom file. It would be nice to have the pom
file contains packaging=zip, but maven central then requires sources and
javadocs. But our zips are really just attached artifacts, so we already
set the packaging type to pom for our zip files. This change just works
around a limitation of the underlying maven publishing library which
silently skips attached artifacts when the packaging type is set to pom.

relates #20164
closes #20375

* Remove unnecessary extra spacing
2016-09-19 15:10:41 -07:00
Jim Ferenczi 1764ec56b3 Fixed naming inconsistency for fields/stored_fields in the APIs (#20166)
This change replaces the fields parameter with stored_fields when it makes sense.
This is dictated by the renaming we made in #18943 for the search API.

The following list of endpoint has been changed to use `stored_fields` instead of `fields`:
* get
* mget
* explain

The documentation and the rest API spec has been updated to cope with the changes for the following APIs:
* delete_by_query
* get
* mget
* explain

The `fields` parameter has been deprecated for the following APIs (it is replaced by _source filtering):
* update: the fields are extracted from the _source directly.
* bulk: the fields parameter is used but fields are extracted from the source directly so it is allowed to have non-stored fields.

Some APIs still have the `fields` parameter for various reasons:
* cat.fielddata: the fields paramaters relates to the fielddata fields that should be printed.
* indices.clear_cache: used to indicate which fielddata fields should be cleared.
* indices.get_field_mapping: used to filter fields in the mapping.
* indices.stats: get stats on fields (stored or not stored).
* termvectors: fields are retrieved from the stored fields if possible and extracted from the _source otherwise.
* mtermvectors:
* nodes.stats: the fields parameter is used to concatenate completion_fields and fielddata_fields so it's not related to stored_fields at all.

Fixes #20155
2016-09-13 20:54:41 +02:00
Jason Tedor e166459bbe Merge branch 'master' into log4j2
* master:
  Increase visibility of deprecation logger
  Skip transport client plugin installed on JDK 9
  Explicitly disable Netty key set replacement
  percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
  Make it possible for Ingest Processors to access AnalysisRegistry
  Allow RestClient to send array-based headers
  Silence rest util tests until the bogusness can be simplified
  Remove unknown HttpContext-based test as it fails unpredictably on different JVMs
  Tests: Improve rest suite names and generated test names for docs tests
  Add support for a RestClient base path
2016-08-31 10:59:27 -04:00
Jason Tedor 6f8a047942 Skip transport client plugin installed on JDK 9
This commit adds an assumption to
PreBuiltTransportClientTests#testPluginInstalled on JDK 9. The
underlying issue is that Netty attempts to access sun.nio.ch but this
package is not exported from java.base on JDK 9. This throws an uncaught
InaccessibleObjectException causing the test to fail. This assumption
can be removed when Netty 4.1.6 is released as it will include a fix for
this scenario.

Relates #20251
2016-08-31 09:46:27 -04:00
Chris Earle b8f4c92d41 Allow RestClient to send array-based headers
This enables the RestClient to send array-based (multi-valued) header values, rather than only sending whatever happened to be the _last_ value of the header.
2016-08-30 18:02:07 -04:00
Chris Earle c05d5f9257 Remove unknown HttpContext-based test as it fails unpredictably on different JVMs 2016-08-30 17:18:01 -04:00
Chris Earle 335c020cd7 Add support for a RestClient base path
This enables simple support for proxies (beyond proxy host and proxy port, which is done via the RequestConfig)) to provide a base path in front of all requests performed by the RestClient.
2016-08-30 13:46:45 -04:00
Jason Tedor 7da0cdec42 Introduce Log4j 2
This commit introduces Log4j 2 to the stack.
2016-08-30 13:31:24 -04:00
Chris Earle bd0b06440e Add "Async" to the end of each Async RestClient method
This makes it much harder to accidentally miss the Response.
2016-08-26 10:51:33 -04:00
Daniel Mitterdorfer 4460998ff8 Remove obsolete NoopSearchRequestBuilder#setNoStoredFields() 2016-08-26 09:58:53 +02:00
Daniel Mitterdorfer 7b81c4ca59 Add client-benchmark-noop-api-plugin to stress clients even more in benchmarks (#20103) 2016-08-26 09:05:47 +02:00
Chris Earle e171d0e0a8 Un-final Core REST Client classes
This removes final from the RestClient, Response, and Sniffer classes so that outside code can mock them. Their constructors are already package private, so there's not much that can go wrong.
2016-08-25 16:02:04 -04:00
Jason Tedor a62740bbd2 Avoid early initializing Netty
Today when we load the Netty plugins, we indirectly cause several Netty
classes to initialize. This is because we attempt to load some classes
by name, and loading these classes is done in a way that triggers a long
chain of class initializers within Netty. We should not do this, this
can lead to log messages before the logger is loader, and it leads to
initialization in cases when the classes would never be needed (for
example, Netty 3 class initialization is never needed if Netty 4 is
used, and vice versa). This commit avoids this early initialization of
these classes by removing the need for the early loading.

Relates #19819
2016-08-05 14:58:33 -04:00
Tanguy Leroux 841d5a210e Update to Jackson 2.8.1
This commit updates Jackson to the 2.8.1 version, which is more strict when it comes to build objects. It also adds the snakeyaml dependency that was previously shaded in jackson libs.

It also closes #18076
2016-08-05 12:26:06 +02:00
Colin Goodheart-Smithe f1110f6f2a fix import statements 2016-08-04 16:45:05 +01:00
Colin Goodheart-Smithe f273981f37 Added failure message to test 2016-08-04 16:28:12 +01:00
Jason Tedor 785624e96b Restore interrupted status on when closing client
When closing a transport client that depends on Netty 4, interrupted
exceptions can be thrown while shutting down some Netty threads. This
commit refactors the handling of these exceptions to finish shutting
down and then just restore the interrupted status.
2016-08-04 11:22:13 -04:00
Jason Tedor 2936810c63 Setting exists equals false instead of not exists
This commit rewrites a boolean expression to check for equality with
false instead of negating the existence check.
2016-08-04 10:03:08 -04:00
Jason Tedor c08557d033 Wait for Netty 4 threads to terminate on close
Today if the PreBuiltTransportClient is using Netty 4 transport, on
shutdown some Netty 4 threads could linger. This commit causes the
client to wait for these threads to shutdown upon termination.
2016-08-04 09:48:38 -04:00
Colin Goodheart-Smithe b0730bb214 Fix PreBuiltTransportClientTests to run and pass
This change does three things:

1. Makes PreBuiltTransportClientTests run since it was silently
failing on a missing dependency
2. Makes PreBuiltTransportClientTests pass
3. Removes the http.type and transport.type from being set in the
transport clients additional settings since these are set to `netty4` by
default anyway.
2016-08-04 14:15:28 +01:00
Jason Tedor 227463c356 Default transport and HTTP to Netty 4
This commit sets the default transport and HTTP implementation to use
transport-netty4.

Relates #19563
2016-08-02 12:19:19 -04:00
Daniel Mitterdorfer b99a482992 Improve client benchmark (#19736)
* Allow to run client benchmark as an uberjar
* Busy wait to avoid accidental skew on low target throughput rates
* Trigger and wait for full GC to happen between trials
* Add missing SuppressForbidden to allow System.gc in client benchmark
2016-08-02 14:17:51 +02:00
Ryan Ernst 05edb0367d Add client jars pom generation, and make transport client depend on
transport client plugin jars
2016-07-29 07:41:39 -07:00
javanna 1ea8f865d6 Add response body to ResponseException error message
Consuming the response body to make it part of the exception message means that it may not be readable anymore later, depending on whether the entity is repeatable or not. Turns out that the response body tells a lot about the error itself, and considering that we don't expect bodies to be incredibly big for errors, we can wrap the entity into a BufferedHttpEntity to make it repeatable.

Closes #19622
2016-07-28 14:44:20 +02:00
Luca Cavanna 8a51cfb5b3 Simplify Sniffer initialization and automatically create the default HostsSniffer (#19599)
Simplify Sniffer initialization and automatically create the default HostsSniffer

Take Sniffer.Builder out to its own top level class. Remove HostsSniffer.Builder and let SnifferBuilder create the default HostsSniffer. This simplifies the Sniffer initialization as the HostsSniffer is not mandatory anymore. It can still be specified though in case the configuration needs to be changed or a different impl has to be used. Also make HostsSniffer an interface.
2016-07-26 17:28:37 +02:00
Daniel Mitterdorfer c33f85bc37 Add client benchmark
With this commit we add a benchmark for the new REST client and the
existing transport client.

Closes #19281
2016-07-26 11:01:22 +02:00
Jason Tedor 2d1b0587dd Introduce Netty 4
This commit adds transport-netty4, a transport and HTTP implementation
based on Netty 4.

Relates #19526
2016-07-22 22:26:35 -04:00
javanna 061ea1bd8c [TEST] move assertions outside of listener in testAsyncRequests for clearer test failures 2016-07-22 22:19:59 +02:00
javanna c9c7af791c update nextHost method javadocs 2016-07-22 21:42:42 +02:00
javanna 46cb3f36ff fix concurrency bug when getting the host for a given request
It can happen that the list of healthy hosts is empty, then we get one from the blacklist. but some other operation might have sneaked in and emptied the blacklist in the meantime, so we have to retry till we manage to get some host, either from the healthy list or from the blacklist.
2016-07-22 21:35:43 +02:00
javanna e6054a931e add async request unit test 2016-07-22 19:01:56 +02:00
javanna 4e8ee1f0ab add some javadocs to clarify internal listeners behaviour 2016-07-22 17:44:50 +02:00
javanna 835d8cecdc [TEST] add unit tests for internal TrackingFailureListener
Any provided listener will always be wrapped into FailureTrackingListener to handle retries
2016-07-22 17:31:11 +02:00
javanna a6a685b0f6 make Response class final 2016-07-22 16:36:14 +02:00
javanna a579866b42 rename mayRetry to isRetryStatus 2016-07-22 15:49:39 +02:00
javanna 37e075a506 Make SyncResponseListener safer
Throw explicit IllegalStateException in unexpected situations, like where both response and exception are set, or when both are unset. Add unit test for SyncResponseListener.
2016-07-22 15:48:15 +02:00
javanna 175c327e17 validate bufferLimit is positive in HeapBufferedAsyncResponseConsumer 2016-07-21 13:59:28 +02:00
javanna 59ccc88c73 rename mustRetry method to mayRetry 2016-07-20 16:24:57 +02:00
javanna fccfe7dcb8 RestClient javadocs adjustments 2016-07-20 16:24:29 +02:00
javanna 569d7b3ecc notify the listener if the request gets cancelled 2016-07-20 15:24:14 +02:00
javanna a9b5c5adbe restore throws IOException clause on all performRequest sync methods
We throw IOException, which is the exception that is going to be thrown in 99% of the cases. A more generic exception can happen, and if it is a runtime one we just let it bubble up as is, otherwise we wrap it into runtime one so that we don't require to catch Exception everywhere, which seems odd.

Also adjusted javadocs for all performRequest methods
2016-07-19 15:18:05 +02:00
javanna 8eccdff9ad add HeapBufferedAsyncResponseConsumer with configurable max buffer size
We keep the default async client behaviour like in BasicAsyncResponseConsumer, but we lower the maximum size of the buffer from Integer.MAX_VALUE (2GB) to 10 MB. This way users will realize they are buffering big responses in heap hence they'll know they have to do something about it, either write their own response consumer or increase the buffer size limit by providing their manually creeted instance of HeapBufferedAsyncResponseConsumer (constructor accept a bufferLimit int argument).
2016-07-19 15:17:12 +02:00
javanna 1bb33cf572 Remove RestClient#JSON_CONTENT_TYPE constant, already available in ContentType class 2016-07-19 15:17:12 +02:00
javanna f2ab597c84 Build: use license mapping for http* and commons-*
This way we reduce the number of LICENSE and NOTICE files with same content for client.rest and client.sniffer projects.
2016-07-19 15:16:45 +02:00
javanna 69309fb834 [TEST] remove one too many SuppressWarnings 2016-07-19 15:16:45 +02:00
javanna 06caea6b80 move RestClient#builder method on top for more visibility 2016-07-19 15:16:45 +02:00
javanna a3f9721751 replace till with until in RestClient javadocs 2016-07-19 15:16:45 +02:00
javanna b6b92c64c0 update Response javadocs 2016-07-19 15:16:45 +02:00
javanna 283090e2ae add check for null hosts in RestClientBuilder, so it fails early
Also delayed call to HttpAsyncClient#start so that if something goes wrong while creating the RestClient, the http client threads don't linger. In fact, if the constructor fails it is not possible to call close against the RestClient.
2016-07-19 15:16:45 +02:00
javanna e5006ed7b5 Rest Client: have RestClientBuilder callback also return the same type as their argument
HttpClientConfigCallback#customizeHttpClient now also returns the HttpClientBuilder so it can be completely replaced
RequestConfigCallback#customizeRequestConfig now also returns the HttpClientBuilder so it can be completely replaced
2016-07-19 15:16:45 +02:00
javanna e27203534a Rest Client: improve listener naming 2016-07-19 15:16:45 +02:00
javanna 41e97a7cb1 RestClient: take builder out to its own class
The RestClient class is getting bigger and bigger, its builder can definitely be taken out to its own top level class: RestClientBuilder
2016-07-19 15:16:45 +02:00
javanna bb21009772 [TEST] add async entities to the randomization for RequestLoggerTests 2016-07-19 15:15:58 +02:00
javanna 1fbec71243 Rest client: introduce async performRequest method and use async client under the hood for sync requests too
The new method accepts the usual parameters (method, endpoint, params, entity and headers) plus a response listener and an async response consumer. Shortcut methods are also added that don't require params, entity and the async response consumer optional.

There are a few relevant api changes as a consequence of the move to async client that affect sync methods:
- Response doesn't implement Closeable anymore, responses don't need to be closed
- performRequest throws Exception rather than just IOException, as that is the the exception that we get from the FutureCallback#failed method in the async http client
- ssl configuration is a bit simpler, one only needs to call setSSLStrategy from a custom HttpClientConfigCallback, that doesn't end up overridng any other default around connection pooling (it used to happen with the sync client and make ssl configuration more complex)

Relates to #19055
2016-07-19 15:15:58 +02:00
javanna 22aa40bb6d Build: add apache async http client dependencies 2016-07-19 15:11:40 +02:00
Simon Willnauer 8394544548 Add a dedicated client/transport project for transport-client (#19435)
The `client/transport` project adds a new jar build project that
pulls in all dependencies and configures all required modules.

Preinstalled modules are:
 * transport-netty
 * lang-mustache
 * reindex
 * percolator

The `TransportClient` classes are still in core
while `TransportClient.Builder` has only a protected construcutor
such that users are redirected to use the new `TransportClientBuilder`
from the new jar.

Closes #19412
2016-07-18 15:42:24 +02:00
javanna 512b8be791 RestClient: simplify ssl configuration and make http config callback functional friendly 2016-07-12 13:25:55 +02:00
javanna fa0b354e66 Rest Client: add callback to customize http client settings
The callback replaces the ability to fully replace the http client instance. By doing that, one used to lose any default that the RestClient had set for the underlying http client. Given that you'd usually override one or two things only, like a couple of timeout values, the ssl factory or the default credentials providers, it is not uder friendly if by doing that users end up replacing the whole http client instance and lose any default set by us.
2016-07-12 12:31:28 +02:00
javanna 942e342662 Rest Client: use short performRequest methods when possible 2016-07-11 10:36:26 +02:00
javanna fd297637a2 Rest Client: add short performRequest method variants without params and/or body
Users wanting to send a request by providing only its method and endpoint, effectively the only two required arguments, shouldn't need to pass in an empty map and a null entity for the body. While at it we can also add a variant to send requests by specifying only method, endpoint and params, but not body. Headers remain a vararg as last argument, so they can always optionally be provided.

 Closes #19312
2016-07-11 10:36:04 +02:00
javanna 2a91a6ac37 Rest Client: check log level against tracer logger for trace logging 2016-07-08 12:52:29 +02:00
javanna c63719d085 Rest Client: wrap log statement in logger.isDebugEnabled 2016-07-08 12:51:48 +02:00
javanna cfc762b70d Rest Client: add slash to log line when missing between host and uri
Closes #19314
2016-07-08 12:25:20 +02:00
javanna 134b73ab33 Rest Client: HostsSniffer to set http as default scheme
The assumption is HostsSniffer is that all of the arguments have been properly provided and validated through HostsSniffer.Builder, except they weren't, as the scheme didn't have a default value and when not set would cause NPEs down the road. Improved tests to catch this also.
2016-07-07 19:11:37 +02:00
Tanguy Leroux 8c40b2b54e Fix order of modifiers 2016-07-01 16:57:14 +02:00
javanna c4b87149c2 Build: set group for client and sniffer, disable publishing for client-test
Closes #19205
2016-07-01 12:09:34 +02:00
Yannick Welsch 33313df0a2 Add ThreadLeakLingering option to Rest client tests
Some Rest tests use the Sun HTTP server which has lingering threads after shutdown. Similar to ESTestCase, this adds an
option to wait 5 seconds for these threads to terminate.
2016-06-23 10:21:37 +02:00
Nik Everett 0bf447c697 Group client projects under :client
:client ---------> :client:rest
:client-sniffer -> :client:sniffer
:client-test ----> :client:test

This lines the client up with how we do things like modules and
plugins.
2016-06-22 14:26:41 -04:00
javanna c2839c1577 [TEST] Add client-test module and make client tests use randomized runner directly
The lucene-test dependency caused issues with IDEs as they would always load the lucene 5 jar although they shouldn't have, which caused jarhell in es core tests.

If we depend directly on randomized runner we don't have this problem. It is luckily still compatible with java 1.7. This requires though adding a thin module that includes the base test class which can be shared between client and client-sniffer.
2016-06-22 19:37:08 +02:00
javanna f0b6abe439 rename onSuccess to onResponse
That makes it a bit clearer that it's about the response and whether we decide if it was a good one or a failure (based on status code)
2016-06-21 17:11:09 +02:00
javanna cb4bfcb864 Take SniffOnFailureListener out of Sniffer and make FailureListener final on RestClient 2016-06-21 15:29:59 +02:00
javanna 8c60374284 Build: do not load integ test class if --skip-integ-tests-in-disguise is specified in NamingConventionsCheck
Projects that don't depend on elasticsearch-test fail otherwise because org.elasticsearch.test.EsIntegTestCase (default integ test class) is not in the classpath. They should provide their onw integ test base class, but having integration tests should not be mandatory. One can simply set skipIntegTestsInDisguise to true to prevent loading of integ test class.
2016-06-17 13:46:27 +02:00
javanna 7c8013f9fd Build: remove explicit targetCompatibility from forbiddenapis config
targetCompatibility set to the project is enough.
2016-06-15 11:43:48 +02:00
javanna 70fb67cc7b Build: targetCompatibility and sourceCompatibility are enough to make sure we compile client and client-sniffer with target and source 1.7 2016-06-14 23:16:38 +02:00
javanna cf93e904e3 remove message parameter from RequestLogger methods
This prevents useless string allocation.
2016-06-14 14:14:03 +02:00
javanna 1932f6bc7c Rename RequestLogger#log methods to distinguish between the two
One method is to log a request that yielded a response, the other one for a failed request
2016-06-14 13:45:38 +02:00
javanna caa6c96259 Build: make client and client-sniffer depend on lucene-test version 5, last 1.7 compatible version 2016-06-14 09:43:31 +02:00
javanna 116805b28b remove TODO around copying hosts when rotating the collection, it's not a problem for now 2016-06-13 14:02:01 +02:00
javanna 8f7b7fb813 added comments to clarify RequestLogger and DeadHostState 2016-06-13 12:59:30 +02:00
javanna 3cd201e67e [TEST] add comment on using animal-sniffer suppress annotation 2016-06-13 12:02:09 +02:00
javanna 50b6f4c02f Build: changed forbidden-apis targetCompatibility to 1.7 for client and client-sniffer 2016-06-13 09:20:39 +02:00
javanna 777d438f48 [TEST] use jdk-internal bundled signature (rather than the previously removed jdk-non-portable) 2016-06-10 12:46:33 +02:00
javanna 3d7186c81f make DeadHostState final 2016-06-10 11:40:35 +02:00
javanna 6db90da25e [TEST] Remove usage of @SuppressForbidden due to sun HttpServer usage, resolve some violations that were hidden by it 2016-06-10 11:40:13 +02:00
javanna a5e329e563 [TEST] use animalsniffer annotation @IgnoreJRERequirement (similar to @SuppressForbidden) rather than exclusion pattern 2016-06-10 10:11:21 +02:00
javanna 0af9d2c767 Build: add animalsniffer to detect usage of java8 apis in client and client-sniffer 2016-06-10 09:51:47 +02:00
javanna f38ce72004 make forbiddenApisTest work for client and client-sniffer 2016-06-09 23:44:34 +02:00
javanna bd773359d5 [TEST] add SuppresForbidden annotation for client project 2016-06-09 17:39:52 +02:00
javanna d8c0fad08f fix failing tests 2016-06-09 17:06:58 +02:00
javanna 9a4971d3fd fix line length 2016-06-09 16:57:57 +02:00
javanna de8b9fd579 use switch for status codes and add comments 2016-06-09 16:51:08 +02:00
javanna cbdffc7965 s/elasticsearchResponse/response 2016-06-09 16:39:59 +02:00
javanna 853ea9385b add comments on retries 2016-06-09 16:38:43 +02:00
javanna 742f9c6eaa nextHost to return Iterable<HttpHost> rather than Iterator 2016-06-09 16:34:01 +02:00
javanna c028bf9250 rename deadUntil to deadUntilNanos 2016-06-09 16:31:09 +02:00
javanna 85606e8a4b renamed all time intervals arguments and members to include the time unit in their name 2016-06-09 16:27:52 +02:00
javanna 437c4f210b rename ElasticsearchResponse to Response and ElasticsearchResponseException to ResponseException 2016-06-09 14:38:32 +02:00
javanna 04d620da74 require hosts when creating RestClient.Builder
Also fix order of arguments when using assertEquals
2016-06-08 12:37:50 +02:00
javanna 2cf04c0877 wrap entity only when not repeatable and improve RequestLoggerTests 2016-06-06 15:41:11 +02:00
javanna 1f7f6e2709 wrap log statement with logger.isDebugEnabled 2016-06-06 15:13:42 +02:00
javanna a461dd84d2 Build: add hamcrest and securemock to version.properties 2016-06-06 15:02:52 +02:00
javanna 05e26a46d7 raise default socket timeout to 10s and default connect timeout to 1s 2016-06-04 01:06:18 +02:00
javanna f2318ed5ae Build: add missing licenses, SHAs and enable dependency licenses check 2016-06-04 00:56:42 +02:00
javanna b15279b5ef Allow to pass socket facttry registry to createDefaultHttpClient method 2016-06-03 23:59:26 +02:00
javanna f17f0f9247 rename ElasticsearchResponse#getFirstHeader to getHeader 2016-06-03 18:28:31 +02:00
javanna 29eee328fe [TEST] expand RequestLoggerTests to all the supported http methods 2016-06-03 18:23:52 +02:00
javanna 13a27a34f8 [TEST] add RestClient integration test
Relies on real HttpServer to test the actual interaction between RestClient and HttpClient, and how requests get sent in real life.
2016-06-03 16:20:12 +02:00
javanna 4572b69011 [TEST] add RestClient unit tests
Unit tests rely on mockito to mock the internal HttpClient instance. No http request is performed, we only simulate interaction between RestClient and its internal HttpClient.
2016-06-03 16:20:12 +02:00
javanna 9ed2d610ec [TEST] rename restClientTests back to RestClientBuilderTests 2016-06-03 16:20:12 +02:00
javanna 24ea585c9e don't use setDefaultHeaders from HttpClient
Store default headers ourselves instead, otherwise default ones cannot be replaced. Don't allow for multiple headers with same key, last one wins and replaces previous ones with same key.

Also fail with null params or headers.
2016-06-03 16:20:11 +02:00
javanna 47e52044e4 [TEST] add setHosts test and rename RestClientBuilderTests to RestClientTests 2016-06-03 16:01:07 +02:00
javanna 35dbdeeae5 check hosts is not null nor empty earlier, remove check from nextHost
if we check at set time, we don't need to check each single time in nextHost
2016-06-03 16:01:07 +02:00
javanna 6d66fbd9c1 add toString to DeadHostState class 2016-06-03 16:01:07 +02:00
javanna c9db111387 add javadocs on closing responses 2016-06-03 16:01:07 +02:00
javanna 83c6e736de add support for PATCH and TRACE methods
Although elasticsearch doesn't support these methods (RestController doesn't even allow to register handler for them), the RestClient should allow to send requests using them.
2016-06-03 16:01:07 +02:00
javanna 51e487fa55 [TEST] remove okhttp test dependency
Use sun HttpServer instead and disable forbidden-apis for test classes. It turns out to be more flexible than okhttp as it allows get & delete with body.
2016-06-03 16:01:07 +02:00
javanna 044a97c740 move client sniffer to its own project
Create a new subproject called client-sniffer that contains the o.e.client.sniff package. Since it is going to go to a separate jar, due to its additional functionalities and dependency on jackson, it makes sense to have it as a separate project that depends on client. This way we make sure that client doesn't depend on it etc.
2016-06-03 16:01:07 +02:00
javanna 7f4807b29e add some javadocs 2016-06-03 16:01:07 +02:00
javanna 29b1f8d44a make some classes and methods package private
ElasticsearchResponseException, as well as ElasticsearchResponse, should only be created from o.e.client package.
RequestLogger should only be used from this package too.
2016-06-03 16:01:07 +02:00
javanna 6490355cb6 make host state immutable
Instead of having a Connection mutable object that holds the state of the connection to each host, we now have immutable objects only. We keep two sets, one with all the hosts, one with the blacklisted ones. Once we blacklist a host we associate it with a DeadHostState which keeps track of the number of failed attempts and when the host should be retried. A new state object is created each and every time the state of the host needs to be updated.
2016-06-03 16:01:07 +02:00
javanna c70e08c393 include response body in ElasticsearchResponseException error message 2016-06-03 16:01:07 +02:00
javanna eae914ae8e Replace rest test client with low level RestClient
We still have a wrapper called RestTestClient that is very specific to Rest tests, as well as RestTestResponse etc. but all the low level bits around http connections etc. are now handled by RestClient.
2016-06-03 16:01:07 +02:00
javanna 325b723930 [TEST] add rest client test dependency and replace usage of HttpRequestBuilder with RestClient in integration tests 2016-06-03 16:01:07 +02:00
javanna 6d3f6c7faf support missing OPTIONS method, it is supported in elasticsearch core 2016-06-03 16:01:07 +02:00
javanna 9a38d81bec Expose whole ElasticsearchResponse within ElasticsearchResponseException
The only small problem is that the response gets closed straightaway and its body read immediately into a string. Should be ok to load it all into memory eagerly though in case of errors. Otherwise it becomes cumbersome to have an exception implement Closeable...
2016-06-03 16:01:07 +02:00
javanna 16ab491016 add getFirstHeader method to ElasticsearchResponse 2016-06-03 16:01:07 +02:00
javanna 1d06916b07 adapt params argument, they can only be Strings in performRequest method 2016-06-03 16:01:07 +02:00
javanna 2735897b36 use versions from versions.properties 2016-06-03 16:01:07 +02:00
javanna 2d7a781195 fix usage of deprecated apis 2016-06-03 16:01:07 +02:00
javanna c0a72c1686 add support for headers: default ones and per request 2016-06-03 16:01:07 +02:00
javanna 85a7721185 simplify Connection class
The connection class can be greatly simplified now that we don't ping anymore. Pings required a special initial state (UNKNOWN) for connections, to indicate that they require pinging although they are not dead. At this point we don't need the State enum anymore, as connections can only be dead or alive based on the number of failed attempts. markResurrected is also not needed, as it was again a way to make pings required. RestClient can simply pick a dead connection now and use it, no need to change its state when picking the connection.
2016-06-03 16:01:07 +02:00
javanna cdffc3d15b remove notion of connection pool, turn around dependency between RestClient and Sniffer
RestClient exposes a setNodes method, which Sniffer can call to update its nodes.
2016-06-03 16:01:07 +02:00
javanna b38ef345e2 remove streams leftover
Given that we don't use streams anymore, we can check straightaway if the connection iterator is empty before returning it and resurrect a connection when needed directly in the connection pool, no lastResortConnection method required.
2016-06-03 16:01:07 +02:00
javanna 599dad560c remove streams and java 8 only api, build with source and target 1.7 2016-06-03 16:01:07 +02:00
javanna 9569ebc262 add builders for simple creation of RestClient and SniffingConnectionPool instances
We have quite some constructor parameters and some defaults should be applied, builders help simplifying creation of objects for users.
2016-06-03 16:01:07 +02:00
javanna e040d2fc77 remove ConnectionPool interface 2016-06-03 16:01:07 +02:00
javanna 17a21f0272 add curl format trace logging for requests and responses 2016-06-03 16:01:07 +02:00
javanna e7fe397c39 add missing parentheses 2016-06-03 16:01:07 +02:00
javanna 530ad227a2 prevent unclosed response entities in ElasticsearchResponseException, eagerly read response string in case of error status code 2016-06-03 16:01:07 +02:00
javanna e77ab87926 return the response as part of ElasticsearchResponseException 2016-06-03 16:01:07 +02:00
javanna e85ed3eb52 remove pinging from static connection pool, can be replaced by a low connect timeout on each request 2016-06-03 16:01:07 +02:00
javanna d7c41764f2 add some javadocs to connection pool classes 2016-06-03 16:01:07 +02:00
javanna 9ffdea9515 remove Scheme enum 2016-06-03 16:01:07 +02:00
javanna bd29dc1572 Remove Transport class, move all of it within RestClient class 2016-06-03 16:01:07 +02:00
javanna 062a21678c merge Connection and StatefulConnection into one class, remove generic type from Transport 2016-06-03 16:01:07 +02:00
javanna a472544ab4 move sniff related stuff to sniff package 2016-06-03 16:01:07 +02:00
javanna ce663e9703 get rid of connection selector predicate 2016-06-03 16:01:07 +02:00
javanna f6a5a0a4ad get rid of Node abstraction 2016-06-03 16:01:07 +02:00
javanna 94cf8437d0 get rid of retry timeout exception 2016-06-03 16:01:07 +02:00
javanna 25892351e7 remove Verb enum 2016-06-03 16:01:07 +02:00
javanna 6ae8be55bd Introduce notion of Transport, Connection and ConnectionPool
There are two implementations of connection pool, a static one that allows to enable/disable pings, and a sniffing one that sniffs nodes from the nodes info api.

Transport retrieves a stream of connections from the connection for each request and calls onSuccess or onFailure depending on the result of the request.

Transport also supports a max retry timeout to control the timeout for the request retries overall.
2016-06-03 16:01:07 +02:00
javanna 5970723945 clarify why jarhell and dependency license check are disabled, add okhttp mockwebserver dependency
Also enable forbiddenApisTest (will fail till we get rid of the sun http web server in RestClientTests that will be replaced with mockwebserver)
2016-06-03 16:01:07 +02:00
javanna c9c33a7719 Fix checkstyle issues, setup thirdPartyAudit checks and enable forbiddenApisMain 2016-06-03 16:01:07 +02:00
Simon Willnauer 3efbe95ca4 RestClient prototype 2016-06-03 16:01:07 +02:00