Commit Graph

1148 Commits

Author SHA1 Message Date
David Roberts b61202b0a8 [ML] Add a limit on line merging in find_file_structure (#42501)
When analysing a semi-structured text file the
find_file_structure endpoint merges lines to form
multi-line messages using the assumption that the
first line in each message contains the timestamp.
However, if the timestamp is misdetected then this
can lead to excessive numbers of lines being merged
to form massive messages.

This commit adds a line_merge_size_limit setting
(default 10000 characters) that halts the analysis
if a message bigger than this is created.  This
prevents significant CPU time being spent subsequently
trying to determine the internal structure of the
huge bogus messages.
2019-06-03 13:45:51 +01:00
Alan Woodward 2129d06643 Create client-only AnalyzeRequest/AnalyzeResponse classes (#42197)
This commit clones the existing AnalyzeRequest/AnalyzeResponse classes
to the high-level rest client, and adjusts request converters to use these new
classes.

This is a prerequisite to removing the Streamable interface from the internal
server version of these classes.
2019-06-03 09:46:36 +01:00
Przemyslaw Gomulka d5061a151a
Remove suppresions for "unchecked" for hamcrest varargs methods Backport(41528) #42749
In hamcrest 2.1 warnings for unchecked varargs were fixed by hamcrest using @SafeVarargs for those matchers where this warning occurred.
This PR is aimed to remove these annotations when Matchers.contains ,Matchers.containsInAnyOrder or Matchers.hasItems was used
backport #41528
2019-05-31 13:58:49 +02:00
Gordon Brown e0dbf6e82a
Refactor HLRC RequestConverters parameters to be more explicit (#42128)
The existing `RequestConverters.Params` is confusing, because it wraps
an underlying request object and mutations of the `Params` object
actually mutate the `Request` that was used in the construction of the
`Params`.

This leads to a situation where we create a `RequestConverter.Params`
object, mutate it, and then it appears nothing happens to it - it
appears to be unused. What happens behind the scenes is that the Request
object is mutated when methods on `Params` are invoked. This results in
unclear, confusing code where mutating one object changes another with
no obvious connection.

This commit refactors `RequestConverters.Params` to be a simple helper
class to produce a `Map` which must be passed explicitly to a Request
object. This makes it apparent that the `Params` are actually used, and
that they have an effect on the `request` object explicit and easier to
understand.

Co-authored-by: Ojas Gulati <ojasgulati100@gmail.com>
2019-05-29 17:08:46 -06:00
kevin fuksman 7c612af6d2
Added param ignore_throttled=false when indicesOptions.ignoreThrottled() is false (#42393)
and fixed test RequestConvertersTests and added ignore_throttled on all request
2019-05-29 13:45:14 +02:00
Hendrik Muhs 345ff21ae5 [ML-DataFrame] rewrite start and stop to answer with acknowledged (#42589)
rewrite start and stop to answer with acknowledged

fixes #42450
2019-05-29 11:14:32 +02:00
Armin Braun 6166fed6f1
Fix BulkProcessorRetryIT (#41700) (#42618)
* Now that we process the bulk requests themselves on the WRITE threadpool, they can run out of retries too like the item requests even when backoff is active
* Fixes #41324 by using the same logic that checks failed item requests for their retry status for the top level bulk requests as well
2019-05-28 17:58:00 +02:00
Hendrik Muhs 6d47ee9268 [ML-DataFrame] add support for fixed_interval, calendar_interval, remove interval (#42427)
* add support for fixed_interval, calendar_interval, remove interval

* adapt HLRC

* checkstyle

* add a hlrc to server test

* adapt yml test

* improve naming and doc

* improve interface and add test code for hlrc to server

* address review comments

* repair merge conflict

* fix date patterns

* address review comments

* remove assert for warning

* improve exception message

* use constants
2019-05-24 20:30:17 +02:00
Luca Cavanna c2af62455f Cut over SearchResponse and SearchTemplateResponse to Writeable (#41855)
Relates to #34389
2019-05-22 18:47:54 +02:00
Luca Cavanna e747326b04 Adapt low-level REST client to java 8 (#41537)
As a follow-up to #38540 we can use lambda functions and method
references where convenient in the low-level REST client.

Also, we need to update the docs to state that the minimum java version
required is 1.8.
2019-05-22 18:47:54 +02:00
Guillaume Darmont 3e231bbad6 StackOverflowError when calling BulkRequest#add (#41672)
Removing of payload in BulkRequest (#39843) had a side effect of making
`BulkRequest.add(DocWriteRequest<?>...)` (with varargs) recursive, thus
leading to StackOverflowError. This PR adds a small change in
RequestConvertersTests to show the error and the corresponding fix in
`BulkRequest`.

Fixes #41668
2019-05-22 11:22:14 -05:00
Ioannis Kakavas cdf9485e33 Allow Kibana user to use the OpenID Connect APIs (#42305)
Add the manage_oidc privilege to the kibana user and to the role 
privileges list
2019-05-22 09:44:37 +03:00
David Kyle 0fd42ce1f5
[ML Data Frame] Start directly data frame rather than via the scheduler (#42224)
Trigger indexer start directly to put the indexer in INDEXING state immediately
2019-05-21 15:48:45 +01:00
David Kyle 24144aead2
[ML] Complete the Data Frame task on stop (#41752) (#42063)
Wait for indexer to stop then complete the persistent task on stop.
If the wait_for_completion is true the request will not return until stopped.
2019-05-21 10:24:20 +01:00
Zachary Tong 6ae6f57d39
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.

This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed.  And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit).  This arrangement is very
error-prone for users.

This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.

The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.

The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
Jay Modi dbbdcea128
Update ciphers for TLSv1.3 and JDK11 if available (#42082)
This commit updates the default ciphers and TLS protocols that are used
when the runtime JDK supports them. New cipher support has been
introduced in JDK 11 and 12 along with performance fixes for AES GCM.
The ciphers are ordered with PFS ciphers being most preferred, then
AEAD ciphers, and finally those with mainstream hardware support. When
available stronger encryption is preferred for a given cipher.

This is a backport of #41385 and #41808. There are known JDK bugs with
TLSv1.3 that have been fixed in various versions. These are:

1. The JDK's bundled HttpsServer will endless loop under JDK11 and JDK
12.0 (Fixed in 12.0.1) based on the way the Apache HttpClient performs
a close (half close).
2. In all versions of JDK 11 and 12, the HttpsServer will endless loop
when certificates are not trusted or another handshake error occurs. An
email has been sent to the openjdk security-dev list and #38646 is open
to track this.
3. In JDK 11.0.2 and prior there is a race condition with session
resumption that leads to handshake errors when multiple concurrent
handshakes are going on between the same client and server. This bug
does not appear when client authentication is in use. This is
JDK-8213202, which was fixed in 11.0.3 and 12.0.
4. In JDK 11.0.2 and prior there is a bug where resumed TLS sessions do
not retain peer certificate information. This is JDK-8212885.

The way these issues are addressed is that the current java version is
checked and used to determine the supported protocols for tests that
provoke these issues.
2019-05-20 09:45:36 -04:00
Ed Savage a68b04e47b [ML] Improve hard_limit audit message (#42086)
Improve the hard_limit memory audit message by reporting how many bytes
over the configured memory limit the job was at the point of the last
allocation failure.

Previously the model memory usage was reported, however this was
inaccurate and hence of limited use -  primarily because the total
memory used by the model can decrease significantly after the models
status is changed to hard_limit but before the model size stats are
reported from autodetect to ES.

While this PR contains the changes to the format of the hard_limit audit
message it is dependent on modifications to the ml-cpp backend to
send additional data fields in the model size stats message. These
changes will follow in a subsequent PR. It is worth noting that this PR
must be merged prior to the ml-cpp one, to keep CI tests happy.
2019-05-17 17:40:08 -04:00
Benjamin Trent febee07dcc
[ML] adding pivot.max_search_page_size option for setting paging size (#41920) (#42079)
* [ML] adding pivot.size option for setting paging size

* Changing field name to address PR comments

* fixing ctor usage

* adjust hlrc for field name change
2019-05-10 13:22:31 -05:00
Hendrik Muhs 8823cb65f7 [ML-DataFrame] migrate to PageParams for get and stats, move PageParams into core (#41851)
migrate hlrc dataframe get and _stats to use PageParams, moves PageParams into core for common 
usage, fix possible NPE in PageParams
2019-05-07 16:16:22 +02:00
Hendrik Muhs 0d9797847a remove validation methods in client (#41754)
remove validation methods in client (#41754)
2019-05-02 20:07:29 +02:00
Benjamin Trent bc333a5cbf
[ML] data frame, adding builder classes for complex config classes (#41638) (#41704)
* [ML] data frame, adding builder classes for complex config classes

* Addressing PR comments, adding some java docs

* cleaning up constructor

* fixing indentation

* change constructors to be package-private
2019-05-01 06:44:29 -05:00
Benjamin Trent a0990ca239
[ML] cleanup + adding description field to transforms (#41554) (#41605)
* [ML] cleanup + adding description field to transforms

* making description length have a max of 1k
2019-04-26 16:50:59 -05:00
David Kyle 1f00cec36f [Ml-Dataframe] Update URLs in Data frame client java doc (#41539) 2019-04-26 12:04:18 +01:00
Christoph Büscher 52495843cc [Docs] Fix common word repetitions (#39703) 2019-04-25 20:47:47 +02:00
Benjamin Trent 08843ba62b
[ML] Adds progress reporting for transforms (#41278) (#41529)
* [ML] Adds progress reporting for transforms

* fixing after master merge

* Addressing PR comments

* removing unused imports

* Adjusting afterKey handling and percentage to be 100*

* Making sure it is a linked hashmap for serialization

* removing unused import

* addressing PR comments

* removing unused import

* simplifying code, only storing total docs and decrementing

* adjusting for rewrite

* removing initial progress gathering from executor
2019-04-25 11:23:12 -05:00
Jim Ferenczi 6184efaff6
Handle unmapped fields in _field_caps API (#34071) (#41426)
Today the `_field_caps` API returns the list of indices where a field
is present only if this field has different types within the requested indices.
However if the request is an index pattern (or an alias, or both...) there
is no way to infer the indices if the response contains only fields that have
the same type in all indices. This commit changes the response to always return
the list of indices in the response. It also adds a way to retrieve unmapped field
in a specific section per field called `unmapped`. This section is created for each field
that is present in some indices but not all if the parameter `include_unmapped` is set to
true in the request (defaults to false).
2019-04-25 18:13:48 +02:00
Luca Cavanna 8a0e5f7b87
Deprecate support for first line empty in msearch API (#41442)
In order to support empty action metadata in the first msearch item,
we need to remove support for prepending msearch request body with an
empty line, which prevents us from parsing the empty line as action
metadata for the first search item.

Relates to #41011
2019-04-25 12:45:18 +02:00
Ryan Ernst 7e3875d781 Upgrade hamcrest to 2.1 (#41464)
hamcrest has some improvements in newer versions, like FileMatchers
that make assertions regarding file exists cleaner. This commit upgrades
to the latest version of hamcrest so we can start using new and improved
matchers.
2019-04-24 23:40:03 -07:00
Armin Braun 381b8e2ece
Fix BulkProcessor Retry ITs (#41338) (#41472)
* The test fails for the retry backoff enabled case because the retry handler in the bulk processor hasn't been adjusted to account for #40866 which now might lead to an outright rejection of the request instead of its items individually
   * Fixed by adding retry functionality to the top level request as well
* Also fixed the duplicate test for the HLRC that wasn't handling the non-backoff case yet the same way the non-client IT did
* closes #41324
2019-04-24 13:46:32 +02:00
Armin Braun 389a13b68e
Mute BulkProcessorRetryIT#testBulkRejectionLoadWithBackoff (#41325) (#41331)
* For #41324
2019-04-18 11:55:28 +02:00
David Kyle 1d2365f5b6 [ML-DataFrame] Refactorings and tidying (#41248)
Remove unnecessary generic params from SingleGroupSource
and unused code from the HLRC
2019-04-17 14:58:26 +01:00
David Roberts 6cc35d3724 [ML] Unmute MachineLearningIT.testDeleteExpiredData (#41186)
The cause of failure was fixed by elastic/ml-cpp#459,
so all that remains on the Java side is to unmute the
test that was failing.

Closes #41070
2019-04-16 16:38:46 +01:00
Christoph Büscher 2980a6c70f Clarify some ToXContent implementations behaviour (#41000)
This change adds either ToXContentObject or ToXContentFragment to classes
directly implementing ToXContent currently. This helps in reasoning about
whether those implementations output full xcontent object or just fragments.

Relates to #16347
2019-04-15 09:42:08 +02:00
Martijn van Groningen 1eff8976a8
Deprecate AbstractHlrc* and AbstractHlrcStreamable* base test classes (#41014)
* moved hlrc parsing tests from xpack to hlrc module and removed dependency on hlrc from xpack core

* deprecated old base test class

* added deprecated jdoc tag

* split test between xpack-core part and hlrc part

* added lang-mustache test dependency, this previously came in via
hlrc dependency.

* added hlrc dependency on a qa module

* duplicated ClusterPrivilegeName class in xpack-core, since x-pack
core no longer has a dependency on hlrc.

* replace ClusterPrivilegeName usages with string literals

* moved tests to dedicated to hlrc packages in order to remove Hlrc part from the name and make sure to use imports instead of full qualified class where possible

* remove ESTestCase. from method invocation and use method directly,
because these tests indirectly extend from ESTestCase
2019-04-10 16:29:17 +02:00
Ed Savage 722362e402 Mute MachineLearningIt#testDeleteExpiredData
Tracked in #41070
2019-04-10 13:07:53 +01:00
Hendrik Muhs f9018ab11b [ML-DataFrame] create checkpoints on every new run (#40725)
Use the checkpoint service to create a checkpoint on every new run. Expose checkpoints stats on _stats endpoint.
2019-04-10 09:14:11 +02:00
Martijn van Groningen 46b0fdae33
Add realistic hlrc request serialization test base class and (#40362)
changed hlrc ccr request tests to use AbstractRequestTestCase base class.

This way the request classes are tested in a more realistic setting.
Note this change also adds a test dependency on xpack core module.

Similar to #39844 but then for hlrc request serialization tests.

Removed iterators from hlrc parsing tests.
Use empty xcontent registries.

Relates to #39745
2019-04-10 08:00:01 +02:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Ed Savage fdc1bdd4d3 [ML][TEST] Fix randomly failing HLRC test (#40973)
Made changes to ensure that unique IDs are generated for model snapshots
used by the deleteExpiredDataTest test in the MachineLearningIT suite.

Previously a sleep of 1s was performed between jobs under the assumption
that this would be sufficient to guarantee that the timestamps used in
the composition of the snapshot IDs would be different.

The new approach is to wait on the condition that the old and new
timestamps are in fact different (to 1s resolution).
2019-04-09 16:55:35 +01:00
Martijn van Groningen 040a4961c7
Revert "Revert "Change HLRC CCR response tests to use AbstractResponseTestCase base class. (#40257)"" (#40971)
This reverts commit df91237a94fd3d3ae954eb1845c434dda692d087.
2019-04-09 15:18:33 +02:00
Martijn van Groningen 84a410d5a8
Revert "Change HLRC CCR response tests to use AbstractResponseTestCase base class. (#40257)"
This reverts commit c29027d99e.
2019-04-08 11:24:06 +02:00
Martijn van Groningen c29027d99e
Change HLRC CCR response tests to use AbstractResponseTestCase base class. (#40257)
This way the response classes are tested in a more realistic setting.

Relates to #39745
2019-04-08 07:09:28 +02:00
Jay Modi f34663282c
Update apache httpclient to version 4.5.8 (#40875)
This change updates our version of httpclient to version 4.5.8, which
contains the fix for HTTPCLIENT-1968, which is a bug where the client
started re-writing paths that contained encoded reserved characters
with their unreserved form.
2019-04-05 13:48:10 -06:00
Martijn van Groningen 809a5f13a4
Make -try xlint warning disabled by default. (#40833)
Many gradle projects specifically use the -try exclude flag, because
there are many cases where auto-closeable resource ignore is never
referenced in body of corresponding try statement. Suppressing this
warning specifically in each case that it happens using
`@SuppressWarnings("try")` would be very verbose.

This change removes `-try` from any gradle project and adds it to the
build plugin. Also this change removes exclude flags from gradle projects
that is already specified in build plugin (for example -deprecation).

Relates to #40366
2019-04-05 08:02:26 +02:00
roy 075078e7e0 HLRC: fix uri encode bug when url path starts with '/' (#34436)
This commit sets the authority of a URI to blank such that it does not
misinterpret slashes in the path as the authority.

Closes #34433
2019-04-04 12:59:59 -05:00
Michael Basnight fb5a0652a8 HLRC: Convert xpack methods to client side objects (#40705)
This commit fixes a problem with BWC that was brought up in #40511. A
newer version of the code was emitting a new value for an enum to an
older version, and the older version could not handle that. It caused
the response to error. The MainResponse is now relaxed, and will accept
whatever values the server expose, and holds most of them as Strings
instead of complex objects.

Fixes #40511
2019-04-04 11:06:44 -05:00
Hendrik Muhs 31e79a73d7 add HLRC protocol tests for transform state and stats (#40766)
adds HLRC protocol tests for state and stats hrlc clients
2019-04-03 12:51:15 +02:00
Hendrik Muhs 1f947054ff add reason to DataFrameTransformState and add hlrc protocol tests (#40736)
add field "reason" to DataFrameTransformState, add hlrc protocol tests and allow unknown fields for DataFrameTransformState
2019-04-03 07:35:07 +02:00
Tim Vernum 2c770ba3cb
Support mustache templates in role mappings (#40571)
This adds a new `role_templates` field to role mappings that is an
alternative to the existing roles field.

These templates are evaluated at runtime to determine which roles should be
granted to a user.
For example, it is possible to specify:

    "role_templates": [
      { "template":{ "source": "_user_{{username}}" } }
    ]

which would mean that every user is assigned to their own role based on
their username.

You may not specify both roles and role_templates in the same role
mapping.

This commit adds support for templates to the role mapping API, the role
mapping engine, the Java high level rest client, and Elasticsearch
documentation.

Due to the lack of caching in our role mapping store, it is currently
inefficient to use a large number of templated role mappings. This will be
addressed in a future change.

Backport of: #39984, #40504
2019-04-02 20:55:10 +11:00
Adrien Grand 7f7d09af2e
Deprecate types in `_graph/explore` calls. (#40466) (#40513)
Any call that uses a path that sets a type will trigger a deprecation warning.
2019-03-28 09:32:26 +01:00
Adrien Grand 65a35c985c
Remove type from VersionConflictEngineException. (#37490) (#40514)
It initially mentioned the type in the exception because the type used to be
required to uniquely identify a document. This is not necessary anymore given
that indices have at most one type.
2019-03-28 09:32:09 +01:00
Andy Bristol c0c6d702a2
ignore 409 conflict in reindex responses (#39543)
The reindex family of APIs (reindex, update-by-query, delete-by-query) can
sometimes return responses that have an error status code (409 Conflict in this
case) but still have a body in the usual BulkByScrollResponse format. When the
HLRC tries to handle such responses, it blows up because it tris to parse it
expecting the error format that errors in general use. This change prompts the
HLRC to parse the response using the expected BulkByScrollResponse format.
2019-03-27 13:27:17 -07:00
David Kyle c990b30019
[ML] Data Frame HLRC Get API (#40509) 2019-03-27 12:40:39 +00:00
Benjamin Trent 12943c5d2c
[ML] Add data frame task state object and field (#40169) (#40490)
* [ML] Add data frame task state object and field

* A new state item is added so that the overall task state can be
accoutned for
* A new FAILED state and reason have been added as well so that failures
can be shown to the user for optional correction

* Addressing PR comments

* adjusting after master merge

* addressing pr comment

* Adjusting auditor usage with failure state

* Refactor, renamed state items to task_state and indexer_state

* Adding todo and removing redundant auditor call

* Address HLRC changes and PR comment

* adjusting hlrc IT test
2019-03-27 06:53:58 -05:00
David Kyle 1354696db9
[ML] Data Frame HLRC Get Stats API (#40443) 2019-03-26 11:17:13 +00:00
Benjamin Trent 7b4f964708
[ML] make source and dest objects in the transform config (#40337) (#40396)
* [ML] make source and dest objects in the transform config

* addressing PR comments

* Fixing compilation post merge

* adding comment for Arrays.hashCode

* addressing changes for moving dest to object

* fixing data_frame yml tests

* fixing API test
2019-03-25 07:16:41 -05:00
David Kyle a4cb92a300
[ML] Data Frame HLRC Preview API (#40258) 2019-03-21 09:38:27 +00:00
Martijn van Groningen a478196f79
Add realistic hlrc response serialization test base class (#39844)
The base class facilitates generating a server side response test instance,
that gets serialized as xcontent, which then gets parsed into a hlrc response
instance, which then gets asserted against the server side response instance.

This way of testing is more realistic then how hlrc response classes are tested
today, which basically tests that serialization works by generating
hlrc response instance, serialize that to xcontent and then parse it back
to a hlrc response instance.

Besides adding a base test class, this change also cuts AcknowledgedResponseTests
and BroadcastResponseTests over to use this base class.

Relates to #39745
2019-03-20 13:35:34 +01:00
David Kyle 387648065d
[ML] Data Frame HLRC start & stop APIs (#40197) 2019-03-19 13:30:01 +00:00
Gordon Brown c8a4a7fc9d
Remove Migration Upgrade and Assistance APIs (#40075)
The Migration Assistance API has been functionally replaced by the
Deprecation Info API, and the Migration Upgrade API is not used for the
transition from ES 6.x to 7.x, and does not need to be kept around to
repair indices that were not properly upgraded before upgrading the
cluster, as was the case in 6.
2019-03-18 13:46:56 -06:00
Jack Conradson b57af6c401 Add a Painless Context REST API (#39382)
This PR adds an internal REST API for querying context information about 
Painless whitelists.

Commands include the following:
GET /_scripts/painless/_context -- retrieves a list of contexts
GET /_scripts/painless/_context?context=%name% retrieves all available 
information about the API for this specific context
2019-03-14 12:42:12 -07:00
Ioannis Kakavas ae0ff05c90 Adjust testGetSslCertificates to run in FIPS (#40046)
As discovered in #40041, when parsing certificates from files, the
SUN Security Provider normalizes DNs from parsed certificates by
adding spaces between RDNs, while the BouncyCastle one (which we
use in FIPS tests) does not.

We could proceed to normalize the DNs in the same manner in this
test by using i.e. the Unbound LDAP SDK but since the goal of this
test is to validate that we do get to read these exact certificates
from our trust sources and not to validate subject DNs, this commit
changes the test to check the serial number instead

Resolves: #40041
2019-03-14 20:25:47 +02:00
David Kyle c02f49e9d3
[ML-Dataframe] Add Data Frame client to the Java HLRC (#40040)
Adds DataFrameClient to the Java HLRC and implements PUT and 
DELETE data frame transform.
2019-03-14 14:57:12 +00:00
Ioannis Kakavas 2512cf3ec8 Mute testGetSslCertificates in FIPS (#40042) 2019-03-14 14:23:46 +02:00
Adrien Grand b841de2e38
Don't emit deprecation warnings on calls to the monitoring bulk API. (#39805) (#39838)
The monitoring bulk API accepts the same format as the bulk API, yet its concept
of types is different from "mapping types" and the deprecation warning is only
emitted as a side-effect of this API reusing the parsing logic of bulk requests.

This commit extracts the parsing logic from `_bulk` into its own class with a
new flag that allows to configure whether usage of `_type` should emit a warning
or not. Support for payloads has been removed for simplicity since they were
unused.

@jakelandis has a separate change that removes this notion of type from the
monitoring bulk API that we are considering bringing to 8.0.
2019-03-11 07:58:28 +01:00
David Kyle 6c2e831e94
[ML-Dataframe] Data frame config HLRC objects (#39825) 2019-03-08 12:18:55 +00:00
Jason Tedor 0250d554b6
Introduce forget follower API (#39718)
This commit introduces the forget follower API. This API is needed in cases that
unfollowing a following index fails to remove the shard history retention leases
on the leader index. This can happen explicitly through user action, or
implicitly through an index managed by ILM. When this occurs, history will be
retained longer than necessary. While the retention lease will eventually
expire, it can be expensive to allow history to persist for that long, and also
prevent ILM from performing actions like shrink on the leader index. As such, we
introduce an API to allow for manual removal of the shard history retention
leases in this case.
2019-03-07 11:08:45 -05:00
David Roberts e94d32d069 Add roles and cluster privileges for data frame transforms (#39661)
This change adds two new cluster privileges:

* manage_data_frame_transforms
* monitor_data_frame_transforms

And two new built-in roles:

* data_frame_transforms_admin
* data_frame_transforms_user

These permit access to the data frame transform endpoints.
(Index privileges are also required on the source and
destination indices for each data frame transform, but
since these indices are configurable they it is not
appropriate to grant them via built-in roles.)
2019-03-05 14:07:25 +00:00
Martijn van Groningen ea4e9ed0a9
Add missing tests for CcrRequestConverters (#39228) 2019-02-25 12:48:43 +01:00
Tal Levy f1c8aa816f fix dangling tag in TasksClientDocumentationIT (#39157)
fix dangling tag in TasksClientDocumentationIT
and fix tag mentions from list-tasks to cancel-tasks
2019-02-20 08:48:58 -08:00
Martijn van Groningen 0594a467f2
Add support for ccr follow info api to HLRC. (#39115)
This API was introduces after #33824 was closed.
2019-02-20 16:10:54 +01:00
Hendrik Muhs 2e2567e827 add debug info for intermittent test failure 2019-02-19 10:54:09 +01:00
Hendrik Muhs 4f662bd289
Add data frame feature (#38934) (#39029)
The data frame plugin allows users to create feature indexes by pivoting a source index. In a
nutshell this can be understood as reindex supporting aggregations or similar to the so called entity
centric indexing.

Full history is provided in: feature/data-frame-transforms
2019-02-18 11:07:29 +01:00
Paul Sanwald 408a800f74
ClusterClientIT refactor (#38872) (#39002)
Add fixes for ClusterClientIT test and unmute tests.
2019-02-17 20:37:26 -05:00
Jason Tedor 6abe99808a
Drop support for the low-level REST client on JDK 7 (#38540)
This commit bumps the minimum compiler version on the low-level REST
client to JDK 8.
2019-02-08 19:48:44 -05:00
Luca Cavanna a7046e001c
Remove support for maxRetryTimeout from low-level REST client (#38085)
We have had various reports of problems caused by the maxRetryTimeout
setting in the low-level REST client. Such setting was initially added
in the attempts to not have requests go through retries if the request
already took longer than the provided timeout.

The implementation was problematic though as such timeout would also
expire in the first request attempt (see #31834), would leave the
request executing after expiration causing memory leaks (see #33342),
and would not take into account the http client internal queuing (see #25951).

Given all these issues, it seems that this custom timeout mechanism 
gives little benefits while causing a lot of harm. We should rather rely 
on connect and socket timeout exposed by the underlying http client 
and accept that a request can overall take longer than the configured 
timeout, which is the case even with a single retry anyways.

This commit removes the `maxRetryTimeout` setting and all of its usages.
2019-02-06 08:43:47 +01:00
Martijn van Groningen 9492dcea42
Update IndexTemplateMetaData to allow unknown fields (#38448)
Updated IndexTemplateMetaData to use ObjectParser.

The IndexTemplateMetaData class used old parsing logic and was not
resiliant to new fields. This commit updates it to use the
ConstructingObjectParser and allow unknown fields.

Relates #36938

Co-authored-by: Michael Basnight <mbasnight@gmail.com>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2019-02-06 08:21:30 +01:00
Lee Hinman a6ce671751
Re-enable TasksClientDocumentationIT.testCancelTasks (#38234)
This test has been disabled since November 2018, but I was not able to reproduce
the failure. Re-enabling this so we can see the full log and get more context if
it fails again.

Relates to #35514
2019-02-05 13:42:43 -07:00
Boaz Leskes 8ad9a07b87 CRUDDocumentationIT fix documentation references 2019-02-05 21:27:28 +01:00
Boaz Leskes 033ba725af
Remove support for internal versioning for concurrency control (#38254)
Elasticsearch has long [supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-versioning) compare and set (a.k.a optimistic concurrency control) operations using internal document versioning. Sadly that approach is flawed and can sometime do the wrong thing. Here's the relevant excerpt from the resiliency status page:

> When a primary has been partitioned away from the cluster there is a short period of time until it detects this. During that time it will continue indexing writes locally, thereby updating document versions. When it tries to replicate the operation, however, it will discover that it is partitioned away. It won’t acknowledge the write and will wait until the partition is resolved to negotiate with the master on how to proceed. The master will decide to either fail any replicas which failed to index the operations on the primary or tell the primary that it has to step down because a new primary has been chosen in the meantime. Since the old primary has already written documents, clients may already have read from the old primary before it shuts itself down. The version numbers of these reads may not be unique if the new primary has already accepted writes for the same document 

We recently [introduced](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/optimistic-concurrency-control.html) a new sequence number based approach that doesn't suffer from this dirty reads problem. 

This commit removes support for internal versioning as a concurrency control mechanism in favor of the sequence number approach.

Relates to #1078
2019-02-05 20:53:35 +01:00
Julie Tibshirani 3ce7d2c9b6
Make sure to reject mappings with type _doc when include_type_name is false. (#38270)
`CreateIndexRequest#source(Map<String, Object>, ... )`, which is used when
deserializing index creation requests, accidentally accepts mappings that are
nested twice under the type key (as described in the bug report #38266).

This in turn causes us to be too lenient in parsing typeless mappings. In
particular, we accept the following index creation request, even though it
should not contain the type key `_doc`:

```
PUT index?include_type_name=false
{
  "mappings": {
    "_doc": {
      "properties": { ... }
    }
  }
}
```

There is a similar issue for both 'put templates' and 'put mappings' requests
as well.

This PR makes the minimal changes to detect and reject these typed mappings in
requests. It does not address #38266 generally, or attempt a larger refactor
around types in these server-side requests, as I think this should be done at a
later time.
2019-02-05 10:52:32 -08:00
Boaz Leskes 12657fda44
`if_seq_no` and `if_primary_term` parameters aren't wired correctly in REST Client's CRUD API (#38411) 2019-02-05 18:05:56 +01:00
Julie Tibshirani 440d1eda8a
Fix failures in BulkProcessorIT#testGlobalParametersAndBulkProcessor. (#38129)
This PR fixes a couple test issues:
* It narrows an assertWarnings call that was too broad, and wasn't always
  applicable with certain random sequences.
* Previously, we could send a typeless bulk request containing '_type: 'null'.
  Now we omit the _type key altogether for typeless requests.
2019-02-05 08:42:37 -08:00
Michael Basnight 8742db3afe
Update Rollup Caps to allow unknown fields (#38339)
This commit ensures that the parts of rollup caps that can allow unknown
fields will allow them. It also modifies the test such that we can use
the features we need for disallowing fields in spots where they would
not be allowed.

Relates #36938
2019-02-05 10:08:08 -06:00
Martijn van Groningen 0beb3c93d1
Clean up duplicate follow config parameter code (#37688)
Introduced FollowParameters class that put follow, resume follow,
put auto follow pattern requests and follow info response classes reuse.

The FollowParameters class had the fields, getters etc. for the common parameters
that all these APIs have.  Also binary and xcontent serialization /
parsing is handled by this class.

The follow, resume follow, put auto follow pattern request classes originally
used optional non primitive fields, so FollowParameters has that too and the follow info api can handle that now too.

Also the followerIndex field can in production only be specified via
the url path. If it is also specified via the request body then
it must have the same value as is specified in the url path. This
option only existed to xcontent testing. However the AbstractSerializingTestCase
base class now also supports createXContextTestInstance() to provide
a different test instance when testing xcontent, so allowing followerIndex
to be specified via the request body is no longer needed.

By moving the followerIndex field from Body to ResumeFollowAction.Request
class and not allowing the followerIndex field to be specified via
the request body the Body class is redundant and can be removed. The
ResumeFollowAction.Request class can then directly use the
FollowParameters class.

For consistency I also removed the ability to specified followerIndex
in the put follow api and the name in put auto follow pattern api via
the request body.
2019-02-05 17:05:19 +01:00
Brandon Kobel 64ff75f04e
Add apm_user reserved role (#38206)
* Adding apm_user

* Fixing SecurityDocumentationIT testGetRoles test

* Adding access to .ml-anomalies-*

* Fixing APM test, we don't have access to the ML state index
2019-02-04 21:45:28 -08:00
Yogesh Gaikwad fe36861ada
Add support for API keys to access Elasticsearch (#38291)
X-Pack security supports built-in authentication service
`token-service` that allows access tokens to be used to 
access Elasticsearch without using Basic authentication.
The tokens are generated by `token-service` based on
OAuth2 spec. The access token is a short-lived token
(defaults to 20m) and refresh token with a lifetime of 24 hours,
making them unsuitable for long-lived or recurring tasks where
the system might go offline thereby failing refresh of tokens.

This commit introduces a built-in authentication service
`api-key-service` that adds support for long-lived tokens aka API
keys to access Elasticsearch. The `api-key-service` is consulted
after `token-service` in the authentication chain. By default,
if TLS is enabled then `api-key-service` is also enabled.
The service can be disabled using the configuration setting.

The API keys:-
- by default do not have an expiration but expiration can be
  configured where the API keys need to be expired after a
  certain amount of time.
- when generated will keep authentication information of the user that
   generated them.
- can be defined with a role describing the privileges for accessing
   Elasticsearch and will be limited by the role of the user that
   generated them
- can be invalidated via invalidation API
- information can be retrieved via a get API
- that have been expired or invalidated will be retained for 1 week
  before being deleted. The expired API keys remover task handles this.

Following are the API key management APIs:-
1. Create API Key - `PUT/POST /_security/api_key`
2. Get API key(s) - `GET /_security/api_key`
3. Invalidate API Key(s) `DELETE /_security/api_key`

The API keys can be used to access Elasticsearch using `Authorization`
header, where the auth scheme is `ApiKey` and the credentials, is the 
base64 encoding of API key Id and API key separated by a colon.
Example:-
```
curl -H "Authorization: ApiKey YXBpLWtleS1pZDphcGkta2V5" http://localhost:9200/_cluster/health
```

Closes #34383
2019-02-05 14:21:57 +11:00
Christoph Büscher d255303584
Add typless client side GetIndexRequest calls and response class (#37778)
The HLRC client currently uses `org.elasticsearch.action.admin.indices.get.GetIndexRequest`
and `org.elasticsearch.action.admin.indices.get.GetIndexResponse` in its get index calls. Both request and
response are designed for the typed APIs, including some return types e.g. for `getMappings()` which in
the maps it returns still use a level including the type name.
In order to change this without breaking existing users of the HLRC API, this PR introduces two new request
and response objects in the `org.elasticsearch.client.indices` client package. These are used by the
IndicesClient#get and IndicesClient#exists calls now by default and support the type-less API. The old request
and response objects are still kept for use in similarly named, but deprecated methods.

The newly introduced client side classes are simplified versions of the server side request/response classes since
they don't need to support wire serialization, and only the response needs fromXContent parsing (but no
xContent-serialization, since this is the responsibility of the server-side class).
Also changing the return type of `GetIndexResponse#getMapping` to
`Map<String, MappingMetaData> getMappings()`, while it previously was returning another map
keyed by the type-name. Similar getters return simple Maps instead of the ImmutableOpenMaps that the 
server side response objects return.
2019-02-05 03:41:05 +01:00
Gordon Brown 292e0f6fb7
Deprecate `_type` in simulate pipeline requests (#37949)
As mapping types are being removed throughout Elasticsearch, the use of
`_type` in pipeline simulation requests is deprecated. Additionally, the
default `_type` used if one is not supplied has been changed to `_doc` for
consistency with the rest of Elasticsearch.
2019-02-04 16:11:44 -07:00
Mayya Sharipova 641704464d
Deprecate types in rollover index API (#38039)
Relates to #35190
2019-02-04 16:07:45 -05:00
Michael Basnight 24db053465
Fix ILM explain response to allow unknown fields (#38054)
IndexLifecycleExplainResponse did not allow unknown fields. This commit
fixes the test and ConstructingObjectParser such that it allows unknown
fields.
2019-02-04 14:05:00 -06:00
Michael Basnight 66530dbde0
Deprecate HLRC security methods (#37883)
This commit deprecates the few methods that had their parameters
reordered to facilitate the move from EmptyResponse to boolean. This
commit also readds the boolean based methods with the proper
signatures.

Relates #37540
Relates #36938
2019-02-04 13:23:00 -06:00
Zachary Tong ab1150378b
Add Composite to AggregationBuilders (#38207) 2019-02-04 13:47:04 -05:00
Christoph Büscher 1899658a38
Mute ClusterClientIT#testClusterHealthYellowSpecificIndex (#38343) 2019-02-04 17:13:54 +01:00
Przemyslaw Gomulka 9b64558efb
Migrating from joda to java.time. Watcher plugin (#35809)
part of the migrating joda time work. Migrating watcher plugin to use JDK's java-time

refers #27330
2019-02-04 15:08:31 +01:00
Daniel Mitterdorfer d975f93967
Use stricter timer in DeadHostStateTests (#38301)
With this commit we add a monotonically strict timer to ensure time is
advancing even if the timer is called in a tight loop in tests. We also
relax a condition in a similar test so it only checks that time is not
moving backwards.

Closes #33747
2019-02-04 15:03:31 +01:00
Benjamin Trent a70f54fc77
Adding ml_settings entry to HLRC and Docs for deprecation_info (#38118) 2019-02-01 12:45:28 -06:00
Paul Sanwald 0d56955d39
mute test, as this one is failing also per #35450 (#38132) 2019-01-31 22:00:09 -05:00
Yuri Astrakhan f3cde06a1d
geotile_grid implementation (#37842)
Implements `geotile_grid` aggregation

This patch refactors previous implementation https://github.com/elastic/elasticsearch/pull/30240

This code uses the same base classes as `geohash_grid` agg, but uses a different hashing
algorithm to allow zoom consistency.  Each grid bucket is aligned to Web Mercator tiles.
2019-01-31 19:11:30 -05:00
Luca Cavanna 622fb7883b
Introduce ability to minimize round-trips in CCS (#37828)
With #37566 we have introduced the ability to merge multiple search responses into one. That makes it possible to expose a new way of executing cross-cluster search requests, that makes CCS much faster whenever there is network latency between the CCS coordinating node and the remote clusters. The coordinating node can now send a single search request to each remote cluster, which gets reduced by each one of them. from + size results are requested to each cluster, and the reduce phase in each cluster is non final (meaning that buckets are not pruned and pipeline aggs are not executed). The CCS coordinating node performs an additional, final reduction, which produces one search response out of the multiple responses received from the different clusters.

This new execution path will be activated by default for any CCS request unless a scroll is provided or inner hits are requested as part of field collapsing. The search API accepts now a new parameter called ccs_minimize_roundtrips that allows to opt-out of the default behaviour.

Relates to #32125
2019-01-31 15:12:14 +01:00