Commit Graph

747 Commits

Author SHA1 Message Date
Armin Braun 389a13b68e
Mute BulkProcessorRetryIT#testBulkRejectionLoadWithBackoff (#41325) (#41331)
* For #41324
2019-04-18 11:55:28 +02:00
David Kyle 1d2365f5b6 [ML-DataFrame] Refactorings and tidying (#41248)
Remove unnecessary generic params from SingleGroupSource
and unused code from the HLRC
2019-04-17 14:58:26 +01:00
David Roberts 6cc35d3724 [ML] Unmute MachineLearningIT.testDeleteExpiredData (#41186)
The cause of failure was fixed by elastic/ml-cpp#459,
so all that remains on the Java side is to unmute the
test that was failing.

Closes #41070
2019-04-16 16:38:46 +01:00
Christoph Büscher 2980a6c70f Clarify some ToXContent implementations behaviour (#41000)
This change adds either ToXContentObject or ToXContentFragment to classes
directly implementing ToXContent currently. This helps in reasoning about
whether those implementations output full xcontent object or just fragments.

Relates to #16347
2019-04-15 09:42:08 +02:00
Martijn van Groningen 1eff8976a8
Deprecate AbstractHlrc* and AbstractHlrcStreamable* base test classes (#41014)
* moved hlrc parsing tests from xpack to hlrc module and removed dependency on hlrc from xpack core

* deprecated old base test class

* added deprecated jdoc tag

* split test between xpack-core part and hlrc part

* added lang-mustache test dependency, this previously came in via
hlrc dependency.

* added hlrc dependency on a qa module

* duplicated ClusterPrivilegeName class in xpack-core, since x-pack
core no longer has a dependency on hlrc.

* replace ClusterPrivilegeName usages with string literals

* moved tests to dedicated to hlrc packages in order to remove Hlrc part from the name and make sure to use imports instead of full qualified class where possible

* remove ESTestCase. from method invocation and use method directly,
because these tests indirectly extend from ESTestCase
2019-04-10 16:29:17 +02:00
Ed Savage 722362e402 Mute MachineLearningIt#testDeleteExpiredData
Tracked in #41070
2019-04-10 13:07:53 +01:00
Hendrik Muhs f9018ab11b [ML-DataFrame] create checkpoints on every new run (#40725)
Use the checkpoint service to create a checkpoint on every new run. Expose checkpoints stats on _stats endpoint.
2019-04-10 09:14:11 +02:00
Martijn van Groningen 46b0fdae33
Add realistic hlrc request serialization test base class and (#40362)
changed hlrc ccr request tests to use AbstractRequestTestCase base class.

This way the request classes are tested in a more realistic setting.
Note this change also adds a test dependency on xpack core module.

Similar to #39844 but then for hlrc request serialization tests.

Removed iterators from hlrc parsing tests.
Use empty xcontent registries.

Relates to #39745
2019-04-10 08:00:01 +02:00
Ed Savage fdc1bdd4d3 [ML][TEST] Fix randomly failing HLRC test (#40973)
Made changes to ensure that unique IDs are generated for model snapshots
used by the deleteExpiredDataTest test in the MachineLearningIT suite.

Previously a sleep of 1s was performed between jobs under the assumption
that this would be sufficient to guarantee that the timestamps used in
the composition of the snapshot IDs would be different.

The new approach is to wait on the condition that the old and new
timestamps are in fact different (to 1s resolution).
2019-04-09 16:55:35 +01:00
Martijn van Groningen 040a4961c7
Revert "Revert "Change HLRC CCR response tests to use AbstractResponseTestCase base class. (#40257)"" (#40971)
This reverts commit df91237a94fd3d3ae954eb1845c434dda692d087.
2019-04-09 15:18:33 +02:00
Martijn van Groningen 84a410d5a8
Revert "Change HLRC CCR response tests to use AbstractResponseTestCase base class. (#40257)"
This reverts commit c29027d99e.
2019-04-08 11:24:06 +02:00
Martijn van Groningen c29027d99e
Change HLRC CCR response tests to use AbstractResponseTestCase base class. (#40257)
This way the response classes are tested in a more realistic setting.

Relates to #39745
2019-04-08 07:09:28 +02:00
roy 075078e7e0 HLRC: fix uri encode bug when url path starts with '/' (#34436)
This commit sets the authority of a URI to blank such that it does not
misinterpret slashes in the path as the authority.

Closes #34433
2019-04-04 12:59:59 -05:00
Michael Basnight fb5a0652a8 HLRC: Convert xpack methods to client side objects (#40705)
This commit fixes a problem with BWC that was brought up in #40511. A
newer version of the code was emitting a new value for an enum to an
older version, and the older version could not handle that. It caused
the response to error. The MainResponse is now relaxed, and will accept
whatever values the server expose, and holds most of them as Strings
instead of complex objects.

Fixes #40511
2019-04-04 11:06:44 -05:00
Hendrik Muhs 31e79a73d7 add HLRC protocol tests for transform state and stats (#40766)
adds HLRC protocol tests for state and stats hrlc clients
2019-04-03 12:51:15 +02:00
Hendrik Muhs 1f947054ff add reason to DataFrameTransformState and add hlrc protocol tests (#40736)
add field "reason" to DataFrameTransformState, add hlrc protocol tests and allow unknown fields for DataFrameTransformState
2019-04-03 07:35:07 +02:00
Tim Vernum 2c770ba3cb
Support mustache templates in role mappings (#40571)
This adds a new `role_templates` field to role mappings that is an
alternative to the existing roles field.

These templates are evaluated at runtime to determine which roles should be
granted to a user.
For example, it is possible to specify:

    "role_templates": [
      { "template":{ "source": "_user_{{username}}" } }
    ]

which would mean that every user is assigned to their own role based on
their username.

You may not specify both roles and role_templates in the same role
mapping.

This commit adds support for templates to the role mapping API, the role
mapping engine, the Java high level rest client, and Elasticsearch
documentation.

Due to the lack of caching in our role mapping store, it is currently
inefficient to use a large number of templated role mappings. This will be
addressed in a future change.

Backport of: #39984, #40504
2019-04-02 20:55:10 +11:00
Adrien Grand 7f7d09af2e
Deprecate types in `_graph/explore` calls. (#40466) (#40513)
Any call that uses a path that sets a type will trigger a deprecation warning.
2019-03-28 09:32:26 +01:00
Adrien Grand 65a35c985c
Remove type from VersionConflictEngineException. (#37490) (#40514)
It initially mentioned the type in the exception because the type used to be
required to uniquely identify a document. This is not necessary anymore given
that indices have at most one type.
2019-03-28 09:32:09 +01:00
Andy Bristol c0c6d702a2
ignore 409 conflict in reindex responses (#39543)
The reindex family of APIs (reindex, update-by-query, delete-by-query) can
sometimes return responses that have an error status code (409 Conflict in this
case) but still have a body in the usual BulkByScrollResponse format. When the
HLRC tries to handle such responses, it blows up because it tris to parse it
expecting the error format that errors in general use. This change prompts the
HLRC to parse the response using the expected BulkByScrollResponse format.
2019-03-27 13:27:17 -07:00
David Kyle c990b30019
[ML] Data Frame HLRC Get API (#40509) 2019-03-27 12:40:39 +00:00
Benjamin Trent 12943c5d2c
[ML] Add data frame task state object and field (#40169) (#40490)
* [ML] Add data frame task state object and field

* A new state item is added so that the overall task state can be
accoutned for
* A new FAILED state and reason have been added as well so that failures
can be shown to the user for optional correction

* Addressing PR comments

* adjusting after master merge

* addressing pr comment

* Adjusting auditor usage with failure state

* Refactor, renamed state items to task_state and indexer_state

* Adding todo and removing redundant auditor call

* Address HLRC changes and PR comment

* adjusting hlrc IT test
2019-03-27 06:53:58 -05:00
David Kyle 1354696db9
[ML] Data Frame HLRC Get Stats API (#40443) 2019-03-26 11:17:13 +00:00
Benjamin Trent 7b4f964708
[ML] make source and dest objects in the transform config (#40337) (#40396)
* [ML] make source and dest objects in the transform config

* addressing PR comments

* Fixing compilation post merge

* adding comment for Arrays.hashCode

* addressing changes for moving dest to object

* fixing data_frame yml tests

* fixing API test
2019-03-25 07:16:41 -05:00
David Kyle a4cb92a300
[ML] Data Frame HLRC Preview API (#40258) 2019-03-21 09:38:27 +00:00
Martijn van Groningen a478196f79
Add realistic hlrc response serialization test base class (#39844)
The base class facilitates generating a server side response test instance,
that gets serialized as xcontent, which then gets parsed into a hlrc response
instance, which then gets asserted against the server side response instance.

This way of testing is more realistic then how hlrc response classes are tested
today, which basically tests that serialization works by generating
hlrc response instance, serialize that to xcontent and then parse it back
to a hlrc response instance.

Besides adding a base test class, this change also cuts AcknowledgedResponseTests
and BroadcastResponseTests over to use this base class.

Relates to #39745
2019-03-20 13:35:34 +01:00
David Kyle 387648065d
[ML] Data Frame HLRC start & stop APIs (#40197) 2019-03-19 13:30:01 +00:00
Gordon Brown c8a4a7fc9d
Remove Migration Upgrade and Assistance APIs (#40075)
The Migration Assistance API has been functionally replaced by the
Deprecation Info API, and the Migration Upgrade API is not used for the
transition from ES 6.x to 7.x, and does not need to be kept around to
repair indices that were not properly upgraded before upgrading the
cluster, as was the case in 6.
2019-03-18 13:46:56 -06:00
Jack Conradson b57af6c401 Add a Painless Context REST API (#39382)
This PR adds an internal REST API for querying context information about 
Painless whitelists.

Commands include the following:
GET /_scripts/painless/_context -- retrieves a list of contexts
GET /_scripts/painless/_context?context=%name% retrieves all available 
information about the API for this specific context
2019-03-14 12:42:12 -07:00
Ioannis Kakavas ae0ff05c90 Adjust testGetSslCertificates to run in FIPS (#40046)
As discovered in #40041, when parsing certificates from files, the
SUN Security Provider normalizes DNs from parsed certificates by
adding spaces between RDNs, while the BouncyCastle one (which we
use in FIPS tests) does not.

We could proceed to normalize the DNs in the same manner in this
test by using i.e. the Unbound LDAP SDK but since the goal of this
test is to validate that we do get to read these exact certificates
from our trust sources and not to validate subject DNs, this commit
changes the test to check the serial number instead

Resolves: #40041
2019-03-14 20:25:47 +02:00
David Kyle c02f49e9d3
[ML-Dataframe] Add Data Frame client to the Java HLRC (#40040)
Adds DataFrameClient to the Java HLRC and implements PUT and 
DELETE data frame transform.
2019-03-14 14:57:12 +00:00
Ioannis Kakavas 2512cf3ec8 Mute testGetSslCertificates in FIPS (#40042) 2019-03-14 14:23:46 +02:00
Adrien Grand b841de2e38
Don't emit deprecation warnings on calls to the monitoring bulk API. (#39805) (#39838)
The monitoring bulk API accepts the same format as the bulk API, yet its concept
of types is different from "mapping types" and the deprecation warning is only
emitted as a side-effect of this API reusing the parsing logic of bulk requests.

This commit extracts the parsing logic from `_bulk` into its own class with a
new flag that allows to configure whether usage of `_type` should emit a warning
or not. Support for payloads has been removed for simplicity since they were
unused.

@jakelandis has a separate change that removes this notion of type from the
monitoring bulk API that we are considering bringing to 8.0.
2019-03-11 07:58:28 +01:00
David Kyle 6c2e831e94
[ML-Dataframe] Data frame config HLRC objects (#39825) 2019-03-08 12:18:55 +00:00
Jason Tedor 0250d554b6
Introduce forget follower API (#39718)
This commit introduces the forget follower API. This API is needed in cases that
unfollowing a following index fails to remove the shard history retention leases
on the leader index. This can happen explicitly through user action, or
implicitly through an index managed by ILM. When this occurs, history will be
retained longer than necessary. While the retention lease will eventually
expire, it can be expensive to allow history to persist for that long, and also
prevent ILM from performing actions like shrink on the leader index. As such, we
introduce an API to allow for manual removal of the shard history retention
leases in this case.
2019-03-07 11:08:45 -05:00
David Roberts e94d32d069 Add roles and cluster privileges for data frame transforms (#39661)
This change adds two new cluster privileges:

* manage_data_frame_transforms
* monitor_data_frame_transforms

And two new built-in roles:

* data_frame_transforms_admin
* data_frame_transforms_user

These permit access to the data frame transform endpoints.
(Index privileges are also required on the source and
destination indices for each data frame transform, but
since these indices are configurable they it is not
appropriate to grant them via built-in roles.)
2019-03-05 14:07:25 +00:00
Martijn van Groningen ea4e9ed0a9
Add missing tests for CcrRequestConverters (#39228) 2019-02-25 12:48:43 +01:00
Tal Levy f1c8aa816f fix dangling tag in TasksClientDocumentationIT (#39157)
fix dangling tag in TasksClientDocumentationIT
and fix tag mentions from list-tasks to cancel-tasks
2019-02-20 08:48:58 -08:00
Martijn van Groningen 0594a467f2
Add support for ccr follow info api to HLRC. (#39115)
This API was introduces after #33824 was closed.
2019-02-20 16:10:54 +01:00
Hendrik Muhs 2e2567e827 add debug info for intermittent test failure 2019-02-19 10:54:09 +01:00
Hendrik Muhs 4f662bd289
Add data frame feature (#38934) (#39029)
The data frame plugin allows users to create feature indexes by pivoting a source index. In a
nutshell this can be understood as reindex supporting aggregations or similar to the so called entity
centric indexing.

Full history is provided in: feature/data-frame-transforms
2019-02-18 11:07:29 +01:00
Paul Sanwald 408a800f74
ClusterClientIT refactor (#38872) (#39002)
Add fixes for ClusterClientIT test and unmute tests.
2019-02-17 20:37:26 -05:00
Martijn van Groningen 9492dcea42
Update IndexTemplateMetaData to allow unknown fields (#38448)
Updated IndexTemplateMetaData to use ObjectParser.

The IndexTemplateMetaData class used old parsing logic and was not
resiliant to new fields. This commit updates it to use the
ConstructingObjectParser and allow unknown fields.

Relates #36938

Co-authored-by: Michael Basnight <mbasnight@gmail.com>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2019-02-06 08:21:30 +01:00
Lee Hinman a6ce671751
Re-enable TasksClientDocumentationIT.testCancelTasks (#38234)
This test has been disabled since November 2018, but I was not able to reproduce
the failure. Re-enabling this so we can see the full log and get more context if
it fails again.

Relates to #35514
2019-02-05 13:42:43 -07:00
Boaz Leskes 8ad9a07b87 CRUDDocumentationIT fix documentation references 2019-02-05 21:27:28 +01:00
Boaz Leskes 033ba725af
Remove support for internal versioning for concurrency control (#38254)
Elasticsearch has long [supported](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-index_.html#index-versioning) compare and set (a.k.a optimistic concurrency control) operations using internal document versioning. Sadly that approach is flawed and can sometime do the wrong thing. Here's the relevant excerpt from the resiliency status page:

> When a primary has been partitioned away from the cluster there is a short period of time until it detects this. During that time it will continue indexing writes locally, thereby updating document versions. When it tries to replicate the operation, however, it will discover that it is partitioned away. It won’t acknowledge the write and will wait until the partition is resolved to negotiate with the master on how to proceed. The master will decide to either fail any replicas which failed to index the operations on the primary or tell the primary that it has to step down because a new primary has been chosen in the meantime. Since the old primary has already written documents, clients may already have read from the old primary before it shuts itself down. The version numbers of these reads may not be unique if the new primary has already accepted writes for the same document 

We recently [introduced](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/optimistic-concurrency-control.html) a new sequence number based approach that doesn't suffer from this dirty reads problem. 

This commit removes support for internal versioning as a concurrency control mechanism in favor of the sequence number approach.

Relates to #1078
2019-02-05 20:53:35 +01:00
Julie Tibshirani 3ce7d2c9b6
Make sure to reject mappings with type _doc when include_type_name is false. (#38270)
`CreateIndexRequest#source(Map<String, Object>, ... )`, which is used when
deserializing index creation requests, accidentally accepts mappings that are
nested twice under the type key (as described in the bug report #38266).

This in turn causes us to be too lenient in parsing typeless mappings. In
particular, we accept the following index creation request, even though it
should not contain the type key `_doc`:

```
PUT index?include_type_name=false
{
  "mappings": {
    "_doc": {
      "properties": { ... }
    }
  }
}
```

There is a similar issue for both 'put templates' and 'put mappings' requests
as well.

This PR makes the minimal changes to detect and reject these typed mappings in
requests. It does not address #38266 generally, or attempt a larger refactor
around types in these server-side requests, as I think this should be done at a
later time.
2019-02-05 10:52:32 -08:00
Boaz Leskes 12657fda44
`if_seq_no` and `if_primary_term` parameters aren't wired correctly in REST Client's CRUD API (#38411) 2019-02-05 18:05:56 +01:00
Julie Tibshirani 440d1eda8a
Fix failures in BulkProcessorIT#testGlobalParametersAndBulkProcessor. (#38129)
This PR fixes a couple test issues:
* It narrows an assertWarnings call that was too broad, and wasn't always
  applicable with certain random sequences.
* Previously, we could send a typeless bulk request containing '_type: 'null'.
  Now we omit the _type key altogether for typeless requests.
2019-02-05 08:42:37 -08:00
Michael Basnight 8742db3afe
Update Rollup Caps to allow unknown fields (#38339)
This commit ensures that the parts of rollup caps that can allow unknown
fields will allow them. It also modifies the test such that we can use
the features we need for disallowing fields in spots where they would
not be allowed.

Relates #36938
2019-02-05 10:08:08 -06:00