Commit Graph

55 Commits

Author SHA1 Message Date
Hendrik Muhs a328a8eaf1
[7.x][Transform] implement node.transform to control where to… (#52998)
implement transform node attributes to disable transform on certain nodes and
test which nodes are allowed to do remote connections

closes #52200
closes #50033
closes #48734

backport #52712
2020-03-02 16:10:57 +01:00
Hendrik Muhs 49f41d127b [Transform] fix NPE in derive stats if shouldStopAtNextCheckpo… (#52940)
fixes a NPE in _stats in case shouldStopAtNextCheckpoint is set.
2020-03-02 08:11:01 +01:00
David Kyle 6e5e64559a
Unwrap cause from remote ActionTransportExceptions (#52842) (#52878)
And log the cause
2020-02-27 11:58:28 +00:00
Jay Modi f3f6ff97ee
Single instance of the IndexNameExpressionResolver (#52604)
This commit modifies the codebase so that our production code uses a
single instance of the IndexNameExpressionResolver class. This change
is being made in preparation for allowing name expression resolution
to be augmented by a plugin.

In order to remove some instances of IndexNameExpressionResolver, the
single instance is added as a parameter of Plugin#createComponents and
PersistentTaskPlugin#getPersistentTasksExecutor.

Backport of #52596
2020-02-21 07:50:02 -07:00
Hendrik Muhs 288ccae23b [Transform] add support for filter aggregation (#52483)
add support for filter aggregations, refactor code for sub-aggregation support in mapping
deduction

fixes #52151
2020-02-21 14:05:11 +01:00
Nik Everett 146def8caa
Implement top_metrics agg (#51155) (#52366)
The `top_metrics` agg is kind of like `top_hits` but it only works on
doc values so it *should* be faster.

At this point it is fairly limited in that it only supports a single,
numeric sort and a single, numeric metric. And it only fetches the "very
topest" document worth of metric. We plan to support returning a
configurable number of top metrics, requesting more than one metric and
more than one sort. And, eventually, non-numeric sorts and metrics. The
trick is doing those things fairly efficiently.

Co-Authored by: Zachary Tong <zach@elastic.co>
2020-02-14 11:19:11 -05:00
Hendrik Muhs efd7542b2a
[7.x][Transform] provide exponential_avg* stats for batch transforms (#52041) (#52323)
provide exponential_avg* stats for batch transforms, avoids confusion
why those values are all 0 otherwise
2020-02-14 07:48:23 +01:00
Jay Modi 5bcc6fce5c
Remove DeprecationLogger from route objects (#52285)
This commit removes the need for DeprecatedRoute and ReplacedRoute to
have an instance of a DeprecationLogger. Instead the RestController now
has a DeprecationLogger that will be used for all deprecated and
replaced route messages.

Relates #51950
Backport of #52278
2020-02-12 15:05:41 -07:00
Hendrik Muhs 5d35eaa1cb [Transform] improve irrecoverable error detection - part 2 (#52003)
base error handling on rest status instead of listing individual exception types

relates to #51820
2020-02-12 14:38:42 +01:00
Hendrik Muhs edaf6d1f79
[Transform] maintain a list of unsupported aggregations in transforms (#52190) (#52222)
add a list of unsupported aggs in transforms and create a test that fails if a new aggregation is
added. Limitation: works only if a new agg is added to either the core or a known plugin
(Analytics, MatrixAggregation).
2020-02-12 07:48:04 +01:00
Jay Modi 3edadfefd0 RestHandlers declare handled routes (#52123)
This commit changes how RestHandlers are registered with the
RestController so that a RestHandler no longer needs to register itself
with the RestController. Instead the RestHandler interface has new
methods which when called provide information about the routes
(method and path combinations) that are handled by the handler
including any deprecated and/or replaced combinations.

This change also makes the publication of RestHandlers safe since they
no longer publish a reference to themselves within their constructors.

Closes #51622

Co-authored-by: Jason Tedor <jason@tedor.me>

Backport of #51950
2020-02-09 22:48:32 -07:00
Hendrik Muhs 03fb5cdaae fallback to float if source type is scaled_float for mapping deduction (#51990)
fallback to float if source type is scaled_float for mapping deduction of min/max aggregation

fixes #51780
2020-02-06 17:27:26 +01:00
Hendrik Muhs c2b08bb72f [Transform] add support for percentile aggs (#51808)
make transform ready for multi value aggregations and add support for percentile

fixes #51663
2020-02-04 12:02:20 +01:00
Hendrik Muhs 5d5f3ce256 [Transform] improve irrecoverable error detection
treat resource not found and illegal argument exceptions as irrecoverable error

relates #50135
2020-02-04 10:36:35 +01:00
Hendrik Muhs ed170cc548
[Transform] Fix stats can return old state information if security is enabled (#51732) (#51738)
do index refresh of the internal transform index with the system user
instead of using the calling user which does not have sufficient rights
if security is enabled

fixes #51728
2020-02-01 19:34:58 +01:00
Ryan Ernst 21224caeaf Remove comparison to true for booleans (#51723)
While we use `== false` as a more visible form of boolean negation
(instead of `!`), the true case is implied and the true value does not
need to explicitly checked. This commit converts cases that have slipped
into the code checking for `== true`.
2020-01-31 16:35:43 -08:00
Gordon Brown 10c8179351
Use exclusions list instead of fake system indices (#51586)
This commit switches the strategy for managing dot-prefixed indices that
should be hidden indices from using "fake" system indices to an explicit
exclusions list that must be updated when those indices are converted to
hidden indices.
2020-01-30 16:31:27 -07:00
Gordon Brown 89c2834b24
Deprecate creation of dot-prefixed index names except for hidden and system indices (#49959)
This commit deprecates the creation of dot-prefixed index names (e.g.
.watches) unless they are either 1) a hidden index, or 2) registered by
a plugin that extends SystemIndexPlugin. This is the first step
towards more thorough protections for system indices.

This commit also modifies several plugins which use dot-prefixed indices
to register indices they own as system indices, and adds a plugin to
register .tasks as a system index.
2020-01-28 10:01:16 -07:00
Hendrik Muhs 2239ba8c6e
[Transform] avoid mapping problems with index templates (#51368) (#51519)
insert explict mappings for objects in nested output to avoid clashes with index templates

fixes #51321
2020-01-28 11:31:07 +01:00
Hendrik Muhs bebce4b190 audit index creation after it the index has been created (#51479)
moves audit message for index creation after the index has been successfully created. This has
been confusing for a user where index creation failed but audit reported index creation.
2020-01-28 10:06:46 +01:00
Hendrik Muhs b233e93014
[Transform] refactor naming leftovers and apply code formating (#51465) (#51470)
refactor renaming leftovers: "data frame transform" to "transforms", touch only internals (variable
names, non-public API's, doc strings, ...) and apply code-formatting (spotless). No logical changes.
2020-01-27 14:04:57 +01:00
Hendrik Muhs 3553f68f5a [Transform] Handle permanent bulk indexing errors (#51307)
check bulk indexing error for permanent problems and ensure the state goes into failed instead of
retry. Corrects the stats API to show the real error and avoids excessive audit logging.

fixes #50122
2020-01-23 16:17:26 +01:00
Hendrik Muhs 51134d9738 check custom meta data to avoid NPE (#51163)
check custom meta data to avoid NPE, fixes a problem introduced in #51072

fixes #51153
2020-01-20 13:53:42 +01:00
Hendrik Muhs 13343b15c9 [Transform] Improve force stop robustness in case of an error (#51072)
If a transform config got lost (e.g. because the internal index disappeared) tasks could not be
stopped using transform API. This change makes it possible to stop transforms without a config,
meaning to remove the background task. In order to do so force must be set to true.
2020-01-17 07:42:21 +01:00
Hendrik Muhs 0178c7c5d0
[7.x][Transform] correctly retrieve checkpoints from remote indices (#50903) (#50969)
uses remote client(s) to correctly retrieve index checkpoints from remote clusters
2020-01-14 15:09:14 +01:00
Dimitris Athanasiou 422422a2bc
[7.x][ML] Reuse SourceDestValidator for data frame analytics (#50841) (#50850)
This commit removes validation logic of source and dest indices
for data frame analytics and replaces it with using the common
`SourceDestValidator` class which is already used by transforms.
This way the validations and their messages become consistent
while we reduce code.

This means that where these validations fail the error messages
will be slightly different for data frame analytics.

Backport of #50841
2020-01-10 14:24:13 +02:00
Nik Everett ae40e22452
Drop "funny" functions building parsers (#50715) (#50814)
Replaces the "funny"
`Function<String, ConstructingObjectParser<T, Void>>` with a much
simpler `ConstructingObjectParser<T, String>`. This makes pretty much
all of our object parsers static.
2020-01-09 15:53:03 -05:00
Benjamin Trent 3e014d39c2
[Transform] fail to start/put on missing pipeline (#50701) (#50795)
If a pipeline referenced by a transform does not exist, we should not allow the transform to be created. 

We do allow the pipeline existence check to be skipped with defer_validations, but if the pipeline still does not exist on `_start`, the pipeline will fail to start.

relates:  #50135
2020-01-09 10:33:22 -05:00
Hendrik Muhs 7c10e9b0e7 [Transform] improve checkpoint reporting (#50369)
fixes empty checkpoints, re-factors checkpoint info creation (moves builder) and always reports
last change detection

relates #43201
relates #50018
2019-12-20 10:49:53 +01:00
Hendrik Muhs de14092ad2 [Transform] refactor source and dest validation to support CCS (#50018)
refactors source and dest validation, adds support for CCS, makes resolve work like reindex/search, allow aliased dest index with a single write index.

fixes #49988
fixes #49851
relates #43201
2019-12-20 10:49:53 +01:00
Hendrik Muhs 1c3ce110bd [Transform] add actual timeout in message (#50140)
add the timeout to the message if stopping a transform times out
2019-12-13 08:10:25 +01:00
Hendrik Muhs c33be29dc7 [Transform] automatic deletion of old checkpoints (#49496)
add automatic deletion of old checkpoints based on count and time
2019-12-04 07:55:57 +01:00
Hendrik Muhs 7aae212287
[Transform] Fix possible audit logging disappearance after rolling upgrade (#49731) (#49767)
ensure audit index template is available during a rolling upgrade before a
transform task can write to it.

fixes #49730
2019-12-03 18:05:06 +01:00
Hendrik Muhs 5256756879 [Transform] add debug log for configuration index (#49484)
add debug log for transform creation and disallow partial results for retrieval
2019-11-25 09:49:17 +01:00
Hendrik Muhs 1fbb248cb7 reenable warning checks in pivot tests (#49436) 2019-11-22 08:50:10 +01:00
Hendrik Muhs c3e4405ddf
[7.x][Transform] Transform fix force stop race condition (#49249) (#49420)
fix force stopping transform if indexer state hasn't been written and/or is set to STOPPED. In certain situations the transform could not be stopped, which means the task could not be removed. Introduces improved abstraction in order to better test state handling in future.
2019-11-21 13:52:14 +01:00
Hendrik Muhs ca912624ec [Transform] improve error handling of script errors (#48887)
improve error handling for script errors, treating it as irrecoverable errors which puts the task
immediately into failed state, also improves the error extraction to properly report the script 
error.

fixes #48467
2019-11-18 10:24:39 +01:00
Hendrik Muhs 5ecde37a68
[7.x][Transform] decouple task and indexer (#48812)
decouple TransformTask and ClientTransformIndexer. Interaction between the 2 classes are
now moved into a context class which holds shared information.

relates #45369
2019-11-01 19:39:35 +01:00
Benjamin Trent 6ea59dd428
[ML][Transforms] add wait_for_checkpoint flag to stop (#47935) (#48591)
Adds `wait_for_checkpoint` for `_stop` API.
2019-10-28 13:02:57 -04:00
Hendrik Muhs ba1c13c47d [Transform] do not fail checkpoint creation due to global checkpoint mismatch (#48423)
Take the max if global checkpoints mismatch instead of throwing an exception. It turned out global
checkpoints can mismatch by design

fixes #48379
2019-10-24 12:22:07 +02:00
Hendrik Muhs b2ce72850b
[7.5][Transform] prevent assignment if any node is older than 7.4 (#48055)
disable task assignment of transforms if any node uses version 7.2 or 7.2 (mixed cluster).

fixes #48019
2019-10-15 16:14:39 +02:00
Hendrik Muhs 4aa7c7bad6
[Transform] add alias for backwards compatibility with 7.4 (#48049)
add alias for backwards compatibility with 7.4

relates #47943
2019-10-15 15:04:09 +02:00
Hendrik Muhs 17d8ee9a9c [Transform] wait for deprecated index shards to get active (#47997)
wait for deprecated index shards to get active
2019-10-14 20:14:30 +02:00
Benjamin Trent 508db4589b
[ML][Transforms] signal listener early on stop failure (#47954) (#48002) 2019-10-14 11:17:11 -04:00
Hendrik Muhs 0ca53bd80e
add BWC alias for internal index
create an alias for old nodes to retrieve new documents in the internal index as they do not know the new index pattern
2019-10-11 17:15:01 +02:00
Hendrik Muhs 3da91d5f7a
[Transform] Rename internal indexes for transform plugin (#47788) (#47900)
rename internal indexes of transform plugin

- rename audit index and create an alias for accessing it, BWC: add an alias for old indexes to
  keep them working, kibana UI will switch to use the read alias
- rename config index and provide BWC to read from old and new ones
2019-10-11 14:16:17 +02:00
Hendrik Muhs 5e0e54f455
[Transform] move root endpoint to _transform with BWC layer (#47127) (#47682)
move the main endpoint to /_transform/ from /_data_frame/transforms/ with providing backwards compatibility and deprecation warnings
2019-10-08 08:59:01 +02:00
Rory Hunter 53a4d2176f
Convert most awaitBusy calls to assertBusy (#45794) (#47112)
Backport of #45794 to 7.x. Convert most `awaitBusy` calls to
`assertBusy`, and use asserts where possible. Follows on from #28548 by
@liketic.

There were a small number of places where it didn't make sense to me to
call `assertBusy`, so I kept the existing calls but renamed the method to
`waitUntil`. This was partly to better reflect its usage, and partly so
that anyone trying to add a new call to awaitBusy wouldn't be able to find
it.

I also didn't change the usage in `TransportStopRollupAction` as the
comments state that the local awaitBusy method is a temporary
copy-and-paste.

Other changes:

  * Rework `waitForDocs` to scale its timeout. Instead of calling
    `assertBusy` in a loop, work out a reasonable overall timeout and await
    just once.
  * Some tests failed after switching to `assertBusy` and had to be fixed.
  * Correct the expect templates in AbstractUpgradeTestCase.  The ES
    Security team confirmed that they don't use templates any more, so
    remove this from the expected templates. Also rewrite how the setup
    code checks for templates, in order to give more information.
  * Remove an expected ML template from XPackRestTestConstants The ML team
    advised that the ML tests shouldn't be waiting for any
    `.ml-notifications*` templates, since such checks should happen in the
    production code instead.
  * Also rework the template checking code in `XPackRestTestHelper` to give
    more helpful failure messages.
  * Fix issue in `DataFrameSurvivesUpgradeIT` when upgrading from < 7.4
2019-09-29 12:21:46 +01:00
Hendrik Muhs 7377ac4637 [Transform] Replace transforms with transform, index constants (#47023)
- replace "transforms" with "transform" for consistency
 - use constants for internal index naming wherever possible and document required changes
2019-09-25 08:31:43 +02:00
Hendrik Muhs abe889af75
[7.5][Transform] rename classes in transform plugin (#46867)
rename classes and settings in transform plugin, provide BWC for old settings
2019-09-20 10:43:00 +02:00