Commit Graph

22 Commits

Author SHA1 Message Date
Hendrik Muhs ca912624ec [Transform] improve error handling of script errors (#48887)
improve error handling for script errors, treating it as irrecoverable errors which puts the task
immediately into failed state, also improves the error extraction to properly report the script 
error.

fixes #48467
2019-11-18 10:24:39 +01:00
Rory Hunter c46a0e8708
Apply 2-space indent to all gradle scripts (#49071)
Backport of #48849. Update `.editorconfig` to make the Java settings the
default for all files, and then apply a 2-space indent to all `*.gradle`
files. Then reformat all the files.
2019-11-14 11:01:23 +00:00
Hendrik Muhs 5ecde37a68
[7.x][Transform] decouple task and indexer (#48812)
decouple TransformTask and ClientTransformIndexer. Interaction between the 2 classes are
now moved into a context class which holds shared information.

relates #45369
2019-11-01 19:39:35 +01:00
Benjamin Trent 6ea59dd428
[ML][Transforms] add wait_for_checkpoint flag to stop (#47935) (#48591)
Adds `wait_for_checkpoint` for `_stop` API.
2019-10-28 13:02:57 -04:00
Hendrik Muhs ba1c13c47d [Transform] do not fail checkpoint creation due to global checkpoint mismatch (#48423)
Take the max if global checkpoints mismatch instead of throwing an exception. It turned out global
checkpoints can mismatch by design

fixes #48379
2019-10-24 12:22:07 +02:00
Hendrik Muhs b2ce72850b
[7.5][Transform] prevent assignment if any node is older than 7.4 (#48055)
disable task assignment of transforms if any node uses version 7.2 or 7.2 (mixed cluster).

fixes #48019
2019-10-15 16:14:39 +02:00
Hendrik Muhs 4aa7c7bad6
[Transform] add alias for backwards compatibility with 7.4 (#48049)
add alias for backwards compatibility with 7.4

relates #47943
2019-10-15 15:04:09 +02:00
Hendrik Muhs 17d8ee9a9c [Transform] wait for deprecated index shards to get active (#47997)
wait for deprecated index shards to get active
2019-10-14 20:14:30 +02:00
Benjamin Trent 508db4589b
[ML][Transforms] signal listener early on stop failure (#47954) (#48002) 2019-10-14 11:17:11 -04:00
Hendrik Muhs 0ca53bd80e
add BWC alias for internal index
create an alias for old nodes to retrieve new documents in the internal index as they do not know the new index pattern
2019-10-11 17:15:01 +02:00
Hendrik Muhs 3da91d5f7a
[Transform] Rename internal indexes for transform plugin (#47788) (#47900)
rename internal indexes of transform plugin

- rename audit index and create an alias for accessing it, BWC: add an alias for old indexes to
  keep them working, kibana UI will switch to use the read alias
- rename config index and provide BWC to read from old and new ones
2019-10-11 14:16:17 +02:00
Hendrik Muhs 0e7869128a
[7.5][Transform] introduce new roles and deprecate old ones (#47780) (#47819)
deprecate data_frame_transforms_{user,admin} roles and introduce transform_{user,admin} roles as replacement
2019-10-10 10:31:24 +02:00
Hendrik Muhs 5e0e54f455
[Transform] move root endpoint to _transform with BWC layer (#47127) (#47682)
move the main endpoint to /_transform/ from /_data_frame/transforms/ with providing backwards compatibility and deprecation warnings
2019-10-08 08:59:01 +02:00
Rory Hunter 53a4d2176f
Convert most awaitBusy calls to assertBusy (#45794) (#47112)
Backport of #45794 to 7.x. Convert most `awaitBusy` calls to
`assertBusy`, and use asserts where possible. Follows on from #28548 by
@liketic.

There were a small number of places where it didn't make sense to me to
call `assertBusy`, so I kept the existing calls but renamed the method to
`waitUntil`. This was partly to better reflect its usage, and partly so
that anyone trying to add a new call to awaitBusy wouldn't be able to find
it.

I also didn't change the usage in `TransportStopRollupAction` as the
comments state that the local awaitBusy method is a temporary
copy-and-paste.

Other changes:

  * Rework `waitForDocs` to scale its timeout. Instead of calling
    `assertBusy` in a loop, work out a reasonable overall timeout and await
    just once.
  * Some tests failed after switching to `assertBusy` and had to be fixed.
  * Correct the expect templates in AbstractUpgradeTestCase.  The ES
    Security team confirmed that they don't use templates any more, so
    remove this from the expected templates. Also rewrite how the setup
    code checks for templates, in order to give more information.
  * Remove an expected ML template from XPackRestTestConstants The ML team
    advised that the ML tests shouldn't be waiting for any
    `.ml-notifications*` templates, since such checks should happen in the
    production code instead.
  * Also rework the template checking code in `XPackRestTestHelper` to give
    more helpful failure messages.
  * Fix issue in `DataFrameSurvivesUpgradeIT` when upgrading from < 7.4
2019-09-29 12:21:46 +01:00
Hendrik Muhs 7377ac4637 [Transform] Replace transforms with transform, index constants (#47023)
- replace "transforms" with "transform" for consistency
 - use constants for internal index naming wherever possible and document required changes
2019-09-25 08:31:43 +02:00
Hendrik Muhs e974f178b5 [Transform] rename data frame transform to transform for hlrc client (#46933)
rename data frame transform to transform for hlrc
2019-09-25 08:31:43 +02:00
Hendrik Muhs abe889af75
[7.5][Transform] rename classes in transform plugin (#46867)
rename classes and settings in transform plugin, provide BWC for old settings
2019-09-20 10:43:00 +02:00
Benjamin Trent 9cf9c64ec2
[7.x] [ML][Transforms] remove `force` flag from _start (#46414) (#46748)
* [ML][Transforms] remove `force` flag from _start (#46414)

* [ML][Transforms] remove `force` flag from _start

* fixing expected error message

* adjusting bwc version
2019-09-18 10:06:05 -04:00
Benjamin Trent 92acc732de
[ML][Transform] Use field caps for mapping deductino (#46703) (#46742) 2019-09-16 10:05:55 -04:00
Hendrik Muhs c8f52ec4ff
[Transform] Rename data frame plugin to transform: classes in xpack.core (#46644) (#46734)
rename classes in xpack.core of transform plugin from "data frame transform" to "transform"
2019-09-16 13:39:22 +02:00
David Roberts 07a0140260
[ML-DataFrame] Ensure latest index template exists before indexing docs (#46595)
When upgrading data nodes to a newer version before
master nodes there was a risk that a transform running
on an upgraded data node would index a document into
the new transforms internal index before its index
template was created.  This would cause the index to
be created with entirely dynamic mappings.

This change introduces a check before indexing any
internal transforms document to ensure that the required
index template exists and create it if it doesn't.

Backport of #46553
2019-09-11 16:27:26 +01:00
Hendrik Muhs efea581dcc
[7.x][Transform]Rename data frame plugin to transform: plugin and package names (#46583)
rename data frame transform plugin to transform:

 - rename plugin data-frame to transform
 - change all package names from o.e.*.dataframe.* to o.e.*.transform.*
 - necessary changes to fix loading/testing
2019-09-11 14:50:08 +02:00