Commit Graph

40 Commits

Author SHA1 Message Date
Hendrik Muhs e177a38504
[7.x][Transform] add throttling (#56007) (#56184)
add throttling to transform, throttling will slow down search requests by
delaying the execution based on a documents per second metric.

fixes #54862
2020-05-05 13:09:02 +02:00
Hendrik Muhs 4b93f17b24 [Transform] improve TransformRestTestCase robustness (#55786)
handles/retries temporary SearchPhaseExecutionErrors

fixes #54810
2020-04-27 17:17:53 +02:00
William Brafford 49e30b15a2
Deprecate disabling basic-license features (#54816) (#55405)
We believe there's no longer a need to be able to disable basic-license
features completely using the "xpack.*.enabled" settings. If users don't
want to use those features, they simply don't need to use them. Having
such features always available lets us build more complex features that
assume basic-license features are present.

This commit deprecates settings of the form "xpack.*.enabled" for
basic-license features, excluding "security", which is a special case.
It also removes deprecated settings from integration tests and unit
tests where they're not directly relevant; e.g. monitoring and ILM are
no longer disabled in many integration tests.
2020-04-17 15:04:17 -04:00
Hendrik Muhs 223fbb2ae7 [Transform] fix sporadic test failure due to unavailable notif… (#54939)
move no initializing shards check before dumping audit messages

fixes #54810
2020-04-09 08:04:42 +02:00
Jason Tedor 5fcda57b37
Rename MetaData to Metadata in all of the places (#54519)
This is a simple naming change PR, to fix the fact that "metadata" is a
single English word, and for too long we have not followed general
naming conventions for it. We are also not consistent about it, for
example, METADATA instead of META_DATA if we were trying to be
consistent with MetaData (although METADATA is correct when considered
in the context of "metadata"). This was a simple find and replace across
the code base, only taking a few minutes to fix this naming issue
forever.
2020-03-31 17:24:38 -04:00
Benjamin Trent 374e76d7cd
[Transform] fixing naming in HLRC and _cat to match API content (#54300) (#54408)
Fixing the naming of the HLRC values to match the ToXContent field names (i.e. the field names returned from an API call).

Also fixes the names in the _cat API as well.

closes #53946
2020-03-30 08:57:02 -04:00
Hendrik Muhs cb0ecafdd8 [Transform] fix transform failure case for percentiles and spa… (#54202)
index null if percentiles could not be calculated due to sparse data

fixes #54201
2020-03-25 19:28:51 +01:00
Hendrik Muhs 7dcacf531f
[7.x][Transform][Rollup] add processing stats to record the ti… (#54027)
add 2 additional stats: processing time and processing total which capture the
time spent for processing results and how often it ran. The 2 new stats
correspond to the existing indexing and search stats. Together with indexing
and search this now allows the user to see the full picture, all 3 stages.
2020-03-24 09:22:02 +01:00
Gordon Brown 10cabbbade
Transition Transforms to using hidden indices for notifcations index (#53773)
This commit changes the Transforms notifications index to be hidden
index, with a hidden alias.

This commit also removes the temporary hack in
MetaDataCreateIndexService that prevents deprecation warnings for known
dot-prefixed index names which are not hidden/system indices, as this
was the last index pattern to need that hack.
2020-03-20 15:40:58 -06:00
Hendrik Muhs a6dca577e5 [Transform] data nanos/date histogram IT (#53654)
add an integration test for date nanos in combination with date_histogram
2020-03-17 20:58:57 +01:00
Hendrik Muhs 696aa4ddaf
[7.x][Transform] add support for script in group_by (#53167) (#53324)
add the possibility to base the group_by on the output of a script.

closes #43152
backport #53167
2020-03-10 11:12:58 +01:00
Hendrik Muhs d9258e210e [Transform] fix sporadic race condition in TransformUsageIT (#52946)
relax the test for trigger count

fixes #52931
2020-03-03 10:27:36 +01:00
Hendrik Muhs 288ccae23b [Transform] add support for filter aggregation (#52483)
add support for filter aggregations, refactor code for sub-aggregation support in mapping
deduction

fixes #52151
2020-02-21 14:05:11 +01:00
Hendrik Muhs 2071f85e1a forward audits to logs (#52394)
audit messages are stored in the notifications index, so audit information is lost for integration
tests. This change forwards audit messages to logs, so they can help to debug issues.

relates: #51627
2020-02-18 08:47:27 +01:00
Hendrik Muhs f0747e607d delete the transform to delete any docs which might have been written by the (#52360)
delete the transform to delete any docs which might have been written by the task after deleting
the index

fixes #51347
2020-02-16 11:23:06 +01:00
Hendrik Muhs efd7542b2a
[7.x][Transform] provide exponential_avg* stats for batch transforms (#52041) (#52323)
provide exponential_avg* stats for batch transforms, avoids confusion
why those values are all 0 otherwise
2020-02-14 07:48:23 +01:00
Hendrik Muhs c2b08bb72f [Transform] add support for percentile aggs (#51808)
make transform ready for multi value aggregations and add support for percentile

fixes #51663
2020-02-04 12:02:20 +01:00
Hendrik Muhs 53e4d1ef07 [Transform] fix TransformRobustnessIT intermittent test failures part 2 (#51523)
add wait for completion in transform robustness test to avoid occasional test failures during cleanup

fixes #51347
2020-01-28 13:37:01 +01:00
Hendrik Muhs 2239ba8c6e
[Transform] avoid mapping problems with index templates (#51368) (#51519)
insert explict mappings for objects in nested output to avoid clashes with index templates

fixes #51321
2020-01-28 11:31:07 +01:00
Hendrik Muhs 61663b495e add an integration test using date_nanos as timestamp (#51477)
add a test for using date_nanos as timestamp field in a continuous transform
2020-01-28 10:10:23 +01:00
Hendrik Muhs b233e93014
[Transform] refactor naming leftovers and apply code formating (#51465) (#51470)
refactor renaming leftovers: "data frame transform" to "transforms", touch only internals (variable
names, non-public API's, doc strings, ...) and apply code-formatting (spotless). No logical changes.
2020-01-27 14:04:57 +01:00
Hendrik Muhs d177747f66 fix TransformRobustnessIT intermittent test failures
ensure the cluster is not in some intermediate state when cleaning up.

fixes #51347
2020-01-24 15:19:11 +01:00
Hendrik Muhs ded7407b4d
[Transform] Adapt tests for error message to 7.x format
adapt messages to 7.x format (#51398)

fixes #51360
2020-01-24 12:17:32 +01:00
Zachary Tong 2e314a133f Mute TransformTaskFailedStateIT#testStartFailedTransform
Tracking issue: https://github.com/elastic/elasticsearch/issues/51360
2020-01-23 12:53:02 -05:00
Zachary Tong feb2a25761 Mute TransformTaskFailedStateIT#testForceStopFailedTransform
Tracking issue: https://github.com/elastic/elasticsearch/issues/51360
2020-01-23 11:58:36 -05:00
Hendrik Muhs 3553f68f5a [Transform] Handle permanent bulk indexing errors (#51307)
check bulk indexing error for permanent problems and ensure the state goes into failed instead of
retry. Corrects the stats API to show the real error and avoids excessive audit logging.

fixes #50122
2020-01-23 16:17:26 +01:00
Hendrik Muhs 13343b15c9 [Transform] Improve force stop robustness in case of an error (#51072)
If a transform config got lost (e.g. because the internal index disappeared) tasks could not be
stopped using transform API. This change makes it possible to stop transforms without a config,
meaning to remove the background task. In order to do so force must be set to true.
2020-01-17 07:42:21 +01:00
Hendrik Muhs c3e4405ddf
[7.x][Transform] Transform fix force stop race condition (#49249) (#49420)
fix force stopping transform if indexer state hasn't been written and/or is set to STOPPED. In certain situations the transform could not be stopped, which means the task could not be removed. Introduces improved abstraction in order to better test state handling in future.
2019-11-21 13:52:14 +01:00
Hendrik Muhs ca912624ec [Transform] improve error handling of script errors (#48887)
improve error handling for script errors, treating it as irrecoverable errors which puts the task
immediately into failed state, also improves the error extraction to properly report the script 
error.

fixes #48467
2019-11-18 10:24:39 +01:00
Rory Hunter c46a0e8708
Apply 2-space indent to all gradle scripts (#49071)
Backport of #48849. Update `.editorconfig` to make the Java settings the
default for all files, and then apply a 2-space indent to all `*.gradle`
files. Then reformat all the files.
2019-11-14 11:01:23 +00:00
Benjamin Trent 6ea59dd428
[ML][Transforms] add wait_for_checkpoint flag to stop (#47935) (#48591)
Adds `wait_for_checkpoint` for `_stop` API.
2019-10-28 13:02:57 -04:00
Hendrik Muhs 3da91d5f7a
[Transform] Rename internal indexes for transform plugin (#47788) (#47900)
rename internal indexes of transform plugin

- rename audit index and create an alias for accessing it, BWC: add an alias for old indexes to
  keep them working, kibana UI will switch to use the read alias
- rename config index and provide BWC to read from old and new ones
2019-10-11 14:16:17 +02:00
Hendrik Muhs 0e7869128a
[7.5][Transform] introduce new roles and deprecate old ones (#47780) (#47819)
deprecate data_frame_transforms_{user,admin} roles and introduce transform_{user,admin} roles as replacement
2019-10-10 10:31:24 +02:00
Hendrik Muhs 5e0e54f455
[Transform] move root endpoint to _transform with BWC layer (#47127) (#47682)
move the main endpoint to /_transform/ from /_data_frame/transforms/ with providing backwards compatibility and deprecation warnings
2019-10-08 08:59:01 +02:00
Hendrik Muhs 7377ac4637 [Transform] Replace transforms with transform, index constants (#47023)
- replace "transforms" with "transform" for consistency
 - use constants for internal index naming wherever possible and document required changes
2019-09-25 08:31:43 +02:00
Hendrik Muhs e974f178b5 [Transform] rename data frame transform to transform for hlrc client (#46933)
rename data frame transform to transform for hlrc
2019-09-25 08:31:43 +02:00
Hendrik Muhs abe889af75
[7.5][Transform] rename classes in transform plugin (#46867)
rename classes and settings in transform plugin, provide BWC for old settings
2019-09-20 10:43:00 +02:00
Benjamin Trent 9cf9c64ec2
[7.x] [ML][Transforms] remove `force` flag from _start (#46414) (#46748)
* [ML][Transforms] remove `force` flag from _start (#46414)

* [ML][Transforms] remove `force` flag from _start

* fixing expected error message

* adjusting bwc version
2019-09-18 10:06:05 -04:00
Hendrik Muhs c8f52ec4ff
[Transform] Rename data frame plugin to transform: classes in xpack.core (#46644) (#46734)
rename classes in xpack.core of transform plugin from "data frame transform" to "transform"
2019-09-16 13:39:22 +02:00
Hendrik Muhs efea581dcc
[7.x][Transform]Rename data frame plugin to transform: plugin and package names (#46583)
rename data frame transform plugin to transform:

 - rename plugin data-frame to transform
 - change all package names from o.e.*.dataframe.* to o.e.*.transform.*
 - necessary changes to fix loading/testing
2019-09-11 14:50:08 +02:00