Commit Graph

30037 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe 76cd7b1eb2
Fixes compile errors in Eclipse due to generics
PersistentTasksCustomMetadata was using a generic param named `Params`. This conflicted with the imported interface `ToXContent.Params`. The java compiler was preferring the generic param over the interface so everything was fine but Eclipse apparently prefers the interface int his case which was screwing up the Hierarchy and causing compile errors in Eclipse. This changes fixes it by renaming the Generic param to `P`
2018-01-31 12:27:34 +01:00
Igor Motov fc524bc9b5
Persistent Tasks: force writeable name of params and status to be the same as their task (#1072)
Changes persistent task serialization and forces params and status to have the same writeable name as the task itself.
2018-01-31 12:27:34 +01:00
Martijn van Groningen 4771965931
Use task builder instead of creating persistent tasks directly. 2018-01-31 12:27:34 +01:00
Igor Motov abd9ae399c
Persistent Tasks: PersistentTaskRequest -> PersistTaskParams (#1057)
Removes the last pieces of ActionRequest from PersistentTaskRequest and renames it into PersistTaskParams, which is now just an interface that extends NamedWriteable and ToXContent.
2018-01-31 12:27:33 +01:00
Igor Motov 6bfea09dd6
Persistent Tasks: switch from long task ids to string task ids (#1035)
This commit switches from long persistent task ids to caller-supplied string persistent task ids.
2018-01-31 12:27:33 +01:00
Hendrik Muhs 0a1f25588b
Added PersistentTasksService#waitForPersistentTasksStatus(...) method to allow callers to wait when an executor node has updated its task status. 2018-01-31 12:27:33 +01:00
Igor Motov 0a1abd430d
Persistent Tasks: remove listener from PersistentTasksExecutor#nodeOperation (#1032)
Instead of having a separate listener for indicating that the current task is finished, this commit is switching to use allocated object itself.
2018-01-31 12:27:32 +01:00
Igor Motov 95c6005f6f
Persistent Tasks: remove retries on notification failures (#977)
Retries should be already handled by TransportMasterNodeAction, there is no need to introduce another retry layer in Persistent Tasks code.
2018-01-31 12:27:32 +01:00
Martijn van Groningen fab0dc449a
Remove PersistentTask#isCurrentStatus() usages 2018-01-31 12:27:32 +01:00
Igor Motov 5a8512bf4e
Persistent Tasks: refactor PersistentTasksService to use ActionListener (#937)
PersistentTasksService methods are not using ActionListener<PersistentTask<?>> instead of PersistentTaskOperationListener.
2018-01-31 12:27:29 +01:00
Jason Tedor 97822dbea3
Respond to rename random ASCII helper methods
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.
2018-01-31 12:00:10 +01:00
Igor Motov 5b45b167bd
Persistent Tasks: check the current state in waitForPersistentTaskStatus (#935)
Add a check for the current state waitForPersistentTaskStatus before waiting for the next one. This fixes sporadic failure in testPersistentActionStatusUpdate test.

Fixes #928
2018-01-31 12:00:09 +01:00
Martijn van Groningen a5acb556b0
Use PersistentTasksService#waitForPersistentTaskStatus(...) to wait for job and datafeed status and use PersistentTasksService#removeTask(...) to force close job and force stop datafeed. 2018-01-31 12:00:09 +01:00
Igor Motov 1b0f5b9572
Persistent Tasks: require correct allocation id for status updates (#923)
In order to prevent tasks state updates by stale executors, this commit adds a check for correct allocation id during status update operation.
2018-01-31 12:00:09 +01:00
Igor Motov 6ca044736e
Persistent Tasks: Add waitForPersistentTaskStatus method (#901)
This method allows to wait for tasks to change their status to match the supplied predicate.
2018-01-31 12:00:09 +01:00
Martijn van Groningen 78b844e79b
Check allocationIdOnLastStatusUpdate when trying to detect whether a task is stale. 2018-01-31 11:59:02 +01:00
Igor Motov b142d7e29c
Persistent Tasks: Remove unused stopped and removeOnCompletion flags (#853)
The stopped and removeOnCompletion flags are not currently used, this commit removes them for now to simplify things.
2018-01-31 11:59:01 +01:00
Igor Motov 37fad04879
Persistent Tasks: Merge NodePersistentTask and RunningPersistentTask (#842)
Refactors NodePersistentTask and RunningPersistentTask into a single AllocatedPersistentTask. Makes it possible to update Persistent Task Status via AllocatedPersistentTask.
2018-01-31 11:59:01 +01:00
Igor Motov 19f39fd392
Persistent Tasks: remove task restart on failure (#815)
If a persistent task throws an exception, the persistent tasks framework will no longer try to restart the task. This is a temporary measure to prevent threshing the cluster with endless restart attempt. We will revisit this in the future version to make the restart process more robust. Please note, however, that if node executing the task goes down, the task will still be restarted on another node.
2018-01-31 11:59:01 +01:00
Igor Motov 9bd24418d5
Make PersistentAction independent from TransportActions (#742)
Removes the transport layer dependency from PersistentActions, makes PersistentActionRegistry immutable and rename actions into tasks in class and variable names.
2018-01-31 11:59:01 +01:00
Igor Motov 810d9335c0
Simplify names of PersistentTasks-related classes
PersistentTask -> NodePersistentTask
PersistentTasksInProgress -> PersistentTasks
PersistentTaskInProgress -> PersistentTask
2018-01-31 11:59:00 +01:00
Igor Motov b33fc05492
Request and Status in Persistent Tasks should be serialized using their writable names
Refactors xcontent serialization of Request and Status to use their writable names instead of action name. That simplifies the parsing logic, allows reuse of the same status object for multiple actions and is consistent with how named objects in xcontent are used.
2018-01-31 11:59:00 +01:00
Igor Motov 5eeb480d97
Add persistent task assignment explanations.
This commit allows persistent actions to indicate why a task was or wasn't assigned to a certain node.
2018-01-31 11:59:00 +01:00
Martijn van Groningen 479429c6ef
In order to keep track of restarted tasks, `allocationIdOnLastStatusUpdate` field was added to `PersistentTaskInProgress` class.
This will allow persistent task implementors to detect whether the executor node has changed or has been unset since the last status update has occured.
2018-01-31 11:58:07 +01:00
Igor Motov 16e661c34b
Make persistent task persist full cluster restart
This commit moves persistent tasks from ClusterState.Custom to MetaData.Custom and adds ability for the task to remain in the metadata after completion.
2018-01-31 11:58:07 +01:00
Martijn van Groningen 243b7e4499
Moved job lifecycle over to persistent tasks
Also replaced the DELETING status from JobState with a boolean flag on Job. The state of a job is now stored inside a persistent task in cluster state. Jobs that aren't running don't have a persistent task, so I moved that notion of being deleted to the job config itself.

Original commit: elastic/x-pack@21cd19ca1c
2018-01-31 11:58:07 +01:00
Igor Motov d340c190b2
Replace List with Map in PersistentTasksInProgress
Store currently running persistent tasks in a map instead of a list.

Original commit: elastic/x-pack@f88c9adef5
2018-01-31 11:58:06 +01:00
David Kyle 32e406181e
Fix check style error after upgrade
Original commit: elastic/x-pack@3bf4025f78
2018-01-31 11:58:06 +01:00
Igor Motov ac67d02bc3
Add support for task status on persistent tasks
Similarly to task status on normal tasks it's now possible to update task status on the persistent tasks. This should allow updating the state of the running tasks (such as loading, started, etc) as well as store intermediate state or progress.

Original commit: elastic/x-pack@048006b467
2018-01-31 11:58:06 +01:00
Martijn van Groningen 777b21f2ef
Add a number of auxiliary methods to persistent tasks classes.
Original commit: elastic/x-pack@7f44b41b7a
2018-01-31 11:57:02 +01:00
Vlad Holubiev eea9ee57dd [Docs] Fix typo in inner-hits.asciidoc (#27998) 2018-01-31 11:55:53 +01:00
Igor Motov f136bfa6e0
Adds support for persistent actions
A persistent action is a transport-like action that is using the cluster state instead of transport to start tasks. This allows persistent tasks to survive restart of executing nodes. A persistent action can be implemented by extending TransportPersistentAction. TransportPersistentAction will start the task by using PersistentActionService, which controls persistent tasks lifecycle.  See TestPersistentActionPlugin for an example implementing a persistent action.

Original commit: elastic/x-pack@5e83f1bfa3
2018-01-31 11:08:56 +01:00
Jim Ferenczi cb1fef7f6e
Fix intermittent failure in InternalEngineTest#testRefreshScopedSearcher (#28417)
This change switches the merge policy to none (for this specific test) in order to make sure that refreshes are always triggered
 by a change in the writer.

 Closes #27514
2018-01-31 09:24:15 +01:00
Jim Ferenczi 7edb978256
RandomDocumentPicks#randomFieldName can produce invalid field name (#28419)
This change makes sure that this function does not create field names that end with a '.', more precisely it only allows
alpha-numeric characters to compose the leaf field name.

Closes #27373
2018-01-31 09:21:09 +01:00
Christoph Büscher 202d28be86
[Test] Fix InstallPluginCommandTests failure on Windows (#28447)
The `testMetaPluginPolicyConfirmation` needs to close the file streams it is
iterating over, otherwise some OSes (like Windows) might not be able to delete
all temporary folders, which in turn leads to test failures.

Closes #28415
2018-01-31 09:20:51 +01:00
Ryan Ernst 139deb535a
Test: Use --illegal-access=warn for tests (#28437)
This commit conditionally adds the --illegal-access=warn flag when tests
are run with java 9. Currently, testing on java 9 triggers a warning
about illegal access from mockito. While that should be fixed (by
updating to a newer mockito base for securemock), the stderr warning we
get is only the first one. Thankfully that is the only one, but this
change will enable finding all such illegal accesses in the future.
2018-01-30 17:40:18 -08:00
Nik Everett 3b6af15a60
XContent: Factor deprecation handling into callback (#28449)
Factors the way in which XContent parsing handles deprecated fields
into a callback that is set at parser construction time. The goals here
are:
1. Remove Log4J as a dependency of XContent so that XContent can be used
by clients without forcing log4j and our particular deprecation handling
scheme.
2. Simplify handling of deprecated fields in tests. Now tests can listen
directly for the deprecation callback rather than digging through a
ThreadLocal.

More accurately, this change begins this work. It deprecates a number of
methods, pointing folks to the new versions of those methods that take
`DeprecationHandler`. The plan is to slowly drop these deprecated
methods. Once they are entirely removed we can remove Log4j as
dependency of XContent.
2018-01-30 18:21:10 -05:00
Nik Everett 9aeed74fe9 Reindex: Raise timeout on flaky test
This gives the test longer to block its updates. Now that we're checking
if the updates actually blocked saw that they may not do so in the
normal 10 seconds on a highly loaded system. And our jenkins machines
often function like highly loaded systems. Maybe this fixes #26758!
2018-01-30 17:59:58 -05:00
Nik Everett 6f64e9728b Reindex: More digging on flaky test
This adds more logging and a missing assertion to a flaky reindex test.
2018-01-30 17:42:40 -05:00
Simon Willnauer a2f3c274bf Remove left-over tribe reference 2018-01-30 21:44:21 +01:00
Christoph Büscher 1c296fe7ed Update bwc version for rank_eval rest tests 2018-01-30 21:02:19 +01:00
Simon Willnauer 3bf8554114
Remove tribe node support (#28443)
Tribe node has been superseeded by Cross-Cluster-Search. This change
removes the tribe node support entirely.
2018-01-30 20:40:19 +01:00
Alexander Reelsen 1d311dfb65 Versions: Add 6.1.4/5.6.8 snapshot versions 2018-01-30 20:13:34 +01:00
Christoph Büscher 6731c76900
Add ranking evaluation API to High Level Rest Client (#28357)
This change adds support for the new ranking evaluation API to the High Level Rest Client.
This mostly means adding support for parsing the various response objects back from the
REST representation. It includes one change to the response syntax where previously we didn't
print the type of the metric details section but we now need it to pick the right parser to
parse this section back.

Closes #28198
2018-01-30 17:48:09 +01:00
Jack Conradson fa8e62d48f
Removes Painless Type in favor of Java Class from the expression nodes related to function references and lambdas. (#28433) 2018-01-30 08:18:14 -08:00
Luca Cavanna b779fb392b
[TEST] add test for case where search source is null (#28398)
Closes #28393
2018-01-30 13:19:43 +01:00
Luca Cavanna 2c99bfc947
REST high-level client: Fix parsing of script fields (#28395)
Script fields can get a bit more complicated than just stored fields. A script can return null, an object and also an array. Extended parsing to support such valid values. Also renamed util method from `parseStoredFieldsValue` to `parseFieldsValue` given that it can parse stored fields but also script fields, anything that's returned as `fields`.

Closes #28380
2018-01-30 13:19:08 +01:00
Christoph Büscher f6a7ee91c9 Add @AwaitsFix for failing InstallPluginCommandTests
`testMetaPluginPolicyConfirmation` fails consistently on windows (#28415).
2018-01-30 11:59:05 +01:00
Boaz Leskes 613fc1654f testTranslogReplayWithFailure: checkStyle 2018-01-30 11:26:06 +01:00
Yannick Welsch 9dd0886265
Fix NullPointerException in MockUncasedHostProvider (#28424)
The MockUncasedHostProvider accesses nodes that are not fully built yet, where TransportService.getNode() returns null, which means that the null entries end up in the list of seedNodes that UnicastZenPing then uses.
2018-01-30 10:44:19 +01:00