Commit Graph

146 Commits

Author SHA1 Message Date
Luca Cavanna d860971572
REST high-level client: add support for split and shrink index API (#28425)
Relates to #27205
2018-02-01 16:37:01 +01:00
Martijn van Groningen 61806802fb
Add persistent tasks
Persistent tasks are build on top of node tasks and provide functionality to restart a task to run on a different coordination node in case the coordinating node is no longer available.
It is up to a persistent task implementation to keep track of status, so that in case the task is restarted, the task can continue were it left off before it was restarted.
2018-02-01 15:26:17 +01:00
Alan Woodward 9500513ba1
Move leftover aliases test from core/ to server/ (#28463) 2018-02-01 11:40:29 +00:00
Colin Goodheart-Smithe 65157e9428
[TEST] Replaces flaky breaker IT test with unit test (#28418)
This change remove the `CircuitBreakerIT. testParentChecking` test method which fails intermittently in unexpected ways with a `MemoryCircuitBreakerTests. testBorrowingSiblingBreakerMemory` unit test method which can test the borrowing functionality more directly

Closes #28223
2018-02-01 08:32:46 +00:00
Jim Ferenczi dd40b984c4
Add a shallow copy method to aggregation builders (#28430)
This change adds a shallow copy method for aggregation builders. This method returns a copy of the builder replacing the factoriesBuilder and metaDada
This method is used when the builder is rewritten (AggregationBuilder#rewrite) in order to make sure that we create a new instance of the parent builder when sub aggregations are rewritten.

Relates #27782
2018-02-01 09:22:32 +01:00
Jim Ferenczi c7d5a54b42
Fix AIOOB on indexed geo_shape query (#28458)
This change fixes a possible AIOOB during the parsing of the document that contains the indexed shape.
This change ensures that the parsing does not continue when the field that contains the shape has been found.

Closes #28456
2018-02-01 09:01:48 +01:00
Jason Tedor 1b3d529bef Introduce secure security manager to project
This commit migrates SecureSM, our secure security manager
implementation, from its own repository to being a sub-project of
Elasticsearch.
2018-01-31 18:23:28 -05:00
Nhat Nguyen 5e0be61774
Add logging to index commit deletion policy (#28448)
This would help us to figure out which index commit that an engine 
started with or used in peer-recovery.

Relates #28405
2018-01-31 11:09:49 -05:00
markharwood 77d2dd203e
Search - add allow_partial_search_results flag with default setting false (#28440)
Adds allow_partial_search_results flag to search requests with default setting = true.
When false, will error if search either timeouts, has partial errors or has missing shards rather
than returning partial search results. A cluster-level setting provides a default for search requests with no flag.

Closes #27435
2018-01-31 15:51:29 +00:00
David Turner 4c154b70d3
Fix rounding of time values near to overlapping days (#28151)
Sometimes, in some places, the clocks are set back across midnight, leading to
overlapping days. This was not handled as expected, and this change fixes this.

Additionally, in this situation it is not true that rounding a time down to the
nearest day is a monotonic operation, as asserted in these tests. This change 
suppresses those assertions in those rare cases.

Fixes #27966.
2018-01-31 15:10:47 +00:00
kel 5819e57baa Replace Bits with new abstract class to respresent documents that have a value (#24088) (#28334) 2018-01-31 15:42:11 +01:00
Martijn van Groningen 592eedbf49
Make persistent tasks work.
Made persistent tasks executors pluggable.
2018-01-31 12:28:06 +01:00
Martijn van Groningen 07e727c769
Removed ClientHelper dependency from PersistentTasksService. 2018-01-31 12:28:06 +01:00
David Roberts cc16f9d9c9
Added AllocatedPersistentTask#waitForPersistentTaskStatus(...) that delegates to PersistentTasksService#waitForPersistentTaskStatus(...)
This allows persistent tasks executor implementations to not have an instance of PersistentTasksService.
2018-01-31 12:28:06 +01:00
Igor Motov 41071e4711
Add adding ability to associate an ID with tasks.
Persistent tasks portion of elastic/elasticsearch#23250
2018-01-31 12:28:06 +01:00
Jay Modi 8521b2d11e
Remove InternalClient and InternalSecurityClient (#3054)
This change removes the InternalClient and the InternalSecurityClient. These are replaced with
usage of the ThreadContext and a transient value, `action.origin`, to indicate which component the
request came from. The security code has been updated to look for this value and ensure the
request is executed as the proper user. This work comes from #2808 where @s1monw suggested
that we do this.

While working on this, I came across index template registries and rather than updating them to use
the new method, I replaced the ML one with the template upgrade framework so that we could
remove this template registry. The watcher template registry is still needed as the template must be
updated for rolling upgrades to work (see #2950).
2018-01-31 12:28:05 +01:00
Martijn van Groningen 4dd69951f3
Make the persistent task status available to PersistentTasksExecutor.nodeOperation(...) method 2018-01-31 12:28:05 +01:00
Colin Goodheart-Smithe 1c489ee867
Refactor/to x content fragments2 (#2329)
* Moves more classes over to ToXContentObject/Fragment

* Removes ToXContentToBytes

* Removes ToXContent from Enums

* review comment fix

* slight change to use XContantHelper
2018-01-31 12:28:05 +01:00
David Roberts 7313ad5b29
Make AllocatedPersistentTask members volatile (#2297)
These members are default initialized on contruction and then set by the
init() method.  It's possible that another thread accessing the object
after init() is called could still see the null/0 values, depending on how
the compiler optimizes the code.
2018-01-31 12:28:05 +01:00
Colin Goodheart-Smithe b0de3c38d6
Moves more classes over to ToXContentObject/Fragment (#2283) 2018-01-31 12:28:04 +01:00
Luca Cavanna 65ce2276eb
Adapt to upstream changes made to AbstractStreamableXContentTestCase (#2117) 2018-01-31 12:28:04 +01:00
Yannick Welsch b5f281386a
Move tribe to a module (#2088)
Companion PR to elastic/elasticsearch#25778
2018-01-31 12:28:04 +01:00
Igor Motov ffdb05e48e
Persistent Tasks: remove unused isCurrentStatus method (#2076)
Removes a method that is no longer used in production code.

Relates to #957
2018-01-31 12:28:04 +01:00
David Kyle 0d50f9c6a9
Call initialising constructor of BaseTasksRequest (#1771) 2018-01-31 12:28:04 +01:00
Chris Earle 1cef531165
Always Accumulate Transport Exceptions (#1619)
This is the x-pack side of the removal of `accumulateExceptions()` for both `TransportNodesAction` and `TransportTasksAction`.

There are occasional, random failures that occur during API calls that are silently ignored from the caller's perspective, which also leads to weird API responses that have no response and also no errors, which is obviously untrue.
2018-01-31 12:28:03 +01:00
Hendrik Muhs 614aef2527
Pass down the provided timeout. 2018-01-31 12:28:03 +01:00
Simon Willnauer 292e383d2c
Fix static / version based BWC tests (#1456)
With the leniency in Version.java we missed to really setup BWC
testing for static indices. This change brings back the testing and adds
missing bwc indices.

Relates to elastic/elasticsearch#24732
2018-01-31 12:28:03 +01:00
Yannick Welsch e69317b24b
Don't call ClusterService.state() in a ClusterStateUpdateTask
The current state is readily available as a parameter
2018-01-31 12:28:02 +01:00
Yannick Welsch 44ea5d6b3e
Separate publishing from applying cluster states
Companion commit to elastic/elasticsearch#24236
2018-01-31 12:28:02 +01:00
Igor Motov a08e2d9e5e
Persistent tasks: require allocation id on task completion (#1107)
Persistent tasks should verify that completion notification is done for correct version of the task, otherwise a delayed notification from an old node can accidentally close a newly reassigned task.
2018-01-31 12:28:01 +01:00
Colin Goodheart-Smithe 76cd7b1eb2
Fixes compile errors in Eclipse due to generics
PersistentTasksCustomMetadata was using a generic param named `Params`. This conflicted with the imported interface `ToXContent.Params`. The java compiler was preferring the generic param over the interface so everything was fine but Eclipse apparently prefers the interface int his case which was screwing up the Hierarchy and causing compile errors in Eclipse. This changes fixes it by renaming the Generic param to `P`
2018-01-31 12:27:34 +01:00
Igor Motov fc524bc9b5
Persistent Tasks: force writeable name of params and status to be the same as their task (#1072)
Changes persistent task serialization and forces params and status to have the same writeable name as the task itself.
2018-01-31 12:27:34 +01:00
Martijn van Groningen 4771965931
Use task builder instead of creating persistent tasks directly. 2018-01-31 12:27:34 +01:00
Igor Motov abd9ae399c
Persistent Tasks: PersistentTaskRequest -> PersistTaskParams (#1057)
Removes the last pieces of ActionRequest from PersistentTaskRequest and renames it into PersistTaskParams, which is now just an interface that extends NamedWriteable and ToXContent.
2018-01-31 12:27:33 +01:00
Igor Motov 6bfea09dd6
Persistent Tasks: switch from long task ids to string task ids (#1035)
This commit switches from long persistent task ids to caller-supplied string persistent task ids.
2018-01-31 12:27:33 +01:00
Hendrik Muhs 0a1f25588b
Added PersistentTasksService#waitForPersistentTasksStatus(...) method to allow callers to wait when an executor node has updated its task status. 2018-01-31 12:27:33 +01:00
Igor Motov 0a1abd430d
Persistent Tasks: remove listener from PersistentTasksExecutor#nodeOperation (#1032)
Instead of having a separate listener for indicating that the current task is finished, this commit is switching to use allocated object itself.
2018-01-31 12:27:32 +01:00
Igor Motov 95c6005f6f
Persistent Tasks: remove retries on notification failures (#977)
Retries should be already handled by TransportMasterNodeAction, there is no need to introduce another retry layer in Persistent Tasks code.
2018-01-31 12:27:32 +01:00
Martijn van Groningen fab0dc449a
Remove PersistentTask#isCurrentStatus() usages 2018-01-31 12:27:32 +01:00
Igor Motov 5a8512bf4e
Persistent Tasks: refactor PersistentTasksService to use ActionListener (#937)
PersistentTasksService methods are not using ActionListener<PersistentTask<?>> instead of PersistentTaskOperationListener.
2018-01-31 12:27:29 +01:00
Jason Tedor 97822dbea3
Respond to rename random ASCII helper methods
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.
2018-01-31 12:00:10 +01:00
Igor Motov 5b45b167bd
Persistent Tasks: check the current state in waitForPersistentTaskStatus (#935)
Add a check for the current state waitForPersistentTaskStatus before waiting for the next one. This fixes sporadic failure in testPersistentActionStatusUpdate test.

Fixes #928
2018-01-31 12:00:09 +01:00
Martijn van Groningen a5acb556b0
Use PersistentTasksService#waitForPersistentTaskStatus(...) to wait for job and datafeed status and use PersistentTasksService#removeTask(...) to force close job and force stop datafeed. 2018-01-31 12:00:09 +01:00
Igor Motov 1b0f5b9572
Persistent Tasks: require correct allocation id for status updates (#923)
In order to prevent tasks state updates by stale executors, this commit adds a check for correct allocation id during status update operation.
2018-01-31 12:00:09 +01:00
Igor Motov 6ca044736e
Persistent Tasks: Add waitForPersistentTaskStatus method (#901)
This method allows to wait for tasks to change their status to match the supplied predicate.
2018-01-31 12:00:09 +01:00
Martijn van Groningen 78b844e79b
Check allocationIdOnLastStatusUpdate when trying to detect whether a task is stale. 2018-01-31 11:59:02 +01:00
Igor Motov b142d7e29c
Persistent Tasks: Remove unused stopped and removeOnCompletion flags (#853)
The stopped and removeOnCompletion flags are not currently used, this commit removes them for now to simplify things.
2018-01-31 11:59:01 +01:00
Igor Motov 37fad04879
Persistent Tasks: Merge NodePersistentTask and RunningPersistentTask (#842)
Refactors NodePersistentTask and RunningPersistentTask into a single AllocatedPersistentTask. Makes it possible to update Persistent Task Status via AllocatedPersistentTask.
2018-01-31 11:59:01 +01:00
Igor Motov 19f39fd392
Persistent Tasks: remove task restart on failure (#815)
If a persistent task throws an exception, the persistent tasks framework will no longer try to restart the task. This is a temporary measure to prevent threshing the cluster with endless restart attempt. We will revisit this in the future version to make the restart process more robust. Please note, however, that if node executing the task goes down, the task will still be restarted on another node.
2018-01-31 11:59:01 +01:00
Igor Motov 9bd24418d5
Make PersistentAction independent from TransportActions (#742)
Removes the transport layer dependency from PersistentActions, makes PersistentActionRegistry immutable and rename actions into tasks in class and variable names.
2018-01-31 11:59:01 +01:00