Commit Graph

1484 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe 1e6ed023ee Fixes ReplicasActionTests
Fixes a test failure and adds a new test that was missing
2018-01-15 09:37:00 +00:00
Colin Goodheart-Smithe 53c3d040f7 Add ReplicaAction for Index Lifecycle (#3535)
* Add ReplicaAction for Index Lifecycle

* Add validation
2018-01-12 13:48:09 +00:00
Martijn van Groningen 18788531c8 Fixed test failure.
Relates to #3256
2018-01-12 14:04:26 +01:00
Colin Goodheart-Smithe 5eb912539d Fixes Logging usage
log line had an extra parameter
2018-01-12 12:17:14 +00:00
Colin Goodheart-Smithe 6a370251fc Improves handling of exceptions in Index Lifecycle (#3511)
* Improves handling of exceptions in Index Lifecycle

This change improves a few different aspects:

* If an exception occurs executing the lifecycle of one index it is caught, logged and other indexes are still processed
* If the lifecycle policy specified in the settings does not exist an error is logged
* Fixes the exception when the delete action is run which occurs because Phase attempts to update the phase and action settings for the deleted index. A `LifecycleAction.indexSurvives()` method is introduced which defaults to `true` but can be overridden to indicate whether the index survives following completion of the action.

* Adds test
2018-01-12 11:52:56 +00:00
Tal Levy e3a6e738ea Fix InternalIndexLifecycleContext to update state in memory (#3505)
* Fix InternalIndexLifecycleContext to update state in memory

The internal and the mock index-lifecycle-context implementations differed
in that the InternalIndexLifecycleContext assumed no one would be using it after
it mutated state. This is not the case. We assume that the current context is updated after
a `#setAction` is called so that the listener can then appropriately use the newly modified
cluster state. since idxMeta was not being updated, any call to `context.getAction` was stale and
either returning null or the previous action, not the next action that was updated by `#setAction`.
Same goes for `setPhase`.

This PR should fix this so that the Mock and Internal implementations are more in line.
2018-01-10 08:21:17 -08:00
Martijn van Groningen a56b0479e1 Changed the chunk coordinator to process the chunks concurrently. (#3256)
The shard follow task executor determines the range of translog operations
between the leader shard's global checkpoint and the last know processed
seqno by the current shard follow task that are missing.

Then the chunks coordinator can then chunk this range up in smaller ranges
if the requested range is above the configured max chunk size. If it is
smaller than the entire range then the chunk coordinator has just one
chuck to coordinate.

Each chunk is added to a queue and is processed by the ChunkProcessor,
that reads the translog ops from the leader shard and then indexes
these translog ops into the follow shard. After that a new chuck is polled
from the queue and the ChunkProcessor performs the same actions until
there are no more chunks in the queue to process. After that the shard
follow task executor will determine a new range of translog operations
to process.

This change changes the chunk coordinator to start polling from the chunk
queue with multiple threads at the same time to handle dealing with a higher
indexing load on the leader side better.
2018-01-10 15:29:11 +01:00
Colin Goodheart-Smithe 57c343bc21 Use the XContentRegistry provided by the parser 2018-01-08 14:14:43 +00:00
Colin Goodheart-Smithe c4621d9dc7 Mutes IndexLifecycleInitialisationIT.testMasterFailover
Due to a bug with unassigned shards in force merge action
2018-01-05 12:43:13 +00:00
Martijn van Groningen 38aea9588b Fixed compile errors after merge. 2018-01-04 18:18:46 +01:00
Colin Goodheart-Smithe 148688a991 Reduces poll time to 1s for integ test
`IndexLifecycleInitialisationIT.testMasterFailover()` intermittently failed because the timeout of 10 seconds to check if the index had been deleted was not long enough sometimes with the poll interval set to 3 seconds. This change sets the poll interval to 1 seconds for the test so that the lifecycle is more responsive. This also means the default value for the poll interval can be safely changed without affecting the test.
2018-01-03 12:29:19 +00:00
Colin Goodheart-Smithe 52e416d0bc Creates index-lifecycle gradle module
This doesn’t currently contain any code but is there in preparation for the x-pack-elasticsearch:plugin split
2018-01-03 12:03:54 +00:00
Colin Goodheart-Smithe 043c8db84d Fixes Tests so build passes 2018-01-03 12:01:12 +00:00
Colin Goodheart-Smithe a37df9092e Moves Actions and related to plugin:core
This change moves the Action classes and referenced data model classes to the new :x-pack-elasticsearch:plugin:core project in preparation for splitting the x-pack features into their own gradle modules.

Note that the TransportAction classes had to be promoted to their own class file (rather than being inner classes to their Action) so they can remain in the plugin project (and will late be move to the `index-lifecycle` project when its created.
2018-01-03 10:45:33 +00:00
Colin Goodheart-Smithe 33afe0f012 Merge branch 'master' into index-lifecycle 2018-01-03 10:21:50 +00:00
Tal Levy 6a47b4fa6e fix infinite loop in test and correct logic in timeseries type 2017-12-21 13:14:37 -08:00
Tal Levy 66af56320e fix tests 2017-12-21 12:56:56 -08:00
Tal Levy e433b1eaa8 Adds ForceMerge action to Index Lifecycle (#3309)
add forcemerge index-lifecycle action
2017-12-21 11:59:44 -08:00
Martijn van Groningen dbdefc5689 Fix issue after merging 2017-12-21 14:53:12 +01:00
Colin Goodheart-Smithe d5609a408b fixes integ test when running as part of the build 2017-12-21 12:38:34 +00:00
Colin Goodheart-Smithe 3627feded7 Refactors lifecycle type to its own class (#3381)
To clean up the parsing of the LifecyclePolicy this change moves the LifecycleType to its own class so it can be created in the normal parsing of LifecyclePolicy rather than having to parse to an intermediary object first. The LifecycleType is an interface which can be implemented for different lifecycle types. These types shiould be singletons and are register with the NamedXContentRegistry and NamedWriteableRegistry only so they are available when reading from a stream or parsing.
2017-12-21 11:20:49 +00:00
Tal Levy fb59f54637 make the lifecycle poll-interval configurable (#3319)
Removes the poll-interval from the IndexLifecycleMetadata and introduces it in 
the form of a cluster setting that is configurable. Changes to this poll interval setting 
will reflect in the Lifecycle Scheduler.
2017-12-20 09:39:39 -08:00
Colin Goodheart-Smithe 586065fca7 Changes to enable the build to run
This includes changing NOCOMMIT comments to be NORELEASE comments so the build passes with them. We have tasks inGH for all these NORELEASE comments so they should be caught before merging to master
2017-12-20 12:27:26 +00:00
Martijn van Groningen 973f126c24 Merge branch 'master' into ccr 2017-12-18 16:52:09 +01:00
Colin Goodheart-Smithe b12f1a9526 More phase tests
This time for when setting the new action fails
2017-12-15 11:20:59 +00:00
Colin Goodheart-Smithe 33a1a92e3d As phase test for when the action execution fails 2017-12-15 11:07:49 +00:00
Colin Goodheart-Smithe 9e693b544c Test improvements 2017-12-15 10:52:37 +00:00
Colin Goodheart-Smithe 3da42f5603 Adds rollover action to index lifecycle plugin (#3266)
This action will rollover an index when executed if the provided conditions are met.
Users may specify the maximum age, maximum index size in bytes or maximum index size in number of documents as conditions for rollover.

When the action executes it firsts checks the local cluster state to find out if the alias exists on the index. If the alias does not exist then the index was either rolled over by a previous run or something else has rolled over the index so the action can be marked as completed. If the index still has the alias set the action will make a rollover index request using the Client. When that request returns and the listener is called the action will only be marked as complete if the response indicates the index was rolled over. If the index was not rolled over (because the conditions are not yet met) the action is not marked as complete and will be re-evaluated on the next call to execute.
2017-12-15 09:36:39 +00:00
Tal Levy 8d6e53356b enable rest of await-fix index-lifecycle integ tests. all green 2017-12-13 13:40:58 -08:00
Tal Levy c0c7e02de4 fix javadoc usage 2017-12-11 16:24:44 -08:00
Tal Levy 288515ddc5 initial attempt towards an ordered execution plan for timeseries phases (#3248)
* initial attempt towards an ordered execution plan for timeseries phases

* respond to feedback and fix bug
2017-12-07 11:59:31 -08:00
Martijn van Groningen 6c06aacb57 Fixed NumberFormatException in the rest layer by only setting batch_size if it has been specified. 2017-12-07 09:34:25 +01:00
Tal Levy ae7e4882e4 general indexlifecycle cleanup, rest tests, reintroduce integ-test (#3243)
* cleanup, rest tests, reintroduce integ-test

- cleaned up some long lines
- re-introduced one of the integration tests
- added rest-spec tests
2017-12-06 07:51:06 -08:00
Tal Levy d847355b58 add timeseries policy action stubs (#3214) 2017-12-05 09:09:18 -08:00
Martijn van Groningen 0dc5a89bc5 fixed compile error 2017-12-01 14:14:17 +01:00
Martijn van Groningen cc5665bce6 pass down batch size correctly 2017-12-01 14:14:17 +01:00
Martijn van Groningen 3cd5eed91e fixed code violations 2017-12-01 14:14:17 +01:00
Martijn van Groningen c3f7d4f580 * Made ChunkCoordinator unittest testable
* Fixed a small issue where each batch would fetch / index the previous batch last operation
* Made batch size a request param on the follow existing index api request.
  This makes is easy to tune this param when running tests from scripts.
* Changed default batch size from 256 to 1024.
2017-12-01 14:14:17 +01:00
Martijn van Groningen 2e382bf7f3 Disallow dynamic mapping updated in follow shards
I forgot to configure a mapping in the follow shard shard, which caused
a dynamic update (due to type auto creation), but this was ignored.

Subsequent searches in follow index then failed due to a mapping missing.
(The _id couldn't be fetched during fetch phase, because the mapping was missing)

We should at a later stage investigate how to best solve this, but for
know to avoid confusion just fail if a dynamic update happens in a
follow shard.
2017-12-01 09:33:06 +01:00
Tal Levy 685231e786 Introduce specific types of index lifecycle policies (#3157)
Introduce specific types of index lifecycle policies
2017-11-30 10:09:41 -08:00
Jason Tedor be5f83a6bd Implement translog operation bulk action
This commit adds a bulk action for apply translog operations in bulk to
an index. This action is then used in the persistent task for CCR to
apply shard changes from a leader shard.

Relates #3147
2017-11-29 07:29:56 -05:00
Jason Tedor 11aa83011c Fix get operations between test
This test was broken by an upstream change that no longer guarantees we
see the operations from the upstream translog in the order they appear
in that translog. As such, the assertions in this test were too strong
so this commit relaxes them.

Relates #3153
2017-11-29 07:16:52 -05:00
Jason Tedor abd263fca9 Merge branch 'master' into ccr
* master:
  [DOCS] Fixed cross cluster search docs issue (#3113)
  Watcher: Add transform input for chained input (#2861)
  [DOCS] Fix cleaning up ML resources after tests
  [ML] Specify ML_ORIGIN when calling the job update endpoint internally (#3110)
  Refactor CLI commands as logging-aware commands
  [ML] Stop datafeed when job fails (#3107)
2017-11-27 14:03:10 -05:00
Colin Goodheart-Smithe 922060d766 Typo 2017-11-27 15:46:47 +00:00
Colin Goodheart-Smithe 7199e2fa5c Small refactoring 2017-11-27 12:32:55 +00:00
Colin Goodheart-Smithe 87766cd4ef Adds JavaDocs and comments and small refactoring 2017-11-27 12:22:10 +00:00
Colin Goodheart-Smithe 9ab2121fcf When actions complete the next action is now run 2017-11-27 11:07:26 +00:00
Colin Goodheart-Smithe 152d5d45e5 Adds listeners to Lifecycle actions
This means that the result of the action can now be async and we can then implement moving immediately to the next action if the current one is complete
2017-11-24 19:27:50 +00:00
Colin Goodheart-Smithe 26ad2142d5 minor tweak 2017-11-24 17:29:38 +00:00
Colin Goodheart-Smithe 9e891de777 refactors code to allow better testing
The client and index metadata have now been abstracted away from the Lifecycle classes behind IndexLifecycleContext. This allow us to test the state machine without having to worry about how the state is persisted and read. It also makes the classes much easier to read and reason about.
2017-11-24 17:22:51 +00:00
Colin Goodheart-Smithe e113189347 only creates initialisation service ifeature is enabled 2017-11-24 10:23:04 +00:00
Colin Goodheart-Smithe 06958949ab Uses LongSupplier for nowSupplier
Rather than Supplier<Long>. This is in line with the `nowSupplier`s used in other areas of Elasticsearch
2017-11-24 10:18:06 +00:00
Colin Goodheart-Smithe 31ded906d0 Adds settings checking to tests
This isn’t an ideal solution so might well be temporary but at least the tests are checking the settings changes are correct now
2017-11-24 10:10:46 +00:00
Colin Goodheart-Smithe 016ba2a57d Adds another test
This test needs more work
2017-11-24 09:40:32 +00:00
Colin Goodheart-Smithe 49b2aaa878 Adds more tests 2017-11-23 18:16:47 +00:00
Colin Goodheart-Smithe 75704510e1 More test changes 2017-11-23 15:48:37 +00:00
Colin Goodheart-Smithe a9703d4a56 Mutes failing integration test for now 2017-11-23 14:12:31 +00:00
Colin Goodheart-Smithe 0e829843fc More unit tests 2017-11-23 14:11:09 +00:00
Colin Goodheart-Smithe db502bef6d Adds some unit tests 2017-11-23 12:44:12 +00:00
Colin Goodheart-Smithe f571dc3000 Remove references to InternalClient which no longer exists 2017-11-23 10:35:25 +00:00
Colin Goodheart-Smithe a3adcb680d Trying to get the test working 2017-11-23 10:00:23 +00:00
Colin Goodheart-Smithe 9cba84b6fc Obeys the after parameter on phases excpet the first 2017-11-22 16:09:04 +00:00
Colin Goodheart-Smithe 3f0cf05dcc Started writing/reading phase and action settings 2017-11-22 13:26:05 +00:00
Colin Goodheart-Smithe a26a5969dd iterates over index and executes correct policy 2017-11-22 10:30:21 +00:00
Martijn van Groningen e6ad1c3e9d ccr: Added apis and persistent tasks for following an index and log changes instead of indexing. 2017-11-22 09:04:59 +01:00
Colin Goodheart-Smithe eba5afefb7 Gets the scheduler to use poll_interval 2017-11-21 16:54:28 +00:00
Colin Goodheart-Smithe 3f3f7b9b47 Adds PUT/GET/DELETE lifecycle APIs
The lifecycles are stored as custom metadata objects in the cluster state. This change also cleans up the parsing of the lifecycle state so that it can be parsed properly
2017-11-21 15:55:45 +00:00
Tal Levy 1fad59b295 tal-temp migration to not use settings for the lifecycle policy 2017-11-17 09:22:54 -07:00
Jason Tedor 8e0b34b507 Override indexing plans
Operations from a leader shard will be indexed into the engine with the
origin set to primary. The problem is here is that then we have primary
semantics in the engine such as assertions about sequence numbers being
unassigned, and we do not have correct semantics for out-of-order
delivery of operations (as we should on a following engine, whether or
not it is primary since the ordering is determined from the
leader). This commit handles this by always using the replica plan for
indexing into a following engine, whether or not the engine is for a
primary shard.

Relates #3000
2017-11-15 13:38:46 -05:00
Jason Tedor 913936f2b0 Maintain order of operations semantics on follower
A following engine even for a primary shard needs to maintain order of
operations semantics as if it were behaving like a replica. That is,
rather than assuming that the order of operations presented to the
engine is the de facto order of operations as is the case for a leader
engine for a primary shard, a following engine must behave like all
replicas behave which is that they resolve order of operations based on
sequence numbers. This commit causes this to be the case for following
engines.

Relates #2931
2017-11-09 15:18:54 -05:00
Martijn van Groningen bf4e18fdfc fixed compile error 2017-11-02 15:12:35 +01:00
Tal Levy dd6d192e25 cleanup tests 2017-10-31 17:39:25 -07:00
Colin Goodheart-Smithe c393944dc0 Adds settings, phase and action objects which control how and when things are executed 2017-10-31 15:30:42 +00:00
Tal Levy 3455445d2c add integration test to indexlifecyclemanagement
This test verifies that we have sufficient failover code so that
a newly elected master re-registers old schedules and fires them off.
All times are relative to the index creation date.
2017-10-30 17:51:01 -07:00
Jason Tedor 769349a9ab Add following engine implementation
This commit is a first step towards a following engine
implementation. Future work will build on this by using this engine to
execute operations on a following engine from another engine (typically
a remote leader engine) that has already assigned sequence numbers to
such operations.

Relates #2776
2017-10-30 13:38:02 -04:00
Tal Levy de4d87c6e6 test 2017-10-26 15:32:08 -07:00
Martijn van Groningen 41c3dc91c1 Changed the shard changes api to use get threadpool instead of management threadpool. 2017-10-17 10:02:28 +02:00
Martijn van Groningen 2663b5ef39 remove unused import 2017-10-16 10:36:22 +02:00
Jason Tedor 85c1935d9d Increase generations in operations between test
This commit sets an index setting for the size of a translog generation
and increases the number of documents indexed to increase the chance of
multiple generations being present when testing getting operations
between two sequence numbers.
2017-10-16 08:59:18 +02:00
Jason Tedor bcd61bfca5 Fix off-by-one error in shard changes action test
This commit fixes an off-by-one error in the shard changes action test
for getting operations between two sequence numbers. The off-by-one
error arises because sequence numbers are indexed from zero, so if N
documents are indexed then the maximum sequence number starting from
zero would be N - 1.
2017-10-16 08:54:04 +02:00
Martijn van Groningen 1f495f59a1 [CCR] Add an internal api to read translog operations between a sequence number range (#2563)
* xdcr: Add an internal api to read translog operations between a sequence number range.

This api will be used later by the persistent task for the following index to pull data from the leader index.

The persistent task can fetch the global checkpoint from the shard stats for each primary shard of the leader index.
Based on the global checkpoint of the primary shards of the following index, the persistent task can send several
calls to the internal api added in this commit to replicate changes from follow index to leader index in a batched manner.
2017-10-16 07:38:03 +02:00
Colin Goodheart-Smithe 4aaec2ef81 Creates a new X-Pack feature for index lifecycle
Feature consists of a shell of a persistant task which will later be used to inspect the index settings and apply curator like changes to the index (move from hot to warm, rollover, shrink etc.)
2017-10-12 16:22:42 +01:00
Jason Tedor 3033aba67b Introduce pluggable engine factory for CCR
This commit utilizes the pluggable engine factory feature in core to
introduce a pluggable engine factory for XDCR. For now this is only a
skeleton implementation to proof out the pluggable engine factory
concept. Future work will implement a genuine following engine for XDCR.

Relates #2655
2017-10-12 10:52:33 -04:00
Jason Tedor 063a76c3db Introduce CCR container class
This commit introduces the container class for CCR functionality. Future
work will expose more specific CCR functionality to the X-Pack plugin
through this class.

Relates #2704
2017-10-06 14:46:39 -04:00