Commit Graph

132 Commits

Author SHA1 Message Date
Tal Levy d63cd8c9c3 step by step 2018-03-26 08:00:03 -07:00
Tal Levy fe22c08207 stop using illegal logging of Object 2018-03-12 13:18:02 -07:00
Tal Levy ebff02c1b2 fix compilation changes 2018-03-12 12:29:22 -07:00
Tal Levy 9850f44b95 fix state left over by spec test 2018-02-13 13:40:09 -08:00
Tal Levy 8e282f21c8 Merge branch 'master' into index-lifecycle 2018-02-12 14:17:24 -08:00
Tal Levy 0a3dcebca0 disallow deleting lifecycle policies that are referenced by existing indices (#3806) 2018-02-01 14:16:06 -08:00
Tal Levy ab49e40f91 add phase and action timestamps to cluster-state (#3726)
This commit sets `index.lifecycle.phase_time` and `index.lifecycle.action_time`
appropriately during setPhase and setAction in IndexLifecycleContext.
2018-01-25 09:02:26 -08:00
Colin Goodheart-Smithe faf067080b Makes sure creation date garentees execution in tests 2018-01-25 13:21:50 +00:00
Tal Levy 9920d6be2d keep plugin up to date with structure changes (#3702)
Looks like we need to split out the tests of core classes to core
and index-lifecycle ones stay in index-lifecycle.

I believe I got everything, although I may have missed at least one thing

checked status with

$ ./gradlew :x-pack-elasticsearch:plugin:index-lifecycle:check -Dtests.seed=39838421912001B4
$ ./gradlew :x-pack-elasticsearch:plugin:core:check -Dtests.seed=39838421912001B4

other things done in this PR:
- removal of a few unused variables/thrown exceptions/imports
- fix TimeseriesLifecycleTypeTests
  - an all null AllocateAction was created
- fix AllocateActionTests
  - woops. -Dtests.seed=39838421912001B4 resulted in two `null`s and an emptyMap.
    this resulted in a test failure.
2018-01-24 08:47:21 +00:00
Tal Levy 4856baf66b fix guice and plugin initialization after split (#3677)
the x-pack-split required updating the IndexLifecycle plugin to
be a proper plugin with the Plugin API methods instead of the XPackPlugin API
methods.
2018-01-23 10:41:22 -08:00
Colin Goodheart-Smithe fd502aa3e6 Makes some fields in Index Lifecycle API optional (#3687)
Specifically this change makes it optional to:
* Specify `includes`, `excludes` and `requires`maps in the allocate action as long as at least one fo the options is specified and is not an empty map
* Specify an `after` parameter on a phase. If no `after` value is specified `TimeValue.ZERO` is used and the phase will be moved to as soon as the previous phase reports `ACTIONS COMPLETED`. `after` is always non-null when we are serialising the Phase.
* Specify a `type` for a LifecyclePolicy. If no `type` is specified `TimeSeriesLifecycleType.INSTANCE` is used since this is currently the only production `type`. `type` is always non-null when we are serialising the LifecyclePolicy.
2018-01-23 17:02:11 +00:00
Colin Goodheart-Smithe f2fa988f2f Fixes Checkstyle 2018-01-23 08:24:03 +00:00
Tal Levy 07b28fd721 migrate lifecycle settings to plugin:core:LifecycleSettings (#3670) 2018-01-22 10:43:16 -08:00
Tal Levy 34d6b61209 add `index.lifecycle.date` setting (#3545)
This PR adds a new setting called `index.lifecycle.date` that 
the ShrinkAction will be responsible for populating in the newly created index.

This way, we can continue to know when we should be executing the next phase
relative to the original index creation date, and not that of the shrunken index.
2018-01-22 09:27:33 -08:00
Colin Goodheart-Smithe 9032d61516 Fixes silly issues created by me 2018-01-22 16:48:32 +00:00
Colin Goodheart-Smithe 2d721136c2 Adds logic to cancel job if master is un-elected (#3613)
A node can stop being the master node whilst it is running, e.g. if it can’t access `minimumMasterNodes` number of master eligible nodes. Because of this we need logic in `IndexLifecycleService` that cancels the scheduled job if the node is no longer master and re-adds the job if the node becomes master again.
2018-01-22 15:45:37 +00:00
Colin Goodheart-Smithe ee774594e2 Adds test for TimeValueSchedule 2018-01-22 15:21:55 +00:00
Colin Goodheart-Smithe a6f6335e86 Moves Index-Lifecycle classes over to new module
These changes are necessary following the x-pack split.
2018-01-22 13:52:22 +00:00
Colin Goodheart-Smithe 3cdf1216cd Removes un-needed licence files 2018-01-19 10:27:57 +00:00
Colin Goodheart-Smithe cb4cbf80ac Catches more that just ElasticsearchException 2018-01-18 05:42:22 +00:00
Tal Levy 84fd234835 Add the Shrink Action to Index Lifecycle Management (#3491)
This does the following in sequential service polls

1. sets the index to read-only and runs shrink with a modified `index.lifecycle.name` setting set to `null`.
2. checks to see if shrink is complete, if it is...
    b. set target index's `index.lifecycle.*` settings to the original index's values.
3. if not complete, just wait till next iteration
4. if operating on shrunken index, delete old index and add it as an alias to shrunken index
2018-01-17 11:22:06 -08:00
Colin Goodheart-Smithe ff883d574b Adds Allocate lifecycle action (#3484)
* Adds Allocate lifcycle action

* Addresses review comments

Still need to make a change in core for the FilterAllocationDecider to make the execute logic simpler

* Addresses more review comments

* Adds randomMap method to AllocateActionTests

* Addresses further review comments
2018-01-17 13:27:40 +00:00
Tal Levy db3a88f487 fix TimeseriesLifecycle ordering behavior 2018-01-16 18:08:05 -08:00
Colin Goodheart-Smithe 971685654d Adds more tests in IndexLifecycleServiceTests 2018-01-15 15:29:07 +00:00
Colin Goodheart-Smithe 1e6ed023ee Fixes ReplicasActionTests
Fixes a test failure and adds a new test that was missing
2018-01-15 09:37:00 +00:00
Colin Goodheart-Smithe 53c3d040f7 Add ReplicaAction for Index Lifecycle (#3535)
* Add ReplicaAction for Index Lifecycle

* Add validation
2018-01-12 13:48:09 +00:00
Colin Goodheart-Smithe 5eb912539d Fixes Logging usage
log line had an extra parameter
2018-01-12 12:17:14 +00:00
Colin Goodheart-Smithe 6a370251fc Improves handling of exceptions in Index Lifecycle (#3511)
* Improves handling of exceptions in Index Lifecycle

This change improves a few different aspects:

* If an exception occurs executing the lifecycle of one index it is caught, logged and other indexes are still processed
* If the lifecycle policy specified in the settings does not exist an error is logged
* Fixes the exception when the delete action is run which occurs because Phase attempts to update the phase and action settings for the deleted index. A `LifecycleAction.indexSurvives()` method is introduced which defaults to `true` but can be overridden to indicate whether the index survives following completion of the action.

* Adds test
2018-01-12 11:52:56 +00:00
Tal Levy e3a6e738ea Fix InternalIndexLifecycleContext to update state in memory (#3505)
* Fix InternalIndexLifecycleContext to update state in memory

The internal and the mock index-lifecycle-context implementations differed
in that the InternalIndexLifecycleContext assumed no one would be using it after
it mutated state. This is not the case. We assume that the current context is updated after
a `#setAction` is called so that the listener can then appropriately use the newly modified
cluster state. since idxMeta was not being updated, any call to `context.getAction` was stale and
either returning null or the previous action, not the next action that was updated by `#setAction`.
Same goes for `setPhase`.

This PR should fix this so that the Mock and Internal implementations are more in line.
2018-01-10 08:21:17 -08:00
Colin Goodheart-Smithe 57c343bc21 Use the XContentRegistry provided by the parser 2018-01-08 14:14:43 +00:00
Colin Goodheart-Smithe c4621d9dc7 Mutes IndexLifecycleInitialisationIT.testMasterFailover
Due to a bug with unassigned shards in force merge action
2018-01-05 12:43:13 +00:00
Colin Goodheart-Smithe 148688a991 Reduces poll time to 1s for integ test
`IndexLifecycleInitialisationIT.testMasterFailover()` intermittently failed because the timeout of 10 seconds to check if the index had been deleted was not long enough sometimes with the poll interval set to 3 seconds. This change sets the poll interval to 1 seconds for the test so that the lifecycle is more responsive. This also means the default value for the poll interval can be safely changed without affecting the test.
2018-01-03 12:29:19 +00:00
Colin Goodheart-Smithe 52e416d0bc Creates index-lifecycle gradle module
This doesn’t currently contain any code but is there in preparation for the x-pack-elasticsearch:plugin split
2018-01-03 12:03:54 +00:00
Colin Goodheart-Smithe 043c8db84d Fixes Tests so build passes 2018-01-03 12:01:12 +00:00
Colin Goodheart-Smithe a37df9092e Moves Actions and related to plugin:core
This change moves the Action classes and referenced data model classes to the new :x-pack-elasticsearch:plugin:core project in preparation for splitting the x-pack features into their own gradle modules.

Note that the TransportAction classes had to be promoted to their own class file (rather than being inner classes to their Action) so they can remain in the plugin project (and will late be move to the `index-lifecycle` project when its created.
2018-01-03 10:45:33 +00:00
Colin Goodheart-Smithe 33afe0f012 Merge branch 'master' into index-lifecycle 2018-01-03 10:21:50 +00:00
Tal Levy 6a47b4fa6e fix infinite loop in test and correct logic in timeseries type 2017-12-21 13:14:37 -08:00
Tal Levy 66af56320e fix tests 2017-12-21 12:56:56 -08:00
Tal Levy e433b1eaa8 Adds ForceMerge action to Index Lifecycle (#3309)
add forcemerge index-lifecycle action
2017-12-21 11:59:44 -08:00
Colin Goodheart-Smithe d5609a408b fixes integ test when running as part of the build 2017-12-21 12:38:34 +00:00
Colin Goodheart-Smithe 3627feded7 Refactors lifecycle type to its own class (#3381)
To clean up the parsing of the LifecyclePolicy this change moves the LifecycleType to its own class so it can be created in the normal parsing of LifecyclePolicy rather than having to parse to an intermediary object first. The LifecycleType is an interface which can be implemented for different lifecycle types. These types shiould be singletons and are register with the NamedXContentRegistry and NamedWriteableRegistry only so they are available when reading from a stream or parsing.
2017-12-21 11:20:49 +00:00
Tal Levy fb59f54637 make the lifecycle poll-interval configurable (#3319)
Removes the poll-interval from the IndexLifecycleMetadata and introduces it in 
the form of a cluster setting that is configurable. Changes to this poll interval setting 
will reflect in the Lifecycle Scheduler.
2017-12-20 09:39:39 -08:00
Colin Goodheart-Smithe 586065fca7 Changes to enable the build to run
This includes changing NOCOMMIT comments to be NORELEASE comments so the build passes with them. We have tasks inGH for all these NORELEASE comments so they should be caught before merging to master
2017-12-20 12:27:26 +00:00
Colin Goodheart-Smithe b12f1a9526 More phase tests
This time for when setting the new action fails
2017-12-15 11:20:59 +00:00
Colin Goodheart-Smithe 33a1a92e3d As phase test for when the action execution fails 2017-12-15 11:07:49 +00:00
Colin Goodheart-Smithe 9e693b544c Test improvements 2017-12-15 10:52:37 +00:00
Colin Goodheart-Smithe 3da42f5603 Adds rollover action to index lifecycle plugin (#3266)
This action will rollover an index when executed if the provided conditions are met.
Users may specify the maximum age, maximum index size in bytes or maximum index size in number of documents as conditions for rollover.

When the action executes it firsts checks the local cluster state to find out if the alias exists on the index. If the alias does not exist then the index was either rolled over by a previous run or something else has rolled over the index so the action can be marked as completed. If the index still has the alias set the action will make a rollover index request using the Client. When that request returns and the listener is called the action will only be marked as complete if the response indicates the index was rolled over. If the index was not rolled over (because the conditions are not yet met) the action is not marked as complete and will be re-evaluated on the next call to execute.
2017-12-15 09:36:39 +00:00
Tal Levy 8d6e53356b enable rest of await-fix index-lifecycle integ tests. all green 2017-12-13 13:40:58 -08:00
Tal Levy c0c7e02de4 fix javadoc usage 2017-12-11 16:24:44 -08:00
Tal Levy 288515ddc5 initial attempt towards an ordered execution plan for timeseries phases (#3248)
* initial attempt towards an ordered execution plan for timeseries phases

* respond to feedback and fix bug
2017-12-07 11:59:31 -08:00