Commit Graph

5416 Commits

Author SHA1 Message Date
Andy Bristol 67c0cf1dbf
[test] include oss tar in packaging tests (#30155)
Add the oss tar distribution to the packaging test plugin. Test the oss
tar distribution in the core packaging tests, and the non-oss tar
distribution in the x-pack packaging tests.
2018-04-26 06:58:41 -07:00
Martijn van Groningen 4bcecd34a1
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  Watcher: Fold two smoke test projects into smoke-test-watcher (#30137)
  In the field capabilities API, deprecate support for providing fields in the request body. (#30157)
  Set JAVA_HOME before forking setup commands (#29647)
  Remove animal sniffer from low-level REST client (#29646)
  Cleanup .gitignore (#30145)
  Do not add noop from local translog to translog again (#29637)
  Build: Assert jar LICENSE and NOTICE files match
  Correct transport compression algorithm in docs (#29645)
  [Test] Fix docs check for DEB package in packaging tests (#30126)
  Painless: Docs Clean Up (#29592)
  Fixes Eclipse build for sql jdbc project (#30114)
  Remove reference to `not_analyzed`.
  [Docs] Add community analysis plugin (#29612)
2018-04-26 09:25:19 +02:00
Alexander Reelsen fadcce8367
Watcher: Fold two smoke test projects into smoke-test-watcher (#30137)
In order to have less qa/ projects, this commit removes the
watcher-smoke-test-mustache and the watcher-smoke-test-painless projects
and moves the tests of both into the watcher-smoke-test projects.

This results in less projects to parse when calling gradle and a shorter test
time due to being able to run everything within one elasticsearch
instance.
2018-04-26 09:23:54 +02:00
Tal Levy b2e0d6af18 fix syntax error in info.asciidoc 2018-04-25 17:38:17 -07:00
Nik Everett a8f40b3e04 Build: Assert jar LICENSE and NOTICE files match
Adds tasks that check that the all jars that we build have LICENSE.txt
and NOTICE.txt files and that the files are correct. Sets check to
depend on these task.

This is mostly there for extra parnoia because we automatically
configure all Jar tasks to include the LICENSE.txt and NOTICE.txt
files anyway. But it is quite possible to add configuration to those
tasks that would override either file.

This causes check to depend on several more things than it used to.
Take, for example, javadoc:

check depends on the new verifyJavadocJarNotice which depends on
extractJavadocJar which depends on javadocJar which depends on
javadoc, this check now depends on javadoc.
2018-04-25 19:53:24 -04:00
Tal Levy 1df6b6c65d Applying changes required for ILM after merge 2018-04-25 13:04:06 -07:00
Tal Levy d74fb9eb45 Opened x-pack ILM code 2018-04-25 12:22:53 -07:00
Colin Goodheart-Smithe c02b895653
Fixes Eclipse build for sql jdbc project (#30114)
The bundled configuration isn't recognised by eclipse so these
dependencies are missed when it imports the `x-pack:plugin:sql:jdbc`
project. This change makes these dependencies compile dependencies if
the build is running for Eclipse.
2018-04-25 16:10:32 +01:00
Martijn van Groningen b126ffa1cb
Updated sha files 2018-04-25 16:00:48 +02:00
Martijn van Groningen 596fab9eb7
fixed compile error after merge 2018-04-25 15:59:22 +02:00
Martijn van Groningen e6b88fa5a0
removed duplicated license 2018-04-25 12:18:02 +02:00
Martijn van Groningen 831e7e0777
Fixed test failure after migrating xpack ccr code to elasticsearch repository. 2018-04-25 10:10:06 +02:00
Martijn van Groningen 5a67a0f78f
Applying changes required for ccr after moving ccr code to elasticsearch 2018-04-25 08:03:29 +02:00
Martijn van Groningen cbd38983f4
Opened x-pack ccr code 2018-04-25 08:03:28 +02:00
Michael Basnight 72f57c8e72
Add comments inadvertently removed during migrate
A few files had their first comment removed even though it did not
contain a license. This re-adds those comments.
2018-04-24 13:41:09 -05:00
David Kyle cfc66a1fd5 [ML] Wait for updates to established memory usage
Tests need to wait for changes to the job's established memory usage to
propagate and an over enthusiastic optimisation meant jobs were updated
from stale state causing recent change to be lost.
2018-04-24 13:46:58 -04:00
Jason Tedor ab101976d6 Fix SQL CLI on Windows
This commit fixes the classpath for the SQL CLI tool on Windows. As the
x-pack bin folder was collapsed into the distribution bin folder, the
location of the classpath here needed to no longer contain the old
plugins directory.
2018-04-20 15:34:01 -07:00
Jason Tedor d99d0fa669 Add distribution type to startup scripts
This commit adds the distribution type to the startup scripts so that we
can discern from log output and the main response the type of the
distribution (deb/rpm/tar/zip).
2018-04-20 15:34:01 -07:00
Jason Tedor c7f6b35485 Fix classpath for X-Pack scripts on Windows
With the move of X-Pack to a module, the classpath for the scripts needs
to be adjusted. This was done on Unix, but not for Windows. This commit
addresses Windows.
2018-04-20 15:34:01 -07:00
Ryan Ernst d8636d0fe3 Rename syskeygen
This commit renames syskeygen to elasticsearch-syskeygen
2018-04-20 15:34:01 -07:00
Jason Tedor c7c0e330b8 Rename users
This commit renames users to elasticsearch-users.
2018-04-20 15:34:01 -07:00
Jason Tedor b05c8bf781 Rename migrate
This commit renames migrate to elasticsearch-migrate.
2018-04-20 15:34:00 -07:00
Jason Tedor 2d3fac307a Rename saml-metadata
This commit renames saml-metadata to elasticsearch-saml-metadata.
2018-04-20 15:34:00 -07:00
Jason Tedor 8a8a1dfc2e Rename sql-cli
This commit renames sql-cli to elasticsearch-sql-cli.
2018-04-20 15:34:00 -07:00
Ryan Ernst 6dbb7be067 Rename croneval
This commit renames croneval to elasticsearch-croneval
2018-04-20 15:34:00 -07:00
Jason Tedor 7ba2defdbf Rename setup-passwords
This commit renames setup-passwords to elasticsearch-setup-passwords.
2018-04-20 15:34:00 -07:00
Jason Tedor 03ffd16921 Rename certutil
This commit renames certutil to elasticsearch-certutil.
2018-04-20 15:33:59 -07:00
Jason Tedor a6b4682711 Rename certgen
This commit renames certgen to elasticsearch-certgen.
2018-04-20 15:33:59 -07:00
Ryan Ernst 0d8aa7527e Reorganize license files
This commit moves the apache and elastic license files into a new
root level `licenses` directory and rewrites the top level LICENSE.txt
to clarify the repository has a mix of apache and elastic licensed code.
2018-04-20 15:33:59 -07:00
Ryan Ernst 9a45662309 Convert license header check to use the new Elastic License
This commit adapts the license headers check to no longer look for the
ealsticsearch confidential license, but instead to look for the new
Elastic License header.
2018-04-20 15:33:59 -07:00
Jason Tedor e64e6d8996 Add distribution flavor to startup scripts
This commit adds the distribution flavor (default versus oss) to the
build process which is passed through the startup scripts to
Elasticsearch. This change will be used to customize the message on
attempting to install/remove x-pack based on the distribution flavor.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst cb56bf49d5 Fixup build after opening x-pack 2018-04-20 15:33:46 -07:00
Ryan Ernst 2efd22454a Migrate x-pack-elasticsearch source to elasticsearch 2018-04-20 15:29:54 -07:00
Martijn van Groningen 9b9d0f9057 Enabled licence header check and fixed unchecked casts. (#4408) 2018-04-20 11:15:52 +02:00
Martijn van Groningen cfd7847628 fixed issues after merging in master 2018-04-20 07:59:13 +02:00
Nhat Nguyen f97aec7b8b Sibling of enforce access to translog via engine
Since elastic/elasticsearch#29542, we no longer expose translog instance
but only provide creating translog snapshot method. This commit adapts
that change in CCR branch.

Relates elastic/elasticsearch#29542
2018-04-18 11:54:00 -04:00
Martijn van Groningen 56ca59a513 Add the ability to the follow index to follow an index in a remote cluster.
The follow index api completely reuses CCS infrastructure that was exposed via:
https://github.com/elastic/elasticsearch/pull/29495

This means that the leader index parameter support the same ccs index
to indicate that an index resides in a different cluster.

I also added a qa module that smoke tests the cross cluster nature of ccr.
The idea is that this test just verifies that ccr can read data from a
remote leader index and that is it, no crazy randomization or indirectly
testing other features.
2018-04-17 07:36:40 +02:00
Tal Levy 689d628d67 move phase-after steps to have the previous phase's phase in its key (#4387)
Previously, phase X's `after` step had `X` as its associated phase. This causes confusion because
we have only entered phase `X` once the `after` step is complete. Therefore, this refactor
pushes the after's phase to be associated with the previous phase. This first phase is an exception. 
The first phase's `after` step is associated with the first phase (not some non-existent prior phase).
2018-04-16 21:19:30 -07:00
Martijn van Groningen c0d42e9cd1 Fixed test 2018-04-16 10:48:46 +02:00
Martijn van Groningen a94b38b88e Fixed compile errors and test failures after merging master into ccr. 2018-04-13 16:35:09 +02:00
Colin Goodheart-Smithe d509834c00 Fixes checkstyle error 2018-04-13 14:39:37 +01:00
Colin Goodheart-Smithe af6456983f Moves registing of NamedWriteables to plugin:core
This is needed so that clients have the ability to serialise and deserialise the objects it creates and gets back.
2018-04-13 14:23:54 +01:00
Colin Goodheart-Smithe bf3db3f1f7 Adds phase and action change times on state transition 2018-04-13 13:38:44 +01:00
Colin Goodheart-Smithe d17886f1b0 Fixes checkstyle error 2018-04-13 10:44:45 +01:00
Tal Levy f04c95f4c2 change policy logs to be DEBUG 2018-04-12 15:40:20 -07:00
Colin Goodheart-Smithe 644d82e165 Renames AliasStep to ShrinkSetAliasStep 2018-04-12 18:29:12 +01:00
Colin Goodheart-Smithe 77be66b8b9 Adds shrink check that original index is deleted 2018-04-12 18:13:57 +01:00
Colin Goodheart-Smithe 59e0979b92 Fixes shrunk shards allocation check step
- Renames EnoughShardsWaitStep to ShrunkShardsAllocatedStep
- Changes ShrunkShardsAllocatedStep to check the shards of the shrunken index rather than the current one
- shrink index prefix is now passed into the steps of the shrink aciton
- Related Test Changes
2018-04-12 18:06:04 +01:00
Tal Levy 34e15011a1 fix ilm rest-tests to reflect latest get_policy changes 2018-04-12 07:47:57 -07:00
Colin Goodheart-Smithe 088894019e Tidies up unit tests
Also fixes failing EnoughShardsWaitStep
2018-04-12 12:37:36 +01:00
Colin Goodheart-Smithe 1c2e7bc431 Increases iterations on step equality tests
To make it in line with the fact that other equality tests use multiple iterations
2018-04-12 12:23:35 +01:00
Colin Goodheart-Smithe e8cb6419f8 Removes unused imports 2018-04-12 12:16:43 +01:00
Colin Goodheart-Smithe 7b247fe41e Fixes compile errors from TimeValue serialisation 2018-04-12 12:15:36 +01:00
Tal Levy 20b6aaf6b6 migrate step tests to extend AbstractStepTestCase 2018-04-11 18:56:53 -07:00
Tal Levy c1b00d0154 cleanup tests
- make DeleteActionTests pass
- remove getType from LifecycleType
2018-04-11 18:28:46 -07:00
Tal Levy 7f4801f03c clean up PhaseTests 2018-04-11 18:16:28 -07:00
Tal Levy 73ee5a4aa0 introduce refactored ShrinkAction
this resulted in a few new steps being introduced

- AliasStep
- EnoughShardsWaitStep
- ShrinkStep
- ShrunkenIndexCheckStep
2018-04-11 18:10:13 -07:00
Colin Goodheart-Smithe 20485cf7fb Adds more missing equality tests
Specifically for the initialPolicyContextStep and the PhaseAfterStep
2018-04-11 13:24:31 +01:00
Colin Goodheart-Smithe 482de191f2 Tidies up some of the unit tests
Adds some missing tests including checking the hashcode and equals methods of `DeleteStep`, `StepKey`, and `TerminalPolicyStep` as well as adding a test for `DeleteAction.toSteps()`
2018-04-11 12:38:55 +01:00
Colin Goodheart-Smithe 66b4e8c4aa Removes unused imports 2018-04-11 10:47:07 +01:00
Tal Levy ef34f982f1 refactor a collection of step-related things + shrink_step
- added ShrinkStep/Tests
- AsyncActionStep now passes in IndexMetaData instead of Index
- Delete usage of ClusterStateActionStep
- with ClusterStateActionStep gone, InitializePolicyContextStep
  is the only other ClusterState-nonWait step
- Migrate setting-updates to UpdateSettingsStep
2018-04-10 17:13:10 -07:00
Colin Goodheart-Smithe 8d91f197d4 Changes update replica and alllocation to async steps
Also renames EnoughShardsWaitStep to ReplicasAllocatedStep, removes it from the allocate action and adds a check that th number of replicas in the cluster state is correct to it.
2018-04-10 16:20:29 +01:00
Colin Goodheart-Smithe aa198b637e Cleans up warnings 2018-04-10 09:53:57 +01:00
Colin Goodheart-Smithe c24dc07346 Removes unused and unneeded code
Various classes had some code that was not used and is not going to be needed so this change cleans up those classes so we don’t have dead code hanging around
2018-04-10 09:28:03 +01:00
Colin Goodheart-Smithe b8d573772f Fixes compilation issue in test 2018-04-10 09:10:52 +01:00
Tal Levy f5b23df349 add more force-merge related steps and tests
The force-merge is an a TODO state due to the
unresolved issue around best_compression.

- updated ReadOnlyStep with tests
- implemented an update to the ForceMergeAction
- added UpdateBestCompressionSettingsStep
- added tests for SegmentCountStep
2018-04-09 16:28:25 -07:00
Colin Goodheart-Smithe 06e969b430 Adds rollover step and implements RolloverAction.toSteps 2018-04-09 15:55:06 +01:00
Colin Goodheart-Smithe 54c6d280a5 Adds Steps for ReplicasAction
Also adds tests for those steps and ReplicasAction.toSteps
2018-04-09 14:43:57 +01:00
Colin Goodheart-Smithe 2eb8fd9336 Adds tests for AllocateAction and its steps
Also tweaks some of the code in the steps of the allocate action
2018-04-09 13:59:25 +01:00
Tal Levy 47ed2e5b23 add more tests 2018-04-05 18:00:41 -07:00
Tal Levy b58159ddad add more tests 2018-04-05 17:43:32 -07:00
Tal Levy 228fc8c842 add some stuff 2018-04-05 17:21:44 -07:00
Tal Levy c0acea3721 :check yourself greeny 2018-04-05 13:16:19 -07:00
Tal Levy 1a2afac98b fix tests 2018-04-05 12:43:10 -07:00
Colin Goodheart-Smithe e3e669067c Adds loggic for TerminalPolicyStep 2018-04-05 17:58:22 +01:00
Colin Goodheart-Smithe 05c2c5655b replaces cause with boolean 2018-04-05 17:53:15 +01:00
Colin Goodheart-Smithe ed260a0430 Removes Cause stuff 2018-04-05 17:10:56 +01:00
Martijn van Groningen d77f756f5c ccr: use indices stats api to fetch global checkpoint of the follower shards and
keep track of shard follow stats inside shard follow stats' node task instead of persistent task status.

By maintaining the shard follow stats inside its node task the stats update is quicker as
no cluster state update is required. The stats are now transient; meaning if the task
is going to run a different node then the stats are gone too. Currently only the processed
global checkpoint is being tracked and this is being restored when a shard follow node task
starts via the indices stats api (the reason of the first change of this change). Other stats
that we may add in the future (like fetch_time, see: https://gist.github.com/s1monw/dba13daf8493bf48431b72365e110717)
it is ok if we start from zero in case a shard follow task moves to another node.
2018-04-05 14:52:20 +02:00
Tal Levy 907586c9e4 add MoveToNextStepUpdateTaskTests 2018-04-04 20:39:00 -07:00
Tal Levy 55d2a62fec add tests for ExecuteStepsUpdateTask 2018-04-04 15:32:36 -07:00
Colin Goodheart-Smithe a8dc8a7c98 Adds more unit tests for IndexLifecycleRunner 2018-04-04 15:59:10 +01:00
Colin Goodheart-Smithe 9ef26dbe51 Unit tests static methods in IndeexLifecycleRunner 2018-04-04 13:29:30 +01:00
Tal Levy 47dcc8fe9c more changes + updated api 2018-04-03 17:42:25 -07:00
Tal Levy 1ac1ee413f moar changes 2018-04-02 16:21:48 -07:00
Tal Levy d2e87a66e5 moar changes
- set `indexSurvives` for Step
- extract the two ClusterStateUpdateTasks to separate classes

- simple Delete Policy works!
2018-03-29 16:23:57 -07:00
Martijn van Groningen d976fa44e7 Removed LocalCheckpointTracker usage. 2018-03-29 07:41:23 +02:00
Tal Levy 74eaba2078 moar changes from Tal 2018-03-28 17:10:50 -07:00
Colin Goodheart-Smithe 179074d3b0 Refactors steps and adds some execution logic
This is far from complete but its necessary to push so the new step classes can be shared and used.
2018-03-28 16:34:34 +01:00
Martijn van Groningen a22a7d079d ccr: Added maximum translog limit that a single shard changes response can return.
This limit is based on the number of estimate bytes in each translog
operation that fall between the minimum and maximum request sequence number.

If this limit is met then the shard follow task executor will make sure
that a subsequent shard changes request will be performed to fetch the
remaining translog operations.

This limit is needed in order to protect against returning too many
translog operations in a single shard changes response.

Relates to #2436
2018-03-28 15:49:57 +02:00
Martijn van Groningen 282740610b Fixed test after merging in master branch. 2018-03-28 09:54:41 +02:00
Tal Levy e6ee5b49d1 have steps point to step-key, not the actual step object 2018-03-27 11:52:08 -07:00
Tal Levy ce4248ec59 slight changes 2018-03-27 11:41:38 -07:00
Colin Goodheart-Smithe 9972710e9e Makes brnach compile
Commented out toSteps implem,entations and other bits needed to get the branch to compile
2018-03-27 12:14:38 +01:00
Tal Levy f429fc0b3e begin making sense of types 2018-03-26 18:48:59 -07:00
Tal Levy 083c563cf6 meh 2018-03-26 18:12:38 -07:00
Tal Levy 57821cd55a moar refactor for steps 2018-03-26 17:54:15 -07:00
Nhat Nguyen 51111a8106 CCR: Stop FollowExistingIndexAction after report failure (#4111)
We check for the existence of both leader and follower index, then properly 
report to the caller. However, we do not return after reporting failure. This
causes the caller receive exception twice: IllegalArgumentException then
NullPointerException. This commit makes sure to stop the action after reporting
failure.
2018-03-26 13:56:47 -04:00
Tal Levy d63cd8c9c3 step by step 2018-03-26 08:00:03 -07:00
Martijn van Groningen 9e4c68c389 Fixed compile and test errors after merging in master 2018-03-16 17:47:10 +01:00
Tal Levy fe22c08207 stop using illegal logging of Object 2018-03-12 13:18:02 -07:00
Tal Levy ebff02c1b2 fix compilation changes 2018-03-12 12:29:22 -07:00
Martijn van Groningen 10cfa21a68 required changes after merge master branch into ccr branch. 2018-02-22 15:03:33 +01:00
Tal Levy 9850f44b95 fix state left over by spec test 2018-02-13 13:40:09 -08:00
Tal Levy 8e282f21c8 Merge branch 'master' into index-lifecycle 2018-02-12 14:17:24 -08:00
Martijn van Groningen 1a9a7ffe97 removed hack 2018-02-07 17:54:28 +01:00
Martijn van Groningen c442d14f1d Several changes that were required after merging master into the ccr branch. 2018-02-05 13:25:58 +01:00
Tal Levy 0a3dcebca0 disallow deleting lifecycle policies that are referenced by existing indices (#3806) 2018-02-01 14:16:06 -08:00
Tal Levy ab49e40f91 add phase and action timestamps to cluster-state (#3726)
This commit sets `index.lifecycle.phase_time` and `index.lifecycle.action_time`
appropriately during setPhase and setAction in IndexLifecycleContext.
2018-01-25 09:02:26 -08:00
Colin Goodheart-Smithe faf067080b Makes sure creation date garentees execution in tests 2018-01-25 13:21:50 +00:00
Martijn van Groningen 4e818254ad re-enabled java integration tests 2018-01-25 14:18:34 +01:00
Tal Levy 9920d6be2d keep plugin up to date with structure changes (#3702)
Looks like we need to split out the tests of core classes to core
and index-lifecycle ones stay in index-lifecycle.

I believe I got everything, although I may have missed at least one thing

checked status with

$ ./gradlew :x-pack-elasticsearch:plugin:index-lifecycle:check -Dtests.seed=39838421912001B4
$ ./gradlew :x-pack-elasticsearch:plugin:core:check -Dtests.seed=39838421912001B4

other things done in this PR:
- removal of a few unused variables/thrown exceptions/imports
- fix TimeseriesLifecycleTypeTests
  - an all null AllocateAction was created
- fix AllocateActionTests
  - woops. -Dtests.seed=39838421912001B4 resulted in two `null`s and an emptyMap.
    this resulted in a test failure.
2018-01-24 08:47:21 +00:00
Martijn van Groningen 05d3d2e49c fix packages after merge 2018-01-24 09:28:42 +01:00
Tal Levy 4856baf66b fix guice and plugin initialization after split (#3677)
the x-pack-split required updating the IndexLifecycle plugin to
be a proper plugin with the Plugin API methods instead of the XPackPlugin API
methods.
2018-01-23 10:41:22 -08:00
Colin Goodheart-Smithe fd502aa3e6 Makes some fields in Index Lifecycle API optional (#3687)
Specifically this change makes it optional to:
* Specify `includes`, `excludes` and `requires`maps in the allocate action as long as at least one fo the options is specified and is not an empty map
* Specify an `after` parameter on a phase. If no `after` value is specified `TimeValue.ZERO` is used and the phase will be moved to as soon as the previous phase reports `ACTIONS COMPLETED`. `after` is always non-null when we are serialising the Phase.
* Specify a `type` for a LifecyclePolicy. If no `type` is specified `TimeSeriesLifecycleType.INSTANCE` is used since this is currently the only production `type`. `type` is always non-null when we are serialising the LifecyclePolicy.
2018-01-23 17:02:11 +00:00
Colin Goodheart-Smithe f2fa988f2f Fixes Checkstyle 2018-01-23 08:24:03 +00:00
Jason Tedor 9b6bb2c635 Enable run task for CCR
This commit enables the run task for ccr by specifying that the ccr
project not be evaluated until after core is evaluated. This is
important since ccr is alphabetically before core and thus Gradle
evaluates it first.

Relates #3665
2018-01-22 15:07:20 -05:00
Tal Levy 07b28fd721 migrate lifecycle settings to plugin:core:LifecycleSettings (#3670) 2018-01-22 10:43:16 -08:00
Tal Levy 34d6b61209 add `index.lifecycle.date` setting (#3545)
This PR adds a new setting called `index.lifecycle.date` that 
the ShrinkAction will be responsible for populating in the newly created index.

This way, we can continue to know when we should be executing the next phase
relative to the original index creation date, and not that of the shrunken index.
2018-01-22 09:27:33 -08:00
Colin Goodheart-Smithe 9032d61516 Fixes silly issues created by me 2018-01-22 16:48:32 +00:00
Colin Goodheart-Smithe 2d721136c2 Adds logic to cancel job if master is un-elected (#3613)
A node can stop being the master node whilst it is running, e.g. if it can’t access `minimumMasterNodes` number of master eligible nodes. Because of this we need logic in `IndexLifecycleService` that cancels the scheduled job if the node is no longer master and re-adds the job if the node becomes master again.
2018-01-22 15:45:37 +00:00
Colin Goodheart-Smithe ee774594e2 Adds test for TimeValueSchedule 2018-01-22 15:21:55 +00:00
Colin Goodheart-Smithe a6f6335e86 Moves Index-Lifecycle classes over to new module
These changes are necessary following the x-pack split.
2018-01-22 13:52:22 +00:00
Martijn van Groningen 83a82d83d0 Moved ccr source code to its own gradle module after xpack split. 2018-01-22 11:09:04 +01:00
Colin Goodheart-Smithe 3cdf1216cd Removes un-needed licence files 2018-01-19 10:27:57 +00:00
Colin Goodheart-Smithe cb4cbf80ac Catches more that just ElasticsearchException 2018-01-18 05:42:22 +00:00
Tal Levy 84fd234835 Add the Shrink Action to Index Lifecycle Management (#3491)
This does the following in sequential service polls

1. sets the index to read-only and runs shrink with a modified `index.lifecycle.name` setting set to `null`.
2. checks to see if shrink is complete, if it is...
    b. set target index's `index.lifecycle.*` settings to the original index's values.
3. if not complete, just wait till next iteration
4. if operating on shrunken index, delete old index and add it as an alias to shrunken index
2018-01-17 11:22:06 -08:00
Colin Goodheart-Smithe ff883d574b Adds Allocate lifecycle action (#3484)
* Adds Allocate lifcycle action

* Addresses review comments

Still need to make a change in core for the FilterAllocationDecider to make the execute logic simpler

* Addresses more review comments

* Adds randomMap method to AllocateActionTests

* Addresses further review comments
2018-01-17 13:27:40 +00:00
Tal Levy db3a88f487 fix TimeseriesLifecycle ordering behavior 2018-01-16 18:08:05 -08:00
Martijn van Groningen 7a4860452a Update the persistent task status in the background.
Follow up for #3256
2018-01-16 10:48:55 +01:00
Jason Tedor bad5135d35 Fix checkpoints in assertion
The checkpoints in the assertion message that the follower checkpoint is
less than the leader checkpoint are backwards. This commit fixes this
message.
2018-01-15 11:50:12 -05:00
Colin Goodheart-Smithe 971685654d Adds more tests in IndexLifecycleServiceTests 2018-01-15 15:29:07 +00:00
Colin Goodheart-Smithe 1e6ed023ee Fixes ReplicasActionTests
Fixes a test failure and adds a new test that was missing
2018-01-15 09:37:00 +00:00
Colin Goodheart-Smithe 53c3d040f7 Add ReplicaAction for Index Lifecycle (#3535)
* Add ReplicaAction for Index Lifecycle

* Add validation
2018-01-12 13:48:09 +00:00
Martijn van Groningen 18788531c8 Fixed test failure.
Relates to #3256
2018-01-12 14:04:26 +01:00
Colin Goodheart-Smithe 5eb912539d Fixes Logging usage
log line had an extra parameter
2018-01-12 12:17:14 +00:00
Colin Goodheart-Smithe 6a370251fc Improves handling of exceptions in Index Lifecycle (#3511)
* Improves handling of exceptions in Index Lifecycle

This change improves a few different aspects:

* If an exception occurs executing the lifecycle of one index it is caught, logged and other indexes are still processed
* If the lifecycle policy specified in the settings does not exist an error is logged
* Fixes the exception when the delete action is run which occurs because Phase attempts to update the phase and action settings for the deleted index. A `LifecycleAction.indexSurvives()` method is introduced which defaults to `true` but can be overridden to indicate whether the index survives following completion of the action.

* Adds test
2018-01-12 11:52:56 +00:00
Tal Levy e3a6e738ea Fix InternalIndexLifecycleContext to update state in memory (#3505)
* Fix InternalIndexLifecycleContext to update state in memory

The internal and the mock index-lifecycle-context implementations differed
in that the InternalIndexLifecycleContext assumed no one would be using it after
it mutated state. This is not the case. We assume that the current context is updated after
a `#setAction` is called so that the listener can then appropriately use the newly modified
cluster state. since idxMeta was not being updated, any call to `context.getAction` was stale and
either returning null or the previous action, not the next action that was updated by `#setAction`.
Same goes for `setPhase`.

This PR should fix this so that the Mock and Internal implementations are more in line.
2018-01-10 08:21:17 -08:00
Martijn van Groningen a56b0479e1 Changed the chunk coordinator to process the chunks concurrently. (#3256)
The shard follow task executor determines the range of translog operations
between the leader shard's global checkpoint and the last know processed
seqno by the current shard follow task that are missing.

Then the chunks coordinator can then chunk this range up in smaller ranges
if the requested range is above the configured max chunk size. If it is
smaller than the entire range then the chunk coordinator has just one
chuck to coordinate.

Each chunk is added to a queue and is processed by the ChunkProcessor,
that reads the translog ops from the leader shard and then indexes
these translog ops into the follow shard. After that a new chuck is polled
from the queue and the ChunkProcessor performs the same actions until
there are no more chunks in the queue to process. After that the shard
follow task executor will determine a new range of translog operations
to process.

This change changes the chunk coordinator to start polling from the chunk
queue with multiple threads at the same time to handle dealing with a higher
indexing load on the leader side better.
2018-01-10 15:29:11 +01:00
Colin Goodheart-Smithe 57c343bc21 Use the XContentRegistry provided by the parser 2018-01-08 14:14:43 +00:00
Colin Goodheart-Smithe c4621d9dc7 Mutes IndexLifecycleInitialisationIT.testMasterFailover
Due to a bug with unassigned shards in force merge action
2018-01-05 12:43:13 +00:00
Martijn van Groningen 38aea9588b Fixed compile errors after merge. 2018-01-04 18:18:46 +01:00
Colin Goodheart-Smithe 148688a991 Reduces poll time to 1s for integ test
`IndexLifecycleInitialisationIT.testMasterFailover()` intermittently failed because the timeout of 10 seconds to check if the index had been deleted was not long enough sometimes with the poll interval set to 3 seconds. This change sets the poll interval to 1 seconds for the test so that the lifecycle is more responsive. This also means the default value for the poll interval can be safely changed without affecting the test.
2018-01-03 12:29:19 +00:00
Colin Goodheart-Smithe 52e416d0bc Creates index-lifecycle gradle module
This doesn’t currently contain any code but is there in preparation for the x-pack-elasticsearch:plugin split
2018-01-03 12:03:54 +00:00
Colin Goodheart-Smithe 043c8db84d Fixes Tests so build passes 2018-01-03 12:01:12 +00:00
Colin Goodheart-Smithe a37df9092e Moves Actions and related to plugin:core
This change moves the Action classes and referenced data model classes to the new :x-pack-elasticsearch:plugin:core project in preparation for splitting the x-pack features into their own gradle modules.

Note that the TransportAction classes had to be promoted to their own class file (rather than being inner classes to their Action) so they can remain in the plugin project (and will late be move to the `index-lifecycle` project when its created.
2018-01-03 10:45:33 +00:00
Colin Goodheart-Smithe 33afe0f012 Merge branch 'master' into index-lifecycle 2018-01-03 10:21:50 +00:00
Tal Levy 6a47b4fa6e fix infinite loop in test and correct logic in timeseries type 2017-12-21 13:14:37 -08:00
Tal Levy 66af56320e fix tests 2017-12-21 12:56:56 -08:00
Tal Levy e433b1eaa8 Adds ForceMerge action to Index Lifecycle (#3309)
add forcemerge index-lifecycle action
2017-12-21 11:59:44 -08:00
Martijn van Groningen dbdefc5689 Fix issue after merging 2017-12-21 14:53:12 +01:00
Colin Goodheart-Smithe d5609a408b fixes integ test when running as part of the build 2017-12-21 12:38:34 +00:00
Colin Goodheart-Smithe 3627feded7 Refactors lifecycle type to its own class (#3381)
To clean up the parsing of the LifecyclePolicy this change moves the LifecycleType to its own class so it can be created in the normal parsing of LifecyclePolicy rather than having to parse to an intermediary object first. The LifecycleType is an interface which can be implemented for different lifecycle types. These types shiould be singletons and are register with the NamedXContentRegistry and NamedWriteableRegistry only so they are available when reading from a stream or parsing.
2017-12-21 11:20:49 +00:00
Tal Levy fb59f54637 make the lifecycle poll-interval configurable (#3319)
Removes the poll-interval from the IndexLifecycleMetadata and introduces it in 
the form of a cluster setting that is configurable. Changes to this poll interval setting 
will reflect in the Lifecycle Scheduler.
2017-12-20 09:39:39 -08:00
Colin Goodheart-Smithe 586065fca7 Changes to enable the build to run
This includes changing NOCOMMIT comments to be NORELEASE comments so the build passes with them. We have tasks inGH for all these NORELEASE comments so they should be caught before merging to master
2017-12-20 12:27:26 +00:00
Martijn van Groningen 973f126c24 Merge branch 'master' into ccr 2017-12-18 16:52:09 +01:00
Colin Goodheart-Smithe b12f1a9526 More phase tests
This time for when setting the new action fails
2017-12-15 11:20:59 +00:00
Colin Goodheart-Smithe 33a1a92e3d As phase test for when the action execution fails 2017-12-15 11:07:49 +00:00
Colin Goodheart-Smithe 9e693b544c Test improvements 2017-12-15 10:52:37 +00:00
Colin Goodheart-Smithe 3da42f5603 Adds rollover action to index lifecycle plugin (#3266)
This action will rollover an index when executed if the provided conditions are met.
Users may specify the maximum age, maximum index size in bytes or maximum index size in number of documents as conditions for rollover.

When the action executes it firsts checks the local cluster state to find out if the alias exists on the index. If the alias does not exist then the index was either rolled over by a previous run or something else has rolled over the index so the action can be marked as completed. If the index still has the alias set the action will make a rollover index request using the Client. When that request returns and the listener is called the action will only be marked as complete if the response indicates the index was rolled over. If the index was not rolled over (because the conditions are not yet met) the action is not marked as complete and will be re-evaluated on the next call to execute.
2017-12-15 09:36:39 +00:00
Tal Levy 8d6e53356b enable rest of await-fix index-lifecycle integ tests. all green 2017-12-13 13:40:58 -08:00
Tal Levy c0c7e02de4 fix javadoc usage 2017-12-11 16:24:44 -08:00
Tal Levy 288515ddc5 initial attempt towards an ordered execution plan for timeseries phases (#3248)
* initial attempt towards an ordered execution plan for timeseries phases

* respond to feedback and fix bug
2017-12-07 11:59:31 -08:00
Martijn van Groningen 6c06aacb57 Fixed NumberFormatException in the rest layer by only setting batch_size if it has been specified. 2017-12-07 09:34:25 +01:00
Tal Levy ae7e4882e4 general indexlifecycle cleanup, rest tests, reintroduce integ-test (#3243)
* cleanup, rest tests, reintroduce integ-test

- cleaned up some long lines
- re-introduced one of the integration tests
- added rest-spec tests
2017-12-06 07:51:06 -08:00
Tal Levy d847355b58 add timeseries policy action stubs (#3214) 2017-12-05 09:09:18 -08:00
Martijn van Groningen 0dc5a89bc5 fixed compile error 2017-12-01 14:14:17 +01:00
Martijn van Groningen cc5665bce6 pass down batch size correctly 2017-12-01 14:14:17 +01:00
Martijn van Groningen 3cd5eed91e fixed code violations 2017-12-01 14:14:17 +01:00
Martijn van Groningen c3f7d4f580 * Made ChunkCoordinator unittest testable
* Fixed a small issue where each batch would fetch / index the previous batch last operation
* Made batch size a request param on the follow existing index api request.
  This makes is easy to tune this param when running tests from scripts.
* Changed default batch size from 256 to 1024.
2017-12-01 14:14:17 +01:00
Martijn van Groningen 2e382bf7f3 Disallow dynamic mapping updated in follow shards
I forgot to configure a mapping in the follow shard shard, which caused
a dynamic update (due to type auto creation), but this was ignored.

Subsequent searches in follow index then failed due to a mapping missing.
(The _id couldn't be fetched during fetch phase, because the mapping was missing)

We should at a later stage investigate how to best solve this, but for
know to avoid confusion just fail if a dynamic update happens in a
follow shard.
2017-12-01 09:33:06 +01:00
Tal Levy 685231e786 Introduce specific types of index lifecycle policies (#3157)
Introduce specific types of index lifecycle policies
2017-11-30 10:09:41 -08:00
Jason Tedor be5f83a6bd Implement translog operation bulk action
This commit adds a bulk action for apply translog operations in bulk to
an index. This action is then used in the persistent task for CCR to
apply shard changes from a leader shard.

Relates #3147
2017-11-29 07:29:56 -05:00
Jason Tedor 11aa83011c Fix get operations between test
This test was broken by an upstream change that no longer guarantees we
see the operations from the upstream translog in the order they appear
in that translog. As such, the assertions in this test were too strong
so this commit relaxes them.

Relates #3153
2017-11-29 07:16:52 -05:00
Jason Tedor abd263fca9 Merge branch 'master' into ccr
* master:
  [DOCS] Fixed cross cluster search docs issue (#3113)
  Watcher: Add transform input for chained input (#2861)
  [DOCS] Fix cleaning up ML resources after tests
  [ML] Specify ML_ORIGIN when calling the job update endpoint internally (#3110)
  Refactor CLI commands as logging-aware commands
  [ML] Stop datafeed when job fails (#3107)
2017-11-27 14:03:10 -05:00
Colin Goodheart-Smithe 922060d766 Typo 2017-11-27 15:46:47 +00:00
Colin Goodheart-Smithe 7199e2fa5c Small refactoring 2017-11-27 12:32:55 +00:00
Colin Goodheart-Smithe 87766cd4ef Adds JavaDocs and comments and small refactoring 2017-11-27 12:22:10 +00:00
Colin Goodheart-Smithe 9ab2121fcf When actions complete the next action is now run 2017-11-27 11:07:26 +00:00
Colin Goodheart-Smithe 152d5d45e5 Adds listeners to Lifecycle actions
This means that the result of the action can now be async and we can then implement moving immediately to the next action if the current one is complete
2017-11-24 19:27:50 +00:00
Colin Goodheart-Smithe 26ad2142d5 minor tweak 2017-11-24 17:29:38 +00:00
Colin Goodheart-Smithe 9e891de777 refactors code to allow better testing
The client and index metadata have now been abstracted away from the Lifecycle classes behind IndexLifecycleContext. This allow us to test the state machine without having to worry about how the state is persisted and read. It also makes the classes much easier to read and reason about.
2017-11-24 17:22:51 +00:00
Colin Goodheart-Smithe e113189347 only creates initialisation service ifeature is enabled 2017-11-24 10:23:04 +00:00
Colin Goodheart-Smithe 06958949ab Uses LongSupplier for nowSupplier
Rather than Supplier<Long>. This is in line with the `nowSupplier`s used in other areas of Elasticsearch
2017-11-24 10:18:06 +00:00
Colin Goodheart-Smithe 31ded906d0 Adds settings checking to tests
This isn’t an ideal solution so might well be temporary but at least the tests are checking the settings changes are correct now
2017-11-24 10:10:46 +00:00
Colin Goodheart-Smithe 016ba2a57d Adds another test
This test needs more work
2017-11-24 09:40:32 +00:00
Colin Goodheart-Smithe 49b2aaa878 Adds more tests 2017-11-23 18:16:47 +00:00
Colin Goodheart-Smithe 75704510e1 More test changes 2017-11-23 15:48:37 +00:00
Colin Goodheart-Smithe a9703d4a56 Mutes failing integration test for now 2017-11-23 14:12:31 +00:00
Colin Goodheart-Smithe 0e829843fc More unit tests 2017-11-23 14:11:09 +00:00
Colin Goodheart-Smithe db502bef6d Adds some unit tests 2017-11-23 12:44:12 +00:00
Colin Goodheart-Smithe f571dc3000 Remove references to InternalClient which no longer exists 2017-11-23 10:35:25 +00:00
Colin Goodheart-Smithe a3adcb680d Trying to get the test working 2017-11-23 10:00:23 +00:00
Colin Goodheart-Smithe 9cba84b6fc Obeys the after parameter on phases excpet the first 2017-11-22 16:09:04 +00:00
Colin Goodheart-Smithe 3f0cf05dcc Started writing/reading phase and action settings 2017-11-22 13:26:05 +00:00
Colin Goodheart-Smithe a26a5969dd iterates over index and executes correct policy 2017-11-22 10:30:21 +00:00
Martijn van Groningen e6ad1c3e9d ccr: Added apis and persistent tasks for following an index and log changes instead of indexing. 2017-11-22 09:04:59 +01:00
Colin Goodheart-Smithe eba5afefb7 Gets the scheduler to use poll_interval 2017-11-21 16:54:28 +00:00
Colin Goodheart-Smithe 3f3f7b9b47 Adds PUT/GET/DELETE lifecycle APIs
The lifecycles are stored as custom metadata objects in the cluster state. This change also cleans up the parsing of the lifecycle state so that it can be parsed properly
2017-11-21 15:55:45 +00:00
Tal Levy 1fad59b295 tal-temp migration to not use settings for the lifecycle policy 2017-11-17 09:22:54 -07:00
Jason Tedor 8e0b34b507 Override indexing plans
Operations from a leader shard will be indexed into the engine with the
origin set to primary. The problem is here is that then we have primary
semantics in the engine such as assertions about sequence numbers being
unassigned, and we do not have correct semantics for out-of-order
delivery of operations (as we should on a following engine, whether or
not it is primary since the ordering is determined from the
leader). This commit handles this by always using the replica plan for
indexing into a following engine, whether or not the engine is for a
primary shard.

Relates #3000
2017-11-15 13:38:46 -05:00
Jason Tedor 913936f2b0 Maintain order of operations semantics on follower
A following engine even for a primary shard needs to maintain order of
operations semantics as if it were behaving like a replica. That is,
rather than assuming that the order of operations presented to the
engine is the de facto order of operations as is the case for a leader
engine for a primary shard, a following engine must behave like all
replicas behave which is that they resolve order of operations based on
sequence numbers. This commit causes this to be the case for following
engines.

Relates #2931
2017-11-09 15:18:54 -05:00
Martijn van Groningen bf4e18fdfc fixed compile error 2017-11-02 15:12:35 +01:00
Tal Levy dd6d192e25 cleanup tests 2017-10-31 17:39:25 -07:00
Colin Goodheart-Smithe c393944dc0 Adds settings, phase and action objects which control how and when things are executed 2017-10-31 15:30:42 +00:00
Tal Levy 3455445d2c add integration test to indexlifecyclemanagement
This test verifies that we have sufficient failover code so that
a newly elected master re-registers old schedules and fires them off.
All times are relative to the index creation date.
2017-10-30 17:51:01 -07:00
Jason Tedor 769349a9ab Add following engine implementation
This commit is a first step towards a following engine
implementation. Future work will build on this by using this engine to
execute operations on a following engine from another engine (typically
a remote leader engine) that has already assigned sequence numbers to
such operations.

Relates #2776
2017-10-30 13:38:02 -04:00
Tal Levy de4d87c6e6 test 2017-10-26 15:32:08 -07:00
Martijn van Groningen 41c3dc91c1 Changed the shard changes api to use get threadpool instead of management threadpool. 2017-10-17 10:02:28 +02:00
Martijn van Groningen 2663b5ef39 remove unused import 2017-10-16 10:36:22 +02:00
Jason Tedor 85c1935d9d Increase generations in operations between test
This commit sets an index setting for the size of a translog generation
and increases the number of documents indexed to increase the chance of
multiple generations being present when testing getting operations
between two sequence numbers.
2017-10-16 08:59:18 +02:00
Jason Tedor bcd61bfca5 Fix off-by-one error in shard changes action test
This commit fixes an off-by-one error in the shard changes action test
for getting operations between two sequence numbers. The off-by-one
error arises because sequence numbers are indexed from zero, so if N
documents are indexed then the maximum sequence number starting from
zero would be N - 1.
2017-10-16 08:54:04 +02:00
Martijn van Groningen 1f495f59a1 [CCR] Add an internal api to read translog operations between a sequence number range (#2563)
* xdcr: Add an internal api to read translog operations between a sequence number range.

This api will be used later by the persistent task for the following index to pull data from the leader index.

The persistent task can fetch the global checkpoint from the shard stats for each primary shard of the leader index.
Based on the global checkpoint of the primary shards of the following index, the persistent task can send several
calls to the internal api added in this commit to replicate changes from follow index to leader index in a batched manner.
2017-10-16 07:38:03 +02:00
Colin Goodheart-Smithe 4aaec2ef81 Creates a new X-Pack feature for index lifecycle
Feature consists of a shell of a persistant task which will later be used to inspect the index settings and apply curator like changes to the index (move from hot to warm, rollover, shrink etc.)
2017-10-12 16:22:42 +01:00
Jason Tedor 3033aba67b Introduce pluggable engine factory for CCR
This commit utilizes the pluggable engine factory feature in core to
introduce a pluggable engine factory for XDCR. For now this is only a
skeleton implementation to proof out the pluggable engine factory
concept. Future work will implement a genuine following engine for XDCR.

Relates #2655
2017-10-12 10:52:33 -04:00
Jason Tedor 063a76c3db Introduce CCR container class
This commit introduces the container class for CCR functionality. Future
work will expose more specific CCR functionality to the X-Pack plugin
through this class.

Relates #2704
2017-10-06 14:46:39 -04:00