Commit Graph

357 Commits

Author SHA1 Message Date
Ioannis Kakavas 4b319d7151
SAML: Process only signed data (#30420)
As conformance to best practices, this changes ensures that if a
SAML Response is signed, we verify the signature before processing
it any further. We were only checking the InResponseTo and
Destination attributes before potential signature validation but
there was no reason to do that up front either.
2018-05-10 11:42:22 +03:00
Ryan Ernst bd24caccaf
Build: Remove xpack specific run task (#30487)
With the opening of xpack, we still retained a run task within
:x-pack:plugin. However, the root level run task also runs with the
default distribution. This change removes the extra run task inside
xpack in favor of using the root level task, and moves the
license/configuration code for run into the main run configuration.
2018-05-09 18:46:14 -07:00
Jack Conradson c87a3ea49b Silence IndexUpgradeIT test failures. (#30430) 2018-05-09 13:53:22 -07:00
Jay Modi 143df3a51d
Test: remove hardcoded list of unconfigured ciphers (#30367)
This commit removes the hardcoded list of unconfigured ciphers in the
SslIntegrationTests. This list may include ciphers that are not
supported on certain JVMs. This list is replaced with code that
dynamically computes the set of ciphers that are not configured for
use by default.
2018-05-09 08:41:17 -06:00
Colin Goodheart-Smithe 54d1efd73b
Adjusts logging levels to be more realistic
Makes most logging at debug rather than warn and error.
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/ExecuteStepsUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleService.java
2018-05-09 11:40:27 +01:00
Colin Goodheart-Smithe a03241d590
Merge branch 'master' into index-lifecycle 2018-05-09 10:50:33 +01:00
Alexander Reelsen f00890ee38
Watcher: Increase HttpClient parallel sent requests (#30130)
The HTTPClient used in watcher is based on the apache http client. The
current client is using a lot of defaults - which are not always
optimal. Two of those defaults are the maximum number of total
connections and the maximum number of connections to a single route.

If one of those limits is reached, the HTTPClient waits for a connection
to be finished thus acting in a blocking fashion. In order to prevent
this when many requests are being executed, we increase the limit of
total connections as well as the connections per route (a route is
basically an endpoint, which also contains proxy information, not
containing an URL, just hosts).

On top of that an additional option has been set to evict
long running connections, which can potentially be reused after some
time. As this requires an additional background thread, this required
some changes to ensure that the httpclient is closed properly. Also the
timeout for this can be configured.
2018-05-09 09:37:47 +02:00
Ryan Ernst ce008c446b
Security: Rename IndexLifecycleManager to SecurityIndexManager (#30442)
This commit renames IndexLifecycleManager to SecurityIndexManager as it
is not actually a general purpose class, but specific to security. It
also removes indirection in code calling the lifecycle service, instead
calling the security index manager directly.
2018-05-08 10:03:11 -07:00
Alexander Reelsen 51418fc158
Watcher: Mark watcher as started only after loading watches (#30403)
Starting watcher should wait for the watcher to be started before
marking the status as started, which is now done via a callback.

Also, reloading watcher could set the execution service to paused. This could
lead to watches not being executed, when run in tests. This fix does not
change the paused flag in the execution service, just clears out the
current queue and executions.

Closes #30381
2018-05-08 17:47:01 +01:00
Alexander Reelsen 71ebed6371
Watcher: Mark watcher as started only after loading watches (#30403)
Starting watcher should wait for the watcher to be started before
marking the status as started, which is now done via a callback.

Also, reloading watcher could set the execution service to paused. This could
lead to watches not being executed, when run in tests. This fix does not
change the paused flag in the execution service, just clears out the
current queue and executions.

Closes #30381
2018-05-07 14:19:01 +02:00
Jason Tedor ec71144040 Merge branch 'master' into index-lifecycle
* master: (35 commits)
  DOCS: Correct mapping tags in put-template api
  DOCS: Fix broken link in the put index template api
  Add put index template api to high level rest client (#30400)
  Relax testAckedIndexing to allow document updating
  [Docs] Add snippets for POS stop tags default value
  Move respect accept header on no handler to 6.3.1
  Respect accept header on no handler (#30383)
  [Test] Add analysis-nori plugin to the vagrant tests
  [Docs] Fix bad link
  [Docs] Fix end of section in the korean plugin docs
  Expose the Lucene Korean analyzer module in a plugin (#30397)
  Docs: remove transport_client from CCS role example (#30263)
  [Rollup] Validate timezone in range queries (#30338)
  Use readFully() to read bytes from CipherInputStream (#28515)
  Fix  docs Recently merged #29229 had a doc bug that broke the doc build. This commit fixes.
  Test: remove cluster permission from CCS user (#30262)
  Add Get Settings API support to java high-level rest client (#29229)
  Watcher: Remove unneeded index deletion in tests
  Set the new lucene version for 6.4.0
  [ML][TEST] Clean up jobs in ModelPlotIT
  ...
2018-05-06 18:17:36 -04:00
Zachary Tong 1c0d339904
[Rollup] Validate timezone in range queries (#30338)
When validating the search request, we make sure any date_histogram
aggregations have timezones that match the jobs.  But we didn't
do any such validation on range queries.

While it wouldn't produce incorrect results, it would be confusing
to the user as no documents would match the aggregation (because we
add a filter clause on the timezone for the agg).

Now the user gets an exception up front, and some helpful text about
why the range query didnt match, and which timezones are acceptable
2018-05-04 10:45:16 -07:00
Jim Ferenczi dbd857341f
Upgrade to 7.4.0-snapshot-1ed95c097b (#30357)
Upgrade to lucene-7.4.0-snapshot-1ed95c097b

This version contains:
* An Analyzer for Korean
* An IntervalQuery and IntervalsSource that retrieve minimum intervals of positional queries.
* A new API to retrieve matches (offsets and positions) of a query for a single document.
* Support for soft deletes in the index writer.
* A fixed shingle filter that handles index time synonyms.
* Support for emoji sequence in ICUTokenizer (with an upgrade to icu 61.1)
2018-05-04 11:44:22 +02:00
Alexander Reelsen 0e6cbbd811
Watcher: Ensure trigger service pauses execution (#30363)
When the watcher service pauses execution due to a cluster state update,
the trigger service and its engines also need to pause properly instead
of keeping going. This is also important when the .watches index is 
deleted, so that watches don't stay in a triggered mode.
2018-05-04 09:02:04 +02:00
Jay Modi aa0d7c73f8
Security: reduce garbage during index resolution (#30180)
The IndexAndAliasesResolver resolves the indices and aliases for each
request and also handles local and remote indices. The current
implementation uses the ResolvedIndices class to hold the resolved
indices and aliases. While evaluating the indices and aliases against
the user's permissions, the final value for ResolvedIndices is
constructed. Prior to this change, this was done by creating a
ResolvedIndices for the first set of indices and for each additional
addition, a new ResolvedIndices object is created and merged with
the existing one. With a small number of indices and aliases this does
not pose a large problem; however as the number of indices/aliases
grows more list allocations and array copies are needed resulting in a
large amount of garbage and severely impacted performance.

This change introduces a builder for ResolvedIndices that appends to
mutable lists until the final value has been constructed, which will
ultimately reduce the amount of garbage generated by this code.
2018-05-03 12:48:23 -06:00
Dimitris Athanasiou a1e23feba2
[ML] Add integration test for model plots (#30359)
Relates #30004
2018-05-03 17:02:45 +01:00
Costin Leau 65dbc17510
SQL: Fix bug caused by empty composites (#30343)
When dealing with filtering, a composite aggregation might return empty
buckets (which have been filtered) which gets sent as is to the client.
Unfortunately this interprets the response as no more data instead of
retrying.

This now has changed and the listener keeps retrying until either the
query has ended or data passes the filter.

Fix #30292
2018-05-03 17:13:32 +03:00
Dimitris Athanasiou 3b260dcfc1
[ML] Account for gaps in data counts after job is reopened (#30294)
This commit fixes an issue with the data diagnostics were
empty buckets are not reported even though they should. Once
a job is reopened, the diagnostics do not get initialized from
the current data counts (especially the latest record timestamp).
The result is that if the data that is sent have a time gap compared
to the previous ones, that gap is not accounted for in the empty bucket
count.

This commit fixes that by initializing the diagnostics with the current
data counts.

Closes #30080
2018-05-03 15:08:24 +01:00
Colin Goodheart-Smithe e9dc190479
Throws ElasticsearchException from cluster state tasks if something
really bad happens

If the `onFailure()` method is called on the cluster state task then
something bad happened that we can't really deal with so this change
throws an ElasticsearchException in that case and the step will be
re-executed when the policy is next triggered. If we can't submit a
cluster state task then we can't move to the error state so there isn't
really anything else we can do here. If the cluster state task fails
like this there are probably bigger issues witht he cluster anyway.

x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/ExecuteStepsUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToNextStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/ClusterStateUpdateStepTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/ExecuteStepsUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToNextStepUpdateTaskTests.java
2018-05-03 11:05:37 +01:00
Colin Goodheart-Smithe 61d88a8fdd
Makes all step tests use `randomStepKey()`
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AbstractStepTestCase.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/DeleteStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ForceMergeStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/InitializePolicyContextStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/PhaseAfterStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ReplicasAllocatedStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/RolloverStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/SegmentCountStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrunkShardsAllocatedStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrunkenIndexCheckStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/TerminalPolicyStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/UpdateSettingsStepTests.java
2018-05-03 10:18:42 +01:00
Colin Goodheart-Smithe b5ae7a1aeb
Adds infrastructure for dealing with errors in step execution (#30341)
* Adds infrastructure for dealing with errors in step execution

This change adds a new ErrorStep which a step can move to if it
encounters an error it cannot automatically recover from by retrying on
the next execution. The error step is special in that it cannot
complete. The intention is that the user will need to call an API to man
ually mocve the step in order to progress the index's lifecycle. The
error step retains the phase and action names of the step before it but
with the step name set to `ERROR`. For this reason no ordinary step can
have this name. `AbstractStepTestCase.testStepNameNotError()` ensures
that no step uses `ERROR` as the step name for either its stepKey or
its nextStepKey.

The new `index.lifecycle.failed_step` setting is used
to store the name of the failed step so the user can know in which step
the error occured. More error information will be added shortly.

The async steps will now move to the error step if listener.onFailure()
is called.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/LifecycleSettings.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/ErrorStep.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistry.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/ErrorStepTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToNextStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistryTests.java

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/LifecycleSettings.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AbstractStepTestCase.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStepTests.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistry.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToNextStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistryTests.java

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/LifecycleSettings.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AbstractStepTestCase.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStepTests.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistry.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToNextStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistryTests.java

* Addresses review comments
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStepTests.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
2018-05-03 09:42:54 +01:00
Colin Goodheart-Smithe 6546535a8b
Merge branch 'master' into index-lifecycle 2018-05-03 08:54:15 +01:00
Alexander Reelsen 2c38d12e23
Watcher: Make start/stop cycle more predictable and synchronous (#30118)
The current implementation starts/stops watcher using an executor. This
can result in our of order operations.

This commit reduces those executor calls to an absolute minimum in order
to be able to do state changes within the cluster state listener method,
which runs in sequence.

When a state change occurs that forces the watcher service to pause
(like no watcher index, no master node, no local shards), the service is
now in a paused state.

Pausing is a super lightweight operation, which marks the
ExecutionService as paused and waits for the currently executing watches
to finish in the background via an executor. The same applies for
stopping, the potentially long running operation is outsourced in to an
executor, as waiting for executed watches is decoupled from the current
state.

The only other long running operation is starting, where watches need to
be loaded. This is also done via an executor, but has an additional
protection by checking the cluster state version it was started with. If
another cluster state version was trying to load the watches, then this
loading will not take effect.

This PR also cleans up some unused states, like the a simple boolean in
the HistoryStore/TriggeredWatchStore marking it as started or stopped,
as this can now be caught in the execution service.

Another advantage of this approach is the fact, that now only triggered
watches are not getting executed, while watches that are run via the
Execute Watch API will still be executed regardless if watcher is
stopped or not.

Lastly the TickerScheduleTriggerEngine thread now only starts on data nodes.
2018-05-03 09:47:12 +02:00
Tim Brooks 226b45e509
Update versions for start_trial after backport (#30218)
This commit is a follow up to #30135. It updates the stream
compatibility versions in the start_trial requests and responses to
reflect that fact that this work has been backported to 6.3.
2018-05-02 15:51:59 -06:00
Ioannis Kakavas cca1a2a7cf
Make licensing FIPS-140 compliant (#30251)
Necessary changes so that the licensing functionality can be
used in a JVM in FIPS 140 approved mode.
* Uses adequate salt length in encryption
* Changes key derivation to PBKDF2WithHmacSHA512 from a custom
  approach with SHA512 and manual key stretching
* Removes redundant manual padding

Other relevant changes:
* Uses the SAH512 hash instead of the encrypted key bytes as the
  key fingerprint to be included in the license specification
* Removes the explicit verification check of the encryption key
  as this is implicitly checked in signature verification.
2018-05-02 22:19:43 +03:00
Ryan Ernst fb0aa562a5
Network: Remove http.enabled setting (#29601)
This commit removes the http.enabled setting. While all real nodes (started with bin/elasticsearch) will always have an http binding, there are many tests that rely on the quickness of not actually needing to bind to 2 ports. For this case, the MockHttpTransport.TestPlugin provides a dummy http transport implementation which is used by default in ESIntegTestCase.

closes #12792
2018-05-02 11:42:05 -07:00
Costin Leau 7790cb5fa9
SQL: Reduce number of ranges generated for comparisons (#30267)
* SQL: Reduce number of ranges generated for comparisons

Rewrote optimization rule for combining ranges by improving the
detection of binary comparisons in a tree to better combine
them in a range, regardless of their place inside an expression.
Additionally, improve the comparisons of Numbers of different types
Also, improve reassembly of conjunction/disjunction into balanced
trees.
Do not promote BinaryComparisons to Ranges since it introduces NULL
boundaries and thus a corner-case that needs too much handling
Compare BinaryComparisons directly between themselves and to Ranges

Fix #30017
2018-05-02 19:35:01 +03:00
Colin Goodheart-Smithe 4ad44a76b0
Merge branch 'master' into index-lifecycle 2018-05-02 09:01:08 +01:00
Tim Vernum 66daaaa1cc
Fix message content in users tool (#30293)
The elasticsearch-users utility had various messages that were
outdated or incorrect. This commit updates the output from this
command to reflect current terminology and configuration.
2018-05-02 11:04:28 +10:00
Michael Basnight 62a9b8909e
Remove RepositoriesMetaData variadic constructor (#29569)
The variadic constructor was only used in a few places and the
RepositoriesMetaData class is backed by a List anyway, so just using a
List will make it simpler to instantiate it.
2018-05-01 15:02:06 -05:00
Nik Everett abe797bd42
SQL: Teach the CLI to ignore empty commands (#30265)
Cause the CLI to ignore commands that are empty or consist only of
newlines. This is a fairly standard thing for SQL CLIs to do.

It looks like:
```
sql> ;
sql>
   |
   | ;
sql> exit;
Bye!
```

I think I *could* have implemented this with a `CliCommand` that throws
out empty string but it felt simpler to bake it in to the `CliRepl`.

Closes #30000
2018-05-01 13:17:24 -04:00
Nik Everett 5e9e6fed90
HTML5ify Javadoc for xpack core (#30277)
xpack core contains a fork of `Cron` from quartz who's javadoc has a
`<table>` with non-html5 compatible stuff. This html5ifies the table and
switches the `:x-pack:plugin:core` project to building javadoc with
HTML5.
2018-05-01 08:32:58 -04:00
Colin Goodheart-Smithe 42711dd46d
Adds single node allocation to shrink (#30206)
* Adds ClusterState to AsyncActionStep.performAction

This is needed so a new step can be created for the shrink action which
can select a node to allocate to based on the current routing rules and
the node attributes on teh discovery nodes.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AsyncActionStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/DeleteStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ForceMergeStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/RolloverStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/UpdateSettingsStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/DeleteStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ForceMergeStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/RolloverStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/UpdateSettingsStepTests.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleService.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java

* Adds single node allocation to shrink

This change adds two new steps as the first steps of the shrink action:

1. A `SetSingleNodeAllocateStep` which:
    1. Determines which of the active nodes match the existing index
allocation rules
    2. Randomly (using Randomness so its deterministic for testing)
picks one of the matching nodes
    3. Updates the index settings to add a require allocation rule for
the node that was picked (using the
`index.routing.allocation.require._name` setting)
2. An `AllocationRoutedStep` which ensures that at least one copy of
each shard is allocated according to the new allocation rules

Note that this change also modifies the `AllocationRoutedStep` to add a
boolean field which determines whether the allocation is complete when
at least one copy of each shard matches the allocation rulees or if it
needs to wait for all shard copies to be allocated according to the
rules.

Lastly, a `randomStepKey()` method is added to `AbstractStepTestCase`
for convenience.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocateAction.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkAction.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/action/admin/indices/
settings/put/UpdateSettingsTestHelper.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AbstractStepTestCase.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkActionTests.java

* Fixes AllocationRoutedStep when `waitForAllShardCopies=false`

This change fixes `AllocationRoutedStep` so that when
`waitForAllShardCopies=false` we wait for any shard copy of each shard
to be allocated according to the allocation rules rather than
specifically the primary of each shard.
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStepTests.java

* Corrects Licence headers and typo
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStepTests.java
2018-05-01 10:27:51 +01:00
Dimitris Athanasiou 057cdffed5
[ML] Refactor DataStreamDiagnostics to use array (#30129)
This commit refactors the DataStreamDiagnostics class
achieving the following advantages:

- simpler code; by encapsulating the moving bucket histogram
into its own class
- better performance; by using an array to store the buckets
instead of a map
- explicit handling of gap buckets; in preparation of fixing #30080
2018-05-01 09:50:32 +01:00
Colin Goodheart-Smithe a6b0f663af
Merge branch 'master' into index-lifecycle 2018-05-01 09:21:57 +01:00
Boaz Leskes 4a537ef03c
Bulk operation fail to replicate operations when a mapping update times out (#30244)
Starting with the refactoring in https://github.com/elastic/elasticsearch/pull/22778 (released in 5.3) we may fail to properly replicate operation when a mapping update on master fails. If a bulk
operations needs a mapping update half way, it will send a request to the master before continuing 
to index the operations. If that request times out or isn't acked (i.e., even one node in the cluster 
didn't process it within 30s), we end up throwing the exception and aborting the entire bulk. This is 
a problem because all operations that were processed so far are not replicated any more to the 
replicas.  Although these operations were never "acked" to the user (we threw an error) it cause the 
local checkpoint on the replicas to lag (on 6.x) and the primary and replica to diverge. 

This PR does a couple of things:
1) Most importantly, treat *any* mapping update failure as a document level failure, meaning only 
    the relevant indexing operation will fail.
2) Removes the mapping update callbacks from `IndexShard.applyIndexOperationOnPrimary` and 
    similar methods for simpler execution. We don't use exceptions any more when a mapping 
    update was successful.

I think we need to do more work here (the fact that a single slow node can prevent those mappings 
updates from being acked and thus fail operations is bad), but I want to keep this as small as I can 
(it is already too big).
2018-05-01 08:15:02 +02:00
David Roberts 225f7093a9
[ML] Include 3rd party C++ component notices (#30132)
The overall NOTICE file for the ML X-Pack module should
include the notices from the 3rd party C++ components as
well as the 3rd party Java components.
2018-04-30 20:05:27 +01:00
Chris Earle 725a5af2c6
_cluster/state should always return cluster_uuid (#30143)
Currently, the only way to get the REST response for the `/_cluster/state`
call to return the `cluster_uuid` is to request the `metadata` metrics,
which is one of the most expensive response structures. However, external
monitoring agents will likely want the `cluster_uuid` to correlate the
response with other API responses whether or not they want cluster
metadata.
2018-04-30 10:16:11 -04:00
Colin Goodheart-Smithe d7d709a5dc
Merge branch 'master' into index-lifecycle 2018-04-30 09:29:48 +01:00
Zachary Tong fee000a37f
[TEST] Redirect links to new locations (#30179)
We had a number of awaitsFix links that weren't updated after the xpack
merge.

Where possible I changed the links to the new locations, but in some
circumstances the original ticket was closed (suggesting the awaitsfix
should be removed) or was otherwise unclear the status.
2018-04-27 09:24:46 -07:00
Tal Levy f6a425e676 fix line length in RolloverStep 2018-04-27 07:48:17 -07:00
Colin Goodheart-Smithe ec9f1a9f3b
Merge branch 'master' into index-lifecycle 2018-04-27 15:32:02 +01:00
Alexander Reelsen 707ba28d48
Watcher: Ensure mail message ids are unique per watch action (#30112)
Email message IDs are supposed to be unique. In order to guarantee this,
we need to take the action id of a watch action into account as well,
not just the watch id from the watch execution context. This prevents
that two actions from the same watch execution end up with the same
message id.
2018-04-27 08:55:25 +02:00
Costin Leau e0b8893645
SQL: Correct error message (#30138)
* SQL: Correct error message

Error messages had placeholders that were not replaced; this PR fixes
that

Fix #30016
2018-04-27 09:24:25 +03:00
Tim Brooks 592481e4ed
Require acknowledgement to start_trial license (#30135)
This is related to #30134. It modifies the start_trial action to require
an acknowledgement parameter in the rest request to actually start the
trial license. There are backwards compatibility issues as prior ES
versions did not support this parameter. To handle this, it is assumed
that a request coming from a node prior to 6.3 is acknowledged. And
attempts to write a non-acknowledged request to a prior to 6.3 node will
throw an exception.

Additionally this PR adds messages about the trial license the user is
generating.
2018-04-26 21:42:44 -04:00
Costin Leau 804c38303f
SQL: Add BinaryMathProcessor to named writeables list (#30127)
BinaryMathProcessor was missing from the list of register  named
writeables causing deserialization errors

Fix #30014
2018-04-27 01:43:44 +03:00
Tal Levy 053e9fb234 Merge branch 'master' into index-lifecycle 2018-04-26 15:15:45 -07:00
Christoph Büscher d0416c76fe
Fix edge cases in CompositeKeyExtractorTests (#30175)
Currently the test picks random java.util.TimeZone ids in some places.
Internally we still need to convert back to joda DateTimeZone by id
occassionally (e.g. when serializing to pre 6.3 versions). There are
some deprecated "SystemV/*" time zones that Jodas DateTimeZone refuses
to convert. This change excludes those rare cases from the set of
allowed random time zones. It would be quiet odd for them to appear in
practice.

Closes #30156
2018-04-26 20:06:47 +02:00
Michael Basnight cb7e3ffd75
Remove licenses missed by the migration (#30128)
A few of the old style license got kept around because their comment
string did not start with a space. This caused the license check to not
see it as a license and skip it. This commit cleans it up.
2018-04-26 11:37:04 -05:00
Tal Levy c10e624c92
update RolloverAction to utilize an index-setting for rollover_alias (#30158)
The purpose of this change is to reflect what is the future way of configuring aliases in
the Rollover action for ILM. With this change, users will be able to edit all desired rollover-related
information in the index's template, so that the index-lifecycle policy does not need to be dependent on it.
2018-04-26 08:18:23 -07:00
Tal Levy 26b42dfbf3 fix typo in xpack-rest test plugin->module 2018-04-26 08:12:02 -07:00
Nik Everett a8f40b3e04 Build: Assert jar LICENSE and NOTICE files match
Adds tasks that check that the all jars that we build have LICENSE.txt
and NOTICE.txt files and that the files are correct. Sets check to
depend on these task.

This is mostly there for extra parnoia because we automatically
configure all Jar tasks to include the LICENSE.txt and NOTICE.txt
files anyway. But it is quite possible to add configuration to those
tasks that would override either file.

This causes check to depend on several more things than it used to.
Take, for example, javadoc:

check depends on the new verifyJavadocJarNotice which depends on
extractJavadocJar which depends on javadocJar which depends on
javadoc, this check now depends on javadoc.
2018-04-25 19:53:24 -04:00
Tal Levy 1df6b6c65d Applying changes required for ILM after merge 2018-04-25 13:04:06 -07:00
Tal Levy d74fb9eb45 Opened x-pack ILM code 2018-04-25 12:22:53 -07:00
Colin Goodheart-Smithe c02b895653
Fixes Eclipse build for sql jdbc project (#30114)
The bundled configuration isn't recognised by eclipse so these
dependencies are missed when it imports the `x-pack:plugin:sql:jdbc`
project. This change makes these dependencies compile dependencies if
the build is running for Eclipse.
2018-04-25 16:10:32 +01:00
Michael Basnight 72f57c8e72
Add comments inadvertently removed during migrate
A few files had their first comment removed even though it did not
contain a license. This re-adds those comments.
2018-04-24 13:41:09 -05:00
David Kyle cfc66a1fd5 [ML] Wait for updates to established memory usage
Tests need to wait for changes to the job's established memory usage to
propagate and an over enthusiastic optimisation meant jobs were updated
from stale state causing recent change to be lost.
2018-04-24 13:46:58 -04:00
Jason Tedor ab101976d6 Fix SQL CLI on Windows
This commit fixes the classpath for the SQL CLI tool on Windows. As the
x-pack bin folder was collapsed into the distribution bin folder, the
location of the classpath here needed to no longer contain the old
plugins directory.
2018-04-20 15:34:01 -07:00
Jason Tedor d99d0fa669 Add distribution type to startup scripts
This commit adds the distribution type to the startup scripts so that we
can discern from log output and the main response the type of the
distribution (deb/rpm/tar/zip).
2018-04-20 15:34:01 -07:00
Jason Tedor c7f6b35485 Fix classpath for X-Pack scripts on Windows
With the move of X-Pack to a module, the classpath for the scripts needs
to be adjusted. This was done on Unix, but not for Windows. This commit
addresses Windows.
2018-04-20 15:34:01 -07:00
Ryan Ernst d8636d0fe3 Rename syskeygen
This commit renames syskeygen to elasticsearch-syskeygen
2018-04-20 15:34:01 -07:00
Jason Tedor c7c0e330b8 Rename users
This commit renames users to elasticsearch-users.
2018-04-20 15:34:01 -07:00
Jason Tedor b05c8bf781 Rename migrate
This commit renames migrate to elasticsearch-migrate.
2018-04-20 15:34:00 -07:00
Jason Tedor 2d3fac307a Rename saml-metadata
This commit renames saml-metadata to elasticsearch-saml-metadata.
2018-04-20 15:34:00 -07:00
Jason Tedor 8a8a1dfc2e Rename sql-cli
This commit renames sql-cli to elasticsearch-sql-cli.
2018-04-20 15:34:00 -07:00
Ryan Ernst 6dbb7be067 Rename croneval
This commit renames croneval to elasticsearch-croneval
2018-04-20 15:34:00 -07:00
Jason Tedor 7ba2defdbf Rename setup-passwords
This commit renames setup-passwords to elasticsearch-setup-passwords.
2018-04-20 15:34:00 -07:00
Jason Tedor 03ffd16921 Rename certutil
This commit renames certutil to elasticsearch-certutil.
2018-04-20 15:33:59 -07:00
Jason Tedor a6b4682711 Rename certgen
This commit renames certgen to elasticsearch-certgen.
2018-04-20 15:33:59 -07:00
Jason Tedor e64e6d8996 Add distribution flavor to startup scripts
This commit adds the distribution flavor (default versus oss) to the
build process which is passed through the startup scripts to
Elasticsearch. This change will be used to customize the message on
attempting to install/remove x-pack based on the distribution flavor.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst 2efd22454a Migrate x-pack-elasticsearch source to elasticsearch 2018-04-20 15:29:54 -07:00
Tal Levy 689d628d67 move phase-after steps to have the previous phase's phase in its key (#4387)
Previously, phase X's `after` step had `X` as its associated phase. This causes confusion because
we have only entered phase `X` once the `after` step is complete. Therefore, this refactor
pushes the after's phase to be associated with the previous phase. This first phase is an exception. 
The first phase's `after` step is associated with the first phase (not some non-existent prior phase).
2018-04-16 21:19:30 -07:00
Colin Goodheart-Smithe d509834c00 Fixes checkstyle error 2018-04-13 14:39:37 +01:00
Colin Goodheart-Smithe af6456983f Moves registing of NamedWriteables to plugin:core
This is needed so that clients have the ability to serialise and deserialise the objects it creates and gets back.
2018-04-13 14:23:54 +01:00
Colin Goodheart-Smithe bf3db3f1f7 Adds phase and action change times on state transition 2018-04-13 13:38:44 +01:00
Colin Goodheart-Smithe d17886f1b0 Fixes checkstyle error 2018-04-13 10:44:45 +01:00
Tal Levy f04c95f4c2 change policy logs to be DEBUG 2018-04-12 15:40:20 -07:00
Colin Goodheart-Smithe 644d82e165 Renames AliasStep to ShrinkSetAliasStep 2018-04-12 18:29:12 +01:00
Colin Goodheart-Smithe 77be66b8b9 Adds shrink check that original index is deleted 2018-04-12 18:13:57 +01:00
Colin Goodheart-Smithe 59e0979b92 Fixes shrunk shards allocation check step
- Renames EnoughShardsWaitStep to ShrunkShardsAllocatedStep
- Changes ShrunkShardsAllocatedStep to check the shards of the shrunken index rather than the current one
- shrink index prefix is now passed into the steps of the shrink aciton
- Related Test Changes
2018-04-12 18:06:04 +01:00
Tal Levy 34e15011a1 fix ilm rest-tests to reflect latest get_policy changes 2018-04-12 07:47:57 -07:00
Colin Goodheart-Smithe 088894019e Tidies up unit tests
Also fixes failing EnoughShardsWaitStep
2018-04-12 12:37:36 +01:00
Colin Goodheart-Smithe 1c2e7bc431 Increases iterations on step equality tests
To make it in line with the fact that other equality tests use multiple iterations
2018-04-12 12:23:35 +01:00
Colin Goodheart-Smithe e8cb6419f8 Removes unused imports 2018-04-12 12:16:43 +01:00
Colin Goodheart-Smithe 7b247fe41e Fixes compile errors from TimeValue serialisation 2018-04-12 12:15:36 +01:00
Tal Levy 20b6aaf6b6 migrate step tests to extend AbstractStepTestCase 2018-04-11 18:56:53 -07:00
Tal Levy c1b00d0154 cleanup tests
- make DeleteActionTests pass
- remove getType from LifecycleType
2018-04-11 18:28:46 -07:00
Tal Levy 7f4801f03c clean up PhaseTests 2018-04-11 18:16:28 -07:00
Tal Levy 73ee5a4aa0 introduce refactored ShrinkAction
this resulted in a few new steps being introduced

- AliasStep
- EnoughShardsWaitStep
- ShrinkStep
- ShrunkenIndexCheckStep
2018-04-11 18:10:13 -07:00
Colin Goodheart-Smithe 20485cf7fb Adds more missing equality tests
Specifically for the initialPolicyContextStep and the PhaseAfterStep
2018-04-11 13:24:31 +01:00
Colin Goodheart-Smithe 482de191f2 Tidies up some of the unit tests
Adds some missing tests including checking the hashcode and equals methods of `DeleteStep`, `StepKey`, and `TerminalPolicyStep` as well as adding a test for `DeleteAction.toSteps()`
2018-04-11 12:38:55 +01:00
Colin Goodheart-Smithe 66b4e8c4aa Removes unused imports 2018-04-11 10:47:07 +01:00
Tal Levy ef34f982f1 refactor a collection of step-related things + shrink_step
- added ShrinkStep/Tests
- AsyncActionStep now passes in IndexMetaData instead of Index
- Delete usage of ClusterStateActionStep
- with ClusterStateActionStep gone, InitializePolicyContextStep
  is the only other ClusterState-nonWait step
- Migrate setting-updates to UpdateSettingsStep
2018-04-10 17:13:10 -07:00
Colin Goodheart-Smithe 8d91f197d4 Changes update replica and alllocation to async steps
Also renames EnoughShardsWaitStep to ReplicasAllocatedStep, removes it from the allocate action and adds a check that th number of replicas in the cluster state is correct to it.
2018-04-10 16:20:29 +01:00
Colin Goodheart-Smithe aa198b637e Cleans up warnings 2018-04-10 09:53:57 +01:00
Colin Goodheart-Smithe c24dc07346 Removes unused and unneeded code
Various classes had some code that was not used and is not going to be needed so this change cleans up those classes so we don’t have dead code hanging around
2018-04-10 09:28:03 +01:00
Colin Goodheart-Smithe b8d573772f Fixes compilation issue in test 2018-04-10 09:10:52 +01:00
Tal Levy f5b23df349 add more force-merge related steps and tests
The force-merge is an a TODO state due to the
unresolved issue around best_compression.

- updated ReadOnlyStep with tests
- implemented an update to the ForceMergeAction
- added UpdateBestCompressionSettingsStep
- added tests for SegmentCountStep
2018-04-09 16:28:25 -07:00
Colin Goodheart-Smithe 06e969b430 Adds rollover step and implements RolloverAction.toSteps 2018-04-09 15:55:06 +01:00
Colin Goodheart-Smithe 54c6d280a5 Adds Steps for ReplicasAction
Also adds tests for those steps and ReplicasAction.toSteps
2018-04-09 14:43:57 +01:00
Colin Goodheart-Smithe 2eb8fd9336 Adds tests for AllocateAction and its steps
Also tweaks some of the code in the steps of the allocate action
2018-04-09 13:59:25 +01:00
Tal Levy 47ed2e5b23 add more tests 2018-04-05 18:00:41 -07:00
Tal Levy b58159ddad add more tests 2018-04-05 17:43:32 -07:00
Tal Levy 228fc8c842 add some stuff 2018-04-05 17:21:44 -07:00
Tal Levy c0acea3721 :check yourself greeny 2018-04-05 13:16:19 -07:00
Tal Levy 1a2afac98b fix tests 2018-04-05 12:43:10 -07:00
Colin Goodheart-Smithe e3e669067c Adds loggic for TerminalPolicyStep 2018-04-05 17:58:22 +01:00
Colin Goodheart-Smithe 05c2c5655b replaces cause with boolean 2018-04-05 17:53:15 +01:00
Colin Goodheart-Smithe ed260a0430 Removes Cause stuff 2018-04-05 17:10:56 +01:00
Tal Levy 907586c9e4 add MoveToNextStepUpdateTaskTests 2018-04-04 20:39:00 -07:00
Tal Levy 55d2a62fec add tests for ExecuteStepsUpdateTask 2018-04-04 15:32:36 -07:00
Colin Goodheart-Smithe a8dc8a7c98 Adds more unit tests for IndexLifecycleRunner 2018-04-04 15:59:10 +01:00
Colin Goodheart-Smithe 9ef26dbe51 Unit tests static methods in IndeexLifecycleRunner 2018-04-04 13:29:30 +01:00
Tal Levy 47dcc8fe9c more changes + updated api 2018-04-03 17:42:25 -07:00
Tal Levy 1ac1ee413f moar changes 2018-04-02 16:21:48 -07:00
Tal Levy d2e87a66e5 moar changes
- set `indexSurvives` for Step
- extract the two ClusterStateUpdateTasks to separate classes

- simple Delete Policy works!
2018-03-29 16:23:57 -07:00
Tal Levy 74eaba2078 moar changes from Tal 2018-03-28 17:10:50 -07:00
Colin Goodheart-Smithe 179074d3b0 Refactors steps and adds some execution logic
This is far from complete but its necessary to push so the new step classes can be shared and used.
2018-03-28 16:34:34 +01:00
Tal Levy e6ee5b49d1 have steps point to step-key, not the actual step object 2018-03-27 11:52:08 -07:00
Tal Levy ce4248ec59 slight changes 2018-03-27 11:41:38 -07:00
Colin Goodheart-Smithe 9972710e9e Makes brnach compile
Commented out toSteps implem,entations and other bits needed to get the branch to compile
2018-03-27 12:14:38 +01:00
Tal Levy f429fc0b3e begin making sense of types 2018-03-26 18:48:59 -07:00
Tal Levy 083c563cf6 meh 2018-03-26 18:12:38 -07:00
Tal Levy 57821cd55a moar refactor for steps 2018-03-26 17:54:15 -07:00
Tal Levy d63cd8c9c3 step by step 2018-03-26 08:00:03 -07:00
Tal Levy fe22c08207 stop using illegal logging of Object 2018-03-12 13:18:02 -07:00
Tal Levy ebff02c1b2 fix compilation changes 2018-03-12 12:29:22 -07:00
Tal Levy 9850f44b95 fix state left over by spec test 2018-02-13 13:40:09 -08:00
Tal Levy 8e282f21c8 Merge branch 'master' into index-lifecycle 2018-02-12 14:17:24 -08:00
Tal Levy 0a3dcebca0 disallow deleting lifecycle policies that are referenced by existing indices (#3806) 2018-02-01 14:16:06 -08:00
Tal Levy ab49e40f91 add phase and action timestamps to cluster-state (#3726)
This commit sets `index.lifecycle.phase_time` and `index.lifecycle.action_time`
appropriately during setPhase and setAction in IndexLifecycleContext.
2018-01-25 09:02:26 -08:00
Colin Goodheart-Smithe faf067080b Makes sure creation date garentees execution in tests 2018-01-25 13:21:50 +00:00
Tal Levy 9920d6be2d keep plugin up to date with structure changes (#3702)
Looks like we need to split out the tests of core classes to core
and index-lifecycle ones stay in index-lifecycle.

I believe I got everything, although I may have missed at least one thing

checked status with

$ ./gradlew :x-pack-elasticsearch:plugin:index-lifecycle:check -Dtests.seed=39838421912001B4
$ ./gradlew :x-pack-elasticsearch:plugin:core:check -Dtests.seed=39838421912001B4

other things done in this PR:
- removal of a few unused variables/thrown exceptions/imports
- fix TimeseriesLifecycleTypeTests
  - an all null AllocateAction was created
- fix AllocateActionTests
  - woops. -Dtests.seed=39838421912001B4 resulted in two `null`s and an emptyMap.
    this resulted in a test failure.
2018-01-24 08:47:21 +00:00
Tal Levy 4856baf66b fix guice and plugin initialization after split (#3677)
the x-pack-split required updating the IndexLifecycle plugin to
be a proper plugin with the Plugin API methods instead of the XPackPlugin API
methods.
2018-01-23 10:41:22 -08:00
Colin Goodheart-Smithe fd502aa3e6 Makes some fields in Index Lifecycle API optional (#3687)
Specifically this change makes it optional to:
* Specify `includes`, `excludes` and `requires`maps in the allocate action as long as at least one fo the options is specified and is not an empty map
* Specify an `after` parameter on a phase. If no `after` value is specified `TimeValue.ZERO` is used and the phase will be moved to as soon as the previous phase reports `ACTIONS COMPLETED`. `after` is always non-null when we are serialising the Phase.
* Specify a `type` for a LifecyclePolicy. If no `type` is specified `TimeSeriesLifecycleType.INSTANCE` is used since this is currently the only production `type`. `type` is always non-null when we are serialising the LifecyclePolicy.
2018-01-23 17:02:11 +00:00
Colin Goodheart-Smithe f2fa988f2f Fixes Checkstyle 2018-01-23 08:24:03 +00:00
Tal Levy 07b28fd721 migrate lifecycle settings to plugin:core:LifecycleSettings (#3670) 2018-01-22 10:43:16 -08:00
Tal Levy 34d6b61209 add `index.lifecycle.date` setting (#3545)
This PR adds a new setting called `index.lifecycle.date` that 
the ShrinkAction will be responsible for populating in the newly created index.

This way, we can continue to know when we should be executing the next phase
relative to the original index creation date, and not that of the shrunken index.
2018-01-22 09:27:33 -08:00
Colin Goodheart-Smithe 9032d61516 Fixes silly issues created by me 2018-01-22 16:48:32 +00:00
Colin Goodheart-Smithe 2d721136c2 Adds logic to cancel job if master is un-elected (#3613)
A node can stop being the master node whilst it is running, e.g. if it can’t access `minimumMasterNodes` number of master eligible nodes. Because of this we need logic in `IndexLifecycleService` that cancels the scheduled job if the node is no longer master and re-adds the job if the node becomes master again.
2018-01-22 15:45:37 +00:00
Colin Goodheart-Smithe ee774594e2 Adds test for TimeValueSchedule 2018-01-22 15:21:55 +00:00
Colin Goodheart-Smithe a6f6335e86 Moves Index-Lifecycle classes over to new module
These changes are necessary following the x-pack split.
2018-01-22 13:52:22 +00:00
Colin Goodheart-Smithe 3cdf1216cd Removes un-needed licence files 2018-01-19 10:27:57 +00:00
Colin Goodheart-Smithe cb4cbf80ac Catches more that just ElasticsearchException 2018-01-18 05:42:22 +00:00
Tal Levy 84fd234835 Add the Shrink Action to Index Lifecycle Management (#3491)
This does the following in sequential service polls

1. sets the index to read-only and runs shrink with a modified `index.lifecycle.name` setting set to `null`.
2. checks to see if shrink is complete, if it is...
    b. set target index's `index.lifecycle.*` settings to the original index's values.
3. if not complete, just wait till next iteration
4. if operating on shrunken index, delete old index and add it as an alias to shrunken index
2018-01-17 11:22:06 -08:00
Colin Goodheart-Smithe ff883d574b Adds Allocate lifecycle action (#3484)
* Adds Allocate lifcycle action

* Addresses review comments

Still need to make a change in core for the FilterAllocationDecider to make the execute logic simpler

* Addresses more review comments

* Adds randomMap method to AllocateActionTests

* Addresses further review comments
2018-01-17 13:27:40 +00:00
Tal Levy db3a88f487 fix TimeseriesLifecycle ordering behavior 2018-01-16 18:08:05 -08:00
Colin Goodheart-Smithe 971685654d Adds more tests in IndexLifecycleServiceTests 2018-01-15 15:29:07 +00:00
Colin Goodheart-Smithe 1e6ed023ee Fixes ReplicasActionTests
Fixes a test failure and adds a new test that was missing
2018-01-15 09:37:00 +00:00
Colin Goodheart-Smithe 53c3d040f7 Add ReplicaAction for Index Lifecycle (#3535)
* Add ReplicaAction for Index Lifecycle

* Add validation
2018-01-12 13:48:09 +00:00
Colin Goodheart-Smithe 5eb912539d Fixes Logging usage
log line had an extra parameter
2018-01-12 12:17:14 +00:00
Colin Goodheart-Smithe 6a370251fc Improves handling of exceptions in Index Lifecycle (#3511)
* Improves handling of exceptions in Index Lifecycle

This change improves a few different aspects:

* If an exception occurs executing the lifecycle of one index it is caught, logged and other indexes are still processed
* If the lifecycle policy specified in the settings does not exist an error is logged
* Fixes the exception when the delete action is run which occurs because Phase attempts to update the phase and action settings for the deleted index. A `LifecycleAction.indexSurvives()` method is introduced which defaults to `true` but can be overridden to indicate whether the index survives following completion of the action.

* Adds test
2018-01-12 11:52:56 +00:00
Tal Levy e3a6e738ea Fix InternalIndexLifecycleContext to update state in memory (#3505)
* Fix InternalIndexLifecycleContext to update state in memory

The internal and the mock index-lifecycle-context implementations differed
in that the InternalIndexLifecycleContext assumed no one would be using it after
it mutated state. This is not the case. We assume that the current context is updated after
a `#setAction` is called so that the listener can then appropriately use the newly modified
cluster state. since idxMeta was not being updated, any call to `context.getAction` was stale and
either returning null or the previous action, not the next action that was updated by `#setAction`.
Same goes for `setPhase`.

This PR should fix this so that the Mock and Internal implementations are more in line.
2018-01-10 08:21:17 -08:00
Colin Goodheart-Smithe 57c343bc21 Use the XContentRegistry provided by the parser 2018-01-08 14:14:43 +00:00
Colin Goodheart-Smithe c4621d9dc7 Mutes IndexLifecycleInitialisationIT.testMasterFailover
Due to a bug with unassigned shards in force merge action
2018-01-05 12:43:13 +00:00
Colin Goodheart-Smithe 148688a991 Reduces poll time to 1s for integ test
`IndexLifecycleInitialisationIT.testMasterFailover()` intermittently failed because the timeout of 10 seconds to check if the index had been deleted was not long enough sometimes with the poll interval set to 3 seconds. This change sets the poll interval to 1 seconds for the test so that the lifecycle is more responsive. This also means the default value for the poll interval can be safely changed without affecting the test.
2018-01-03 12:29:19 +00:00
Colin Goodheart-Smithe 52e416d0bc Creates index-lifecycle gradle module
This doesn’t currently contain any code but is there in preparation for the x-pack-elasticsearch:plugin split
2018-01-03 12:03:54 +00:00
Colin Goodheart-Smithe 043c8db84d Fixes Tests so build passes 2018-01-03 12:01:12 +00:00
Colin Goodheart-Smithe a37df9092e Moves Actions and related to plugin:core
This change moves the Action classes and referenced data model classes to the new :x-pack-elasticsearch:plugin:core project in preparation for splitting the x-pack features into their own gradle modules.

Note that the TransportAction classes had to be promoted to their own class file (rather than being inner classes to their Action) so they can remain in the plugin project (and will late be move to the `index-lifecycle` project when its created.
2018-01-03 10:45:33 +00:00
Colin Goodheart-Smithe 33afe0f012 Merge branch 'master' into index-lifecycle 2018-01-03 10:21:50 +00:00
Tal Levy 6a47b4fa6e fix infinite loop in test and correct logic in timeseries type 2017-12-21 13:14:37 -08:00
Tal Levy 66af56320e fix tests 2017-12-21 12:56:56 -08:00
Tal Levy e433b1eaa8 Adds ForceMerge action to Index Lifecycle (#3309)
add forcemerge index-lifecycle action
2017-12-21 11:59:44 -08:00
Colin Goodheart-Smithe d5609a408b fixes integ test when running as part of the build 2017-12-21 12:38:34 +00:00
Colin Goodheart-Smithe 3627feded7 Refactors lifecycle type to its own class (#3381)
To clean up the parsing of the LifecyclePolicy this change moves the LifecycleType to its own class so it can be created in the normal parsing of LifecyclePolicy rather than having to parse to an intermediary object first. The LifecycleType is an interface which can be implemented for different lifecycle types. These types shiould be singletons and are register with the NamedXContentRegistry and NamedWriteableRegistry only so they are available when reading from a stream or parsing.
2017-12-21 11:20:49 +00:00
Tal Levy fb59f54637 make the lifecycle poll-interval configurable (#3319)
Removes the poll-interval from the IndexLifecycleMetadata and introduces it in 
the form of a cluster setting that is configurable. Changes to this poll interval setting 
will reflect in the Lifecycle Scheduler.
2017-12-20 09:39:39 -08:00
Colin Goodheart-Smithe 586065fca7 Changes to enable the build to run
This includes changing NOCOMMIT comments to be NORELEASE comments so the build passes with them. We have tasks inGH for all these NORELEASE comments so they should be caught before merging to master
2017-12-20 12:27:26 +00:00
Colin Goodheart-Smithe b12f1a9526 More phase tests
This time for when setting the new action fails
2017-12-15 11:20:59 +00:00
Colin Goodheart-Smithe 33a1a92e3d As phase test for when the action execution fails 2017-12-15 11:07:49 +00:00
Colin Goodheart-Smithe 9e693b544c Test improvements 2017-12-15 10:52:37 +00:00
Colin Goodheart-Smithe 3da42f5603 Adds rollover action to index lifecycle plugin (#3266)
This action will rollover an index when executed if the provided conditions are met.
Users may specify the maximum age, maximum index size in bytes or maximum index size in number of documents as conditions for rollover.

When the action executes it firsts checks the local cluster state to find out if the alias exists on the index. If the alias does not exist then the index was either rolled over by a previous run or something else has rolled over the index so the action can be marked as completed. If the index still has the alias set the action will make a rollover index request using the Client. When that request returns and the listener is called the action will only be marked as complete if the response indicates the index was rolled over. If the index was not rolled over (because the conditions are not yet met) the action is not marked as complete and will be re-evaluated on the next call to execute.
2017-12-15 09:36:39 +00:00
Tal Levy 8d6e53356b enable rest of await-fix index-lifecycle integ tests. all green 2017-12-13 13:40:58 -08:00
Tal Levy c0c7e02de4 fix javadoc usage 2017-12-11 16:24:44 -08:00
Tal Levy 288515ddc5 initial attempt towards an ordered execution plan for timeseries phases (#3248)
* initial attempt towards an ordered execution plan for timeseries phases

* respond to feedback and fix bug
2017-12-07 11:59:31 -08:00
Tal Levy ae7e4882e4 general indexlifecycle cleanup, rest tests, reintroduce integ-test (#3243)
* cleanup, rest tests, reintroduce integ-test

- cleaned up some long lines
- re-introduced one of the integration tests
- added rest-spec tests
2017-12-06 07:51:06 -08:00
Tal Levy d847355b58 add timeseries policy action stubs (#3214) 2017-12-05 09:09:18 -08:00
Tal Levy 685231e786 Introduce specific types of index lifecycle policies (#3157)
Introduce specific types of index lifecycle policies
2017-11-30 10:09:41 -08:00
Colin Goodheart-Smithe 922060d766 Typo 2017-11-27 15:46:47 +00:00
Colin Goodheart-Smithe 7199e2fa5c Small refactoring 2017-11-27 12:32:55 +00:00
Colin Goodheart-Smithe 87766cd4ef Adds JavaDocs and comments and small refactoring 2017-11-27 12:22:10 +00:00
Colin Goodheart-Smithe 9ab2121fcf When actions complete the next action is now run 2017-11-27 11:07:26 +00:00
Colin Goodheart-Smithe 152d5d45e5 Adds listeners to Lifecycle actions
This means that the result of the action can now be async and we can then implement moving immediately to the next action if the current one is complete
2017-11-24 19:27:50 +00:00
Colin Goodheart-Smithe 26ad2142d5 minor tweak 2017-11-24 17:29:38 +00:00
Colin Goodheart-Smithe 9e891de777 refactors code to allow better testing
The client and index metadata have now been abstracted away from the Lifecycle classes behind IndexLifecycleContext. This allow us to test the state machine without having to worry about how the state is persisted and read. It also makes the classes much easier to read and reason about.
2017-11-24 17:22:51 +00:00
Colin Goodheart-Smithe e113189347 only creates initialisation service ifeature is enabled 2017-11-24 10:23:04 +00:00
Colin Goodheart-Smithe 06958949ab Uses LongSupplier for nowSupplier
Rather than Supplier<Long>. This is in line with the `nowSupplier`s used in other areas of Elasticsearch
2017-11-24 10:18:06 +00:00
Colin Goodheart-Smithe 31ded906d0 Adds settings checking to tests
This isn’t an ideal solution so might well be temporary but at least the tests are checking the settings changes are correct now
2017-11-24 10:10:46 +00:00
Colin Goodheart-Smithe 016ba2a57d Adds another test
This test needs more work
2017-11-24 09:40:32 +00:00
Colin Goodheart-Smithe 49b2aaa878 Adds more tests 2017-11-23 18:16:47 +00:00
Colin Goodheart-Smithe 75704510e1 More test changes 2017-11-23 15:48:37 +00:00
Colin Goodheart-Smithe a9703d4a56 Mutes failing integration test for now 2017-11-23 14:12:31 +00:00
Colin Goodheart-Smithe 0e829843fc More unit tests 2017-11-23 14:11:09 +00:00
Colin Goodheart-Smithe db502bef6d Adds some unit tests 2017-11-23 12:44:12 +00:00
Colin Goodheart-Smithe f571dc3000 Remove references to InternalClient which no longer exists 2017-11-23 10:35:25 +00:00
Colin Goodheart-Smithe a3adcb680d Trying to get the test working 2017-11-23 10:00:23 +00:00
Colin Goodheart-Smithe 9cba84b6fc Obeys the after parameter on phases excpet the first 2017-11-22 16:09:04 +00:00
Colin Goodheart-Smithe 3f0cf05dcc Started writing/reading phase and action settings 2017-11-22 13:26:05 +00:00
Colin Goodheart-Smithe a26a5969dd iterates over index and executes correct policy 2017-11-22 10:30:21 +00:00
Colin Goodheart-Smithe eba5afefb7 Gets the scheduler to use poll_interval 2017-11-21 16:54:28 +00:00