Commit Graph

6159 Commits

Author SHA1 Message Date
Alexander Reelsen 71ebed6371
Watcher: Mark watcher as started only after loading watches (#30403)
Starting watcher should wait for the watcher to be started before
marking the status as started, which is now done via a callback.

Also, reloading watcher could set the execution service to paused. This could
lead to watches not being executed, when run in tests. This fix does not
change the paused flag in the execution service, just clears out the
current queue and executions.

Closes #30381
2018-05-07 14:19:01 +02:00
Martijn van Groningen ad499fc178
[CCR] added rest specs and simple rest test for follow and unfollow apis (#30123)
[CCR] added rest specs and simple rest test for follow and unfollow apis, also

Added an acknowledge field in follow and unfollow api responses. Currently these api return an empty response and fixed bug in unfollow api that didn't cleanup node tasks properly.
2018-05-07 14:18:28 +02:00
Jason Tedor ec71144040 Merge branch 'master' into index-lifecycle
* master: (35 commits)
  DOCS: Correct mapping tags in put-template api
  DOCS: Fix broken link in the put index template api
  Add put index template api to high level rest client (#30400)
  Relax testAckedIndexing to allow document updating
  [Docs] Add snippets for POS stop tags default value
  Move respect accept header on no handler to 6.3.1
  Respect accept header on no handler (#30383)
  [Test] Add analysis-nori plugin to the vagrant tests
  [Docs] Fix bad link
  [Docs] Fix end of section in the korean plugin docs
  Expose the Lucene Korean analyzer module in a plugin (#30397)
  Docs: remove transport_client from CCS role example (#30263)
  [Rollup] Validate timezone in range queries (#30338)
  Use readFully() to read bytes from CipherInputStream (#28515)
  Fix  docs Recently merged #29229 had a doc bug that broke the doc build. This commit fixes.
  Test: remove cluster permission from CCS user (#30262)
  Add Get Settings API support to java high-level rest client (#29229)
  Watcher: Remove unneeded index deletion in tests
  Set the new lucene version for 6.4.0
  [ML][TEST] Clean up jobs in ModelPlotIT
  ...
2018-05-06 18:17:36 -04:00
Jason Tedor beee5fe004
Respect accept header on no handler (#30383)
Today when processing a request for a URL path for which we can not find
a handler we send back a plain-text response. Yet, we have the accept
header in our hand and can respect the accepted media type of the
request. This commit addresses this.
2018-05-04 18:13:50 -04:00
Jay Modi 13e2223680
Docs: remove transport_client from CCS role example (#30263)
This commit removes the unnecessary transport_client cluster permission
from the role that is used as an example in our documentation. This
permission is not needed to use cross cluster search.
2018-05-04 11:52:11 -06:00
Zachary Tong 1c0d339904
[Rollup] Validate timezone in range queries (#30338)
When validating the search request, we make sure any date_histogram
aggregations have timezones that match the jobs.  But we didn't
do any such validation on range queries.

While it wouldn't produce incorrect results, it would be confusing
to the user as no documents would match the aggregation (because we
add a filter clause on the timezone for the agg).

Now the user gets an exception up front, and some helpful text about
why the range query didnt match, and which timezones are acceptable
2018-05-04 10:45:16 -07:00
Nhat Nguyen 6e0d0feca0 Enable MockHttpTransport in ShardChangsIT
CCR side of #29601
2018-05-04 13:44:18 -04:00
Jay Modi 2426d65dda
Test: remove cluster permission from CCS user (#30262)
This commit updates the multi cluster search test with security so that
the user that is simply performing a multi cluster search does not have
any cluster permissions. This is done as none are needed by this user
and excess privileges could mask a behavior change.
2018-05-04 10:14:09 -06:00
Alexander Reelsen bf51a21b6c Watcher: Remove unneeded index deletion in tests
A leftover invalid TODO comment and `.watches` index deletion command
can be removed again.
2018-05-04 17:10:54 +02:00
Nhat Nguyen f3a87be74a Remove old sha files from dated Lucene snapshot
This old snapshot was used this branch only.
2018-05-04 10:57:54 -04:00
Nhat Nguyen 8fefa8a661 Update InternalEngine tests on ccr side for #30121
Relates #30121
2018-05-04 10:57:54 -04:00
Nhat Nguyen db14717098 Merge branch 'master' into ccr
* master:
  Set the new lucene version for 6.4.0
  [ML][TEST] Clean up jobs in ModelPlotIT
  Upgrade to 7.4.0-snapshot-1ed95c097b (#30357)
  Watcher: Ensure trigger service pauses execution (#30363)
  [DOCS] Added coming qualifiers in changelog
  [DOCS] Commented out empty sections in the changelog to fix the doc build. (#30372)
  Security: reduce garbage during index resolution (#30180)
  Make RepositoriesMetaData contents unmodifiable (#30361)
  Change quad tree max levels to 29. Closes #21191 (#29663)
  Test: use trial license in qa tests with security
  [ML] Add integration test for model plots (#30359)
  SQL: Fix bug caused by empty composites (#30343)
  [ML] Account for gaps in data counts after job is reopened (#30294)
  InternalEngineTests.testConcurrentOutOfOrderDocsOnReplica should use two documents (#30121)
  Change signature of Get Repositories Response (#30333)
  Tests: Use different watch ids per test in smoke test (#30331)
  [Docs] Add term query with normalizer example
  Adds Eclipse config for xpack licence headers (#30299)
  Watcher: Make start/stop cycle more predictable and synchronous (#30118)
  [test] add debug logging for packaging test
  [DOCS] Removed X-Pack Breaking Changes
  [DOCS] Fixes link to TLS LDAP info
  Update versions for start_trial after backport (#30218)
  Packaging: Set elasticsearch user to have non-existent homedir (#29007)
  [DOCS] Fixes broken links to bootstrap user (#30349)
  Fix NPE when CumulativeSum agg encounters null/empty bucket (#29641)
  Make licensing FIPS-140 compliant (#30251)
  [DOCS] Reorganizes authentication details in Stack Overview (#30280)
  Network: Remove http.enabled setting (#29601)
  Fix merging logic of Suggester Options (#29514)
  [DOCS] Adds LDAP realm configuration details (#30214)
  [DOCS] Adds native realm configuration details (#30215)
  ReplicationTracker.markAllocationIdAsInSync may hang if allocation is cancelled (#30316)
  [DOCS] Enables edit links for X-Pack pages (#30278)
  Packaging: Unmark systemd service file as a config file (#29004)
  SQL: Reduce number of ranges generated for comparisons (#30267)
  Tests: Simplify VersionUtils released version splitting (#30322)
  Cancelling a peer recovery on the source can leak a primary permit (#30318)
  Added changelog entry for deb prerelease version change (#30184)
  Convert server javadoc to html5 (#30279)
  Create default ES_TMPDIR on Windows (#30325)
  [Docs] Clarify `fuzzy_like_this` redirect (#30183)
  Post backport of #29658.
  Fix docs of the `_ignored` meta field.
  Remove MapperService#types(). (#29617)
  Remove useless version checks in REST tests. (#30165)
  Add a new `_ignored` meta field. (#29658)
  Move repository-azure fixture test to QA project (#30253)

# Conflicts:
#	buildSrc/version.properties
#	server/src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
2018-05-04 09:40:57 -04:00
Dimitris Athanasiou 2751790aea [ML][TEST] Clean up jobs in ModelPlotIT
Closes #30377
2018-05-04 10:46:19 +01:00
Jim Ferenczi dbd857341f
Upgrade to 7.4.0-snapshot-1ed95c097b (#30357)
Upgrade to lucene-7.4.0-snapshot-1ed95c097b

This version contains:
* An Analyzer for Korean
* An IntervalQuery and IntervalsSource that retrieve minimum intervals of positional queries.
* A new API to retrieve matches (offsets and positions) of a query for a single document.
* Support for soft deletes in the index writer.
* A fixed shingle filter that handles index time synonyms.
* Support for emoji sequence in ICUTokenizer (with an upgrade to icu 61.1)
2018-05-04 11:44:22 +02:00
Alexander Reelsen 0e6cbbd811
Watcher: Ensure trigger service pauses execution (#30363)
When the watcher service pauses execution due to a cluster state update,
the trigger service and its engines also need to pause properly instead
of keeping going. This is also important when the .watches index is 
deleted, so that watches don't stay in a triggered mode.
2018-05-04 09:02:04 +02:00
Jay Modi aa0d7c73f8
Security: reduce garbage during index resolution (#30180)
The IndexAndAliasesResolver resolves the indices and aliases for each
request and also handles local and remote indices. The current
implementation uses the ResolvedIndices class to hold the resolved
indices and aliases. While evaluating the indices and aliases against
the user's permissions, the final value for ResolvedIndices is
constructed. Prior to this change, this was done by creating a
ResolvedIndices for the first set of indices and for each additional
addition, a new ResolvedIndices object is created and merged with
the existing one. With a small number of indices and aliases this does
not pose a large problem; however as the number of indices/aliases
grows more list allocations and array copies are needed resulting in a
large amount of garbage and severely impacted performance.

This change introduces a builder for ResolvedIndices that appends to
mutable lists until the final value has been constructed, which will
ultimately reduce the amount of garbage generated by this code.
2018-05-03 12:48:23 -06:00
jaymode 9573492a9e
Test: use trial license in qa tests with security
QA tests that use security need to use a trial license instead of a
basic license. Basic licenses do not enable security so these tests are
not running in the expected configuration. This can also lead to issues
that seem completely unrelated such as authentication failures right
after cluster formation.

The authentication failure right after cluster formation happens since
a request makes it past the authentication license checks and the
request starts the authentication process. During authentication the
cluster forms and the license is updated to a basic license, which
disables all realms. The request that is being authenticated then tries
to iterate over the realms, but the realms are empty and the request
cannot be authenticated. This results in a authentication failure even
though the credentials provided are correct.

Closes #30306
2018-05-03 10:33:08 -06:00
Dimitris Athanasiou a1e23feba2
[ML] Add integration test for model plots (#30359)
Relates #30004
2018-05-03 17:02:45 +01:00
Costin Leau 65dbc17510
SQL: Fix bug caused by empty composites (#30343)
When dealing with filtering, a composite aggregation might return empty
buckets (which have been filtered) which gets sent as is to the client.
Unfortunately this interprets the response as no more data instead of
retrying.

This now has changed and the listener keeps retrying until either the
query has ended or data passes the filter.

Fix #30292
2018-05-03 17:13:32 +03:00
Dimitris Athanasiou 3b260dcfc1
[ML] Account for gaps in data counts after job is reopened (#30294)
This commit fixes an issue with the data diagnostics were
empty buckets are not reported even though they should. Once
a job is reopened, the diagnostics do not get initialized from
the current data counts (especially the latest record timestamp).
The result is that if the data that is sent have a time gap compared
to the previous ones, that gap is not accounted for in the empty bucket
count.

This commit fixes that by initializing the diagnostics with the current
data counts.

Closes #30080
2018-05-03 15:08:24 +01:00
Colin Goodheart-Smithe e9dc190479
Throws ElasticsearchException from cluster state tasks if something
really bad happens

If the `onFailure()` method is called on the cluster state task then
something bad happened that we can't really deal with so this change
throws an ElasticsearchException in that case and the step will be
re-executed when the policy is next triggered. If we can't submit a
cluster state task then we can't move to the error state so there isn't
really anything else we can do here. If the cluster state task fails
like this there are probably bigger issues witht he cluster anyway.

x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/ExecuteStepsUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToNextStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/ClusterStateUpdateStepTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/ExecuteStepsUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToNextStepUpdateTaskTests.java
2018-05-03 11:05:37 +01:00
Alexander Reelsen 18394540ab
Tests: Use different watch ids per test in smoke test (#30331)
Each test now has its own watch id that is being used.
This ensures there are no old history entries, which can potentially
lead to broken test assertions.
2018-05-03 11:41:06 +02:00
Colin Goodheart-Smithe 61d88a8fdd
Makes all step tests use `randomStepKey()`
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AbstractStepTestCase.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/DeleteStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ForceMergeStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/InitializePolicyContextStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/PhaseAfterStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ReplicasAllocatedStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/RolloverStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/SegmentCountStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrunkShardsAllocatedStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrunkenIndexCheckStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/TerminalPolicyStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/UpdateSettingsStepTests.java
2018-05-03 10:18:42 +01:00
Colin Goodheart-Smithe b5ae7a1aeb
Adds infrastructure for dealing with errors in step execution (#30341)
* Adds infrastructure for dealing with errors in step execution

This change adds a new ErrorStep which a step can move to if it
encounters an error it cannot automatically recover from by retrying on
the next execution. The error step is special in that it cannot
complete. The intention is that the user will need to call an API to man
ually mocve the step in order to progress the index's lifecycle. The
error step retains the phase and action names of the step before it but
with the step name set to `ERROR`. For this reason no ordinary step can
have this name. `AbstractStepTestCase.testStepNameNotError()` ensures
that no step uses `ERROR` as the step name for either its stepKey or
its nextStepKey.

The new `index.lifecycle.failed_step` setting is used
to store the name of the failed step so the user can know in which step
the error occured. More error information will be added shortly.

The async steps will now move to the error step if listener.onFailure()
is called.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/LifecycleSettings.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/ErrorStep.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistry.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/ErrorStepTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToNextStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistryTests.java

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/LifecycleSettings.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AbstractStepTestCase.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStepTests.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistry.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToNextStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistryTests.java

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/LifecycleSettings.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AbstractStepTestCase.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStepTests.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTask.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistry.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToErrorStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/MoveToNextStepUpdateTaskTests.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/PolicyStepsRegistryTests.java

* Addresses review comments
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ErrorStepTests.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
2018-05-03 09:42:54 +01:00
Colin Goodheart-Smithe 6546535a8b
Merge branch 'master' into index-lifecycle 2018-05-03 08:54:15 +01:00
Alexander Reelsen 2c38d12e23
Watcher: Make start/stop cycle more predictable and synchronous (#30118)
The current implementation starts/stops watcher using an executor. This
can result in our of order operations.

This commit reduces those executor calls to an absolute minimum in order
to be able to do state changes within the cluster state listener method,
which runs in sequence.

When a state change occurs that forces the watcher service to pause
(like no watcher index, no master node, no local shards), the service is
now in a paused state.

Pausing is a super lightweight operation, which marks the
ExecutionService as paused and waits for the currently executing watches
to finish in the background via an executor. The same applies for
stopping, the potentially long running operation is outsourced in to an
executor, as waiting for executed watches is decoupled from the current
state.

The only other long running operation is starting, where watches need to
be loaded. This is also done via an executor, but has an additional
protection by checking the cluster state version it was started with. If
another cluster state version was trying to load the watches, then this
loading will not take effect.

This PR also cleans up some unused states, like the a simple boolean in
the HistoryStore/TriggeredWatchStore marking it as started or stopped,
as this can now be caught in the execution service.

Another advantage of this approach is the fact, that now only triggered
watches are not getting executed, while watches that are run via the
Execute Watch API will still be executed regardless if watcher is
stopped or not.

Lastly the TickerScheduleTriggerEngine thread now only starts on data nodes.
2018-05-03 09:47:12 +02:00
Andy Bristol 824e648662 [test] add debug logging for packaging test 2018-05-02 18:09:44 -07:00
lcawley b3516786ec [DOCS] Removed X-Pack Breaking Changes 2018-05-02 15:52:14 -07:00
lcawley 427d6912ea [DOCS] Fixes link to TLS LDAP info 2018-05-02 15:30:49 -07:00
Tim Brooks 226b45e509
Update versions for start_trial after backport (#30218)
This commit is a follow up to #30135. It updates the stream
compatibility versions in the start_trial requests and responses to
reflect that fact that this work has been backported to 6.3.
2018-05-02 15:51:59 -06:00
Lisa Cawley e697299103
[DOCS] Fixes broken links to bootstrap user (#30349) 2018-05-02 13:09:25 -07:00
Ioannis Kakavas cca1a2a7cf
Make licensing FIPS-140 compliant (#30251)
Necessary changes so that the licensing functionality can be
used in a JVM in FIPS 140 approved mode.
* Uses adequate salt length in encryption
* Changes key derivation to PBKDF2WithHmacSHA512 from a custom
  approach with SHA512 and manual key stretching
* Removes redundant manual padding

Other relevant changes:
* Uses the SAH512 hash instead of the encrypted key bytes as the
  key fingerprint to be included in the license specification
* Removes the explicit verification check of the encryption key
  as this is implicitly checked in signature verification.
2018-05-02 22:19:43 +03:00
Lisa Cawley 3e9fe3c9cd
[DOCS] Reorganizes authentication details in Stack Overview (#30280) 2018-05-02 12:08:02 -07:00
Ryan Ernst fb0aa562a5
Network: Remove http.enabled setting (#29601)
This commit removes the http.enabled setting. While all real nodes (started with bin/elasticsearch) will always have an http binding, there are many tests that rely on the quickness of not actually needing to bind to 2 ports. For this case, the MockHttpTransport.TestPlugin provides a dummy http transport implementation which is used by default in ESIntegTestCase.

closes #12792
2018-05-02 11:42:05 -07:00
Lisa Cawley 383856a175
[DOCS] Adds LDAP realm configuration details (#30214) 2018-05-02 11:22:32 -07:00
Lisa Cawley 5064ff6ad4
[DOCS] Adds native realm configuration details (#30215) 2018-05-02 10:56:31 -07:00
Lisa Cawley 0d7ac9a74c
[DOCS] Enables edit links for X-Pack pages (#30278) 2018-05-02 10:13:42 -07:00
Costin Leau 7790cb5fa9
SQL: Reduce number of ranges generated for comparisons (#30267)
* SQL: Reduce number of ranges generated for comparisons

Rewrote optimization rule for combining ranges by improving the
detection of binary comparisons in a tree to better combine
them in a range, regardless of their place inside an expression.
Additionally, improve the comparisons of Numbers of different types
Also, improve reassembly of conjunction/disjunction into balanced
trees.
Do not promote BinaryComparisons to Ranges since it introduces NULL
boundaries and thus a corner-case that needs too much handling
Compare BinaryComparisons directly between themselves and to Ranges

Fix #30017
2018-05-02 19:35:01 +03:00
Nhat Nguyen d621fc7a00
Add tombstone document into Lucene for Noop (#30226)
This commit adds a tombstone document into Lucene for every No-op. 
With this change, Lucene index is expected to have a complete history 
of operations like Translog. In fact, this guarantee is subjected to the
soft-deletes retention merge policy.

Relates #29530
2018-05-02 09:08:29 -04:00
Adrien Grand 231a63fdf8
Remove useless version checks in REST tests. (#30165)
Many tests are added with a version check so that they do not run against a
version that doesn't have the feature yet. Master is 7.0, so all tests that
do not run against 6.0+ can be removed and the version check can be removed
on all tests that always run on 6.0+.
2018-05-02 11:34:15 +02:00
Colin Goodheart-Smithe 4ad44a76b0
Merge branch 'master' into index-lifecycle 2018-05-02 09:01:08 +01:00
Jason Tedor 217d090dcd Merge branch 'master' into ccr
* master:
  Fix message content in users tool (#30293)
  [DOCS] Fixes links to breaking changes
  [DOCS] Adds new installation package details (#29590)
  Revert "Build: Move gradle wrapper jar to a dot dir (#30146)"
2018-05-01 23:49:21 -04:00
Nhat Nguyen eb4281edef CCR side #30244
Relates #30244
2018-05-01 21:08:24 -04:00
Nhat Nguyen d52ca33bd9 Merge branch 'master' into ccr
* master: (68 commits)
  [DOCS] Removes X-Pack Elasticsearch release notes (#30272)
  Correct an example in the top-level suggester documentation. (#30224)
  [DOCS] Removes broken link
  [DOCS] Adds file realm configuration details (#30221)
  [DOCS] Adds PKI realm configuration details (#30225)
  Fix a reference to match_phrase_prefix in the match query docs. (#30282)
  Fix failure for validate API on a terms query (#29483)
  [DOCS] Fix 6.4-specific link in changelog (#30314)
  Remove RepositoriesMetaData variadic constructor (#29569)
  Test: increase authentication logging for debugging
  [DOCS] Removes redundant SAML realm settings (#30196)
  REST Client: Add Request object flavored methods (#29623)
  [DOCS] Adds changelog to Elasticsearch Reference (#30271)
  [DOCS] Fixes section error
  SQL: Teach the CLI to ignore empty commands (#30265)
  [DOCS] Adds Active Directory realm configuration details (#30223)
  [DOCS] Removes redundant file realm settings (#30192)
  [DOCS] Fixes users command name (#30275)
  Build: Move gradle wrapper jar to a dot dir (#30146)
  Build: Log a warning if disabling reindex-from-old (#30304)
2018-05-01 21:07:54 -04:00
Tim Vernum 66daaaa1cc
Fix message content in users tool (#30293)
The elasticsearch-users utility had various messages that were
outdated or incorrect. This commit updates the output from this
command to reflect current terminology and configuration.
2018-05-02 11:04:28 +10:00
lcawley 42cc122a9b [DOCS] Fixes links to breaking changes 2018-05-01 17:21:28 -07:00
Lisa Cawley 092dd6cb89
[DOCS] Removes X-Pack Elasticsearch release notes (#30272) 2018-05-01 16:05:23 -07:00
lcawley 6a55eead94 [DOCS] Removes broken link 2018-05-01 14:42:29 -07:00
Lisa Cawley db44a4cddb
[DOCS] Adds file realm configuration details (#30221) 2018-05-01 14:11:38 -07:00
Lisa Cawley 86addc0c8b
[DOCS] Adds PKI realm configuration details (#30225) 2018-05-01 13:47:24 -07:00
Michael Basnight 62a9b8909e
Remove RepositoriesMetaData variadic constructor (#29569)
The variadic constructor was only used in a few places and the
RepositoriesMetaData class is backed by a List anyway, so just using a
List will make it simpler to instantiate it.
2018-05-01 15:02:06 -05:00
jaymode cdae8ff5a0
Test: increase authentication logging for debugging
This commit increases the logging for authentication in the x-pack
multi-node qa test project. This is needed to assist in debugging HTTP
authorization failures in waiting for the second node in these tests.

See #30306
2018-05-01 13:15:59 -06:00
Lisa Cawley 1a0838bd0b
[DOCS] Removes redundant SAML realm settings (#30196) 2018-05-01 11:53:11 -07:00
Lisa Cawley 5b5c98c96b
[DOCS] Adds changelog to Elasticsearch Reference (#30271) 2018-05-01 10:34:26 -07:00
lcawley 4dec3c36ee Merge remote-tracking branch 'upstream/master' 2018-05-01 10:22:05 -07:00
lcawley 0f688e0867 [DOCS] Fixes section error 2018-05-01 10:21:39 -07:00
Nik Everett abe797bd42
SQL: Teach the CLI to ignore empty commands (#30265)
Cause the CLI to ignore commands that are empty or consist only of
newlines. This is a fairly standard thing for SQL CLIs to do.

It looks like:
```
sql> ;
sql>
   |
   | ;
sql> exit;
Bye!
```

I think I *could* have implemented this with a `CliCommand` that throws
out empty string but it felt simpler to bake it in to the `CliRepl`.

Closes #30000
2018-05-01 13:17:24 -04:00
Lisa Cawley 7933f5e28e
[DOCS] Adds Active Directory realm configuration details (#30223) 2018-05-01 09:15:13 -07:00
Lisa Cawley 846783c2b6
[DOCS] Removes redundant file realm settings (#30192) 2018-05-01 08:55:35 -07:00
Lisa Cawley b34e5cf216
[DOCS] Fixes users command name (#30275) 2018-05-01 08:52:22 -07:00
Nik Everett 5e9e6fed90
HTML5ify Javadoc for xpack core (#30277)
xpack core contains a fork of `Cron` from quartz who's javadoc has a
`<table>` with non-html5 compatible stuff. This html5ifies the table and
switches the `:x-pack:plugin:core` project to building javadoc with
HTML5.
2018-05-01 08:32:58 -04:00
Colin Goodheart-Smithe 42711dd46d
Adds single node allocation to shrink (#30206)
* Adds ClusterState to AsyncActionStep.performAction

This is needed so a new step can be created for the shrink action which
can select a node to allocate to based on the current routing rules and
the node attributes on teh discovery nodes.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AsyncActionStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/DeleteStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ForceMergeStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/RolloverStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/UpdateSettingsStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/DeleteStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ForceMergeStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/RolloverStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/UpdateSettingsStepTests.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleService.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java

* Adds single node allocation to shrink

This change adds two new steps as the first steps of the shrink action:

1. A `SetSingleNodeAllocateStep` which:
    1. Determines which of the active nodes match the existing index
allocation rules
    2. Randomly (using Randomness so its deterministic for testing)
picks one of the matching nodes
    3. Updates the index settings to add a require allocation rule for
the node that was picked (using the
`index.routing.allocation.require._name` setting)
2. An `AllocationRoutedStep` which ensures that at least one copy of
each shard is allocated according to the new allocation rules

Note that this change also modifies the `AllocationRoutedStep` to add a
boolean field which determines whether the allocation is complete when
at least one copy of each shard matches the allocation rulees or if it
needs to wait for all shard copies to be allocated according to the
rules.

Lastly, a `randomStepKey()` method is added to `AbstractStepTestCase`
for convenience.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocateAction.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkAction.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/action/admin/indices/
settings/put/UpdateSettingsTestHelper.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AbstractStepTestCase.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkActionTests.java

* Fixes AllocationRoutedStep when `waitForAllShardCopies=false`

This change fixes `AllocationRoutedStep` so that when
`waitForAllShardCopies=false` we wait for any shard copy of each shard
to be allocated according to the allocation rules rather than
specifically the primary of each shard.
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStepTests.java

* Corrects Licence headers and typo
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStepTests.java
2018-05-01 10:27:51 +01:00
David Turner b9e1860f36 Mark JdbcSqlSpecIT as @AwaitsFix
Relates #30292
2018-05-01 10:03:18 +01:00
Dimitris Athanasiou 057cdffed5
[ML] Refactor DataStreamDiagnostics to use array (#30129)
This commit refactors the DataStreamDiagnostics class
achieving the following advantages:

- simpler code; by encapsulating the moving bucket histogram
into its own class
- better performance; by using an array to store the buckets
instead of a map
- explicit handling of gap buckets; in preparation of fixing #30080
2018-05-01 09:50:32 +01:00
Colin Goodheart-Smithe a6b0f663af
Merge branch 'master' into index-lifecycle 2018-05-01 09:21:57 +01:00
Martijn van Groningen 8a2df6c3b9
[CCR] Only normalize -1 seqno in shard changes request. (#30238)
Prior to this change a -1 seqno would be normalized earlier, which
caused a leader shard containing a single operation to be ignored.

Closes #30227
2018-05-01 08:40:23 +02:00
Boaz Leskes 4a537ef03c
Bulk operation fail to replicate operations when a mapping update times out (#30244)
Starting with the refactoring in https://github.com/elastic/elasticsearch/pull/22778 (released in 5.3) we may fail to properly replicate operation when a mapping update on master fails. If a bulk
operations needs a mapping update half way, it will send a request to the master before continuing 
to index the operations. If that request times out or isn't acked (i.e., even one node in the cluster 
didn't process it within 30s), we end up throwing the exception and aborting the entire bulk. This is 
a problem because all operations that were processed so far are not replicated any more to the 
replicas.  Although these operations were never "acked" to the user (we threw an error) it cause the 
local checkpoint on the replicas to lag (on 6.x) and the primary and replica to diverge. 

This PR does a couple of things:
1) Most importantly, treat *any* mapping update failure as a document level failure, meaning only 
    the relevant indexing operation will fail.
2) Removes the mapping update callbacks from `IndexShard.applyIndexOperationOnPrimary` and 
    similar methods for simpler execution. We don't use exceptions any more when a mapping 
    update was successful.

I think we need to do more work here (the fact that a single slow node can prevent those mappings 
updates from being acked and thus fail operations is bad), but I want to keep this as small as I can 
(it is already too big).
2018-05-01 08:15:02 +02:00
lcawley 51902238f3 [DOCS] Fixes syskeygen command name 2018-04-30 13:20:22 -07:00
David Roberts 225f7093a9
[ML] Include 3rd party C++ component notices (#30132)
The overall NOTICE file for the ML X-Pack module should
include the notices from the 3rd party C++ components as
well as the 3rd party Java components.
2018-04-30 20:05:27 +01:00
Lisa Cawley e95a7aa6f0
[DOCS] Removes redundant Active Directory realm settings (#30190) 2018-04-30 08:52:21 -07:00
Lisa Cawley 05160e6cd8
[DOCS] Removes redundant LDAP realm settings (#30193) 2018-04-30 08:04:15 -07:00
Chris Earle 725a5af2c6
_cluster/state should always return cluster_uuid (#30143)
Currently, the only way to get the REST response for the `/_cluster/state`
call to return the `cluster_uuid` is to request the `metadata` metrics,
which is one of the most expensive response structures. However, external
monitoring agents will likely want the `cluster_uuid` to correlate the
response with other API responses whether or not they want cluster
metadata.
2018-04-30 10:16:11 -04:00
Colin Goodheart-Smithe d7d709a5dc
Merge branch 'master' into index-lifecycle 2018-04-30 09:29:48 +01:00
Lisa Cawley 7eaec6031d
[DOCS] Fixes broken links (#30219) 2018-04-27 13:24:15 -07:00
Lisa Cawley 962c965812
[DOCS] Adds native realm security settings (#30186) 2018-04-27 12:40:05 -07:00
Nik Everett f4ed902698
CCS: Drop http address from remote cluster info (#29568)
They are expensive to fetch and no longer needed by Kibana so they
*shouldn't* be needed by anyone else either.

Closes #29207
2018-04-27 14:19:00 -04:00
Zachary Tong fee000a37f
[TEST] Redirect links to new locations (#30179)
We had a number of awaitsFix links that weren't updated after the xpack
merge.

Where possible I changed the links to the new locations, but in some
circumstances the original ticket was closed (suggesting the awaitsfix
should be removed) or was otherwise unclear the status.
2018-04-27 09:24:46 -07:00
Tanguy Leroux 7ae3b3b155
Move repository-s3 fixture tests to QA test project (#29372)
This commit moves the repository-s3 fixture test added in #29296 in a
new `repository-s3/qa/amazon-s3` project. This new project allows the
REST integration tests to be executed using the real S3 service when
all the required environment variables are provided. When no env var
is provided, then the tests are executed using the fixture added
in #29296.

The REST tests located at the `repository-s3`plugin  project now only 
verify that the plugin is correctly loaded.

The REST tests have been adapted to allow a bucket name and a base 
path to be specified as env vars. This way it is possible to run the tests
with different base paths (could be anything, like a CI job name or a
branch name) without multiplicating buckets.

Related to #29349
2018-04-27 16:49:06 +02:00
Tal Levy f6a425e676 fix line length in RolloverStep 2018-04-27 07:48:17 -07:00
Colin Goodheart-Smithe ec9f1a9f3b
Merge branch 'master' into index-lifecycle 2018-04-27 15:32:02 +01:00
Nhat Nguyen 112b5f1744 Merge branch 'master' into ccr
* master: (24 commits)
  Watcher: Ensure mail message ids are unique per watch action (#30112)
  REST: Remove GET support for clear cache indices (#29525)
  SQL: Correct error message (#30138)
  Require acknowledgement to start_trial license (#30135)
  Fix a bug in FieldCapabilitiesRequest#equals and hashCode. (#30181)
  SQL: Add BinaryMathProcessor to named writeables list (#30127)
  Tests: Use buildDir as base for generated-resources (#30191)
  Fix SliceBuilderTests#testRandom failures
  Build: Fix deb version to use tilde with prerelease versions (#29000)
  Fix edge cases in CompositeKeyExtractorTests (#30175)
  Document time unit limitations for date histograms (#30177)
  Add support for field capabilities to the high-level REST client. (#29664)
  Remove licenses missed by the migration (#30128)
  [DOCS] Updates docker installation package details (#30110)
  Fix TermsSetQueryBuilder.doEquals() method (#29629)
  [Monitoring] Remove unhelpful Monitoring tests (#30144)
  [Test] Fix RenameProcessorTests.testRenameExistingFieldNullValue() (#29655)
  add copyright/scope configuration for intellij to Contributing Guide (#29688)
  [test] include oss tar in packaging tests (#30155)
  TEST: Update settings should go through cluster state (#29682)
  ...
2018-04-27 09:23:31 -04:00
Alexander Reelsen 707ba28d48
Watcher: Ensure mail message ids are unique per watch action (#30112)
Email message IDs are supposed to be unique. In order to guarantee this,
we need to take the action id of a watch action into account as well,
not just the watch id from the watch execution context. This prevents
that two actions from the same watch execution end up with the same
message id.
2018-04-27 08:55:25 +02:00
Costin Leau e0b8893645
SQL: Correct error message (#30138)
* SQL: Correct error message

Error messages had placeholders that were not replaced; this PR fixes
that

Fix #30016
2018-04-27 09:24:25 +03:00
Tim Brooks 592481e4ed
Require acknowledgement to start_trial license (#30135)
This is related to #30134. It modifies the start_trial action to require
an acknowledgement parameter in the rest request to actually start the
trial license. There are backwards compatibility issues as prior ES
versions did not support this parameter. To handle this, it is assumed
that a request coming from a node prior to 6.3 is acknowledged. And
attempts to write a non-acknowledged request to a prior to 6.3 node will
throw an exception.

Additionally this PR adds messages about the trial license the user is
generating.
2018-04-26 21:42:44 -04:00
Costin Leau 804c38303f
SQL: Add BinaryMathProcessor to named writeables list (#30127)
BinaryMathProcessor was missing from the list of register  named
writeables causing deserialization errors

Fix #30014
2018-04-27 01:43:44 +03:00
Ryan Ernst 55e1b1e8b5
Tests: Use buildDir as base for generated-resources (#30191)
This commit moves the generated-resources directory created by many qa
projects into the build directory, so it is not seen as unknown files to
git.
2018-04-26 15:36:14 -07:00
Tal Levy 053e9fb234 Merge branch 'master' into index-lifecycle 2018-04-26 15:15:45 -07:00
Christoph Büscher d0416c76fe
Fix edge cases in CompositeKeyExtractorTests (#30175)
Currently the test picks random java.util.TimeZone ids in some places.
Internally we still need to convert back to joda DateTimeZone by id
occassionally (e.g. when serializing to pre 6.3 versions). There are
some deprecated "SystemV/*" time zones that Jodas DateTimeZone refuses
to convert. This change excludes those rare cases from the set of
allowed random time zones. It would be quiet odd for them to appear in
practice.

Closes #30156
2018-04-26 20:06:47 +02:00
Michael Basnight cb7e3ffd75
Remove licenses missed by the migration (#30128)
A few of the old style license got kept around because their comment
string did not start with a space. This caused the license check to not
see it as a license and skip it. This commit cleans it up.
2018-04-26 11:37:04 -05:00
Lisa Cawley eebcdce7f9
[DOCS] Updates docker installation package details (#30110) 2018-04-26 08:54:29 -07:00
Chris Earle 1d81ec1562
[Monitoring] Remove unhelpful Monitoring tests (#30144)
This removes some monitoring tests that have been silenced for a long
time. These tests don't really provide any value for the upgrade suite and
they just create noise due to their occasional timing-related failures.
2018-04-26 11:32:46 -04:00
Tal Levy c10e624c92
update RolloverAction to utilize an index-setting for rollover_alias (#30158)
The purpose of this change is to reflect what is the future way of configuring aliases in
the Rollover action for ILM. With this change, users will be able to edit all desired rollover-related
information in the index's template, so that the index-lifecycle policy does not need to be dependent on it.
2018-04-26 08:18:23 -07:00
Tal Levy 26b42dfbf3 fix typo in xpack-rest test plugin->module 2018-04-26 08:12:02 -07:00
Andy Bristol 67c0cf1dbf
[test] include oss tar in packaging tests (#30155)
Add the oss tar distribution to the packaging test plugin. Test the oss
tar distribution in the core packaging tests, and the non-oss tar
distribution in the x-pack packaging tests.
2018-04-26 06:58:41 -07:00
Martijn van Groningen 4bcecd34a1
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  Watcher: Fold two smoke test projects into smoke-test-watcher (#30137)
  In the field capabilities API, deprecate support for providing fields in the request body. (#30157)
  Set JAVA_HOME before forking setup commands (#29647)
  Remove animal sniffer from low-level REST client (#29646)
  Cleanup .gitignore (#30145)
  Do not add noop from local translog to translog again (#29637)
  Build: Assert jar LICENSE and NOTICE files match
  Correct transport compression algorithm in docs (#29645)
  [Test] Fix docs check for DEB package in packaging tests (#30126)
  Painless: Docs Clean Up (#29592)
  Fixes Eclipse build for sql jdbc project (#30114)
  Remove reference to `not_analyzed`.
  [Docs] Add community analysis plugin (#29612)
2018-04-26 09:25:19 +02:00
Alexander Reelsen fadcce8367
Watcher: Fold two smoke test projects into smoke-test-watcher (#30137)
In order to have less qa/ projects, this commit removes the
watcher-smoke-test-mustache and the watcher-smoke-test-painless projects
and moves the tests of both into the watcher-smoke-test projects.

This results in less projects to parse when calling gradle and a shorter test
time due to being able to run everything within one elasticsearch
instance.
2018-04-26 09:23:54 +02:00
Tal Levy b2e0d6af18 fix syntax error in info.asciidoc 2018-04-25 17:38:17 -07:00
Nik Everett a8f40b3e04 Build: Assert jar LICENSE and NOTICE files match
Adds tasks that check that the all jars that we build have LICENSE.txt
and NOTICE.txt files and that the files are correct. Sets check to
depend on these task.

This is mostly there for extra parnoia because we automatically
configure all Jar tasks to include the LICENSE.txt and NOTICE.txt
files anyway. But it is quite possible to add configuration to those
tasks that would override either file.

This causes check to depend on several more things than it used to.
Take, for example, javadoc:

check depends on the new verifyJavadocJarNotice which depends on
extractJavadocJar which depends on javadocJar which depends on
javadoc, this check now depends on javadoc.
2018-04-25 19:53:24 -04:00
Tal Levy 1df6b6c65d Applying changes required for ILM after merge 2018-04-25 13:04:06 -07:00
Tal Levy d74fb9eb45 Opened x-pack ILM code 2018-04-25 12:22:53 -07:00
Colin Goodheart-Smithe c02b895653
Fixes Eclipse build for sql jdbc project (#30114)
The bundled configuration isn't recognised by eclipse so these
dependencies are missed when it imports the `x-pack:plugin:sql:jdbc`
project. This change makes these dependencies compile dependencies if
the build is running for Eclipse.
2018-04-25 16:10:32 +01:00
Martijn van Groningen b126ffa1cb
Updated sha files 2018-04-25 16:00:48 +02:00
Martijn van Groningen 596fab9eb7
fixed compile error after merge 2018-04-25 15:59:22 +02:00
Martijn van Groningen e6b88fa5a0
removed duplicated license 2018-04-25 12:18:02 +02:00
Martijn van Groningen 831e7e0777
Fixed test failure after migrating xpack ccr code to elasticsearch repository. 2018-04-25 10:10:06 +02:00
Martijn van Groningen 5a67a0f78f
Applying changes required for ccr after moving ccr code to elasticsearch 2018-04-25 08:03:29 +02:00
Martijn van Groningen cbd38983f4
Opened x-pack ccr code 2018-04-25 08:03:28 +02:00
Michael Basnight 72f57c8e72
Add comments inadvertently removed during migrate
A few files had their first comment removed even though it did not
contain a license. This re-adds those comments.
2018-04-24 13:41:09 -05:00
David Kyle cfc66a1fd5 [ML] Wait for updates to established memory usage
Tests need to wait for changes to the job's established memory usage to
propagate and an over enthusiastic optimisation meant jobs were updated
from stale state causing recent change to be lost.
2018-04-24 13:46:58 -04:00
Jason Tedor ab101976d6 Fix SQL CLI on Windows
This commit fixes the classpath for the SQL CLI tool on Windows. As the
x-pack bin folder was collapsed into the distribution bin folder, the
location of the classpath here needed to no longer contain the old
plugins directory.
2018-04-20 15:34:01 -07:00
Jason Tedor d99d0fa669 Add distribution type to startup scripts
This commit adds the distribution type to the startup scripts so that we
can discern from log output and the main response the type of the
distribution (deb/rpm/tar/zip).
2018-04-20 15:34:01 -07:00
Jason Tedor c7f6b35485 Fix classpath for X-Pack scripts on Windows
With the move of X-Pack to a module, the classpath for the scripts needs
to be adjusted. This was done on Unix, but not for Windows. This commit
addresses Windows.
2018-04-20 15:34:01 -07:00
Ryan Ernst d8636d0fe3 Rename syskeygen
This commit renames syskeygen to elasticsearch-syskeygen
2018-04-20 15:34:01 -07:00
Jason Tedor c7c0e330b8 Rename users
This commit renames users to elasticsearch-users.
2018-04-20 15:34:01 -07:00
Jason Tedor b05c8bf781 Rename migrate
This commit renames migrate to elasticsearch-migrate.
2018-04-20 15:34:00 -07:00
Jason Tedor 2d3fac307a Rename saml-metadata
This commit renames saml-metadata to elasticsearch-saml-metadata.
2018-04-20 15:34:00 -07:00
Jason Tedor 8a8a1dfc2e Rename sql-cli
This commit renames sql-cli to elasticsearch-sql-cli.
2018-04-20 15:34:00 -07:00
Ryan Ernst 6dbb7be067 Rename croneval
This commit renames croneval to elasticsearch-croneval
2018-04-20 15:34:00 -07:00
Jason Tedor 7ba2defdbf Rename setup-passwords
This commit renames setup-passwords to elasticsearch-setup-passwords.
2018-04-20 15:34:00 -07:00
Jason Tedor 03ffd16921 Rename certutil
This commit renames certutil to elasticsearch-certutil.
2018-04-20 15:33:59 -07:00
Jason Tedor a6b4682711 Rename certgen
This commit renames certgen to elasticsearch-certgen.
2018-04-20 15:33:59 -07:00
Ryan Ernst 0d8aa7527e Reorganize license files
This commit moves the apache and elastic license files into a new
root level `licenses` directory and rewrites the top level LICENSE.txt
to clarify the repository has a mix of apache and elastic licensed code.
2018-04-20 15:33:59 -07:00
Ryan Ernst 9a45662309 Convert license header check to use the new Elastic License
This commit adapts the license headers check to no longer look for the
ealsticsearch confidential license, but instead to look for the new
Elastic License header.
2018-04-20 15:33:59 -07:00
Jason Tedor e64e6d8996 Add distribution flavor to startup scripts
This commit adds the distribution flavor (default versus oss) to the
build process which is passed through the startup scripts to
Elasticsearch. This change will be used to customize the message on
attempting to install/remove x-pack based on the distribution flavor.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst cb56bf49d5 Fixup build after opening x-pack 2018-04-20 15:33:46 -07:00
Ryan Ernst 2efd22454a Migrate x-pack-elasticsearch source to elasticsearch 2018-04-20 15:29:54 -07:00
Martijn van Groningen 9b9d0f9057 Enabled licence header check and fixed unchecked casts. (#4408) 2018-04-20 11:15:52 +02:00
Martijn van Groningen cfd7847628 fixed issues after merging in master 2018-04-20 07:59:13 +02:00
Nhat Nguyen f97aec7b8b Sibling of enforce access to translog via engine
Since elastic/elasticsearch#29542, we no longer expose translog instance
but only provide creating translog snapshot method. This commit adapts
that change in CCR branch.

Relates elastic/elasticsearch#29542
2018-04-18 11:54:00 -04:00
Martijn van Groningen 56ca59a513 Add the ability to the follow index to follow an index in a remote cluster.
The follow index api completely reuses CCS infrastructure that was exposed via:
https://github.com/elastic/elasticsearch/pull/29495

This means that the leader index parameter support the same ccs index
to indicate that an index resides in a different cluster.

I also added a qa module that smoke tests the cross cluster nature of ccr.
The idea is that this test just verifies that ccr can read data from a
remote leader index and that is it, no crazy randomization or indirectly
testing other features.
2018-04-17 07:36:40 +02:00
Tal Levy 689d628d67 move phase-after steps to have the previous phase's phase in its key (#4387)
Previously, phase X's `after` step had `X` as its associated phase. This causes confusion because
we have only entered phase `X` once the `after` step is complete. Therefore, this refactor
pushes the after's phase to be associated with the previous phase. This first phase is an exception. 
The first phase's `after` step is associated with the first phase (not some non-existent prior phase).
2018-04-16 21:19:30 -07:00
Martijn van Groningen c0d42e9cd1 Fixed test 2018-04-16 10:48:46 +02:00
Martijn van Groningen a94b38b88e Fixed compile errors and test failures after merging master into ccr. 2018-04-13 16:35:09 +02:00
Colin Goodheart-Smithe d509834c00 Fixes checkstyle error 2018-04-13 14:39:37 +01:00
Colin Goodheart-Smithe af6456983f Moves registing of NamedWriteables to plugin:core
This is needed so that clients have the ability to serialise and deserialise the objects it creates and gets back.
2018-04-13 14:23:54 +01:00
Colin Goodheart-Smithe bf3db3f1f7 Adds phase and action change times on state transition 2018-04-13 13:38:44 +01:00
Colin Goodheart-Smithe d17886f1b0 Fixes checkstyle error 2018-04-13 10:44:45 +01:00
Tal Levy f04c95f4c2 change policy logs to be DEBUG 2018-04-12 15:40:20 -07:00
Colin Goodheart-Smithe 644d82e165 Renames AliasStep to ShrinkSetAliasStep 2018-04-12 18:29:12 +01:00
Colin Goodheart-Smithe 77be66b8b9 Adds shrink check that original index is deleted 2018-04-12 18:13:57 +01:00
Colin Goodheart-Smithe 59e0979b92 Fixes shrunk shards allocation check step
- Renames EnoughShardsWaitStep to ShrunkShardsAllocatedStep
- Changes ShrunkShardsAllocatedStep to check the shards of the shrunken index rather than the current one
- shrink index prefix is now passed into the steps of the shrink aciton
- Related Test Changes
2018-04-12 18:06:04 +01:00
Tal Levy 34e15011a1 fix ilm rest-tests to reflect latest get_policy changes 2018-04-12 07:47:57 -07:00
Colin Goodheart-Smithe 088894019e Tidies up unit tests
Also fixes failing EnoughShardsWaitStep
2018-04-12 12:37:36 +01:00
Colin Goodheart-Smithe 1c2e7bc431 Increases iterations on step equality tests
To make it in line with the fact that other equality tests use multiple iterations
2018-04-12 12:23:35 +01:00
Colin Goodheart-Smithe e8cb6419f8 Removes unused imports 2018-04-12 12:16:43 +01:00
Colin Goodheart-Smithe 7b247fe41e Fixes compile errors from TimeValue serialisation 2018-04-12 12:15:36 +01:00
Tal Levy 20b6aaf6b6 migrate step tests to extend AbstractStepTestCase 2018-04-11 18:56:53 -07:00
Tal Levy c1b00d0154 cleanup tests
- make DeleteActionTests pass
- remove getType from LifecycleType
2018-04-11 18:28:46 -07:00
Tal Levy 7f4801f03c clean up PhaseTests 2018-04-11 18:16:28 -07:00
Tal Levy 73ee5a4aa0 introduce refactored ShrinkAction
this resulted in a few new steps being introduced

- AliasStep
- EnoughShardsWaitStep
- ShrinkStep
- ShrunkenIndexCheckStep
2018-04-11 18:10:13 -07:00
Colin Goodheart-Smithe 20485cf7fb Adds more missing equality tests
Specifically for the initialPolicyContextStep and the PhaseAfterStep
2018-04-11 13:24:31 +01:00
Colin Goodheart-Smithe 482de191f2 Tidies up some of the unit tests
Adds some missing tests including checking the hashcode and equals methods of `DeleteStep`, `StepKey`, and `TerminalPolicyStep` as well as adding a test for `DeleteAction.toSteps()`
2018-04-11 12:38:55 +01:00
Colin Goodheart-Smithe 66b4e8c4aa Removes unused imports 2018-04-11 10:47:07 +01:00
Tal Levy ef34f982f1 refactor a collection of step-related things + shrink_step
- added ShrinkStep/Tests
- AsyncActionStep now passes in IndexMetaData instead of Index
- Delete usage of ClusterStateActionStep
- with ClusterStateActionStep gone, InitializePolicyContextStep
  is the only other ClusterState-nonWait step
- Migrate setting-updates to UpdateSettingsStep
2018-04-10 17:13:10 -07:00
Colin Goodheart-Smithe 8d91f197d4 Changes update replica and alllocation to async steps
Also renames EnoughShardsWaitStep to ReplicasAllocatedStep, removes it from the allocate action and adds a check that th number of replicas in the cluster state is correct to it.
2018-04-10 16:20:29 +01:00
Colin Goodheart-Smithe aa198b637e Cleans up warnings 2018-04-10 09:53:57 +01:00
Colin Goodheart-Smithe c24dc07346 Removes unused and unneeded code
Various classes had some code that was not used and is not going to be needed so this change cleans up those classes so we don’t have dead code hanging around
2018-04-10 09:28:03 +01:00
Colin Goodheart-Smithe b8d573772f Fixes compilation issue in test 2018-04-10 09:10:52 +01:00
Tal Levy f5b23df349 add more force-merge related steps and tests
The force-merge is an a TODO state due to the
unresolved issue around best_compression.

- updated ReadOnlyStep with tests
- implemented an update to the ForceMergeAction
- added UpdateBestCompressionSettingsStep
- added tests for SegmentCountStep
2018-04-09 16:28:25 -07:00
Colin Goodheart-Smithe 06e969b430 Adds rollover step and implements RolloverAction.toSteps 2018-04-09 15:55:06 +01:00
Colin Goodheart-Smithe 54c6d280a5 Adds Steps for ReplicasAction
Also adds tests for those steps and ReplicasAction.toSteps
2018-04-09 14:43:57 +01:00
Colin Goodheart-Smithe 2eb8fd9336 Adds tests for AllocateAction and its steps
Also tweaks some of the code in the steps of the allocate action
2018-04-09 13:59:25 +01:00
Tal Levy 47ed2e5b23 add more tests 2018-04-05 18:00:41 -07:00
Tal Levy b58159ddad add more tests 2018-04-05 17:43:32 -07:00
Tal Levy 228fc8c842 add some stuff 2018-04-05 17:21:44 -07:00
Tal Levy c0acea3721 :check yourself greeny 2018-04-05 13:16:19 -07:00
Tal Levy 1a2afac98b fix tests 2018-04-05 12:43:10 -07:00
Colin Goodheart-Smithe e3e669067c Adds loggic for TerminalPolicyStep 2018-04-05 17:58:22 +01:00
Colin Goodheart-Smithe 05c2c5655b replaces cause with boolean 2018-04-05 17:53:15 +01:00
Colin Goodheart-Smithe ed260a0430 Removes Cause stuff 2018-04-05 17:10:56 +01:00
Martijn van Groningen d77f756f5c ccr: use indices stats api to fetch global checkpoint of the follower shards and
keep track of shard follow stats inside shard follow stats' node task instead of persistent task status.

By maintaining the shard follow stats inside its node task the stats update is quicker as
no cluster state update is required. The stats are now transient; meaning if the task
is going to run a different node then the stats are gone too. Currently only the processed
global checkpoint is being tracked and this is being restored when a shard follow node task
starts via the indices stats api (the reason of the first change of this change). Other stats
that we may add in the future (like fetch_time, see: https://gist.github.com/s1monw/dba13daf8493bf48431b72365e110717)
it is ok if we start from zero in case a shard follow task moves to another node.
2018-04-05 14:52:20 +02:00
Tal Levy 907586c9e4 add MoveToNextStepUpdateTaskTests 2018-04-04 20:39:00 -07:00
Tal Levy 55d2a62fec add tests for ExecuteStepsUpdateTask 2018-04-04 15:32:36 -07:00
Colin Goodheart-Smithe a8dc8a7c98 Adds more unit tests for IndexLifecycleRunner 2018-04-04 15:59:10 +01:00
Colin Goodheart-Smithe 9ef26dbe51 Unit tests static methods in IndeexLifecycleRunner 2018-04-04 13:29:30 +01:00
Tal Levy 47dcc8fe9c more changes + updated api 2018-04-03 17:42:25 -07:00
Tal Levy 1ac1ee413f moar changes 2018-04-02 16:21:48 -07:00
Tal Levy d2e87a66e5 moar changes
- set `indexSurvives` for Step
- extract the two ClusterStateUpdateTasks to separate classes

- simple Delete Policy works!
2018-03-29 16:23:57 -07:00
Martijn van Groningen d976fa44e7 Removed LocalCheckpointTracker usage. 2018-03-29 07:41:23 +02:00
Tal Levy 74eaba2078 moar changes from Tal 2018-03-28 17:10:50 -07:00
Colin Goodheart-Smithe 179074d3b0 Refactors steps and adds some execution logic
This is far from complete but its necessary to push so the new step classes can be shared and used.
2018-03-28 16:34:34 +01:00
Martijn van Groningen a22a7d079d ccr: Added maximum translog limit that a single shard changes response can return.
This limit is based on the number of estimate bytes in each translog
operation that fall between the minimum and maximum request sequence number.

If this limit is met then the shard follow task executor will make sure
that a subsequent shard changes request will be performed to fetch the
remaining translog operations.

This limit is needed in order to protect against returning too many
translog operations in a single shard changes response.

Relates to #2436
2018-03-28 15:49:57 +02:00
Martijn van Groningen 282740610b Fixed test after merging in master branch. 2018-03-28 09:54:41 +02:00
Tal Levy e6ee5b49d1 have steps point to step-key, not the actual step object 2018-03-27 11:52:08 -07:00
Tal Levy ce4248ec59 slight changes 2018-03-27 11:41:38 -07:00
Colin Goodheart-Smithe 9972710e9e Makes brnach compile
Commented out toSteps implem,entations and other bits needed to get the branch to compile
2018-03-27 12:14:38 +01:00
Tal Levy f429fc0b3e begin making sense of types 2018-03-26 18:48:59 -07:00
Tal Levy 083c563cf6 meh 2018-03-26 18:12:38 -07:00
Tal Levy 57821cd55a moar refactor for steps 2018-03-26 17:54:15 -07:00
Nhat Nguyen 51111a8106 CCR: Stop FollowExistingIndexAction after report failure (#4111)
We check for the existence of both leader and follower index, then properly 
report to the caller. However, we do not return after reporting failure. This
causes the caller receive exception twice: IllegalArgumentException then
NullPointerException. This commit makes sure to stop the action after reporting
failure.
2018-03-26 13:56:47 -04:00
Tal Levy d63cd8c9c3 step by step 2018-03-26 08:00:03 -07:00
Martijn van Groningen 9e4c68c389 Fixed compile and test errors after merging in master 2018-03-16 17:47:10 +01:00
Tal Levy fe22c08207 stop using illegal logging of Object 2018-03-12 13:18:02 -07:00
Tal Levy ebff02c1b2 fix compilation changes 2018-03-12 12:29:22 -07:00
Martijn van Groningen 10cfa21a68 required changes after merge master branch into ccr branch. 2018-02-22 15:03:33 +01:00
Tal Levy 9850f44b95 fix state left over by spec test 2018-02-13 13:40:09 -08:00
Tal Levy 8e282f21c8 Merge branch 'master' into index-lifecycle 2018-02-12 14:17:24 -08:00
Martijn van Groningen 1a9a7ffe97 removed hack 2018-02-07 17:54:28 +01:00
Martijn van Groningen c442d14f1d Several changes that were required after merging master into the ccr branch. 2018-02-05 13:25:58 +01:00
Tal Levy 0a3dcebca0 disallow deleting lifecycle policies that are referenced by existing indices (#3806) 2018-02-01 14:16:06 -08:00
Tal Levy ab49e40f91 add phase and action timestamps to cluster-state (#3726)
This commit sets `index.lifecycle.phase_time` and `index.lifecycle.action_time`
appropriately during setPhase and setAction in IndexLifecycleContext.
2018-01-25 09:02:26 -08:00
Colin Goodheart-Smithe faf067080b Makes sure creation date garentees execution in tests 2018-01-25 13:21:50 +00:00
Martijn van Groningen 4e818254ad re-enabled java integration tests 2018-01-25 14:18:34 +01:00
Tal Levy 9920d6be2d keep plugin up to date with structure changes (#3702)
Looks like we need to split out the tests of core classes to core
and index-lifecycle ones stay in index-lifecycle.

I believe I got everything, although I may have missed at least one thing

checked status with

$ ./gradlew :x-pack-elasticsearch:plugin:index-lifecycle:check -Dtests.seed=39838421912001B4
$ ./gradlew :x-pack-elasticsearch:plugin:core:check -Dtests.seed=39838421912001B4

other things done in this PR:
- removal of a few unused variables/thrown exceptions/imports
- fix TimeseriesLifecycleTypeTests
  - an all null AllocateAction was created
- fix AllocateActionTests
  - woops. -Dtests.seed=39838421912001B4 resulted in two `null`s and an emptyMap.
    this resulted in a test failure.
2018-01-24 08:47:21 +00:00
Martijn van Groningen 05d3d2e49c fix packages after merge 2018-01-24 09:28:42 +01:00
Tal Levy 4856baf66b fix guice and plugin initialization after split (#3677)
the x-pack-split required updating the IndexLifecycle plugin to
be a proper plugin with the Plugin API methods instead of the XPackPlugin API
methods.
2018-01-23 10:41:22 -08:00
Colin Goodheart-Smithe fd502aa3e6 Makes some fields in Index Lifecycle API optional (#3687)
Specifically this change makes it optional to:
* Specify `includes`, `excludes` and `requires`maps in the allocate action as long as at least one fo the options is specified and is not an empty map
* Specify an `after` parameter on a phase. If no `after` value is specified `TimeValue.ZERO` is used and the phase will be moved to as soon as the previous phase reports `ACTIONS COMPLETED`. `after` is always non-null when we are serialising the Phase.
* Specify a `type` for a LifecyclePolicy. If no `type` is specified `TimeSeriesLifecycleType.INSTANCE` is used since this is currently the only production `type`. `type` is always non-null when we are serialising the LifecyclePolicy.
2018-01-23 17:02:11 +00:00
Colin Goodheart-Smithe f2fa988f2f Fixes Checkstyle 2018-01-23 08:24:03 +00:00
Jason Tedor 9b6bb2c635 Enable run task for CCR
This commit enables the run task for ccr by specifying that the ccr
project not be evaluated until after core is evaluated. This is
important since ccr is alphabetically before core and thus Gradle
evaluates it first.

Relates #3665
2018-01-22 15:07:20 -05:00
Tal Levy 07b28fd721 migrate lifecycle settings to plugin:core:LifecycleSettings (#3670) 2018-01-22 10:43:16 -08:00
Tal Levy 34d6b61209 add `index.lifecycle.date` setting (#3545)
This PR adds a new setting called `index.lifecycle.date` that 
the ShrinkAction will be responsible for populating in the newly created index.

This way, we can continue to know when we should be executing the next phase
relative to the original index creation date, and not that of the shrunken index.
2018-01-22 09:27:33 -08:00
Colin Goodheart-Smithe 9032d61516 Fixes silly issues created by me 2018-01-22 16:48:32 +00:00
Colin Goodheart-Smithe 2d721136c2 Adds logic to cancel job if master is un-elected (#3613)
A node can stop being the master node whilst it is running, e.g. if it can’t access `minimumMasterNodes` number of master eligible nodes. Because of this we need logic in `IndexLifecycleService` that cancels the scheduled job if the node is no longer master and re-adds the job if the node becomes master again.
2018-01-22 15:45:37 +00:00
Colin Goodheart-Smithe ee774594e2 Adds test for TimeValueSchedule 2018-01-22 15:21:55 +00:00
Colin Goodheart-Smithe a6f6335e86 Moves Index-Lifecycle classes over to new module
These changes are necessary following the x-pack split.
2018-01-22 13:52:22 +00:00
Martijn van Groningen 83a82d83d0 Moved ccr source code to its own gradle module after xpack split. 2018-01-22 11:09:04 +01:00
Colin Goodheart-Smithe 3cdf1216cd Removes un-needed licence files 2018-01-19 10:27:57 +00:00
Colin Goodheart-Smithe cb4cbf80ac Catches more that just ElasticsearchException 2018-01-18 05:42:22 +00:00
Tal Levy 84fd234835 Add the Shrink Action to Index Lifecycle Management (#3491)
This does the following in sequential service polls

1. sets the index to read-only and runs shrink with a modified `index.lifecycle.name` setting set to `null`.
2. checks to see if shrink is complete, if it is...
    b. set target index's `index.lifecycle.*` settings to the original index's values.
3. if not complete, just wait till next iteration
4. if operating on shrunken index, delete old index and add it as an alias to shrunken index
2018-01-17 11:22:06 -08:00
Colin Goodheart-Smithe ff883d574b Adds Allocate lifecycle action (#3484)
* Adds Allocate lifcycle action

* Addresses review comments

Still need to make a change in core for the FilterAllocationDecider to make the execute logic simpler

* Addresses more review comments

* Adds randomMap method to AllocateActionTests

* Addresses further review comments
2018-01-17 13:27:40 +00:00
Tal Levy db3a88f487 fix TimeseriesLifecycle ordering behavior 2018-01-16 18:08:05 -08:00
Martijn van Groningen 7a4860452a Update the persistent task status in the background.
Follow up for #3256
2018-01-16 10:48:55 +01:00
Jason Tedor bad5135d35 Fix checkpoints in assertion
The checkpoints in the assertion message that the follower checkpoint is
less than the leader checkpoint are backwards. This commit fixes this
message.
2018-01-15 11:50:12 -05:00
Colin Goodheart-Smithe 971685654d Adds more tests in IndexLifecycleServiceTests 2018-01-15 15:29:07 +00:00
Colin Goodheart-Smithe 1e6ed023ee Fixes ReplicasActionTests
Fixes a test failure and adds a new test that was missing
2018-01-15 09:37:00 +00:00
Colin Goodheart-Smithe 53c3d040f7 Add ReplicaAction for Index Lifecycle (#3535)
* Add ReplicaAction for Index Lifecycle

* Add validation
2018-01-12 13:48:09 +00:00
Martijn van Groningen 18788531c8 Fixed test failure.
Relates to #3256
2018-01-12 14:04:26 +01:00
Colin Goodheart-Smithe 5eb912539d Fixes Logging usage
log line had an extra parameter
2018-01-12 12:17:14 +00:00
Colin Goodheart-Smithe 6a370251fc Improves handling of exceptions in Index Lifecycle (#3511)
* Improves handling of exceptions in Index Lifecycle

This change improves a few different aspects:

* If an exception occurs executing the lifecycle of one index it is caught, logged and other indexes are still processed
* If the lifecycle policy specified in the settings does not exist an error is logged
* Fixes the exception when the delete action is run which occurs because Phase attempts to update the phase and action settings for the deleted index. A `LifecycleAction.indexSurvives()` method is introduced which defaults to `true` but can be overridden to indicate whether the index survives following completion of the action.

* Adds test
2018-01-12 11:52:56 +00:00
Tal Levy e3a6e738ea Fix InternalIndexLifecycleContext to update state in memory (#3505)
* Fix InternalIndexLifecycleContext to update state in memory

The internal and the mock index-lifecycle-context implementations differed
in that the InternalIndexLifecycleContext assumed no one would be using it after
it mutated state. This is not the case. We assume that the current context is updated after
a `#setAction` is called so that the listener can then appropriately use the newly modified
cluster state. since idxMeta was not being updated, any call to `context.getAction` was stale and
either returning null or the previous action, not the next action that was updated by `#setAction`.
Same goes for `setPhase`.

This PR should fix this so that the Mock and Internal implementations are more in line.
2018-01-10 08:21:17 -08:00
Martijn van Groningen a56b0479e1 Changed the chunk coordinator to process the chunks concurrently. (#3256)
The shard follow task executor determines the range of translog operations
between the leader shard's global checkpoint and the last know processed
seqno by the current shard follow task that are missing.

Then the chunks coordinator can then chunk this range up in smaller ranges
if the requested range is above the configured max chunk size. If it is
smaller than the entire range then the chunk coordinator has just one
chuck to coordinate.

Each chunk is added to a queue and is processed by the ChunkProcessor,
that reads the translog ops from the leader shard and then indexes
these translog ops into the follow shard. After that a new chuck is polled
from the queue and the ChunkProcessor performs the same actions until
there are no more chunks in the queue to process. After that the shard
follow task executor will determine a new range of translog operations
to process.

This change changes the chunk coordinator to start polling from the chunk
queue with multiple threads at the same time to handle dealing with a higher
indexing load on the leader side better.
2018-01-10 15:29:11 +01:00
Colin Goodheart-Smithe 57c343bc21 Use the XContentRegistry provided by the parser 2018-01-08 14:14:43 +00:00
Colin Goodheart-Smithe c4621d9dc7 Mutes IndexLifecycleInitialisationIT.testMasterFailover
Due to a bug with unassigned shards in force merge action
2018-01-05 12:43:13 +00:00
Martijn van Groningen 38aea9588b Fixed compile errors after merge. 2018-01-04 18:18:46 +01:00
Colin Goodheart-Smithe 148688a991 Reduces poll time to 1s for integ test
`IndexLifecycleInitialisationIT.testMasterFailover()` intermittently failed because the timeout of 10 seconds to check if the index had been deleted was not long enough sometimes with the poll interval set to 3 seconds. This change sets the poll interval to 1 seconds for the test so that the lifecycle is more responsive. This also means the default value for the poll interval can be safely changed without affecting the test.
2018-01-03 12:29:19 +00:00
Colin Goodheart-Smithe 52e416d0bc Creates index-lifecycle gradle module
This doesn’t currently contain any code but is there in preparation for the x-pack-elasticsearch:plugin split
2018-01-03 12:03:54 +00:00
Colin Goodheart-Smithe 043c8db84d Fixes Tests so build passes 2018-01-03 12:01:12 +00:00
Colin Goodheart-Smithe a37df9092e Moves Actions and related to plugin:core
This change moves the Action classes and referenced data model classes to the new :x-pack-elasticsearch:plugin:core project in preparation for splitting the x-pack features into their own gradle modules.

Note that the TransportAction classes had to be promoted to their own class file (rather than being inner classes to their Action) so they can remain in the plugin project (and will late be move to the `index-lifecycle` project when its created.
2018-01-03 10:45:33 +00:00
Colin Goodheart-Smithe 33afe0f012 Merge branch 'master' into index-lifecycle 2018-01-03 10:21:50 +00:00
Tal Levy 6a47b4fa6e fix infinite loop in test and correct logic in timeseries type 2017-12-21 13:14:37 -08:00
Tal Levy 66af56320e fix tests 2017-12-21 12:56:56 -08:00
Tal Levy e433b1eaa8 Adds ForceMerge action to Index Lifecycle (#3309)
add forcemerge index-lifecycle action
2017-12-21 11:59:44 -08:00
Martijn van Groningen dbdefc5689 Fix issue after merging 2017-12-21 14:53:12 +01:00
Colin Goodheart-Smithe d5609a408b fixes integ test when running as part of the build 2017-12-21 12:38:34 +00:00
Colin Goodheart-Smithe 3627feded7 Refactors lifecycle type to its own class (#3381)
To clean up the parsing of the LifecyclePolicy this change moves the LifecycleType to its own class so it can be created in the normal parsing of LifecyclePolicy rather than having to parse to an intermediary object first. The LifecycleType is an interface which can be implemented for different lifecycle types. These types shiould be singletons and are register with the NamedXContentRegistry and NamedWriteableRegistry only so they are available when reading from a stream or parsing.
2017-12-21 11:20:49 +00:00
Tal Levy fb59f54637 make the lifecycle poll-interval configurable (#3319)
Removes the poll-interval from the IndexLifecycleMetadata and introduces it in 
the form of a cluster setting that is configurable. Changes to this poll interval setting 
will reflect in the Lifecycle Scheduler.
2017-12-20 09:39:39 -08:00
Colin Goodheart-Smithe 586065fca7 Changes to enable the build to run
This includes changing NOCOMMIT comments to be NORELEASE comments so the build passes with them. We have tasks inGH for all these NORELEASE comments so they should be caught before merging to master
2017-12-20 12:27:26 +00:00
Martijn van Groningen 973f126c24 Merge branch 'master' into ccr 2017-12-18 16:52:09 +01:00
Colin Goodheart-Smithe b12f1a9526 More phase tests
This time for when setting the new action fails
2017-12-15 11:20:59 +00:00
Colin Goodheart-Smithe 33a1a92e3d As phase test for when the action execution fails 2017-12-15 11:07:49 +00:00
Colin Goodheart-Smithe 9e693b544c Test improvements 2017-12-15 10:52:37 +00:00
Colin Goodheart-Smithe 3da42f5603 Adds rollover action to index lifecycle plugin (#3266)
This action will rollover an index when executed if the provided conditions are met.
Users may specify the maximum age, maximum index size in bytes or maximum index size in number of documents as conditions for rollover.

When the action executes it firsts checks the local cluster state to find out if the alias exists on the index. If the alias does not exist then the index was either rolled over by a previous run or something else has rolled over the index so the action can be marked as completed. If the index still has the alias set the action will make a rollover index request using the Client. When that request returns and the listener is called the action will only be marked as complete if the response indicates the index was rolled over. If the index was not rolled over (because the conditions are not yet met) the action is not marked as complete and will be re-evaluated on the next call to execute.
2017-12-15 09:36:39 +00:00
Tal Levy 8d6e53356b enable rest of await-fix index-lifecycle integ tests. all green 2017-12-13 13:40:58 -08:00
Tal Levy c0c7e02de4 fix javadoc usage 2017-12-11 16:24:44 -08:00
Tal Levy 288515ddc5 initial attempt towards an ordered execution plan for timeseries phases (#3248)
* initial attempt towards an ordered execution plan for timeseries phases

* respond to feedback and fix bug
2017-12-07 11:59:31 -08:00
Martijn van Groningen 6c06aacb57 Fixed NumberFormatException in the rest layer by only setting batch_size if it has been specified. 2017-12-07 09:34:25 +01:00
Tal Levy ae7e4882e4 general indexlifecycle cleanup, rest tests, reintroduce integ-test (#3243)
* cleanup, rest tests, reintroduce integ-test

- cleaned up some long lines
- re-introduced one of the integration tests
- added rest-spec tests
2017-12-06 07:51:06 -08:00
Tal Levy d847355b58 add timeseries policy action stubs (#3214) 2017-12-05 09:09:18 -08:00
Martijn van Groningen 0dc5a89bc5 fixed compile error 2017-12-01 14:14:17 +01:00
Martijn van Groningen cc5665bce6 pass down batch size correctly 2017-12-01 14:14:17 +01:00
Martijn van Groningen 3cd5eed91e fixed code violations 2017-12-01 14:14:17 +01:00
Martijn van Groningen c3f7d4f580 * Made ChunkCoordinator unittest testable
* Fixed a small issue where each batch would fetch / index the previous batch last operation
* Made batch size a request param on the follow existing index api request.
  This makes is easy to tune this param when running tests from scripts.
* Changed default batch size from 256 to 1024.
2017-12-01 14:14:17 +01:00
Martijn van Groningen 2e382bf7f3 Disallow dynamic mapping updated in follow shards
I forgot to configure a mapping in the follow shard shard, which caused
a dynamic update (due to type auto creation), but this was ignored.

Subsequent searches in follow index then failed due to a mapping missing.
(The _id couldn't be fetched during fetch phase, because the mapping was missing)

We should at a later stage investigate how to best solve this, but for
know to avoid confusion just fail if a dynamic update happens in a
follow shard.
2017-12-01 09:33:06 +01:00
Tal Levy 685231e786 Introduce specific types of index lifecycle policies (#3157)
Introduce specific types of index lifecycle policies
2017-11-30 10:09:41 -08:00
Jason Tedor be5f83a6bd Implement translog operation bulk action
This commit adds a bulk action for apply translog operations in bulk to
an index. This action is then used in the persistent task for CCR to
apply shard changes from a leader shard.

Relates #3147
2017-11-29 07:29:56 -05:00
Jason Tedor 11aa83011c Fix get operations between test
This test was broken by an upstream change that no longer guarantees we
see the operations from the upstream translog in the order they appear
in that translog. As such, the assertions in this test were too strong
so this commit relaxes them.

Relates #3153
2017-11-29 07:16:52 -05:00
Jason Tedor abd263fca9 Merge branch 'master' into ccr
* master:
  [DOCS] Fixed cross cluster search docs issue (#3113)
  Watcher: Add transform input for chained input (#2861)
  [DOCS] Fix cleaning up ML resources after tests
  [ML] Specify ML_ORIGIN when calling the job update endpoint internally (#3110)
  Refactor CLI commands as logging-aware commands
  [ML] Stop datafeed when job fails (#3107)
2017-11-27 14:03:10 -05:00
Colin Goodheart-Smithe 922060d766 Typo 2017-11-27 15:46:47 +00:00
Colin Goodheart-Smithe 7199e2fa5c Small refactoring 2017-11-27 12:32:55 +00:00
Colin Goodheart-Smithe 87766cd4ef Adds JavaDocs and comments and small refactoring 2017-11-27 12:22:10 +00:00
Colin Goodheart-Smithe 9ab2121fcf When actions complete the next action is now run 2017-11-27 11:07:26 +00:00
Colin Goodheart-Smithe 152d5d45e5 Adds listeners to Lifecycle actions
This means that the result of the action can now be async and we can then implement moving immediately to the next action if the current one is complete
2017-11-24 19:27:50 +00:00
Colin Goodheart-Smithe 26ad2142d5 minor tweak 2017-11-24 17:29:38 +00:00
Colin Goodheart-Smithe 9e891de777 refactors code to allow better testing
The client and index metadata have now been abstracted away from the Lifecycle classes behind IndexLifecycleContext. This allow us to test the state machine without having to worry about how the state is persisted and read. It also makes the classes much easier to read and reason about.
2017-11-24 17:22:51 +00:00
Colin Goodheart-Smithe e113189347 only creates initialisation service ifeature is enabled 2017-11-24 10:23:04 +00:00
Colin Goodheart-Smithe 06958949ab Uses LongSupplier for nowSupplier
Rather than Supplier<Long>. This is in line with the `nowSupplier`s used in other areas of Elasticsearch
2017-11-24 10:18:06 +00:00
Colin Goodheart-Smithe 31ded906d0 Adds settings checking to tests
This isn’t an ideal solution so might well be temporary but at least the tests are checking the settings changes are correct now
2017-11-24 10:10:46 +00:00
Colin Goodheart-Smithe 016ba2a57d Adds another test
This test needs more work
2017-11-24 09:40:32 +00:00
Colin Goodheart-Smithe 49b2aaa878 Adds more tests 2017-11-23 18:16:47 +00:00
Colin Goodheart-Smithe 75704510e1 More test changes 2017-11-23 15:48:37 +00:00
Colin Goodheart-Smithe a9703d4a56 Mutes failing integration test for now 2017-11-23 14:12:31 +00:00
Colin Goodheart-Smithe 0e829843fc More unit tests 2017-11-23 14:11:09 +00:00
Colin Goodheart-Smithe db502bef6d Adds some unit tests 2017-11-23 12:44:12 +00:00
Colin Goodheart-Smithe f571dc3000 Remove references to InternalClient which no longer exists 2017-11-23 10:35:25 +00:00
Colin Goodheart-Smithe a3adcb680d Trying to get the test working 2017-11-23 10:00:23 +00:00
Colin Goodheart-Smithe 9cba84b6fc Obeys the after parameter on phases excpet the first 2017-11-22 16:09:04 +00:00
Colin Goodheart-Smithe 3f0cf05dcc Started writing/reading phase and action settings 2017-11-22 13:26:05 +00:00
Colin Goodheart-Smithe a26a5969dd iterates over index and executes correct policy 2017-11-22 10:30:21 +00:00
Martijn van Groningen e6ad1c3e9d ccr: Added apis and persistent tasks for following an index and log changes instead of indexing. 2017-11-22 09:04:59 +01:00
Colin Goodheart-Smithe eba5afefb7 Gets the scheduler to use poll_interval 2017-11-21 16:54:28 +00:00
Colin Goodheart-Smithe 3f3f7b9b47 Adds PUT/GET/DELETE lifecycle APIs
The lifecycles are stored as custom metadata objects in the cluster state. This change also cleans up the parsing of the lifecycle state so that it can be parsed properly
2017-11-21 15:55:45 +00:00
Tal Levy 1fad59b295 tal-temp migration to not use settings for the lifecycle policy 2017-11-17 09:22:54 -07:00
Jason Tedor 8e0b34b507 Override indexing plans
Operations from a leader shard will be indexed into the engine with the
origin set to primary. The problem is here is that then we have primary
semantics in the engine such as assertions about sequence numbers being
unassigned, and we do not have correct semantics for out-of-order
delivery of operations (as we should on a following engine, whether or
not it is primary since the ordering is determined from the
leader). This commit handles this by always using the replica plan for
indexing into a following engine, whether or not the engine is for a
primary shard.

Relates #3000
2017-11-15 13:38:46 -05:00
Jason Tedor 913936f2b0 Maintain order of operations semantics on follower
A following engine even for a primary shard needs to maintain order of
operations semantics as if it were behaving like a replica. That is,
rather than assuming that the order of operations presented to the
engine is the de facto order of operations as is the case for a leader
engine for a primary shard, a following engine must behave like all
replicas behave which is that they resolve order of operations based on
sequence numbers. This commit causes this to be the case for following
engines.

Relates #2931
2017-11-09 15:18:54 -05:00
Martijn van Groningen bf4e18fdfc fixed compile error 2017-11-02 15:12:35 +01:00
Tal Levy dd6d192e25 cleanup tests 2017-10-31 17:39:25 -07:00
Colin Goodheart-Smithe c393944dc0 Adds settings, phase and action objects which control how and when things are executed 2017-10-31 15:30:42 +00:00
Tal Levy 3455445d2c add integration test to indexlifecyclemanagement
This test verifies that we have sufficient failover code so that
a newly elected master re-registers old schedules and fires them off.
All times are relative to the index creation date.
2017-10-30 17:51:01 -07:00
Jason Tedor 769349a9ab Add following engine implementation
This commit is a first step towards a following engine
implementation. Future work will build on this by using this engine to
execute operations on a following engine from another engine (typically
a remote leader engine) that has already assigned sequence numbers to
such operations.

Relates #2776
2017-10-30 13:38:02 -04:00
Tal Levy de4d87c6e6 test 2017-10-26 15:32:08 -07:00
Martijn van Groningen 41c3dc91c1 Changed the shard changes api to use get threadpool instead of management threadpool. 2017-10-17 10:02:28 +02:00
Martijn van Groningen 2663b5ef39 remove unused import 2017-10-16 10:36:22 +02:00
Jason Tedor 85c1935d9d Increase generations in operations between test
This commit sets an index setting for the size of a translog generation
and increases the number of documents indexed to increase the chance of
multiple generations being present when testing getting operations
between two sequence numbers.
2017-10-16 08:59:18 +02:00
Jason Tedor bcd61bfca5 Fix off-by-one error in shard changes action test
This commit fixes an off-by-one error in the shard changes action test
for getting operations between two sequence numbers. The off-by-one
error arises because sequence numbers are indexed from zero, so if N
documents are indexed then the maximum sequence number starting from
zero would be N - 1.
2017-10-16 08:54:04 +02:00
Martijn van Groningen 1f495f59a1 [CCR] Add an internal api to read translog operations between a sequence number range (#2563)
* xdcr: Add an internal api to read translog operations between a sequence number range.

This api will be used later by the persistent task for the following index to pull data from the leader index.

The persistent task can fetch the global checkpoint from the shard stats for each primary shard of the leader index.
Based on the global checkpoint of the primary shards of the following index, the persistent task can send several
calls to the internal api added in this commit to replicate changes from follow index to leader index in a batched manner.
2017-10-16 07:38:03 +02:00
Colin Goodheart-Smithe 4aaec2ef81 Creates a new X-Pack feature for index lifecycle
Feature consists of a shell of a persistant task which will later be used to inspect the index settings and apply curator like changes to the index (move from hot to warm, rollover, shrink etc.)
2017-10-12 16:22:42 +01:00
Jason Tedor 3033aba67b Introduce pluggable engine factory for CCR
This commit utilizes the pluggable engine factory feature in core to
introduce a pluggable engine factory for XDCR. For now this is only a
skeleton implementation to proof out the pluggable engine factory
concept. Future work will implement a genuine following engine for XDCR.

Relates #2655
2017-10-12 10:52:33 -04:00
Jason Tedor 063a76c3db Introduce CCR container class
This commit introduces the container class for CCR functionality. Future
work will expose more specific CCR functionality to the X-Pack plugin
through this class.

Relates #2704
2017-10-06 14:46:39 -04:00