5108 Commits

Author SHA1 Message Date
David Roberts
2b72adc8ac
[TEST] Reduce forecast overflow to disk test memory limit (#30727)
By default ML native processes are only allowed to use
30% of RAM, so the previous 2GB setting prevented the
test passing on VMs with only 4GB RAM.  This change
reduces the limit to 1200MB, which means it can now
pass on VMs with 4GB RAM.
2018-05-18 19:01:43 +01:00
Ryan Ernst
b3f3a4312b
Plugins: Remove meta plugins (#30670)
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
2018-05-18 10:56:08 -07:00
Lisa Cawley
e750462e0c
[DOCS] Moves X-Pack configurationg pages in table of contents (#30702) 2018-05-18 10:26:03 -07:00
Dimitris Athanasiou
6bb2a1da22
[ML][TEST] Fix bucket count assertion in ModelPlotsIT (#30717)
As the first record is random, there's a chance it will
be aligned on a bucket start. Thus we need to check the
bucket count is in [23, 24].

Closes #30715
2018-05-18 17:59:01 +03:00
Dimitris Athanasiou
1484a31be5
[ML][TEST] Make AutodetectMemoryLimitIT less fragile (#30716)
These tests aim to check the set model memory limit is
respected. Additionally, it was asserting counts of
partition, by, over fields in an attempt to check that
the used memory is spent meaningfully. However, this
made the tests fragile, as changes in the ml-cpp could
lead to CI failures.

This commit removes those assertions. We are working on
adding tests in ml-cpp that will compensate.
2018-05-18 17:57:20 +03:00
Hendrik Muhs
6c313a9871 This implementation lazily (on 1st forecast request) checks for available
diskspace and creates a subfolder for storing data outside of Lucene
indexes, but as part of the ES data paths.

Details:
 - tmp storage is managed and does not allow allocation if disk space is
   below a threshold (5GB at the moment)
 - tmp storage is supposed to be managed by the native component but in
   case this fails cleanup is provided:
    - on job close
    - on process crash
    - after node crash, on restart
 - available space is re-checked for every forecast call (the native
   component has to check again before writing)

Note: The 1st path that has enough space is chosen on job open (job
close/reopen triggers a new search)
2018-05-18 14:04:09 +02:00
Colin Goodheart-Smithe
c5d31e30dd
Merge branch 'master' into index-lifecycle 2018-05-18 11:14:10 +01:00
Alexander Reelsen
b5a793b569
Tests: Fail if test watches could not be triggered (#30392)
Watcher tests now always fail hard when watches that were 
tried to be triggered in a test using the trigger() method, 
but could not because they were not found on any of the 
nodes in the cluster.
2018-05-18 08:51:43 +02:00
Hendrik Muhs
d893041634
[ML] add version information in case of crash of native ML process (#30674)
This change adds version information in case a native ML process crashes, the version is important for choosing the right symbol files when analyzing the crash. Adding the version combines all necessary information on one line.

relates elastic/ml-cpp#94
2018-05-18 07:46:52 +02:00
Tal Levy
5b5f33a956 Merge branch 'master' into index-lifecycle 2018-05-17 22:42:31 -07:00
Lisa Cawley
a8b4a98ee5
[DOCS] Fixes edit URLs for stack overview (#30583) 2018-05-17 14:20:03 -07:00
Ryan Ernst
30e0f33d16 Silence sleep based watcher test
see https://github.com/elastic/elasticsearch/issues/30699
2018-05-17 13:04:10 -07:00
Tal Levy
ca9f307b0a
add move-to-step endpoint for lifecycles (#30636)
This change introduces a new rest endpoint for lifecycles that
allows users to explicitely jump to earlier or later steps in the
policy's execution. This is useful for re-running tasks that may
be stuck, or were incorrectly configured.

Endpoint can be found in this format:

POST _xpack/index_lifecycle/_move/<index_name>
{
  current_step: ...
  next_step: ...
}

This operates on a per-index basis and does not resolve the param to
multiple indices.

The action is validated so that the index's state is only modified if
all of the following are true:

- <index_name> has an existing policy associated with it
- current_step is the actual step the index is currently on (for sanity)
- next_step is a valid step within the policy-step-registry

* respond to reviewer

refactor to stop using MoveToNextStepUpdateTask directly

* remove getPolicyRegistry

* rename validateMoveToNextStep
2018-05-17 13:02:15 -07:00
lcawl
663295d635 [DOCS] Replace X-Pack terms with attributes 2018-05-17 09:57:11 -07:00
Tal Levy
190a7efd2a Merge branch 'master' into index-lifecycle 2018-05-17 09:21:17 -07:00
Dimitris Athanasiou
75665a2d3e
[ML] Clean left behind model state docs (#30659)
It is possible for state documents to be
left behind in the state index. This may be
because of bugs or uncontrollable scenarios.
In any case, those documents may take up quite
some disk space when they add up. This commit
adds a step in the expired data deletion that
is part of the daily maintenance service. The
new step searches for state documents that
do not belong to any of the current jobs and
deletes them.

Closes #30551
2018-05-17 17:51:26 +03:00
Tal Levy
2814557679
add Rest tests to index-lifecycle (#30159)
* add QA-style Rest tests to index-lifecycle

This PR introduces a `qa` module within the index-lifecycle project.

the idea is to have both complex policies tested, as well as
policies with isolated/singular actions. So far, only tests with policies
containing one action are implemented.

Following Actions have implemented tests in this commit

- AllocateAction
- DeleteAction
- ForceMergeAction
- ReadOnlyAction
- ReplicasAction

tests to be added later

- RolloverAction
- ShrinkAction

* respond to review and enable integTests

* fix dependsOn fiasco

* fix license

* update to new proj structure

* move to new integTest with x-pack-core as module

* remove unused imports

* update to use module instead of plugin
2018-05-17 06:17:15 -07:00
Christoph Büscher
712473b558
[Docs] Replace InetSocketTransportAddress with TransportAdress (#30673)
The former class has been removed in 6.0, the documentation code
snippets should be updated accordingly.
2018-05-17 14:23:08 +02:00
David Roberts
ef0daee850
[TEST] Account for increase in ML C++ memory usage (#30675)
Recent changes to the ML C++ have resulted in higher memory usage,
so fewer "by" fields can be analyzed in a given amount of model
memory.
2018-05-17 12:59:20 +01:00
Tim Vernum
9f824c4aa8
Add detailed assert message to IndexAuditUpgradeIT (#30669)
Print out the returned buckets if the size does not match the
expectation.
2018-05-17 21:36:13 +10:00
Ioannis Kakavas
35fa934971
Adjust fast forward for token expiration test (#30668)
Adjust fast forward for token expiration test

Adjusts the maximum fast forward time for token expiration tests
to be 5 seconds before actual token expiration so that the test
won't fail even when upperlimit is randomly selected.

Resolves: #30062
2018-05-17 14:10:49 +03:00
Alexander Reelsen
11d776ecf0
Watcher: Fix watch history template for dynamic slack attachments (#30172)
The part of the history template responsible for slack attachments had a
dynamic mapping configured which could lead to problems, when a string
value looking like a date was configured in the value field of an
attachment.

This commit fixes the template by setting this field always to text.
This also requires a change in the template numbering to be sure this
will be applied properly when starting watcher.
2018-05-17 11:57:54 +02:00
Martijn van Groningen
5298237847
Merge remote-tracking branch 'es/master' into ccr
* es/master: (74 commits)
  Preserve REST client auth despite 401 response (#30558)
  [test] packaging: add windows boxes (#30402)
  Make xpack modules instead of a meta plugin (#30589)
  Mute ShrinkIndexIT
  [ML] DeleteExpiredDataAction should use client with origin (#30646)
  Reindex: Fixed typo in assertion failure message (#30619)
  [DOCS] Fixes list of unconverted snippets in build.gradle
  [DOCS] Reorganizes RBAC documentation
  SQL: Remove dependency for server's version from JDBC driver (#30631)
  Test: increase search logging for LicensingTests
  Adjust serialization version in IndicesOptions
  [TEST] Fix compilation
  Remove version argument in RangeFieldType (#30411)
  Remove unused DirectoryUtils class. (#30582)
  Mitigate date histogram slowdowns with non-fixed timezones. (#30534)
  Add a MovingFunction pipeline aggregation, deprecate MovingAvg agg (#29594)
  Removes AwaitsFix on IndicesOptionsTests
  Template upgrades should happen in a system context (#30621)
  Fix bug in BucketMetrics path traversal (#30632)
  Fixes IndiceOptionsTests to serialise correctly (#30644)
  ...
2018-05-17 10:34:44 +02:00
Tal Levy
4e757fff21 Merge branch 'master' into index-lifecycle 2018-05-16 15:46:44 -07:00
Ryan Ernst
a4c9c2fa2a
Make xpack modules instead of a meta plugin (#30589)
This commit removes xpack from being a meta-plugin-as-a-module.
It also fixes a couple tests which were missing task dependencies, which
failed once the gradle execution order changed.
2018-05-16 15:35:57 -07:00
Dimitris Athanasiou
01bdfcde6f
[ML] DeleteExpiredDataAction should use client with origin (#30646)
This is an admin action that should be allowed to operate on
ML indices with full permissions.
2018-05-16 23:35:23 +03:00
lcawl
8ff9baeb82 [DOCS] Fixes list of unconverted snippets in build.gradle 2018-05-16 13:11:06 -07:00
lcawl
4f41018753 [DOCS] Reorganizes RBAC documentation 2018-05-16 12:22:52 -07:00
Igor Motov
7b6d0faec2
SQL: Remove dependency for server's version from JDBC driver (#30631)
Removes dependency for server's version from the JDBC driver code. This
should allow us to dramatically reduce driver's size by removing the
server dependency from the driver.

Relates #29856
2018-05-16 15:07:14 -04:00
jaymode
d912fa4357
Test: increase search logging for LicensingTests
This commit increases the logging level around search to aid in
debugging failures in LicensingTests#testSecurityActionsByLicenseType
where we are seeing all shards failed error while trying to search the
security index.

See #30301
2018-05-16 11:59:49 -06:00
Van0SS
4478f10a2a Rest High Level client: Add List Tasks (#29546)
This change adds a `listTasks` method to the high level java
ClusterClient which allows listing running tasks through the 
task management API.

Related to #27205
2018-05-16 13:31:37 +02:00
Colin Goodheart-Smithe
f4fad07113
Merge branch 'master' into index-lifecycle 2018-05-16 11:43:26 +01:00
Colin Goodheart-Smithe
a75b8adce5
Refactors ClientHelper to combine header logic (#30620)
* Refactors ClientHelper to combine header logic

This change removes all the `*ClientHelper` classes which were
repeating logic between plugins and instead adds
`ClientHelper.executeWithHeaders()` and
`ClientHelper.executeWithHeadersAsync()` methods to centralise the
logic for executing requests with stored security headers.

* Removes Watcher headers constant
2018-05-16 11:38:24 +01:00
Martijn van Groningen
7942e4082a
build: enhance check task instead of overwriting it.
(test task didn't run when check task ran)
2018-05-16 10:54:15 +02:00
David Kyle
16f5a515f3
[ML] Wait for ML indices in rolling upgrade tests (#30615) 2018-05-16 09:52:25 +01:00
Alexander Reelsen
6ec5e4408c
Watcher: Ensure secrets integration tests also run triggered watch (#30478)
When the encrpytion of sensitive date is enabled, test that a
scheduled watch is executed as expected and produces the correct value
from a secret in the basic auth header.
2018-05-16 10:32:22 +02:00
Martijn van Groningen
596ec1848e
[CCR] Add validation checks that were left out of #30120 (#30463) 2018-05-16 09:46:03 +02:00
Colin Goodheart-Smithe
79be4d2976
Adds step info to ClusterStateWaitSteps (#30609)
The `ClusterStateWaitStep.isConditionMet()` method now returns a
`Result` object which contains a boolean for if the condition is met
and an `ToXContentObject` to provide information in the case where the
condition is not met.
If the condition is not met, the step information is stored in the
cluster state
2018-05-16 08:15:23 +01:00
Martijn van Groningen
23204e3d09
[CCR] Fixed follow and unfollow api url path according to design.
The TODOs in the rest actions was incorrect. The problem was that
these rest actions used `follow_index` as first named variable in the path
under which the rest actions were registered. Other candidate rest actions that
also have a named variable as first element in the path (but with a different
name) get resolved as rest parameters too and passed down to the rest
action that actually ends up getting executed.

In the case of the follow index api, a `index` parameter got passed down
to `RestFollowExistingAction`, but that param was never used. This caused the
follow index api call to fail, because of unused http parameters.

This change doesn't fixes that problem, but works around it by using
`index` as named variable for the follow index (instead of `follow_index`).

Relates to #30102
2018-05-16 09:07:50 +02:00
Martijn van Groningen
64b97313d5
[CCR] Make cross cluster replication work with security (#30239)
If security is enabled today with ccr then the follow index api will
fail with the fact that system user does not have privileges to use
the shard changes api. The reason that system user is used is because
the persistent tasks that keep the shards in sync runs in the background
and the user that invokes the follow index api only start those background
processes.

I think it is better that the system user isn't used by the persistent
tasks that keep shards in sync, but rather runs as the same user that
invoked the follow index api and use the permissions that that user has.
This is what this PR does, and this is done by keeping track of
security headers inside  the persistent task (similar to how rollup does this).

This PR also adds a cluster ccr priviledge that allows a user to follow
or unfollow an index. Finally if a user that wants to follow an index,
it needs to have read and monitor privileges on the leader index and
monitor and write privileges on the follow index.
2018-05-16 07:48:32 +02:00
Tal Levy
8c12a59c5e Merge branch 'master' into index-lifecycle 2018-05-15 22:41:23 -07:00
Ioannis Kakavas
2b09e90237
Replace custom reloadable Key/TrustManager (#30509)
Make SSLContext reloadable

This commit replaces all customKeyManagers and TrustManagers 
(ReloadableKeyManager,ReloadableTrustManager, 
EmptyKeyManager, EmptyTrustManager) with instances of 
X509ExtendedKeyManager and X509ExtendedTrustManager. 
This change was triggered by the effort to allow Elasticsearch to 
run in a FIPS-140 environment. In JVMs running in FIPS approved 
mode, only SunJSSE TrustManagers and KeyManagers can be used. 
Reloadability is now ensured by a volatile instance of SSLContext
in SSLContectHolder.
SSLConfigurationReloaderTests use the reloadable SSLContext to
initialize HTTP Clients and Servers and use these for testing the
key material and trust relations.
2018-05-16 08:32:13 +03:00
Ryan Ernst
c7d82b378b
Build: Add task interdependencies for ssl configuration (#30633)
This commit fixes the tasks creating ssl certs for tests to have correct
dependsOn to ensure the right tasks are run before tests run.
2018-05-15 16:09:15 -07:00
Tim Brooks
99b9ab58e2
Add nio http server transport (#29587)
This commit is related to #28898. It adds an nio driven http server
transport. Currently it only supports basic http features. Cors,
pipeling, and read timeouts will need to be added in future PRs.
2018-05-15 16:37:14 -06:00
Jason Tedor
abc06d5b79
Expose master version in REST test context (#30623)
This commit exposes the master version to the REST test context. This
will be needed in a follow-up where the master version will be used to
determine whether or not a certain warning header is expected.
2018-05-15 17:26:43 -04:00
Costin Leau
03dd2ab499 SQL: eliminate disabled tests 2018-05-15 22:49:05 +03:00
Costin Leau
09329eb84f
SQL: Verify GROUP BY ordering on grouped columns (#30585)
Due to the way composite aggregation works, ordering in GROUP BY can be
applied only through grouped columns which now the analyzer verifier
enforces.

Fix 29900
2018-05-15 22:46:46 +03:00
Ryan Ernst
21b9170dec
Security: Remove SecurityLifecycleService (#30526)
This commit removes the SecurityLifecycleService, relegating its former
functions of listening for cluster state updates to SecurityIndexManager
and IndexAuditTrail.
2018-05-15 12:13:24 -07:00
Ioannis Kakavas
17d65c1f06 Unmute IndexUpgradeIT tests
After additional fixes to the Jenkins job, these tests can now be
unmuted.

Closes #30430
2018-05-15 13:48:15 +03:00
Yannick Welsch
af2b9dd779 Revert "Mute ML upgrade test (#30458)"
This reverts commit 4b36ea74334e2664412af3540a64b82e0845e710.
2018-05-15 11:20:57 +02:00