1521 Commits

Author SHA1 Message Date
Igor Motov
ffcd54de77 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@e67ceb1d1e
2017-11-17 13:33:28 -05:00
Albert Zaharovits
5b73c77011 SetupPasswordTool handle url cmd option correctly (elastic/x-pack-elasticsearch#2899)
Fixes bug when the url option had trailing slashes. The URL built
was invalid (consecutive fwd slashes) but the failure errors of
the subsequent requests were ignored.

URL is built correctly from the option spec.
True HTTP errors and Exceptions are logged and the cmd fails.

relates elastic/x-pack-elasticsearch#2778

Original commit: elastic/x-pack-elasticsearch@62b2d94ca0
2017-11-17 20:30:22 +02:00
Nhat Nguyen
418a6632e5 Registers a new handler for shard snapshot status
Relates elastic/x-pack-elasticsearch#27165

Original commit: elastic/x-pack-elasticsearch@11199642bb
2017-11-17 13:13:50 -05:00
Nik Everett
b8e082107f Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@bbc72c0368
2017-11-17 12:05:47 -05:00
David Roberts
6741a62e80 [ML] Adjust memory limit test to account for greater accuracy (elastic/x-pack-elasticsearch#3047)
Due to elastic/machine-learning-cpp#409 the ML C++ code now instruments
memory more accurately.  This change modifies the expectations of the Java
integration test to account for the change.

Original commit: elastic/x-pack-elasticsearch@2ed7a75af4
2017-11-17 17:02:19 +00:00
Alexander Reelsen
0f97e28074 Watcher: Further preparations for source repo split (elastic/x-pack-elasticsearch#3006)
Changes to further prepare for feature split with regards to watcher:

- CryptoService has been moved into watcher
- CryptoService.generateKey() has been moved into SystemKeyTools, only
  used there
- The creation of the http client/notification classes have been moved
  into watcher, no further dependencies on watcher in XPackPlugin
- Each subproject now registers it's own named writeables

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a60c98ba7e
2017-11-17 17:05:07 +01:00
Alexander Reelsen
3d5fb54522 Watcher: Create templates on nodes newer than the master (elastic/x-pack-elasticsearch#2950)
This problem was introduced due to distributed watch execution.

When a node newer than the master node joins the cluster and gets a
.watches shard assigned it is supposed to start watcher. However
when a new version of the watch history template is part of that new
node (and we might increase that version anytime), this template does
not get installed, because only the master node is updating watcher
templates.

This commit checks if the local node version is higher than the master
node version and then also puts missing templates.

Currently this is done for all watcher templates, not only the watcher history.

relates elastic/x-pack-elasticsearch#2944

Original commit: elastic/x-pack-elasticsearch@4960231ea7
2017-11-17 16:56:29 +01:00
Hendrik Muhs
a583cf270b [ML-FC] implement endpoint parameter that takes a duration (elastic/x-pack-elasticsearch#3027)
Adds a duration parameter to the forecast API. Also fixes issue if no parameter is given (forecast 1 day from time of last bucket), in which case it lets autodetect decide

depends on elastic/machine-learning-cpp#407

Original commit: elastic/x-pack-elasticsearch@3387478872
2017-11-17 06:37:03 +01:00
Tim Brooks
f94e6d427b Modify x-pack to for TcpTransport changes
This is related to elatic/elasticsearch#27407. That commit removed
parameterization from TcpTranport. This commit modifies the security
transport to be compatible with those changes.

Original commit: elastic/x-pack-elasticsearch@9878c26e14
2017-11-16 14:32:40 -07:00
Hendrik Muhs
5efdb54eb8 add memoryUsage stats (elastic/x-pack-elasticsearch#3025)
adds memory usage statistics to forecast request stats

depends on: elastic/machine-learning-cpp#406

Original commit: elastic/x-pack-elasticsearch@abf90030fc
2017-11-16 18:38:13 +01:00
Dimitrios Athanasiou
80f4797f7a [ML][TEST] Add integ test for memory redution in population analysis
Original commit: elastic/x-pack-elasticsearch@ccd1f4835d
2017-11-16 17:10:59 +00:00
Igor Motov
8a1dd59178 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@4805de1874
2017-11-15 17:27:45 -05:00
Tim Brooks
d45c2d784c Update x-pack to support TcpChannel (elastic/x-pack-elasticsearch#2983)
This is related to elastic/elasticsearch#27132. This commit updates
x-pack to support signature changes introduced in that PR.

Original commit: elastic/x-pack-elasticsearch@461c96ffe6
2017-11-15 12:38:56 -07:00
Lee Hinman
889b008298 Add "client-api-objects" dependency for xpack plugin and transport-client (elastic/x-pack-elasticsearch#2995)
* Add "client-api-objects" dependency for xpack plugin and transport-client

This adds another gradle project, "client-api-objects" which is intended to be a
common dependency so that the xpack plugin and transport-client can share the
same Request and Response objects.

Relates to elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a6d83299d0
2017-11-15 09:49:00 -07:00
Tim Brooks
5d47b96905 Modify x-pack to for TransportChannel changes (elastic/x-pack-elasticsearch#3013)
This is related to elastic/elasticsearch#elastic/x-pack-elasticsearch#27388. It modifies x-pack to
be compatible with the removal of the delegating transport channel.

Original commit: elastic/x-pack-elasticsearch@3bd7bf6773
2017-11-15 09:48:17 -07:00
Alexander Reelsen
36105231c3 Watcher: Return useful error message when no accounts are found (elastic/x-pack-elasticsearch#2897)
When there were no accounts configured, watcher returned a cryptic
error message containing 'null' in the description. This fix returns
a more clear error message.

On top a dedicated NotificationServiceTests class was added to remove
redundant test cases in the hipchat/jira/slack unit tests, that all
basically tested NotificationService capabilties.

relates elastic/x-pack-elasticsearch#2666

Original commit: elastic/x-pack-elasticsearch@45d0d1df31
2017-11-15 11:51:23 +01:00
Alexander Reelsen
f286f9b0f2 Watcher: Remove unused test helper class
Original commit: elastic/x-pack-elasticsearch@09f66a5f52
2017-11-15 11:05:59 +01:00
Clinton Gormley
d833af2046 Rest spec fixes (elastic/x-pack-elasticsearch#2965)
* Rename REST spec xpack.deprecation.info to xpack.migration.deprecations

* Fixed parameter-type naming in xpack.ml.get_model_snapshots

* Fixed QS multi-cluster search test to use cluster.remote_info

Original commit: elastic/x-pack-elasticsearch@ccd35b4a6c
2017-11-15 09:33:19 +01:00
Martijn van Groningen
56e7512500 Make the persistent task status available to PersistentTasksExecutor.nodeOperation(...) method
Original commit: elastic/x-pack-elasticsearch@ce6d788e77
2017-11-15 07:54:05 +01:00
Martijn van Groningen
439830890f Move the initialization of persistent tasks from the machine learning level to xpack level.
Today persistent tasks is only usable from machine learning, but others like ccr will need to use it too.
With this change ccr and other will be able to make use of it too.

Original commit: elastic/x-pack-elasticsearch@c90f01d5f6
2017-11-15 07:44:53 +01:00
Nik Everett
1a434636fe Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@4c504025ce
2017-11-14 14:30:12 -05:00
jaymode
2f8cd77349 Test: TribeWithSecurityIT should wait for security index to be writeable
This commit adds checks to the TribeWithSecurityIT tests to ensure that the security index is
writeable before making modification operations. Otherwise, we hit errors in tests that are not
always reproducible.

relates elastic/x-pack-elasticsearch#2977

Original commit: elastic/x-pack-elasticsearch@c29bdff7ae
2017-11-14 08:18:55 -07:00
Jay Modi
5888174f24 Cleanup leftover tribe references in the plugin build.gradle file (elastic/x-pack-elasticsearch#2987)
In elastic/x-pack-elasticsearch#2901, the dependency on the tribe module was removed but a few leftover references were missed
in the build.gradle file of the x-pack-elasticsearch plugin. This commit removes these leftover
references.

Original commit: elastic/x-pack-elasticsearch@03f1cae1f5
2017-11-14 08:11:21 -07:00
Alexander Reelsen
dc42887396 Watcher: Move watcher-only packages into watcher hierarchy (elastic/x-pack-elasticsearch#2933)
In order to prepare for separate source directories, this commit moves
a few packages back into the watcher namespaces. A few of them have been
moved out previously as we thought that it might make sense to have a
dedicated notification API. This wont be the case for watcher on ES
anymore, so we can safely move those back into the watcher space.

Packages affected by this move:

* org.elasticsearch.xpack.common.http
* org.elasticsearch.xpack.common.text
* org.elasticsearch.xpack.common.secret
* org.elasticsearch.xpack.common.stats
* org.elasticsearch.xpack.support
* org.elasticsearch.xpack.notification

Tests have been moved accordingly.

The class `XContentUtils` has been split into one implementation for
watcher and one for security as different methods were used.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@0aec64a7e2
2017-11-14 11:35:10 +01:00
Dimitris Athanasiou
e00b4bd197 [ML][TEST] Improve RevertModelSnapshotIT (elastic/x-pack-elasticsearch#2954)
X-Pack side testing of elastic/machine-learning-cpp#390



Original commit: elastic/x-pack-elasticsearch@64e8ca85eb
2017-11-14 10:32:42 +00:00
Igor Motov
ea0e58f971 SQL: introduce setting to disable SQL (elastic/x-pack-elasticsearch#2966)
Adds xpack.sql.enabled setting to provide ability to disable SQL on elasticsearch nodes.

relates elastic/x-pack-elasticsearch#2872

Original commit: elastic/x-pack-elasticsearch@d13b72e9ea
2017-11-13 15:10:47 -05:00
Igor Motov
774f423d9e Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@79f79ea1c2
2017-11-13 15:09:35 -05:00
jaymode
1e30b12f97 Test: increase security index wait times
Our tests currently rely on waiting for the security index to be available in some cases and in
CI, these checks have been timing out. This commit increases the amount of time that we will wait
for the index before failing to account for slow machines.

Original commit: elastic/x-pack-elasticsearch@639dccd5cb
2017-11-13 12:32:06 -07:00
Chris Earle
def8c48dcb [Monitoring] Add "apm" to "stack_stats" for Phone Home (elastic/x-pack-elasticsearch#2848)
This checks if `apm-*` indices exist in the cluster to try to determine if APM is in use on the Elasticsearch cluster.

Original commit: elastic/x-pack-elasticsearch@7f9a9a4eee
2017-11-13 14:24:07 -05:00
Simon Willnauer
92f87cab14 Xpack side of elastic/elasticsearch#27161 (elastic/x-pack-elasticsearch#2851)
Original commit: elastic/x-pack-elasticsearch@31fdcdf583
2017-11-13 12:07:04 +01:00
Hendrik Muhs
fd392627a0 [ML-FC] Forecast status message rebase (elastic/x-pack-elasticsearch#2936)
Re-enables persistence of the forecast request stats document and adds more fields to it. Depends on https://github.com/elastic/machine-learning-cpp/pull/382.

Original commit: elastic/x-pack-elasticsearch@b6762005c0
2017-11-13 09:03:29 +01:00
Ryan Ernst
9a2ae4b7f2 Update security policy to use versionless codebase properties (elastic/x-pack-elasticsearch#2602)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/26756

Original commit: elastic/x-pack-elasticsearch@a219f5b6c0
2017-11-10 11:00:34 -08:00
David Roberts
742a052619 [ML] Account for the possibility of C++ log messages being UTF-16 (elastic/x-pack-elasticsearch#2952)
On Windows, log4cxx always writes to stderr in UTF-16, and we get the
logs from C++ to Java by redirecting stderr to our named pipe.  Hence
the log handler in Java needs to tolerate the log stream it's reading
being either UTF-16 (for Windows) or UTF-8 (for other platforms).

Fixes elastic/machine-learning-cpp#385

Original commit: elastic/x-pack-elasticsearch@89237d7125
2017-11-10 15:15:10 +00:00
Nik Everett
b2285ae66e Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b9d07ccd0f
2017-11-10 09:34:10 -05:00
Tim Vernum
039cf42fdf Extend wait time to 20s in SecurityIntegTest
This kind of sucks, because we shouldn't have to wait that long for tests to run.
But they're failing CI with some regularity, and we rely on these integration tests.

Original commit: elastic/x-pack-elasticsearch@3f4acb2a32
2017-11-10 15:20:30 +11:00
Tal Levy
8c489b1a98 Prevent 7.x nodes from joining cluster with un-upgraded 6.x .security indices (elastic/x-pack-elasticsearch#2921) (elastic/x-pack-elasticsearch#2940)
This is a forward-port of elastic/x-pack-elasticsearch/pull/2921.

original commit message:

Before this commit, a cluster with security enabled and backed by
native-realm user permissions allowed rolled upgrades to clusters without
upgrading the `.security` index. This resulted in the newly established
6.0 cluster not able to register the native-realm users previously established
in the `.security` index. In order to fix this, one would have to rely on file-based
users to re-configure and upgrade the `.security` index. Since this state is easily
avoidable with an upgrade, this commit rejects the joining of upgraded nodes without
upgrading the security index beforehand.

modifications:

Test with 7.x vs 6.x nodes.

Original commit: elastic/x-pack-elasticsearch@56f81bfb20
2017-11-09 12:49:59 -08:00
jaymode
9d85f377c7 Test: update the branch logic for BWC tests
This commit updates the logic for determining which branch to use to make it consistent with the
logic in elasticsearch. This change means that testing BWC within the same major picks the correct
branch.

Original commit: elastic/x-pack-elasticsearch@2d75d15c41
2017-11-09 13:03:40 -07:00
Chris Earle
efb5b8827b [Monitoring] Add Rolling Upgrade Tests (elastic/x-pack-elasticsearch#2832)
This adds a rolling upgrade test for X-Pack monitoring. It works by using the `_xpack/monitoring/_bulk` endpoint to send arbitrary data, then verify that it exists.

This forces a few things to happen, thereby testing the behavior: 

1. The templates must exist.
2. The elected master node must be "ready" to work (hence the first
point).
3. The same "system_api_version" is accepted by every version of ES.

Original commit: elastic/x-pack-elasticsearch@012e5738bb
2017-11-09 12:49:37 -05:00
Hendrik Muhs
2693c6a730 [ML] improve logging for autodetect crashes (elastic/x-pack-elasticsearch#2866)
Improving logging for unexpected autodetect termination (crash, oom). Output to the log pipe not conforming to the json log output format are treated as fatal error and logged, so that the crash as well as a proper error message if available gets logged.

Original commit: elastic/x-pack-elasticsearch@ae5d792d3f
2017-11-09 15:47:23 +01:00
David Roberts
142b59a4d5 [ML] Use the correct timeout for the process context lock (elastic/x-pack-elasticsearch#2935)
This change should have been made in elastic/x-pack-elasticsearch#2913.  Now we hold the process
context lock throughout the job close procedure, the timeout for trying
to lock it should be the timeout used for job open/close rather than the
timeout for connecting named pipes.

Original commit: elastic/x-pack-elasticsearch@79672b0825
2017-11-09 13:50:15 +00:00
Luca Cavanna
62b8e54247 Build: add aggs-matrix-stats to license mapping and ignore sha list (elastic/x-pack-elasticsearch#2932)
Original commit: elastic/x-pack-elasticsearch@d33a5b95bc
2017-11-09 11:32:36 +01:00
Jay Modi
e29649a7bc Remove the xpack plugin's dependency on the tribe module (elastic/x-pack-elasticsearch#2901)
This change removes the xpack plugin's dependency on the tribe module, which is not a published
artifact. For the most part this just involves moving some test classes around, but for the
security and tribe integration the usage of constant settings was removed and replaced with the
string names. This is a bit unfortunate, but a test was added in a QA project that depends on tribe
that will alert us if a new setting is added that we need to be aware of.

relates elastic/x-pack-elasticsearch#2656

Original commit: elastic/x-pack-elasticsearch@649a8033e4
2017-11-08 12:39:02 -07:00
jaymode
96d0a374a4 Test: fix check for security version after template updater change
This change fixes the check for the version of the security template after the template updater was
changed to only run on the master node in elastic/elasticsearch#27294. Additionally, the wait time
for the cluster to have a yellow status has been increased to account for delayed shards and slower
machines.

Original commit: elastic/x-pack-elasticsearch@a2e72bed12
2017-11-08 10:46:53 -07:00
David Kyle
cba4421a75 [ML] Fix streaming the process update request (elastic/x-pack-elasticsearch#2928)
Original commit: elastic/x-pack-elasticsearch@cf76c13a2b
2017-11-08 16:29:12 +00:00
Igor Motov
a72879acb2 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@f3b4897936
2017-11-08 11:01:45 -05:00
Christoph Büscher
17ae4899c8 [Test] Change expected exception type after changes in core
Original commit: elastic/x-pack-elasticsearch@0ad2e06970
2017-11-08 12:54:54 +01:00
David Roberts
7b25e7d9ed [ML] Remove Watcher middleman from ML dependency on core (elastic/x-pack-elasticsearch#2926)
I imagine this needless indirection arose from accepting the wrong
IntelliJ suggestion for an import.

Original commit: elastic/x-pack-elasticsearch@54d7e854d3
2017-11-08 10:33:48 +00:00
Tim Vernum
59b453e1c8 [Security] Fix concrete security index name (elastic/x-pack-elasticsearch#2905)
The 5.6 Upgrade API will reindex .security to .security-6 and create a .security alias.
But the 6.0 default was to create a .security-v6 index and a .security alias if none existed (e.g. fresh x-pack install)

Having two different index names based on the method of install/upgrade complicates the code and testing, so we're unifying on the .security-6 index name that already exists in the wild.

Original commit: elastic/x-pack-elasticsearch@d78f569c5f
2017-11-08 10:22:42 +11:00
David Roberts
027f64b221 [ML] Fix a race condition simultaneous close requests are made for a job (elastic/x-pack-elasticsearch#2913)
When simultaneous close requests were made for the same job it was possible
that one of the requests would inappropriately log error messages about the
job having failed.  This change prevents that problem, whilst continuing to
adhere to the requirement that close requests for already closing jobs do not
return until the close request that is doing the work completes.

relates elastic/x-pack-elasticsearch#2912

Original commit: elastic/x-pack-elasticsearch@513b7fa1d6
2017-11-07 14:30:59 +00:00
Tim Vernum
8e5855e62e Allow XPack user read-only access to index audit log (elastic/x-pack-elasticsearch#2906)
The default internal XPack user no longer has access to the security index, but it should have read-only access to the audit log so that watches can be triggered based on audit events (but cannot write audit records)

Original commit: elastic/x-pack-elasticsearch@5c37720dad
2017-11-07 19:31:24 +11:00