1608 Commits

Author SHA1 Message Date
Lee Hinman
889b008298 Add "client-api-objects" dependency for xpack plugin and transport-client (elastic/x-pack-elasticsearch#2995)
* Add "client-api-objects" dependency for xpack plugin and transport-client

This adds another gradle project, "client-api-objects" which is intended to be a
common dependency so that the xpack plugin and transport-client can share the
same Request and Response objects.

Relates to elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a6d83299d0
2017-11-15 09:49:00 -07:00
Tim Brooks
5d47b96905 Modify x-pack to for TransportChannel changes (elastic/x-pack-elasticsearch#3013)
This is related to elastic/elasticsearch#elastic/x-pack-elasticsearch#27388. It modifies x-pack to
be compatible with the removal of the delegating transport channel.

Original commit: elastic/x-pack-elasticsearch@3bd7bf6773
2017-11-15 09:48:17 -07:00
Alexander Reelsen
36105231c3 Watcher: Return useful error message when no accounts are found (elastic/x-pack-elasticsearch#2897)
When there were no accounts configured, watcher returned a cryptic
error message containing 'null' in the description. This fix returns
a more clear error message.

On top a dedicated NotificationServiceTests class was added to remove
redundant test cases in the hipchat/jira/slack unit tests, that all
basically tested NotificationService capabilties.

relates elastic/x-pack-elasticsearch#2666

Original commit: elastic/x-pack-elasticsearch@45d0d1df31
2017-11-15 11:51:23 +01:00
Alexander Reelsen
f286f9b0f2 Watcher: Remove unused test helper class
Original commit: elastic/x-pack-elasticsearch@09f66a5f52
2017-11-15 11:05:59 +01:00
Clinton Gormley
d833af2046 Rest spec fixes (elastic/x-pack-elasticsearch#2965)
* Rename REST spec xpack.deprecation.info to xpack.migration.deprecations

* Fixed parameter-type naming in xpack.ml.get_model_snapshots

* Fixed QS multi-cluster search test to use cluster.remote_info

Original commit: elastic/x-pack-elasticsearch@ccd35b4a6c
2017-11-15 09:33:19 +01:00
Martijn van Groningen
56e7512500 Make the persistent task status available to PersistentTasksExecutor.nodeOperation(...) method
Original commit: elastic/x-pack-elasticsearch@ce6d788e77
2017-11-15 07:54:05 +01:00
Martijn van Groningen
439830890f Move the initialization of persistent tasks from the machine learning level to xpack level.
Today persistent tasks is only usable from machine learning, but others like ccr will need to use it too.
With this change ccr and other will be able to make use of it too.

Original commit: elastic/x-pack-elasticsearch@c90f01d5f6
2017-11-15 07:44:53 +01:00
Nik Everett
1a434636fe Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@4c504025ce
2017-11-14 14:30:12 -05:00
jaymode
2f8cd77349 Test: TribeWithSecurityIT should wait for security index to be writeable
This commit adds checks to the TribeWithSecurityIT tests to ensure that the security index is
writeable before making modification operations. Otherwise, we hit errors in tests that are not
always reproducible.

relates elastic/x-pack-elasticsearch#2977

Original commit: elastic/x-pack-elasticsearch@c29bdff7ae
2017-11-14 08:18:55 -07:00
Jay Modi
5888174f24 Cleanup leftover tribe references in the plugin build.gradle file (elastic/x-pack-elasticsearch#2987)
In elastic/x-pack-elasticsearch#2901, the dependency on the tribe module was removed but a few leftover references were missed
in the build.gradle file of the x-pack-elasticsearch plugin. This commit removes these leftover
references.

Original commit: elastic/x-pack-elasticsearch@03f1cae1f5
2017-11-14 08:11:21 -07:00
Alexander Reelsen
dc42887396 Watcher: Move watcher-only packages into watcher hierarchy (elastic/x-pack-elasticsearch#2933)
In order to prepare for separate source directories, this commit moves
a few packages back into the watcher namespaces. A few of them have been
moved out previously as we thought that it might make sense to have a
dedicated notification API. This wont be the case for watcher on ES
anymore, so we can safely move those back into the watcher space.

Packages affected by this move:

* org.elasticsearch.xpack.common.http
* org.elasticsearch.xpack.common.text
* org.elasticsearch.xpack.common.secret
* org.elasticsearch.xpack.common.stats
* org.elasticsearch.xpack.support
* org.elasticsearch.xpack.notification

Tests have been moved accordingly.

The class `XContentUtils` has been split into one implementation for
watcher and one for security as different methods were used.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@0aec64a7e2
2017-11-14 11:35:10 +01:00
Dimitris Athanasiou
e00b4bd197 [ML][TEST] Improve RevertModelSnapshotIT (elastic/x-pack-elasticsearch#2954)
X-Pack side testing of elastic/machine-learning-cpp#390



Original commit: elastic/x-pack-elasticsearch@64e8ca85eb
2017-11-14 10:32:42 +00:00
Igor Motov
ea0e58f971 SQL: introduce setting to disable SQL (elastic/x-pack-elasticsearch#2966)
Adds xpack.sql.enabled setting to provide ability to disable SQL on elasticsearch nodes.

relates elastic/x-pack-elasticsearch#2872

Original commit: elastic/x-pack-elasticsearch@d13b72e9ea
2017-11-13 15:10:47 -05:00
Igor Motov
774f423d9e Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@79f79ea1c2
2017-11-13 15:09:35 -05:00
jaymode
1e30b12f97 Test: increase security index wait times
Our tests currently rely on waiting for the security index to be available in some cases and in
CI, these checks have been timing out. This commit increases the amount of time that we will wait
for the index before failing to account for slow machines.

Original commit: elastic/x-pack-elasticsearch@639dccd5cb
2017-11-13 12:32:06 -07:00
Chris Earle
def8c48dcb [Monitoring] Add "apm" to "stack_stats" for Phone Home (elastic/x-pack-elasticsearch#2848)
This checks if `apm-*` indices exist in the cluster to try to determine if APM is in use on the Elasticsearch cluster.

Original commit: elastic/x-pack-elasticsearch@7f9a9a4eee
2017-11-13 14:24:07 -05:00
Simon Willnauer
92f87cab14 Xpack side of elastic/elasticsearch#27161 (elastic/x-pack-elasticsearch#2851)
Original commit: elastic/x-pack-elasticsearch@31fdcdf583
2017-11-13 12:07:04 +01:00
Hendrik Muhs
fd392627a0 [ML-FC] Forecast status message rebase (elastic/x-pack-elasticsearch#2936)
Re-enables persistence of the forecast request stats document and adds more fields to it. Depends on https://github.com/elastic/machine-learning-cpp/pull/382.

Original commit: elastic/x-pack-elasticsearch@b6762005c0
2017-11-13 09:03:29 +01:00
Ryan Ernst
9a2ae4b7f2 Update security policy to use versionless codebase properties (elastic/x-pack-elasticsearch#2602)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/26756

Original commit: elastic/x-pack-elasticsearch@a219f5b6c0
2017-11-10 11:00:34 -08:00
David Roberts
742a052619 [ML] Account for the possibility of C++ log messages being UTF-16 (elastic/x-pack-elasticsearch#2952)
On Windows, log4cxx always writes to stderr in UTF-16, and we get the
logs from C++ to Java by redirecting stderr to our named pipe.  Hence
the log handler in Java needs to tolerate the log stream it's reading
being either UTF-16 (for Windows) or UTF-8 (for other platforms).

Fixes elastic/machine-learning-cpp#385

Original commit: elastic/x-pack-elasticsearch@89237d7125
2017-11-10 15:15:10 +00:00
Nik Everett
b2285ae66e Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b9d07ccd0f
2017-11-10 09:34:10 -05:00
Tim Vernum
039cf42fdf Extend wait time to 20s in SecurityIntegTest
This kind of sucks, because we shouldn't have to wait that long for tests to run.
But they're failing CI with some regularity, and we rely on these integration tests.

Original commit: elastic/x-pack-elasticsearch@3f4acb2a32
2017-11-10 15:20:30 +11:00
Tal Levy
8c489b1a98 Prevent 7.x nodes from joining cluster with un-upgraded 6.x .security indices (elastic/x-pack-elasticsearch#2921) (elastic/x-pack-elasticsearch#2940)
This is a forward-port of elastic/x-pack-elasticsearch/pull/2921.

original commit message:

Before this commit, a cluster with security enabled and backed by
native-realm user permissions allowed rolled upgrades to clusters without
upgrading the `.security` index. This resulted in the newly established
6.0 cluster not able to register the native-realm users previously established
in the `.security` index. In order to fix this, one would have to rely on file-based
users to re-configure and upgrade the `.security` index. Since this state is easily
avoidable with an upgrade, this commit rejects the joining of upgraded nodes without
upgrading the security index beforehand.

modifications:

Test with 7.x vs 6.x nodes.

Original commit: elastic/x-pack-elasticsearch@56f81bfb20
2017-11-09 12:49:59 -08:00
jaymode
9d85f377c7 Test: update the branch logic for BWC tests
This commit updates the logic for determining which branch to use to make it consistent with the
logic in elasticsearch. This change means that testing BWC within the same major picks the correct
branch.

Original commit: elastic/x-pack-elasticsearch@2d75d15c41
2017-11-09 13:03:40 -07:00
Chris Earle
efb5b8827b [Monitoring] Add Rolling Upgrade Tests (elastic/x-pack-elasticsearch#2832)
This adds a rolling upgrade test for X-Pack monitoring. It works by using the `_xpack/monitoring/_bulk` endpoint to send arbitrary data, then verify that it exists.

This forces a few things to happen, thereby testing the behavior: 

1. The templates must exist.
2. The elected master node must be "ready" to work (hence the first
point).
3. The same "system_api_version" is accepted by every version of ES.

Original commit: elastic/x-pack-elasticsearch@012e5738bb
2017-11-09 12:49:37 -05:00
Hendrik Muhs
2693c6a730 [ML] improve logging for autodetect crashes (elastic/x-pack-elasticsearch#2866)
Improving logging for unexpected autodetect termination (crash, oom). Output to the log pipe not conforming to the json log output format are treated as fatal error and logged, so that the crash as well as a proper error message if available gets logged.

Original commit: elastic/x-pack-elasticsearch@ae5d792d3f
2017-11-09 15:47:23 +01:00
David Roberts
142b59a4d5 [ML] Use the correct timeout for the process context lock (elastic/x-pack-elasticsearch#2935)
This change should have been made in elastic/x-pack-elasticsearch#2913.  Now we hold the process
context lock throughout the job close procedure, the timeout for trying
to lock it should be the timeout used for job open/close rather than the
timeout for connecting named pipes.

Original commit: elastic/x-pack-elasticsearch@79672b0825
2017-11-09 13:50:15 +00:00
Luca Cavanna
62b8e54247 Build: add aggs-matrix-stats to license mapping and ignore sha list (elastic/x-pack-elasticsearch#2932)
Original commit: elastic/x-pack-elasticsearch@d33a5b95bc
2017-11-09 11:32:36 +01:00
Jay Modi
e29649a7bc Remove the xpack plugin's dependency on the tribe module (elastic/x-pack-elasticsearch#2901)
This change removes the xpack plugin's dependency on the tribe module, which is not a published
artifact. For the most part this just involves moving some test classes around, but for the
security and tribe integration the usage of constant settings was removed and replaced with the
string names. This is a bit unfortunate, but a test was added in a QA project that depends on tribe
that will alert us if a new setting is added that we need to be aware of.

relates elastic/x-pack-elasticsearch#2656

Original commit: elastic/x-pack-elasticsearch@649a8033e4
2017-11-08 12:39:02 -07:00
jaymode
96d0a374a4 Test: fix check for security version after template updater change
This change fixes the check for the version of the security template after the template updater was
changed to only run on the master node in elastic/elasticsearch#27294. Additionally, the wait time
for the cluster to have a yellow status has been increased to account for delayed shards and slower
machines.

Original commit: elastic/x-pack-elasticsearch@a2e72bed12
2017-11-08 10:46:53 -07:00
David Kyle
cba4421a75 [ML] Fix streaming the process update request (elastic/x-pack-elasticsearch#2928)
Original commit: elastic/x-pack-elasticsearch@cf76c13a2b
2017-11-08 16:29:12 +00:00
Igor Motov
a72879acb2 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@f3b4897936
2017-11-08 11:01:45 -05:00
Christoph Büscher
17ae4899c8 [Test] Change expected exception type after changes in core
Original commit: elastic/x-pack-elasticsearch@0ad2e06970
2017-11-08 12:54:54 +01:00
David Roberts
7b25e7d9ed [ML] Remove Watcher middleman from ML dependency on core (elastic/x-pack-elasticsearch#2926)
I imagine this needless indirection arose from accepting the wrong
IntelliJ suggestion for an import.

Original commit: elastic/x-pack-elasticsearch@54d7e854d3
2017-11-08 10:33:48 +00:00
Tim Vernum
59b453e1c8 [Security] Fix concrete security index name (elastic/x-pack-elasticsearch#2905)
The 5.6 Upgrade API will reindex .security to .security-6 and create a .security alias.
But the 6.0 default was to create a .security-v6 index and a .security alias if none existed (e.g. fresh x-pack install)

Having two different index names based on the method of install/upgrade complicates the code and testing, so we're unifying on the .security-6 index name that already exists in the wild.

Original commit: elastic/x-pack-elasticsearch@d78f569c5f
2017-11-08 10:22:42 +11:00
David Roberts
027f64b221 [ML] Fix a race condition simultaneous close requests are made for a job (elastic/x-pack-elasticsearch#2913)
When simultaneous close requests were made for the same job it was possible
that one of the requests would inappropriately log error messages about the
job having failed.  This change prevents that problem, whilst continuing to
adhere to the requirement that close requests for already closing jobs do not
return until the close request that is doing the work completes.

relates elastic/x-pack-elasticsearch#2912

Original commit: elastic/x-pack-elasticsearch@513b7fa1d6
2017-11-07 14:30:59 +00:00
Tim Vernum
8e5855e62e Allow XPack user read-only access to index audit log (elastic/x-pack-elasticsearch#2906)
The default internal XPack user no longer has access to the security index, but it should have read-only access to the audit log so that watches can be triggered based on audit events (but cannot write audit records)

Original commit: elastic/x-pack-elasticsearch@5c37720dad
2017-11-07 19:31:24 +11:00
Igor Motov
de33803d85 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@bfdbc2bb75
2017-11-06 18:51:23 -05:00
Jay Modi
be773363c9 Do not fail requests on exceptions from native roles store (elastic/x-pack-elasticsearch#2857)
This commit changes the handling of exceptions when retrieving roles from the native roles store.
Previously, exceptions would have caused the request to terminate and the exception would be
sent back to the user. This makes for a bad experience when a cluster hasn't been upgraded to the
latest index format and anonymous access is enabled with a native role as all requests without
preemptive basic authentication would result in an exception. The change here is to allow the
request to continue processing. Once the security index is up to date, the roles cache is cleared
so that the native roles can be picked up.

relates elastic/x-pack-elasticsearch#2686

Original commit: elastic/x-pack-elasticsearch@ef5149140f
2017-11-06 10:27:56 -07:00
Simon Willnauer
457c49c332 Apply Renames from split shards (elastic/x-pack-elasticsearch#2716)
XPack side of elastic/elasticsearch#26931

Original commit: elastic/x-pack-elasticsearch@6e7c3d4242
2017-11-06 11:38:21 +01:00
Tim Vernum
dd3a800745 Fix ASN.1 encoding of "cn" OtherName in CertGen/CertUtil (elastic/x-pack-elasticsearch#2858)
Certgen was generating "other name" SANs without the explicit [0] tag that is required.
This was masked by the fact that the JRE X.509 classes always wrap the "other name" name-value in a [0] tag  (even if it already has one)

Also switched to a UTF8 String from an IA5 string to match the configuration being used for testing in openssl.

Original commit: elastic/x-pack-elasticsearch@1b87964ec7
2017-11-06 10:04:17 +11:00
Nik Everett
00d30285e1 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@33905ed7be
2017-11-04 19:12:40 -04:00
David Roberts
7b36046f33 Use TestEnvironment factory method to create test Environment objects (elastic/x-pack-elasticsearch#2860)
This is the X-Pack side of elastic/elasticsearch#27235.  To force people
who construct an Environment object in production code to think about the
correct setting of configPath there is no longer a single argument
constructor in the Environment class.  Instead there is a factory method
in the test framework to replace it.  Having this in the test framework
ensures that there is no way to use it in production code.

Original commit: elastic/x-pack-elasticsearch@4860e92d90
2017-11-04 13:25:56 +00:00
Nik Everett
d96317fbca SQL: fix known actions test after action move
Original commit: elastic/x-pack-elasticsearch@a44932a25a
2017-11-03 21:58:09 -04:00
Nik Everett
1bf1521b37 SQL: Remove a package (elastic/x-pack-elasticsearch#2837)
Collapses a package into its parent package because they only contain a
single class together. No need for two layers.

Original commit: elastic/x-pack-elasticsearch@c947e57952
2017-11-04 01:54:21 +00:00
Nik Everett
25a0ec42f6 SQL: Reorganize the translate action (elastic/x-pack-elasticsearch#2841)
Organizes the SQL translate action to match the way that x-pack has been
organizing new actions for a while. All of the pieces are put into the
same class file.

Original commit: elastic/x-pack-elasticsearch@def911c0ab
2017-11-03 23:25:52 +00:00
Chris Earle
5b85453c9a [TEST] Use the test's Settings.Builder in the test rather than Settings.EMPTY
Original commit: elastic/x-pack-elasticsearch@5b7d7bc652
2017-11-03 16:56:01 -04:00
Nik Everett
562117a7b7 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@c8368be422
2017-11-03 16:16:50 -04:00
David Roberts
ba5dbc4daf Remove uses of single argument Environment constructor from production code (elastic/x-pack-elasticsearch#2852)
Following elastic/elasticsearch#27235 the single argument Environment constructor
is forbidden in production code.  This change removes the last such uses from
X-Pack.

Original commit: elastic/x-pack-elasticsearch@87e72d0d07
2017-11-03 09:12:35 +00:00
David Roberts
14211b47f2 Use the correct Environment object in NativeControllerHolder (elastic/x-pack-elasticsearch#2847)
We should not be constructing a temporary Environment object in production
code.  This currently isn't causing any problems, but it might in the future
if elastic/elasticsearch#27144 or something similar is ever merged.  Instead
the master Environment of the node should always be used.

Original commit: elastic/x-pack-elasticsearch@6276a54a45
2017-11-02 16:36:09 +00:00