Commit Graph

1573 Commits

Author SHA1 Message Date
jaymode 34ecd18e76 Remove use of forbidden API URL#getPath
tool. However, this is a forbidden API so this commit replaces it with URI#getPath. Additionally,
the tests fail with a security manager permission error due to the use of Mockito for exception
throwing. This commit still uses Mockito for throwing exceptions but does it differently in a way
that is acceptable by our test security policy.

Original commit: elastic/x-pack-elasticsearch@5e1d45acf8
2017-11-17 14:03:26 -07:00
Igor Motov d011247970 SQL: Fix testJdbcActionLicense test
The testJdbcActionLicense test broke because we changed the error serialization mechanism in elastic/x-pack-elasticsearch#3034. This commit updates that check to capture an exception instead of checking for ErrorResponse.

Original commit: elastic/x-pack-elasticsearch@17c41426d0
2017-11-17 15:37:02 -05:00
Igor Motov ffcd54de77 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@e67ceb1d1e
2017-11-17 13:33:28 -05:00
Albert Zaharovits 5b73c77011 SetupPasswordTool handle url cmd option correctly (elastic/x-pack-elasticsearch#2899)
Fixes bug when the url option had trailing slashes. The URL built
was invalid (consecutive fwd slashes) but the failure errors of
the subsequent requests were ignored.

URL is built correctly from the option spec.
True HTTP errors and Exceptions are logged and the cmd fails.

relates elastic/x-pack-elasticsearch#2778

Original commit: elastic/x-pack-elasticsearch@62b2d94ca0
2017-11-17 20:30:22 +02:00
Nhat Nguyen 418a6632e5 Registers a new handler for shard snapshot status
Relates elastic/x-pack-elasticsearch#27165

Original commit: elastic/x-pack-elasticsearch@11199642bb
2017-11-17 13:13:50 -05:00
Nik Everett b8e082107f Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@bbc72c0368
2017-11-17 12:05:47 -05:00
David Roberts 6741a62e80 [ML] Adjust memory limit test to account for greater accuracy (elastic/x-pack-elasticsearch#3047)
Due to elastic/machine-learning-cpp#409 the ML C++ code now instruments
memory more accurately.  This change modifies the expectations of the Java
integration test to account for the change.

Original commit: elastic/x-pack-elasticsearch@2ed7a75af4
2017-11-17 17:02:19 +00:00
Alexander Reelsen 0f97e28074 Watcher: Further preparations for source repo split (elastic/x-pack-elasticsearch#3006)
Changes to further prepare for feature split with regards to watcher:

- CryptoService has been moved into watcher
- CryptoService.generateKey() has been moved into SystemKeyTools, only
  used there
- The creation of the http client/notification classes have been moved
  into watcher, no further dependencies on watcher in XPackPlugin
- Each subproject now registers it's own named writeables

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a60c98ba7e
2017-11-17 17:05:07 +01:00
Alexander Reelsen 3d5fb54522 Watcher: Create templates on nodes newer than the master (elastic/x-pack-elasticsearch#2950)
This problem was introduced due to distributed watch execution.

When a node newer than the master node joins the cluster and gets a
.watches shard assigned it is supposed to start watcher. However
when a new version of the watch history template is part of that new
node (and we might increase that version anytime), this template does
not get installed, because only the master node is updating watcher
templates.

This commit checks if the local node version is higher than the master
node version and then also puts missing templates.

Currently this is done for all watcher templates, not only the watcher history.

relates elastic/x-pack-elasticsearch#2944

Original commit: elastic/x-pack-elasticsearch@4960231ea7
2017-11-17 16:56:29 +01:00
Hendrik Muhs a583cf270b [ML-FC] implement endpoint parameter that takes a duration (elastic/x-pack-elasticsearch#3027)
Adds a duration parameter to the forecast API. Also fixes issue if no parameter is given (forecast 1 day from time of last bucket), in which case it lets autodetect decide

depends on elastic/machine-learning-cpp#407

Original commit: elastic/x-pack-elasticsearch@3387478872
2017-11-17 06:37:03 +01:00
Tim Brooks f94e6d427b Modify x-pack to for `TcpTransport` changes
This is related to elatic/elasticsearch#27407. That commit removed
parameterization from TcpTranport. This commit modifies the security
transport to be compatible with those changes.

Original commit: elastic/x-pack-elasticsearch@9878c26e14
2017-11-16 14:32:40 -07:00
Hendrik Muhs 5efdb54eb8 add memoryUsage stats (elastic/x-pack-elasticsearch#3025)
adds memory usage statistics to forecast request stats

depends on: elastic/machine-learning-cpp#406

Original commit: elastic/x-pack-elasticsearch@abf90030fc
2017-11-16 18:38:13 +01:00
Dimitrios Athanasiou 80f4797f7a [ML][TEST] Add integ test for memory redution in population analysis
Original commit: elastic/x-pack-elasticsearch@ccd1f4835d
2017-11-16 17:10:59 +00:00
Igor Motov 8a1dd59178 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@4805de1874
2017-11-15 17:27:45 -05:00
Tim Brooks d45c2d784c Update x-pack to support TcpChannel (elastic/x-pack-elasticsearch#2983)
This is related to elastic/elasticsearch#27132. This commit updates
x-pack to support signature changes introduced in that PR.

Original commit: elastic/x-pack-elasticsearch@461c96ffe6
2017-11-15 12:38:56 -07:00
Lee Hinman 889b008298 Add "client-api-objects" dependency for xpack plugin and transport-client (elastic/x-pack-elasticsearch#2995)
* Add "client-api-objects" dependency for xpack plugin and transport-client

This adds another gradle project, "client-api-objects" which is intended to be a
common dependency so that the xpack plugin and transport-client can share the
same Request and Response objects.

Relates to elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@a6d83299d0
2017-11-15 09:49:00 -07:00
Tim Brooks 5d47b96905 Modify x-pack to for `TransportChannel` changes (elastic/x-pack-elasticsearch#3013)
This is related to elastic/elasticsearch#elastic/x-pack-elasticsearch#27388. It modifies x-pack to
be compatible with the removal of the delegating transport channel.

Original commit: elastic/x-pack-elasticsearch@3bd7bf6773
2017-11-15 09:48:17 -07:00
Alexander Reelsen 36105231c3 Watcher: Return useful error message when no accounts are found (elastic/x-pack-elasticsearch#2897)
When there were no accounts configured, watcher returned a cryptic
error message containing 'null' in the description. This fix returns
a more clear error message.

On top a dedicated NotificationServiceTests class was added to remove
redundant test cases in the hipchat/jira/slack unit tests, that all
basically tested NotificationService capabilties.

relates elastic/x-pack-elasticsearch#2666

Original commit: elastic/x-pack-elasticsearch@45d0d1df31
2017-11-15 11:51:23 +01:00
Alexander Reelsen f286f9b0f2 Watcher: Remove unused test helper class
Original commit: elastic/x-pack-elasticsearch@09f66a5f52
2017-11-15 11:05:59 +01:00
Clinton Gormley d833af2046 Rest spec fixes (elastic/x-pack-elasticsearch#2965)
* Rename REST spec xpack.deprecation.info to xpack.migration.deprecations

* Fixed parameter-type naming in xpack.ml.get_model_snapshots

* Fixed QS multi-cluster search test to use cluster.remote_info

Original commit: elastic/x-pack-elasticsearch@ccd35b4a6c
2017-11-15 09:33:19 +01:00
Martijn van Groningen 56e7512500 Make the persistent task status available to PersistentTasksExecutor.nodeOperation(...) method
Original commit: elastic/x-pack-elasticsearch@ce6d788e77
2017-11-15 07:54:05 +01:00
Martijn van Groningen 439830890f Move the initialization of persistent tasks from the machine learning level to xpack level.
Today persistent tasks is only usable from machine learning, but others like ccr will need to use it too.
With this change ccr and other will be able to make use of it too.

Original commit: elastic/x-pack-elasticsearch@c90f01d5f6
2017-11-15 07:44:53 +01:00
Nik Everett 1a434636fe Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@4c504025ce
2017-11-14 14:30:12 -05:00
jaymode 2f8cd77349 Test: TribeWithSecurityIT should wait for security index to be writeable
This commit adds checks to the TribeWithSecurityIT tests to ensure that the security index is
writeable before making modification operations. Otherwise, we hit errors in tests that are not
always reproducible.

relates elastic/x-pack-elasticsearch#2977

Original commit: elastic/x-pack-elasticsearch@c29bdff7ae
2017-11-14 08:18:55 -07:00
Jay Modi 5888174f24 Cleanup leftover tribe references in the plugin build.gradle file (elastic/x-pack-elasticsearch#2987)
In elastic/x-pack-elasticsearch#2901, the dependency on the tribe module was removed but a few leftover references were missed
in the build.gradle file of the x-pack-elasticsearch plugin. This commit removes these leftover
references.

Original commit: elastic/x-pack-elasticsearch@03f1cae1f5
2017-11-14 08:11:21 -07:00
Alexander Reelsen dc42887396 Watcher: Move watcher-only packages into watcher hierarchy (elastic/x-pack-elasticsearch#2933)
In order to prepare for separate source directories, this commit moves
a few packages back into the watcher namespaces. A few of them have been
moved out previously as we thought that it might make sense to have a
dedicated notification API. This wont be the case for watcher on ES
anymore, so we can safely move those back into the watcher space.

Packages affected by this move:

* org.elasticsearch.xpack.common.http
* org.elasticsearch.xpack.common.text
* org.elasticsearch.xpack.common.secret
* org.elasticsearch.xpack.common.stats
* org.elasticsearch.xpack.support
* org.elasticsearch.xpack.notification

Tests have been moved accordingly.

The class `XContentUtils` has been split into one implementation for
watcher and one for security as different methods were used.

Relates elastic/x-pack-elasticsearch#2925

Original commit: elastic/x-pack-elasticsearch@0aec64a7e2
2017-11-14 11:35:10 +01:00
Dimitris Athanasiou e00b4bd197 [ML][TEST] Improve RevertModelSnapshotIT (elastic/x-pack-elasticsearch#2954)
X-Pack side testing of elastic/machine-learning-cpp#390



Original commit: elastic/x-pack-elasticsearch@64e8ca85eb
2017-11-14 10:32:42 +00:00
Igor Motov ea0e58f971 SQL: introduce setting to disable SQL (elastic/x-pack-elasticsearch#2966)
Adds xpack.sql.enabled setting to provide ability to disable SQL on elasticsearch nodes.

relates elastic/x-pack-elasticsearch#2872

Original commit: elastic/x-pack-elasticsearch@d13b72e9ea
2017-11-13 15:10:47 -05:00
Igor Motov 774f423d9e Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@79f79ea1c2
2017-11-13 15:09:35 -05:00
jaymode 1e30b12f97 Test: increase security index wait times
Our tests currently rely on waiting for the security index to be available in some cases and in
CI, these checks have been timing out. This commit increases the amount of time that we will wait
for the index before failing to account for slow machines.

Original commit: elastic/x-pack-elasticsearch@639dccd5cb
2017-11-13 12:32:06 -07:00
Chris Earle def8c48dcb [Monitoring] Add "apm" to "stack_stats" for Phone Home (elastic/x-pack-elasticsearch#2848)
This checks if `apm-*` indices exist in the cluster to try to determine if APM is in use on the Elasticsearch cluster.

Original commit: elastic/x-pack-elasticsearch@7f9a9a4eee
2017-11-13 14:24:07 -05:00
Simon Willnauer 92f87cab14 Xpack side of elastic/elasticsearch#27161 (elastic/x-pack-elasticsearch#2851)
Original commit: elastic/x-pack-elasticsearch@31fdcdf583
2017-11-13 12:07:04 +01:00
Hendrik Muhs fd392627a0 [ML-FC] Forecast status message rebase (elastic/x-pack-elasticsearch#2936)
Re-enables persistence of the forecast request stats document and adds more fields to it. Depends on https://github.com/elastic/machine-learning-cpp/pull/382.

Original commit: elastic/x-pack-elasticsearch@b6762005c0
2017-11-13 09:03:29 +01:00
Ryan Ernst 9a2ae4b7f2 Update security policy to use versionless codebase properties (elastic/x-pack-elasticsearch#2602)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/26756

Original commit: elastic/x-pack-elasticsearch@a219f5b6c0
2017-11-10 11:00:34 -08:00
David Roberts 742a052619 [ML] Account for the possibility of C++ log messages being UTF-16 (elastic/x-pack-elasticsearch#2952)
On Windows, log4cxx always writes to stderr in UTF-16, and we get the
logs from C++ to Java by redirecting stderr to our named pipe.  Hence
the log handler in Java needs to tolerate the log stream it's reading
being either UTF-16 (for Windows) or UTF-8 (for other platforms).

Fixes elastic/machine-learning-cpp#385

Original commit: elastic/x-pack-elasticsearch@89237d7125
2017-11-10 15:15:10 +00:00
Nik Everett b2285ae66e Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b9d07ccd0f
2017-11-10 09:34:10 -05:00
Tim Vernum 039cf42fdf Extend wait time to 20s in SecurityIntegTest
This kind of sucks, because we shouldn't have to wait that long for tests to run.
But they're failing CI with some regularity, and we rely on these integration tests.

Original commit: elastic/x-pack-elasticsearch@3f4acb2a32
2017-11-10 15:20:30 +11:00
Tal Levy 8c489b1a98 Prevent 7.x nodes from joining cluster with un-upgraded 6.x .security indices (elastic/x-pack-elasticsearch#2921) (elastic/x-pack-elasticsearch#2940)
This is a forward-port of elastic/x-pack-elasticsearch/pull/2921.

original commit message:

Before this commit, a cluster with security enabled and backed by
native-realm user permissions allowed rolled upgrades to clusters without
upgrading the `.security` index. This resulted in the newly established
6.0 cluster not able to register the native-realm users previously established
in the `.security` index. In order to fix this, one would have to rely on file-based
users to re-configure and upgrade the `.security` index. Since this state is easily
avoidable with an upgrade, this commit rejects the joining of upgraded nodes without
upgrading the security index beforehand.

modifications:

Test with 7.x vs 6.x nodes.

Original commit: elastic/x-pack-elasticsearch@56f81bfb20
2017-11-09 12:49:59 -08:00
jaymode 9d85f377c7 Test: update the branch logic for BWC tests
This commit updates the logic for determining which branch to use to make it consistent with the
logic in elasticsearch. This change means that testing BWC within the same major picks the correct
branch.

Original commit: elastic/x-pack-elasticsearch@2d75d15c41
2017-11-09 13:03:40 -07:00
Chris Earle efb5b8827b [Monitoring] Add Rolling Upgrade Tests (elastic/x-pack-elasticsearch#2832)
This adds a rolling upgrade test for X-Pack monitoring. It works by using the `_xpack/monitoring/_bulk` endpoint to send arbitrary data, then verify that it exists.

This forces a few things to happen, thereby testing the behavior: 

1. The templates must exist.
2. The elected master node must be "ready" to work (hence the first
point).
3. The same "system_api_version" is accepted by every version of ES.

Original commit: elastic/x-pack-elasticsearch@012e5738bb
2017-11-09 12:49:37 -05:00
Hendrik Muhs 2693c6a730 [ML] improve logging for autodetect crashes (elastic/x-pack-elasticsearch#2866)
Improving logging for unexpected autodetect termination (crash, oom). Output to the log pipe not conforming to the json log output format are treated as fatal error and logged, so that the crash as well as a proper error message if available gets logged.

Original commit: elastic/x-pack-elasticsearch@ae5d792d3f
2017-11-09 15:47:23 +01:00
David Roberts 142b59a4d5 [ML] Use the correct timeout for the process context lock (elastic/x-pack-elasticsearch#2935)
This change should have been made in elastic/x-pack-elasticsearch#2913.  Now we hold the process
context lock throughout the job close procedure, the timeout for trying
to lock it should be the timeout used for job open/close rather than the
timeout for connecting named pipes.

Original commit: elastic/x-pack-elasticsearch@79672b0825
2017-11-09 13:50:15 +00:00
Luca Cavanna 62b8e54247 Build: add aggs-matrix-stats to license mapping and ignore sha list (elastic/x-pack-elasticsearch#2932)
Original commit: elastic/x-pack-elasticsearch@d33a5b95bc
2017-11-09 11:32:36 +01:00
Jay Modi e29649a7bc Remove the xpack plugin's dependency on the tribe module (elastic/x-pack-elasticsearch#2901)
This change removes the xpack plugin's dependency on the tribe module, which is not a published
artifact. For the most part this just involves moving some test classes around, but for the
security and tribe integration the usage of constant settings was removed and replaced with the
string names. This is a bit unfortunate, but a test was added in a QA project that depends on tribe
that will alert us if a new setting is added that we need to be aware of.

relates elastic/x-pack-elasticsearch#2656

Original commit: elastic/x-pack-elasticsearch@649a8033e4
2017-11-08 12:39:02 -07:00
jaymode 96d0a374a4 Test: fix check for security version after template updater change
This change fixes the check for the version of the security template after the template updater was
changed to only run on the master node in elastic/elasticsearch#27294. Additionally, the wait time
for the cluster to have a yellow status has been increased to account for delayed shards and slower
machines.

Original commit: elastic/x-pack-elasticsearch@a2e72bed12
2017-11-08 10:46:53 -07:00
David Kyle cba4421a75 [ML] Fix streaming the process update request (elastic/x-pack-elasticsearch#2928)
Original commit: elastic/x-pack-elasticsearch@cf76c13a2b
2017-11-08 16:29:12 +00:00
Igor Motov a72879acb2 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@f3b4897936
2017-11-08 11:01:45 -05:00
Christoph Büscher 17ae4899c8 [Test] Change expected exception type after changes in core
Original commit: elastic/x-pack-elasticsearch@0ad2e06970
2017-11-08 12:54:54 +01:00
David Roberts 7b25e7d9ed [ML] Remove Watcher middleman from ML dependency on core (elastic/x-pack-elasticsearch#2926)
I imagine this needless indirection arose from accepting the wrong
IntelliJ suggestion for an import.

Original commit: elastic/x-pack-elasticsearch@54d7e854d3
2017-11-08 10:33:48 +00:00
Tim Vernum 59b453e1c8 [Security] Fix concrete security index name (elastic/x-pack-elasticsearch#2905)
The 5.6 Upgrade API will reindex .security to .security-6 and create a .security alias.
But the 6.0 default was to create a .security-v6 index and a .security alias if none existed (e.g. fresh x-pack install)

Having two different index names based on the method of install/upgrade complicates the code and testing, so we're unifying on the .security-6 index name that already exists in the wild.

Original commit: elastic/x-pack-elasticsearch@d78f569c5f
2017-11-08 10:22:42 +11:00
David Roberts 027f64b221 [ML] Fix a race condition simultaneous close requests are made for a job (elastic/x-pack-elasticsearch#2913)
When simultaneous close requests were made for the same job it was possible
that one of the requests would inappropriately log error messages about the
job having failed.  This change prevents that problem, whilst continuing to
adhere to the requirement that close requests for already closing jobs do not
return until the close request that is doing the work completes.

relates elastic/x-pack-elasticsearch#2912

Original commit: elastic/x-pack-elasticsearch@513b7fa1d6
2017-11-07 14:30:59 +00:00
Tim Vernum 8e5855e62e Allow XPack user read-only access to index audit log (elastic/x-pack-elasticsearch#2906)
The default internal XPack user no longer has access to the security index, but it should have read-only access to the audit log so that watches can be triggered based on audit events (but cannot write audit records)

Original commit: elastic/x-pack-elasticsearch@5c37720dad
2017-11-07 19:31:24 +11:00
Igor Motov de33803d85 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@bfdbc2bb75
2017-11-06 18:51:23 -05:00
Jay Modi be773363c9 Do not fail requests on exceptions from native roles store (elastic/x-pack-elasticsearch#2857)
This commit changes the handling of exceptions when retrieving roles from the native roles store.
Previously, exceptions would have caused the request to terminate and the exception would be
sent back to the user. This makes for a bad experience when a cluster hasn't been upgraded to the
latest index format and anonymous access is enabled with a native role as all requests without
preemptive basic authentication would result in an exception. The change here is to allow the
request to continue processing. Once the security index is up to date, the roles cache is cleared
so that the native roles can be picked up.

relates elastic/x-pack-elasticsearch#2686

Original commit: elastic/x-pack-elasticsearch@ef5149140f
2017-11-06 10:27:56 -07:00
Simon Willnauer 457c49c332 Apply Renames from split shards (elastic/x-pack-elasticsearch#2716)
XPack side of elastic/elasticsearch#26931

Original commit: elastic/x-pack-elasticsearch@6e7c3d4242
2017-11-06 11:38:21 +01:00
Tim Vernum dd3a800745 Fix ASN.1 encoding of "cn" OtherName in CertGen/CertUtil (elastic/x-pack-elasticsearch#2858)
Certgen was generating "other name" SANs without the explicit [0] tag that is required.
This was masked by the fact that the JRE X.509 classes always wrap the "other name" name-value in a [0] tag  (even if it already has one)

Also switched to a UTF8 String from an IA5 string to match the configuration being used for testing in openssl.

Original commit: elastic/x-pack-elasticsearch@1b87964ec7
2017-11-06 10:04:17 +11:00
Nik Everett 00d30285e1 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@33905ed7be
2017-11-04 19:12:40 -04:00
David Roberts 7b36046f33 Use TestEnvironment factory method to create test Environment objects (elastic/x-pack-elasticsearch#2860)
This is the X-Pack side of elastic/elasticsearch#27235.  To force people
who construct an Environment object in production code to think about the
correct setting of configPath there is no longer a single argument
constructor in the Environment class.  Instead there is a factory method
in the test framework to replace it.  Having this in the test framework
ensures that there is no way to use it in production code.

Original commit: elastic/x-pack-elasticsearch@4860e92d90
2017-11-04 13:25:56 +00:00
Nik Everett d96317fbca SQL: fix known actions test after action move
Original commit: elastic/x-pack-elasticsearch@a44932a25a
2017-11-03 21:58:09 -04:00
Nik Everett 1bf1521b37 SQL: Remove a package (elastic/x-pack-elasticsearch#2837)
Collapses a package into its parent package because they only contain a
single class together. No need for two layers.

Original commit: elastic/x-pack-elasticsearch@c947e57952
2017-11-04 01:54:21 +00:00
Nik Everett 25a0ec42f6 SQL: Reorganize the translate action (elastic/x-pack-elasticsearch#2841)
Organizes the SQL translate action to match the way that x-pack has been
organizing new actions for a while. All of the pieces are put into the
same class file.

Original commit: elastic/x-pack-elasticsearch@def911c0ab
2017-11-03 23:25:52 +00:00
Chris Earle 5b85453c9a [TEST] Use the test's Settings.Builder in the test rather than Settings.EMPTY
Original commit: elastic/x-pack-elasticsearch@5b7d7bc652
2017-11-03 16:56:01 -04:00
Nik Everett 562117a7b7 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@c8368be422
2017-11-03 16:16:50 -04:00
David Roberts ba5dbc4daf Remove uses of single argument Environment constructor from production code (elastic/x-pack-elasticsearch#2852)
Following elastic/elasticsearch#27235 the single argument Environment constructor
is forbidden in production code.  This change removes the last such uses from
X-Pack.

Original commit: elastic/x-pack-elasticsearch@87e72d0d07
2017-11-03 09:12:35 +00:00
David Roberts 14211b47f2 Use the correct Environment object in NativeControllerHolder (elastic/x-pack-elasticsearch#2847)
We should not be constructing a temporary Environment object in production
code.  This currently isn't causing any problems, but it might in the future
if elastic/elasticsearch#27144 or something similar is ever merged.  Instead
the master Environment of the node should always be used.

Original commit: elastic/x-pack-elasticsearch@6276a54a45
2017-11-02 16:36:09 +00:00
Chris Earle 2acd0afdd5 [Monitoring] Add Data to Warn on lack of Transport TLS (elastic/x-pack-elasticsearch#2820)
This adds the data necessary to add a warning to the alerts UI representing each cluster when xpack.security.transport.tls.enabled is not set to true for a trial licensed cluster running with
xpack.security.enabled.

Original commit: elastic/x-pack-elasticsearch@28fe8bad76
2017-11-02 15:26:04 +00:00
Nik Everett 28dc53ac5e Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@ad5707a44f
2017-11-02 00:14:51 -04:00
Nik Everett d66d88c5cd Fix tset compile
Test wasn't compiling after a change to core.

Original commit: elastic/x-pack-elasticsearch@8017b29f0b
2017-11-02 00:14:27 -04:00
Albert Zaharovits fb13299714 Log when encountered cert but expecting key and vice-versa.. (elastic/x-pack-elasticsearch#2670)
Log when encountered cert but expecting key and vice-versa.

relates elastic/x-pack-elasticsearch#2657

Original commit: elastic/x-pack-elasticsearch@4e26d8044f
2017-11-01 15:20:28 +02:00
Chris Earle 31741a85d9 [Monitoring] Add Shard-level Details to Index Stats Collection (elastic/x-pack-elasticsearch#2817)
This adds details about the shards and the health of the index. By adding these stats directly to the document, the UI can avoid many aggregations and enable better searching and sorting against indices.

Original commit: elastic/x-pack-elasticsearch@f38ae5ce69
2017-10-31 16:41:40 +00:00
Tanguy Leroux f416c5b3c9 Replace empty index block checks with global block checks in get/put license actions (elastic/x-pack-elasticsearch#2766)
Related to elastic/x-pack-elasticsearch#10530

Original commit: elastic/x-pack-elasticsearch@f4c9924d62
2017-10-31 16:15:14 +01:00
Nik Everett 33f4a8317c JDBC metadata integration with security (elastic/x-pack-elasticsearch#2806)
I realized that we weren't running our DatabaseMetaData tests. One thing led to another and I made these changes:
1. Got the DatabaseMetaData tests running in all three of our QA projects.
2. Fixed the SecurityCatalogFilter to work with `SqlGetIndicesAction`. It worked before, but only for requests that were a `SqlAction` as well as `SqlGetIndicesAction`.
3. Added security test for the JDBC DatabaseMetaData requests. These mirror exactly the security tests that we use for `SHOW TABLES` and `DESCRIBE` but cover the JDBC actions.

Original commit: elastic/x-pack-elasticsearch@7026d83c06
2017-10-30 23:22:12 +00:00
Nik Everett b0dc14f639 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@7af47176fc
2017-10-30 13:50:33 -04:00
Jay Modi 4f65d9b527 Retry startup for IndexAuditTrail and version templates (elastic/x-pack-elasticsearch#2755)
This commit removes the FAILED state for the IndexAuditTrail so that we always try to keep starting
the service. Previously, on any exception during startup we moved to a failed state and never tried
to start again. The users only option was to restart the node. This was problematic in the case of
large clusters as there could be common timeouts of cluster state listeners that would cause the
startup of this service to fail.

Additionally, the logic in the IndexAuditTrail to update the template on the current cluster has
been removed and replaced with the use of the TemplateUpgradeService. However, we still need to
maintain the ability to determine if a template on a remote cluster should be PUT. To avoid always
PUTing the template, the version field has been added so it only needs to be PUT once on upgrade.

Finally, the default queue size has been increased as this is another common issue that users hit
with high traffic clusters.

relates elastic/x-pack-elasticsearch#2658

Original commit: elastic/x-pack-elasticsearch@27e2ce7223
2017-10-30 09:11:18 -06:00
Chris Earle c335b00c9b [Monitoring] Add "cluster_state.nodes_hash" to document (elastic/x-pack-elasticsearch#2798)
Adding this field enables a very simple mechanism for detecting node changes in the cluster state via Watcher (and other mechanisms). The next step is to add the cluster alert that uses it.

Original commit: elastic/x-pack-elasticsearch@1eacc25cff
2017-10-30 13:13:57 +00:00
Martijn van Groningen 9706d07209 disabled test assertion
Original commit: elastic/x-pack-elasticsearch@5a05c607e5
2017-10-30 10:15:45 +01:00
Martijn van Groningen f96153143b test: added logging and don't use replicas
Original commit: elastic/x-pack-elasticsearch@917ef8177d
2017-10-30 09:39:10 +01:00
Martijn van Groningen 2d89394896 test: removed refresh workaround
Original commit: elastic/x-pack-elasticsearch@75b571c23f
2017-10-30 09:29:43 +01:00
Martijn van Groningen 9a1c103bb2 security: Fail search request if profile is used and DLS is active.
Original commit: elastic/x-pack-elasticsearch@b83536460d
2017-10-30 09:12:27 +01:00
Tim Vernum 0c7caabea1 Usability enhancements for certificate generation (elastic/x-pack-elasticsearch#2561)
This commit adds a new `certutil` command and deprecates the `certgen` command.
 
The new certuil consists of sub commands that are (by default) are simpler to use than the old monolithic command, but still support all the previous behaviours.

Original commit: elastic/x-pack-elasticsearch@3f57687da9
2017-10-30 13:08:31 +11:00
Nhat ba29971323 test: updates DocsStats with totalSizeInBytes
Relates https://github.com/elastic/elasticsearch/pull/27117

Original commit: elastic/x-pack-elasticsearch@9bf177d90b
2017-10-28 13:04:21 -04:00
Nik Everett b91e9d85e9 Fix SHAs
Original commit: elastic/x-pack-elasticsearch@15c2669416
2017-10-27 17:12:15 -04:00
Alexander Reelsen 940eabd5f3 Watcher: Add thread pool rejection to execution state (elastic/x-pack-elasticsearch#2805)
The execution state of a watch did not differentiate between failures of
the execution like a broken painless script and a thread pool rejection.

This adds an own state, which allows to aggregate on such data in the
watch history, which should ease debugging issues a bit.

Original commit: elastic/x-pack-elasticsearch@351e64e14d
2017-10-27 16:37:14 +02:00
Martijn van Groningen 96b0b4e96d test: refresh only once to workaround phrase suggester edge case
Relates to elastic/x-pack-elasticsearch#2804

Original commit: elastic/x-pack-elasticsearch@3f2b6b7eea
2017-10-27 15:05:18 +02:00
Hendrik Muhs f74e680142 [ML] add detectorIndex to modelplot and forecast (elastic/x-pack-elasticsearch#2796)
add detector_index to model plots and forecast

relates elastic/x-pack-elasticsearch#2547

corresponding ml-cpp change: elastic/machine-learning-cpp#361

Original commit: elastic/x-pack-elasticsearch@5927d8578e
2017-10-27 12:54:42 +02:00
Dimitris Athanasiou c7e94b3b4c [ML] Enable overall buckets aggregation at a custom bucket span (elastic/x-pack-elasticsearch#2782)
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.

This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.

The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.

Original commit: elastic/x-pack-elasticsearch@981f7a40e5
2017-10-27 11:14:13 +01:00
Martijn van Groningen e028716bec test: use a single primary shard to workaround an edge case with the phrase suggester
Relates to elastic/x-pack-elasticsearch#2804

Original commit: elastic/x-pack-elasticsearch@afd028faf7
2017-10-27 10:41:19 +02:00
Nik Everett 52d9de1de7 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@dd027d9ca5
2017-10-26 09:51:04 -04:00
Martijn van Groningen 62215f1fae security: Fail request if suggesters are used and DLS is active.
Original commit: elastic/x-pack-elasticsearch@056c735e77
2017-10-26 08:02:31 +02:00
Tim Vernum 8985625ea5 [Security] BulkShardRequest may have multiple indices (elastic/x-pack-elasticsearch#2742)
If a bulk update references aliases rather than concrete indices,
it is possible that a single shard level request could have multiple distinct "index names", potentially including "date math".
Those names will resolve to the same concrete index, but they might have different privileges.

Original commit: elastic/x-pack-elasticsearch@34cfd11df8
2017-10-26 15:34:58 +11:00
Jason Tedor 70a38ec545 Enable BWC testing against other remotes
This commit enables BWC testing against remotes on GitHub other than
elastic/elasticsearch.git and elastic/x-pack-elasticsearch.git.

Relates elastic/x-pack-elasticsearch#2707

Original commit: elastic/x-pack-elasticsearch@9028f2e089
2017-10-25 22:39:58 -04:00
Lee Hinman cdaa047d56 [TEST] Fix compilation for ARS stats exposing
Original commit: elastic/x-pack-elasticsearch@eb0ad99434
2017-10-24 11:09:19 -06:00
Albert Zaharovits 403912b8a2 SecureSettings ignored by customAuditIndexSettings (elastic/x-pack-elasticsearch#2748)
customAuditIndexSettings does not submit SecureSettings with putIndexMapping.

relates elastic/x-pack-elasticsearch#2705

* Randomize SecureSetting in testcase

* Apply feedback

Original commit: elastic/x-pack-elasticsearch@1a5414b057
2017-10-24 13:50:35 +03:00
Alexander Reelsen 6f437c973a Watcher: Ensure all templates exist before starting watcher (elastic/x-pack-elasticsearch#2765)
This is to ensure that the required templates exist (which are added by
the WatcherIndexTemplateRegistry) before actually starting watcher.

Relates elastic/x-pack-elasticsearch#2761 

Original commit: elastic/x-pack-elasticsearch@568088061f
2017-10-23 11:57:40 +02:00
Martijn van Groningen c9682d02d4 fix test
Original commit: elastic/x-pack-elasticsearch@7ca5e0fb54
2017-10-23 09:58:33 +02:00
Martijn van Groningen 652f6560b8 security: Always allow access to a rootdoc's nested documents if access to rootdoc is allowed
relates elastic/x-pack-elasticsearch#2665

Original commit: elastic/x-pack-elasticsearch@2bbddd1dd2
2017-10-23 09:28:53 +02:00
Nik Everett 65f2b9fe01 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@9fc67cbfee
2017-10-18 10:27:19 -04:00
Simon Willnauer 2d1ce76194 Adopt core that `_flush` and `_force_merge` doesn't refresh anymore (elastic/x-pack-elasticsearch#2752)
Relates to elastic/elasticsearch#27000

Original commit: elastic/x-pack-elasticsearch@52e9951094
2017-10-16 10:16:50 +02:00
Igor Motov e1d4fdc6d3 SQL: Unknown filter type [pattern_capture] warning in SQL IT Tests (elastic/x-pack-elasticsearch#2746)
Fixes java.lang.IllegalArgumentException: Unknown filter type [pattern_capture] for [email] warnings that pollute log files during execution of the AbstractSqlIntegTestCase-based tests.

Related to elastic/x-pack-elasticsearch#2501

Original commit: elastic/x-pack-elasticsearch@b9ec3c20d0
2017-10-13 11:55:48 -04:00
Nik Everett 99fea5f448 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@d521ecda35
2017-10-13 10:39:28 -04:00
Tanguy Leroux a6776cef97 [Monitoring] Add interval_ms to Monitoring documents (elastic/x-pack-elasticsearch#2650)
This commit adds a new interval_ms field to the monitoring documents. 
This field indicates the current collection interval for Elasticsearch or 
external monitored systems. The value is indexed as a long.

Related to elastic/x-pack-elasticsearch#212

Original commit: elastic/x-pack-elasticsearch@2ceb20455c
2017-10-13 11:18:47 +02:00
Jay Modi 9028c0a642 Allow PkiRealm to use truststore.password setting (elastic/x-pack-elasticsearch#2727)
This change fixes an incorrect check for a missing password setting for the PKI realm. The check
only allowed the secure setting to be used for the PkiRealm password even though the legacy setting
is still valid. This change fixes the check.

Relates elastic/x-pack-elasticsearch#2487

Original commit: elastic/x-pack-elasticsearch@a4524c2c05
2017-10-12 10:07:08 -06:00
Tanguy Leroux 0299886388 [Tests] Use XPack Usage API to verify Monitoring exporters are disabled (elastic/x-pack-elasticsearch#2648)
This commit changes the MonitoringIt and XPackRestIT tests so that the
disableMonitoring() method now use the XPack Usage API in order to check
that the monitoring exporters are correctly disabled. It checks at the
beginning of the tests (all exporters must be disabled before running
the test) and also at the end of the test.

This commit also fixes a bug in MonitoringIT where the Bulk thread pool
active queue was wrongly extracted from the response's map, forcing the
test to always wait for 30sec.

relates elastic/x-pack-elasticsearch#2459

Original commit: elastic/x-pack-elasticsearch@2d349e840f
2017-10-12 09:36:44 +02:00
Costin Leau 2026198dd4 Add size for column tests (elastic/x-pack-elasticsearch#2685)
Add displaySize to columnInfo

Original commit: elastic/x-pack-elasticsearch@ed1d265e98
2017-10-12 00:03:41 +03:00
Tanguy Leroux ea4bff1d43 [Monitoring] Align MonitoringBulkDoc serialization with 6.0 (elastic/x-pack-elasticsearch#2736)
The version used in serialization must be aligned with 6.0/6.x.

Original commit: elastic/x-pack-elasticsearch@db63b91bc6
2017-10-11 17:56:24 +02:00
Tim Vernum a4f7db4f66 [Security] Improve error messages in setup-passwords (elastic/x-pack-elasticsearch#2724)
Provides more verbose messaging around errors and possible causes when the tool aborts.

This change is primarily focused on errors connecting to the Elasticsearch node when TLS is enabled on the HTTP connection.

Original commit: elastic/x-pack-elasticsearch@aa8f7c6143
2017-10-11 12:32:35 +10:00
Tim Vernum bc038b323d [Security] Apply validation when parsing certgen input (elastic/x-pack-elasticsearch#2711)
When certgen configuration was read from an input file (`-in` option) validation errors were collected but never reported. Depending on the type of error this may have caused the tool to exit with an internal error (e.g. NPE).

Validation is now applied after parsing the file and if errors are found the tool exits.

Original commit: elastic/x-pack-elasticsearch@b2262ed1d7
2017-10-11 12:30:19 +10:00
David Roberts 5d0388ccb3 [TEST] Fix ML node attribute test
When ML is disabled the attribute checking is stricter, but the test
did not reflect this

Original commit: elastic/x-pack-elasticsearch@613e97c595
2017-10-10 16:22:03 +01:00
David Roberts ab9cc25a8e [ML] Prevent ML node attributes being set directly (elastic/x-pack-elasticsearch#2725)
ML uses node attributes to ensure that the master node knows how many
ML jobs may be allocated to each node.  This change prevents a user
messing up the way these attributes are used by setting them differently
using node.attr.* entries in their elasticsearch.yml.

This covers the "very short term" change outlined in elastic/x-pack-elasticsearch#2649

Original commit: elastic/x-pack-elasticsearch@9c381801d9
2017-10-10 15:12:59 +01:00
Dimitris Athanasiou 5eea355b33 [ML] Add overall buckets api (elastic/x-pack-elasticsearch#2713)
Adds the GET overall_buckets API.

The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets

The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.

relates elastic/x-pack-elasticsearch#2693

Original commit: elastic/x-pack-elasticsearch@ba6061482d
2017-10-10 14:41:24 +01:00
Alexander Reelsen 80593fb23c Watcher: Add execution state to watch status (elastic/x-pack-elasticsearch#2699)
The execution state is kind of a global indicator if a watch has been
running successfully and is used by the watcher UI.

However this field is only stored in the watch history but not part of
the watch status, thus it is not available everywhere. In order to
simplify the watcher UI this commit also adds the field to the
watch status which is stored together with the watch.

It is stored under the `status.execution_state` field as `status.state`
is already taken. This is also reflects with the name of the java class.

The WatchStatus class does not contain serialization checks, as this is
intended to be backported to 6.x, where those checks will be added.

Once the backport is done, the old execution state field can be fully
deleted from the master branch in another commit (syncing with Kibana
folks required).

relates elastic/x-pack-elasticsearch#2385

* fix doc tests

Original commit: elastic/x-pack-elasticsearch@26e8f99571
2017-10-10 09:07:33 +02:00
Alexander Reelsen cadfd03529 Watcher: Allow JIRA path to be custom chosen (elastic/x-pack-elasticsearch#2682)
The path of a JIRA endpoint used to be fixed. This commit allows the
path to be dynamic, so that users can deploy their JIRA instance under
an arbitrary prefix.

Original commit: elastic/x-pack-elasticsearch@7702505114
2017-10-10 08:55:28 +02:00
Chris Earle 69ab7797be [Monitoring] Cleaner Service should be able to cleanup .watcher-history* (elastic/x-pack-elasticsearch#2696)
This adds a dynamic setting, which defaults to `false` currently, that can be used to delete all `.watcher-history*` indices that match the same age requirements as Monitoring indices.

Original commit: elastic/x-pack-elasticsearch@8ca3bdbca3
2017-10-09 15:46:07 -06:00
Igor Motov 79d6b88763 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@7503023447
2017-10-09 15:47:35 -04:00
Albert Zaharovits 98347088f9 Fix LDAP Authc connections deadlock (elastic/x-pack-elasticsearch#2587)
Do not execute bind on on the LDAP reader thread

Each LDAP connection has a single associated thread, executing the handlers for async requests; this is managed by the LDAP library. The bind operation is blocking for the connection. It is a deadlock to call bind, if on the LDAP reader thread for the same connection, because waiting for the bind response blocks the thread processing responses (for this connection).
This will execute the bind operation (and the subsequent runnable) on a thread pool after checking for the conflict above.

Closes: elastic/x-pack-elasticsearch#2570, elastic/x-pack-elasticsearch#2620

Original commit: elastic/x-pack-elasticsearch@404a3d8737
2017-10-09 13:06:12 +03:00
Simon Willnauer cd14f33ae2 Return List instead of an array from settings (elastic/x-pack-elasticsearch#2694)
XPack side of elastic/elasticsearch#26903

Original commit: elastic/x-pack-elasticsearch@f0390974ab
2017-10-09 09:52:34 +02:00
Nik Everett 5806b620c5 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@141332d3fc
2017-10-06 13:57:55 -04:00
Chris Earle f5561006f5 [Monitoring] Fix HttpExporterIT.testHostChangeReChecksTemplate (elastic/x-pack-elasticsearch#2703)
This changes the test to use the correct version (the Watch version).

Original commit: elastic/x-pack-elasticsearch@197ec3869b
2017-10-06 10:11:23 -06:00
David Roberts dc1ed43ac1 Bump monitoring cluster alert versions (elastic/x-pack-elasticsearch#2673)
If the monitoring cluster alert versions are too far behind the
monitoring index template versions then it causes tests in HttpExporterIT
to fail.  This change increases the versions for the cluster alerts to
7.0.0-alpha1 to match the increase in index template version from elastic/x-pack-elasticsearch#2614.

relates elastic/x-pack-elasticsearch#2671

Original commit: elastic/x-pack-elasticsearch@b3cc3c03fe
2017-10-06 16:06:25 +01:00
Chris Earle 22c804ed24 [Monitoring] Fix HttpExporterIT (elastic/x-pack-elasticsearch#2702)
Uses the appropriate overload of `generateRandomStringArray` to disallow empty arrays from being returned.

Original commit: elastic/x-pack-elasticsearch@2596653ca1
2017-10-06 08:59:44 -06:00
Jay Modi f73d0c7a07 Add transport ssl enabled value back to security usage (elastic/x-pack-elasticsearch#2695)
Since the transport ssl enabled setting is usable in 6.x again, this change adds back the value to
the xpack security usage so that it can be included in phone home data.

Original commit: elastic/x-pack-elasticsearch@52f6176df0
2017-10-06 08:43:32 -06:00
Tim Vernum ec5a038f98 [Security] Support "type" field in role-mappings (elastic/x-pack-elasticsearch#2681)
The upgrade API adds a "type" field to role mapping documents.
The parser would reject these docs due to an unexpected field. We now ignore the "type" field instead.

Original commit: elastic/x-pack-elasticsearch@538f5adab2
2017-10-06 13:50:55 +11:00
Costin Leau 31a952993a Merge remote-tracking branch 'remotes/upstream/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@2ac0dab27b
2017-10-05 19:16:36 +03:00
Dimitris Athanasiou 686eb0ab65 [ML] Refactor [Bucket|Record]QueryBuilder classes (elastic/x-pack-elasticsearch#2684)
Those classes used to be elasticsearch-agnostic wrappers
of the query parameters. However, we now do not need that
layer of abstraction. Instead we can make those builders
own the building of the SearchSourceBuilder, which
simplifies the JobProvider and makes them reusable.

Original commit: elastic/x-pack-elasticsearch@b079cce1d6
2017-10-05 11:47:18 +01:00
Simon Willnauer acba5a3c87 Lists are now represented as actual lists in Settings
Relates to elastic/elasticsearch#26878

Original commit: elastic/x-pack-elasticsearch@de6cfe26ed
2017-10-05 09:26:07 +02:00
Nhat 0c48f2c313 test: do not use deprecated shard preferences (elastic/x-pack-elasticsearch#2630)
This commit makes sure that we won't use the deprecated shard
preferences.

Relates elastic/elasticsearch#26335

Original commit: elastic/x-pack-elasticsearch@273e968407
2017-10-04 07:49:39 -04:00
David Roberts d0e3b8f524 [TEST] Mute failing test suite: HttpExporterIT
See https://github.com/elastic/x-pack-elasticsearch/issues/2671

Original commit: elastic/x-pack-elasticsearch@3f63d00057
2017-10-04 10:33:23 +01:00
Tim Vernum b228ad0511 [Security] Cache action privilege testing for bulk items (elastic/x-pack-elasticsearch#2526)
Since we are authorising on a single shard of a single index,
and there are only 3 possible actions that an item might represent,
we can test which items are authorised with a maximum of 3 permission
evaluations, regardless of how many items are actually in the shard
request. Previously we would test them all independently which had
a much higher overhead for large bulk requests.

Relates: elastic/x-pack-elasticsearch#2369 

Original commit: elastic/x-pack-elasticsearch@aceacf0aa3
2017-10-04 18:46:37 +11:00
Tim Vernum 8980357a29 [Security] Handle no-content gracefully (elastic/x-pack-elasticsearch#2610)
A number of REST requests require a body but did not explicitly validate for it.
This would typically cause a NPE if they were called with no body.

Original commit: elastic/x-pack-elasticsearch@863ac89429
2017-10-04 18:45:40 +11:00
Simon Willnauer f5864c7291 Move away from `Settings#getAsMap()` (elastic/x-pack-elasticsearch#2661)
Relates to elastic/elasticsearch#26845

Original commit: elastic/x-pack-elasticsearch@0323ea07a5
2017-10-04 01:21:59 -06:00
Alexander Reelsen 8268cecb80 Tests: Replace script with search transform to remove plugin in test code (elastic/x-pack-elasticsearch#2470)
The test also used the timewarp trigger for watches to be executed, but it is sufficient to just call the execute watch API to make this test faster.

Original commit: elastic/x-pack-elasticsearch@3a4165f72c
2017-10-04 09:12:13 +02:00
David Roberts 0be7255029 [ML] Allow dynamic updates to the xpack.ml.max_model_memory_limit setting (elastic/x-pack-elasticsearch#2503)
A key limitation with this is that the updated setting only applies when
jobs are created or updated.  It does NOT automatically restrict existing
jobs.

relates elastic/x-pack-elasticsearch#2462

Original commit: elastic/x-pack-elasticsearch@73bd08db3f
2017-10-03 14:22:46 +01:00
David Roberts c7b4ef9a89 Add cgroup memory usage/limit to OS stats on Linux (elastic/x-pack-elasticsearch#2614)
This change adapts the monitoring tests to account for an addition to
the OS stats made in elastic/elasticsearch#26166.

Original commit: elastic/x-pack-elasticsearch@9e36764857
2017-10-03 12:09:59 +01:00
Alexander Reelsen 3e0644b891 Tests: Fix test assertion in execution service test
No need for exact duration value check, just make sure it is
not zero based.

relates elastic/x-pack-elasticsearch#2525

Original commit: elastic/x-pack-elasticsearch@1fe3a0bf5a
2017-10-02 14:33:10 +02:00
David Roberts 90b2b74e76 [ML] Tolerate a body without timestamp for get_buckets with a timestamp (elastic/x-pack-elasticsearch#2640)
When getting a single bucket, the get_buckets API can take a timestamp
either in the body or in the URL.  Prior to this change, if a timestamp
was specified in the URL but a body not containing a timestamp was specified
(either empty or containing other parameters like exclude_interim or sort)
then it would cause a bad_request exception.  This in turn causes problems
for clients that cannot send a body when GETting and always send a body when
POSTing.

This change fixes get_buckets to always read any timestamp in the URL, even
when a body is sent.

relates elastic/x-pack-elasticsearch#2515

Original commit: elastic/x-pack-elasticsearch@5c23dd972e
2017-09-29 09:17:36 +01:00
David Roberts 0a89abdd7b [ML] snapshot_id is required when reverting a model snapshot (elastic/x-pack-elasticsearch#2641)
Previously the API spec did not say this.

Original commit: elastic/x-pack-elasticsearch@eaf214411d
2017-09-28 09:24:50 +01:00
Igor Motov fdd98f01ed Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@273e0a110e
2017-09-26 11:59:49 -04:00
Dimitris Athanasiou b3ae022985 [ML] Remove unused member in NativeAutodetectProcessFactory (elastic/x-pack-elasticsearch#2629)
Original commit: elastic/x-pack-elasticsearch@31f265f0e5
2017-09-26 14:12:27 +01:00
Dimitrios Athanasiou 5ad170a7ed Fix compilation due to upstream change
Original commit: elastic/x-pack-elasticsearch@57d834841d
2017-09-26 12:11:53 +01:00
Dimitris Athanasiou c46e09902d [ML] Fix close job when the process has not launched yet (elastic/x-pack-elasticsearch#2616)
If a job close is requested after a job was opened but before
its process was launched, the job close returns successfully
without doing anything. The result is that the process hangs
around. This has been causing test failures as documented
int elastic/x-pack-elasticsearch#2360 and elastic/x-pack-elasticsearch#1270.

This commit fixes this problem by refactoring the
AutodetectProcessManager. It introduces a state pattern
to make clear the states of the process and it uses locking
to ensure a close waits for the job process to be created.

relates elastic/x-pack-elasticsearch#1270

Original commit: elastic/x-pack-elasticsearch@ff858bd136
2017-09-26 11:46:01 +01:00
Tanguy Leroux a3984f7baa [Monitoring] Remove MonitoringSettings (elastic/x-pack-elasticsearch#2596)
Original commit: elastic/x-pack-elasticsearch@28dea7b699
2017-09-26 09:37:59 +02:00
Nik Everett e6cd81458b Fix KnownActionTests after removing jdbc transport action
Original commit: elastic/x-pack-elasticsearch@2cb7cdd122
2017-09-25 18:12:18 -04:00
jaymode 7b8d92a2a7 Test: fix AuthenticationService tests timeouts due to incorrect stream sizes
The AuthenticationService#testInvalidToken would cause a suite timeout in the case of an exception
due to a incorrect stream size as the latch was never counted down. This fixes the missing latch
countdown.

relates elastic/x-pack-elasticsearch#2615

Original commit: elastic/x-pack-elasticsearch@e838e6e912
2017-09-25 13:02:22 -06:00
Nik Everett c7c79bc1c0 Add scrolling support to jdbc (elastic/x-pack-elasticsearch#2524)
* Switch `ResultSet#getFetchSize` from returning the *requested*
fetch size to returning the size of the current page of results.
For scrolling searches without parent/child this'll match the
requested fetch size but for other things it won't. The nice thing
about this is that it lets us tell users a thing to look at if
they are wondering why they are using a bunch of memory.
* Remove all the entire JDBC action and implement it on the REST
layer instead.
* Create some code that can be shared between the cli and jdbc
actions to properly handle errors and request deserialization so
there isn't as much copy and paste between the two. This helps
because it calls out explicitly the places where they are different.
  * I have not moved the CLI REST handle to shared code because
I think it'd be clearer to make that change in a followup PR.
* There is now no more need for constructs that allow iterating
all of the results in the same request. I have not removed these
because I feel that that cleanup is big enough to deserve its own
PR.

Original commit: elastic/x-pack-elasticsearch@3b12afd11c
2017-09-25 14:41:46 -04:00
Tim Brooks 6099660643 Update LicenseMetadata version check to be 6.1
This is related to elastic/x-pack-elasticsearch#1941. It modifies to the version check in
LicenseMetadata because this elastic/x-pack-elasticsearch#2419 has been backported to 6.x.

Original commit: elastic/x-pack-elasticsearch@052358b550
2017-09-25 09:42:37 -06:00
Tanguy Leroux 9bd2da86db [Tests] Reenable MonitoringIT
Checking the size of the map doesn't make sense when each filter is
checked independently right after.

Original commit: elastic/x-pack-elasticsearch@58e5d3401d
2017-09-25 16:01:18 +02:00
Nik Everett 216058035b Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b3f6d3fd70
2017-09-25 09:55:17 -04:00
Simon Willnauer e7b5702f50 Adopt settings cleanups from core (elastic/x-pack-elasticsearch#2605)
Relates to elastic/elasticsearch#26739

Original commit: elastic/x-pack-elasticsearch@dd13d099de
2017-09-25 12:27:07 +02:00
Tanguy Leroux 1bb9c4fe71 [Tests] Mute MonitoringIT
see https://github.com/elastic/x-pack-elasticsearch/issues/2609

Original commit: elastic/x-pack-elasticsearch@e2e2393e62
2017-09-25 09:14:50 +02:00
Martijn van Groningen e93e7cf816 Revert "reset me"
This reverts commit elastic/x-pack-elasticsearch@3d44c14cd9.

Original commit: elastic/x-pack-elasticsearch@00203223b4
2017-09-25 08:56:01 +02:00
Martijn van Groningen ad3eb997c6 reset me
Original commit: elastic/x-pack-elasticsearch@3d44c14cd9
2017-09-25 08:52:51 +02:00
Jason Tedor b4f7d56c35 Whitelist global checkpoint sync actions
This commit whitelists the global checkpoint sync actions as otherwise
these actions do not have privileges to run as the system user and will
be denied as unauthorized.

Relates elastic/x-pack-elasticsearch#2604

Original commit: elastic/x-pack-elasticsearch@598ae1ff50
2017-09-22 16:27:51 -04:00
Tim Brooks 6b2e7fbed8 Add api to upgrade from basic to trial license (elastic/x-pack-elasticsearch#2419)
This is related to elastic/x-pack-elasticsearch#1941.

Currently we support self-generating either a basic or trial license at
cluster startup. With the addition of the basic option, it is possible
that a user would choose to self-generate and eventually register a
basic license.

This commit allows a user to upgrade to a 30-day trial license if they
have not already utilized this 30-day trial license before. Additionally
it adds a get route to check if the user is eligible to upgrade. This
route will allow kibana to implement a cleaner UI.

Original commit: elastic/x-pack-elasticsearch@7f19b33a08
2017-09-22 14:18:07 -06:00
Tanguy Leroux 6957282b50 [Monitoring] Fix MonitoringIT on Windows
Original commit: elastic/x-pack-elasticsearch@7a835c179d
2017-09-22 18:21:44 +02:00
Chris Earle 4cf8348209 [Monitoring] Remove Beats Support until wanted (elastic/x-pack-elasticsearch#2592)
This removes the creation and handling of the Beats monitoring template and its data until we actually expect to support it (most likely 6.2 - 6.3).

Original commit: elastic/x-pack-elasticsearch@2dc8abbb37
2017-09-22 11:12:44 -04:00
Nik Everett 2c183d566e Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@53dc6d4ce7
2017-09-21 15:20:20 -04:00
Nik Everett 8a05c1b81f Move all sql integration tests into qa (elastic/x-pack-elasticsearch#2432)
Builds on elastic/x-pack-elasticsearch#2403 to move all of sql's integration testing into
qa modules with different running server configurations. The
big advantage of this is that it allows us to test the cli and
jdbc with security present.

Creating a project that depends on both cli and jdbc and the
server has some prickly jar hell issues because cli and jdbc
package their dependencies in the jar. This works around it
in a few days:
1. Include only a single copy of the JDBC dependencies with
careful gradle work.
2. Do not include the CLI on the classpath at all and instead
run it externally.

I say "run it externally" rather than "fork it" because Elasticsearch
tests aren't allowed to fork other processes. This is forbidden
by seccomp on linux and seatbelt on osx and cannot be explicitly
requested like additional security manager settings. So instead
of forking the CLI process directly the tests interact with a test
fixture that isn't bound by Elasticsearch's rules and *can* fork
it.

This forking of the CLI has a nice side effect: it forces us to
make sure that things like security and connection strings other
than `localhost:9200` work. The old test could and did work around
missing features like that. The new tests cannot so I added the
ability to set the connection string. Configuring usernames and
passwords was also not supported but I did not add support for
that, only created the failing test and marked it as `@AwaitsFix`.

Original commit: elastic/x-pack-elasticsearch@560c6815e3
2017-09-21 09:58:52 -04:00
David Kyle ce77b076ad [ML] Adapt memory limit thresholds in test (elastic/x-pack-elasticsearch#2581)
* Adapt memory limit thresholds in test. Caused by elastic/machine-learning-cpp#305

Original commit: elastic/x-pack-elasticsearch@2984f35b2c
2017-09-21 12:52:31 +01:00
Albert Zaharovits f435145060 Fixes Javadoc generation for @SuppressForb adnot (elastic/x-pack-elasticsearch#2583)
Original commit: elastic/x-pack-elasticsearch@75ce6ac028
2017-09-21 14:28:57 +03:00
Albert Zaharovits a8ea4f2d90 Authz FieldExtractor support for TermInSet query (elastic/x-pack-elasticsearch#2539)
FieldExtractor for TermInSet query.

Closes: elastic/x-pack-elasticsearch#1104

Original commit: elastic/x-pack-elasticsearch@7a46361c86
2017-09-21 11:35:34 +03:00
Albert Zaharovits 6b51f5e6ca Update BouncyCastle version from 1.55 to 1.58 (elastic/x-pack-elasticsearch#2548)
Update BouncyCastle version from 1.55 to 1.58

Requires regeneration of .project and .classpath files, e.g.
gradle cleanEclipse & gradle eclipse

Closes: elastic/x-pack-elasticsearch#2332

Original commit: elastic/x-pack-elasticsearch@5806fd4204
2017-09-21 11:30:42 +03:00
Albert Zaharovits c84c48fa01 Usability improvement for the password bootstrap tool (reserved users) (elastic/x-pack-elasticsearch#2444)
Generate or prompt for passwords of the reserved users, validating strength
and typos, then issue change requests, keeping elastic user change request last.

Closes: elastic/x-pack-elasticsearch#2424

Original commit: elastic/x-pack-elasticsearch@1f827d393c
2017-09-21 11:26:28 +03:00
Tanguy Leroux 34bd72e69b [Monitoring] Remove monitoring resolvers (elastic/x-pack-elasticsearch#2566)
Now all monitoring documents are indexed in timestamped indices dedicated 
to each product, the old monitoring's resolvers and all the associated plumbing 
can be removed from the Elasticsearch monitoring plugin.

This pull request merges the feature branch feature/monitoring-summer-cleanup 
into master. All commits have been independently reviewed as part of elastic/x-pack-elasticsearch#2226:

* [Monitoring] Clean up monitoring doc (elastic/x-pack-elasticsearch#2208)
* [Monitoring] Remove ClusterStatsResolver (elastic/x-pack-elasticsearch#2209)
* [Monitoring] Remove IndexRecoveryResolver (elastic/x-pack-elasticsearch#2229)
* [Monitoring] Remove JobStatsResolver (elastic/x-pack-elasticsearch#2230)
* [Monitoring] Remove ShardsResolver (elastic/x-pack-elasticsearch#2233)
* [Monitoring] Fix missing instanceof in exporter bulks
* [Monitoring] Remove NodeStatsResolver (elastic/x-pack-elasticsearch#2244)
* [Monitoring] Remove IndexStatsResolver & IndicesStatsResolver  (elastic/x-pack-elasticsearch#2247)
* [Monitoring] Remove source node attributes and add timestamp (elastic/x-pack-elasticsearch#2285)
* [Monitoring] Remove last resolver (elastic/x-pack-elasticsearch#2330)
* [Monitoring] Make MonitoringDoc an abstract class (elastic/x-pack-elasticsearch#2451)
* [Monitoring] Fix TransportMonitoringBulkActionTests
* [Monitoring] Remove old resolver tests and add a single integration test (elastic/x-pack-elasticsearch#2453)

It does not introduce any new feature, so it could also go in 6.0.

relates elastic/x-pack-elasticsearch#2226


Original commit: elastic/x-pack-elasticsearch@e3d5cec28c
2017-09-21 09:36:03 +02:00
Tim Sullivan 723d894753 [Monitoring/Cluster Alerts] Fix the email message when cluster license expiration is resolved (elastic/x-pack-elasticsearch#2557)
* [Monitoring/Cluster Alerts] Fix the email message for cluster license expiration resolved

* fix making payload.message show only when new

Original commit: elastic/x-pack-elasticsearch@6d54b02913
2017-09-20 15:44:14 -07:00
Igor Motov 59892ba654 SQL: Fix KnownActionsTests after introduction of the _sql/translate endpoint
Original commit: elastic/x-pack-elasticsearch@a6f2c05a5e
2017-09-20 17:45:58 -04:00
Costin Leau 6cc3c067b7 Introduce _sql/translate endpoint
Builds on RestSqlAction and in fact, extends SqlAction to keep up
with future request settings.

Original commit: elastic/x-pack-elasticsearch@7bbef4bdff
2017-09-20 21:48:53 +03:00
Igor Motov 4964239f4c Tests: Remove AwaitsFix from TemplateUpgraderTests
I believe this problem should have been already fixed by elastic/elasticsearch#26698.

Original commit: elastic/x-pack-elasticsearch@0ec93f2803
2017-09-20 14:12:24 -04:00
Dimitrios Athanasiou a5e8589b38 [ML][TEST] Add integ test for elastic/x-pack-elasticsearch#2519
Relates elastic/x-pack-elasticsearch#2519

Original commit: elastic/x-pack-elasticsearch@91b38a62bb
2017-09-20 18:59:34 +01:00
Nik Everett 2df8b0c144 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@dff8c22d99
2017-09-20 12:06:27 -04:00
Dimitris Athanasiou fad98d784f [ML] Align aggregated data extraction to histogram interval (elastic/x-pack-elasticsearch#2553)
When the datafeed uses aggregations and in order to accommodate
derivatives, an extra bucket is queried at the beginning of each
search. In order to avoid visiting the same bucket twice, we need
to search buckets aligned to the histogram interval. This allows
us to steer away from partial buckets, and thus avoid the problem
of dropping or duplicating data.

relates elastic/x-pack-elasticsearch#2519

Original commit: elastic/x-pack-elasticsearch@e03dde5fea
2017-09-20 16:45:06 +01:00
Nik Everett 6ea902f913 SQL: NOCOMMIT cleanup
Removes a few NOCOMMITs that are tracked other places and updates
a few with plans on how to work on them.

Original commit: elastic/x-pack-elasticsearch@8d1cfdf4ee
2017-09-20 11:19:05 -04:00
Hendrik Muhs 3c517902f2 re-enable categorization IT (elastic/x-pack-elasticsearch#2534)
re-enables categorization IT after fixing https://github.com/elastic/machine-learning-cpp/issues/279 upstream

Original commit: elastic/x-pack-elasticsearch@398a668125
2017-09-20 11:39:37 +02:00
Yannick Welsch 8648153f0e Deguice ActionFilter (elastic/x-pack-elasticsearch#2533)
Companion PR to elastic/elasticsearch#26691

Original commit: elastic/x-pack-elasticsearch@3fceb54809
2017-09-20 10:30:23 +02:00
Tal Levy 8b1021ccad blacklist two license rest tests when build.snapshot=false (elastic/x-pack-elasticsearch#2559)
There are two rest tests that rely on hardcoded license
signatures that use the dev public key. These tests fail
when tests are run with build.snapshot=false. This Commit
blacklists these two tests in that scenario

relates elastic/x-pack-elasticsearch#2527.

Original commit: elastic/x-pack-elasticsearch@7581e8d699
2017-09-19 16:56:11 -07:00
Nik Everett ea66433899 Mark periodically failing test as AwaitsFix
This should reduce our build failure emails. The failure is already
tracked at https://github.com/elastic/x-pack-elasticsearch/issues/2421.
It isn't actively being worked but will be soon.

Original commit: elastic/x-pack-elasticsearch@6baf55de42
2017-09-19 09:59:26 -04:00
David Roberts 67055877a5 [ML] Avoid creating spurious 0 "actual" values for model plot documents (elastic/x-pack-elasticsearch#2535)
Some model plot documents should not have an "actual" value, for example
when no input events were seen for a meean/min/max detector in a particular
bucket.  Prior to this change we would set the "actual" value to 0 for such
model plot documents.  Following this change no "actual" value will be
present in these documents.

Only newly created model plot documents are affected.  Model plot documents
that were incorrectly written in the past will remain wrong forever.

relates elastic/x-pack-elasticsearch#2528

Original commit: elastic/x-pack-elasticsearch@47a7365f59
2017-09-19 10:30:43 +01:00
Tim Vernum aec2308228 Allow AD realm to perform 'run-as' lookups (elastic/x-pack-elasticsearch#2531)
- Marks the AD Session factory as supporting "lookup" (Refer: elastic/x-pack-elasticsearch@40b07b3)
- Adds "pool.enabled" as a registered setting on AD realm (Refer: elastic/x-pack-elasticsearch@40b07b3)
- Fixes LDAP user lookup that has been broken since 6.x (Refer: elastic/x-pack-elasticsearch@f796949)

Original commit: elastic/x-pack-elasticsearch@62ff6129a1
2017-09-19 14:50:26 +10:00
Nik Everett 641db10605 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@81ed649a9b
2017-09-18 16:33:15 -04:00
Tal Levy 5a090c14c1 convert more request objects to writeable (elastic/x-pack-elasticsearch#2457)
* convert more to writeable

* migrate streamable tests to writeable tests

Original commit: elastic/x-pack-elasticsearch@56794e5760
2017-09-18 13:20:02 -07:00
Suyog Rao 3a9aad5ece [Logstash] Remove version field from config mgmt
relates elastic/x-pack-elasticsearch#2405

Original commit: elastic/x-pack-elasticsearch@5bfd1b7a6d
2017-09-18 12:35:21 -07:00
Simon Willnauer 47214426e9 Use InputStreamStreamInputs validation to limit size of tokens (elastic/x-pack-elasticsearch#2537)
Relates to elastic/elasticsearch#26692
relates elastic/x-pack-elasticsearch#2493

Original commit: elastic/x-pack-elasticsearch@8e23868743
2017-09-18 19:25:08 +02:00
Nik Everett 52ee02da27 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@c25c179ce6
2017-09-18 12:32:46 -04:00
Dimitris Athanasiou 9b25d0edf7 [ML] Correctly ignore extra aggregation buckets (elastic/x-pack-elasticsearch#2530)
The problem here is that the code was ignoring buckets
whose start time was before the start time of the extractor.
However, this is not a good enough condition. For example,
when there are no data in the bucket extra bucket that is
being queried, the first bucket will be the one containing
the start time.

This commit fixes the issue by changing the condition to
ignore buckets before the first bucket that includes the
start time of the extraction.

relates elastic/x-pack-elasticsearch#2519 

Original commit: elastic/x-pack-elasticsearch@15c7d2655f
2017-09-18 12:10:19 +01:00
Tim Vernum dea82a07a2 Infer KeyStore type from pathname (elastic/x-pack-elasticsearch#2514)
If the keystore type is not explicitly specified, infer it from the filename.
Treats .p12, .pfx and .pkcs12 as being PKCS12, all others as jks.

This will allow certgen to produce PKCSelastic/x-pack-elasticsearch#12 files by default and make it easy to use them as x-pack keystores

Original commit: elastic/x-pack-elasticsearch@fc361f0d87
2017-09-18 14:21:19 +10:00
Chris Earle 24c2c62ca2 [Monitoring] Ignore .marvel* indices (elastic/x-pack-elasticsearch#2520)
Beginning with 7.0, the cleaner service will no longer automatically cleanup .marvel indices regardless of their age.

Original commit: elastic/x-pack-elasticsearch@5b90e6f62a
2017-09-15 13:35:45 -07:00
Tim Brooks b3914afd30 Reenable TribeWithSecurityIT tests (elastic/x-pack-elasticsearch#2511)
This is related to elastic/x-pack-elasticsearch#1996. These tests were disabled during the bootstrap
password work. They can now be reenabled. Additionally, I made the test
password used in tests consistent.

Original commit: elastic/x-pack-elasticsearch@5b490c8231
2017-09-15 12:50:54 -06:00
Simon Willnauer 96e01dce47 Only require TLS for standard/gold/platinum licenses elastic/x-pack-elasticsearch#2517
relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@5213cf24f1
2017-09-15 20:21:15 +02:00
Chris Earle 9c9da2e1e4 [Monitoring] Remove Legacy Monitoring Indices (elastic/x-pack-elasticsearch#2513)
This changes Monitoring's Cleaner Service to remove any legacy Monitoring index that is appropriately old.

This includes any `.marvel-*` index and also the "data" indices used by both Marvel and 5.0 - 5.4 versions of X-Pack monitoring, as well as the legacy alerts index.

Original commit: elastic/x-pack-elasticsearch@8d99f5518b
2017-09-15 08:46:10 -07:00
jaymode 84dd719ab9 remove outdated comment
Original commit: elastic/x-pack-elasticsearch@06a51abb65
2017-09-15 09:03:36 -06:00
jaymode 344603e40f update text in TLSLicenseBootstrapCheck
Original commit: elastic/x-pack-elasticsearch@4ee6827566
2017-09-15 08:56:34 -06:00
Simon Willnauer 4d20586b24 [TEST] add integration test that ensures we reject license upgrades if TLS is not enabled
Original commit: elastic/x-pack-elasticsearch@dfbadb5e5f
2017-09-15 14:47:28 +02:00
Simon Willnauer c3066d1a51 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@9ce33bc7c3
2017-09-15 09:51:16 +02:00
Simon Willnauer 023bdb72b2 Add common-analysis plugin to several xpack integ tests (elastic/x-pack-elasticsearch#2501)
Several tests miss the common-analysis plugin in the old-style integ
tests causing odd exceptions in the test logs. This adds the missing plugin reference.

relates elastic/x-pack-elasticsearch#2363

Original commit: elastic/x-pack-elasticsearch@e4e6735408
2017-09-15 09:25:36 +02:00
jaymode 8997792875 Test: use TLS for plugin integ tests
Original commit: elastic/x-pack-elasticsearch@99971d7256
2017-09-14 15:57:28 -06:00
Michael Basnight fa0b854fb6 Update rest-api-spec to use bad_request (elastic/x-pack-elasticsearch#2507)
ref #elastic/elasticsearch#26539

Original commit: elastic/x-pack-elasticsearch@8b79a0769a
2017-09-14 15:59:29 -05:00
Andy Bristol 279c7e14fd [TEST] fix security template version check in rest tests (elastic/x-pack-elasticsearch#2506)
Since the template upgrade service was added, upgrades should
be performed by a node with the highest version in the cluster,
which may not be the master node.

Original commit: elastic/x-pack-elasticsearch@d66145de54
2017-09-14 12:16:20 -07:00
Jay Modi 57de66476c Disable TLS by default (elastic/x-pack-elasticsearch#2481)
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.

Relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@abc66ec67d
2017-09-14 12:18:54 -06:00
Simon Willnauer 1e14e14571 Prevent licenses to be upgraded to production unless TLS is configured (elastic/x-pack-elasticsearch#2502)
if a user tries to upgrade a license to a production license and has security
enabled we prevent the upgrade unless TLS is setup. This is a requirement now
if a cluster with security is running in prodcution.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@d61ef3bcb1
2017-09-14 20:14:27 +02:00
Nik Everett bac9afee7e Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@270ab71b19
2017-09-14 11:43:48 -04:00
Nik Everett 5d3f5cc4f8 Support scrolling in SQL's CLI (elastic/x-pack-elasticsearch#2494)
* Move CLI to TransportSqlAction
    * Moves REST endpoint from `/_cli` to `/_sql/cli`
    * Removes the special purpose CLI transport action instead
    implements the CLI entirely on the REST layer, delegating
    all SQL stuff to the same action that backs the `/_sql` REST
    API.
    * Reworks "embedded testing mode" to use a `FilterClient` to
    bounce capture the sql transport action and execute in embedded.
    * Switches CLI formatting from consuming the entire response
    to consuming just the first page of the response and returning
    a `cursor` that can be used to read the next page. That read is
    not yet implemented.
    * Switch CLI formatting from the consuming the `RowSetCursor` to
    consuming the `SqlResponse` object.
    * Adds tests for CLI formatting.
* Support next page in the cli
    * Rename cli's CommandRequest/CommandResponse to
    QueryInitRequest/QueryInitResponse to line up with jdbc
    * Implement QueryPageRequest/QueryPageResponse in cli
    * Use `byte[]` to represent the cursor in the cli. Those bytes
    mean something, but only to the server. The only reasonint that
    the client does about them is "if length == 0 then there isn't a
    next page."
    * Pull common code from jdbc's QueryInitRequest, QueryPageRequest,
    QueryInitResponse, and QueryPageResponse into the shared-proto
    project
        * By implication this switches jdbc's QueryPageRequest to using
     the same cursor implementation as the cli

Original commit: elastic/x-pack-elasticsearch@193586f1ee
2017-09-14 10:26:42 -04:00