Commit Graph

6315 Commits

Author SHA1 Message Date
Lisa Cawley 1bb3ad38c7 [DOCS] Update APIs for multiple jobs or datafeeds (elastic/x-pack-elasticsearch#2105)
* [DOCS] Update APIs for multiple jobs or datafeeds

* [DOCS] Fix syntax diagrams for ML stop/close APIs

* [DOCS] Removed TBD authorization for ML APIs

Original commit: elastic/x-pack-elasticsearch@1a9137a5a7
2017-08-09 08:30:21 -07:00
Colin Goodheart-Smithe a4dd177978 Migrates ToXContent classes (elastic/x-pack-elasticsearch#2024)
* Migrates ToXContent classes

* review comments

Original commit: elastic/x-pack-elasticsearch@74ce3755ab
2017-08-09 15:55:04 +01:00
Lisa Cawley ccf0b6f2ed [DOCS] Fix typo (elastic/x-pack-elasticsearch#2211)
Original commit: elastic/x-pack-elasticsearch@6efb78c1b2
2017-08-08 12:52:52 -07:00
Alexander Reelsen 0b5909fc65 Watcher: Stop swallowing exceptions, always return them instead of message (elastic/x-pack-elasticsearch#1933)
It is really hard to debug some issues with watcher, when only the
e.getMessage() is returned as failure reasons instead of the whole
stack trace.

This commit gets rid of ExceptionsHelper.detailedMessage(e) and always
returns the whole exception.

This commit also extends the watch history to have all fields named
error be treated like an object to be sure they do not get
indexed. No matter where it's placed in the hierarchy

In addition a few Field interface classes were removed, that only contained parse fields.

relates elastic/x-pack-elasticsearch#1816

Original commit: elastic/x-pack-elasticsearch@b2ce680139
2017-08-08 18:36:22 +02:00
David Roberts 22da5cf89e [ML] Add max open jobs per node as a node attribute (elastic/x-pack-elasticsearch#2203)
This commit adds the max_running_jobs setting from elasticsearch.yml
into a node attribute called ml.max_open_jobs.  Previously there was
an assumption that max_running_jobs would be the same for all nodes in
the cluster.  However, during a rolling cluster restart where the value
of the setting is being changed this clearly cannot be the case, and
would cause unexpected/unpredictable limits to be used during the period
when different nodes had different settings.

For backwards compatibility, if another node in the cluster has not added
its setting for max_running_jobs to the cluster state then the old
(flawed but better than nothing) approach is applied, i.e. assume the
remote node's setting for max_running_jobs is equal to that of the node
deciding the job allocation.

Relates elastic/x-pack-elasticsearch#2185

Original commit: elastic/x-pack-elasticsearch@1e62b89183
2017-08-08 16:16:27 +01:00
Dimitris Athanasiou 2864078da2 [ML] Move job group validation after parsing (elastic/x-pack-elasticsearch#2207)
Validating job groups during parsing results into
the validation error being wrapped into a parse
exception. The UI then does not display the cause of the
error. Finally, it is conceptually not a parse error, so
it belongs outside the parsing phase.

Original commit: elastic/x-pack-elasticsearch@a03f002bdc
2017-08-08 15:59:04 +01:00
Alexander Reelsen cff3418e96 Tests: Only restart watcher on watcher tests
In order to not restart watcher on every test, this checks
if this is a watcher test and only restarts watcher in that case.

In addition this also checks if watcher is not marked as started.
as otherwise restarting does not make sense. Lastly, this waits until
watcher is marked as started before proceeding.

Original commit: elastic/x-pack-elasticsearch@a8d72f3ebb
2017-08-08 14:28:44 +02:00
David Roberts 6a159d2127 [ML] Fix fallout from bulk action requiring newlines (elastic/x-pack-elasticsearch#2205)
Only unit tests were broken.  Production ML code was always terminating
bulk requests with newlines.

Original commit: elastic/x-pack-elasticsearch@96ed06fed3
2017-08-08 11:07:13 +01:00
Alexander Reelsen 55e88d6857 Watcher: Ignore if template is missing when upgrade is running (elastic/x-pack-elasticsearch#2199)
If one of the old watcher templates does not exist when we try 
to delete it, the upgrade should just continue.

Original commit: elastic/x-pack-elasticsearch@6a52bad329
2017-08-08 10:17:58 +02:00
Chris Earle 1d6f82dbe3 [Monitoring] Remove Dedicated IndicesStatsCollector (elastic/x-pack-elasticsearch#2192)
This removes the `IndicesStatsCollector` and, instead, it reuses the superset version of the call from the `IndexStatsCollector`.

On clusters with a large number of indices, this should actually help a good amount in reducing wasted calls and memory allocation without any difference in the output.

Original commit: elastic/x-pack-elasticsearch@93b09878e4
2017-08-07 13:00:41 -04:00
Tim Sullivan 7d86d75aa4 [Monitoring/Cluster Alerts] default admin email is snake_case (elastic/x-pack-elasticsearch#2177)
Original commit: elastic/x-pack-elasticsearch@d6129a0065
2017-08-07 09:15:14 -07:00
Alexander Reelsen 2d08477093 Tests: Fix upgrade tests by not running stats against arbitrary hosts
Original commit: elastic/x-pack-elasticsearch@1b858aad52
2017-08-07 13:27:36 +02:00
Alexander Reelsen 8f6874abf9 Tests: Increase logging in watcher upgrade tests to debug test failures
Original commit: elastic/x-pack-elasticsearch@380a8541dc
2017-08-07 11:31:16 +02:00
Luca Cavanna b530da945d Adapt to yaml test client upstream changes (elastic/x-pack-elasticsearch#2172)
Original commit: elastic/x-pack-elasticsearch@d984f71801
2017-08-07 11:02:25 +02:00
David Roberts 05cbe8dc0c [ML] Disallow creating a job against a closed results or state index (elastic/x-pack-elasticsearch#2186)
Previously if this was attempted you'd get an NPE (5.x) or hang (6.x).
Following this change you get an error message telling you what the
problem is.

relates elastic/x-pack-elasticsearch#2170

Original commit: elastic/x-pack-elasticsearch@ea12a9ff46
2017-08-07 08:53:12 +01:00
Chris Earle 87bc215b91 [ML] Use try-with-resources for InputStream (elastic/x-pack-elasticsearch#2182)
Original commit: elastic/x-pack-elasticsearch@88d5e73fec
2017-08-04 13:26:35 -04:00
Alexander Reelsen 36bad2079d Tests: Fix watcher test when aliases were used
Original commit: elastic/x-pack-elasticsearch@d1c38cb85a
2017-08-04 18:01:26 +02:00
Lisa Cawley a6a76f8d2a [DOCS] Clarify xpack.security.enabled info (elastic/x-pack-elasticsearch#2166)
Original commit: elastic/x-pack-elasticsearch@140e330973
2017-08-04 08:59:50 -07:00
Jay Modi 8b0fb5eae8 Re-enable OpenLDAP tests and run against vagrant instance (elastic/x-pack-elasticsearch#2121)
This commit re-enables the OpenLDAP tests that were previously running against a one-off instance
in AWS but now run against a vagrant fixture. There were some IntegTests that would run against the
OpenLDAP instance randomly but with this change they no longer run against OpenLDAP. This is ok as
the functionality that is tested by these has coverage elsewhere.

relates elastic/x-pack-elasticsearch#1823

Original commit: elastic/x-pack-elasticsearch@ac9bc82297
2017-08-04 09:44:08 -06:00
David Roberts 39bc5886f2 [ML] Remove record_count from ML mappings (elastic/x-pack-elasticsearch#2183)
record_count is no longer written to new results, but is still tolerated
for backwards compatibility.  However, in the backwards compatibility case
the results index must already contain the required mapping.  There's no
need to add this mapping to newly created results indices.

Original commit: elastic/x-pack-elasticsearch@e586f3ba96
2017-08-04 10:52:35 +01:00
Jay Modi a7d6138f83 Fix the building of the default URL for the setup password tool (elastic/x-pack-elasticsearch#2176)
This commit fixes the building of the default URL for the setup password tool so that a default
elasticsearch.yml file will still result in a succesful run of the tool.

relates elastic/x-pack-elasticsearch#2174

Original commit: elastic/x-pack-elasticsearch@2291b14875
2017-08-03 15:14:24 -06:00
Dimitris Athanasiou 2d9d4c41d8 [ML] Add ability to assign groups to jobs (elastic/x-pack-elasticsearch#2155)
In particular:

  - adds a `groups` field in a job
  - group names can be used in multi-job APIs

relates elastic/x-pack-elasticsearch#2097

Original commit: elastic/x-pack-elasticsearch@c8517221ae
2017-08-03 17:32:05 +01:00
David Kyle 4b531c4dbb [ML] Check histogram interval is a divisor of bucketspan (elastic/x-pack-elasticsearch#2153)
Original commit: elastic/x-pack-elasticsearch@356dfa719c
2017-08-03 11:58:58 +01:00
Alexander Reelsen 65ea1b3bc4 Tests: Remove unneeded classes, fix AwaitsFix watcher tests (elastic/x-pack-elasticsearch#2139)
Fix TemplateTransformMappingTests to work, even if date rolls over
during execution.

Reenable test in BootStrapTests, was forgotten.

Remove the SecurityF/MonitoringF/WatcherF classes, as there is a gradle
command to easily start elasticsearch with xpack

Remove HasherBenchmark, as it is not a test and relies on RandomContext
that is not available anymore (also I think a JMH benchmark would be
needed here).

Remove ManualPublicSmtpServersTester, was not usable anymore.

Remove OldWatcherIndicesBackwardsCompatibilityTests, now in dedicated
rolling upgrade tests.

Remove unused EvalCron class.

Original commit: elastic/x-pack-elasticsearch@100fa9e9b0
2017-08-03 12:51:35 +02:00
Lisa Cawley a8fc1255be [DOCS] Update ml.enabled description (elastic/x-pack-elasticsearch#2148)
Original commit: elastic/x-pack-elasticsearch@8a32f0a66f
2017-08-02 10:32:24 -07:00
Tim Brooks dd899956e2 Use nio transport in x-pack tests (elastic/x-pack-elasticsearch#2159)
This commit is related to elastic/elasticsearch#25986. It updates x-pack
to use the randomized transport work in elasticsearch.

Original commit: elastic/x-pack-elasticsearch@eba2c0f815
2017-08-02 11:29:53 -05:00
Lisa Cawley bf92450fc4 [DOCS] Update multivariate_by_fields (elastic/x-pack-elasticsearch#2147)
Original commit: elastic/x-pack-elasticsearch@a26025ac5e
2017-08-02 08:25:46 -07:00
Jay Modi 573f365b56 Fix validation of username and password in CreateTokenRequest (elastic/x-pack-elasticsearch#2145)
This change fixes the validation of the the username and password field in the CreateTokenRequest
and adds a unit test to validate the fix.

relates elastic/x-pack-elasticsearch#2127

Original commit: elastic/x-pack-elasticsearch@b870683d39
2017-08-02 07:58:54 -06:00
David Kyle 7795d70414 [ML] Get Jobs/Datafeed Actions can run on local node (elastic/x-pack-elasticsearch#2095)
* No need to execute Get Jobs/Datafeed Actions on master node

Original commit: elastic/x-pack-elasticsearch@9d19010c5f
2017-08-02 11:24:28 +01:00
Dimitris Athanasiou 0125a332a1 [ML] Add support for mutli-job/multi-datafeed APIs (elastic/x-pack-elasticsearch#2079)
This commit enhances job/datafeed APIs that support acting
on multiple jobs/datafeeds at once so that they accept
expressions that may contain comma-separated lists or
wildcard patterns.

The APIs that are enhances are:

  - get jobs API
  - get job stats API
  - close job API
  - get datafeeds API
  - get datafeed stats API
  - stop datafeed API

relates elastic/x-pack-elasticsearch#1876

Original commit: elastic/x-pack-elasticsearch@45a1139d97
2017-08-02 11:10:06 +01:00
David Roberts c09430f3bf [TEST] Fix ML licensing tests following change in core test framework (elastic/x-pack-elasticsearch#2152)
The change made in elastic/elasticsearch#25986 means that half the time
tests will use the NIO transport rather than the mock TCP transport.
But the NIO plugin was not added to the TestXPackTransportClient.

Original commit: elastic/x-pack-elasticsearch@e465b0f801
2017-08-02 09:51:03 +01:00
Luca Cavanna 9fcb230d90 Adapt to upstream changes made to AbstractStreamableXContentTestCase (elastic/x-pack-elasticsearch#2117)
Original commit: elastic/x-pack-elasticsearch@0b1be31ffa
2017-08-02 08:43:09 +02:00
Jay Modi a35234de56 Setup password tool builds default URL from settings (elastic/x-pack-elasticsearch#2146)
This change makes the setup password tool build the default URL from the settings provided by the
environment. This will ease the amount of work a user would have to do in order to run the tool as
http vs https will be selected automatically and the port/host will as well.

Original commit: elastic/x-pack-elasticsearch@79affe4a79
2017-08-01 14:13:08 -06:00
Ryan Ernst 45a55d16cf Bump master version to 7.0.0-alpha1 (elastic/x-pack-elasticsearch#2135)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25876

Original commit: elastic/x-pack-elasticsearch@c86ea25009
2017-08-01 15:48:04 -04:00
Jay Modi ec11799003 Read the token passphrase earlier in the bootstrap check (elastic/x-pack-elasticsearch#2144)
This commit moves the reading of the token passphrase to the creation of the bootstrap check to
avoid issues with the secure settings keystore already being closed and thus causing issues during
startup.

Original commit: elastic/x-pack-elasticsearch@bba1cc832d
2017-08-01 13:04:34 -06:00
Alexander Reelsen 80baa1b83e Docs: Fixing failing docs build by removing leftover marker
Original commit: elastic/x-pack-elasticsearch@606a516529
2017-08-01 17:53:10 +02:00
David Kyle 466d421abe [ML] Missing validations in analysis config (elastic/x-pack-elasticsearch#2103)
* Don’t set detector field names to empty strings
* Check summary count field and categorisation field names are not empty strings
* Check a detector has a by field when using multivariate by fields

Original commit: elastic/x-pack-elasticsearch@461be12f9f
2017-08-01 15:54:46 +01:00
David Roberts 8487e1e0fe [ML] Better handling of errors if native controller dies (elastic/x-pack-elasticsearch#2141)
If the native controller dies or is killed then requests to open jobs
now immediately return with an error that says what the problem is.
The error that is logged also now clearly records the problem.

Previously open job requests would time out if the native controller
was not running, and logged errors were not easy to understand without
in-depth knowledge of the ML code.

relates elastic/x-pack-elasticsearch#2140

Original commit: elastic/x-pack-elasticsearch@fc7f074d4a
2017-08-01 15:53:58 +01:00
Jay Modi 7291eb55fe Automatically enable AES 256 bit TLS ciphers when available (elastic/x-pack-elasticsearch#2137)
This commit adds detection of support for AES 256 bit ciphers and enables their use when the JVM
supports them. For OpenJDK, this is often the case without any changes but for the Oracle JVM, the
unlimited policy file needs to be installed. In order to simplify the work a user would need to do
we can detect this support and automatically enable the AES 256 bit versions of the ciphers we
already enable.

Original commit: elastic/x-pack-elasticsearch@5f23b18a1e
2017-08-01 07:36:35 -06:00
Alexander Reelsen 4bf5d9536a Tests: Remove @ClusterScope tests, create REST tests (elastic/x-pack-elasticsearch#2131)
Replacing integration tests with rest tests and unit tests, thus removing integration tests that require start of a new cluster. Removing unused testing methods

Original commit: elastic/x-pack-elasticsearch@265966d80c
2017-08-01 14:15:36 +02:00
Alexander Reelsen 547b0ebc1b X-Pack: Remove deprecated handlers, remove watcher restart action (elastic/x-pack-elasticsearch#2133)
The deprecated handlers should have been removed earlier, but are now
going to to away finally.

Also the watcher restart action has been removed, mainly because users
should not blindly restart, but always make sure, that watcher is
stopped correctly before restarting. This had been removed from the
transport action previously.

Original commit: elastic/x-pack-elasticsearch@78a5ec3c05
2017-08-01 14:14:51 +02:00
Alexander Reelsen 681ce6e2aa Watcher: Replace flaky WatchStatsTests with unit tests (elastic/x-pack-elasticsearch#2118)
This flaky tests was using sleep, latches and a custom script plugin,
causing issues with stopping/starting tests.

This removes the integration tests and replaces it with a unit test.

Also removed a couple of unused ctor/method parameters as cleanup.

relates elastic/x-pack-elasticsearch#1639

Original commit: elastic/x-pack-elasticsearch@2a42faf2db
2017-08-01 09:50:32 +02:00
Alexander Reelsen 9ea032998b Watcher: Remove REST hijacking operations (elastic/x-pack-elasticsearch#2132)
As all the triggering is now done on the shards, people can use
bulk, *-by-query and regular index/delete operations on the watcher
index and do not need to use the dedicated APIs.

We can now remove the long existing rest operation hijacking code.

Original commit: elastic/x-pack-elasticsearch@08f4f4c3de
2017-08-01 09:49:28 +02:00
Jason Tedor 5b1bf9a31e Remove max script compilation settings in tests
Some standalone tests set the max script compilation limit. However,
this setting is now set in the main cluster setup in core so it is no
longer needed here. This commit removes these obviated settings in
standalone tests.

Original commit: elastic/x-pack-elasticsearch@089328c8d7
2017-08-01 14:57:57 +09:00
Jay Modi 7fecca6329 Fix handling of exceptions for concurrent attempts to create security index (elastic/x-pack-elasticsearch#2120)
This commit fixes the handling of some exceptions when we attempt to create the security index and
alias. The issue here is provoked by a test that is currently muted with an AwaitsFix,
GroupMappingTests, which will be unmuted in another change.

Original commit: elastic/x-pack-elasticsearch@55f6b656cb
2017-07-31 12:34:14 -06:00
Lisa Cawley af050a2da6 [DOCS] Move Reporting and Security out of X-Pack Reference (elastic/x-pack-elasticsearch#2134)
Original commit: elastic/x-pack-elasticsearch@3e007e0679
2017-07-31 09:55:08 -07:00
Luca Cavanna dac28fe4e0 Adapt to AbstractQueryBuilder upstream changes
AbstractQueryBuilder doesn't extend ToXContentToBytes anymore

Original commit: elastic/x-pack-elasticsearch@6c1b948ccb
2017-07-31 17:38:56 +02:00
Alexander Reelsen 9c937449b4 Docs: Add http timeout configuration parameters
Original commit: elastic/x-pack-elasticsearch@be4de8f8cc
2017-07-31 17:27:10 +02:00
Jason Tedor c51e6e1cd6 Remove unnecessary directory push/pop from scripts
This commit proposes removing an unnecessary directory push/pop from the
X-Pack scripts. It is not clear exactly why these were added, the
original change was almost three years ago in
elastic/x-pack@ea9ba7cdd0 but
unfortunately the commit message does not elucidate the exact the
problem, nor is there an associated pull request. This change has
propogated into all of the X-Pack scripts yet still the reasons are
unclear. The little that we can glean from the commit message is that
there was a problem with the default paths if the script was executed
outside of the Elasticsearch home. It seems that such issues have been
addressed by the recent introduction of elasticsearch-env so maybe we
can simplify these scripts here?

Relates elastic/x-pack-elasticsearch#2125

Original commit: elastic/x-pack-elasticsearch@9548c47743
2017-07-31 23:13:22 +09:00
Jason Tedor a80b1e4de1 Exit Windows scripts promptly on failure
When invoking the elasticsearch-env.bat or x-pack-env.bat batch scripts
on Windows, if these scripts exits due to an error (e.g., Java can not
be found, or the wrong version of Java is found), then the script
exits. Sadly, on Windows, this does not also terminate the caller,
instead returning control. This means we have to explicitly exit so that
is what we do in this commit.

Relates elastic/x-pack-elasticsearch#2126

Original commit: elastic/x-pack-elasticsearch@18645db62c
2017-07-31 20:39:59 +09:00