Commit Graph

1309 Commits

Author SHA1 Message Date
Dimitrios Athanasiou 5ad170a7ed Fix compilation due to upstream change
Original commit: elastic/x-pack-elasticsearch@57d834841d
2017-09-26 12:11:53 +01:00
Dimitris Athanasiou c46e09902d [ML] Fix close job when the process has not launched yet (elastic/x-pack-elasticsearch#2616)
If a job close is requested after a job was opened but before
its process was launched, the job close returns successfully
without doing anything. The result is that the process hangs
around. This has been causing test failures as documented
int elastic/x-pack-elasticsearch#2360 and elastic/x-pack-elasticsearch#1270.

This commit fixes this problem by refactoring the
AutodetectProcessManager. It introduces a state pattern
to make clear the states of the process and it uses locking
to ensure a close waits for the job process to be created.

relates elastic/x-pack-elasticsearch#1270

Original commit: elastic/x-pack-elasticsearch@ff858bd136
2017-09-26 11:46:01 +01:00
Tanguy Leroux a3984f7baa [Monitoring] Remove MonitoringSettings (elastic/x-pack-elasticsearch#2596)
Original commit: elastic/x-pack-elasticsearch@28dea7b699
2017-09-26 09:37:59 +02:00
jaymode 7b8d92a2a7 Test: fix AuthenticationService tests timeouts due to incorrect stream sizes
The AuthenticationService#testInvalidToken would cause a suite timeout in the case of an exception
due to a incorrect stream size as the latch was never counted down. This fixes the missing latch
countdown.

relates elastic/x-pack-elasticsearch#2615

Original commit: elastic/x-pack-elasticsearch@e838e6e912
2017-09-25 13:02:22 -06:00
Tim Brooks 6099660643 Update LicenseMetadata version check to be 6.1
This is related to elastic/x-pack-elasticsearch#1941. It modifies to the version check in
LicenseMetadata because this elastic/x-pack-elasticsearch#2419 has been backported to 6.x.

Original commit: elastic/x-pack-elasticsearch@052358b550
2017-09-25 09:42:37 -06:00
Tanguy Leroux 9bd2da86db [Tests] Reenable MonitoringIT
Checking the size of the map doesn't make sense when each filter is
checked independently right after.

Original commit: elastic/x-pack-elasticsearch@58e5d3401d
2017-09-25 16:01:18 +02:00
Simon Willnauer e7b5702f50 Adopt settings cleanups from core (elastic/x-pack-elasticsearch#2605)
Relates to elastic/elasticsearch#26739

Original commit: elastic/x-pack-elasticsearch@dd13d099de
2017-09-25 12:27:07 +02:00
Tanguy Leroux 1bb9c4fe71 [Tests] Mute MonitoringIT
see https://github.com/elastic/x-pack-elasticsearch/issues/2609

Original commit: elastic/x-pack-elasticsearch@e2e2393e62
2017-09-25 09:14:50 +02:00
Martijn van Groningen e93e7cf816 Revert "reset me"
This reverts commit elastic/x-pack-elasticsearch@3d44c14cd9.

Original commit: elastic/x-pack-elasticsearch@00203223b4
2017-09-25 08:56:01 +02:00
Martijn van Groningen ad3eb997c6 reset me
Original commit: elastic/x-pack-elasticsearch@3d44c14cd9
2017-09-25 08:52:51 +02:00
Jason Tedor b4f7d56c35 Whitelist global checkpoint sync actions
This commit whitelists the global checkpoint sync actions as otherwise
these actions do not have privileges to run as the system user and will
be denied as unauthorized.

Relates elastic/x-pack-elasticsearch#2604

Original commit: elastic/x-pack-elasticsearch@598ae1ff50
2017-09-22 16:27:51 -04:00
Tim Brooks 6b2e7fbed8 Add api to upgrade from basic to trial license (elastic/x-pack-elasticsearch#2419)
This is related to elastic/x-pack-elasticsearch#1941.

Currently we support self-generating either a basic or trial license at
cluster startup. With the addition of the basic option, it is possible
that a user would choose to self-generate and eventually register a
basic license.

This commit allows a user to upgrade to a 30-day trial license if they
have not already utilized this 30-day trial license before. Additionally
it adds a get route to check if the user is eligible to upgrade. This
route will allow kibana to implement a cleaner UI.

Original commit: elastic/x-pack-elasticsearch@7f19b33a08
2017-09-22 14:18:07 -06:00
Tanguy Leroux 6957282b50 [Monitoring] Fix MonitoringIT on Windows
Original commit: elastic/x-pack-elasticsearch@7a835c179d
2017-09-22 18:21:44 +02:00
Chris Earle 4cf8348209 [Monitoring] Remove Beats Support until wanted (elastic/x-pack-elasticsearch#2592)
This removes the creation and handling of the Beats monitoring template and its data until we actually expect to support it (most likely 6.2 - 6.3).

Original commit: elastic/x-pack-elasticsearch@2dc8abbb37
2017-09-22 11:12:44 -04:00
David Kyle ce77b076ad [ML] Adapt memory limit thresholds in test (elastic/x-pack-elasticsearch#2581)
* Adapt memory limit thresholds in test. Caused by elastic/machine-learning-cpp#305

Original commit: elastic/x-pack-elasticsearch@2984f35b2c
2017-09-21 12:52:31 +01:00
Albert Zaharovits f435145060 Fixes Javadoc generation for @SuppressForb adnot (elastic/x-pack-elasticsearch#2583)
Original commit: elastic/x-pack-elasticsearch@75ce6ac028
2017-09-21 14:28:57 +03:00
Albert Zaharovits a8ea4f2d90 Authz FieldExtractor support for TermInSet query (elastic/x-pack-elasticsearch#2539)
FieldExtractor for TermInSet query.

Closes: elastic/x-pack-elasticsearch#1104

Original commit: elastic/x-pack-elasticsearch@7a46361c86
2017-09-21 11:35:34 +03:00
Albert Zaharovits 6b51f5e6ca Update BouncyCastle version from 1.55 to 1.58 (elastic/x-pack-elasticsearch#2548)
Update BouncyCastle version from 1.55 to 1.58

Requires regeneration of .project and .classpath files, e.g.
gradle cleanEclipse & gradle eclipse

Closes: elastic/x-pack-elasticsearch#2332

Original commit: elastic/x-pack-elasticsearch@5806fd4204
2017-09-21 11:30:42 +03:00
Albert Zaharovits c84c48fa01 Usability improvement for the password bootstrap tool (reserved users) (elastic/x-pack-elasticsearch#2444)
Generate or prompt for passwords of the reserved users, validating strength
and typos, then issue change requests, keeping elastic user change request last.

Closes: elastic/x-pack-elasticsearch#2424

Original commit: elastic/x-pack-elasticsearch@1f827d393c
2017-09-21 11:26:28 +03:00
Tanguy Leroux 34bd72e69b [Monitoring] Remove monitoring resolvers (elastic/x-pack-elasticsearch#2566)
Now all monitoring documents are indexed in timestamped indices dedicated 
to each product, the old monitoring's resolvers and all the associated plumbing 
can be removed from the Elasticsearch monitoring plugin.

This pull request merges the feature branch feature/monitoring-summer-cleanup 
into master. All commits have been independently reviewed as part of elastic/x-pack-elasticsearch#2226:

* [Monitoring] Clean up monitoring doc (elastic/x-pack-elasticsearch#2208)
* [Monitoring] Remove ClusterStatsResolver (elastic/x-pack-elasticsearch#2209)
* [Monitoring] Remove IndexRecoveryResolver (elastic/x-pack-elasticsearch#2229)
* [Monitoring] Remove JobStatsResolver (elastic/x-pack-elasticsearch#2230)
* [Monitoring] Remove ShardsResolver (elastic/x-pack-elasticsearch#2233)
* [Monitoring] Fix missing instanceof in exporter bulks
* [Monitoring] Remove NodeStatsResolver (elastic/x-pack-elasticsearch#2244)
* [Monitoring] Remove IndexStatsResolver & IndicesStatsResolver  (elastic/x-pack-elasticsearch#2247)
* [Monitoring] Remove source node attributes and add timestamp (elastic/x-pack-elasticsearch#2285)
* [Monitoring] Remove last resolver (elastic/x-pack-elasticsearch#2330)
* [Monitoring] Make MonitoringDoc an abstract class (elastic/x-pack-elasticsearch#2451)
* [Monitoring] Fix TransportMonitoringBulkActionTests
* [Monitoring] Remove old resolver tests and add a single integration test (elastic/x-pack-elasticsearch#2453)

It does not introduce any new feature, so it could also go in 6.0.

relates elastic/x-pack-elasticsearch#2226


Original commit: elastic/x-pack-elasticsearch@e3d5cec28c
2017-09-21 09:36:03 +02:00
Tim Sullivan 723d894753 [Monitoring/Cluster Alerts] Fix the email message when cluster license expiration is resolved (elastic/x-pack-elasticsearch#2557)
* [Monitoring/Cluster Alerts] Fix the email message for cluster license expiration resolved

* fix making payload.message show only when new

Original commit: elastic/x-pack-elasticsearch@6d54b02913
2017-09-20 15:44:14 -07:00
Igor Motov 4964239f4c Tests: Remove AwaitsFix from TemplateUpgraderTests
I believe this problem should have been already fixed by elastic/elasticsearch#26698.

Original commit: elastic/x-pack-elasticsearch@0ec93f2803
2017-09-20 14:12:24 -04:00
Dimitrios Athanasiou a5e8589b38 [ML][TEST] Add integ test for elastic/x-pack-elasticsearch#2519
Relates elastic/x-pack-elasticsearch#2519

Original commit: elastic/x-pack-elasticsearch@91b38a62bb
2017-09-20 18:59:34 +01:00
Dimitris Athanasiou fad98d784f [ML] Align aggregated data extraction to histogram interval (elastic/x-pack-elasticsearch#2553)
When the datafeed uses aggregations and in order to accommodate
derivatives, an extra bucket is queried at the beginning of each
search. In order to avoid visiting the same bucket twice, we need
to search buckets aligned to the histogram interval. This allows
us to steer away from partial buckets, and thus avoid the problem
of dropping or duplicating data.

relates elastic/x-pack-elasticsearch#2519

Original commit: elastic/x-pack-elasticsearch@e03dde5fea
2017-09-20 16:45:06 +01:00
Hendrik Muhs 3c517902f2 re-enable categorization IT (elastic/x-pack-elasticsearch#2534)
re-enables categorization IT after fixing https://github.com/elastic/machine-learning-cpp/issues/279 upstream

Original commit: elastic/x-pack-elasticsearch@398a668125
2017-09-20 11:39:37 +02:00
Yannick Welsch 8648153f0e Deguice ActionFilter (elastic/x-pack-elasticsearch#2533)
Companion PR to elastic/elasticsearch#26691

Original commit: elastic/x-pack-elasticsearch@3fceb54809
2017-09-20 10:30:23 +02:00
Tal Levy 8b1021ccad blacklist two license rest tests when build.snapshot=false (elastic/x-pack-elasticsearch#2559)
There are two rest tests that rely on hardcoded license
signatures that use the dev public key. These tests fail
when tests are run with build.snapshot=false. This Commit
blacklists these two tests in that scenario

relates elastic/x-pack-elasticsearch#2527.

Original commit: elastic/x-pack-elasticsearch@7581e8d699
2017-09-19 16:56:11 -07:00
Nik Everett ea66433899 Mark periodically failing test as AwaitsFix
This should reduce our build failure emails. The failure is already
tracked at https://github.com/elastic/x-pack-elasticsearch/issues/2421.
It isn't actively being worked but will be soon.

Original commit: elastic/x-pack-elasticsearch@6baf55de42
2017-09-19 09:59:26 -04:00
David Roberts 67055877a5 [ML] Avoid creating spurious 0 "actual" values for model plot documents (elastic/x-pack-elasticsearch#2535)
Some model plot documents should not have an "actual" value, for example
when no input events were seen for a meean/min/max detector in a particular
bucket.  Prior to this change we would set the "actual" value to 0 for such
model plot documents.  Following this change no "actual" value will be
present in these documents.

Only newly created model plot documents are affected.  Model plot documents
that were incorrectly written in the past will remain wrong forever.

relates elastic/x-pack-elasticsearch#2528

Original commit: elastic/x-pack-elasticsearch@47a7365f59
2017-09-19 10:30:43 +01:00
Tim Vernum aec2308228 Allow AD realm to perform 'run-as' lookups (elastic/x-pack-elasticsearch#2531)
- Marks the AD Session factory as supporting "lookup" (Refer: elastic/x-pack-elasticsearch@40b07b3)
- Adds "pool.enabled" as a registered setting on AD realm (Refer: elastic/x-pack-elasticsearch@40b07b3)
- Fixes LDAP user lookup that has been broken since 6.x (Refer: elastic/x-pack-elasticsearch@f796949)

Original commit: elastic/x-pack-elasticsearch@62ff6129a1
2017-09-19 14:50:26 +10:00
Tal Levy 5a090c14c1 convert more request objects to writeable (elastic/x-pack-elasticsearch#2457)
* convert more to writeable

* migrate streamable tests to writeable tests

Original commit: elastic/x-pack-elasticsearch@56794e5760
2017-09-18 13:20:02 -07:00
Suyog Rao 3a9aad5ece [Logstash] Remove version field from config mgmt
relates elastic/x-pack-elasticsearch#2405

Original commit: elastic/x-pack-elasticsearch@5bfd1b7a6d
2017-09-18 12:35:21 -07:00
Simon Willnauer 47214426e9 Use InputStreamStreamInputs validation to limit size of tokens (elastic/x-pack-elasticsearch#2537)
Relates to elastic/elasticsearch#26692
relates elastic/x-pack-elasticsearch#2493

Original commit: elastic/x-pack-elasticsearch@8e23868743
2017-09-18 19:25:08 +02:00
Dimitris Athanasiou 9b25d0edf7 [ML] Correctly ignore extra aggregation buckets (elastic/x-pack-elasticsearch#2530)
The problem here is that the code was ignoring buckets
whose start time was before the start time of the extractor.
However, this is not a good enough condition. For example,
when there are no data in the bucket extra bucket that is
being queried, the first bucket will be the one containing
the start time.

This commit fixes the issue by changing the condition to
ignore buckets before the first bucket that includes the
start time of the extraction.

relates elastic/x-pack-elasticsearch#2519 

Original commit: elastic/x-pack-elasticsearch@15c7d2655f
2017-09-18 12:10:19 +01:00
Tim Vernum dea82a07a2 Infer KeyStore type from pathname (elastic/x-pack-elasticsearch#2514)
If the keystore type is not explicitly specified, infer it from the filename.
Treats .p12, .pfx and .pkcs12 as being PKCS12, all others as jks.

This will allow certgen to produce PKCSelastic/x-pack-elasticsearch#12 files by default and make it easy to use them as x-pack keystores

Original commit: elastic/x-pack-elasticsearch@fc361f0d87
2017-09-18 14:21:19 +10:00
Chris Earle 24c2c62ca2 [Monitoring] Ignore .marvel* indices (elastic/x-pack-elasticsearch#2520)
Beginning with 7.0, the cleaner service will no longer automatically cleanup .marvel indices regardless of their age.

Original commit: elastic/x-pack-elasticsearch@5b90e6f62a
2017-09-15 13:35:45 -07:00
Tim Brooks b3914afd30 Reenable TribeWithSecurityIT tests (elastic/x-pack-elasticsearch#2511)
This is related to elastic/x-pack-elasticsearch#1996. These tests were disabled during the bootstrap
password work. They can now be reenabled. Additionally, I made the test
password used in tests consistent.

Original commit: elastic/x-pack-elasticsearch@5b490c8231
2017-09-15 12:50:54 -06:00
Simon Willnauer 96e01dce47 Only require TLS for standard/gold/platinum licenses elastic/x-pack-elasticsearch#2517
relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@5213cf24f1
2017-09-15 20:21:15 +02:00
Chris Earle 9c9da2e1e4 [Monitoring] Remove Legacy Monitoring Indices (elastic/x-pack-elasticsearch#2513)
This changes Monitoring's Cleaner Service to remove any legacy Monitoring index that is appropriately old.

This includes any `.marvel-*` index and also the "data" indices used by both Marvel and 5.0 - 5.4 versions of X-Pack monitoring, as well as the legacy alerts index.

Original commit: elastic/x-pack-elasticsearch@8d99f5518b
2017-09-15 08:46:10 -07:00
jaymode 84dd719ab9 remove outdated comment
Original commit: elastic/x-pack-elasticsearch@06a51abb65
2017-09-15 09:03:36 -06:00
jaymode 344603e40f update text in TLSLicenseBootstrapCheck
Original commit: elastic/x-pack-elasticsearch@4ee6827566
2017-09-15 08:56:34 -06:00
Simon Willnauer 4d20586b24 [TEST] add integration test that ensures we reject license upgrades if TLS is not enabled
Original commit: elastic/x-pack-elasticsearch@dfbadb5e5f
2017-09-15 14:47:28 +02:00
Simon Willnauer c3066d1a51 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@9ce33bc7c3
2017-09-15 09:51:16 +02:00
Simon Willnauer 023bdb72b2 Add common-analysis plugin to several xpack integ tests (elastic/x-pack-elasticsearch#2501)
Several tests miss the common-analysis plugin in the old-style integ
tests causing odd exceptions in the test logs. This adds the missing plugin reference.

relates elastic/x-pack-elasticsearch#2363

Original commit: elastic/x-pack-elasticsearch@e4e6735408
2017-09-15 09:25:36 +02:00
jaymode 8997792875 Test: use TLS for plugin integ tests
Original commit: elastic/x-pack-elasticsearch@99971d7256
2017-09-14 15:57:28 -06:00
Michael Basnight fa0b854fb6 Update rest-api-spec to use bad_request (elastic/x-pack-elasticsearch#2507)
ref #elastic/elasticsearch#26539

Original commit: elastic/x-pack-elasticsearch@8b79a0769a
2017-09-14 15:59:29 -05:00
Andy Bristol 279c7e14fd [TEST] fix security template version check in rest tests (elastic/x-pack-elasticsearch#2506)
Since the template upgrade service was added, upgrades should
be performed by a node with the highest version in the cluster,
which may not be the master node.

Original commit: elastic/x-pack-elasticsearch@d66145de54
2017-09-14 12:16:20 -07:00
Jay Modi 57de66476c Disable TLS by default (elastic/x-pack-elasticsearch#2481)
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.

Relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@abc66ec67d
2017-09-14 12:18:54 -06:00
Simon Willnauer 1e14e14571 Prevent licenses to be upgraded to production unless TLS is configured (elastic/x-pack-elasticsearch#2502)
if a user tries to upgrade a license to a production license and has security
enabled we prevent the upgrade unless TLS is setup. This is a requirement now
if a cluster with security is running in prodcution.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@d61ef3bcb1
2017-09-14 20:14:27 +02:00
Jay Modi 8d8baffe24 Add specific client and user for security index access (elastic/x-pack-elasticsearch#2492)
This change removes security index access from the xpack user by creating its own specific role
and adds a xpack security user that maintains the superuser role so that it can perform all
operations necessary for security.

Original commit: elastic/x-pack-elasticsearch@ad906bc913
2017-09-14 08:09:14 -06:00
David Roberts 104a3a323f [BUILD] Make AWS error message more informative
Original commit: elastic/x-pack-elasticsearch@42cca7ed82
2017-09-14 14:46:48 +01:00
Simon Willnauer 91b57ee63f Add bootstrap check that enforces TLS if a production license is in the local clusterstate (elastic/x-pack-elasticsearch#2499)
This change will enforce transport SSL to be enforced if security is enabled and the
license in the clusterstate is a production license. The cluster state is loaded from
local storage such that we don't need to join a cluster to make these checks. Yet, the cluster
might have already got a different license if the node got disconnected while the license got
downgraded and then TLS got disabled. This corner case requires manual intervention which
we consider ok given the simplicity of this change.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@5765b7cd21
2017-09-14 13:52:53 +02:00
Hendrik Muhs 7d19264363 [ML-FC] Branch landing feature/ml (elastic/x-pack-elasticsearch#2500)
integrate forecasting feature branch into master

    - add endpoint xpack/ml/job/forecast to request forecasting on data of ml-jobs
       - current parameters: end time
    - persists forecast results into shared or own index
       - different runs are separated by a 'forecast id'

relates elastic/x-pack-elasticsearch#1838

Original commit: elastic/x-pack-elasticsearch@f9d701a6bc
2017-09-14 12:31:20 +02:00
Simon Willnauer 3b00251a96 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@4a36f0c2be
2017-09-14 07:43:19 +02:00
Jason Tedor 4f3e740ba8 Refactor bootstrap check results and error messages
This commit refactors the X-Pack bootstrap checks to respond to a change
in core Elasticsearch where the checks now return a single result
object.

Relates elastic/x-pack-elasticsearch#2495

Original commit: elastic/x-pack-elasticsearch@230b050529
2017-09-13 21:30:51 -04:00
Nik Everett f15666f82e Fix links in deprecation checks (elastic/x-pack-elasticsearch#2490)
Some links must have moved since we wrote the tests and released
5.6.0.

relates elastic/x-pack-elasticsearch#2488

Original commit: elastic/x-pack-elasticsearch@ebceee7f3d
2017-09-13 16:32:40 -04:00
Simon Willnauer 01a921a8e3 Accept BootstrapContext in xpack (elastic/x-pack-elasticsearch#2486)
This is the xpack side of elastic/elasticsearch#26628

Original commit: elastic/x-pack-elasticsearch@f6c0599ee2
2017-09-13 22:14:29 +02:00
Jay Modi f30e5c3fee Register the legacy truststore password setting for the PKI realm (elastic/x-pack-elasticsearch#2487)
After the addition of the secure settings in 5.6, the truststore.password setting for the PKI realm
was no longer registered. This would cause new nodes to fail for customers that were upgrading and
had configured a PKI realm with a truststore. This change registers the setting and adds a test to
ensure a realm configuration with the old setting passes validation.

Relates elastic/support-dev-help#2505

Original commit: elastic/x-pack-elasticsearch@54da044a27
2017-09-13 13:11:54 -06:00
Simon Willnauer 0680e41f36 Prevent nodes from joining a non-TLS enabled cluster with a production license (elastic/x-pack-elasticsearch#2484)
This change prevents a node from joining a cluster with a production license (gold, platinum, standard) iff the cluster doesn't have TLS setup. This is mainly a BWC oriented change that prevents joining old 5.x clusters without a TLS setup.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@21f5a58472
2017-09-13 20:40:35 +02:00
Dimitris Athanasiou 99ffbb1cd6 [ML] Add random offset to the maintenance task execution time (elastic/x-pack-elasticsearch#2483)
Currently the maintenance task is executed at 30 minutes past
midnight of each day. In the scenario where multiple clusters
are running on the same hardware infrastructure they all will
be running at the same time, competing for resources.

This commit changes this by adding a random offset to the
execution time which ranges from 0 to 119 minutes. The
minute granularity means that different offsets give at
least 1 minute for the maintenance task to end. Moreover,
the 2 hour window gives enough slots for different offsets
to occur and remains within what most people would think
as "middle of the night".

relates elastic/x-pack-elasticsearch#2273

Original commit: elastic/x-pack-elasticsearch@b538923aca
2017-09-13 14:53:44 +01:00
Dimitris Athanasiou e4753656bc [ML] Randomize default datafeed query delay (elastic/x-pack-elasticsearch#2475)
Changes the default query delay from 1m to a random
value between 1m and 2m. The motivation is to avoid
having multiple jobs firing their searches at the same
time which may potentially lead to increased load
on the machine.

relates elastic/x-pack-elasticsearch#2472

Original commit: elastic/x-pack-elasticsearch@3224e836fa
2017-09-13 09:12:39 +01:00
David Roberts 2e3aca414b [ML] Remove obsolete dynamic mapper setting (elastic/x-pack-elasticsearch#2477)
Since 5.6 we have only used one type per ML index, so this setting
is not necessary.

Original commit: elastic/x-pack-elasticsearch@64c434adec
2017-09-12 19:25:30 +01:00
jaymode 940e699e31 Remove index.mapper.dynamic from watcher and security templates
index.mapper.dynamic should not be used for 6.0+ indices, so this commit removes it from the
templates used by security and watcher.

Relates elastic/elasticsearch#25734

Original commit: elastic/x-pack-elasticsearch@766eebe660
2017-09-12 11:12:25 -06:00
Simon Willnauer 2f5aeb6c6f Remove token passphrase setting (elastic/x-pack-elasticsearch#2318)
This change removes `xpack.security.authc.token.passphrase` entirely since from
6.0 onwards we use randomly generated keys by the master there is no need for
this setting anymore. This setting will be deprecated from 6.0 onwards.

Original commit: elastic/x-pack-elasticsearch@37ba90359e
2017-09-12 15:34:41 +02:00
Alexander Reelsen c3f3ae5391 Watcher: Remove all traces from execution on master node (elastic/x-pack-elasticsearch#2383)
As there are no master node operations anymore.

* TransportActions are regular Actions now
* Watcher requests are now ActionRequests, no MasterNodeRequests anymore
* REST spec does not contain master node timeout parameters anymore
* WatcherLifeCycleService does not have a check anymore if watcher is able to run distributed, this will be a given in 7.0
* Some serialization BWC checks against version 5 have been removed

Original commit: elastic/x-pack-elasticsearch@4607dd538c
2017-09-12 15:05:26 +02:00
Dimitris Athanasiou e4882b36b7 [ML] Ensure datafeed runs on time (elastic/x-pack-elasticsearch#2471)
The datafeed runs on frequency-aligned intervals behind
query_delay. Currently, when a real-time run is triggered,
we subtract query_delay from now and then we take the aligned
interval. This results into running frequency + query_delay
behind now. The fix involves simply adding the query_delay
into the time real-time runs occur.

Relates elastic/x-pack-elasticsearch#2426

Original commit: elastic/x-pack-elasticsearch@61ceaaca8f
2017-09-12 13:24:55 +01:00
David Roberts 59d94eba40 [TEST] Mute failing test: CategorizationIT testBasicCategorization
See elastic/machine-learning-cpp#279

Original commit: elastic/x-pack-elasticsearch@03f0c307b7
2017-09-12 11:04:19 +01:00
David Roberts 1500074cb2 [ML] Add method to find the established memory use for a job (elastic/x-pack-elasticsearch#2449)
"Established" memory use will be one of the building blocks for smarter node
allocation.

In order for a job to be considered to have established memory usage it must:
- Have generated at least 20 buckets of results
- Have generated at least one model size stats document
- Have low variability of model bytes in model size stats documents in the
  time period covered by the last 20 buckets, which is defined as having a
  coefficient of variation of no more than 0.1

Relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@5032eb01d8
2017-09-12 10:25:53 +01:00
David Kyle bad65b4186 [ML] Add setting for job max model memory limit (elastic/x-pack-elasticsearch#2460)
* Add setting for job max model memory limit

* Address review comments

Original commit: elastic/x-pack-elasticsearch@5cec3a1abf
2017-09-11 14:53:46 +01:00
Ryan Ernst 53294f217c Use versionless alias to ES rest client codebase (elastic/x-pack-elasticsearch#2441)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/26521

Original commit: elastic/x-pack-elasticsearch@b650e9e433
2017-09-06 18:58:17 -07:00
Tim Brooks 84a47e2690 Disable `HipChatServiceTests` integration tests
This is related to elastic/x-pack-elasticsearch#2429. These tests are currently disabled due to
calls to the hipchat integration api failing. There is an open infra
issue for this (elastic/infra#2726).

Original commit: elastic/x-pack-elasticsearch@4aa9fe0387
2017-09-05 11:13:39 -06:00
David Roberts 500b4ac6b9 [TEST] Improve ML security tests (elastic/x-pack-elasticsearch#2417)
The changes made for elastic/x-pack-elasticsearch#2369 showed that the ML security tests were seriously
weakened by the decision to grant many "minimal" privileges to all users
involved in the tests.  A better solution is to override the auth header
such that a superuser runs setup actions and assertions that work by
querying raw documents in ways that an end user wouldn't.  Then the ML
endpoints can be called with the privileges provided by the ML roles and
nothing else.

Original commit: elastic/x-pack-elasticsearch@4de42d9e54
2017-09-05 10:49:41 +01:00
Alexander Reelsen cd5e001ca6 Watcher: Only load active watches on load (elastic/x-pack-elasticsearch#2408)
When watcher is loading it must only load the watches
which are active instead of all possible watches.

This loading happens on start up as well as when shards
relocate.

Original commit: elastic/x-pack-elasticsearch@29df56b99d
2017-09-01 16:20:42 +02:00
David Roberts 32b4c18ea3 [ML] Ensure internal client is used where appropriate (elastic/x-pack-elasticsearch#2415)
Implementation details of ML endpoints should be performed using the
internal client, so that the end user only requires permissions for
the public ML endpoints and does not need to know how they are
implemented.  This change fixes some instances where this rule was
not adhered to.

Original commit: elastic/x-pack-elasticsearch@01c8f5172c
2017-09-01 13:16:48 +01:00
David Kyle 6ba02e8087 [ML] Use the internal client for DeleteModelSnapshotAction (elastic/x-pack-elasticsearch#2407)
Original commit: elastic/x-pack-elasticsearch@7d3cbfa0cf
2017-08-31 18:16:30 +01:00
Tim Vernum 57a07d6b5a Authorize on shard requests for bulk actions (elastic/x-pack-elasticsearch#2369)
* Add support for authz checks at on shard requests

* Add Rest Tests for authorization

* Bulk security - Only reject individual items, rather than a whole shard

* Sync with core change

* Grant "delete" priv in ML smoketest

This role had index and+bulk privileges but it also needs delete (in order to delete ML model-snapshots)

Original commit: elastic/x-pack-elasticsearch@830e89e652
2017-08-31 11:49:46 -04:00
Alexander Reelsen 2033b027ed Tests: Ensure watcher is enabled/disabled during tests (elastic/x-pack-elasticsearch#2392)
The method to check if watcher was enabled was returning
`randomBoolean()` and thus could change during test runs.

This fixes the test to ensure that always the same value
is returned and documents this requirement.

relates elastic/x-pack-elasticsearch#1783

Original commit: elastic/x-pack-elasticsearch@97bf3cfc29
2017-08-31 12:55:05 +02:00
Alexander Reelsen 4cada797d7 Watcher: Replace System.currentTimeMillis() with nanotime() (elastic/x-pack-elasticsearch#2393)
Enjoy the luxury of monotonically increasing clocks. So that
the duration will never be zero.

Original commit: elastic/x-pack-elasticsearch@c934ff0adb
2017-08-31 11:57:25 +02:00
Alexander Reelsen 1121a15eb7 Watcher: Ensure emit_stacktraces parameter works in watcher stats (elastic/x-pack-elasticsearch#2399)
This parameter ceased to work when Elasticsearch 5 introduced strict
parameter handling, because of a missing test.

This commit adds the parameter to the rest handler responseParams()
and adds a test along with the needed YAML definition.

relates elastic/x-pack-elasticsearch#2396

Original commit: elastic/x-pack-elasticsearch@8638df336c
2017-08-31 09:19:07 +02:00
Hendrik Muhs 381bb4ed7d add a test case to ensure NaN's aren't accepted (elastic/x-pack-elasticsearch#2395)
Original commit: elastic/x-pack-elasticsearch@72879c4f7c
2017-08-31 07:41:48 +02:00
David Kyle b818ee285b Revert "[ML] Check influencer names are valid (elastic/x-pack-elasticsearch#2073)"
This reverts commit elastic/x-pack-elasticsearch@75869cacb3.

Original commit: elastic/x-pack-elasticsearch@1f582ed260
2017-08-30 15:19:49 +01:00
Alexander Reelsen 03ec9df1d8 Tests: Remove is_true assertion that can return zero
Original commit: elastic/x-pack-elasticsearch@220d87fe4e
2017-08-30 15:46:38 +02:00
Dimitrios Athanasiou 5c145224a5 [ML] Add integrations tests for memory limit compliance
Relates elastic/x-pack-elasticsearch#2098

Original commit: elastic/x-pack-elasticsearch@730835edfa
2017-08-30 14:27:06 +01:00
Alexander Reelsen 200cb6a5b3 Tests: Remove is_true check in REST test
If the duration time was 0 (and this might happen due to
using System.currentTimeMillis), the is_true check
still returns false.

The correct fix will be done later to replace the offending
measurement calls and replace them. Then we can add back this
line.

Original commit: elastic/x-pack-elasticsearch@076a9a37cc
2017-08-30 14:04:24 +02:00
Alexander Reelsen 58d16c3579 Tests: Disable watcher in tribe TribeTransportTestCase
Watcher does not need to run in those tests. Disabling it,
no need for careful stopping then.

Relates elastic/x-pack-elasticsearch#1783

Original commit: elastic/x-pack-elasticsearch@ecca7f2ebd
2017-08-30 13:48:54 +02:00
Alexander Reelsen 80df309bbd Tests: Dont check for execution speed, only that field exists
Execution might be fast in combination with the clock not being counted
fast enough. As we are only interested in the field existing, we dont
need to check for the value.

relates elastic/x-pack-elasticsearch#2391

Original commit: elastic/x-pack-elasticsearch@8f3df967d9
2017-08-30 13:38:50 +02:00
Alexander Reelsen 4ac13fa9c5 Watcher: Replace integration tests with unit and REST tests (elastic/x-pack-elasticsearch#2322)
A bunch of integration tests should have been built as unit tests
or already have unit test equivalents.

This commit removes integration tests as well as adding REST equivalents
or creating unit tests instead of extending from AbstractWatcherIntegrationTestCase

Original commit: elastic/x-pack-elasticsearch@a97b99467d
2017-08-30 11:46:28 +02:00
Alexander Reelsen 17de601e21 Watcher: Remove the BWC compatible old watcher stats (elastic/x-pack-elasticsearch#2323)
Those were needed for the 5.x to 6.x transition phase, but can
be removed for 7.x.

Original commit: elastic/x-pack-elasticsearch@75572bd0c8
2017-08-30 11:45:57 +02:00
Ryan Ernst 3ef93c55b0 Build: Add build metadata generation and usage to xpack (elastic/x-pack-elasticsearch#2368)
This commit adds generation of build_metadata files from bwc checkouts
as well as the sibling ES checkout of the xpack ci script.

relates elastic/elasticsearch#26397

Original commit: elastic/x-pack-elasticsearch@e17d904beb
2017-08-28 14:03:02 -07:00
Jim Ferenczi 27d8b4c79c Remove the _all metadata field (elastic/x-pack-elasticsearch#2356)
This change removes the `_all` metadata field. This field is deprecated in 6
and cannot be activated for indices created in 6 so it can be safely removed in
the next major version (e.g. 7).

Relates https://github.com/elastic/elasticsearch/pull/26356

Original commit: elastic/x-pack-elasticsearch@a47133c94e
2017-08-28 13:01:27 +02:00
Michael Basnight e18f04f3eb Revert "Use shaded rest client dependencies" (elastic/x-pack-elasticsearch#2352)
This reverts commit elastic/x-pack-elasticsearch@8605560232.

Relates elastic/elasticsearch#26367

Original commit: elastic/x-pack-elasticsearch@e4cd960504
2017-08-25 14:13:16 -05:00
Alexander Reelsen 9b0a1a34e0 Upgrade: Remove watcher/security upgrade checks (elastic/x-pack-elasticsearch#2338)
The checks are used for the 5.6 to 6.x transition, thus they do
not make sense to keep in 7.x.

Original commit: elastic/x-pack-elasticsearch@c6c6fa819e
2017-08-25 17:24:49 +02:00
David Kyle 5a63c4d9a2 [ML] Remove trailing slashes from ML rest spec (elastic/x-pack-elasticsearch#2350)
Original commit: elastic/x-pack-elasticsearch@f2a3d70562
2017-08-25 09:35:07 +01:00
David Kyle 175e8db3aa [ML] Don’t count incomplete buckets in data stream diagnostics (elastic/x-pack-elasticsearch#2351)
* Don’t count incomplete buckets in data stream diagnostics

* Fix tests now bucket_count doesn’t include partial buckets

Original commit: elastic/x-pack-elasticsearch@bc1a7bd9e7
2017-08-25 09:25:15 +01:00
Ryan Ernst 13672dad13 Build: Set xpack to require keystore
See https://github.com/elastic/elasticsearch/pull/26329

Original commit: elastic/x-pack-elasticsearch@e77361a6d5
2017-08-24 14:09:07 -07:00
Jay Modi 8e4ded82c0 Adapt to resync rename and add assertion for missing user when sending a request (elastic/x-pack-elasticsearch#2345)
This commit adapts to the renaming of the TransportResyncReplicationAction in core and also adds
an assertion to the check for a user being present when sending a request. The assertion is added
so that we can hopefully catch these scenarios in our testing as the assertion error will cause the
node to die but the ISE will just be seen in the logs. Since we do not run with assertions enabled
in production, the ISE is left to handle those cases.

relates elastic/x-pack-elasticsearch#2335

Original commit: elastic/x-pack-elasticsearch@c5ce0c93af
2017-08-24 11:04:41 -06:00
Colin Goodheart-Smithe d0103d6cab [TEST] Fixes PageParamsTests to no underflow from and size
In `mutateInstance()` the from or size could become negative if the other one was pushed over the limit for `from + size`. This change fixes this case to make sure after the mutate method is called the from and size obey the limit but are also both `>= 0`

relates elastic/x-pack-elasticsearch#2344

Original commit: elastic/x-pack-elasticsearch@a8a7072fcc
2017-08-24 09:29:36 +01:00
Christoph Büscher f05568e7b3 Revert adding bin to .gitignore (elastic/x-pack-elasticsearch#2346)
Original commit: elastic/x-pack-elasticsearch@cfd2be3831
2017-08-23 22:43:01 +02:00
Jason Tedor 85e494d90f Add x-pack-env.bat
This file was missing from the commit which depends on this file due to
a .gitignore rule. This commit adds this vital file.

Relates elastic/x-pack-elasticsearch#2124

Original commit: elastic/x-pack-elasticsearch@78125b56de
2017-08-23 14:16:36 -04:00
Stacey Gammon 1bfa4cb8bb Add reserved dashboards_only_user role (elastic/x-pack-elasticsearch#2250)
* Add reserved dashboards_only_user role

* Fix line too long

* add tests for new reserved role

* rename role, hopefully fix tests

* Fix test

Original commit: elastic/x-pack-elasticsearch@99f6718c7c
2017-08-23 13:16:17 -04:00
David Kyle 71063825be [MLFix bucket setting count (elastic/x-pack-elasticsearch#2339)
Original commit: elastic/x-pack-elasticsearch@6801e90d54
2017-08-23 15:46:51 +01:00
Alexander Reelsen 6ee4fe6a0b Watcher: Improve upgrade API logging message to include index names
Original commit: elastic/x-pack-elasticsearch@9cecc11f88
2017-08-23 16:11:49 +02:00
Albert Zaharovits 026729e911 TOKEN_SERVICE_ENABLED_SETTING enabled if HTTP_SSL_ENABLED (elastic/x-pack-elasticsearch#2321)
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.

* TokenService default enabled if HTTP_ENABLED == false

* Fixed tests that need TokenService explicitly enabled

* [DOC] Default value for `xpack.security.authc.token.enabled`

Original commit: elastic/x-pack-elasticsearch@bd154d16eb
2017-08-23 13:21:30 +03:00
Colin Goodheart-Smithe de6c275dca Refactor/to x content fragments2 (elastic/x-pack-elasticsearch#2329)
* Moves more classes over to ToXContentObject/Fragment

* Removes ToXContentToBytes

* Removes ToXContent from Enums

* review comment fix

* slight change to use XContantHelper

Original commit: elastic/x-pack-elasticsearch@0f2d3f328b
2017-08-23 08:17:28 +01:00
David Kyle 18d15ef2d2 Revert "[ML] Missing validations in analysis config (elastic/x-pack-elasticsearch#2103)"
This reverts commit elastic/x-pack-elasticsearch@461be12f9f.

Original commit: elastic/x-pack-elasticsearch@1ce7196710
2017-08-22 13:33:40 +01:00
Yannick Welsch b4353b55ad Allow build to directly run under JDK 9 (elastic/x-pack-elasticsearch#2320)
With Gradle 4.1 and newer JDK versions, we can finally invoke Gradle directly using a JDK9 JAVA_HOME without requiring a JDK8 to "bootstrap" the build. As the thirdPartyAudit task runs within the JVM that Gradle runs in, it needs to be adapted now to be JDK9 aware.

Relates to elastic/elasticsearch#25859

Original commit: elastic/x-pack-elasticsearch@4bf266e0b0
2017-08-22 14:46:37 +09:30
Alexander Reelsen 27f39c615b Watcher: Create two index ugprade checks for watcher upgrade (elastic/x-pack-elasticsearch#2298)
As there are two indices to upgrade for watcher, it makes a lot of sense
to also have two upgrade checks.

There is one upgrader for the watches index, which deletes
old templates, adds the new one before and then does the reindexing.
Same for the triggered watches index.

This also means, that there will be two entries popping up in the kibana
UI.

Note: Each upgrade check checks if the other index (for the .watches
upgrade check the triggered watches index and vice versa) is already
upgraded and only if that is true, watcher is restarted.

relates elastic/x-pack-elasticsearch#2238

Original commit: elastic/x-pack-elasticsearch@2c92040ed6
2017-08-21 17:36:16 +02:00
Jason Tedor 8a5be5b58e Replace atomic move writer
I noticed this while working on a previous issue with atomic move writer
(silent swallowing of exceptions). Namely, atomic move writer has
dangerous semantics. The problem is as follows: atomic move writer works
by writing lines to a temporary file, and then in its close method it
replaces the target path with the temporary file. However, the close
method is invoked whether or not all writes to the temporary file
succeeded (because writers obtained from atomic move writer are used in
try-with-resources blocks, as they should be). There is no way to
distinguish that the writer is being closed in a successful scenario
versus a failure scenario. In the close method for atomic move writer,
the target file is replaced by the temporary file. This means that the
target file is replaced whether or not writing to the temporary file
actually succeeded. Since these atomic move writers are used for user
configuration files (users and user_roles), a failure here can lead to
data loss for the user, a tragedy!

There is another (less serious) problem with the atomic move
writer. Since the close method tries to move the temporary file in place
of the existing file, the temporary file can be left behind if there is
another failure in the close method (e.g., closing the underlying file
after writing, or setting the permissions on the temporary file). This
means that in some situations, atomic move writer will leave temporary
files behind (which is not definitively not atomic).

This commit replaces the atomic move writer with a safer mechanism. We
still perform the write atomically in the following sense: we write to a
temporary file. Either writing to that file succeeds or it fails. If
writing succeeds, we replace the existing file with the temporary
file. If writing fails, we clean up the temporary file and the existing
file remains in place.

Relates elastic/x-pack-elasticsearch#2299


Original commit: elastic/x-pack-elasticsearch@3199decb0a
2017-08-21 10:35:37 -04:00
David Roberts e428c58dff [TEST] Fix logic in one branch of random mutate test
Certain types of datafeeds cannot have null chunking configs, so
setting chunking config to null sometimes doesn't stick as null

Original commit: elastic/x-pack-elasticsearch@3a52bad460
2017-08-21 14:51:04 +01:00
Colin Goodheart-Smithe 0a1225f934 [TEST] Adds mutate function to some tests (elastic/x-pack-elasticsearch#2263)
Original commit: elastic/x-pack-elasticsearch@560d7e9a80
2017-08-21 11:20:14 +01:00
Alexander Reelsen 1b6d9d430c Watcher: Load for watch for execution as late as possible (elastic/x-pack-elasticsearch#2151)
When a watch is executed currently, it gets passed an in-memory
watch object, that was loaded, before the execution started.

This means there is a window of time, where an old watch could still
be executing, then a watch gets loaded for execution, then the old watch
execution finishes and updates the watch status and thus reindexes the
watch.

Now the watch, that got loaded for execution, executes and tries to
store its watch status, but fails, because the version of the watch
has changed.

This commit changes the point in time where the watch is loaded. Now
this only happens, while a watch is in its protected execution block,
and thus we can be sure, that there is no other execution of the watch
happening.

This will primarily impact watches, that execute often, but their
runtime is longer than the configured interval between executions.

Side fix: Removed some duplicate testing method and moved into 
WatcherTestUtils, fixed a tests with a ton of if's with random booleans
into separate tests.

relates elastic/x-pack-elasticsearch#395

Original commit: elastic/x-pack-elasticsearch@bf393023d7
2017-08-21 10:43:41 +02:00
Simon Willnauer 44c3d5b3d9 Allow to change bwc refspec in xpack (elastic/x-pack-elasticsearch#2311)
Today it's impossible to run xpack bwc tests against any other branch
or remote than upstream. This allows to pass `-Dtests.bwc.refspec` to
change the refspec to use for the bwc tests.

Original commit: elastic/x-pack-elasticsearch@4d365f5a6e
2017-08-21 09:59:45 +02:00
Simon Willnauer 846f40ba15 [TEST] allow test to start an additional node even if max number of random nodes is reached
relates elastic/x-pack-elasticsearch#2315

Original commit: elastic/x-pack-elasticsearch@d989c06198
2017-08-19 11:03:46 +02:00
Simon Willnauer 71827b70a0 Bump token service BWC version to 6.0.0-beta2
Original commit: elastic/x-pack-elasticsearch@ef688f02cb
2017-08-18 17:06:45 +02:00
David Roberts 4f17335629 [ML] Add unit tests for ML roles (elastic/x-pack-elasticsearch#2309)
I forgot to add these when the roles were originally added

Original commit: elastic/x-pack-elasticsearch@0e72141b11
2017-08-18 14:04:41 +01:00
Simon Willnauer ac9ab974f4 Ensure token service can boostrap itself without a pre-shared key (elastic/x-pack-elasticsearch#2240)
Today we require a pre-shared key to use the token service. Beside the
additional setup step it doesn't allow for key-rotation which is a major downside.

This change adds a TokenService private ClusterState.Custom that is used to distribute
the keys used to encrypt tokens. It also has the infrastructur to add automatic key
rotation which is not in use yet but included here to illustrate how it can work down
the road.

This is considered a prototype and requires additioanl integration testing. Yet, it's fully
BWC with a rolling / full cluster restart from a previous version (also from 5.6 to 6.x)
since if the password is set it will just use it instead of generating a new one.
Once we implement the automatic key rotation via the clusterstate we need to ensure that we are
fully upgraded before we do that.
Also note that the ClusterState.Custom is fully transient and will never be serialized to disk.

Original commit: elastic/x-pack-elasticsearch@1ae22f5d41
2017-08-18 14:23:43 +02:00
David Roberts 269f0f6a19 [ML] Default model memory limit to 1GB for newly created jobs (elastic/x-pack-elasticsearch#2300)
This change means that newly created jobs will get an explicit 1GB
model memory limit if no model memory limit is specified when creating
the job.  Existing jobs that had a null model memory limit will carry
on using the default model memory limit defined in the C++ code.

Relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@a4e6b73c2b
2017-08-18 09:36:09 +01:00
Jason Tedor 5125978f60 Enable command extensions
We rely on command extensions in our scripts but we do not actually
guarantee that they are enabled (usually they are, by default, but they
can be disabled outside of our control). This commit ensures that they
are enabled.

Relates elastic/x-pack-elasticsearch#2307

Original commit: elastic/x-pack-elasticsearch@a5eec8ca7b
2017-08-17 22:50:52 -04:00
Simon Willnauer 724325f161 Fallback to `keystore.seed` as a bootstrap password if actual password is not present (elastic/x-pack-elasticsearch#2295)
Today we require the `bootstrap.password` to be present in the keystore in order to
bootstrap xpack. With the addition of `keystore.seed` we have a randomly generated password
per node to do the bootstrapping. This will improve the initial user experience significantly
since the user doesn't need to create a keystore and add a password, they keystore is created
automatically unless already present and is always created with this random seed.

Relates to elastic/elasticsearch#26253

Original commit: elastic/x-pack-elasticsearch@5a984b4fd8
2017-08-17 16:42:32 +02:00
David Roberts 2ae5634dc9 [ML] Improve error message when auto-close isn't attempted (elastic/x-pack-elasticsearch#2296)
The old message of "Cannot auto close job" implied the problem was with
closing the job.  This change makes it clearer that the problem is that
the datafeed could not be stopped and hence auto-close will not even be
attempted.

Original commit: elastic/x-pack-elasticsearch@065e9930ce
2017-08-17 15:07:06 +01:00
David Roberts 44857d71b3 Make AllocatedPersistentTask members volatile (elastic/x-pack-elasticsearch#2297)
These members are default initialized on contruction and then set by the
init() method.  It's possible that another thread accessing the object
after init() is called could still see the null/0 values, depending on how
the compiler optimizes the code.

Original commit: elastic/x-pack-elasticsearch@668121e274
2017-08-17 14:54:31 +01:00
Colin Goodheart-Smithe 751680e7b2 Moves more classes over to ToXContentObject/Fragment (elastic/x-pack-elasticsearch#2283)
Original commit: elastic/x-pack-elasticsearch@73c6802523
2017-08-17 11:16:56 +01:00
Alexander Reelsen 6d30806996 Watcher: Improvements on the rolling restart tests (elastic/x-pack-elasticsearch#2286)
This improves the rolling restart tests (tests different paths in
different ways) and aligns the upgrade code with the 5.6 branch from

Relates elastic/x-pack-elasticsearch#2238

Original commit: elastic/x-pack-elasticsearch@01b0954558
2017-08-17 11:41:11 +02:00
Simon Willnauer 8f15324a08 Don't bootstrap security index on start-up but authenticate bootstrap password locally (elastic/x-pack-elasticsearch#2272)
Today we try to bootstrap the security index with the bootstrap password and recommend the user to change the password with the user tool. This is trappy for instance if you happen to configure multiple nodes with a different bootstrap passwords (which is possible) it's unclear which password made it too bootstrap. Yet, we tell in the logs but it can still be very confusing. In general it should be possible to bootstrap with the user tool from any node unless the user is already created in the native user store. This change uses the bootstrap.password from the local node and always authenticate against it until the user is bootstrapped even if the passwords are different on different nodes. This will also work for authenticating against the cluster for instance if a user deletes the .security index or if that index has not been upgraded.

Original commit: elastic/x-pack-elasticsearch@8cebecb287
2017-08-17 08:36:26 +02:00
Jason Tedor 2bf8f4b0bc Remove print writer wrapping for users tools
When writing the users and users_roles files, we wrap a custom writer in
a print writer. There is a problem with this though: when print writer
closes it closes our underlying custom writer and the close
implementation for our custom writer is not trivial, it executes code
that can throw an I/O exception. When print writer invokes this close
and an I/O exception is thrown, it swallows that exception and sets the
status on the print writer to error. One would think that we could
simply check this status but alas print writer is broken here. The act
of checking the status causes print writer to try to flush the
underyling stream which is going to be completely undefined because the
underlying stream might or might not be closed. This might cause another
exception to be thrown, losing the original. Print writer screwed the
pooch here, there is no good reason to try to do any I/O after the
underlying writer entered a failed state. To address this we remove the
use of print writer, we use our custom writer directly. This allows any
thrown exceptions to bubble up.

Relates elastic/x-pack-elasticsearch#2288

Original commit: elastic/x-pack-elasticsearch@11b8dd5641
2017-08-16 12:50:39 -04:00
David Roberts 6fcc3be438 [ML] Preserve _meta on results index mapping update (elastic/x-pack-elasticsearch#2274)
When mappings are updated for an index are updated most settings are
merged, but not _meta.  This change ensures that _meta is set when we
add per-job term mappings to our results index mappings.  In order to
keep the logic for updating mappings after upgrade working, we now
have to put ALL the mappings for our results along with the latest _meta
section when updating per-job term mappings.

relates elastic/x-pack-elasticsearch#2265

Original commit: elastic/x-pack-elasticsearch@f58c11a13e
2017-08-16 16:16:30 +01:00
Simon Willnauer 8b23f133c7 Create security bootstrap checks early to access secure settings safely (elastic/x-pack-elasticsearch#2282)
We close the secure settings in core before we pull bootstrap checks.
This means if a bootstrap check like the `TokenPassphraseBootstrapCheck`
accesses a secure setting that late it will fail due to an exception in
the `PKCS12KeyStore`. This change moves the bootstrap check creation
to the plugin constructor and adds a dummy setting to the integTest
that triggers the bootstrap checks.

Original commit: elastic/x-pack-elasticsearch@2b20865d1c
2017-08-16 13:01:52 +02:00
Yannick Welsch fd76651d92 Expose timeout of acknowledged requests in REST layer (elastic/x-pack-elasticsearch#2259)
Companion PR to elastic/elasticsearch#26189

Original commit: elastic/x-pack-elasticsearch@f561e22835
2017-08-16 07:43:18 +08:00
David Roberts db8885a46e [ML] Do not download the ml-cpp zip when building it locally (elastic/x-pack-elasticsearch#2262)
When the machine-learning-cpp repo is built locally, the zip file it
creates is preferred over that downloaded from s3 when creating the
overall x-pack-elasticsearch zip.  However, prior to this change the
build would ALSO download an ml-cpp zip from s3, and just not use it.

Original commit: elastic/x-pack-elasticsearch@bd71637edd
2017-08-15 16:31:23 +01:00
Jason Tedor f3a7d46698 Rename CONF_DIR to ES_PATH_CONF
This commit is following upstream Elasticsearch which has renamed the
environment variable used to specify a custom configuration directory
from CONF_DIR to ES_PATH_CONF.

Relates elastic/x-pack-elasticsearch#2261

Original commit: elastic/x-pack-elasticsearch@9ae29941e5
2017-08-15 06:19:39 +09:00
Tim Vernum a27dc257c9 Gracefully handle no content(-type) in Put License (elastic/x-pack-elasticsearch#2258)
PUT /_xpack/license with no content or content-type should fail with an appropriate error message rather than throwing NPE.

Original commit: elastic/x-pack-elasticsearch@f8c744d2a2
2017-08-14 20:39:39 +10:00
Alexander Reelsen 5416a6afd4 Tests: Fix timeout in watcher history template test
Due to an invalid timeout the test failed earlier than it should have.

relates elastic/x-pack-elasticsearch#2222

Original commit: elastic/x-pack-elasticsearch@2265c419e3
2017-08-14 09:53:52 +02:00
David Roberts b5d159bc1c [ML] Handle simultaneous force delete datafeed and stop datafeed (elastic/x-pack-elasticsearch#2243)
This is an important case as the UI force stops datafeeds now.

Fixes elastic/x-pack-kibana#2083

Original commit: elastic/x-pack-elasticsearch@4d0f62ad2d
2017-08-11 14:43:24 +01:00
David Roberts cb3f3d2d04 [ML] Switch from max_running_jobs to xpack.ml.max_open_jobs (elastic/x-pack-elasticsearch#2232)
This change makes 2 improvements to the max_running_jobs setting:

1. Namespaces it by adding the xpack.ml. prefix
2. Renames "running" to "open", because the "running" terminology
   is not used elsewhere

The old max_running_jobs setting is used as a fallback if the new
xpack.ml.max_open_jobs setting is not specified.  max_running_jobs
is deprecated and (to ease backporting in the short term) will be
removed from 7.0 in a different PR closer to release of 7.0.

Relates elastic/x-pack-elasticsearch#2185

Original commit: elastic/x-pack-elasticsearch@18c539f9bb
2017-08-11 09:00:33 +01:00
Alexander Reelsen 5f30508efd Tests: Remove AwaitsFix annotation and increase logging
This test does not reproduce locally but fails regularly in CI.
Added more logging and proper comments.

Relates elastic/x-pack-elasticsearch#2222

Original commit: elastic/x-pack-elasticsearch@bf6b590629
2017-08-11 09:44:09 +02:00
Alexander Reelsen 11334b2df3 Tests: Fix TimeThrottleIntegrationTests to not rely on shard actions (elastic/x-pack-elasticsearch#2234)
These tests used to fail rarely, because during a watch execution
one of the watcher shards was relocated resulting in a second execution
of watch.

In order to prevent this, the tests do not need to actually create any
shards, which causes watcher potentially to be rebalanced.

This simplifies and speeds up the test as well.

relates elastic/x-pack-elasticsearch#1608

Original commit: elastic/x-pack-elasticsearch@1cfac1145d
2017-08-11 09:19:25 +02:00
Alexander Reelsen 35ee552923 Tests: Fix logger initialization to include hostname in timewarped watcher
This cleans up logging, when starting several elasticsearch instances,
as otherwise you cannot see, which node emits this log message.

Original commit: elastic/x-pack-elasticsearch@c8c2819d86
2017-08-10 14:09:14 +02:00
Alexander Reelsen ceb13988e3 Watcher: Do not update watch status state during execution (elastic/x-pack-elasticsearch#2204)
When a watch is executed, it sends an update request to the watch to
udpate its status.

This update request also updates the status.state field, which contains
information, if the watch is active. If the watch gets executed, and
during execution a watch gets disabled, then the current execution will
set the watch back to active.

This commit fixes the current behaviour and never changes the state of
a watch when updating the status after executing, allowing
activate/deactivate calls to work as expected, regardless if a watch
is being executed.

This will fix not only the current behaviour but also some flaky tests.

Original commit: elastic/x-pack-elasticsearch@ca69109ecb
2017-08-10 13:16:39 +02:00
Alexander Reelsen 4012da662d Tests: Add @AwaitsFix to failing test
Original commit: elastic/x-pack-elasticsearch@545155ec13
2017-08-10 10:17:22 +02:00
Colin Goodheart-Smithe a4dd177978 Migrates ToXContent classes (elastic/x-pack-elasticsearch#2024)
* Migrates ToXContent classes

* review comments

Original commit: elastic/x-pack-elasticsearch@74ce3755ab
2017-08-09 15:55:04 +01:00
Alexander Reelsen 0b5909fc65 Watcher: Stop swallowing exceptions, always return them instead of message (elastic/x-pack-elasticsearch#1933)
It is really hard to debug some issues with watcher, when only the
e.getMessage() is returned as failure reasons instead of the whole
stack trace.

This commit gets rid of ExceptionsHelper.detailedMessage(e) and always
returns the whole exception.

This commit also extends the watch history to have all fields named
error be treated like an object to be sure they do not get
indexed. No matter where it's placed in the hierarchy

In addition a few Field interface classes were removed, that only contained parse fields.

relates elastic/x-pack-elasticsearch#1816

Original commit: elastic/x-pack-elasticsearch@b2ce680139
2017-08-08 18:36:22 +02:00
David Roberts 22da5cf89e [ML] Add max open jobs per node as a node attribute (elastic/x-pack-elasticsearch#2203)
This commit adds the max_running_jobs setting from elasticsearch.yml
into a node attribute called ml.max_open_jobs.  Previously there was
an assumption that max_running_jobs would be the same for all nodes in
the cluster.  However, during a rolling cluster restart where the value
of the setting is being changed this clearly cannot be the case, and
would cause unexpected/unpredictable limits to be used during the period
when different nodes had different settings.

For backwards compatibility, if another node in the cluster has not added
its setting for max_running_jobs to the cluster state then the old
(flawed but better than nothing) approach is applied, i.e. assume the
remote node's setting for max_running_jobs is equal to that of the node
deciding the job allocation.

Relates elastic/x-pack-elasticsearch#2185

Original commit: elastic/x-pack-elasticsearch@1e62b89183
2017-08-08 16:16:27 +01:00
Dimitris Athanasiou 2864078da2 [ML] Move job group validation after parsing (elastic/x-pack-elasticsearch#2207)
Validating job groups during parsing results into
the validation error being wrapped into a parse
exception. The UI then does not display the cause of the
error. Finally, it is conceptually not a parse error, so
it belongs outside the parsing phase.

Original commit: elastic/x-pack-elasticsearch@a03f002bdc
2017-08-08 15:59:04 +01:00
David Roberts 6a159d2127 [ML] Fix fallout from bulk action requiring newlines (elastic/x-pack-elasticsearch#2205)
Only unit tests were broken.  Production ML code was always terminating
bulk requests with newlines.

Original commit: elastic/x-pack-elasticsearch@96ed06fed3
2017-08-08 11:07:13 +01:00
Alexander Reelsen 55e88d6857 Watcher: Ignore if template is missing when upgrade is running (elastic/x-pack-elasticsearch#2199)
If one of the old watcher templates does not exist when we try 
to delete it, the upgrade should just continue.

Original commit: elastic/x-pack-elasticsearch@6a52bad329
2017-08-08 10:17:58 +02:00
Chris Earle 1d6f82dbe3 [Monitoring] Remove Dedicated IndicesStatsCollector (elastic/x-pack-elasticsearch#2192)
This removes the `IndicesStatsCollector` and, instead, it reuses the superset version of the call from the `IndexStatsCollector`.

On clusters with a large number of indices, this should actually help a good amount in reducing wasted calls and memory allocation without any difference in the output.

Original commit: elastic/x-pack-elasticsearch@93b09878e4
2017-08-07 13:00:41 -04:00
Tim Sullivan 7d86d75aa4 [Monitoring/Cluster Alerts] default admin email is snake_case (elastic/x-pack-elasticsearch#2177)
Original commit: elastic/x-pack-elasticsearch@d6129a0065
2017-08-07 09:15:14 -07:00
David Roberts 05cbe8dc0c [ML] Disallow creating a job against a closed results or state index (elastic/x-pack-elasticsearch#2186)
Previously if this was attempted you'd get an NPE (5.x) or hang (6.x).
Following this change you get an error message telling you what the
problem is.

relates elastic/x-pack-elasticsearch#2170

Original commit: elastic/x-pack-elasticsearch@ea12a9ff46
2017-08-07 08:53:12 +01:00
Chris Earle 87bc215b91 [ML] Use try-with-resources for InputStream (elastic/x-pack-elasticsearch#2182)
Original commit: elastic/x-pack-elasticsearch@88d5e73fec
2017-08-04 13:26:35 -04:00
Alexander Reelsen 36bad2079d Tests: Fix watcher test when aliases were used
Original commit: elastic/x-pack-elasticsearch@d1c38cb85a
2017-08-04 18:01:26 +02:00