1521 Commits

Author SHA1 Message Date
Tanguy Leroux
0299886388 [Tests] Use XPack Usage API to verify Monitoring exporters are disabled (elastic/x-pack-elasticsearch#2648)
This commit changes the MonitoringIt and XPackRestIT tests so that the
disableMonitoring() method now use the XPack Usage API in order to check
that the monitoring exporters are correctly disabled. It checks at the
beginning of the tests (all exporters must be disabled before running
the test) and also at the end of the test.

This commit also fixes a bug in MonitoringIT where the Bulk thread pool
active queue was wrongly extracted from the response's map, forcing the
test to always wait for 30sec.

relates elastic/x-pack-elasticsearch#2459

Original commit: elastic/x-pack-elasticsearch@2d349e840f
2017-10-12 09:36:44 +02:00
Costin Leau
2026198dd4 Add size for column tests (elastic/x-pack-elasticsearch#2685)
Add displaySize to columnInfo

Original commit: elastic/x-pack-elasticsearch@ed1d265e98
2017-10-12 00:03:41 +03:00
Tanguy Leroux
ea4bff1d43 [Monitoring] Align MonitoringBulkDoc serialization with 6.0 (elastic/x-pack-elasticsearch#2736)
The version used in serialization must be aligned with 6.0/6.x.

Original commit: elastic/x-pack-elasticsearch@db63b91bc6
2017-10-11 17:56:24 +02:00
Tim Vernum
a4f7db4f66 [Security] Improve error messages in setup-passwords (elastic/x-pack-elasticsearch#2724)
Provides more verbose messaging around errors and possible causes when the tool aborts.

This change is primarily focused on errors connecting to the Elasticsearch node when TLS is enabled on the HTTP connection.

Original commit: elastic/x-pack-elasticsearch@aa8f7c6143
2017-10-11 12:32:35 +10:00
Tim Vernum
bc038b323d [Security] Apply validation when parsing certgen input (elastic/x-pack-elasticsearch#2711)
When certgen configuration was read from an input file (`-in` option) validation errors were collected but never reported. Depending on the type of error this may have caused the tool to exit with an internal error (e.g. NPE).

Validation is now applied after parsing the file and if errors are found the tool exits.

Original commit: elastic/x-pack-elasticsearch@b2262ed1d7
2017-10-11 12:30:19 +10:00
David Roberts
5d0388ccb3 [TEST] Fix ML node attribute test
When ML is disabled the attribute checking is stricter, but the test
did not reflect this

Original commit: elastic/x-pack-elasticsearch@613e97c595
2017-10-10 16:22:03 +01:00
David Roberts
ab9cc25a8e [ML] Prevent ML node attributes being set directly (elastic/x-pack-elasticsearch#2725)
ML uses node attributes to ensure that the master node knows how many
ML jobs may be allocated to each node.  This change prevents a user
messing up the way these attributes are used by setting them differently
using node.attr.* entries in their elasticsearch.yml.

This covers the "very short term" change outlined in elastic/x-pack-elasticsearch#2649

Original commit: elastic/x-pack-elasticsearch@9c381801d9
2017-10-10 15:12:59 +01:00
Dimitris Athanasiou
5eea355b33 [ML] Add overall buckets api (elastic/x-pack-elasticsearch#2713)
Adds the GET overall_buckets API.

The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets

The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.

relates elastic/x-pack-elasticsearch#2693

Original commit: elastic/x-pack-elasticsearch@ba6061482d
2017-10-10 14:41:24 +01:00
Alexander Reelsen
80593fb23c Watcher: Add execution state to watch status (elastic/x-pack-elasticsearch#2699)
The execution state is kind of a global indicator if a watch has been
running successfully and is used by the watcher UI.

However this field is only stored in the watch history but not part of
the watch status, thus it is not available everywhere. In order to
simplify the watcher UI this commit also adds the field to the
watch status which is stored together with the watch.

It is stored under the `status.execution_state` field as `status.state`
is already taken. This is also reflects with the name of the java class.

The WatchStatus class does not contain serialization checks, as this is
intended to be backported to 6.x, where those checks will be added.

Once the backport is done, the old execution state field can be fully
deleted from the master branch in another commit (syncing with Kibana
folks required).

relates elastic/x-pack-elasticsearch#2385

* fix doc tests

Original commit: elastic/x-pack-elasticsearch@26e8f99571
2017-10-10 09:07:33 +02:00
Alexander Reelsen
cadfd03529 Watcher: Allow JIRA path to be custom chosen (elastic/x-pack-elasticsearch#2682)
The path of a JIRA endpoint used to be fixed. This commit allows the
path to be dynamic, so that users can deploy their JIRA instance under
an arbitrary prefix.

Original commit: elastic/x-pack-elasticsearch@7702505114
2017-10-10 08:55:28 +02:00
Chris Earle
69ab7797be [Monitoring] Cleaner Service should be able to cleanup .watcher-history* (elastic/x-pack-elasticsearch#2696)
This adds a dynamic setting, which defaults to `false` currently, that can be used to delete all `.watcher-history*` indices that match the same age requirements as Monitoring indices.

Original commit: elastic/x-pack-elasticsearch@8ca3bdbca3
2017-10-09 15:46:07 -06:00
Igor Motov
79d6b88763 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@7503023447
2017-10-09 15:47:35 -04:00
Albert Zaharovits
98347088f9 Fix LDAP Authc connections deadlock (elastic/x-pack-elasticsearch#2587)
Do not execute bind on on the LDAP reader thread

Each LDAP connection has a single associated thread, executing the handlers for async requests; this is managed by the LDAP library. The bind operation is blocking for the connection. It is a deadlock to call bind, if on the LDAP reader thread for the same connection, because waiting for the bind response blocks the thread processing responses (for this connection).
This will execute the bind operation (and the subsequent runnable) on a thread pool after checking for the conflict above.

Closes: elastic/x-pack-elasticsearch#2570, elastic/x-pack-elasticsearch#2620

Original commit: elastic/x-pack-elasticsearch@404a3d8737
2017-10-09 13:06:12 +03:00
Simon Willnauer
cd14f33ae2 Return List instead of an array from settings (elastic/x-pack-elasticsearch#2694)
XPack side of elastic/elasticsearch#26903

Original commit: elastic/x-pack-elasticsearch@f0390974ab
2017-10-09 09:52:34 +02:00
Nik Everett
5806b620c5 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@141332d3fc
2017-10-06 13:57:55 -04:00
Chris Earle
f5561006f5 [Monitoring] Fix HttpExporterIT.testHostChangeReChecksTemplate (elastic/x-pack-elasticsearch#2703)
This changes the test to use the correct version (the Watch version).

Original commit: elastic/x-pack-elasticsearch@197ec3869b
2017-10-06 10:11:23 -06:00
David Roberts
dc1ed43ac1 Bump monitoring cluster alert versions (elastic/x-pack-elasticsearch#2673)
If the monitoring cluster alert versions are too far behind the
monitoring index template versions then it causes tests in HttpExporterIT
to fail.  This change increases the versions for the cluster alerts to
7.0.0-alpha1 to match the increase in index template version from elastic/x-pack-elasticsearch#2614.

relates elastic/x-pack-elasticsearch#2671

Original commit: elastic/x-pack-elasticsearch@b3cc3c03fe
2017-10-06 16:06:25 +01:00
Chris Earle
22c804ed24 [Monitoring] Fix HttpExporterIT (elastic/x-pack-elasticsearch#2702)
Uses the appropriate overload of `generateRandomStringArray` to disallow empty arrays from being returned.

Original commit: elastic/x-pack-elasticsearch@2596653ca1
2017-10-06 08:59:44 -06:00
Jay Modi
f73d0c7a07 Add transport ssl enabled value back to security usage (elastic/x-pack-elasticsearch#2695)
Since the transport ssl enabled setting is usable in 6.x again, this change adds back the value to
the xpack security usage so that it can be included in phone home data.

Original commit: elastic/x-pack-elasticsearch@52f6176df0
2017-10-06 08:43:32 -06:00
Tim Vernum
ec5a038f98 [Security] Support "type" field in role-mappings (elastic/x-pack-elasticsearch#2681)
The upgrade API adds a "type" field to role mapping documents.
The parser would reject these docs due to an unexpected field. We now ignore the "type" field instead.

Original commit: elastic/x-pack-elasticsearch@538f5adab2
2017-10-06 13:50:55 +11:00
Costin Leau
31a952993a Merge remote-tracking branch 'remotes/upstream/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@2ac0dab27b
2017-10-05 19:16:36 +03:00
Dimitris Athanasiou
686eb0ab65 [ML] Refactor [Bucket|Record]QueryBuilder classes (elastic/x-pack-elasticsearch#2684)
Those classes used to be elasticsearch-agnostic wrappers
of the query parameters. However, we now do not need that
layer of abstraction. Instead we can make those builders
own the building of the SearchSourceBuilder, which
simplifies the JobProvider and makes them reusable.

Original commit: elastic/x-pack-elasticsearch@b079cce1d6
2017-10-05 11:47:18 +01:00
Simon Willnauer
acba5a3c87 Lists are now represented as actual lists in Settings
Relates to elastic/elasticsearch#26878

Original commit: elastic/x-pack-elasticsearch@de6cfe26ed
2017-10-05 09:26:07 +02:00
Nhat
0c48f2c313 test: do not use deprecated shard preferences (elastic/x-pack-elasticsearch#2630)
This commit makes sure that we won't use the deprecated shard
preferences.

Relates elastic/elasticsearch#26335

Original commit: elastic/x-pack-elasticsearch@273e968407
2017-10-04 07:49:39 -04:00
David Roberts
d0e3b8f524 [TEST] Mute failing test suite: HttpExporterIT
See https://github.com/elastic/x-pack-elasticsearch/issues/2671

Original commit: elastic/x-pack-elasticsearch@3f63d00057
2017-10-04 10:33:23 +01:00
Tim Vernum
b228ad0511 [Security] Cache action privilege testing for bulk items (elastic/x-pack-elasticsearch#2526)
Since we are authorising on a single shard of a single index,
and there are only 3 possible actions that an item might represent,
we can test which items are authorised with a maximum of 3 permission
evaluations, regardless of how many items are actually in the shard
request. Previously we would test them all independently which had
a much higher overhead for large bulk requests.

Relates: elastic/x-pack-elasticsearch#2369 

Original commit: elastic/x-pack-elasticsearch@aceacf0aa3
2017-10-04 18:46:37 +11:00
Tim Vernum
8980357a29 [Security] Handle no-content gracefully (elastic/x-pack-elasticsearch#2610)
A number of REST requests require a body but did not explicitly validate for it.
This would typically cause a NPE if they were called with no body.

Original commit: elastic/x-pack-elasticsearch@863ac89429
2017-10-04 18:45:40 +11:00
Simon Willnauer
f5864c7291 Move away from Settings#getAsMap() (elastic/x-pack-elasticsearch#2661)
Relates to elastic/elasticsearch#26845

Original commit: elastic/x-pack-elasticsearch@0323ea07a5
2017-10-04 01:21:59 -06:00
Alexander Reelsen
8268cecb80 Tests: Replace script with search transform to remove plugin in test code (elastic/x-pack-elasticsearch#2470)
The test also used the timewarp trigger for watches to be executed, but it is sufficient to just call the execute watch API to make this test faster.

Original commit: elastic/x-pack-elasticsearch@3a4165f72c
2017-10-04 09:12:13 +02:00
David Roberts
0be7255029 [ML] Allow dynamic updates to the xpack.ml.max_model_memory_limit setting (elastic/x-pack-elasticsearch#2503)
A key limitation with this is that the updated setting only applies when
jobs are created or updated.  It does NOT automatically restrict existing
jobs.

relates elastic/x-pack-elasticsearch#2462

Original commit: elastic/x-pack-elasticsearch@73bd08db3f
2017-10-03 14:22:46 +01:00
David Roberts
c7b4ef9a89 Add cgroup memory usage/limit to OS stats on Linux (elastic/x-pack-elasticsearch#2614)
This change adapts the monitoring tests to account for an addition to
the OS stats made in elastic/elasticsearch#26166.

Original commit: elastic/x-pack-elasticsearch@9e36764857
2017-10-03 12:09:59 +01:00
Alexander Reelsen
3e0644b891 Tests: Fix test assertion in execution service test
No need for exact duration value check, just make sure it is
not zero based.

relates elastic/x-pack-elasticsearch#2525

Original commit: elastic/x-pack-elasticsearch@1fe3a0bf5a
2017-10-02 14:33:10 +02:00
David Roberts
90b2b74e76 [ML] Tolerate a body without timestamp for get_buckets with a timestamp (elastic/x-pack-elasticsearch#2640)
When getting a single bucket, the get_buckets API can take a timestamp
either in the body or in the URL.  Prior to this change, if a timestamp
was specified in the URL but a body not containing a timestamp was specified
(either empty or containing other parameters like exclude_interim or sort)
then it would cause a bad_request exception.  This in turn causes problems
for clients that cannot send a body when GETting and always send a body when
POSTing.

This change fixes get_buckets to always read any timestamp in the URL, even
when a body is sent.

relates elastic/x-pack-elasticsearch#2515

Original commit: elastic/x-pack-elasticsearch@5c23dd972e
2017-09-29 09:17:36 +01:00
David Roberts
0a89abdd7b [ML] snapshot_id is required when reverting a model snapshot (elastic/x-pack-elasticsearch#2641)
Previously the API spec did not say this.

Original commit: elastic/x-pack-elasticsearch@eaf214411d
2017-09-28 09:24:50 +01:00
Igor Motov
fdd98f01ed Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@273e0a110e
2017-09-26 11:59:49 -04:00
Dimitris Athanasiou
b3ae022985 [ML] Remove unused member in NativeAutodetectProcessFactory (elastic/x-pack-elasticsearch#2629)
Original commit: elastic/x-pack-elasticsearch@31f265f0e5
2017-09-26 14:12:27 +01:00
Dimitrios Athanasiou
5ad170a7ed Fix compilation due to upstream change
Original commit: elastic/x-pack-elasticsearch@57d834841d
2017-09-26 12:11:53 +01:00
Dimitris Athanasiou
c46e09902d [ML] Fix close job when the process has not launched yet (elastic/x-pack-elasticsearch#2616)
If a job close is requested after a job was opened but before
its process was launched, the job close returns successfully
without doing anything. The result is that the process hangs
around. This has been causing test failures as documented
int elastic/x-pack-elasticsearch#2360 and elastic/x-pack-elasticsearch#1270.

This commit fixes this problem by refactoring the
AutodetectProcessManager. It introduces a state pattern
to make clear the states of the process and it uses locking
to ensure a close waits for the job process to be created.

relates elastic/x-pack-elasticsearch#1270

Original commit: elastic/x-pack-elasticsearch@ff858bd136
2017-09-26 11:46:01 +01:00
Tanguy Leroux
a3984f7baa [Monitoring] Remove MonitoringSettings (elastic/x-pack-elasticsearch#2596)
Original commit: elastic/x-pack-elasticsearch@28dea7b699
2017-09-26 09:37:59 +02:00
Nik Everett
e6cd81458b Fix KnownActionTests after removing jdbc transport action
Original commit: elastic/x-pack-elasticsearch@2cb7cdd122
2017-09-25 18:12:18 -04:00
jaymode
7b8d92a2a7 Test: fix AuthenticationService tests timeouts due to incorrect stream sizes
The AuthenticationService#testInvalidToken would cause a suite timeout in the case of an exception
due to a incorrect stream size as the latch was never counted down. This fixes the missing latch
countdown.

relates elastic/x-pack-elasticsearch#2615

Original commit: elastic/x-pack-elasticsearch@e838e6e912
2017-09-25 13:02:22 -06:00
Nik Everett
c7c79bc1c0 Add scrolling support to jdbc (elastic/x-pack-elasticsearch#2524)
* Switch `ResultSet#getFetchSize` from returning the *requested*
fetch size to returning the size of the current page of results.
For scrolling searches without parent/child this'll match the
requested fetch size but for other things it won't. The nice thing
about this is that it lets us tell users a thing to look at if
they are wondering why they are using a bunch of memory.
* Remove all the entire JDBC action and implement it on the REST
layer instead.
* Create some code that can be shared between the cli and jdbc
actions to properly handle errors and request deserialization so
there isn't as much copy and paste between the two. This helps
because it calls out explicitly the places where they are different.
  * I have not moved the CLI REST handle to shared code because
I think it'd be clearer to make that change in a followup PR.
* There is now no more need for constructs that allow iterating
all of the results in the same request. I have not removed these
because I feel that that cleanup is big enough to deserve its own
PR.

Original commit: elastic/x-pack-elasticsearch@3b12afd11c
2017-09-25 14:41:46 -04:00
Tim Brooks
6099660643 Update LicenseMetadata version check to be 6.1
This is related to elastic/x-pack-elasticsearch#1941. It modifies to the version check in
LicenseMetadata because this elastic/x-pack-elasticsearch#2419 has been backported to 6.x.

Original commit: elastic/x-pack-elasticsearch@052358b550
2017-09-25 09:42:37 -06:00
Tanguy Leroux
9bd2da86db [Tests] Reenable MonitoringIT
Checking the size of the map doesn't make sense when each filter is
checked independently right after.

Original commit: elastic/x-pack-elasticsearch@58e5d3401d
2017-09-25 16:01:18 +02:00
Nik Everett
216058035b Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b3f6d3fd70
2017-09-25 09:55:17 -04:00
Simon Willnauer
e7b5702f50 Adopt settings cleanups from core (elastic/x-pack-elasticsearch#2605)
Relates to elastic/elasticsearch#26739

Original commit: elastic/x-pack-elasticsearch@dd13d099de
2017-09-25 12:27:07 +02:00
Tanguy Leroux
1bb9c4fe71 [Tests] Mute MonitoringIT
see https://github.com/elastic/x-pack-elasticsearch/issues/2609

Original commit: elastic/x-pack-elasticsearch@e2e2393e62
2017-09-25 09:14:50 +02:00
Martijn van Groningen
e93e7cf816 Revert "reset me"
This reverts commit elastic/x-pack-elasticsearch@3d44c14cd9.

Original commit: elastic/x-pack-elasticsearch@00203223b4
2017-09-25 08:56:01 +02:00
Martijn van Groningen
ad3eb997c6 reset me
Original commit: elastic/x-pack-elasticsearch@3d44c14cd9
2017-09-25 08:52:51 +02:00
Jason Tedor
b4f7d56c35 Whitelist global checkpoint sync actions
This commit whitelists the global checkpoint sync actions as otherwise
these actions do not have privileges to run as the system user and will
be denied as unauthorized.

Relates elastic/x-pack-elasticsearch#2604

Original commit: elastic/x-pack-elasticsearch@598ae1ff50
2017-09-22 16:27:51 -04:00