Commit Graph

6555 Commits

Author SHA1 Message Date
Tanguy Leroux 34bd72e69b [Monitoring] Remove monitoring resolvers (elastic/x-pack-elasticsearch#2566)
Now all monitoring documents are indexed in timestamped indices dedicated 
to each product, the old monitoring's resolvers and all the associated plumbing 
can be removed from the Elasticsearch monitoring plugin.

This pull request merges the feature branch feature/monitoring-summer-cleanup 
into master. All commits have been independently reviewed as part of elastic/x-pack-elasticsearch#2226:

* [Monitoring] Clean up monitoring doc (elastic/x-pack-elasticsearch#2208)
* [Monitoring] Remove ClusterStatsResolver (elastic/x-pack-elasticsearch#2209)
* [Monitoring] Remove IndexRecoveryResolver (elastic/x-pack-elasticsearch#2229)
* [Monitoring] Remove JobStatsResolver (elastic/x-pack-elasticsearch#2230)
* [Monitoring] Remove ShardsResolver (elastic/x-pack-elasticsearch#2233)
* [Monitoring] Fix missing instanceof in exporter bulks
* [Monitoring] Remove NodeStatsResolver (elastic/x-pack-elasticsearch#2244)
* [Monitoring] Remove IndexStatsResolver & IndicesStatsResolver  (elastic/x-pack-elasticsearch#2247)
* [Monitoring] Remove source node attributes and add timestamp (elastic/x-pack-elasticsearch#2285)
* [Monitoring] Remove last resolver (elastic/x-pack-elasticsearch#2330)
* [Monitoring] Make MonitoringDoc an abstract class (elastic/x-pack-elasticsearch#2451)
* [Monitoring] Fix TransportMonitoringBulkActionTests
* [Monitoring] Remove old resolver tests and add a single integration test (elastic/x-pack-elasticsearch#2453)

It does not introduce any new feature, so it could also go in 6.0.

relates elastic/x-pack-elasticsearch#2226


Original commit: elastic/x-pack-elasticsearch@e3d5cec28c
2017-09-21 09:36:03 +02:00
Tim Sullivan 723d894753 [Monitoring/Cluster Alerts] Fix the email message when cluster license expiration is resolved (elastic/x-pack-elasticsearch#2557)
* [Monitoring/Cluster Alerts] Fix the email message for cluster license expiration resolved

* fix making payload.message show only when new

Original commit: elastic/x-pack-elasticsearch@6d54b02913
2017-09-20 15:44:14 -07:00
Igor Motov 4964239f4c Tests: Remove AwaitsFix from TemplateUpgraderTests
I believe this problem should have been already fixed by elastic/elasticsearch#26698.

Original commit: elastic/x-pack-elasticsearch@0ec93f2803
2017-09-20 14:12:24 -04:00
Dimitrios Athanasiou a5e8589b38 [ML][TEST] Add integ test for elastic/x-pack-elasticsearch#2519
Relates elastic/x-pack-elasticsearch#2519

Original commit: elastic/x-pack-elasticsearch@91b38a62bb
2017-09-20 18:59:34 +01:00
lcawley 57f987b3b2 [DOCS] Added index-shared4 and index-shared5.asciidoc
Original commit: elastic/x-pack-elasticsearch@ef407e2677
2017-09-20 10:53:40 -07:00
lcawley 0ec98e0190 [DOCS] Fixed broken Kibana link
Original commit: elastic/x-pack-elasticsearch@10db543680
2017-09-20 09:30:02 -07:00
Lisa Cawley 8f1984a86e [DOCS] Enable read-only access for kibana_system user (elastic/x-pack-elasticsearch#2465)
Original commit: elastic/x-pack-elasticsearch@a262acb1b0
2017-09-20 09:25:58 -07:00
Dimitris Athanasiou fad98d784f [ML] Align aggregated data extraction to histogram interval (elastic/x-pack-elasticsearch#2553)
When the datafeed uses aggregations and in order to accommodate
derivatives, an extra bucket is queried at the beginning of each
search. In order to avoid visiting the same bucket twice, we need
to search buckets aligned to the histogram interval. This allows
us to steer away from partial buckets, and thus avoid the problem
of dropping or duplicating data.

relates elastic/x-pack-elasticsearch#2519

Original commit: elastic/x-pack-elasticsearch@e03dde5fea
2017-09-20 16:45:06 +01:00
Hendrik Muhs 3c517902f2 re-enable categorization IT (elastic/x-pack-elasticsearch#2534)
re-enables categorization IT after fixing https://github.com/elastic/machine-learning-cpp/issues/279 upstream

Original commit: elastic/x-pack-elasticsearch@398a668125
2017-09-20 11:39:37 +02:00
Yannick Welsch 8648153f0e Deguice ActionFilter (elastic/x-pack-elasticsearch#2533)
Companion PR to elastic/elasticsearch#26691

Original commit: elastic/x-pack-elasticsearch@3fceb54809
2017-09-20 10:30:23 +02:00
Tal Levy 8b1021ccad blacklist two license rest tests when build.snapshot=false (elastic/x-pack-elasticsearch#2559)
There are two rest tests that rely on hardcoded license
signatures that use the dev public key. These tests fail
when tests are run with build.snapshot=false. This Commit
blacklists these two tests in that scenario

relates elastic/x-pack-elasticsearch#2527.

Original commit: elastic/x-pack-elasticsearch@7581e8d699
2017-09-19 16:56:11 -07:00
lcawley af2b2ca542 [DOCS] Add random default datafeed query delay
Original commit: elastic/x-pack-elasticsearch@d886f09192
2017-09-19 09:54:11 -07:00
Nik Everett ea66433899 Mark periodically failing test as AwaitsFix
This should reduce our build failure emails. The failure is already
tracked at https://github.com/elastic/x-pack-elasticsearch/issues/2421.
It isn't actively being worked but will be soon.

Original commit: elastic/x-pack-elasticsearch@6baf55de42
2017-09-19 09:59:26 -04:00
David Roberts 67055877a5 [ML] Avoid creating spurious 0 "actual" values for model plot documents (elastic/x-pack-elasticsearch#2535)
Some model plot documents should not have an "actual" value, for example
when no input events were seen for a meean/min/max detector in a particular
bucket.  Prior to this change we would set the "actual" value to 0 for such
model plot documents.  Following this change no "actual" value will be
present in these documents.

Only newly created model plot documents are affected.  Model plot documents
that were incorrectly written in the past will remain wrong forever.

relates elastic/x-pack-elasticsearch#2528

Original commit: elastic/x-pack-elasticsearch@47a7365f59
2017-09-19 10:30:43 +01:00
Tanguy Leroux 0aef18333f Add packaging test for bootstrap password setup (elastic/x-pack-elasticsearch#2509)
relates elastic/x-pack-elasticsearch#2388

Original commit: elastic/x-pack-elasticsearch@cc750155d0
2017-09-19 10:07:39 +02:00
Tim Vernum aec2308228 Allow AD realm to perform 'run-as' lookups (elastic/x-pack-elasticsearch#2531)
- Marks the AD Session factory as supporting "lookup" (Refer: elastic/x-pack-elasticsearch@40b07b3)
- Adds "pool.enabled" as a registered setting on AD realm (Refer: elastic/x-pack-elasticsearch@40b07b3)
- Fixes LDAP user lookup that has been broken since 6.x (Refer: elastic/x-pack-elasticsearch@f796949)

Original commit: elastic/x-pack-elasticsearch@62ff6129a1
2017-09-19 14:50:26 +10:00
Lisa Cawley 4ffaec5173 [DOCS] Remove redundant certgen info (elastic/x-pack-elasticsearch#2542)
Original commit: elastic/x-pack-elasticsearch@6147e32fd1
2017-09-18 14:22:34 -07:00
Lisa Cawley 679ef6a744 [DOCS] Added _xpack_security internal user (elastic/x-pack-elasticsearch#2541)
Original commit: elastic/x-pack-elasticsearch@d1c87af335
2017-09-18 13:32:11 -07:00
Tal Levy 5a090c14c1 convert more request objects to writeable (elastic/x-pack-elasticsearch#2457)
* convert more to writeable

* migrate streamable tests to writeable tests

Original commit: elastic/x-pack-elasticsearch@56794e5760
2017-09-18 13:20:02 -07:00
Suyog Rao 3a9aad5ece [Logstash] Remove version field from config mgmt
relates elastic/x-pack-elasticsearch#2405

Original commit: elastic/x-pack-elasticsearch@5bfd1b7a6d
2017-09-18 12:35:21 -07:00
lcawley 3ff175b541 [DOCS] Remove old certgen examples
Original commit: elastic/x-pack-elasticsearch@f97d0ab00c
2017-09-18 12:22:45 -07:00
Lisa Cawley f71dc549a5 [DOCS] Update X-Pack installation re TLS changes (elastic/x-pack-elasticsearch#2523)
* [DOCS] Update xpack installation re TLS changes

* [DOCS] Clarify multi-node TLS requirement

Original commit: elastic/x-pack-elasticsearch@a961488fa1
2017-09-18 11:20:18 -07:00
Simon Willnauer 47214426e9 Use InputStreamStreamInputs validation to limit size of tokens (elastic/x-pack-elasticsearch#2537)
Relates to elastic/elasticsearch#26692
relates elastic/x-pack-elasticsearch#2493

Original commit: elastic/x-pack-elasticsearch@8e23868743
2017-09-18 19:25:08 +02:00
Lisa Cawley 371953488b [DOCS] Create certgen command reference (elastic/x-pack-elasticsearch#2456)
* [DOCS] Create certgen command reference

* [DOCS] Added missing certgen parameters

* [DOCS] Finalized certgen command reference

Original commit: elastic/x-pack-elasticsearch@6d0b795eb4
2017-09-18 08:36:13 -07:00
Dimitris Athanasiou 9b25d0edf7 [ML] Correctly ignore extra aggregation buckets (elastic/x-pack-elasticsearch#2530)
The problem here is that the code was ignoring buckets
whose start time was before the start time of the extractor.
However, this is not a good enough condition. For example,
when there are no data in the bucket extra bucket that is
being queried, the first bucket will be the one containing
the start time.

This commit fixes the issue by changing the condition to
ignore buckets before the first bucket that includes the
start time of the extraction.

relates elastic/x-pack-elasticsearch#2519 

Original commit: elastic/x-pack-elasticsearch@15c7d2655f
2017-09-18 12:10:19 +01:00
Tim Vernum dea82a07a2 Infer KeyStore type from pathname (elastic/x-pack-elasticsearch#2514)
If the keystore type is not explicitly specified, infer it from the filename.
Treats .p12, .pfx and .pkcs12 as being PKCS12, all others as jks.

This will allow certgen to produce PKCSelastic/x-pack-elasticsearch#12 files by default and make it easy to use them as x-pack keystores

Original commit: elastic/x-pack-elasticsearch@fc361f0d87
2017-09-18 14:21:19 +10:00
Chris Earle 24c2c62ca2 [Monitoring] Ignore .marvel* indices (elastic/x-pack-elasticsearch#2520)
Beginning with 7.0, the cleaner service will no longer automatically cleanup .marvel indices regardless of their age.

Original commit: elastic/x-pack-elasticsearch@5b90e6f62a
2017-09-15 13:35:45 -07:00
jaymode 19de38665e Docs: remove incorrect name attribute from role snippet
relates elastic/x-pack-elasticsearch#2497

Original commit: elastic/x-pack-elasticsearch@ef15a1e36c
2017-09-15 12:51:37 -06:00
Tim Brooks b3914afd30 Reenable TribeWithSecurityIT tests (elastic/x-pack-elasticsearch#2511)
This is related to elastic/x-pack-elasticsearch#1996. These tests were disabled during the bootstrap
password work. They can now be reenabled. Additionally, I made the test
password used in tests consistent.

Original commit: elastic/x-pack-elasticsearch@5b490c8231
2017-09-15 12:50:54 -06:00
Simon Willnauer 96e01dce47 Only require TLS for standard/gold/platinum licenses elastic/x-pack-elasticsearch#2517
relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@5213cf24f1
2017-09-15 20:21:15 +02:00
Chris Earle 9c9da2e1e4 [Monitoring] Remove Legacy Monitoring Indices (elastic/x-pack-elasticsearch#2513)
This changes Monitoring's Cleaner Service to remove any legacy Monitoring index that is appropriately old.

This includes any `.marvel-*` index and also the "data" indices used by both Marvel and 5.0 - 5.4 versions of X-Pack monitoring, as well as the legacy alerts index.

Original commit: elastic/x-pack-elasticsearch@8d99f5518b
2017-09-15 08:46:10 -07:00
jaymode 84dd719ab9 remove outdated comment
Original commit: elastic/x-pack-elasticsearch@06a51abb65
2017-09-15 09:03:36 -06:00
jaymode 344603e40f update text in TLSLicenseBootstrapCheck
Original commit: elastic/x-pack-elasticsearch@4ee6827566
2017-09-15 08:56:34 -06:00
Jay Modi 53d6d945f0 Update documentation to reflect the latest TLS changes and licensing (elastic/x-pack-elasticsearch#2508)
This commit updates to documentation and adds notes about TLS being required to install a
license.

Relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@0d8bfb98ea
2017-09-15 08:44:03 -06:00
Simon Willnauer 4d20586b24 [TEST] add integration test that ensures we reject license upgrades if TLS is not enabled
Original commit: elastic/x-pack-elasticsearch@dfbadb5e5f
2017-09-15 14:47:28 +02:00
Simon Willnauer c3066d1a51 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@9ce33bc7c3
2017-09-15 09:51:16 +02:00
Simon Willnauer 023bdb72b2 Add common-analysis plugin to several xpack integ tests (elastic/x-pack-elasticsearch#2501)
Several tests miss the common-analysis plugin in the old-style integ
tests causing odd exceptions in the test logs. This adds the missing plugin reference.

relates elastic/x-pack-elasticsearch#2363

Original commit: elastic/x-pack-elasticsearch@e4e6735408
2017-09-15 09:25:36 +02:00
Lisa Cawley 895d28f462 [DOCS] Remove redundant users command info (elastic/x-pack-elasticsearch#2504)
Original commit: elastic/x-pack-elasticsearch@1c9fa91293
2017-09-14 15:47:21 -07:00
jaymode 8997792875 Test: use TLS for plugin integ tests
Original commit: elastic/x-pack-elasticsearch@99971d7256
2017-09-14 15:57:28 -06:00
lcawley c3b658df88 [DOCS] Fixed broken link to put watch API
Original commit: elastic/x-pack-elasticsearch@f36caaa371
2017-09-14 14:12:20 -07:00
Michael Basnight fa0b854fb6 Update rest-api-spec to use bad_request (elastic/x-pack-elasticsearch#2507)
ref #elastic/elasticsearch#26539

Original commit: elastic/x-pack-elasticsearch@8b79a0769a
2017-09-14 15:59:29 -05:00
Lisa Cawley df1e4e85a4 Format Watcher APIs (elastic/x-pack-elasticsearch#2382)
* [DOCS] Format Watcher APIs

* [DOCS] Removed master_timeout from Watcher APIs

* [DOCS] Added authority info to watcher APIs

Original commit: elastic/x-pack-elasticsearch@1e6de3b036
2017-09-14 13:01:47 -07:00
Andy Bristol 279c7e14fd [TEST] fix security template version check in rest tests (elastic/x-pack-elasticsearch#2506)
Since the template upgrade service was added, upgrades should
be performed by a node with the highest version in the cluster,
which may not be the master node.

Original commit: elastic/x-pack-elasticsearch@d66145de54
2017-09-14 12:16:20 -07:00
Jay Modi 57de66476c Disable TLS by default (elastic/x-pack-elasticsearch#2481)
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.

Relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@abc66ec67d
2017-09-14 12:18:54 -06:00
Simon Willnauer 1e14e14571 Prevent licenses to be upgraded to production unless TLS is configured (elastic/x-pack-elasticsearch#2502)
if a user tries to upgrade a license to a production license and has security
enabled we prevent the upgrade unless TLS is setup. This is a requirement now
if a cluster with security is running in prodcution.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@d61ef3bcb1
2017-09-14 20:14:27 +02:00
Jay Modi 8d8baffe24 Add specific client and user for security index access (elastic/x-pack-elasticsearch#2492)
This change removes security index access from the xpack user by creating its own specific role
and adds a xpack security user that maintains the superuser role so that it can perform all
operations necessary for security.

Original commit: elastic/x-pack-elasticsearch@ad906bc913
2017-09-14 08:09:14 -06:00
David Roberts 104a3a323f [BUILD] Make AWS error message more informative
Original commit: elastic/x-pack-elasticsearch@42cca7ed82
2017-09-14 14:46:48 +01:00
Simon Willnauer 91b57ee63f Add bootstrap check that enforces TLS if a production license is in the local clusterstate (elastic/x-pack-elasticsearch#2499)
This change will enforce transport SSL to be enforced if security is enabled and the
license in the clusterstate is a production license. The cluster state is loaded from
local storage such that we don't need to join a cluster to make these checks. Yet, the cluster
might have already got a different license if the node got disconnected while the license got
downgraded and then TLS got disabled. This corner case requires manual intervention which
we consider ok given the simplicity of this change.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@5765b7cd21
2017-09-14 13:52:53 +02:00
Hendrik Muhs 7d19264363 [ML-FC] Branch landing feature/ml (elastic/x-pack-elasticsearch#2500)
integrate forecasting feature branch into master

    - add endpoint xpack/ml/job/forecast to request forecasting on data of ml-jobs
       - current parameters: end time
    - persists forecast results into shared or own index
       - different runs are separated by a 'forecast id'

relates elastic/x-pack-elasticsearch#1838

Original commit: elastic/x-pack-elasticsearch@f9d701a6bc
2017-09-14 12:31:20 +02:00
Simon Willnauer 3b00251a96 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@4a36f0c2be
2017-09-14 07:43:19 +02:00